Class prototype-based sparse supervision point cloud object detection method
The method for sparse supervised point cloud object detection using class prototypes addresses the limitations of existing methods by propagating annotations across scenes and combining annotation types, achieving superior performance and accuracy with reduced manual effort.
Patent Information
- Application Number
- CN202411191747.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-28
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-08-28
AI Technical Summary
Existing sparse supervision methods for point cloud object detection are not suitable for indoor scenes due to the need for complex optimization modules and are not designed to handle scene priors, requiring extensive manual annotation.
A method for sparse supervised point cloud object detection based on class prototypes, involving a prototype-assisted object mining module and a multi-label collaborative optimization module, which utilizes class prototypes to propagate annotations across scenes and combine sparse, pseudo, and prototype annotations for improved generalization and accuracy.
The method achieves robust performance and high accuracy in indoor scenes with reduced annotation effort, surpassing state-of-the-art methods under sparse supervision and maintaining performance comparable to fully supervised methods with minimal annotations.
Smart Images

Figure CN119068257A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of embodied intelligence, and specifically, is a method for sparse supervised point cloud object detection based on class prototypes. Background Art
[0002] Point cloud object detection, as part of the perception of indoor scenes, provides more complete and accurate environmental perception data for embodied intelligent systems, thereby achieving safer and more reliable decisions. Traditional indoor point cloud object detection relies on a large number of precise annotations, which are time-consuming and labor-intensive to construct. In complex indoor scenes, such annotation settings require a lot of manual work because it requires constant rotation of the view to obtain accurate annotations. Therefore, there is an urgent need for a method that can achieve more accurate detection while reducing annotations.
[0003] Sparse supervision only requires a few objects to be labeled in each point cloud scene, greatly reducing the manual time required for labeling. Existing sparse supervision methods have only explored outdoor applications and are designed for outdoor scenes. CoIn (CoIn: Contrastive Instance Feature Mining for Outdoor 3D Object Detection with Very Limited Annotations. In ICCV, 2023.) takes advantage of the sparse characteristics of outdoor scenes and ensures that all categories of objects are displayed in one scene, including cars, people, and bicycles, by copying objects to blank areas. With such a strategy, it can mine all categories of objects from a single scene. However, such methods are not suitable for indoor scenes. Indoor scenes have scene priors. It is unreasonable to copy the bathtub in the bathroom scene to the living room scene. Therefore, outdoor methods cannot be used in indoor scenes. In addition, sparsely supervised models are often accompanied by optimization modules, which require the design of complex structures to ensure a balance between the precision and recall of pseudo-annotation. Summary of the invention
[0004] The purpose of the present invention is to provide a method for sparse supervised point cloud object detection based on class prototypes to solve the problems existing in the existing methods.
[0005] The technical solution to achieve the purpose of the present invention is: a method for sparse supervised point cloud object detection based on class prototypes, including two processes: a training phase and a testing phase.
[0006] The training phase consists of the following steps:
[0007] Step 1: Build a detector to obtain proposal features and detection results from the input point cloud scene;
[0008] Step 2: Construct a prototype-assisted object mining module, and use the labeled parts in the proposal features obtained in Step 1 to cluster the class prototypes of each class; calculate the similarity between the unlabeled proposals and each class prototype in Step 2, and propagate the prototype labels to the unlabeled parts according to the similarity.
[0009] Step 3: Calculate the loss function values of the sparsely labeled parts and the prototype-labeled parts, and train the initial detector.
[0010] Step 4: Construct a multi-label collaborative optimization module, load the parameters of the initial detector to obtain pseudo-labels and collaborate with the prototype labels in Step 3, and at the same time calculate the loss function values of the sparse labels, pseudo-labels and prototype labels, and train the detector again to obtain the final detector.
[0011] In the test phase, the point cloud scene to be detected is input into the trained final detector to obtain the detection result.
[0012] Furthermore, in Step 1 of the training phase, the original input point cloud is input into the detector, and the detector extracts the proposal feature F p ∈R N×d and the detection result, where the detection result is divided into a classification score C ∈ R N×c and a regression result B ∈ R N×b ; the detector consists of sparse convolutions, R represents real numbers, N represents the number of proposals, d is the feature dimension, c is the number of classifications, and b is the dimension of regression.
[0013] Furthermore, the construction of the prototype-assisted object mining module described in Step 2 of the training phase includes two parts: clustering of class prototypes and matching of prototype labels.
[0014] (1) Clustering of class prototypes: This part receives the proposal features in Step 1 and outputs the class prototype features P i ={P 1 , P 2 , …, P c} of c categories; the specific implementation process is as follows: First, according to the indication of the sparse label, extract the labeled part M << N of the proposal feature; then use the multi-step Sinkhorn-Knopp iteration algorithm to obtain the correspondence L with each prototype in c , where O is the number of prototypes, set to a number greater than 1; the correspondence here is obtained using the following formula:
[0015]
[0016] Where u and v are renormalization vectors obtained by multi-step Sinkhorn-Knopp iteration; k is the temperature coefficient used to adjust the smoothness of the distribution;
[0017] Finally, use the corresponding relationship to update P c ; Here the update method of the i-th prototype of category c is as follows:
[0018]
[0019] in is the characteristic of l2 regularization; μ is the coordination variable, N k Represents the number of sparsely labeled proposal features matched by the i-th prototype of category c;
[0020] (2) Prototype annotation matching: First, the proposal feature F obtained in step 1 of the training phase is p , filter out the unmarked areas And calculate its similarity with the class prototype Next, use A to calculate the propagation probability
[0021] W=C u ⊙A′
[0022] whereA′=argmax(A)i=1,…,O#(3)
[0023] Among them, ⊙ is the Hadamard product, W represents the probability that the proposal feature of the unlabeled area belongs to a certain category, and A′ uses the class prototype with the highest probability of each category to represent the current category. Represents the classification score of the unlabeled area; then based on W, the category label of each unlabeled area can be obtained. The process is as follows:
[0024] C f =argmax(W)i=1,…,C#(4)
[0025] Finally, by screening out the parts with classification scores lower than the set threshold and the parts beyond the detection range, the final prototype annotation, that is, the remaining category labels, is obtained.
[0026] Furthermore, in step 3 of the training phase, the initial detector is trained. After obtaining the prototype annotation, the loss function of the detector is as follows:
[0027] The entire scene is divided into annotated parts and unannotated parts. First, the annotated part still uses the annotated detection loss L det , including classification loss and regression loss; secondly, the unlabeled part uses the prototype annotation as its annotation to calculate the prototype loss L pro ; The total loss is Ldet +L pro ; Use the gradient descent method to update the network parameters of the initial detector until the model converges to obtain a trained initial detector.
[0028] Furthermore, in step 4 of the training phase, a multi-label collaborative optimization module is constructed. First, the initial detector trained in step 3 of the training phase is used to obtain pseudo labels. In this step, the calculation of the detection loss for pseudo labels is additionally used. pse ; The total loss is L det +L pro +L pse ; Use the gradient descent method to update the network parameters of the initial detector until the model converges to obtain the trained final detector.
[0029] Compared with the prior art, the present invention has the following significant advantages:
[0030] (1) The present invention constructs a prototype-assisted object mining module, breaks through the limitation of scenes, obtains class prototypes from the entire data and assigns prototype annotations, and realizes the propagation of annotations from annotated objects to class prototypes, and then from class prototypes to unannotated objects. Compared with previous methods, the present invention realizes cross-scene annotation propagation, utilizes the information of the entire data set, and obtains more robust performance.
[0031] (2) The present invention constructs a multi-annotation collaborative optimization module that collaboratively combines sparse annotation, pseudo-annotation, and prototype annotation. Compared with previous methods, the present invention implements a simpler and more efficient optimization module that can better utilize multiple annotation information and improve the generalization ability and accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Figure 1 It is an overall framework diagram of the method of the present invention, which includes a framework diagram of a prototype-assisted object mining module and a framework diagram of a multi-annotation collaborative optimization module.
[0033] Figure 2 is a schematic diagram of the prototype labels mined by the sparse annotation used in the present invention and the proposed prototype-assisted object mining module, Figure 2 (a) and Figure 2 (b) Sparse annotations and mined prototype labels of the same scene, respectively.
[0034] Figure 3 It is a comparison of the detection results of the method proposed in this invention and other methods under sparse annotation. DETAILED DESCRIPTION
[0035] The present invention discloses a method for sparse supervised point cloud object detection based on class prototypes. The method uses a prototype-assisted object mining module to break through the limitation of a single scene through cross-scene aggregated class prototypes, and realizes the mining of all categories in the scene; then a multi-annotation collaborative optimization module is used to simplify the design of the optimization module by combining sparse annotations, pseudo-annotations and prototype annotations, and at the same time achieve a balance between precision and recall rate. The method includes two processes: a training phase and a testing phase.
[0036] The specific steps of the training phase process are:
[0037] Step 1_1: Input the original input point cloud into the detector and use the detector to extract the proposal features F p ∈R N×d And the test results, where the test results are divided into classification scores C∈R N×c And the regression result B∈R N×b The detector consists of sparse convolutions. represents a real number, N represents the number of proposals, d is the feature dimension, c is the number of categories and b is the regression dimension.
[0038] Step 1_2: Construct a prototype-assisted object mining module, including two parts: clustering of class prototypes and prototype annotation matching.
[0039] (1) Clustering of class prototypes: This part receives the proposal features in step 1_1 Output c categories of prototype features P i = {P 1 ,P 2 ,…,P c The specific implementation process is as follows: Assuming the cth category as an example, first extract the part of the proposal feature annotation according to the sparse annotation instructions Then use multi-step Sinkhorn-Knopp iteration (Lightspeed Computation of Optimal Transport. In NeurIPS, 2013) to obtain and The corresponding relationship L of each prototype in c , where O>1 is to ensure the diversity of objects in category c. The correspondence here is obtained using the following formula:
[0040]
[0041] Where u and v are renormalized vectors obtained by multiple steps of Sinkhorn-Knopp iteration. k is 0.05.
[0042] Finally, use the corresponding relationship to update Pc Here, the update method of the i-th prototype of category c is as follows:
[0043]
[0044] in is the feature of l2 regularization. μ is 0.9, N k Represents the number of sparsely labeled proposal features matched by the i-th prototype of category c.
[0045] (2) Prototype annotation matching: First, for the proposal feature F obtained in step 1_1 p , filter out the unmarked areas And calculate the similarity between it and the class prototype of each category obtained in step 1-2 (1) Next, use A to calculate the propagation probability
[0046] W=C u ⊙A′
[0047] whereA′=argmax(A)i=1,…,O#(3)
[0048] Where ⊙ is the Hadamard product. W represents the probability that the proposal feature of the unlabeled region belongs to a certain category. A′ uses the class prototype with the highest probability of each category to represent the current category. Represents the classification score of the unlabeled area. Then, based on W, the category label of each unlabeled area can be obtained. Because this part of the label is obtained from the prototype, it is called prototype labeling in the present invention. The process is as follows:
[0049] C f =argmax(W)i=1,…,C#(4)
[0050] Finally, the final prototype annotation is obtained by screening out the parts with low classification scores and the parts beyond the detection range.
[0051] Step 1_3: Train the initial detector. After obtaining the prototype annotation, the loss function of the detector is as follows: In this step, the present invention divides the entire scene into an annotated part and an unannotated part. First, the annotated part still uses the annotated detection loss L det , including classification loss and regression loss. Secondly, the unlabeled part uses the prototype annotation as its annotation to calculate the prototype loss L pro , because there is only category loss here, so only the classification loss is calculated. The total loss is L det +L pro Using the gradient descent method, the network parameters of the initial detector are updated until the model converges to obtain a trained initial detector.
[0052] Step 1_4: Construct a multi-label collaborative optimization module. First, use the initial detector trained in step 1_3 to obtain pseudo labels. The threshold set here is 0.3 to ensure the accuracy of pseudo labels. In order to solve the problem of low recall rate of pseudo labels, the present invention re-introduces the prototype annotation proposed in step 1_2 in this step. Similar to step 1_3, the difference in this step is that the calculation of the detection loss of pseudo labels is added. pse The total loss is changed to L det +L pro +L pse Using the gradient descent method, the network parameters of the initial detector are updated until the model converges to obtain the trained final detector.
[0053] The specific steps of the test phase process are to input the point cloud scene to be detected into the trained final detector to obtain the detection result.
[0054] The present invention is further described below in conjunction with the accompanying drawings and embodiments of the specification. The described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0055] The present invention is described in detail below with reference to the accompanying drawings. The overall framework of the method is as follows: Figure 1 As shown in FIG. 1 , a method for sparse supervised point cloud object detection based on class prototypes includes two processes: a training phase and a testing phase. The specific steps of the training phase are:
[0056] Step 1_1: Input the original input point cloud into the detector and use the detector to extract proposal features and the test results, where the test results are divided into classification scores And the regression results The detector consists of sparse convolutions. represents a real number, N represents the number of proposals, d is the feature dimension, c is the number of categories and b is the regression dimension. In the method of the present invention, N is dynamically changing and has no fixed value. d is set to 128, c is set to 18 and b is set to 6.
[0057] Step 1_2: Build a prototype auxiliary object mining module. The module structure is as follows Figure 1 As shown in Figure 2, it includes two parts: clustering of class prototypes and prototype annotation matching.
[0058] (1) Clustering of class prototypes: This part receives the proposal features in step 1_1 Output c categories of prototype features P i = {P 1 ,P2 ,…,P c The specific implementation process is as follows: Assuming the cth category as an example, first extract the part of the proposal feature annotation according to the sparse annotation instructions Then use multi-step Sinkhorn-Knopp iteration (Lightspeed Computation of Optimal Transport. In NeurIPS, 2013) to obtain and The corresponding relationship L of each prototype in c , where O>1 is to ensure the diversity of objects in category c. The correspondence here is obtained using the following formula:
[0059]
[0060] Where u and v are renormalized vectors obtained by multiple steps of Sinkhorn-Knopp iteration. k is 0.05.
[0061] Finally, use the corresponding relationship to update P c Here, the update method of the i-th prototype of category c is as follows:
[0062]
[0063] in is the feature of l2 regularization. μ is 0.9, N k Represents the number of sparsely labeled proposal features matched by the i-th prototype of category c.
[0064] (2) Prototype annotation matching: First, for the proposal feature F obtained in step 1_1 p , filter out the unmarked areas And calculate the similarity between it and the class prototype of each category obtained in step 1-2 (1) Next, use A to calculate the propagation probability
[0065] W=C u ⊙A′
[0066] whereA′=argmax(A)i=1,…,O#(3)
[0067] Where ⊙ is the Hadamard product. W represents the probability that the proposal feature of the unlabeled region belongs to a certain category. A′ uses the class prototype with the highest probability of each category to represent the current category. Represents the classification score of the unlabeled area. Then, based on W, the category label of each unlabeled area can be obtained. Because this part of the label is obtained from the prototype, it is called prototype labeling in the present invention. The process is as follows:
[0068] C f =argmax(W),i=1,…,C#(4)
[0069] Finally, the final prototype annotation is obtained by filtering out the parts with low classification scores and the parts beyond the detection range. Figure 2 As shown in (b), it can be seen that the present invention detects the categories of doors, trash cans, etc. when only the sofa is marked in the scene, which proves the effectiveness of mining class prototypes from the entire data set and using them to mine labels.
[0070] Step 1_3: Train the initial detector. After obtaining the prototype annotation, the loss function of the detector is as follows: In this step, the present invention divides the entire scene into an annotated part and an unannotated part. First, the annotated part still uses the annotated detection loss L det , including classification loss and regression loss. Secondly, the unlabeled part uses the prototype annotation as its annotation to calculate the prototype loss L pro , because there is only category loss here, so only the classification loss is calculated. The total loss is L det +L pro Using the gradient descent method, the network parameters of the initial detector are updated until the model converges to obtain a trained initial detector.
[0071] Step 1_4: Construct a multi-label collaborative optimization module with the following structure: Figure 1 As shown. First, the initial detector trained in step 1_3 is used to obtain pseudo labels. The threshold set here is 0.3 to ensure the accuracy of pseudo labels. In order to solve the problem of low recall rate of pseudo labels, the present invention re-introduces the prototype annotation proposed in step 1_2 in this step. Similar to step 1_3, the difference in this step is that the calculation of the detection loss of pseudo labels is added. pse The total loss is changed to L det +L pro +L pse Using the gradient descent method, the network parameters of the initial detector are updated until the model converges to obtain the trained final detector.
[0072] The specific steps of the test phase process are to input the point cloud scene to be detected into the trained final detector to obtain the detection result.
[0073] In order to further verify the feasibility and effectiveness of the method of the present invention, the technical effect of the present invention is further described in combination with experiments. The hardware platform used in the experiment of the present invention is: Intel (R) Xeon (R) Gold 6230CPU @ 2.10GHz × 16, 64G memory, two GPUs with 24G video memory, model NVIDIA TITAN 3090. The software platform used in the experiment of the present invention is: Ubuntu 18.04.6LTS operating system and Python 3.8.17, Pytorch1.8.0 deep learning framework. The ScanNet V2 data set validation set is used to analyze the effect of the present invention. Here, the results of mean average precision (mAP) when IOU is 0.25 and 0.5 are used as the evaluation index of detection performance.
[0074] The sparse supervision set by the present invention is that only one object is labeled for each scene in the ScanNet V2 training set, and the final labeling rate is only 7% of the full supervision. The method of the present invention is used to detect each point cloud scene on the validation set of the ScanNet V2 dataset to obtain the detection results corresponding to each scene. In order to test the performance of the method of the present invention, the proposed method of sparse supervised point cloud object detection based on class prototypes is compared with the fully labeled ground truth and the most advanced fully supervised method. The comparison methods include: FCAF3D (FCAF3D: Fully Convolutional Anchor-Free 3D Object Detection. In ECCV, 2022), TR3D (TR3D: Towards Real-Time Indoor 3D Object Detection. In ICIP, 2023) and SPGroup3D (SPGroup3D: Superpoint Grouping Network for Indoor 3D Object Detection). The quantitative results are listed in Table 1. The above-mentioned evaluation index mAP is used as the evaluation standard, and the larger the value, the better.
[0075] Table 1 Quantitative test results of the method of the present invention and the prior art on the validation set
[0076]
[0077] From the results in Table 1, it can be seen that the method of the present invention significantly surpasses the performance of the FCAF3D and TR3D models designed for full supervision under sparse supervision, exceeding the state-of-the-art TR3D by 17.2 points in mAP@0.25 and 19.0 points in mAP@0.5. In addition, even compared with the most advanced fully supervised methods, the method of the present invention still achieves 76% of the performance with only 7% of the annotations. This proves the effectiveness of the prototype-assisted object mining module and the multi-label collaborative optimization module proposed in the present invention. Figure 3 It can be seen from the figure that the present invention can obtain more accurate detection results under the same classification threshold (0.3).
[0078] The above embodiments should be regarded as illustrative and non-restrictive, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the invention and within the spirit and principles of the present invention should be covered by the protection scope of the present invention.
Claims
1. A method for sparse supervised point cloud object detection based on class prototypes, including two processes: a training phase and a testing phase, characterized in that: The training phase consists of the following steps: Step 1: Build a detector to obtain proposal features and detection results from the input point cloud scene; Step 2: Build a prototype-assisted object mining module, use the annotated part of the proposal features obtained in step 1 to cluster and obtain the class prototype of each class; calculate the similarity between the proposal of the unannotated part and each class prototype in step 2, and propagate the prototype annotation to the unannotated part according to the similarity; Step 3: Calculate the loss function values of the sparsely labeled part and the prototype labeled part, and train the initial detector; Step 4: Build a multi-annotation collaborative optimization module, load the parameters of the initial detector to obtain pseudo-annotations and coordinate with the prototype annotations in step 3, calculate the loss function values of sparse annotations, pseudo-annotations, and prototype annotations, train the detector again, and obtain the final detector. In the testing phase, the point cloud scene to be detected is input into the trained final detector to obtain the detection result.
2. The method for sparse supervised point cloud object detection based on class prototype according to claim 1, characterized in that: In the first step of the training phase, the original input point cloud is input into the detector, and the detector is used to extract the proposal feature F p ∈R N×d And the test results, where the test results are divided into classification scores C∈R N×c And the regression result B∈R N×b ; The detector consists of sparse convolutions, R represents a real number, N represents the number of proposals, d is the feature dimension, c is the number of categories, and b is the regression dimension.
3. The method for sparse supervised point cloud object detection based on class prototype according to claim 1, characterized in that: The prototype-assisted object mining module is constructed as described in step 2 of the training phase, including clustering of class prototypes and prototype annotation matching; (1) Clustering of class prototypes: This part receives the proposed features in step 1 Output the class prototype features P of c categories i ={P 1 , P 2 , …, P c}; The specific implementation process is as follows: First, according to the indication of sparse annotation, extract the part of the proposed feature annotation M << N; Then use the multi-step Sinkhorn-Knopp iteration to obtain The correspondence L with each prototype in , where O is the number of prototypes, set to a number greater than 1; The correspondence here is obtained using the following formula: Where u and v are renormalization vectors obtained by multi-step Sinkhorn-Knopp iteration; k is the temperature coefficient used to adjust the smoothness of the distribution; Finally, use the corresponding relationship to update P c ; Here the update method of the i-th prototype of category c is as follows: in is the characteristic of l2 regularization; μ is the coordination variable, N k Represents the number of sparsely labeled proposal features matched by the i-th prototype of category c; (2) Prototype annotation matching: First, the proposal feature F obtained in step 1 of the training phase is p , filter out the unlabeled areas And calculate its similarity with the class prototype Next, use A to calculate the propagation probability W=C u ⊙A′ whereA′=argmax(A)i=1,…,O#(3) Among them, ⊙ is the Hadamard product, W represents the probability that the proposal feature of the unlabeled area belongs to a certain category, and A′ uses the class prototype with the highest probability of each category to represent the current category. Represents the classification score of the unlabeled area; then based on W, the category label of each unlabeled area can be obtained. The process is as follows: C f =argmax(W)i=1,…,C#(4) Finally, by screening out the parts with classification scores lower than the set threshold and the parts beyond the detection range, the final prototype annotation, that is, the remaining category labels, is obtained.
4. The method for sparse supervised point cloud object detection based on class prototype according to claim 1, characterized in that: In step 3 of the training phase, the initial detector is trained. After obtaining the prototype annotation, the loss function of the detector is as follows: The entire scene is divided into annotated parts and unannotated parts. First, the annotated part still uses the annotated detection loss L dey , including classification loss and regression loss; secondly, the unlabeled part uses the prototype annotation as its annotation to calculate the prototype loss L pro ; The total loss is L det +L pro ; Use the gradient descent method to update the network parameters of the initial detector until the model converges to obtain a trained initial detector.
5. The method for sparse supervised point cloud object detection based on class prototype according to claim 1, characterized in that: In step 4 of the training phase, a multi-label collaborative optimization module is constructed. First, the initial detector trained in step 3 of the training phase is used to obtain pseudo labels. In this step, the calculation of the detection loss for pseudo labels is additionally used. pse ; The total loss is L det +L pro +L pse ; Using the gradient descent method, the network parameters of the initial detector are updated until the model converges to obtain the trained final detector.
Citation Information
Patent Citations
Method for densifying sparse mode in 3D point cloud detection
CN116129423A
3D point cloud target detection method based on dynamic adaptive data enhancement
CN117315612A
Weak supervision point cloud semantic segmentation method combining noise mining and correction strategy
CN117876685A
Implicit scene completion method based on dense pixel representation
CN118135519A
Unsupervised domain adaptation method, device, system and storage medium of semantic segmentation based on uniform clustering
US20220383052A1
