A method for sparse supervised point cloud object detection based on class prototypes

Abstract

The application discloses a kind of based on sparse supervision point cloud object detection of class prototype method, including training phase and test phase.Training phase includes: first, a detector is extracted to construct the feature of proposal;Again, prototype auxiliary object mining module is constructed to use class prototype to mine prototype label to enhance supervision signal;Finally, multi-label collaborative optimization module is constructed using sparse label, pseudo label and sparse label to provide more comprehensive supervision signal.The prototype auxiliary object mining module and multi-label collaborative optimization module mentioned in the above module are only used in training phase, in test phase, only the point cloud scene to be detected is input into the trained detector to obtain the detection result.The application can realize the mining of object under the condition of sparse supervision, and the robustness of the detector to the perception of object is realized by using cross-scene class prototype, and the effect of 76% full supervision model is achieved under the condition that only 7% objects are labeled.

Description

Technical Field

[0001] This invention belongs to the field of embodied intelligence, specifically, it is a method for sparse supervised point cloud object detection based on class prototypes. Background Technology

[0002] Point cloud object detection, as part of the bottom-up perception of indoor scenes, provides embodied intelligence systems with more complete and accurate environmental perception data, thereby enabling safer and more reliable decision-making. Traditional indoor point cloud object detection relies on a large amount of precise annotation, the construction of which is time-consuming and labor-intensive. In complex indoor scenes, such annotation setup requires a significant amount of manual work, as it necessitates constant rotation of the viewing angle to obtain accurate annotations. Therefore, there is an urgent need for a method that can achieve relatively accurate detection while reducing the amount of annotation required.

[0003] Sparse supervision only requires labeling a few objects within each point cloud scene, significantly reducing the manual time required for annotation. Existing sparse supervision methods have only explored outdoor applications and are designed for outdoor scenes. CoIn (CoIn: Contrastive Instance Feature Mining for Outdoor 3D Object Detection with Very Limited Annotations. In ICCV, 2023.) leverages the sparse nature of outdoor scenes by copying objects to empty areas to ensure that all categories of objects, including cars, people, and bicycles, are displayed in a single scene. With this strategy, it can mine objects of all categories from a single scene. However, this method is not suitable for indoor scenes. Indoor scenes have scene priors; copying a bathtub from a bathroom scene to a living room scene is unreasonable. Therefore, outdoor methods cannot be used in indoor scenes. Furthermore, sparse supervised models are often accompanied by optimization modules, which require complex structures to ensure a balance between the precision and recall of pseudo-labels. Summary of the Invention

[0004] The purpose of this invention is to provide a method for sparse supervised point cloud object detection based on class prototypes, in order to solve the problems existing in the existing methods.

[0005] The technical solution to achieve the purpose of this invention is: a method for sparse supervised point cloud object detection based on class prototypes, comprising two processes: a training phase and a testing phase.

[0006] The training phase includes the following steps:

[0007] Step 1: Construct a detector to obtain proposal features and detection results from the input point cloud scene;

[0008] Step 2: construct a prototype-assisted object mining module, and perform clustering on the labeled parts of the proposal features obtained in Step 1 to obtain the class prototype of each class; calculate the similarity between the unlabeled proposals and each class prototype in Step 2, and propagate prototype labels to the unlabeled parts according to the similarity;

[0009] Step 3: calculate the loss function value of the sparsely labeled parts and the prototype-labeled parts, and train an initial detector;

[0010] Step 4: construct a multi-label collaborative optimization module, load the parameters of the initial detector to obtain pseudo-labels, and collaborate the pseudo-labels with the prototype labels obtained in Step 3; meanwhile calculate the loss function values of sparse labels, pseudo-labels and prototype labels, train the detector again, and obtain the final detector;

[0011] In the testing phase, the point cloud scene to be detected is input into the trained final detector to obtain the detection result.

[0012] Further, in Step 1 of the training phase, the original input point cloud is input into the detector, and the detector extracts proposal features F p ∈R N×d and detection results, wherein the detection results are divided into classification scores C∈R N×c and regression results B∈R N×b ; the detector is composed of sparse convolutions, R represents a real number, N represents the number of proposals, d is the feature dimension, c is the number of classes, and b is the regression dimension.

[0013] Further, constructing the prototype-assisted object mining module described in Step 2 of the training phase includes two parts: class prototype clustering and prototype label matching;

[0014] (1) Class prototype clustering: this part receives the proposal features in Step 1 and outputs class prototype features P of c classes i ={P 1 , P 2 , …, P c}; the specific implementation process is: first extract the labeled parts of the proposal features according to the indication of sparse labels M<<N; then use multi-step Sinkhorn-Knopp iteration to obtain and the correspondence L between each prototype in c , wherein O is the number of prototypes, which is set as a number greater than 1; the correspondence here is obtained by the following formula:

[0015]

[0016] ​Where u and v are renormalization vectors, obtained through multi-step Sinkhorn-Knopp iteration; k is a temperature coefficient used to adjust the smoothness of the distribution;

[0017] Finally, update P using the correspondence relationship. c The update method for the i-th prototype of category c is as follows:

[0018]

[0019] in This is a characteristic of L2 regularization; μ is the coordinating variable, N k This represents the number of sparsely labeled proposal features that match the i-th prototype of category c;

[0020] (2) Prototype label matching: First, for the proposal features F obtained in step one of the training phase... p Select the unlabeled areas. And calculate its similarity to the class prototype. Next, we use A to calculate the propagation probability.

[0021] W=C u ⊙A′

[0022] whereA′=argmax(A)i=1,…,O#(3)

[0023] Where ⊙ represents the Hadamard product, W represents the probability that the proposal feature of the unlabeled region belongs to a certain category, and A′ uses the class prototype with the highest probability for each category to represent the current category. The classification score represents the unlabeled region; then, based on W, the category label for each unlabeled region can be obtained, as follows:

[0024] C f =argmax(W)i=1,…,C#(4)

[0025] Finally, by filtering out the parts with classification scores below the set threshold and those outside the detection range, the final prototype label, i.e. the remaining category label, is obtained.

[0026] Furthermore, in step three of the training phase, the initial detector is trained. After obtaining the prototype annotations, the loss function of the detector is as follows:

[0027] The entire scene is divided into labeled and unlabeled parts. First, the labeled part still uses the labeled detection loss L. det This includes classification loss and regression loss; secondly, the unlabeled portion is calculated using the prototype label as its label to calculate the prototype loss L. pro The total loss is Ldet +L pro The gradient descent method is used to update the network parameters of the initial detector until the model converges, resulting in a well-trained initial detector.

[0028] Furthermore, in step four of the training phase, a multi-label collaborative optimization module is constructed. First, pseudo-labels are obtained using the initial detector trained in step three of the training phase. This step further incorporates the calculation of the detection loss L for the pseudo-labels. pse The total loss is L det +L pro +L pse The gradient descent method is used to update the network parameters of the initial detector until the model converges, resulting in the trained final detector.

[0029] Compared with the prior art, the present invention has the following significant advantages:

[0030] (1) This invention constructs a prototype-assisted object mining module, which breaks through the limitations of the scene, obtains class prototypes from the entire data and assigns prototype labels, realizing the propagation of labels from labeled objects to class prototypes, and then from class prototypes to unlabeled objects. Compared with previous methods, this invention realizes cross-scene label propagation, utilizes the information of the entire dataset, and achieves more robust performance.

[0031] (2) This invention constructs a multi-label collaborative optimization module that collaboratively combines sparse labels, pseudo labels, and prototype labels. Compared with previous methods, this invention achieves a simpler and more efficient optimization module, which can better utilize various labeling information and improve the generalization ability and accuracy of the model. Attached Figure Description

[0032] Figure 1 This is an overall framework diagram of the method of the present invention, including a framework diagram of the prototype-assisted object mining module and a framework diagram of the multi-annotation collaborative optimization module.

[0033] Figure 2 This is a schematic diagram illustrating the sparse annotation used in this invention and the prototype tags mined by the proposed prototype-assisted object mining module. Figure 2 (a) and Figure 2 (b) are sparse annotations and prototype labels mined from the same scene.

[0034] Figure 3 This is a comparison of the detection results of the method proposed in this invention with other methods under sparse annotation. Detailed Implementation

[0035] This invention discloses a method for sparse supervised point cloud object detection based on class prototypes. This method uses a prototype-assisted object mining module to overcome the limitations of a single scene by aggregating class prototypes across scenes, achieving the mining of all categories within a scene. Subsequently, a multi-annotation collaborative optimization module simplifies the design of the optimization module by combining sparse annotations, pseudo-annotations, and prototype annotations, while achieving a balance between accuracy and recall. The method comprises two phases: a training phase and a testing phase.

[0036] The specific steps of the training phase process are as follows:

[0037] Step 1_1: Input the original input point cloud into the detector and use the detector to extract the proposal features F. p ∈R N×d And the detection results, where the detection results are divided into classification scores C∈R N×c And regression results B∈R N×b The detector consists of sparse convolutions. Let N represent a real number, d be the feature dimension, c be the number of categories, and b be the regression dimension.

[0038] Step 1_2: Construct a prototype-assisted object mining module, which includes two parts: class prototype clustering and prototype annotation matching.

[0039] (1) Clustering of class prototypes: This part receives the proposal features from step 1_1. Output the class prototype features P of c categories i ={P 1 ,P 2 ,…,P c The specific implementation process is as follows: Taking the c-th category as an example, firstly, according to the instructions of sparse annotation, extract the part of the proposal feature annotation. Then, using multi-step Sinkhorn-Knopp iteration (Lightspeed Computation of Optimal Transport. In NeurIPS, 2013) to obtain... and The correspondence between each prototype L c Where O>1 is to ensure the diversity of objects within category c. The correspondence here is obtained using the following formula:

[0040]

[0041] Where u and v are renormalization vectors, obtained through multiple Sinkhorn-Knopp iterations. k is 0.05.

[0042] Finally, update P using the correspondence relationship.c The update method for the i-th prototype of category c is as follows:

[0043]

[0044] in This is a characteristic of L2 regularization. μ is 0.9, N k This represents the number of sparsely labeled proposal features that match the i-th prototype of category c.

[0045] (2) Prototype labeling matching: First, for the proposal features F obtained in step 1_1, p Select the unlabeled areas. And calculate the similarity between it and the class prototypes of each category obtained in step 1-2(1). Next, we use A to calculate the propagation probability.

[0046] W=C u ⊙A′

[0047] whereA′=argmax(A)i=1,…,O#(3)

[0048] Where ⊙ represents the Hadamard product. W represents the probability that the proposed feature of the unlabeled region belongs to a certain category. A′ uses the class prototype with the highest probability for each category to represent the current category. This represents the classification score for the unlabeled region. Then, based on W, the category label for each unlabeled region can be obtained. Since these labels are derived from the prototype, they are referred to as prototype labeling in this invention. This process is as follows:

[0049] C f =argmax(W)i=1,…,C#(4)

[0050] Finally, the prototype label is obtained by filtering out parts with low classification scores and parts that are outside the detection range.

[0051] Step 1_3: Train the initial detector. After obtaining the prototype annotations, the detector's loss function is as follows: In this step, the entire scene is divided into an annotated part and an unannotated part. First, the annotated part still uses the annotated detection loss L. det This includes classification loss and regression loss. Secondly, for the unlabeled portion, the prototype label is used as its label to calculate the prototype loss L. pro Since only class loss is considered here, only the classification loss is calculated. The total loss is L. det +L pro The gradient descent method is used to update the network parameters of the initial detector until the model converges, resulting in a well-trained initial detector.

[0052] Step 1_4: Construct a multi-label collaborative optimization module. First, use the initial detector trained in Step 1_3 to obtain pseudo-labels. The threshold is set to 0.3 to ensure the accuracy of the pseudo-labels. To address the low recall rate of pseudo-labels, this invention reintroduces the prototype annotation proposed in Step 1_2 in this step. Similar to Step 1_3, the difference in this step is the addition of calculating the detection loss L for pseudo-labels. pse The total loss has been changed to L. det +L pro +L pse The gradient descent method is used to update the network parameters of the initial detector until the model converges, resulting in the trained final detector.

[0053] The specific steps of the testing phase are as follows: input the point cloud scene to be detected into the trained final detector to obtain the detection result.

[0054] The present invention will be further described below with reference to the accompanying drawings and embodiments. The described embodiments are merely some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] The present invention will now be described in detail with reference to the accompanying drawings. The overall framework of the method is as follows: Figure 1 As shown, a method for sparse supervised point cloud object detection based on class prototypes includes two processes: a training phase and a testing phase. The specific steps of the training phase are as follows:

[0056] Step 1_1: Input the original input point cloud into the detector and use the detector to extract proposal features. And the test results, which are divided into category scores. and regression results The detector consists of sparse convolutions. Let N represent the number of proposals, d be the feature dimension, c be the number of categories, and b be the regression dimension. In the method of this invention, N is dynamically changing and does not have a fixed value; d is set to 128, c to 18, and b to 6.

[0057] Step 1_2: Construct a prototype-assisted object mining module, the module structure is as follows Figure 1 As shown, it includes two parts: clustering of class prototypes and prototype annotation matching.

[0058] (1) Clustering of class prototypes: This part receives the proposal features from step 1_1. Output the class prototype features P of c categories i ={P 1 ,P2 ,…,P c The specific implementation process is as follows: Taking the c-th category as an example, firstly, according to the instructions of sparse annotation, extract the part of the proposal feature annotation. Then, using multi-step Sinkhorn-Knopp iteration (Lightspeed Computation of Optimal Transport. In NeurIPS, 2013) to obtain... and The correspondence between each prototype L c Where O>1 is to ensure the diversity of objects within category c. The correspondence here is obtained using the following formula:

[0059]

[0060] Where u and v are renormalization vectors, obtained through multiple Sinkhorn-Knopp iterations. k is 0.05.

[0061] Finally, update P using the correspondence relationship. c The update method for the i-th prototype of category c is as follows:

[0062]

[0063] in This is a characteristic of L2 regularization. μ is 0.9, N k This represents the number of sparsely labeled proposal features that match the i-th prototype of category c.

[0064] (2) Prototype labeling matching: First, for the proposal features F obtained in step 1_1, p Select the unlabeled areas. And calculate the similarity between it and the class prototypes of each category obtained in step 1-2(1). Next, we use A to calculate the propagation probability.

[0065] W=C u ⊙A′

[0066] whereA′=argmax(A)i=1,…,O#(3)

[0067] Where ⊙ represents the Hadamard product. W represents the probability that the proposed feature of the unlabeled region belongs to a certain category. A′ uses the class prototype with the highest probability for each category to represent the current category. This represents the classification score for the unlabeled region. Then, based on W, the category label for each unlabeled region can be obtained. Since these labels are derived from the prototype, they are referred to as prototype labeling in this invention. This process is as follows:

[0068] C f =argmax(W),i=1,…,C#(4)

[0069] Finally, the prototype annotations are obtained by filtering out low-scoring segments and those outside the detection range. The mined prototype annotations are as follows: Figure 2 As shown in (b), it can be seen that the present invention can detect the categories of doors, trash cans, etc. in the scene when only the sofa is labeled, which proves the effectiveness of the present invention in mining class prototypes from the entire dataset and mining labels accordingly.

[0070] Step 1_3: Train the initial detector. After obtaining the prototype annotations, the detector's loss function is as follows: In this step, the entire scene is divided into an annotated part and an unannotated part. First, the annotated part still uses the annotated detection loss L. det This includes classification loss and regression loss. Secondly, for the unlabeled portion, the prototype label is used as its label to calculate the prototype loss L. pro Since only class loss is considered here, only the classification loss is calculated. The total loss is L. det +L pro The gradient descent method is used to update the network parameters of the initial detector until the model converges, resulting in a well-trained initial detector.

[0071] Step 1_4: Construct a multi-label collaborative optimization module, with the following structure: Figure 1 As shown. First, pseudo-labels are obtained using the initial detector trained in step 1_3. A threshold of 0.3 is set here to ensure the accuracy of the pseudo-labels. To address the low recall rate of pseudo-labels, this invention reintroduces the prototype annotation proposed in step 1_2 in this step. Similar to step 1_3, the difference in this step is the addition of calculating the detection loss L for pseudo-labels. pse The total loss has been changed to L. det +L pro +L pse The gradient descent method is used to update the network parameters of the initial detector until the model converges, resulting in the trained final detector.

[0072] The specific steps of the testing phase are as follows: input the point cloud scene to be detected into the trained final detector to obtain the detection result.

[0073] To further verify the feasibility and effectiveness of the method of this invention, the technical effects of this invention are further described in conjunction with experiments. The hardware platform used in the experiments of this invention is: Intel(R) Xeon(R) Gold 6230 CPU@2.10GHz×16, 64G of memory, and two GPUs with 24G of video memory each, model NVIDIA TITAN 3090. The software platform used in the experiments of this invention is: Ubuntu 18.04.6LTS operating system and Python 3.8.17 and PyTorch 1.8.0 deep learning framework. The ScanNet V2 dataset validation set is used to analyze the effect of this invention. Here, the mean Average Precision (mAP) at IOU of 0.25 and 0.5 is used as the evaluation index of detection performance.

[0074] The sparse supervision set in this invention involves labeling only one object per scene in the ScanNet V2 training set, resulting in a labeling rate of only 7% of that achieved with full supervision. The method of this invention is used to detect each point cloud scene on the ScanNet V2 dataset validation set, yielding detection results for each scene. To test the performance of this method, the proposed prototype-based sparsely supervised point cloud object detection method is compared with fully labeled ground truth and state-of-the-art fully supervised methods. The comparison methods include: FCAF3D (Fully Convolutional Anchor-Free 3D Object Detection, ECCV, 2022), TR3D (Towards Real-Time Indoor 3D Object Detection, ICIP, 2023), and SPGroup3D (Superpoint Grouping Network for Indoor 3D Object Detection). Quantitative results are listed in Table 1, using the aforementioned evaluation metric mAP as the evaluation standard; a higher value is better.

[0075] Table 1. Quantitative test results on the validation set using the method of the present invention and existing technologies.

[0076]

[0077] As shown in Table 1, the method of this invention significantly outperforms the performance of the FCAF3D and TR3D models designed for full supervision under sparse supervision, exceeding the state-of-the-art TR3D by 17.2 points at mAP@0.25 and by 19.0 points at mAP@0.5. Furthermore, even compared to the state-of-the-art fully supervised method, the method of this invention achieves 76% performance with only 7% annotation. This demonstrates the effectiveness of the prototype-assisted object mining module and the multi-label collaborative optimization module proposed in this invention. Figure 3 As can be seen, the present invention can obtain more accurate detection results under the same classification threshold (0.3).

[0078] The above embodiments should be considered as exemplary and non-limiting, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by those skilled in the art within the scope of the technology disclosed in the invention and within the spirit and principles of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A method for sparse supervised point cloud object detection based on class prototypes, comprising two processes: a training phase and a testing phase, characterized in that: The training phase includes the following steps: Step 1: Construct a detector to obtain proposal features and detection results from the input point cloud scene; input the original input point cloud into the detector, and use the detector to extract proposal features. And the test results, which are divided into category scores. and regression results The detector consists of sparse convolutions, where R represents a real number, N represents the number of proposals, d is the feature dimension, c is the number of classifications, and b is the regression dimension. Step 2: Construct a prototype-assisted object mining module. Use the labeled parts of the proposal features obtained in Step 1 to cluster and obtain the class prototype of each class. Calculate the similarity between the unlabeled proposals and each class prototype in Step 2, and propagate the prototype label to the unlabeled parts based on the similarity. The construction of the prototype-assisted object mining module includes two parts: class prototype clustering and prototype label matching. (1) Clustering of class prototypes: This part receives the proposal features from step 1. Output Class prototype features of each category The specific implementation process is as follows: First, according to the instructions of sparse annotation, extract the part of the proposal feature annotation. , << Then, the multi-step Sinkhorn-Knoppiteration algorithm is used to obtain... and Correspondence between each prototype Where O is the number of prototypes, set to a number greater than 1; the correspondence here is obtained using the following formula: in and These are all renormalized vectors, obtained through multiple Sinkhorn-Knopp iterations; It is a temperature coefficient used to adjust the smoothness of the distribution; Finally, update using the correspondence relationship. ; Category here No. The prototype update method is as follows: in This is a characteristic of L2 regularization; As a coordinating variable, Indicates category No. The number of prototypes that match the sparsely labeled proposal features; (2) Prototype labeling matching: First, for the proposal features obtained in step one of the training phase... ,filter Exit the unmarked areas ( ), and calculate its similarity to the class prototype. Next, utilize Calculate the propagation probability in, For Hadama accumulation, The probability that a proposal from an unlabeled region belongs to a certain category. The current category is represented by the class prototype with the highest probability for each category. The classification score represents the unlabeled area; then based on The category label for each unlabeled region can be obtained through the following process: Finally, by filtering out the parts with classification scores below the set threshold and the parts that are outside the detection range, the final prototype label is obtained, which is the remaining category label. Step 3: Calculate the loss function values ​​for the sparse labeled part and the prototype labeled part, and train the initial detector; Step 4: Construct a multi-label collaborative optimization module, load the parameters of the initial detector to obtain pseudo labels and collaborate with the prototype labels in Step 3, and simultaneously calculate the loss function values ​​of sparse labels, pseudo labels and prototype labels, and retrain the detector to obtain the final detector. During the testing phase, the point cloud scene to be detected is input into the trained final detector to obtain the detection results.

2. The method for sparse supervised point cloud object detection based on class prototypes according to claim 1, characterized in that: In step three of the training phase, the initial detector is trained. After obtaining the prototype annotations, the loss function of the detector is as follows: The entire scene is divided into labeled and unlabeled parts. First, the labeled part still uses the labeled detection loss. This includes classification loss and regression loss; secondly, for unlabeled parts, the prototype label is used as its label to calculate the prototype loss. The total loss is ; The network parameters of the initial detector are updated using gradient descent until the model converges, resulting in a well-trained initial detector.

3. The method for sparse supervised point cloud object detection based on class prototypes according to claim 1, characterized in that: In step four of the training phase, a multi-label collaborative optimization module is constructed. First, pseudo-labels are obtained using the initial detector trained in step three of the training phase. This step further incorporates the calculation of the detection loss for the pseudo-labels. The total loss is + ; The gradient descent method is used to update the network parameters of the initial detector until the model converges, resulting in the trained final detector.

Images