Sparse point cloud understanding method based on joint learning and hierarchical self-distillation

Abstract

The invention relates to a sparse point cloud understanding method based on joint learning and hierarchical self-distillation, and belongs to the technical field of computer vision and graphics. Compared with an existing method, the method has the advantages that a plug-and-play hierarchical self-distillation (HSD) framework is added to maximize mutual information from multiple coverage scales, the most unique implicit features in the deepest hyper-space are transmitted to other scales, the last-layer classifier of a series of different scales is designated as a teacher network (the deepest layer), and the classification efficiency is improved. In this way, the classifier in the previous network layer can be supervised, so that important knowledge can be transmitted to different superspaces. According to the method, joint learning is emphasized, and parameter distribution in upstream and downstream networks is coordinated by using MAE and HSD, so that the performance in reconstruction and understanding tasks is enhanced at the same time.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and graphics, and specifically relates to a sparse point cloud understanding method based on joint learning and hierarchical self-distillation. Background Art

[0002] Point cloud technology is an important field in 3D computer vision and graphics, which involves using a large number of points in space to represent the three-dimensional shape of objects and scenes. Point cloud technology has been widely used in fields such as autonomous driving and augmented reality.

[0003] Point cloud completion and understanding play a key role in many 3D computer vision tasks such as autonomous driving, augmented reality, industrial manufacturing, and robotics. Especially in interactive scenarios such as robot grasping, the incompleteness of point cloud data poses a challenge to accurate interaction. This challenge is mainly caused by two reasons: on the one hand, in order to perform collision detection and select grasping postures, it is necessary to reconstruct planes and objects from the point clouds collected by sensors; on the other hand, the interaction method and strength are closely related to the uneven distribution and sparsity of the point cloud, which depends on the understanding of the object.

[0004] In recent years, such as PointNet [1] 、PointNet++ [2] , DGCNN [3] Deep learning models such as CNNs and CNNs have begun to directly use point cloud data for understanding. These methods have improved the tolerance to irregularities and sparsity of point clouds by extracting global and local features. However, they still have difficulty dealing with the problem of large amounts of missing data or surfaces that are too sparse due to factors such as fast scanning, occlusion, viewing angle limitations, or insufficient sensor resolution. To this end, researchers such as X.Yan [4] and Jaritz, J.Gu [5] et al. proposed a processing method for sparse point clouds.

[0005] [1] CRQi, H.Su, K.Mo, and LJGuibas, "Pointnet: Deep learning on pointsets for 3d classification and segmentation," in Proceedings of the IEEEconference on computer vision and pattern recognition, 2017, pp.652–660.

[0006] [2] CRQi, L.Yi, H.Su, and LJGuibas, "Pointnet++: Deep hierarchical feature learning on point sets in a metric space," in Advances in neuralinformation processing systems, 2017, pp.5099–5108.

[0007] [3]Y.Wang, Y.Sun, Z.Liu, SESarma, MMBronstein, and JMSolomon, "Dynamic graph cnn for learning on point clouds," Acm Transactions On Graphics(tog), vol.38, no.5, pp.1–12, 2019.

[0008] [4]

[0009] [5]Jaritz, J.Gu, and H.Su, "Multi-view pointnet for 3d scene understanding," in Proceedings of the IEEE / CVF International Conference on Computer Vision Workshops, 2019, pp.0–0.

[0010] However, the above-mentioned methods often rely on complex voxelization processes or multi-view photometric information, are limited by memory or require additional supervisory signals, and therefore lack versatility. Summary of the invention

[0011] 1. Technical issues to be resolved

[0012] The technical problem to be solved by the present invention is: to design a sparse point cloud understanding method that can complete an incomplete point cloud and identify the shape category of the completed point cloud.

[0013] (II) Technical solution

[0014] In order to solve the above technical problems, the present invention provides a sparse point cloud understanding method based on joint learning and hierarchical self-distillation, comprising the following steps:

[0015] Step 1: Design a cascade network model including an upstream network and a downstream network; the upstream network acts as a mask autoencoder to reconstruct the incomplete input point cloud into a complete shape; the downstream network includes a hierarchical feature extraction HFE module and a fully connected FC classification head to pass the large receptive field information of the deepest network back to the shallow network to complete knowledge self-distillation and realize the classification of the reconstructed point cloud;

[0016] Step 2: Based on the benchmark dataset, the farthest point sampling FPS algorithm is used to sample the centroid of each random area, and the k-nearest neighbor kNN algorithm is used to sample the points adjacent to the centroid. The centroid point and the points adjacent to the centroid are used as input data, and the input data is divided into training data and test data. The input data is a scattered sparse point cloud; the benchmark dataset includes the synthetic dataset ModelNet40 and the real dataset ScanObjectNN;

[0017] Step 3, training and optimizing the entire cascade network model designed in step 1 based on the designed loss function and the training data;

[0018] Step 4: Based on the cascade network model trained in step 3, test the test data in the input data, complete and understand the incomplete point cloud data, complete the incomplete point cloud, and identify the shape category of the completed point cloud.

[0019] The present invention also provides a system for implementing the method.

[0020] (III) Beneficial effects

[0021] Compared with the existing methods, the present invention adds a plug-and-play hierarchical self-distillation (HSD) framework (KL divergence plus cross entropy) to maximize the mutual information from multiple coverage scales and pass the most unique implicit features in the deepest hyperspace to other scales. The last layer of classifiers in the series of different scales is designated as the teacher network (deepest layer) to supervise the classifiers in the previous network layers, thereby transferring important knowledge to different hyperspaces. The present invention emphasizes joint learning and uses MAE and HSD to coordinate the parameter distribution in the upstream and downstream networks, thereby enhancing the performance on reconstruction and understanding tasks at the same time.

[0022] In summary, this paper proposes an end-to-end partial point cloud processing network (cascade network model) and a hierarchical self-distillation (HSD) framework, which aims to maximize the mutual information captured in the scale-aware hyperspace, and HSD can be applied to any hierarchical point cloud benchmark model. And it is highly competitive in many datasets. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A structural diagram of the cascade network model designed for the present invention;

[0024] Figure 2 The point cloud data graph of ModelNet40 input in the embodiment of the present invention;

[0025] Figure 3 The result image of point cloud completion by the upstream model;

[0026] Figure 4 The result of the downstream model's understanding of ModelNet40 point cloud data;

[0027] Figure 5 This is the result of the downstream model’s understanding of ScanObjectNN point cloud data. DETAILED DESCRIPTION

[0028] In order to make the purpose, content and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below in conjunction with the drawings and examples.

[0029] The present invention provides a sparse point cloud understanding method based on joint learning and hierarchical self-distillation. The method is based on joint learning of hierarchical self-distillation (HSD), which completes incomplete point clouds and identifies the shape categories of completed point clouds. It is implemented by two cascade networks, upstream and downstream. In the upstream network, the input data consists of a small number of neighborhood point clouds generated based on randomly selected centroids, which are used to act as occluded point clouds. Then, a masked autoencoder (MAE) is used to learn the asymmetric mapping between unoccluded (complete) to occluded (incomplete) point clouds. In the downstream network, any state-of-the-art (SOTA) hierarchical point cloud classification model can be used for point cloud classification.

[0030] The present invention provides a sparse point cloud understanding method based on joint learning and hierarchical self-distillation, such as Figure 1 As shown, the following steps are included:

[0031] Step 1: Design a cascade network model including an upstream network and a downstream network. The upstream network acts as a mask autoencoder to reconstruct the incomplete input point cloud into a complete shape; the downstream network includes a hierarchical feature extraction (HFE) module and a fully connected (FC) classification head to pass the large receptive field information of the deepest network back to the shallow network to complete knowledge self-distillation;

[0032] Step 2: Based on commonly used benchmark datasets (including synthetic dataset ModelNet40 [6] And the real dataset ScanObjectNN [7] ), the farthest point sampling (FPS) algorithm is used to sample the centroid of each random area, and a small number of points adjacent to the centroid are sampled using the k-nearest neighbor (kNN) algorithm. The centroid point and the points adjacent to the centroid are used as input data, and the input data is divided into training data and test data. The input data is a dispersed sparse point cloud;

[0033] Step 2 is specifically as follows: In order to generate a dispersed sparse point cloud, this embodiment does not use random masking points, but uses the farthest point sampling FPS algorithm to sample the centroid of each random area, and uses the k nearest neighbor kNN algorithm to sample a small number of points adjacent to the centroid, and the centroid point and the centroid adjacent points are used as input data. In particular, for the ModelNet40 dataset, this example uses three types of input data: 32×16=512, 32×8=256, and 64×8=512 in size, respectively. The last input data has 8 adjacent points sampled based on the real value (ground truth, GT) with dense resolution, making the last type more sparse than the first two types.

[0034] refer to Figure 2 Input point cloud data graph for ModelNet40 in the embodiment.

[0035] Step 3, training and optimizing the entire cascade network model designed in step 1 based on the designed loss function and training data;

[0036] Furthermore, in step 3, based on the intrinsic connection between the reconstructed data and the target data, a loss function is designed based on the chamfer distance (CD) to optimize the completion effect:

[0037]

[0038] Where Q and X represent the given complete point set and its corresponding reconstruction point set It represents the squared Euclidean 2-norm of the chamfer distance, and the point set is the set of point clouds.

[0039] Furthermore, in step 3, different downstream networks and corresponding loss functions are used according to different downstream tasks (including classification and segmentation). For point cloud classification problems, the present invention uses simplified versions of three classic hierarchical structures, namely PointNet++, CurveNet [8] and PointMLP [9] , as downstream networks. These downstream networks have achieved good results based on local perception algorithms and are based on cross entropy As the loss function, the formula is:

[0040]

[0041] Where Y is the one-hot encoding representation of the true label of the point cloud, that is, a vector with a length of the number of categories C, in which only one element is 1 and the rest are 0. is the probability distribution predicted by the cascade network model, which is also a vector of length C, with each element Indicates the probability that a point cloud sample belongs to the i-th category. i is an index that traverses all categories.

[0042] Furthermore, in step 3, according to the hierarchical feature extraction and self-distillation method, the difference between the deepest layer and the shallow layer is quantified based on the KL divergence, and the following loss function is designed to balance the ratio of cross entropy and KL divergence:

[0043]

[0044] L is the total number of layers, Z is the implicit variable, where Z L and Z l represents the deepest and intermediate implicit expressions, β is a Lagrange multiplier used to constrain the expected distortion to achieve control over the degree of self-distillation; γ is the weight; I(a; b) represents the mutual information of a and b.

[0045] Furthermore, the step 3 specifically performs weighted addition of the three loss functions to obtain a comprehensive loss function, and the formula is as follows:

[0046]

[0047] Wherein, α, β, and γ are weights, α and β (i.e., the above-mentioned Lagrange multipliers) are used to control the weight ratio of reconstruction and understanding tasks, and γ is used to control the amount of information flowing back to the shallow network, which are set to 1, 0.001, and 0.8 respectively according to experience. The entire cascade network model of the present invention is trained and optimized through a comprehensive loss function.

[0048] Step 4: Based on the cascade network model trained in step 3, test the test data in the input data, complete and understand the incomplete point cloud data, complete the incomplete point cloud, and identify the shape category of the completed point cloud, so as to obtain higher overall accuracy (OA), average class accuracy (mAcc) and smaller reconstruction error (CD).

[0049] Figure 3 It is the result of the upstream network completing the point cloud. Figure 4 It is the result of the downstream model's understanding of ModelNet40 point cloud data. Figure 5 It is the result of the downstream model's understanding of ScanObjectNN point cloud data.

[0050] Compared with the prior art, the end-to-end partial point cloud processing network (cascade network model) proposed in the present invention is more general in more environments. In various cases where the input point cloud data is 32×16=512, 32×8=256 and 64×8=512, it achieves higher overall accuracy (OA) and average class accuracy (mAcc) and smaller reconstruction error (CD) than previous methods in both the completion of incomplete point cloud data and the realization of downstream tasks of complete point cloud data, which also proves that the present invention has strong generalization and good robustness.

[0051] [6] Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, and J. Xiao, "3d shapenets: A deep representation for volumetric shapes," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 1912–1920.

[0052] [7]MAUy, Q.-H.Pham, B.-S.Hua, T.Nguyen, and S.-K.Yeung, "Revisitingpoint cloud classification: A new benchmark dataset and classification model on real-world data," in Proceedings of the IEEE / CVF international conference on computer vision, 2019, pp.1588–1597.

[0053] [8] T. Xiang, C. Zhang, Y. Song, J. Yu, and W. Cai, "Walk in the cloud: Learning curves for point clouds shape analysis," in Proceedings of the IEEE / CVFInternational Conference on Computer Vision, 2021, pp. 915–924.

[0054] [9]X.Ma, C.Qin, H.You, H.Ran, and Y.Fu, "Rethinking network design and local geometry in point cloud: A simple residual MLP framework," in International Conference on Learning Representations, 2022.

[0055] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A sparse point cloud understanding method based on joint learning and hierarchical self-distillation, characterized in that: The following steps are involved: Step 1, design a cascade network model including an upstream network and a downstream network; the upstream network acts as a mask autoencoder to reconstruct the incomplete input point cloud into a complete shape; The downstream network includes a hierarchical feature extraction HFE module and a fully connected FC classification head, which is used to pass the large receptive field information of the deepest network back to the shallow network to complete knowledge self-distillation and realize the classification of the reconstructed point cloud; Step 2: Based on the benchmark data set, the farthest point sampling FPS algorithm is used to sample the centroid of each random area, and the k-nearest neighbor kNN algorithm is used to sample the points adjacent to the centroid. The centroid point and the points adjacent to the centroid are used as input data, and the input data is divided into training data and test data. The input data is a scattered sparse point cloud. The benchmark datasets include the synthetic dataset ModelNet40 and the real dataset ScanObjectNN; Step 3, training and optimizing the entire cascade network model designed in step 1 based on the designed loss function and the training data; Step 4: Based on the cascade network model trained in step 3, test the test data in the input data, complete and understand the incomplete point cloud data, complete the incomplete point cloud, and identify the shape category of the completed point cloud.

2. The method according to claim 1, characterized in that In step 2, for the synthetic dataset ModelNet40, three input data types are used: 32×16=512, 32×8=256, and 64×8=512 in size. The 8 neighboring points of the last input data are sampled based on the true value with dense resolution, making the last one sparser than the first two types.

3. The method according to claim 2, characterized in that In step 3, based on the intrinsic connection between the reconstructed data and the target data, a loss function is designed to optimize the completion effect based on the chamfer distance: Where Q and X represent the given complete point set and its corresponding reconstructed point set. It represents the squared Euclidean 2-norm of the chamfer distance, and the point set is the set of point clouds.

4. The method according to claim 3, characterized in that In step 3, different downstream networks and corresponding loss functions are used according to different downstream tasks (including classification and segmentation). For the point cloud classification problem, three hierarchical structures, namely PointNet++, CurveNet and PointMLP, are used as downstream networks. These downstream networks are based on cross entropy As the loss function, the formula is: Among them, Y is the one-hot encoding representation of the true label of the point cloud, that is, a vector with a length of the number of categories C, in which only one element is 1 and the rest are 0. is the probability distribution predicted by the cascade network model, which is also a vector of length C, with each element Represents the probability that the point cloud sample belongs to the i-th category.

5. The method according to claim 4, characterized in that Furthermore, in step 3, according to the hierarchical feature extraction and self-distillation method, the difference between the deepest layer and the shallow layer is quantified based on the KL divergence, and the following loss function is designed to balance the ratio of cross entropy and KL divergence: Among them, Z L and Z l represents the deepest and intermediate implicit expressions, β is a Lagrange multiplier used to constrain the expected distortion to achieve control over the degree of self-distillation; L is the total number of layers, Z is the implicit variable, γ is the weight; I(a; b) represents the mutual information between a and b.

6. The method according to claim 5, characterized in that In step 3, the three loss functions are weighted and added to obtain the comprehensive loss function, the formula is as follows: Where α, β, and γ are weights. α and β are used to control the weight ratio of reconstruction and understanding tasks, and γ is used to control the amount of information flowing back to the shallow network. The comprehensive loss function is used to train and optimize the entire cascade network model.

7. The method according to claim 6, characterized in that α, β, and γ are set to 1, 0.001, and 0.8, respectively.

8. The method according to any one of claims 1 to 7, characterized in that This method is applied in the field of computer vision technology.

9. The method according to any one of claims 1 to 7, characterized in that This method is used in the field of graphics technology.

10. A system for implementing the method according to any one of claims 1 to 9.

Images