A point cloud 3D object detection method not affected by rotation transformation
By aligning the neural network weights with point cloud features, rotation-invariant features are constructed, solving the instability problem of point cloud detection under rotation transformation and achieving high-accuracy point cloud detection in various scenarios.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG UNIV
- Filing Date
- 2022-10-21
- Publication Date
- 2026-06-09
AI Technical Summary
Existing point cloud detection methods are unstable under rotation transformations. In particular, rotation-invariant methods rely on data augmentation, while rotation-invariant methods are redundant and uninterpretable, limiting their application in more general scenarios.
By treating the weights of the neural network as a set of vectors distributed in a feature space of the same dimension as the point cloud features, and by using principal component analysis and rotation alignment, point cloud features unaffected by rotation transformations are constructed, thus achieving point cloud detection.
It maintains feature consistency under rotational transformations, improving the accuracy and interpretability of point cloud detection, and is suitable for point cloud tasks in various scenarios.
Smart Images
Figure CN115601607B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer image processing, and in particular relates to a point cloud 3D object detection method that is unaffected by rotation transformation. Background Technology
[0002] Unlike two-dimensional images, three-dimensional point clouds are highly sensitive to geometric transformations. Translational geometric transformations include translation, scaling, and rotation. Translation and scaling can be eliminated by normalizing the point cloud. Therefore, the biggest challenge lies in rotation. Point clouds are so sensitive to rotation that adding rotation-based data augmentation during training is less effective than not adding it at all. Therefore, exploring the rotation robustness of point clouds has become an important and popular research topic.
[0003] The field of rotational robustness of point clouds can be mainly divided into two major research directions:
[0004] 1) Rotation-equivariance: The main goal of this type of work is to explore a rotation-equivariant feature extraction method. Representative works include "Learning SO(3) Equivariant Representations with Spherical CNNs" published at the European Conference on Computer Vision, ECCV 2018, and "3D-Rotation-Equivariant Quaternion Neural Networks" published at the European Conference on Computer Vision, ECCV 2020. The rotation-equivariance method aims to ensure that the output features are rotated in a consistent manner with the input. Therefore, this type of method cannot guarantee that the output is completely invariant to any rotation. However, the rotation-equivariance method cannot guarantee that the output remains completely invariant to rotation. That is, for the same input data, the output will change under different rotation angles. Therefore, whether or not rotation data augmentation is applied during the training and testing phases of the model will result in a large difference in performance. Therefore, this type of method only achieves robustness to rotation, but still relies on adding rotation augmentation during training to ensure that the performance does not drop too much.
[0005] 2) Rotation-invariance: The main goal of this type of work is to transform the xyz Cartesian coordinates of the original point cloud into a low-level feature that is invariant to any rotation. This type of work can be mainly divided into two categories: (1) Constructing a rotation-invariant feature, such as "A Rotation-Invariant Framework for Deep Point Cloud Analysis" published in IEEE Transactions on Visualization and Computer Graphics, and "Triangle-Net: Towards Robustness in Point Cloud Learning" published in IEEE Winter Conference on Applications of Computer Vision, WACV 2021. This type of method constructs a feature that is invariant to any rotation from the relative positional relationship between points, such as based on the edge and angle relationship between neighboring points. However, rotation-invariant features in this type of method are redundant and uninterpretable, that is, it is impossible to explain which feature design is optimal. (2) Constructing a rotation-invariant coordinate system, such as "Learning Rotation-Invariant Representations of Point Clouds Using Aligned Edge Convolutional Neural Networks" published at the 2020 IEEE / ACS 17th International Conference on Computer Systems and Applications, and "A Closer Look at Rotation-Invariant Deep Point Cloud Analysis" published at the Proceedings of the IEEE / CVF International Conference on Computer Vision, ICCV 2021. These methods transform the point cloud coordinates from the original xyz coordinate system to a coordinate system with principal components as the axes. Because the principal components rotate consistently with the point cloud, the coordinates of 3D points in the new coordinate system remain unchanged under any rotation. However, they face the problems of over-reliance on global distribution and low interpretability, which limits the application of these methods in more generalized scenarios. Summary of the Invention
[0006] This invention provides a point cloud 3D object detection method that is unaffected by rotation transformation. By treating the weights of the neural network as a set of vectors distributed in a feature space with the same dimension as the point cloud features, the point cloud features are aligned with the weights of the neural network to construct a point cloud feature that does not change with the rotation transformation of the point cloud, thus realizing point cloud 3D object detection that is unaffected by rotation transformation.
[0007] A point cloud 3D object detection method unaffected by rotation transformation includes the following steps:
[0008] (1) The network weights of the first layer of the neural network are regarded as a set of vectors distributed in the feature space with the same dimension as the point cloud features;
[0009] (2) Perform seed point sampling and neighborhood aggregation on the input point cloud data to obtain the local point cloud around each seed point;
[0010] (3) Perform principal component analysis on the network weights in step (1) and the local point cloud obtained in step (2);
[0011] (4) Align the network weights with the weights of the local point cloud to obtain rotation-invariant features;
[0012] (5) Input the local point cloud features from step (4) into the neural network and perform feedforward propagation to detect the prediction of the 3D object box output by the head of the network.
[0013] (6) Train the neural network through gradient backpropagation;
[0014] (7) After the neural network is trained, perform the 3D object detection task of the point cloud.
[0015] This invention can perform 3D object detection, or other classification or recognition tasks on a given frame of 3D point cloud without being affected by rotation transformation.
[0016] In step (1), the network weights W∈R of the first layer of the neural network are... 3×d Consider them as d three-dimensional vectors, distributed in the same space as the input point cloud, where d is the feature dimension of the first layer network output.
[0017] The specific process of step (2) is as follows:
[0018] (2-1) For the input point cloud P = [p1, p2, ..., p...] N ]∈R 3×N Sampling k seed points using the farthest point sampling FPS, let the set of indices of these seed points be .
[0019] (2-2) For the i-th point p iObtain a spherical neighborhood centered at the seed point with radius r. Sample n points within this neighborhood to form the local point cloud surrounding the seed point. Let the set of indices of the local point cloud be denoted as . The local point cloud is then represented as
[0020] In step (3), the network weights W from step (1) and the local point cloud of each seed point obtained in step (2) are compared. The principal component analysis is performed as follows:
[0021] (3-1) Translate the centroid of the local point cloud to the origin, and we get:
[0022]
[0023] (3-2) Find Feature vectors:
[0024]
[0025] Among them, Λ i =diag(λ i，1 , λ i，2 , λ i，3 The diagonal matrix (λ) is constructed using eigenvalues. i，1 ≥λ i，2 ≥λ i，3 V i =[v i，1 v i，2 v i，3 [] represents the eigenvectors, i.e., the principal components;
[0026] (3-3) Shifting the network weight W to the origin, we get:
[0027]
[0028] (3-4) Find Feature vectors:
[0029]
[0030] Among them, A w =diag(λ w，1 , λ w，2 , λ w，3 The diagonal matrix (λ) is constructed using eigenvalues. w，1 ≥λ w，2 ≥λ w，3 U = [u1, u2, u3] is the eigenvector, i.e., the principal component.
[0031] In step (4), the specific process of aligning the network weights with the weights of the local point cloud is as follows:
[0032] Principal component V i =[v i，1 v i，2 v i，3 After a rotational transformation R i Then, make it coincide with U = [u1, u2, u3]; where R i The following formula is used to obtain...
[0033]
[0034] The local point cloud obtained in step (2) Perform the same rotational transformation:
[0035]
[0036] In step (5), the process of the neural network performing the first layer of feedforward propagation is as follows:
[0037]
[0038] Where, X′ i ∈R 3×n for The matrix is formed by arranging the elements, where b is the learnable bias of the first linear fully connected layer.
[0039] Compared with the prior art, the present invention has the following beneficial effects:
[0040] 1. This invention can be applied to point clouds in all scenarios. The construction of rotation-invariant features depends only on the local point cloud and not on the global scene.
[0041] 2. This invention has excellent interpretability. For the patterns encoded by the network weights, aligning the features with them ensures that the output reflects the degree of pattern matching, without being affected by rotation transformation. Attached Figure Description
[0042] Figure 1 This is a flowchart of a point cloud 3D object detection method that is unaffected by rotation transformation according to the present invention. Detailed Implementation
[0043] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be noted that the embodiments described below are intended to facilitate the understanding of the present invention and do not constitute any limitation thereof.
[0044] like Figure 1 As shown, a point cloud 3D object detection method unaffected by rotation transformations specifically includes the following steps:
[0045] S01, assign the weights W∈R of the first layer of the neural network. 3×d(d is the feature dimension of the first layer network output) is regarded as d three-dimensional vectors.
[0046] S02, for the input point cloud P = [p1, p2, ..., p...] N ]∈R 3×N We sample k seed points using FPS (Furthest Point Sampling), and let the set of indices of these seed points be denoted as . For the i-th point p i Obtain a spherical neighborhood centered at the seed point with radius r. Sample n points within this neighborhood to form the local point cloud surrounding the seed point. Let the set of indices of the local point cloud be denoted as . The local point cloud can then be represented as
[0047] S03, The network weights W and the local point cloud of each seed point. Perform principal component analysis.
[0048] For point p i To obtain the local point cloud, first translate the centroid of the local point cloud to the origin, and you will get:
[0049]
[0050] Then ask Feature vectors:
[0051]
[0052] Among them, Λ i =diag(λ i，1 , λ i，2 , λ i，3 The diagonal matrix (λ) is constructed using eigenvalues. i，1 ≥λ i，2 ≥λ i，3 V i =[v i，1 v i，2 v i，3 ] represents the eigenvectors, i.e., the principal components.
[0053] For the network weight W, first shift it to the origin, and we get:
[0054]
[0055] Then ask Feature vectors:
[0056]
[0057] Among them, Λ w=diag(λ w，1 , λ w，2 , λ w，3 The diagonal matrix (λ) is constructed using eigenvalues. w，1 ≥λ w，2 ≥λ w，3 U = [u1, u2, u3] is the eigenvector, i.e., the principal component.
[0058] S04 aligns the network weights in S01 with the weights of the local point cloud in S02 to obtain rotation-invariant features. The alignment method is to align the weights of the local point cloud in S02... Perform rotational transformation:
[0059]
[0060] Among them, R i The following formula is used to obtain...
[0061]
[0062] S05: The aligned local point cloud features from S04 are input into the neural network and fed forward. The feedforward process of the first layer of the neural network:
[0063]
[0064] in, The matrix is formed by arranging the elements, where b is the learnable bias of the first linear fully connected layer.
[0065] Features are extracted through neural networks, and the final detection network head output predicts the 3D object.
[0066] S06, train the neural network through gradient backpropagation.
[0067] S07 uses the trained network to be applied to 3D object detection tasks of point clouds or other tasks (such as point cloud classification, segmentation, etc.).
[0068] To verify the effectiveness of this invention, experiments were conducted on the ModelNet40 and ShapeNet datasets. ModelNet40 contains CAD models across 40 categories. The preprocessed training set contained 9843 models, and the test set contained 2468 models. The ShapeNet dataset contains 16881 models across 16 categories, with 3D points annotated using 50 partial labels. After standard splitting, the training set contained 14007 samples, and the test set contained 2874 samples. The experimental results are shown in Tables 1 and 2, respectively.
[0069] Table 1
[0070]
[0071] Table 1 shows the accuracy comparison on the ModelNet40 dataset. z / z indicates that random rotations around the z-axis were applied during both training and testing; AR / AR indicates that arbitrary random rotations were applied during both training and testing; z / AR indicates that random rotations around the z-axis were applied during training and arbitrary random rotations were applied during testing.
[0072] By comparing existing network structures, published literature [1], published literature [2] and the method of the present invention, it can be seen that the method of the present invention achieves the highest accuracy in the three rotation modes of z / z, AR / AR and z / AR.
[0073] Table 2
[0074]
[0075] Table 2 shows the results on the ShapeNet dataset, with arbitrary random rotations applied during both testing and training; Class mIoU represents the average IoU for each class; Insta.mIoU represents the average IoU for each test sample.
[0076] By comparing existing network structures, published literature [1], published literature [2] and the method of the present invention, it can be seen that the method of the present invention has achieved the highest accuracy.
[0077] Among them, the published literature [1] is "Zhang, J.; Yu, M.-Y.; Vasudevan, R.; and Johnson-Roberson, M. 2020. Learning rotation-invariant representations of point clouds using aligned edge convolutional neural networks. In 2020 International Conference on 3D Vision (3DV), 200–209. IEEE."; the published literature [2] is "Li, F.; Fujiwara, K.; Okura, F.; and Matsushita, Y. 2021. A Closer Look at Rotation-Invariant Deep PointCloud Analysis. In Proceedings of the IEEE / CVF International Conference on Computer Vision, 16218–16227.".
[0078] The embodiments described above provide a detailed explanation of the technical solutions and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A point cloud 3D object detection method unaffected by rotation transformation, characterized in that, Includes the following steps: (1) The network weights of the first layer of the neural network are regarded as a set of vectors distributed in a feature space with the same dimension as the point cloud features; specifically, the network weights of the first layer of the neural network are... Considered There are three-dimensional vectors, where... The feature dimension output by the first layer of the network; (2) Seed point sampling and neighborhood aggregation are performed on the input point cloud data to obtain the local point cloud around each seed point; the specific process is as follows: (2-1) Input point cloud FPS sampling by sampling the furthest point Let there be n seed points, and let the set of indices of these seed points be y = n. ; (2-2) For the first i Points To obtain data centered on that point, Sampling within a spherical neighborhood of radius . Let each point be a local point cloud surrounding the seed point, and let the set of indices of the local point cloud be denoted as . Then the local point cloud is represented as ; (3) Perform principal component analysis on the network weights in step (1) and the local point cloud obtained in step (2); the specific process is as follows: (3-1) By translating the centroid of the local point cloud to the origin, we get: (3-2) Find Feature vectors: in, The diagonal matrix is formed by the eigenvalues. , These are the eigenvectors, i.e., the principal components; (3-3) Network weights Translate to the origin, and we get: (3-4) Find Feature vectors: in, The diagonal matrix is formed by the eigenvalues. , These are the eigenvectors, i.e., the principal components; (4) Align the network weights with the weights of the local point cloud to obtain rotation-invariant features; (5) Input the local point cloud features from step (4) into the neural network and perform feedforward propagation to detect the prediction of the 3D object box output by the head of the network. (6) Train the neural network through gradient backpropagation; (7) After the neural network is trained, perform the 3D object detection task of point cloud.
2. The point cloud 3D object detection method unaffected by rotation transformation according to claim 1, characterized in that, In step (4), the specific process of aligning the network weights with the weights of the local point cloud is as follows: Principal components After a rotational transformation Then, make it with Overlap; among them, The following formula is used to obtain... 。