A point cloud completion method based on mesh-agnostic point removal
Patent Information
- Application Number
- CN202311829693.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-28
- Publication Date
- 2026-09-25
- Estimated Expiration
- 2043-12-28
AI Technical Summary
目前先进的点云补全方法采用的点偏移、点代理、二维平面折叠等方法在面对高形状复杂度的点云补全任务时具有明显的短板：1)点偏移技术在面对复杂点云时需要多步偏移，最终带来的误差累计非常大；2)点代理技术在有限的代理空间中也很难表示复杂的点云形状；3)二维平面折叠技术在物体表面复杂的情况下也难以在折叠过程中很好地拟合
[0029]1、本发明在补全过程中，由于使用神经网络学习卷积核参数来排除规则化网格中的无关点，进而完成补全，可以有效提高补全网络对残缺点云局部形状的补全能力，使得点云补全网络在高复杂度残缺点云的补全性能上表现良好。
Smart Images
Figure CN118096596B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the fields of artificial intelligence and computer vision, and relates to point cloud completion technology, specifically a novel method for point cloud completion by removing grid-independent points. Background Technology
[0002] Point cloud completion, which aims to physically and realistically restore the complete shape of incomplete point clouds due to limitations of the sensor itself or the sensor scanning distance, plays an important role in fields such as 3D scene understanding and robotics, and is an important downstream task in the field of point cloud.
[0003] The most advanced point cloud completion methods currently employ techniques such as point offset, point proxy, and 2D plane folding to achieve good performance on point cloud completion datasets such as PCN. However, it should be noted that datasets such as PCN only have a small number of point cloud categories and the point cloud shapes are relatively simple. Under these conditions, classic point cloud completion network structures such as point offset, point proxy, and 2D plane folding can support accurate point cloud completion.
[0004] However, these methods perform poorly when faced with more complex point cloud shapes and a wider variety of point cloud categories. In the real world, the shapes of objects are far more complex than those in point clouds in datasets like PCN, and the categories of objects are also more diverse. Current advanced point cloud completion methods, such as point offset, point proxy, and 2D plane folding, have significant shortcomings when dealing with point cloud completion tasks with high shape complexity: 1) Point offset techniques require multiple offset steps when dealing with complex point clouds, resulting in a large cumulative error; 2) Point proxy techniques struggle to represent complex point cloud shapes within a limited proxy space; 3) 2D plane folding techniques also struggle to fit well during the folding process when the object surface is complex. The commonly used point cloud completion methods mentioned above perform poorly on complex point clouds and multi-category point clouds. Therefore, there is an urgent need for a method that can adapt to point cloud completion with high shape complexity and multiple categories. Summary of the Invention
[0005] This invention provides a point cloud completion method based on mesh-independent point removal. It utilizes a deep neural network to learn convolutional kernel parameters for removing mesh-independent points, effectively improving the completion performance of the point cloud completion network on complex shapes and point clouds of various categories. Furthermore, this method employs uniformly sampled point meshing to fill regularized meshes, effectively enhancing the robustness of the completion network against point clouds with various types of defects. This invention is achieved through the following technical solutions:
[0006] A point cloud completion method based on mesh-independent point removal, characterized in that the method includes the following steps:
[0007] Step 1: Input the training residual cloud;
[0008] Step two: Construct a regularized grid. Within the regularized grid, uniformly sample a set of points and grid the sampled points. Then, grid the residual defect cloud using the same parameters as the regularized grid and fill it into the regularized grid. The method is as follows:
[0009] (1) Obtain the coordinate range x of the complete point cloud min / y min / z min and x max / y max / z max , where x min / y min / z min These represent the minimum values of the coordinates of the complete point cloud on the x-axis, y-axis, and z-axis, respectively. max / y max / z max These represent the maximum values of the coordinates of the complete point cloud on the x-axis, y-axis, and z-axis, respectively.
[0010] (2) Set the range of the regularized grid according to the coordinate range of the complete point cloud, and set a uniform sampling interval according to the range of the regularized grid and the number of samplings T for each dimension to form a sampling point set S;
[0011] (3) The sampling point set S and the residual defect cloud P are spliced together to obtain the point set B. Based on the range of the regularized grid and the set size of the regularized grid H×W×M, two gridding layers are used to grid the residual defect cloud P and the spliced point set B respectively using the same gridding parameters to obtain the residual defect cloud grid. and splicing point set mesh
[0012] Step 3: Use an encoder and decoder that includes 3D convolutional layers and pooling layers to extract multi-scale and multi-level mesh features;
[0013] Step 4: Generate a set of three-dimensional convolution kernel parameters K using a three-dimensional convolutional layer based on the features extracted by the encoder and decoder;
[0014] Step 5: Convolve the learned 3D convolution kernel parameters onto the regularized mesh to remove irrelevant points and obtain a preliminary, coarsely complete point cloud. The method is as follows: stitch the point set mesh... Pointwise convolution is performed using kernel parameter K. The kernel parameter K is used to gather the features of adjacent vertices around the current vertices to determine whether the current vertices are occupied, thereby identifying irrelevant points in the grid and removing them to obtain the grid G after removing irrelevant points. The demeshing layer is used to obtain a preliminary coarse point cloud C based on the grid G after removing irrelevant points.
[0015] Step 6: Using a fully connected layer, learn multiple sets of offsets for each point based on the initially completed coarse point cloud C. Use the offsets to obtain new points based on the coarse point cloud, and generate a densely completed point cloud.
[0016] Step 7: Optimize the neural network using gradient descent based on the chamfer distance loss;
[0017] Step 8: Iterate through neural network training and save the neural network model;
[0018] Step 9: Using the saved neural network model, input the test point cloud for testing, and the neural network model will output the point cloud completion result.
[0019] Furthermore, the features of the grid vertices are calculated by interpolation of points in the neighboring grids of the grid vertices. The method is as follows: Grid vertex v i Feature w i The calculation method is as follows:
[0020]
[0021] Wherein, N(v) i ) represents the mesh vertex v i The set of points within the surrounding grid range, x i y i , z i Represents the grid vertex v i The coordinates of point p are x, y, and z.
[0022] Furthermore, the specific method for extracting multi-scale and multi-level mesh features using encoder and decoder modules composed of 3D convolutional layers and pooling layers is as follows:
[0023] (1) An encoder is constructed using multiple sets of 3D convolutional layers with a kernel size of 4×4×4 and pooling layers with a kernel size of 2×2×2. The encoder is used to process residual cloud meshes. Capture the multi-level, multi-scale features F of residual clouds;
[0024] (2) A decoder is constructed using two fully connected layers and N sets of 3D deconvolution layers with a kernel size of 4×4×4. The feature map F is enlarged to the original mesh size, and multiple feature maps are obtained from the N sets of deconvolution layers of the decoder.
[0025] Furthermore, a fully connected layer is used to learn multiple sets of offsets for each point in the coarse point cloud. New points are then obtained based on the offsets, generating a densely completed point cloud. The specific method is as follows:
[0026] (1) Based on the preliminary completed coarse point cloud C, the multi-layer feature map in the decoder is used to obtain the feature U of the corresponding point position of the coarse point cloud using the point cloud feature sampling layer;
[0027] (2) Using multiple fully connected layers, multiple offsets are generated for each point in the coarse point cloud C based on feature U. Multiple new point sets are obtained based on the coarse point cloud C using the offsets. Compare the rough point cloud C with the point set The points are stitched together to obtain a densely completed point cloud R.
[0028] The beneficial effects of the technical solution provided by this invention are:
[0029] 1. In the completion process, this invention uses a neural network to learn convolution kernel parameters to eliminate irrelevant points in the regularized grid, thereby completing the completion. This can effectively improve the completion network's ability to complete the local shape of incomplete point clouds, making the point cloud completion network perform well in completing highly complex incomplete point clouds.
[0030] 2. In the completion process, this invention uses a set of uniformly sampled points to be meshed and then filled with a regularized grid, which provides a prerequisite for the grid-independent point exclusion method, avoids the adverse effects of null values in the grid on network training, and improves the robustness of the completion network when facing various types of point cloud shapes. Attached Figure Description
[0031] Figure 1 A flowchart of a point cloud completion method based on the removal of mesh-independent points;
[0032] Figures 2 to 3 This is a comparison of the experimental results of the method of the present invention with the existing best method; Detailed Implementation
[0033] The technical solutions of this invention will now be clearly and completely described with reference to the accompanying drawings. All other embodiments obtained by those skilled in the art based on the technical solutions of this invention without inventive effort are within the scope of protection of this invention.
[0034] (a) Input training defect cloud.
[0035] (II) Construct a regularized grid. Within the scope of the regularized grid, uniformly sample a set of points and grid the sampled points. Then, grid the residual defect cloud using the same parameters as the regularized grid and fill it into the regularized grid. The specific steps are as follows:
[0036] (1) Obtain the coordinate range x of the complete point cloud min / y min / z min and x max / y max / z max , where x min / y min / zmin These represent the minimum values of the coordinates of the complete point cloud on the x-axis, y-axis, and z-axis, respectively. max / y max / z max These represent the maximum values of the coordinates of the complete point cloud on the x-axis, y-axis, and z-axis, respectively.
[0037] (2) Set the range of the regularized grid based on the coordinate range of the complete point cloud. Set a uniform sampling interval based on the range of the regularized grid and the number of samplings T for each dimension. Sample according to the sampling interval O = T. 3 The points form a point set S.
[0038] (3) The sampling point set S and the residual defect cloud P are spliced together to obtain the point set B. Based on the range of the regularized grid and the set size of the regularized grid H×W×M, two gridding layers are used to grid the residual defect cloud P and the spliced point set B respectively using the same gridding parameters to obtain the residual defect cloud grid. and splicing point set mesh The specific description of the meshing operation is as follows:
[0039]
[0040] Here, Gridding represents the gridding operation, the main steps of which are to calculate the grid vertex features by interpolating the points in the eight grid cells surrounding the grid vertex, where the grid vertex v is the grid vertex. i Feature w i The calculation method is as follows:
[0041]
[0042] Wherein, N(v) i ) represents the mesh vertex v i The set of points within the surrounding grid range, x i y i , z i Represents the grid vertex v i The coordinates of point p are x, y, and z.
[0043] (III) Extract multi-scale, multi-level mesh features using an encoder and decoder module composed of 3D convolutional layers and pooling layers. The specific steps are as follows:
[0044] (1) An encoder is constructed using multiple sets of 3D convolutional layers with a kernel size of 4×4×4 and pooling layers with a kernel size of 2×2×2. The encoder is used to process residual cloud meshes. Capture the multi-level, multi-scale features F of residual clouds.
[0045]
[0046] Here, Enc represents the encoder encoding process.
[0047] (2) A decoder is constructed using two fully connected layers and N sets of 3D deconvolution layers with a kernel size of 4×4×4. The feature map F is enlarged to the original mesh size, and multiple feature maps are obtained from the N sets of deconvolution layers of the decoder.
[0048]
[0049] Here, Dec represents the decoding process of the decoder.
[0050] (iv) Based on the features extracted from the encoding / decoding structure, a set of three-dimensional convolutional kernel parameters is generated using a three-dimensional convolutional layer. The specific steps are as follows:
[0051] (1) Based on the feature maps F learned by the encoder and decoder N A set of convolution kernel parameters K is generated using a 3D convolution with 1 output channel.
[0052] K = Conv(F) N )
[0053] Here, Conv represents the three-dimensional convolution operation.
[0054] (v) The learned 3D convolution kernel parameters are used to convolve the regularized mesh, removing irrelevant points to obtain a preliminary, coarse point cloud. The specific steps are as follows:
[0055] (1) In the splicing point set grid Then, pointwise convolution is performed using kernel parameter K. The kernel parameter K is used to aggregate the features of adjacent vertices around a current grid vertex to determine whether the current grid vertex is occupied, thereby identifying irrelevant points in the grid and removing them to obtain the grid G after removing irrelevant points.
[0056]
[0057] in, p represents pointwise convolution. i It is a grid For any vertex in the array, k i This indicates that the convolution kernel parameter K is at vertex p. i The value at that location, Represents a grid Mid-vertex p i If the set of surrounding vertices has a convolution value of 0, it is considered an irrelevant point.
[0058] (2) Using the inverse meshing layer, a preliminary coarse point cloud C is obtained from the mesh G after excluding irrelevant points. The specific operation of the inverse meshing layer is as follows.
[0059] C = GriddingReverse(G)
[0060] Here, GriddingReverse represents the inverse meshing operation. The main steps are to calculate the coordinates of points in the mesh using the vertex coordinates and vertex features. The calculation method is as follows.
[0061]
[0062] Among them, c j This represents the coordinates of point j. w represents the coordinates of the k-th vertex of the grid containing point j. k This represents the feature of the k-th grid vertex.
[0063] (vi) Use a fully connected layer to learn multiple sets of offsets for each point in the coarse point cloud, and use the offsets to obtain new points based on the coarse point cloud, generating a densely completed point cloud. The specific steps are as follows:
[0064] (1) Based on the initially completed coarse point cloud C, the multi-layer feature maps in the decoder are... The feature U of the corresponding point location in the coarse point cloud is obtained using a point cloud feature sampling layer. The calculation method is as follows.
[0065]
[0066] Here, FeatureSampling represents the sampling operation of the point cloud feature sampling layer. The main steps are to obtain point features based on the vertex features of the grid where the points in the coarse point cloud C are located. The calculation method is as follows.
[0067]
[0068] in, This represents the nth vertex of the grid containing point m, and Concat represents the concatenation operation of features along the channel dimension.
[0069] (2) Multiple offsets are generated for each point in the coarse point cloud C based on feature u using multi-layer fully connected layers.
[0070] D = Reshape(FC(U))
[0071] Here, FC represents a fully connected layer, and Reshape represents a feature deformation operation. In the implementation, the number of channels of the output value of the fully connected layer is split into multiple groups to represent multiple offsets. Using the offsets D, multiple new point sets are obtained on the basis of the coarse point cloud C.
[0072]
[0073] Compare the rough point cloud C with the point set By stitching them together, we obtain a densely completed point cloud R.
[0074]
[0075] (vii) Optimize the neural network using gradient descent based on the chamfer distance loss. The formula for calculating the chamfer distance is as follows:
[0076]
[0077] Here, P1 and P2 represent two point clouds, signifying a completed point cloud and a whole point cloud in the point cloud completion task. n1 and n2 represent the number of points in point clouds P1 and P2, respectively, where p1 represents the points in point cloud P1 and p2 represents the points in point cloud P2. Assuming I represents a whole point cloud, here...
[0078] Loss = CD(R, I)
[0079] Here, Loss represents the chamfer distance loss between the densely completed point cloud R and the complete point cloud I, which is used for backpropagation during neural network training.
[0080] (viii) Train the neural network for 50 iterations and save the neural network model. During the training process, check whether the dataset has been trained for 50 iterations to ensure that the neural network parameters converge and achieve the best point cloud completion effect. If the dataset has not been trained for 50 iterations, continue with steps one through seven, inputting the training residual point cloud to train the neural network model, and save the corresponding neural network model parameters after each iteration; if the dataset has been trained for 50 iterations, proceed to step nine.
[0081] (ix) Use the saved neural network model, input the test point cloud for testing, and output the point cloud completion result from the neural network model.
[0082] The feasibility of the method of the present invention is verified below with specific examples, as detailed in the following description:
[0083] Comparative experiments were conducted on two commonly used and challenging point cloud completion datasets: ShapeNet-55 and KITTI. ShapeNet-55, derived from the PCN dataset, includes more categories to approximate the diversity of 3D objects in the 3D world. It comprises 41,952 models for training and 10,518 models for validation and testing. ShapeNet-55 includes only complete point clouds, each containing 16,384 points. 2,048 points were randomly sampled from the complete point clouds to form incomplete point clouds for completion. The KITTI dataset, on the other hand, uses point clouds captured by LiDAR sensors and includes incomplete point clouds of 2,400 cars taken from 426 different time stamps. Each incomplete point cloud contains 2,048 points. Complete point clouds are not included in the KITTI dataset.
[0084] On the ShapeNet-55 dataset, chamfer distance is used as a metric for point cloud completion quality. Chamfer distance calculates the similarity between the completed result and the complete point cloud.
[0085] Since the KITTI dataset does not contain a complete point cloud, a consistency metric is used to measure the quality of point cloud completion. The consistency metric can, to some extent, reflect the robustness of point cloud completion across different instances in the KITTI dataset. The formula for calculating the consistency metric is as follows:
[0086]
[0087] Where L represents the number of frames containing the i-th vehicle. This represents the completion result of the i-th car in the j-th frame containing the i-th car.
[0088] according to Figure 2 The chamfer distance results obtained by comparing the proposed method with existing state-of-the-art point cloud completion methods (PointTr based on point proxy and PMP-Net based on point offset, respectively) on the ShapeNe-55 dataset show that the completion quality of the proposed method is higher than that of the current state-of-the-art methods when completing incomplete point clouds on datasets with more categories and higher shape complexity. Figure 3 The consistency results obtained by performing point cloud completion on the KITTI dataset using the proposed method and the existing best point cloud completion methods (PointTr based on point proxy method and PMP-Net based on point offset method, respectively) demonstrate that the proposed method has good robustness. Therefore, the feasibility and superiority of the proposed method can be demonstrated.
Claims
1. A point cloud completion method based on mesh-independent point removal, characterized in that, The method includes the following steps: Step 1: Input the training residual cloud; Step two: Construct a regularized grid. Within the regularized grid, uniformly sample a set of points and grid the sampled points. Then, grid the residual defect cloud using the same parameters as the regularized grid and fill it into the regularized grid. The method is as follows: (1) Obtain the coordinate range x of the complete point cloud min / y min / z min and x max / y max / z max , where x min / y min / z min These represent the minimum values of the coordinates of the complete point cloud on the x-axis, y-axis, and z-axis, respectively. max / y max / z max These represent the maximum values of the coordinates of the complete point cloud on the x-axis, y-axis, and z-axis, respectively. (2) Set the range of the regularized grid according to the coordinate range of the complete point cloud, and set a uniform sampling interval according to the range of the regularized grid and the number of samplings T for each dimension to form a sampling point set S; (3) The sampling point set S and the residual defect cloud P are spliced together to obtain the point set B. Based on the range of the regularized grid and the set size of the regularized grid H×W×M, two gridding layers are used to grid the residual defect cloud P and the spliced point set B respectively using the same gridding parameters to obtain the residual defect cloud grid. and splicing point set mesh Step 3: Use an encoder and decoder that includes 3D convolutional layers and pooling layers to extract multi-scale and multi-level mesh features; Step 4: Generate a set of three-dimensional convolution kernel parameters K using a three-dimensional convolutional layer based on the features extracted by the encoder and decoder; Step 5: Convolve the learned 3D convolution kernel parameters onto the regularized mesh to remove irrelevant points and obtain a preliminary, coarsely complete point cloud. The method is as follows: stitch the point set mesh... Pointwise convolution is performed using kernel parameter K. The kernel parameter K is used to gather the features of adjacent vertices around the current vertices to determine whether the current vertices are occupied, thereby identifying irrelevant points in the grid and removing them to obtain the grid G after removing irrelevant points. The demeshing layer is used to obtain a preliminary coarse point cloud C based on the grid G after removing irrelevant points. Step 6: Using a fully connected layer, learn multiple sets of offsets for each point based on the initially completed coarse point cloud C. Use the offsets to obtain new points based on the coarse point cloud, and generate a densely completed point cloud. Step 7: Optimize the neural network using gradient descent based on the chamfer distance loss; Step 8: Iterate through neural network training and save the neural network model; Step 9: Using the saved neural network model, input the test point cloud for testing, and the neural network model will output the point cloud completion result.
2. The point cloud completion method based on mesh-independent point removal according to claim 1, characterized in that, The features of a grid vertex are calculated by interpolating points in the neighboring grid of the grid vertex. The method is as follows: Grid vertex v i Feature w i The calculation method is as follows: Wherein, N(v) i ) represents the mesh vertex v i The set of points within the surrounding grid range, x i y i , z i Represents the grid vertex v i The coordinates of point p are x, y, and z.
3. The point cloud completion method based on mesh-independent point removal according to claim 1, characterized in that, The specific method for extracting multi-scale, multi-level mesh features using an encoder and decoder module composed of 3D convolutional layers and pooling layers is as follows: (1) An encoder is constructed using multiple sets of 3D convolutional layers with a kernel size of 4×4×4 and pooling layers with a kernel size of 2×2×2. The encoder is then used to process the residual cloud mesh. Capture the multi-level, multi-scale features F of residual clouds; (2) A decoder is constructed using two fully connected layers and N sets of 3D deconvolution layers with a kernel size of 4×4×4. The feature map F is enlarged to the original mesh size, and multiple feature maps are obtained from the N sets of deconvolution layers of the decoder.
4. The point cloud completion method based on mesh-independent point removal according to claim 1, characterized in that, The specific method for generating a densely completed point cloud is as follows: Multiple offsets are learned for each point in the coarse point cloud using a fully connected layer. New points are then obtained based on these offsets. (1) Based on the preliminary completed coarse point cloud C, the multi-layer feature map in the decoder is used to obtain the feature U of the corresponding point position of the coarse point cloud using the point cloud feature sampling layer; (2) Using multiple fully connected layers, multiple offsets are generated for each point in the coarse point cloud C based on feature U. Multiple new point sets are obtained based on the coarse point cloud C using the offsets. Compare the rough point cloud C with the point set The points are stitched together to obtain a densely completed point cloud R.