Point Cloud Completion Method and System Based on Multi-Resolution Dual-Feature Folding

Abstract

The present invention relates to a point cloud completion method and system based on multi-resolution dual-feature folding. The method includes: inputting a defective point cloud into a point cloud completion network; the point cloud completion network is a pyramid network model with multi-resolution input and multi-level output, and the point cloud completion network includes a multi-resolution encoder, a feature folding layer, and a decoder; using the multi-resolution encoder to extract the global feature and local feature of the defective point cloud; the multi-resolution encoder includes a global feature extractor and a local residual feature extractor; using the feature folding layer to fuse the global feature and the local feature to generate intermediate point features; using the decoder to decode the intermediate point features to generate a complete point cloud. The present invention optimizes the deficiencies of previous methods in local detail extraction and generation of point cloud completion.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a point cloud completion method and system based on multi-resolution dual-feature folding. Background Art

[0002] At present, with the development of computer vision, technologies such as 3D reconstruction and unmanned driving are also booming. In the application of these technologies, 3D data is a very important carrier. There are many ways to represent 3D data, such as point cloud, grid, voxel, etc. Due to the advantages of simple representation, unified structure, easy storage and modification of point cloud, point cloud data is often used to represent 3D objects in the fields of 3D scanning and autonomous driving.

[0003] However, in the process of acquiring point cloud data, it is inevitable that holes or partial shape loss may occur in the objects represented by the point cloud due to mutual occlusion between objects, distance limitation of depth sensors or reflection of the object surface. Such missing incomplete data often cannot accurately represent the true shape of the object in the computer, and it will be difficult to apply to downstream tasks such as object segmentation or surface reconstruction. Therefore, it has become an important task to infer the complete structure of the object by using the incomplete point cloud as input, that is, to complete these incomplete point clouds representing the object.

[0004] In recent years, methods for point cloud completion have been continuously evolving and can generally be divided into two categories: geometry-based point cloud completion methods and learning-based point cloud completion methods. Geometry-based methods utilize the geometric information of the existing input point cloud to complete the overall shape completion. This type of method includes surface reconstruction methods and symmetry methods. The surface reconstruction method (Wang, Y., Li, H., Ning, X., & Shi, Z. (2011). A new interpolation method in mesh reconstruction from 3D point cloud. Proceedings of the 10th International Conference on Virtual Reality Continuum and Its Applications in Industry.) fills the locally incomplete point cloud holes through an interpolation algorithm that generates smooth regions; the symmetry method (Venkatesh, M. V., & Cheung, S. C. S. (2006, October). Symmetric Shape Completion Under Severe Occlusions. In ICIP (pp. 709-712).) identifies the symmetry axes and repeated regular structures in the input incomplete point cloud so as to copy the point cloud in the non-incomplete region to the incomplete region, thereby achieving the purpose of point cloud completion. Shen Yang (Yang, S., Qi, Y., & Qin, H. (2011). Simultaneous structure and geometry detail completion based on interactive user sketches. Science China Information Sciences, 55, 1123-1137.) et al. decompose the initial incomplete model into a base model and high-frequency components, representing the global rough shape and geometric details respectively, then repair the base model through smooth hole filling, and use the high-frequency information to calculate the geometric detail image, thereby repairing the local geometric details and the global structure; M Sung (Sung, M., Kim, V. G., Angst, R., & Guibas, L. J. (2015). Data-driven structural priors for shape completion. ACM Transactions on Graphics (TOG), 34, 1-11) et al. proposed a method that fuses object symmetry and database query to perform shape completion of the point cloud.Although this type of method can be directly implemented through the algorithm and the completion speed is fast, it has high requirements for the input point cloud data. When the incomplete area of the input point cloud is too large, it is difficult to estimate the geometric shape of the missing area, and thus the missing area cannot be completed. At the same time, the generalization is also poor. When the geometric shapes of the incomplete areas of different point cloud data are quite different, it is necessary to build a completion algorithm for each missing geometric shape. As a result, the algorithm cost is too high to complete large-scale, multi-category point cloud data.

[0005] With the development of deep learning, some learning-based point cloud completion methods have emerged. However, the disorder of point clouds makes it very difficult for these methods to extract features from point clouds. Some methods (Dai, A., Ruizhongtai Qi, C., & Nieβner, M. (2017). Shape completion using 3d-encoder-predictor cnns and shapesynthesis. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 5868-5877).) (Stutz, D., & Geiger, A. (2020). Learning 3d shape completion under weak supervision. International Journal of Computer Vision, 128(5), 1162-1181.) (Liu, Z., Tang, H., Lin, Y., & Han, S. (2019). Point-voxel cnn for efficient 3d deep learning. Advances in Neural Information Processing Systems, 32.) convert point clouds into voxels and use 3D convolution to extract their features. Such methods have more improvements in dealing with point clouds with overly large missing areas. However, such methods have a major drawback: the voxelization degree of point clouds is proportional to the resolution and computational cost. The higher the voxelization degree, the higher the resolution, and the richer the details of the obtained object, but the computational cost also increases cubically; The proposed PointNet (Qi, C., Su, H., Mo, K., & Guibas, L. J. (2017). PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 77-85.) solves the problem of directly extracting features from point clouds and uses a Multilayer Perceptron (MLP) to directly process point cloud data. This technology has been further applied to many point cloud completion methods, such as L-GAN (Achlioptas, P., Diamanti, O., Mitliagkas, I., & Guibas, L.J. (2018). Learning Representations and Generative Models for 3D Point Clouds. ICML.), PCN (Yuan, W., Khot, T., Held, D., Mertz, C., & Hebert, M. (2018). PCN: Point Completion Network. 2018 International Conference on 3D Vision (3DV), 728 - 737.), TopNet (Tchapmi, L.P., Kosaraju, V., Rezatofighi, H., Reid, I.D., & Savarese, S. (2019). TopNet: Structural Point Cloud Decoder. 2019 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 383 - 392.), PFNet (Huang, Z., Yu, Y., Xu, J., Ni, F., & Le, X. (2020). PF - Net: Point Fractal Network for 3D Point Cloud Completion. 2020 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 7659 - 7667.), PoinTr (Yu, X., Rao, Y., Wang, Z., Liu, Z., Lu, J., & Zhou, J. (2021). PoinTr: Diverse Point Cloud Completion with Geometry - Aware Transformers. 2021 IEEE / CVF International Conference on Computer Vision (ICCV), 12478 - 12487.), etc. These methods design encoder - decoder - based architectures to generate complete point clouds. These methods solve the problem of point cloud voxelization and can generate relatively complete point clouds. However, they suffer from the loss of fine - grained features during the encoding stage and are difficult to recover, resulting in a significant difference between the detailed parts of the completed point cloud and the real samples. There are still deficiencies in the smoothness of the point cloud details and the degree of shape fitting. These methods can extract and generate the global features of point cloud objects. However, their performance in local details still differs significantly from the target point cloud. Summary of the Invention

[0006] The object of the present invention is to provide a point cloud completion method and system based on multi - resolution dual - feature folding to solve the deficiencies in local detail extraction and generation.

[0007] To achieve the above object, the present invention provides the following solutions:

[0008] A point cloud completion method based on multi - resolution dual - feature folding, comprising:

[0009] Input the incomplete point cloud into the point cloud completion network; the point cloud completion network is a pyramid network model with multi - resolution input and multi - level output, and the point cloud completion network includes a multi - resolution encoder, a feature folding layer, and a decoder;

[0010] Use the multi - resolution encoder to extract the global feature and the local feature of the incomplete point cloud; the multi - resolution encoder includes a global feature extractor and a local residual feature extractor; the global feature extractor adopts a multi - resolution point cloud global feature extraction method, and the local residual feature extractor adopts a point cloud segmentation residual feature extraction method;

[0011] Use the feature folding layer to fuse the global feature and the local feature to generate intermediate point features;

[0012] Use the decoder to decode the intermediate point features to generate a complete point cloud.

[0013] Optionally, after inputting the incomplete point cloud into the point cloud completion network, it further includes:

[0014] Perform downsampling on the incomplete point cloud twice using the farthest point sampling algorithm to obtain three incomplete point clouds with different resolutions respectively;

[0015] Evenly divide the incomplete point cloud into four point cloud blocks.

[0016] Optionally, the step of using the multi - resolution encoder to extract the global feature and the local feature of the incomplete point cloud specifically includes:

[0017] Input the three incomplete point clouds with different resolutions into the global feature extractor to extract the global feature;

[0018] Input the point cloud blocks into the local residual feature extractor to extract the local feature.

[0019] Optionally, the step of inputting the three incomplete point clouds with different resolutions into the global feature extractor to extract the global feature specifically includes:

[0020] The defective point clouds with three different resolutions are respectively passed through a fusion multi-layer perceptron, and each defective point cloud is respectively mapped to six dimensions;

[0021] The features of each dimension are fused into the final feature vector as local context to generate the fused features;

[0022] The fused features are dimension-reduced through one-dimensional convolution to obtain global features.

[0023] Optionally, inputting the point cloud block into the local residual feature extractor to extract local features specifically includes:

[0024] Using a multi-layer perceptron to convert the point cloud block from data into a feature matrix;

[0025] Using a 3*1 convolution kernel to extract the deep features of the feature matrix and performing three upsampling operations to obtain 16384-dimensional features;

[0026] Outputting 2048-dimensional local features of the 16384-dimensional features through a linear layer.

[0027] Optionally, using the feature folding layer to fuse the global features and the local features to generate intermediate point features specifically includes:

[0028] Mapping the global features and the local features to multiple 4*4 2D grids to obtain the grid feature vectors corresponding to the global features and the grid feature vectors corresponding to the local features;

[0029] Distorting and deforming the two grid feature vectors and connecting the two deformed grid feature vectors;

[0030] Passing the connected grid feature vectors through a multi-layer perceptron to obtain the folded intermediate point features.

[0031] Optionally, using the decoder to decode the intermediate point features to generate a complete point cloud specifically includes:

[0032] Generating three original point cloud vectors from the intermediate point features through a fully connected layer;

[0033] Generating three roughly dense point clouds from the three original point cloud vectors through a multi-layer perceptron;

[0034] Starting from the first roughly dense point cloud, adding each folded point in the first roughly dense point cloud to the second roughly dense point cloud to form a new second roughly dense point cloud;

[0035] Add each folded point in the new second rough point cloud to the third rough point cloud to generate a complete point cloud.

[0036] A point cloud completion system based on multi-resolution dual-feature folding, comprising:

[0037] A defective point cloud input module for inputting a defective point cloud into a point cloud completion network; the point cloud completion network is a pyramid network model with multi-resolution input and multi-level output, and the point cloud completion network includes a multi-resolution encoder, a feature folding layer, and a decoder;

[0038] A global feature and local feature extraction module for extracting the global feature and local feature of the defective point cloud by using the multi-resolution encoder; the multi-resolution encoder includes a global feature extractor and a local residual feature extractor; the global feature extractor adopts a multi-resolution point cloud global feature extraction method, and the local residual feature extractor adopts a point cloud segmentation residual feature extraction method;

[0039] An intermediate point feature generation module for fusing the global feature and the local feature by using the feature folding layer to generate an intermediate point feature;

[0040] A complete point cloud generation module for decoding the intermediate point feature by using the decoder to generate a complete point cloud.

[0041] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects: The present invention provides a point cloud completion method and system based on multi-resolution dual-feature folding, which completes the input defective point cloud through a pyramid network model with multi-resolution input and multi-level output. Among them, the multi-resolution point cloud global feature extraction method and the point cloud segmentation residual feature extraction method are respectively used to extract the global feature and local feature of the defective point cloud. The extraction of dual features strengthens the extraction ability of the detailed part on the basis of obtaining the global feature of the point cloud, and optimizes the problem of insufficient local detail extraction in the previous method when extracting point cloud features; The folding method is introduced to fuse the global feature and the local feature, making the surface of the point cloud decoded by the decoder smoother and containing more local details, making it closer to the real point cloud. Description of the Drawings

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0043] Figure 1Flow chart of the point cloud completion method based on multi - resolution dual - feature folding provided by the present invention;

[0044] Figure 2 Structural diagram of the point cloud completion network provided by the present invention;

[0045] Figure 3 Structural diagram of the global feature extractor provided by the present invention;

[0046] Figure 4 Structural diagram of the local residual feature extractor provided by the present invention;

[0047] Figure 5 Schematic diagram of the feature folding layer provided by the present invention;

[0048] Figure 6 Structural diagram of the decoder provided by the present invention;

[0049] Figure 7 Another flow chart of the point cloud completion method based on multi - resolution dual - feature folding provided by the present invention;

[0050] Figure 8 Output comparison chart of the present invention and different methods under the same input data. Detailed implementation manners

[0051] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0052] The purpose of the present invention is to provide a point cloud completion method and system based on multi - resolution dual - feature folding, which optimizes the deficiencies in the local detail extraction and generation of the previous methods in point cloud completion.

[0053] To make the above - mentioned objects, features, and advantages of the present invention more obvious and understandable, the present invention will be further described in detail below in conjunction with the drawings and specific implementation manners.

[0054] Figure 1 Flow chart of the point cloud completion method based on multi - resolution dual - feature folding provided by the present invention, as Figure 1 shown, a point cloud completion method based on multi - resolution dual - feature folding, characterized by comprising:

[0055] Step 101: Input the defective point cloud into the point cloud completion network; the point cloud completion network is a pyramid network model with multi-resolution input and multi-level output, and the point cloud completion network includes a multi-resolution encoder, a feature folding layer, and a decoder.

[0056] In practical applications, after the step 101, it further includes: using the farthest point sampling algorithm to perform two downsamplings on the defective point cloud to obtain three defective point clouds with different resolutions respectively; dividing the defective point cloud into four point cloud blocks on average. Among them, the specific method of point cloud downsampling is: using the farthest point sampling algorithm to perform two downsamplings on the input defective point cloud and the target point cloud to obtain three point clouds with different resolutions respectively, namely the original point cloud with size N, the first downsampled point cloud with size N / 2, and the second downsampled point cloud with size N / 4, where N is the number of points in the point cloud, and each point is composed of three coordinates X, Y, and Z.

[0057] The specific method of point cloud segmentation is: dividing the input defective point cloud into four point cloud blocks on average, so that the size of the point cloud block is N / 4.

[0058] Use the Completion3D dataset for training point cloud completion. This dataset consists of 28,974 point clouds of 8 categories and 800 samples for training and validation respectively. Each of its samples contains 2048 points.

[0059] Figure 2 The structural diagram of the point cloud completion network provided by the present invention is as Figure 2 shown. The network first extracts features from the point cloud, then performs feature folding, and finally uses the decoder to generate a complete point cloud. The present invention uses the Chamfer Distance as the loss function for training, and obtains the weight parameter model with the best effect after 157 rounds of training. The detailed process includes the following steps:

[0060] Step 1, preprocess the point cloud in the dataset, which mainly includes two parts: the first part is to downsample the input point cloud; the second part is to segment the input point cloud.

[0061] Step 2, neural network model construction: Based on the encoder-decoder architecture, a multi-resolution decoder and dual feature folding are proposed to enhance the feature extraction ability of the encoder and the point cloud generation ability of the decoder.

[0062] Step 3: Selection of the loss function: Use the Chamfer Distance to evaluate the quality of the output point cloud. The Chamfer Distance is used to measure the difference between two point clouds. If the distance is large, it means that the two point clouds are quite different; if the distance is small, the two point clouds are more similar.

[0063] Step 4: Model parameter saving: Save the parameters when the parameters in this round of training are better than those in the previous round. After reaching the maximum number of training rounds, select the optimal parameters as the final model.

[0064] Step 102: Use the multi-resolution encoder to extract the global features and local features of the defective point cloud; the multi-resolution encoder includes a global feature extractor and a local residual feature extractor; the global feature extractor uses a multi-resolution point cloud global feature extraction method, and the local residual feature extractor uses a point cloud segmentation residual feature extraction method.

[0065] In practical applications, step 102 specifically includes: Input three defective point clouds with different resolutions into the global feature extractor to extract global features; input the point cloud block into the local residual feature extractor to extract local features.

[0066] The step of inputting three defective point clouds with different resolutions into the global feature extractor to extract global features specifically includes: Pass the three defective point clouds with different resolutions through a fusion multi-layer perceptron respectively, and map each defective point cloud to six dimensions respectively; use the features of each dimension as local context to fuse into the final feature vector to generate the fused features; reduce the dimension of the fused features through one-dimensional convolution to obtain global features.

[0067] Figure 3 The structural diagram of the global feature extractor provided by the present invention is as Figure 3 shown. The three defective point clouds with different resolutions are processed through the structure of a fusion multi-layer perceptron (Fusion Multilayer Perceptron) respectively. The fusion multi-layer perceptron maps each point cloud to six dimensions respectively, which are 64, 128, 256, 512, 1024, and 2048. At the same time, the features of each dimension are used as local context to fuse into the final feature vector, and finally the dimension is reduced through one-dimensional convolution to obtain global features.

[0068] The step of inputting the point cloud block into the local residual feature extractor to extract local features specifically includes: Use a multi-layer perceptron to convert the point cloud block from data into a feature matrix; use a 3*1 convolution kernel to extract the deep features of the feature matrix and perform three upsampling operations to obtain features of 16384 dimensions; output 2048-dimensional local features through a linear layer for the 16384-dimensional features.

[0069] Figure 4 The structural diagram of the local residual feature extractor provided by the present invention is as Figure 4As shown in the figure, first, a Multilayer Perceptron (MLP) is used to convert the input point cloud from N*3 data into a feature matrix. Then, a 3*1 convolutional kernel is used to extract deep features, and three upsampling operations are performed. Finally, the obtained 16384-dimensional feature vector is output as a 2048-dimensional local feature vector through a linear layer.

[0070] Step 103: Use the feature folding layer to fuse the global feature and the local feature to generate intermediate point features.

[0071] In practical applications, step 103 specifically includes: mapping the global feature and the local feature onto multiple 4*4 2D grids to obtain the grid feature vectors corresponding to the global feature and the grid feature vectors corresponding to the local feature; performing warping on the two grid feature vectors and connecting the two warped grid feature vectors; processing the connected grid feature vectors through a multilayer perceptron to obtain the folded intermediate point features.

[0072] Figure 5 Schematic diagram of the feature folding layer provided by the present invention, as Figure 5 shown, the purpose of the feature folding layer is to fuse the global feature and the local feature to make it a vector that contains both the global feature and the local feature. Specifically: taking the two features obtained from the encoder as inputs, taking the 16*n-dimensional local feature vector L and the 16*m-dimensional global feature vector G as inputs, mapping the two feature vectors onto multiple 4*4 2D grids to obtain the grid feature vectors corresponding to the two features, then performing warping on the grids, and finally connecting the two grid feature vectors, and then processing through a multilayer perceptron to obtain the folded intermediate point features.

[0073] Step 104: Use the decoder to decode the intermediate point features to generate a complete point cloud.

[0074] In practical applications, step 104 specifically includes: generating three original point cloud vectors by passing the intermediate point features through a fully connected layer; generating three roughly point clouds with different densities by passing the three original point cloud vectors through a multilayer perceptron; starting from the first roughly point cloud, adding each folded point in the first roughly point cloud to the second roughly point cloud to form a new second roughly point cloud; adding each folded point in the new second roughly point cloud to the third roughly point cloud to generate a complete point cloud.

[0075] The purpose of the decoder is to learn the features extracted by the encoder and generate a complete point cloud. Figure 6 Structural diagram of the decoder provided by the present invention, as Figure 6As shown, specifically: the folded intermediate point feature V0 is passed through a fully connected layer to generate three original point cloud vectors V1, V2, and V3; then the original point cloud vectors are passed through a multi-layer perceptron to generate three roughly point clouds P1, P2, and P3 with different densities, and their densities are 128, 512, and 2048 respectively; finally, starting from P1, each folded point in P1 is added to P2, and a rough point patch of this layer is generated centered on this point; at the same time, P1 and P2 are matched with real samples sampled with the same number of points to improve the accuracy of P3 in generating point clouds; similarly, each point of P2 is added to P3, and the finally generated P3 point cloud is the complete output point cloud.

[0076] The formula for the loss function Chamfer Distance is as follows:

[0077]

[0078] where d CD (S1, S2) represents the Chamfer Distance between two point clouds S1 and S2, x is a point in point cloud S1, y is a point in point cloud S2, the first term represents the mean of the sum of the minimum distances from any point x in S1 to S2, and the second term represents the mean of the sum of the minimum distances from any point y in S2 to S1.

[0079] This formula calculates the average nearest square distance between the complete output point cloud S1 and the real sample S2. The smaller the distance, the more realistic the generated point cloud. In the decoder, three point clouds with different densities will be output, and the point cloud loss with densities of 128 and 512 is used to optimize the finally generated point cloud with a density of 2048.

[0080] In order to execute the method corresponding to the above-mentioned Embodiment 1 to achieve the corresponding functions and technical effects, a point cloud completion system based on multi-resolution dual feature folding is provided below.

[0081] A point cloud completion system based on multi-resolution dual feature folding includes:

[0082] A defective point cloud input module for inputting a defective point cloud into the point cloud completion network; the point cloud completion network is a pyramid network model with multi-resolution input and multi-level output, and the point cloud completion network includes a multi-resolution encoder, a feature folding layer, and a decoder.

[0083] Global feature and local feature extraction module, which is used to extract the global feature and local feature of the defective point cloud by using the multi-resolution encoder; the multi-resolution encoder includes a global feature extractor and a local residual feature extractor; the global feature extractor adopts a multi-resolution point cloud global feature extraction method, and the local residual feature extractor adopts a point cloud segmentation residual feature extraction method.

[0084] Intermediate point feature generation module, which is used to fuse the global feature and the local feature by using the feature folding layer to generate intermediate point features.

[0085] Complete point cloud generation module, which is used to decode the intermediate point features by using the decoder to generate a complete point cloud.

[0086] The present invention proposes a new multi-resolution and dual-feature point cloud completion network based on an encoder-decoder architecture. In order to enhance the feature extraction ability of the traditional encoder-decoder architecture network for defective point clouds and improve the fitting degree of the generated complete point cloud, a pyramid network with multi-resolution input and multi-level output is designed to learn point cloud features and complete the input point cloud.

[0087] Figure 7 Another flowchart of the point cloud completion method based on multi-resolution dual-feature folding provided by the present invention is as Figure 7 shown. The present invention extracts the global and local features of the point cloud based on the multi-resolution point cloud global feature extraction method (Multi-resolution global feature, MRGF) and the point cloud segmentation residual feature extraction method (segment res feature, SRF), which enhances the encoder's ability to extract local features of the point cloud and enables the network to better capture the local detail information of the point cloud.

[0088] In order to smoothly fuse the global feature and the local feature of the point cloud and generate a smooth surface, the present invention introduces a folding method to fuse the global feature and the local feature, making the point cloud surface generated by the decoder smoother.

[0089] In order to verify the effectiveness of the point cloud completion network of the present invention, the point cloud completion network of the present invention, FoldingNet, PCN, and TopNet were trained on the Completion3D dataset, and the performance of the four methods on this dataset was statistically analyzed. Using the Chamfer Distance as the evaluation index, the results are analyzed from a quantitative perspective as shown in Table 1. Table 1 is a schematic table for evaluating using the Chamfer Distance on the Completion3D dataset. The lower the evaluation value, the more similar the generated point cloud is to the real sample.

[0090] Table 1

[0091]

[0092]

[0093] As can be seen from Table 1, the method of the present invention has a higher completion accuracy compared with similar methods and performs optimally in the point clouds of eight categories. Although the Chamfer Distance in the two categories of Lamp and Table is slightly higher, it is still the lowest value among several methods. Generally speaking, the point cloud completion network of the present invention is also significantly superior to other methods on average.

[0094] Of course, the present invention also analyzes the superiority of the present invention from the perspective of visualization. Figure 8 is a comparison chart of the outputs of the present invention and different methods under the same input data. As Figure 8 shown, the first column is the defective point cloud input to the network, the second to fourth columns are the three comparison methods, the fifth column is the output result of the point cloud completion network of the present invention, and the sixth column is the target point cloud. It can be intuitively seen that the present invention is relatively comprehensive in generating the overall shape of the object in various shapes and performs better in restoring the local details of the missing part of the object. For example, for the missing tire part of the car point cloud, the point cloud completion network of the present invention can well complete it. Although other methods restore the overall shape of the car, they cannot complete the details of the tire part; although FoldingNet and TopNet can complete the missing part, the overall shape of the generated point cloud has a large gap with the sample point cloud. The present invention is superior to the other three methods in terms of the structural integrity and detail similarity in the completion of eight types of point clouds.

[0095] The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0096] In this specification, each embodiment is described in a progressive manner. The key point of each embodiment is to illustrate the differences from other embodiments. The same or similar parts among the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method part.

[0097] In this article, specific examples are used to illustrate the principle and implementation of the present invention. The description of the above embodiments is only for helping to understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation and application scope. To sum up, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. A point cloud completion method based on multi-resolution dual feature folding, characterized in that Including: Inputting the defective point cloud into the point cloud completion network; the point cloud completion network is a pyramid network model with multi-resolution input and multi-level output, and the point cloud completion network includes a multi-resolution encoder, a feature folding layer, and a decoder; the multi-resolution encoder includes a global feature extractor and a local residual feature extractor; Performing two downsamplings on the defective point cloud using the farthest point sampling algorithm to obtain three defective point clouds with different resolutions respectively; Dividing the defective point cloud into four point cloud blocks on average; Inputting the three defective point clouds with different resolutions into the global feature extractor to extract global features, specifically including: Passing the three defective point clouds with different resolutions through a fusion multi-layer perceptron respectively, and mapping each defective point cloud to six dimensions; Taking the features of each dimension as local context and fusing them into the final feature vector to generate the fused features; Reducing the dimension of the fused features through one-dimensional convolution to obtain global features; Inputting the point cloud blocks into the local residual feature extractor to extract local features, specifically including: Using a multi-layer perceptron to convert the point cloud blocks from data into a feature matrix; Extracting the deep features of the feature matrix using a 3*1 convolution kernel and performing three upsampling operations to obtain features of 16384 dimensions; Outputting local features of 2048 dimensions through a linear layer for the 16384-dimensional features; Using the feature folding layer to fuse the global features and the local features to generate intermediate point features, specifically including: Mapping the global features and the local features onto multiple 4*4 2D grids to obtain the grid feature vectors corresponding to the global features and the grid feature vectors corresponding to the local features; Distorting and deforming the two grid feature vectors and connecting the two deformed grid feature vectors; Passing the connected grid feature vectors through a multi-layer perceptron to obtain the folded intermediate point features; Using the decoder to decode the intermediate point features to generate a complete point cloud.

2. The point cloud completion method based on multi-resolution dual feature folding according to claim 1, wherein The step of using the decoder to decode the intermediate point features to generate a complete point cloud specifically includes: Generating three original point cloud vectors from the intermediate point features through a fully connected layer; Generating three rough point clouds with different densities from the three original point cloud vectors through a multi-layer perceptron; Starting from the first rough point cloud, adding each folded point in the first rough point cloud to the second rough point cloud to form a new second rough point cloud; Adding each folded point in the new second rough point cloud to the third rough point cloud to generate a complete point cloud.

3. A point cloud completion system based on multi-resolution dual feature folding, characterized in that, The point cloud completion system based on multi-resolution dual feature folding adopts the point cloud completion method based on multi-resolution dual feature folding described in any one of claims 1-2. The point cloud completion system based on multi-resolution dual feature folding includes: A defective point cloud input module for inputting a defective point cloud into the point cloud completion network; the point cloud completion network is a pyramid network model with multi-resolution input and multi-level output, and the point cloud completion network includes a multi-resolution encoder, a feature folding layer, and a decoder; A global feature and local feature extraction module for extracting the global features and local features of the defective point cloud by using the multi-resolution encoder; the multi-resolution encoder includes a global feature extractor and a local residual feature extractor; the global feature extractor adopts a multi-resolution point cloud global feature extraction method, and the local residual feature extractor adopts a point cloud segmentation residual feature extraction method; An intermediate point feature generation module for fusing the global features and the local features by using the feature folding layer to generate intermediate point features; A complete point cloud generation module for decoding the intermediate point features by using the decoder to generate a complete point cloud.

Images