A point cloud completion method based on local geometric consistency and feature consistency

Abstract

The present invention relates to a point cloud completion method based on local geometric consistency and feature consistency, which is used to solve the problem of serious loss of details in the completion result and uneven distribution of the completed point cloud in the point cloud completion task. The specific scheme includes: the method based on feature consistency solves the problem of detail loss by strengthening the consistency of the point cloud distribution in the corresponding local area between the predicted value and the true value; in addition, in the point cloud generation process, the present invention adopts a "rough to detailed" multi-stage method to generate point clouds of different scales. Therefore, the method based on feature consistency maps point clouds of different scales to feature space, and strengthens the consistency of point clouds of different scales in feature space, so that point clouds of different scales are more consistent in geometry, so that the final completion result is closer to the true value. Compared with the existing methods, the present invention has largely overcome the above-mentioned problems, and the present invention has a significant improvement effect.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and relates to a point cloud completion method with local geometric consistency and feature consistency, and specifically to technologies such as deep learning neural network methods and three-dimensional image feature extraction. Background Art

[0002] In the field of computer vision, a three-dimensional point cloud is a set of vectors in a three-dimensional coordinate system, which is used to represent the geometric shape of an object. Point cloud data is widely valued and used because of its advantages such as easy data acquisition, ability to effectively represent complex three-dimensional shapes, and no need to maintain topological connection relationships. At the same time, compared with two-dimensional images, three-dimensional point clouds have the advantage of high-dimensional information volume, which has a natural advantage in describing objective things. In recent years, with the development of autonomous driving technology, virtual reality, augmented reality technology, and telecommunications technology, point cloud data has been used more and more widely. However, point cloud data is limited by occlusion, reflection, human operation errors, and equipment resolution during the acquisition process, resulting in missing point cloud data, which will limit the application of downstream tasks (such as target detection and recognition). Therefore, point cloud completion has received more and more attention from researchers. Point cloud completion methods can be mainly divided into three categories: geometry-based completion methods, alignment-based methods, and learning-based methods.

[0003] (1) Geometry-based completion method

[0004] Geometry-based methods use the geometric information contained in incomplete point cloud data to complete the point cloud shape without any external auxiliary information. For example, surface reconstruction methods fill in the holes generated by incomplete scanning when collecting point cloud data by generating smooth interpolations. Shape symmetry-based methods complete the missing area by determining the symmetry axis to copy part of the shape of the known area. These methods all assume that the input data is relatively complete and the geometric structure of the missing area can be inferred from the directly observed area.

[0005] The advantage of this method is that it is simple and can be easily implemented through a simple algorithm. The disadvantages are also very obvious. First, it has very high requirements for the input point cloud data. When the incomplete area of ​​the input point cloud is too large, it is impossible to estimate the geometric shape of the missing area, and thus it is impossible to complete the missing area. Second, the generalization of this method is poor. When the geometric shapes of the incomplete areas of different point cloud data are very different, it is necessary to construct a completion algorithm for each missing geometric shape, resulting in a very high algorithm cost and the inability to complete large-scale point cloud data.

[0006] (2) Alignment-based completion method

[0007] Alignment-based methods complete the missing point clouds by matching them with point cloud models in a large database. Some methods directly retrieve the complete shape for matching, while others match partial shapes of objects and then combine these partial shapes to obtain the complete shape. Some methods deform the retrieved models to synthesize a shape that is more consistent with the input. Still other methods use geometric substitutions, such as planes, quadratic surfaces, and other geometric structures to replace the structures in the shape database. These methods require high optimization costs during the reasoning process, are impractical for online application, and are also easily affected by noise.

[0008] In summary, the alignment-based method manually creates the same or similar CAD model, and then completes the matching alignment with the CAD model according to the features of the missing point cloud, thereby completing the shape restoration of the 3D point cloud. In the actual scanning process, the scanned scenes are generally complex and changeable. Although traditional algorithms are effective for known scenes, they are not very practical.

[0009] (3) Learning-based point cloud completion method

[0010] With the rise of deep learning and the widespread application of neural networks, current point cloud completion methods are mainly focused on learning-based methods. By learning a parameterized model (generally a deep neural network), the missing point cloud is directly mapped to the complete point cloud. Or based on the information of the missing point cloud, the point cloud of the missing area is directly generated. This paper adopts the method of only generating the missing area. This method is defined as: given the coordinates of the missing point cloud data: X = {x1, x2, ..., x n},x i ∈R N*3 , and the corresponding coordinates of the missing area point cloud data are given: Y = {y1, y2, ..., y m},y i ∈R M*3 Given a missing point cloud X, the completion network predicts the missing area Y of the point cloud.

[0011] The learning-based method can be roughly decomposed into two goals in the completion process: first, retain the original input point cloud geometry information to ensure that it is not changed or introduced into noise as much as possible during the network completion process. Second, predict the missing area under the given input conditions. Compared with traditional non-learning methods, this type of method has faster reasoning ability and better generalization ability. In the early development of this type of method, in order to apply the traditional convolution idea to irregular data such as three-dimensional point clouds, voxelization methods were proposed, such as Figure 1, discretize the point cloud, confine the point cloud to a regular regularly arranged grid, and then apply convolution to extract features. However, this method of voxelizing point clouds produces high memory consumption, increases computational complexity, limits output resolution, and further, loses detail information during the discretization process. Until the emergence of Point-Net, features were extracted directly from point cloud data without the need for voxelization, thus avoiding a series of disadvantages brought about by voxelization. After that, most point cloud processing networks adopted methods based on Point-Net. Existing learning-based point cloud completion methods include Folding-Net, Top-Net, PCN, PF-Net, VRC-Net, etc.

[0012] Compared with the earlier alignment-based and geometry-based point cloud completion methods, the learning-based completion methods have high generalization performance, fast completion speed, and are not easily affected by noise. Most of the existing learning-based point cloud completion methods use the Point-Net-based structure, which uses an encoder based on a multi-layer perceptron to extract features from missing point cloud features, and then decodes and reconstructs the extracted features through a decoder based on a fully connected layer and a multi-layer perceptron. Through a multi-stage generation method of "from rough to detailed", a point cloud containing rich details is finally obtained. The main framework adopted by the present invention is based on this structure, such as Figure 2 However, this method uses the chamfer distance (CD) as the loss function when training the model, as shown in the following formula: The smaller the chamfer distance, the smaller the reconstruction loss, and the closer the predicted value is to the true value.

[0013]

[0014] Among them, S1 represents the predicted point cloud, S2 represents the real point cloud, the first term represents the average distance from each point in the predicted point cloud to the nearest point in the real point cloud, which measures the difference between the prediction and the reality; the second term represents the average distance from each point in the real point cloud to the nearest point in the predicted point cloud, which measures the degree to which the real point cloud is covered by the predicted point cloud. Since the decoder generates the point cloud of the missing area in multiple stages, the total loss function is:

[0015]

[0016] However, this method simply considers the point-to-point loss between the predicted point cloud and the real point cloud, such as Figure 3 As shown in (left), the loss of the corresponding local area is not considered, and the local area is not constrained, so the local geometric relationship cannot be well preserved, resulting in distortion of local area details of the point cloud and uneven distribution of the point cloud. Summary of the invention

[0017] The problem solved by the present invention is to overcome the shortcomings of existing learning-based point cloud completion methods and effectively improve the problems of detail loss and uneven point cloud distribution in the point cloud completion process of existing learning-based methods. The present invention provides a point cloud completion method based on local geometric consistency and feature consistency.

[0018] Based on local geometric consistency. Since the traditional learning-based method does not consider the loss of local areas, this paper uses the idea of ​​local geometric consistency to make the geometric relationship of the local areas corresponding to the predicted point cloud and the real point cloud consistent, thereby optimizing the shortcomings of the existing learning-based methods. The specific implementation process is as follows: First, the predicted point cloud and the real point cloud are sampled at the farthest point (FPS) to obtain M key points that can represent the entire point cloud. Then, with these key points as the center, the point cloud is divided into M different local small areas. When reconstructing the point cloud, the loss between the predicted point cloud and the local small areas corresponding to the real point cloud is considered, so that the local geometric relationship of the predicted point cloud is consistent with the local geometric relationship of the real point cloud, and the predicted point cloud retains more details. Figure 3 As shown in (right), the bold points are M (here M=20 is taken as an example) key points obtained by the farthest point sampling (FPS) method. Although sparse, they can still represent the geometric shape of the object; with the key point as the center, k (here k=7 is taken as an example) neighboring points of each key point are taken to form multiple local small areas centered on the key point, thus completing the point cloud area division. Afterwards, in order to establish the correspondence between the predicted point cloud and the local small areas of the real point cloud, firstly, the correspondence between the predicted and real point cloud key points is established according to the Euclidean distance. Since the key point is the center of each local small area, the correspondence between the key points can be used as the correspondence between the predicted and real point cloud local small areas. With the correspondence, the loss between the corresponding local small areas can be calculated. Compared with the simple point cloud point-to-point loss method, this local-to-local loss method will generate a constraint on the local area, which can retain more local geometric relationships, and to a large extent maintain the consistency of the predicted point cloud and the local area of ​​the real point cloud, thereby making the predicted point cloud retain more details. In the specific operation of the experiment, the method of the present invention collects 128 (i.e., M=128) key points from the predicted and real point clouds respectively, and divides them into 128 local small areas according to the key points. Each local area contains 1 key point and its 7 neighboring points (i.e., k=7). The overall loss function after adding the local area loss is as follows:

[0019]

[0020] where d CD_localIt means that after dividing the point cloud into regions, the loss is calculated using the CD chamfer distance.

[0021] Based on the consistency of point cloud features. Since the point cloud completion network uses a multi-stage method of "rough to detailed" to generate point clouds, and the rough point cloud has a guiding role in the generation of detailed point clouds, point clouds of different scales have a certain correlation in the feature space. Therefore, the correlation of point cloud features of different scales can be enhanced to enhance the guiding role of rough point clouds in the generation of fine point clouds, so that the generated point clouds of different scales are more consistent and coherent in geometric structure, and the generated fine point clouds are more reasonable and more realistic. First, the true value of the point cloud in the missing area is reconstructed and trained through a multi-scale point cloud reconstruction neural network to obtain a stable pre-training model. Figure 5 This network is similar to the point cloud completion network and is also composed of two parts: an encoder and a decoder. The encoder is used to extract point cloud features, and the decoder decodes and reconstructs the missing area of ​​the point cloud based on the extracted features. When training the point cloud completion network, we only use the encoder of the pre-trained model to extract the features of the point cloud generated by the completion network.

[0022] When training the point cloud completion network, the decoder of the completion network predicts and generates point clouds of missing areas of different scales, that is, Among them and The encoders of the pre-trained model are used to extract feature vectors C with dimensions n (taking n = 1920 as an example) from point clouds of missing areas of different scales. s With C d , respectively represent the extracted rough With delicate The features of the point cloud. By optimizing the distance between feature vectors, C s With C d The distribution of is as close as possible, thereby maintaining the consistency of the two, so that the completion network generates missing area point clouds of different scales in a consistent and coherent geometric shape, which is more reasonable. When optimizing the distance between features, the present invention uses different methods to conduct experiments, such as cosine similarity, Manhattan distance, Euclidean distance, etc. Finally, according to the best test results, the Euclidean distance is used as the optimization method in this article. Therefore, the loss between two feature vectors is as follows:

[0023]

[0024] Its C s =(x1,x2,...,x n ) T and C d =(y1,y2,...,y n )T Respectively represent and Extracted features. So the loss function after adding feature vector loss is:

[0025]

[0026] Since the point cloud completion network uses a multi-stage point cloud generation method, it is natural to apply local consistency to multi-scale levels, thereby further enhancing the completion performance of the model. Combined with the point cloud feature consistency method, the final point cloud completion network based on multi-scale local geometric consistency and feature consistency is formed, such as Figure 2 . Therefore, the final loss function is:

[0027]

[0028]

[0029] Beneficial effects: Through the method of local geometric consistency, the predicted point cloud and the real point cloud are divided into regions, and the loss is calculated for the corresponding regions, which overcomes the problem of missing details caused by only considering point-to-point loss in the traditional point cloud completion process. Then, through the method of feature consistency, the point clouds of different scales predicted by the network are mapped to the feature space, and the distance loss function is used to make the distribution of point clouds of different scales in the feature space consistent, and finally the point clouds of different scales generated by the network are geometrically consistent and coherent. Experimental results show that the content of the present invention improves the completion performance by about 14% compared with the existing methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 Point cloud voxelization process;

[0031] Figure 2 The overall framework of the point cloud completion network;

[0032] Figure 3 Comparison between finding loss point by point and finding loss in small areas;

[0033] Figure 4 Pre-trained models;

[0034] Figure 5 Subjective quality comparison of different methods on ShapeNet-Part dataset. DETAILED DESCRIPTION

[0035] The above is a general description of the method of the present invention. This section will explain the specific implementation process. The method predicts and completes the point cloud of the missing area through the point cloud completion network. The pre-training involved extracts the predicted point cloud by obtaining a stable encoder model. and The features are further used to calculate the loss function of the completion network. The following is a description of the pre-training stage, the point cloud completion network training stage, and the prediction stage.

[0036] The pre-training stage is to use the neural network, Figure 4 As shown in the figure, a stable model for reconstructing missing areas is obtained. The purpose is to use its encoder to extract the features of the point cloud in the missing area and use it in the feature loss function, so that point clouds of different scales are similar in the feature space, and then the point clouds of different scales reconstructed by the decoder are geometrically consistent.

[0037] Step 1: Randomly select a viewpoint, take the viewpoint as the center, and select M (M = 512 for example) points closest to the viewpoint in the complete point cloud as the missing area of ​​the point cloud, so as to obtain the training data of this stage, and also serve as the ground truth of the network loss function of this stage. The missing area of ​​the point cloud is downsampled m times (m = 2 for example) to obtain M, M / k, M / k respectively. 2 (k is the downsampling coefficient, taking k=2 as an example) point clouds of different scales.

[0038] Step 2: Input the point clouds of the three different scales into the encoder. The encoder has three parallel branches, each of which is a multi-layer perceptron with the same structure but different parameters. A point cloud of one scale is input into one multi-layer perceptron, and each multi-layer perceptron is used to extract the features of a point cloud of one scale. The feature dimensions of the three scales are all 1×d (taking d=1920 as an example).

[0039] Step 3: Concatenate these three features (CONCAT) to obtain 3×d features. After the convolution (CONV) operation, the concatenated features are fused to finally obtain the 1×d fused feature F.

[0040] Step 4: Input feature F into the decoder through a linear layer to obtain the reconstructed missing area; the decoder gradually generates a detailed point cloud through a multi-stage generation method from "rough to detailed". The decoder is a three-layer structure. In order to facilitate the generation of point clouds of different scales, the decoder first performs three linear transformations on feature F to obtain feature vectors F of different dimensions. i , where i = 1, 2, 3. Then the three feature vectors are decoded and reconstructed respectively. The first layer consists of a linear layer, which is used to decode and reconstruct F1, and its output is The key points representing the missing area are sparse, but they can represent the overall shape. The second layer consists of a Linear layer and a CONV layer, which is used to reconstruct the F2 decoding. The output is Represents The third layer is also composed of Linear layer and CONV convolution layer, which is used to decode and reconstruct F3, and the output is Represents Upsampling is performed at the center to obtain the densest point cloud, which contains the richest details. It is the final desired result.

[0041] In the pre-training process, an end-to-end training method is adopted, that is, the network input is the missing area, and the output is the missing area reconstructed by the network. The CD chamfer distance is used as the loss function of the network during the training process. After 200 iterations, the loss value tends to be stable, that is, a stable pre-training model is obtained.

[0042] Among them, the loss function is expressed as follows:

[0043]

[0044] in, represents the point clouds of missing areas of different scales reconstructed by the pre-trained model, Y gt , Y″ gt , Y′ gt is with The corresponding ground truth. gt , Y″ gt is through different downsampling coefficients, gt Down-sampled, Y gt It is the area containing M points removed according to the viewpoint.

[0045] The training of the point cloud completion network is similar to the pre-trained model, where the encoder extracts features and the decoder predicts the missing areas of the point cloud.

[0046] Step 1: First, randomly select a viewpoint, take the viewpoint as the center, select the M (M=512 for example) points closest to the viewpoint in the complete point cloud and remove them, so as to obtain a cloud with missing points (containing N points, N=2048 for example) as the training data of this stage, and the missing area containing M points is used as the groundtruth of the network loss function of this stage. The missing point cloud is downsampled m times (taking m=2 for example) to obtain N, N / k, N / k respectively. 2 (k is the downsampling coefficient, taking k=2 as an example) point clouds of different scales are used as input.

[0047] Step 2: Input the point clouds of the three different scales into the encoder. The encoder has three parallel branches, each of which is a multi-layer perceptron with the same structure but different parameters. A point cloud of one scale is input into one multi-layer perceptron, and each multi-layer perceptron is used to extract the features of a point cloud of one scale. The feature dimensions of the three scales are all 1×d (taking d=1920 as an example).

[0048] Step 3: Concatenate (CONCAT) these three features to obtain 3×d features. After a convolution operation (CONV), the concatenated features are fused to obtain a 1×d fused feature V.

[0049] Step 4: The feature V is input to the decoder through a linear layer. The decoder gradually generates a detailed point cloud through a multi-stage generation method from "rough to detailed". Its structure is the same as the decoder of the training model. The key points representing the missing area, although sparse, can represent the overall shape. Represents Take it as the center and upsample it to get a slightly denser point cloud; Represents Taking the center as the center, upsampling is performed to obtain the densest points, which contain the richest details. This is the prediction result of the point cloud completion network.

[0050] Step 5: Separately and The corresponding features C1 and C2 are extracted by the encoder of the pre-trained model, and the Euclidean distance is used as the loss function to measure the difference between the two. The network is optimized by reducing the difference so that the distribution of the two in the feature space is consistent. and its corresponding true value Y′ gt , and its corresponding true value Y gt The local geometric consistency method is applied to keep the local geometric relationship between the predicted value and the true value consistent. The specific loss function is as follows:

[0051]

[0052] in, Point clouds of missing areas of different scales predicted by the point cloud completion network, Y gt , Y′ gt , Y″ gt is with The corresponding ground truth. gt , Y″ gt By using different downsampling coefficients, Y gtDown-sampled, Y gt It is the area containing M points removed according to the viewpoint.

[0053] In the formal training process, an end-to-end training method is adopted, that is, the network input is a point cloud with missing points, the training data comes from the public dataset Shapenet-Part, and the output is the point cloud of the missing area. The loss function of the network is composed of CD chamfer distance and Euclidean distance. After 200 iterations, the loss value tends to be stable, that is, a stable point cloud model is obtained. After that, this model can be used to complete the missing point cloud in the test set.

[0054] After the above training process is completed, the point cloud completion network predicts and completes the point cloud data to be completed. The specific process is the first four steps of the point cloud completion network training process, which will not be repeated here.

[0055] Figure 5 From top to bottom are the input data, the completion results of PCN, VRC-Net, PF-Net, and LGP-Net (the method of the present invention), and the true value. The light-colored area represents the input with missing points, and the dark-colored area represents the predicted value of the missing area. From a subjective visual comparison, the present invention overcomes the problems of detail loss and uneven distribution brought about by the existing learning-based point cloud completion method. In terms of subjective quality, the experimental results of the method of the present invention are significantly better than the results of other methods. For example, the sixth column is to complete the missing notebook. It can be clearly seen that the point cloud of the missing area generated by the PCN method is unevenly distributed; the point cloud of the missing area generated by VRC-Net is very irregular at the edges; the missing area generated by PF-Net produces obvious point clustering; and the point cloud of the missing area generated by the method of the present invention (LGP-Net) is evenly distributed, relatively regular at the edges, and very close to the true value.

[0056]

[0057] Table 1 Comparison of objective quality of reconstruction by different methods on ShapeNet-Part dataset

[0058] From Table 1, we can see that compared with other methods, our method has achieved the best reconstruction results in 7 out of 8 types of data in terms of objective quality. From the perspective of the average loss, our method is 14.15% lower than the second place. It can be seen that the method proposed in this invention can improve the quality of point cloud completion.

[0059] comprehensive Figure 5 The visualization results and the objective indicators in Table 1 show that the point cloud completion method proposed in the present invention can effectively retain more detailed information in the predicted point cloud and reduce the reconstruction loss. This shows that the method proposed in the present invention is feasible and effective.

Claims

1. A point cloud completion method based on local geometric consistency and feature consistency, characterized by: The point cloud completion network is used to predict and complete the point cloud of the missing area. The working process of the point cloud completion network is as follows: Step 1: Downsample the missing point cloud m times to obtain N, N / k, N / k points respectively. 2 Point clouds of different scales of points; Step 2: Input the point clouds of the three different scales into the encoder. The encoder has three parallel branches, each of which is a multi-layer perceptron with the same structure but different parameters. A point cloud of one scale is input into one multi-layer perceptron, and each multi-layer perceptron is used to extract the features of a point cloud of one scale. The feature dimensions of the three scales are all 1×d. Step 3: Concatenate these three features to obtain 3×d features. After the convolution CONV operation, the concatenated features are fused to obtain a 1×d fused feature V. Step 4: Input the feature V into the decoder through a linear layer to obtain the reconstructed missing area; the decoder is a three-layer structure, in which the first layer consists of a linear layer, and its output is Represents the key points of the missing area; the second layer consists of a linear layer and a convolutional layer in sequence, and the output is Representative Upsampling is performed at the center to obtain a slightly denser point cloud; the third layer consists of a linear layer and a convolutional layer in sequence, and the output is Represents Upsampling is performed at the center to obtain the densest point cloud, which contains the richest details. It is the final desired result; The loss function of the point cloud completion network is as follows: in, Y is the point cloud of missing areas of different scales generated by the point cloud completion network; gt , Y′ gt , Y″ gt They are respectively Corresponding Ground truth; Y′ gt , Y″ gt Through different downsampling coefficients, Y gt Down-sampled, Y gt That is, the missing area containing M points is removed according to the viewpoint, C s With C d They represent the encoder of the pre-trained model from the point cloud and Extracted features; In the pre-training process, an end-to-end training method is adopted, that is, the network input is the missing area, and the output is the missing area reconstructed by the network. The CD chamfer distance is used as the loss function in the pre-training stage during the training process. The purpose of pre-training is to obtain a stable encoder model for extracting predicted point clouds. and The features of are used to complete the calculation of the network loss function. The specific process includes: Step 1: Randomly select a viewpoint, take the viewpoint as the center, and select the M points closest to the viewpoint in the complete point cloud as the missing area of ​​the point cloud, so as to obtain the training data of the pre-training stage, which is also used as the ground truth of the network loss function in this stage; downsample the missing area of ​​the point cloud m times to obtain M, M / k, M / k respectively. 2 Point clouds of different scales of points, k is the downsampling coefficient; Step 2: Input the point clouds of the three different scales into the encoder. The encoder has three parallel branches, each of which is a multi-layer perceptron with the same structure but different parameters. A point cloud of one scale is input into one multi-layer perceptron, and each multi-layer perceptron is used to extract the features of a point cloud of one scale. The feature dimensions of the three scales are all 1×d. Step 3: Concatenate these three features to obtain 3×d features. After the convolution CONV operation, the concatenated features are fused to finally obtain a 1×d fused feature V. Step 4: Input the feature V into the decoder through a linear layer to obtain the reconstructed missing area; the decoder is a three-layer structure, in which the first layer consists of a linear layer, and its output is Represents the key points of the missing area; the second layer consists of a linear layer and a convolutional layer in sequence, and the output is Representative Upsampling is performed at the center to obtain a slightly denser point cloud; the third layer consists of a linear layer and a convolutional layer in sequence, and the output is Represents Upsampling is performed at the center to obtain the densest point cloud, which contains the richest details. It is the final desired result.

Images