A point cloud completion method based on view generator

Abstract

The present invention discloses a point cloud completion method based on a view generator, which first uses an invisible point removal algorithm to preprocess a data set; then designs a view-based point cloud completion network PV‑Net, which includes an encoder E and a decoder D; wherein the encoder E is responsible for extracting the global feature f of the input residual point cloud, and the decoder D is responsible for decoding the global feature f to obtain a complete point cloud, and the decoder adopts a view-based strategy from part to whole, which is implemented by a view generator. The point cloud completion method proposed in the present invention generates an intermediate point cloud under adjacent view angles by making full use of the view correlation between residual point clouds; gradually reduces the degree of incompleteness of the residual point cloud by expanding the view angle of the residual point cloud, so as to better infer the complete shape of the residual point cloud, and at the same time uses the residual point cloud for mutual supervision to promote the decoder to pay better attention to local details, thereby achieving a better completion effect.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology; in particular, it relates to a point cloud completion technology based on a view generator. Background Art

[0002] In recent years, point cloud completion tasks have attracted widespread attention in the fields of computer vision and robotics. Due to limitations such as resolution or occlusion, the original point cloud data obtained by devices such as depth cameras or lidars are usually sparse and incomplete. Incomplete 3D objects greatly limit their applications in real life, such as object detection, scene segmentation and other downstream tasks. Therefore, it is very necessary to predict or infer the missing structure from the partially observed incomplete point clouds, which has very important potential and value in the field of 3D vision, especially in application scenarios such as autonomous driving and robotics.

[0003] Many works have attempted to solve the point cloud completion problem. These methods can be divided into three categories according to the representation method of 3D data: voxel-based methods, grid-based methods, and point cloud-based methods. Voxel-based methods perform shape completion on volume data, such as using occupancy grids or TSDF voxel representations. The advantage of using voxel representation methods is that convolution operations can be used directly, but the disadvantage is that it consumes a lot of memory. Grid-based methods represent 3D objects as a set of vertices and edges. Although grids can describe complex structures, it is difficult to change the topological shape during model training due to their fixed vertex connection patterns. In contrast, point clouds are an intuitive and concise representation of 3D data, and more and more research focuses on shape completion methods based on point clouds. However, the disorder and irregularity of point clouds pose great challenges to point cloud completion tasks. The decoder structure of the current mainstream point cloud completion methods mostly adopts a resolution-based coarse-to-fine strategy, which obtains fine output by upsampling sparse coarse point clouds. The coarse-to-fine strategy is suitable for the task of completing residual point clouds caused by resolution, but it does not perform well for missing points caused by view occlusion. How to better handle the point cloud completion task caused by perspective is the main content of this patent. Summary of the invention

[0004] To solve the above problems, the present invention discloses a point cloud completion method based on a view generator, which comprises the following steps:

[0005] S1. Dataset generation: Use the invisible point removal algorithm to process the dataset and generate the residual point cloud P. v ;

[0006] S2. Network model construction: Based on the encoder-decoder structure, a viewpoint-based point cloud generator is proposed to construct the PV-Net network model;

[0007] S3, model training: Use the generated training set and the optimization algorithm to reduce the loss function so that the network converges and completes the point cloud completion.

[0008] The present invention is further preferred: the invisible point removal algorithm in step S1 specifically includes viewing angle selection, spherical mapping and visible point selection.

[0009] The specific method of viewing angle selection is: the twelve vertices of the unit icosahedron are used as the viewing angle C, and the viewing angle direction is toward the coordinate origin (0,0,0); the specific steps of the spherical mapping operation are: arbitrarily select the viewing angle c k ∈C, any point p in the point cloud P i Mapped onto a sphere with radius r The spherical mapping formula is:

[0010]

[0011] The specific steps of the visible point selection operation are: select the point from the perspective c k and mapping sets The convex hull points in the set are combined to obtain the residual defect cloud P containing only visible points. v .

[0012] The present invention further preferably: the PV-Net network model in step S2 is composed of an encoder E and a decoder D,

[0013] The encoder E is responsible for extracting the global features f of the input residual point cloud, and the decoder D is responsible for decoding the global features f to obtain the complete point cloud;

[0014] The specific structure of encoder E is: Input residual defect cloud P v First, the convolution layer Conv1 composed of three multi-layer perceptrons (MLP) is used to obtain 256-dimensional features, and then the global feature f2 is obtained through the maximum pooling layer. The formula is:

[0015] f2 = maxpool(Conv1(P v )),

[0016] Repeat the global feature f2 256 times in dimension 2 and concatenate it with f1 to obtain the mixed feature f3. The formula is:

[0017] f3=[repeat(f2)||f1],

[0018] Then, the final global feature f is obtained through the convolution layer Conv2 composed of three MLPs and the maximum pooling layer. The formula is:

[0019] f = maxpool(Conv2(f3));

[0020] The specific structure of decoder D is as follows: the global feature f obtained by input decoder E is passed through encoder D1 composed of k perspective generators to obtain the intermediate point cloud Y k , the formula is:

[0021]

[0022] Among them, Γ is the farthest point sampling operation, ∪ is the point-by-point splicing operation, For the output of each perspective generator, the formula is:

[0023]

[0024] Among them, v i is the view parameter of the adjacent view, f is the global feature obtained by the encoder, and g is the two-dimensional grid parameter generated by the two-dimensional grid generator.

[0025] After passing through the encoder D2 composed of j perspective generators, the final complete point cloud Y is obtained, and the formula is:

[0026]

[0027] Y ji =MLPs[v i ,Y k ,g]

[0028] In the point cloud completion method based on the view generator, the generated training set in step S3 includes residual point cloud and complete point cloud pairs, the optimization algorithm is the gradient descent method, and the loss function uses the chamfer distance (CD) and the earth mover's distance (EMD), which are defined as:

[0029]

[0030]

[0031] Among them, S1 and S2 are any two point cloud sets. It should be noted that the EMD distance requires the same number of points in the two point cloud sets, while the number of points in the two CD subsets can be arbitrary. The loss function formula of PV-Net is:

[0032]

[0033] Among them, Y k is the output of each view generator in the first stage of the decoder, Y j is the output of each view generator in the second stage of the decoder, Y is the final output complete point cloud of the network, is true value.

[0034] Beneficial effects of the present invention:

[0035] 1. Aiming at the problem that the point cloud completion method is prone to produce overall shape coarse estimation errors and poor local detail recovery when dealing with point cloud missing caused by perspective occlusion, the present invention proposes a point cloud completion method based on perspective generator, which effectively improves the completion quality of the overall point cloud and local details.

[0036] 2. The present invention designs a perspective-based point cloud completion network PV-Net by adopting an encoder-decoder based structure, in which the decoder utilizes the encoder and perspective information to effectively process the incomplete point clouds under different perspectives, restore the missing point clouds by gradually expanding the perspective of the incomplete point clouds, and fully explore the relationship between the incomplete point clouds under different perspectives, so as to obtain a more complete point cloud with stronger integrity, while retaining the local details of the point cloud, achieving a better completion effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is the whole flow chart of the present invention;

[0038] Figure 2 is a perspective selection diagram in an embodiment of the present invention;

[0039] Figure 3 It is the network structure diagram of PV-Net of the present invention;

[0040] Figure 4 It is a structural diagram of the encoder of the present invention;

[0041] Figure 5 It is a structural diagram of a decoder of the present invention;

[0042] Figure 6 8 types of point cloud simulation result diagrams provided by the embodiments of the present invention;

[0043] Figure 7 It is a simulation result diagram of the aircraft point cloud at different viewing angles provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. It should be noted that the words "front", "rear", "left", "right", "upper" and "lower" used in the following description refer to directions in the accompanying drawings, and the words "inner" and "outer" refer to directions toward or away from the geometric center of a specific component, respectively.

[0045] A point cloud completion method based on a view generator in this embodiment, the algorithm flow chart is as follows Figure 1 As shown. Specifically including the following steps:

[0046] Step 1: Dataset generation: Use the invisible point removal algorithm to process the dataset and generate the residual point cloud P v , including three modules: perspective selection, spherical mapping and visible point selection.

[0047] The specific method of viewing angle selection is: the twelve vertices of the unit icosahedron are used as the viewing angle C, and the viewing angle direction is toward the coordinate origin (0,0,0), such as Figure 2 As shown; the specific steps of the spherical mapping operation are: arbitrarily select the viewing angle c k ∈C, any point p in the point cloud P i Mapped onto a sphere with radius r The spherical mapping formula is:

[0048]

[0049] The specific steps of the visible point selection operation are: select the point from the perspective c k and mapping sets The convex hull points in the set are combined to obtain the residual defect cloud P containing only visible points. v .

[0050] Step 2: Network model construction: Figure 3 As shown in the figure, based on the encoder-decoder structure, a viewpoint-based point cloud generator is proposed to construct the PV-Net network model. It consists of an encoder E and decoders D1 and D2. Encoder E is responsible for extracting the global feature f of the input residual point cloud, and decoders D1 and D2 are responsible for decoding the global feature f to obtain a complete point cloud.

[0051] The specific structure of encoder E is as follows Figure 4 As shown, input residual cloud P v First, the convolution layer Conv1 composed of three multi-layer perceptrons (MLP) is used to obtain 256-dimensional features, and then the global feature f2 is obtained through the maximum pooling layer. The formula is:

[0052] f2 = maxpool(Conv1(P v )),

[0053] Repeat the global feature f2 256 times in dimension 2 and concatenate it with f1 to obtain the mixed feature f3. The formula is:

[0054] f3=[repeat(f2)||f1],

[0055] Then, the final global feature f is obtained through the convolution layer Conv2 composed of three MLPs and the maximum pooling layer. The formula is:

[0056] f = maxpool(Conv2(f3));

[0057] The specific structure of the decoder is as follows Figure 5 As shown, the global feature f obtained by the input decoder E passes through the encoder D1 composed of k perspective generators to obtain the intermediate point cloud Y k , the formula is:

[0058]

[0059] Among them, Γ is the farthest point sampling operation, ∪ is the point-by-point splicing operation, For the output of each perspective generator, the formula is:

[0060]

[0061] Among them, v i is the view parameter of the adjacent view, f is the global feature obtained by the encoder, and g is the two-dimensional grid parameter generated by the two-dimensional grid generator.

[0062] After passing through the encoder D2 composed of j perspective generators, the final complete point cloud Y is obtained, and the formula is:

[0063]

[0064] Y ji =MLPs[v i ,Y k ,g].

[0065] Step 3: Model training: Use the generated training set and the optimization algorithm to reduce the loss function so that the network converges and completes the point cloud completion. The training set includes the residual point cloud and the complete point cloud pair. The optimization algorithm is the gradient descent method. The loss function uses the chamfer distance (CD) and the earth mover's distance (EMD), which are defined as:

[0066]

[0067]

[0068] Among them, S1 and S2 are any two point cloud sets. It should be noted that the EMD distance requires the same number of points in the two point cloud sets, while the number of points in the two CD subsets can be arbitrary. The loss function formula of PV-Net is:

[0069]

[0070] Among them, Y k is the output of each view generator in the first stage of the decoder, Y jis the output of each view generator in the second stage of the decoder, Y is the final output complete point cloud of the network, is true value.

[0071] The training set includes residual point cloud and complete point cloud pairs. The optimization algorithm is gradient descent, and the loss function uses Chamfer Distance (CD) and Earth Mover's Distance (EMD), which are defined as:

[0072]

[0073]

[0074] Among them, S1 and S2 are any two point cloud sets. It should be noted that the EMD distance requires the same number of points in the two point cloud sets, while the number of points in the two CD subsets can be arbitrary. The loss function formula of PV-Net is:

[0075]

[0076] Among them, Y k is the output of each view generator in the first stage of the decoder, Y j is the output of each view generator in the second stage of the decoder, Y is the final output complete point cloud of the network, is true value.

[0077] The technical effects of the present invention are further described below in conjunction with simulation experiments.

[0078] The simulation experiment of the present invention adopts the present invention and the existing point cloud completion network to complete the test set in the PCN data set, and obtains the qualitative result diagram of point cloud completion, such as Figure 6 shown.

[0079] In the simulation experiment, the existing technologies used include: ① the point cloud completion network proposed by Wentao Yuan et al. in their paper “Pcn: Point completion network. In 2018 International Conference on 3D Vision (pp. 728-737). IEEE.”; ② the point cloud completion network proposed by Yaoqing Yang et al. in their paper “FoldingNet: Point Cloud Auto-encoder via Deep Grid Deformation. In 2018 IEEE / CVF Conference on Computer Vision and Pattern Recognition (pp. 206-215). IEEE”; ③ the point cloud completion network proposed by Lyne P. Tchapmi et al. in their paper “TopNet: Structural Point Cloud Decoder. 2019 IEEE / CVF Conference on Computer Vision and Pattern Recognition (pp. 383-392). IEEE”.

[0080] Combine the following Figure 6 The simulation diagram of the present invention is further described. Figure 6 The first and seventh columns are the incomplete input point cloud and the complete true value point cloud processed to a fixed point number of 512 and 2048 points according to step 1, respectively. The second to fifth columns are the completion results of the prior art with an output point resolution of 2048 points, respectively. The sixth column is the completion result of the method of the present invention with an output point resolution of 2048 points. Figure 6 It can be clearly seen from the qualitative test results that compared with the completion results of the present invention, the prior art cannot well preserve the detailed prominent parts of the input incomplete point cloud, the reasoning effect on the incomplete parts is not accurate enough, and the point distribution of the completed point cloud is not uniform enough.

[0081] Using the chamfer distance mentioned in step 3 as the evaluation criterion for the point cloud completion results, the chamfer distances between all completion results and the true point cloud are plotted in Table 1.

[0082] Table 1 Quantitative test results of the present invention and the prior art in the simulation experiment

[0083]

[0084] It can be seen from Table 1 that the chamfer distance of the completion result of the present invention when the output resolution is 2048 points is smaller than that of the prior art, which proves that the present invention can obtain a completed point cloud that is more similar to the true point cloud.

[0085] It should be noted that the above-mentioned embodiments are merely examples for the purpose of clear explanation, and are not intended to limit the implementation methods. It is not necessary and impossible to list all the implementation methods exhaustively. All components not specified in this embodiment can be implemented using existing technologies. For those skilled in the art, several improvements and modifications can be made without departing from the principles of the present invention, and these improvements and modifications should also be considered as the scope of protection of the present invention.

Claims

1. A point cloud completion method based on a view generator, characterized in that: The following steps are involved: S1. Dataset generation: Use the invisible point removal algorithm to process the dataset and generate the residual point cloud P. v ; S2. Network model construction: Based on the encoder-decoder structure, a viewpoint-based point cloud generator is proposed to construct a PV-Net network model; in step S2, the PV-Net network model consists of an encoder E and a decoder D, wherein the encoder E is responsible for extracting the global feature f of the input residual point cloud, and the decoder D is responsible for decoding the global feature f to obtain a complete point cloud; the specific structure of the encoder E is: the residual point cloud P is input v First, the convolution layer Conv1 composed of three multi-layer perceptrons is used to obtain 256-dimensional features, and then the global feature f2 is obtained through the maximum pooling layer. The formula is: f2=maxpool(Conv1(P v )), Repeat the global feature f2 256 times in dimension 2 and concatenate it with f1 to obtain the mixed feature f3. The formula is: f3=[repeat(f2)||f1], Then, the final global feature f is obtained through the convolution layer Conv2 composed of three MLPs and the maximum pooling layer. The formula is: f = maxpool(Conv2(f3)); The specific structure of the decoder D is as follows: the global feature f obtained by the input decoder E is passed through the encoder D1 composed of k view generators to obtain the intermediate point cloud Y k , the formula is: Among them, Γ is the farthest point sampling operation, ∪ is the point-by-point splicing operation, For the output of each perspective generator, the formula is: Among them, v i is the view parameter of the adjacent view, f is the global feature obtained by the encoder, and g is the two-dimensional grid parameter generated by the two-dimensional grid generator; After passing through the encoder D2 composed of j perspective generators, the final complete point cloud Y is obtained, and the formula is: S3, model training: Use the generated training set and the optimization algorithm to reduce the loss function so that the network converges and completes the point cloud completion.

2. A point cloud completion method based on a view generator according to claim 1, characterized in that: In the step S1, the invisible point removal algorithm specifically includes viewing angle selection, spherical mapping and visible point selection.

3. A point cloud completion method based on a view generator according to claim 2, characterized in that: The specific method of selecting the viewing angle is as follows: The twelve vertices of the unit icosahedron are used as the viewing angle C, and the viewing angle direction is toward the coordinate origin (0,0,0); the specific steps of the spherical mapping operation are: arbitrarily select the viewing angle c k ∈C, any point p in the point cloud P i Mapped onto a sphere with radius r The spherical mapping formula is: The specific steps of the visible point selection operation are: select k and mapping sets The convex hull points in the set are combined to obtain the residual defect cloud P containing only visible points. v .

4. The point cloud completion method based on a view generator according to claim 1, characterized in that: In step S3, the generated training set includes residual point cloud and complete point cloud pairs, the optimization algorithm is gradient descent method, and the loss function uses chamfer distance and earth mover distance, which is defined as: Among them, S1 and S2 are any two point cloud sets; it should be noted that the EMD distance requires the same number of points in the two point cloud sets, and the number of points in the two CD subsets can be arbitrary; the loss function formula of PV-Net is: Among them, Y k is the output of each view generator in the first stage of the decoder, Y j is the output of each view generator in the second stage of the decoder, Y is the final output complete point cloud of the network, is true value.

Images