A point cloud completion method based on multi-scale feature extraction

Abstract

The application discloses a point cloud completion method based on multi-scale feature extraction, comprising the following steps: randomly selecting a point from an original point cloud as an initial point, then calculating the distance from the remaining original points to the nearest farthest point in a loop, and then selecting the point with the largest distance from the points as the next farthest point; adopting an improved multilayer perceptron to extract the global information of a low-resolution point cloud, simultaneously adopting EdgeConv to extract the detailed information of a high-resolution original point cloud, and fusing the two through the multilayer perceptron in a recursive cascade mode, so as to obtain a latent feature vector; taking the latent feature vector as an input to generate a rough point cloud with a point number of 1024; conveying to a shared multilayer perceptron, and finally adding the rough point cloud when outputting. The point cloud completion method based on multi-scale feature extraction can pay attention to global features while considering the topological information between points, guarantee the comprehensiveness and integrity of the features, narrow the gap between points, and enhance the generalization ability of the model.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of visual processing, and particularly relates to a point cloud completion method based on multi-scale feature extraction. BACKGROUND

[0002] With the development of the times and the progress of science, three-dimensional vision technology has become one of the current research hotspots due to its unique advantages, and is widely used in computer vision, robot technology, remote sensing and other fields. Among the many types of three-dimensional data description, three-dimensional point cloud has the characteristics of small data volume and fine object presentation, so it is often used as an upstream task for three-dimensional target classification, three-dimensional scene semantic segmentation and other researches. However, due to the limitations of sensor resolution, self-occlusion and sparsity of data points, the acquired data becomes incomplete during the acquisition of point cloud data. Therefore, it is crucial to complete the input incomplete point cloud data with high quality. The point cloud completion method can be roughly divided into geometry-based method, alignment-based method and learning-based method. Due to the limitations of the first two completion methods, more and more researchers tend to use learning-based completion methods. The learning-based completion method directly learns the structural features of the incomplete point cloud according to its geometric information, so as to train a model with shape completion ability. In order to make the shape of the completed point cloud variable and the surface smoother, Yang et al. introduced a folding-based decoder under the premise of extracting global features of the model, which outputs the completed point cloud by superimposing 2D grids on the 3D surface of the point cloud. Liu et al. made further research on this basis, generated a rough point cloud framework using a deformation network, and then refined it using a sampling algorithm and a residual network to generate the final complete point cloud. Tchapmi et al. proposed to use a hierarchical tree structure to complete the point cloud without assuming any topological structure. Since the missing structures of incomplete point clouds are different, most learning-based completion methods only extract global features of single-resolution original point clouds, thereby losing detailed information. Therefore, how to make the generated point cloud have global structure while having detailed information, and ensure that the point cloud features have integrity and comprehensiveness, is the key point of point cloud completion research, and is also the technical research focus of the present application.

[0003] The prior art one adopts a point cloud completion mechanism of compression first and then restoration, that is, a classic encoder-decoder structure framework, which realizes completion by learning point features through an encoder and generating a dense point cloud by a decoder. In the encoder, the geometric information of the input incomplete point cloud is globally encoded by a shared multi-layer perceptron (Shared MLP) and a symmetric function to form a feature vector and input the decoder; in the decoder, the input feature vector is decoded into a complete point cloud skeleton, and the final dense complete point cloud is generated with the point coordinates as the center. In addition, according to the research of the prior art one, a point cloud completion method from sparse to dense is adopted, which maps the input incomplete point cloud to the shared multi-layer perceptron point by point to obtain the point feature vector and forms the point feature matrix, and then uses the maximum pooling layer to collect the point features to generate the latent feature vector. In order to effectively extract the global features of the original point cloud, the above operation is stacked twice to obtain enhanced features, form a multi-dimensional global feature vector and input the decoder. The decoder stage uses a fully connected layer to decode the input global feature vector into a rough point cloud framework, and then combines a folding network to integrate the global feature vectors and input them into the shared multi-layer perceptron to output a dense complete point cloud.

[0004] The prior art one only focuses on global features when learning the geometric features of the input point cloud, and ignores the structure information of the original incomplete point cloud in the process of generating a dense point cloud, so it lacks attention to detail information. Only single-resolution incomplete input is used for feature extraction, and there is a lack of consideration of the multi-faceted features, resulting in the extracted features being single. The shared multi-layer perceptron is used for feature extraction, which ignores part of the hierarchical point cloud features, resulting in the lack of completeness of the point cloud features.

[0005] The second prior art mainly adopts an iterative framework to realize point cloud completion by simulating the moving behavior of objects and moving points one by one. That is, the moving distance of all points is constrained, and a unique point moving path is predicted for each point to form a final complete point cloud. According to the research of the second prior art, a way of learning multi-step point moving paths is adopted to realize point cloud completion. In order to extract global features with detailed information, the points selected by the farthest point sampling method are used as the center coordinates for grouping by using the ball query method, and then the local features are extracted by using a multi-layer perception machine. Finally, the operation of repeated sampling, grouping and feature extraction is performed to obtain global features. Then the global features are propagated to the point-by-point feature module, the path information of the previous point movement is stored, and the next step position of each point is inferred to obtain the current reserved feature, i.e. the point-by-point feature. Finally, the point-by-point feature process is iteratively calculated, and the third output result is sent to the multi-layer perception machine to output the point-by-point displacement and be combined with the original point cloud to generate the final complete point cloud. The second technology calculates the point-by-point displacement by iteration, selectively retains or ignores the time sequence information by using a recurrent neural network, thereby constraining the geometric structure of the generated point cloud and enhancing the completion effect.

[0006] The second prior art uses the point-by-point displacement method for completion, but the generated effect diagram has problems such as large gaps between points, uneven distribution of points, etc. Although the sampling and grouping operations are added to the multi-layer perception machine to make the completion method pay attention to local features, it ignores the spatial stereo geometric relationship between points and is also affected by the density of points. The above two technical solutions only focus on global features, causing the loss of detailed topological structure between points, incomplete features extracted by a single resolution, incomplete features extracted by a multi-layer perception machine, and gaps between points in the generated point cloud. SUMMARY

[0007] Based on the deficiencies of the above prior art, the technical problem solved by the present application is to provide a point cloud completion method based on multi-scale feature extraction, which pays attention to global features while considering the topological information between points, ensures the comprehensiveness and integrity of the features, reduces the gap between points, and enhances the generalization ability of the model.

[0008] In order to solve the above technical problems, the present application realizes the following technical scheme: the present application provides a point cloud completion method based on multi-scale feature extraction, comprising:

[0009] Step S1, randomly select a point from the original point cloud as an initial point, then calculate the distance from the remaining original points to the nearest farthest point in the loop, then select the point with the largest distance from these points as the next farthest point;

[0010] Step S2: While using an improved multilayer perceptron to extract global information of low-resolution point cloud, EdgeConv is used to extract detailed information of high-resolution original point cloud. The two are then fused through a multilayer perceptron in a recursive cascade manner to obtain the latent feature vector.

[0011] Step S3: Take the latent feature vector as input, and then use a three-layer fully connected layer with dimensions of 2048, 1024, and 1024 to map it onto a 1024×3 output unit. Finally, reshape it into a 1024×3 matrix to generate a coarse point cloud with 1024 points.

[0012] Step S4: The original residual point cloud and the coarse point cloud are downsampled to 512 points using farthest point sampling and mirroring operations respectively, and then the two are spliced ​​together to form a synthetic point cloud. Taking each point of the synthetic point cloud as the center point, 16 points are generated around each point through folded 2D mesh, and the extended latent feature vector is embedded. Combined with the connection method of the residual network, the synthetic point cloud, 2D mesh and latent feature vector are concatenated and sent to the shared multilayer perceptron. Finally, the coarse point cloud is added at the output.

[0013] Furthermore, in step S2, the improved multilayer perceptron MCMLP is used to extract two low-resolution point cloud features respectively, and then the MLP is used to fuse them.

[0014] Furthermore, the technical steps of MCMLP are as follows:

[0015] Step 21: Map the input point cloud to different feature dimensions, and perform non-linear operations on each layer using batch normalization and ReLU activation function;

[0016] Step 22: Concatenate the obtained multidimensional features in the last layer;

[0017] Step 23: Feed the concatenated features into global max pooling and global average pooling in parallel to aggregate semantic features simultaneously;

[0018] Step 24: Concatenate pooled features to form a latent feature vector.

[0019] Furthermore, in step S2, local detail information of the point cloud is extracted using EdgeConv. The extraction method is to stack two EdgeConv layers and then use max pooling and average pooling in parallel to aggregate features to form local features.

[0020] Furthermore, the technical steps of EdgeConv are as follows:

[0021] Step 25: Use the K-nearest neighbor algorithm to find the answer at a point x. iSelect the nearest k points around it and form directed edges;

[0022] Step 26: Calculate the edge feature e of the point and its nearest k points ij = h(x i ,x j ), where h is a feature extraction function;

[0023] Step 27: Dynamically update the point x i feature value using the aggregation function.

[0024] From the above, the point cloud completion method based on multi-scale feature extraction of the present application has the following beneficial effects:

[0025] 1. For the problem of single resolution feature extraction, the present application uses twice downsampling to obtain multi-scale point cloud, so that the method pays attention to more representative feature points, ensures the comprehensiveness of the features, and improves the robustness of the method.

[0026] 2. The present application designs a new feature extraction module (MCMLP), which can retain complete point cloud features, enhance the representation ability of point cloud data, and improve the completion effect of the method.

[0027] 3. The present application proposes to extract low-resolution point cloud as global feature using MCMLP, and at the same time, extract high-resolution point cloud as local feature using EdgeConv for embedding. This method of parallelly mining point cloud features by taking different feature extraction methods effectively enhances the extraction ability of the method for global and local features of point cloud.

[0028] 4. The present application combines the original point cloud for point cloud refinement in the process of generating dense point cloud, preventing local details from being affected by incomplete point cloud, and thus improving the completion performance of the method.

[0029] The above description is only a summary of the technical solutions of the present application, in order to more clearly understand the technical means of the present application, which can be implemented according to the content of the specification, and in order to make the above and other purposes, features and advantages of the present application more simple and easy to understand, the following will be described in detail in combination with the preferred embodiments and the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0031] Figure 1A point cloud completion framework based on multi-scale feature extraction of the application is shown in the figure;

[0032] Figure 2 A structure diagram of the feature extraction module is shown in the figure;

[0033] Figure 3 A structure diagram of the MCMLP is shown in the figure;

[0034] Figure 4 A structure diagram for generating a dense point cloud is shown in the figure;

[0035] Figure 5 A visualization comparison effect diagram of the application and other methods in the ShapeNet dataset is shown in the figure;

[0036] Figure 6 A visualization comparison effect diagram of the application and other methods in the KITTI dataset is shown in the figure. DETAILED DESCRIPTION

[0037] The specific implementation of the point cloud completion method based on multi-scale feature extraction of the application will be described in detail below with reference to the accompanying drawings.

[0038] The main technical solution of the application is shown in the figure Figure 1 , which mainly includes feature point sampling, point cloud feature extraction, coarse point cloud generation, dense point cloud generation, and final performance evaluation.

[0039] 1. Feature point sampling

[0040] In a point cloud, a feature point is usually explained as a point cloud framework with fewer points. To effectively obtain feature points and ensure feature integrity, the application adopts the farthest point sampling method to obtain a multi-scale point cloud with a resolution of , where N is the number of original point clouds, and the value is 2048.

[0041] The application first randomly selects a point from the original point cloud (i.e., the incomplete point cloud) as an initial point, then calculates the distance from the remaining original points to the nearest farthest point in the loop, and then selects the point with the largest distance from these points as the next farthest point, where the number of loop iterations is the number of points required for the actual model to be down-sampled.

[0042] 2. Feature extraction

[0043] To better mine the global and local features of the point cloud, the application proposes to use an improved multi-layer perceptron to extract the global information of the low-resolution point cloud while using EdgeConv to extract the detailed information of the high-resolution original point cloud, and to fuse the two through a multi-layer perceptron in a recursive form, so as to obtain a latent feature vector, and further lay a foundation for the generation of a complete point cloud. The structure diagram of the specific feature extraction module is shown in the figure 2.

[0044] (1) Global feature extraction network-MCMLP

[0045] The present application designs an improved multilayer perceptron, called MCMLP (Multilayer Combined Multi-layer Perceptron), which effectively makes up for the deficiency of traditional multilayer perceptron focusing only on the highest dimensional features, and guarantees feature integrity. The present application uses MCMLP to extract features of two low-resolution point clouds respectively, and uses MLP for fusion. The MCMLP structure diagram is as shown in Figure 3 The technical steps of MCMLP are as follows:

[0046] First, the input point cloud is mapped to different feature dimensions f = [64, 64, 128, 256, 512, 1024], and batch normalization (BN) and ReLU activation function are used for nonlinear operation at each layer; then the obtained multi-dimensional features are concatenated at the last layer; then the concatenated features are sent into global maximum pooling and global average pooling in parallel to simultaneously aggregate semantic features; finally, the pooled features are concatenated to form a latent feature vector. The calculation formula is expressed as:

[0047]

[0048] Where i = 0, 1, 2, 3, 4, 5, f i is an element in f, GMP is global maximum pooling, GAP is global average pooling, is concatenated, so the feature channel number of MCMLP is 2048.

[0049] (2) Local feature extraction network-EdgeConv

[0050] The present application uses EdgeConv to extract local detailed information of point cloud, and its extraction method is to stack two layers of EdgeConv, and then use maximum pooling and average pooling in parallel for feature aggregation to form local features. The calculation expression is:

[0051]

[0052] Where V' is the feature extracted by using EdgeConv twice, GMP is global maximum pooling, GAP is global average pooling, and P N＝2048 represents an input point cloud with 2048 points.

[0053] The technical steps of EdgeConv are as follows:

[0054] First, the K-neighbor algorithm is used to find the K-neighbor points of a point x iSelect the nearest k points around it and form directed edges (i,j i1 ),...,(i,j ik ), then calculate the edge feature e ij = h(x i ,x j ) of the point and its nearest k points, where h is the feature extraction function; finally, the feature value of the point x i is dynamically updated by using the aggregation function.

[0055] 3 Coarse point cloud generation

[0056] The present application mainly adopts a full connection network to generate a coarse point cloud. First, the latent feature vector is taken as input, then it is mapped to the output unit of 1024x3 by using three layers of full connection with the dimension of 2048, 1024, 1024, and finally reshaped into a 1024x3 matrix to generate a coarse point cloud with the number of points of 1024. The calculation expression is:

[0057] P coarse = R(FC(V)) (3)

[0058] Where FC represents full connection operation, R represents matrix reset, and V represents latent feature vector.

[0059] 4 Generation of dense point cloud

[0060] The structural diagram for generating a dense point cloud is shown in Figure 4 . First, the farthest point sampling and mirror operation are used to downsample the original incomplete point cloud and the coarse point cloud into 512 points respectively, and the two are spliced to form a synthetic point cloud. Then, taking each point of the synthetic point cloud as the center point, 16 points are generated around each point through the folded 2D grid, and the expanded latent feature vector is embedded. Next, combined with the connection mode of the residual network, the synthetic point cloud, 2D grid and latent feature vector are connected in series and delivered to the shared multilayer perceptron, and finally the addition coarse point cloud is output. In the optimization process, the following loss function is adopted for training:

[0061]

[0062] Loss = d CD1 (P coarse ,P gt )+μd CD2 (P dense ,P gt ) (5)

[0063] Formula (4) is a chamfer distance (CD) calculation formula, S1 and S2 respectively represent the generated point cloud and the ground true point cloud. Formula (5) is a joint loss function of the method training, which is mainly composed of CD linear combination. The first part of formula (5) is to calculate the CD value between the coarse point cloud and the ground true point cloud, the second part is to calculate the CD value between the dense point cloud and the ground true point cloud, and μ is a hyperparameter that balances the above two parts.

[0064] 5Evaluation index

[0065] The CD, earth mover's distance (EMD), F-score and consistency are selected as the evaluation indexes of the point cloud completion method with the highest universality.

[0066] As shown in Table 1, the CD value, EMD value and F-score value of the method of the application on the ShapeNet dataset are 0.0109, 0.0242 and 0.6339 respectively, which are higher than those of the comparative models without adding the original incomplete point cloud when using MLP, PointNet++ and refining the coarse point cloud. The experimental results verify the effectiveness of the mentioned MCMLP, EdgeConv and the addition of the original incomplete point cloud in the refining process.

[0067] Table 1 Experimental results on the ShapeNet dataset

[0068]

[0069] As shown in Table 2 and Table 3, the quantitative comparison results of the application and other methods on the ShapeNet and KITTI datasets are shown. In Table 2, compared with FoldingNet, PCN, PointFCAE and PMPNet, the CD value is reduced by 26.4%, 9.4%, 12.4% and 2.8% respectively; the EMD value is reduced by 68.6%, 8%, 34.9% and 52% respectively; and the F-score value is increased by 19.3%, 7.5%, 11% and 6.7% respectively. In Table 3, compared with FoldingNet, PCN, PointFCAE and PMPNet, the output consistency is reduced by 22.4%, 10.6%, 5.6% and 39.8% respectively. The experimental results show that the method of the application is much better than other methods, and also verify the generalization of the method of the application.

[0070] Table 2 Comparison experimental results of other methods on the ShapeNet dataset

[0071]

[0072] Table 3 comparative experimental results of KITTI dataset with other methods

[0073]

[0074] As shown in Figure 5 , Figure 6 qualitative comparison results of the present application and other methods on ShapeNet and KITTI dataset. As can be seen from the figure, compared with other methods, the complete point cloud edge generated by the present application is clearer, the surface is smoother, and the point distribution is more uniform. This shows that the method has better grasp of the details of the original point cloud while controlling the global structure, and the shape filled is more reasonable. This fact also verifies the robustness and feasibility of the present application method.

[0075] Finally, it should be noted that the above is the preferred embodiment of the present application, of course, cannot be limited to the scope of the present application, it should be noted that for those skilled in the art, without departing from the principles of the present application, can make a number of improvements and changes, these improvements and changes are also considered to be within the scope of the present application.

Claims

1. A point cloud completion method based on multi-scale feature extraction, characterized in that, Comprise: Step S1, obtain a multi-scale point cloud with a resolution of from the original point cloud by taking the farthest point sampling method, where N is the number of the original point cloud, and the value is 2048; Step S2, adopt improved multi-layer perception to extract the global information of low-resolution point cloud, and adopt EdgeConv to extract the detailed information of high-resolution original point cloud, and fuse the two in a recursive form of cascade through multi-layer perception, so as to obtain a latent feature vector; Step S3, taking the latent feature vector as input, then using three layers of full connection with dimension of 2048, 1024, 1024 to map it to an output unit of 1024x3, and finally reshape it into a 1024x3 matrix to generate a rough point cloud with 1024 points; Step S4, adopt farthest point sampling and mirror operation to downsample the original incomplete point cloud and the rough point cloud into 512 points respectively, and splice the two to form a synthetic point cloud, take each point of the synthetic point cloud as a center point, generate 16 points around each point through a folded 2D grid, and embed the expanded latent feature vector, combine the connection mode of the residual network, concatenate the synthetic point cloud, 2D grid and latent feature vector, and deliver them to the shared multi-layer perception, and finally add the rough point cloud in the output; In the step S2, two low-resolution point cloud features are extracted by using the improved multi-layer perception MCMLP, and are fused by using the MLP; The technical steps of MCMLP are as follows: Step 21: map the input point cloud to different feature dimensions, and use batch normalization and ReLU activation function in each layer to perform nonlinear operation; Step 22: splice the obtained multi-dimensional features in the last layer; Step 23: parallelly send the spliced features into global maximum pooling and global average pooling to simultaneously aggregate semantic features; Step 24: concatenate the pooled features to form a latent feature vector.

2. The method of claim 1, wherein, In the step S2, the local detailed information of the point cloud is extracted by using EdgeConv, and the extraction method is to stack two layers of EdgeConv, and then use maximum pooling and average pooling in parallel to aggregate features to form local features.

3. The method of claim 2, wherein, The technical steps of EdgeConv are as follows: Step 25: Use K-Nearest Neighbor algorithm to select the k nearest points to a point x i and form directed edges Step 26: Calculate the edge feature e of this point and its nearest k points ij = h(x i , x j ), where h is a feature extraction function Step 27: Dynamically update point x with aggregation function i Eigenvalue.

Images