Graph convolution-based cascade refinement three-dimensional point cloud completion method

Abstract

The invention discloses a cascade refinement three-dimensional point cloud completion method based on graph convolution, and relates to the technical field of image processing methods. The method comprises the following steps: firstly, extracting global features and local features of an incomplete point cloud # imgabs0 # in a coarse point cloud generation module by adopting a double-path structure, and processing the global features and the local features to generate fusion features; secondly, predicting fusion features by using a full connection layer, and outputting a coarse point cloud # imgabs1 #; thirdly, the fused features and the coarse point cloud # imgabs2 # are input into a refined point cloud generation module for feature splicing, and the spliced features are processed to obtain a refined complete point cloud # imgabs3 #; according to the method, the complete point cloud with rich details and uniform distribution can be effectively generated, and the precision and authenticity of the completion effect are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing methods, and in particular to a cascaded refinement three-dimensional point cloud completion method based on graph convolution. Background Art

[0002] There are mainly the following point cloud completion methods in the prior art: 1) Convolution-based point cloud completion algorithm: With the rapid development of deep learning in the field of two-dimensional image processing, especially breakthroughs in tasks such as image completion and object recognition, deep learning technology has also been gradually applied to 3D shape completion tasks, and has made significant progress. Unlike two-dimensional images, 3D point clouds are highly irregular and disordered data, and traditional two-dimensional convolution operations cannot be directly applied. In order to overcome the contradiction between two-dimensional convolution and the disordered structure of 3D point clouds, the current common practice is to convert 3D point clouds into 2D grids or 3D voxels so that convolution operations can proceed smoothly.

[0003] Among them, the sparse voxel completion network SVCN can restore the three-dimensional surface and refine it using the sparse convolution U-Net by introducing the prior knowledge of the real-world 3D surface shape. This method effectively handles the disorder of the 3D point cloud and optimizes the completion result through the refinement process. However, the limitation of this method is that the results it generates cannot always reach high resolution. In order to further overcome this problem, GRNet came into being. GRNet can better handle disordered point clouds and generate complete point clouds using the geometric detail information in the input by using 3D grids as intermediate representations. However, GRNet is still limited by the resolution of its voxel representation and cannot generate a high-detail point cloud model. VE-PCN proposes to embed point clouds into regular voxel grids to help the generation of shape edges and the reconstruction of 3D shapes. VE-PCN fully mines the geometric information of 3D shapes through a multi-scale strategy, generates fine-grained complete point clouds, and can handle higher-resolution details, significantly improving the accuracy and detail quality of completion.

[0004] 2) Graph-based point cloud completion algorithm: Point cloud is a data form that is independent but has potential spatial relationships with surrounding neighboring points. Due to the unstructured characteristics of point cloud, it has a high degree of fit with graph structure. Graph structure can effectively mine the spatial and geometric relationships between points, thereby improving the processing effect of point cloud data. In order to generate fine local geometric structures, many point cloud completion networks based on graph convolution have been proposed. For example, the edge-aware point cloud completion network ECG uses a graph convolution network to perceive the edge features of point clouds. Its method includes two stages: first, a point cloud with a rough outline is generated through an encoder-decoder framework; then, local geometric details are refined using graph convolution to generate a more delicate point cloud shape. In addition, the graph-guided deformation network combines the graph-guided strategy with the grid deformation method to adaptively capture the changes in geometric details of different objects, thereby better restoring the fine-grained shape of the point cloud. This method makes full use of the relationship between points and improves the geometric consistency of the completion result. CRA-Net proposes a cross-region attention unit to calculate the spatial relationship between all local features based on the graph attention mechanism, thereby generating a point cloud with the clearest details. By deeply modeling cross-region features, this method can capture the global and local feature relationships of point clouds. FBNet further optimizes the point cloud completion process by combining feedback mechanisms with graph structures. FBNet uses layered graph networks and feedback-aware completion blocks to refine the completion results, gradually refining the details of the point cloud from sparse to dense, and improving the integrity of the overall geometric structure.

[0005] These methods introduce graph structures into point cloud completion and fully explore the potential relationship between points, which promotes the development of point cloud completion algorithms in geometric detail recovery and global shape preservation. These studies show that graph-based models have significant advantages in processing unstructured data and are an important direction in point cloud research.

[0006] (3) Point cloud completion algorithm based on generative adversarial Point cloud completion based on generative adversarial networks inputs the generated point cloud into the discriminator together with the real point cloud data, and refines the geometric structure of the generated point cloud through the feedback of the discriminator. This method optimizes the generator to make the generated point cloud more realistic in terms of shape, details and realism, thereby effectively improving the quality of point cloud completion. Rl-Gan-Net uses an autoencoder to capture the latent space representation of the point cloud, and uses GAN for training to generate missing areas that are highly similar to the real point cloud. The Cycle4Completion network explores the potential relationship from complete shapes to incomplete shapes by introducing a reverse learning strategy. This method uses a two-time synchronous conversion network to generate a high-fidelity point cloud model through bidirectional mapping learning during the completion process. This bidirectional mapping mechanism not only improves the detail recovery ability of point cloud completion, but also maintains a high degree of structural consistency and fluency when dealing with complex geometric shapes. ShapeInversion first introduced the inversion of a generative adversarial network to complete the shape. The network uses a GAN pre-trained on the complete shape to search for a latent code that gives a complete shape that can reconstruct a given partial input. ShapeInversion no longer requires paired training data and is able to incorporate the captured rich priors into the trained generative model. PointALCR combines a GAN-based and autoencoder-based framework with contrastive regularization to improve the representativeness and generative power of the completed point cloud. The AdversarialLatent GAN module is used to learn the latent space of input / target point cloud representations and extend the generative and discriminative capabilities of the GAN training procedure. Contrastive regularization ensures that the reconstructed items are close to the true values ​​and away from the incomplete input in the feature space.

[0007] These methods can not only restore the missing point cloud areas but also ensure the shape, details and authenticity of the generated point clouds by optimizing the adversarial training mechanism of the generator and the discriminator, making the application of point cloud data more extensive and efficient in different application scenarios.

[0008] 4) Transformer-based point cloud completion algorithm: PMP-Net adopts the idea of ​​gradual iteration, gradually completing the missing areas by utilizing the local features and global information of the point cloud in each round of iteration. This iterative mechanism can effectively capture the subtle changes and structures of the point cloud, thereby accurately restoring the missing geometric information. However, this method may face a trade-off between computational efficiency and accuracy when dealing with complex scenes. In order to further improve the performance of point cloud completion, PMP-Net++ is optimized on the basis of PMP-Net, especially the introduction of the Transformer enhanced learning network. This improvement significantly improves the accuracy and robustness of point cloud completion, especially in the processing of large-scale point cloud data, the Transformer's self-attention mechanism can better capture the long-distance dependencies in the point cloud, thereby improving the ability to extract global features. AnchorFormer learns a set of discriminant nodes (anchor points) to complete the object shape problem. By using anchor points to represent different local geometric patterns, AnchorFormer can establish reasonable geometric connections between different local areas, thereby achieving efficient point cloud completion. ProxyFormer divides the point cloud into input and prediction, and each part transmits information through its proxy. In order to better perceive the location of missing points, a missing part sensitive Transformer is designed to transform the random normal distribution into reasonable location information, and proxy alignment is used to refine the missing proxy. GeoFormer designs a feature enhancement point generator to integrate Figure 1 We extract image features from canonical coordinate maps and align them with pure point features to enhance global geometric features, while enhancing local details via cross-attention between multi-scale features extracted from partial inputs and features derived from previously estimated points.

[0009] Although existing point cloud completion methods have made some progress in completion accuracy and speed, many methods still do not deeply explore the local topological features of point clouds, which may cause the relationship between points to be lost during the feature extraction process, resulting in the loss of details of the completion effect. Summary of the invention

[0010] The technical problem to be solved by the present invention is how to provide a cascaded refinement three-dimensional point cloud completion method based on graph convolution that can effectively generate a complete point cloud with rich details and uniform distribution, thereby improving the accuracy and authenticity of the completion effect.

[0011] In order to solve the above technical problems, the technical solution adopted by the present invention is: a cascaded refinement three-dimensional point cloud completion method based on graph convolution, comprising the following steps: First, a dual-path structure is used in the coarse point cloud generation module to extract the residual point cloud. The global and local features are obtained by fusing the global and local features, and the fused features are predicted using the fully connected layer to output the coarse point cloud. ; Secondly, the fusion features and the coarse point cloud Input to the refined point cloud generation module for feature splicing, and process the spliced ​​features to obtain a refined and complete point cloud .

[0012] The beneficial effect of adopting the above technical solution is that the method described in this application adopts a multi-stage processing strategy, decomposing the point cloud completion task into multiple processing stages, and gradually refining the generated point cloud. Through the cascade structure, it is possible to make full use of the local information and global information generated in each stage, so as to effectively generate a complete point cloud with rich details and uniform distribution, thereby improving the accuracy and authenticity of the completion effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0014] Figure 1 is an overall flow chart of the method described in the embodiment of the present invention; Figure 2 is a framework diagram of a coarse point cloud generation module in the method according to an embodiment of the present invention; Figure 3 It is a branch framework diagram of local feature extraction in the method described in the embodiment of the present invention; Figure 4 is a processing flow chart of the edge graph convolution module in the method according to an embodiment of the present invention; Figure 5 is a comparison diagram between the existing sampling method and graph convolution sampling in an embodiment of the present invention; Figure 6 It is a framework diagram of a refinement point cloud generation module in the method according to an embodiment of the present invention; Figure 7 It is a framework diagram of the global self-enhancement module in the method described in the embodiment of the present invention. DETAILED DESCRIPTION

[0015] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0016] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0017] In order to generate a complete point cloud with rich details and uniform distribution, the method proposed in this application is based on a two-stage completion architecture and processes the point cloud completion task in a coarse-to-fine manner. The overall process of this method is as follows Figure 1 As shown, this method mainly includes two parts: coarse point cloud generation and refined point cloud generation.

[0018] In general, the embodiment of the present invention discloses a cascaded refinement three-dimensional point cloud completion method based on graph convolution, comprising the following steps: First, a dual-path structure is used in the coarse point cloud generation module to extract the residual point cloud. The global and local features are obtained by fusing the global and local features, and the fused features are predicted using the fully connected layer to output the coarse point cloud. ; Secondly, the fusion features and the coarse point cloud Input to the refined point cloud generation module for feature splicing, and process the spliced ​​features to obtain a refined and complete point cloud .

[0019] Coarse point cloud generation module: This application adopts a dual-path structure in the coarse point cloud generation module, and the residual point cloud Input to the global and local feature extraction branches for processing, respectively, to obtain features and Features , the feature and Features Obtain features after fusion , fully extract the residual cloud features to obtain the features , and then use three fully connected layers to decode the information and predict the rough point cloud Its specific structure and processing flow are as follows Figure 2 shown.

[0020] The global feature extraction module uses the PointNet feature extraction method. The input point cloud size of this module is , generating features as the output of this module.

[0021] In the local feature extraction branch, edge graph convolution module and graph convolution sampling are used to fully extract the multi-scale geometric features of the local point cloud. The structure diagram of the local feature extraction branch is as follows: Figure 3As shown in Figure 2, the edge graph convolution module is used to extract the local features and neighborhood relationships of the point cloud, so that the geometric characteristics of the input data can be better captured. The graph convolution sampling module is used to retain richer multi-scale features, including local features and global structural features, while reducing the computational complexity.

[0022] The local feature extraction branch first uses the first edge graph convolution module to transform the input residual cloud Mapping to high-dimensional space gives , using graph convolution sampling to downsample the input and extracted features by 4 times, we get Next, the second edge graph convolution module is used to extract the sampled features and expand the dimensions to obtain , use the second graph convolution sampling module to perform 2 times downsampling on the sampled points and features, and get Subsequently, the third edge graph convolution module is used to extract local features from the downsampled features to obtain the hierarchical features under 8 times downsampling. .

[0023] Through multi-level downsampling and gradual feature extraction, the model not only learns local details, but also gradually aggregates the global contour information of the point cloud. In the case of the last graph convolution sampling, the extracted features can describe the global contour information of the entire point cloud and become the features representing the overall contour of the entire point cloud. Finally, the aggregated features are obtained. .

[0024] Edge graph convolution module: In the case of sparse input, graph convolution has good feature perception ability. The edge graph convolution module obtains the edge features between a point and its neighboring points by constructing a local neighborhood graph, and then aggregates the edge features to update the feature value of the point. The local neighborhood graph of each layer is composed of the neighborhood of each node dynamically selected by the distance metric, so it can capture local neighborhood information while maintaining the invariance of point arrangement. Figure 4 This is the structure diagram of the edge graph convolution module. This application uses the K-nearest neighbor (KNN) algorithm to select the K points closest to a point and form a directed edge, then calculates the edge features between the point and its K nearest points, and finally uses the aggregation operation to dynamically update the features of the known points.

[0025] Graph convolution sampling module: This application uses a graph convolution sampling module to adaptively sample some input points and their features. Specifically, by weighting the neighbors of the sampling points in the geometric space and feature space, pooled points and features are generated.

[0026] This method uses the geometric and semantic relationships between a single point cloud and its neighborhood as the basis for learning weights, which can better understand the location information of the point. The input point cloud is represented as , the corresponding feature is , where N represents the number of points and D represents the dimension of the feature. The weight matrix w is learned from the relationship between the key point and its neighbors and is defined as: (1) in, and It is a nonlinear mapping function that can learn the relationship between point clouds and features. and represents key points and their corresponding features, and Represents one of the neighborhood points of the key point and its features. The points and features finally obtained by sampling can be expressed as: (2) (3) in It is an undirected graph The set of edges, the undirected graph is constructed by KNN combination, is the weight of the neighborhood points obtained by calculation, is the weight of the calculated neighborhood feature.

[0027] When the weight of a point is 1, the point becomes a new sampling point. When a point may affect the prediction of the coarse point cloud, the weight of the point is set to a minimum value so that the feature of the point will not be considered during the pooling process. By introducing the graph convolution sampling module, the weight of the node is better considered during the sampling process, thereby improving the performance and robustness of the point cloud completion task. Figure 5 As shown in Figure 2, previous sampling methods did not consider the weight of the nodes, while graph convolution sampling calculated the different contributions of each node during the sampling process.

[0028] (2) Refined point cloud generation module: The refined point cloud generation module receives the merged features and coarse point cloud As input, output is a refined complete point cloud , the structure diagram is as follows Figure 6 shown.

[0029] Coarse point cloud and Features Generate two point feature matrices respectively and , and then perform point-by-point splicing to obtain the feature In this way, each point feature in the matrix maintains both the local details of the shape and the global geometric structure of the shape. Next, the global self-enhancement module and the edge graph convolution module are alternately stacked to gradually upsample the point features. Then, the upsampled point feature matrix is ​​connected point by point With point feature matrix Perform splicing to obtain the feature matrix Finally, the comprehensive features are processed and the corresponding feature vectors are converted into three-dimensional position coordinates to obtain the refined complete point cloud. .

[0030] Global self-enhancement module: The global self-enhancement module introduces self-attention to establish the spatial relationship between points to capture global information. The structure is as follows Figure 7 It accepts the input feature matrix X and the upsampling ratio u, where X is of size Matrix. Multi-head self-attention is used to integrate information from different points of X in the form of residuals to obtain the feature matrix F: (4) (5) in The execution method is similar to that of transformer. and is the linear transformation matrix. After the linear transformation operation, the dimension of the point increases u times. Similar to the local cross attention mechanism, the output of the global self-enhancement module is (6) Among them, QJ represents the global self-enhancement module, the output of which is a size The matrix of .

[0031] The loss function is a measure of the gap between the output point cloud and the real point cloud, and is used to reduce the difference between the real value and the generated point cloud. This application uses Chamfer Distance (CD) to measure the distance between two point clouds. The smaller the distance, the closer and more similar the two point clouds are. This application uses multi-stage completion loss to increase the proportion of feature points, better focus on feature points with high-frequency information, and has excellent results in completing the details of the point cloud. The multi-stage completion loss function is shown in the formula: (7) Among them, let the coarse point cloud and the output point cloud generated by the refined point cloud be recorded as At the same time, the ground truth point cloud is downsampled by FPS to obtain three sub-clouds , and their densities are same. Represents the contribution weight of each module. In this application implementation, each Set to 1.

[0032] CD is the average of the shortest distances between two point clouds, that is, for each point in one point cloud, calculate its distance to the nearest point in the other point cloud and average all distances. CD calculation for incomplete point cloud and real point cloud The distance can be defined as follows: (8); The experimental environment configuration of this application is as follows: CPU model is 15 vCPU Intel(R) Xeon(R) Platinum8358P CPU @ 2.60GHz, GPU model is A40, video memory size is 48GB, memory is 80GB. The development environment is PyCharm, using Cuda11.1, Python3.8 and Pytorch1.9.0 framework to build the network, using Adam optimizer to randomly initialize the network weights, the maximum number of iterations is set to 400 times, the learning rate is initialized to 1E-4, and the learning rate decays by 0.7 every 40 times. The batch size is set to 32.

[0033] In summary, the method described in this application proposes a cascade-based three-dimensional point cloud completion network, which improves the completion performance by calculating the loss function between the coarse point cloud, dense point cloud and real point cloud respectively. Experimental results show that the method described in this application can effectively capture and obtain local and global point cloud features, and better restore the shape of the missing part of the point cloud.

Claims

1. A cascaded refinement method for 3D point cloud completion based on graph convolution, characterized in that The steps include: First, a dual-path structure is used in the coarse point cloud generation module to extract the residual point cloud. The local features and global features are combined to obtain the fused features, and the fused features are predicted using the fully connected layer to output the coarse point cloud. ; Secondly, the fusion features and the coarse point cloud Input to the refined point cloud generation module for feature splicing, and process the spliced ​​features to obtain a refined and complete point cloud .

2. The cascaded refinement three-dimensional point cloud completion method based on graph convolution as claimed in claim 1, characterized in that: The processing method of the rough point cloud generation module comprises the following steps: Residual cloud The features are respectively input into the global extraction module branch and the local feature extraction branch for processing to obtain the features. and Features ; The characteristics and Features Obtain features after fusion , using three fully connected layers to feature Decode information and predict and generate coarse point cloud .

3. The cascaded refinement three-dimensional point cloud completion method based on graph convolution as claimed in claim 2, characterized in that: The local feature extraction module includes three edge graph convolution modules, two graph convolution sampling modules and a maximum pooling module. The input end of the first edge graph convolution module is the input end of the local feature extraction branch. The output end of the first edge graph convolution module is connected to the input end of the maximum pooling module via the first graph convolution sampling module, the second edge graph convolution module, the second graph convolution sampling module and the third edge graph convolution module in sequence. The output end of the maximum pooling module is the output end of the local feature extraction module.

4. The cascaded refinement three-dimensional point cloud completion method based on graph convolution as claimed in claim 3, characterized in that: The local feature extraction module first uses the first edge graph convolution module to convert the input residual cloud Mapping to high-dimensional space to obtain features , use the first graph convolution sampling module to downsample the input and extracted features by 4 times to obtain the feature ; Next, the second edge graph convolution module is used to convolutionally transform the sampled features. Extract and expand dimensions to obtain features , use the second graph convolution sampling module to perform 2 times downsampling on the sampled points and features to obtain the features , then, the third edge graph convolution module is used to convolution the downsampled features Realize the extraction of local features and obtain the hierarchical structure features under 8 times downsampling , the final hierarchical feature After processing through the maximum pooling module, the aggregated features are obtained .

5. The cascaded refinement three-dimensional point cloud completion method based on graph convolution as claimed in claim 4, characterized in that: The edge graph convolution module is used to obtain edge features between a point and its neighboring points by constructing a local neighborhood graph, and then aggregate the edge features to update the feature value of the point, thereby capturing local neighborhood information while maintaining the invariance of the point arrangement.

6. The cascaded refinement three-dimensional point cloud completion method based on graph convolution according to claim 4, characterized in that: The graph convolution sampling module generates pooled points and features by weighting the neighbors of the sampling points in the geometric space and feature space: The input point cloud is represented as , the corresponding feature is , where N represents the number of points and D represents the dimension of the feature; The weight matrix w is learned from the relationship between the keypoint and its neighbors and is defined as: （1）； in, and It is a nonlinear mapping function used to learn the relationship between point clouds and features. and represents key points and their corresponding features, and Represents one of the neighborhood points of the key point and its features; the points and features finally obtained by sampling are expressed as: （2）； （3）； in, It is an undirected graph The set of edges, the undirected graph is constructed by KNN combination, is the weight of the neighborhood point obtained by calculation, is the weight of the calculated neighborhood feature; When the weight of a point is 1, the point becomes a new sampling point; when a point may affect the prediction of the coarse point cloud, the weight of the point is set to a minimum value so that the feature of the point will not be considered during the pooling process.

7. The cascaded refinement three-dimensional point cloud completion method based on graph convolution according to claim 1, characterized in that: The input of the refined point cloud generation module is the coarse point cloud and Features , the coarse point cloud and Features Generate two point feature matrices respectively and , and then perform point-by-point concatenation to obtain the feature matrix ; The feature matrix The input is fed into a network that alternately stacks the global self-enhancement module and the edge graph convolution module to gradually upsample the point features, and then the upsampled point feature matrix is ​​connected point by point With point feature matrix Perform splicing to obtain a comprehensive feature matrix Finally, for the comprehensive features Processing is performed and the corresponding feature vectors are converted into three-dimensional position coordinates to obtain a complete point cloud after refinement. .

8. The cascaded refinement three-dimensional point cloud completion method based on graph convolution as claimed in claim 7, characterized in that: The global self-enhancement module introduces self-attention to establish the spatial relationship between points to capture global information: The global self-enhancement module accepts the input feature matrix X and the upsampling ratio u, where X is the size of ; Multi-head self-attention is used to integrate the information from different points of the input feature matrix X in the form of residuals to obtain the feature matrix F: （4）； （5）； in, and is the linear transformation matrix; after the linear transformation operation, the dimension of the point increases u times, and the output of the global self-enhancement module is: （6）； Among them, QJ represents the global self-enhancement module, the output of which is a size The matrix of .

9. The cascaded refinement three-dimensional point cloud completion method based on graph convolution according to claim 1, characterized in that: The loss function is a measure of the gap between the output point cloud and the real point cloud. It is used to reduce the difference between the real value and the generated point cloud. The chamfer distance CD is used to measure the distance between the two point clouds. The smaller the distance, the closer and more similar the two point clouds are.

10. The cascaded refinement three-dimensional point cloud completion method based on graph convolution according to claim 9, characterized in that: A multi-stage completion loss function is used to measure the distance between two point clouds. The multi-stage completion loss function is shown as follows: （7）； The output point clouds generated by the coarse point cloud and the refined point cloud are denoted as ; At the same time, the ground truth point cloud is downsampled by FPS to obtain three sub-clouds , and their densities are The same; among them, Represents the contribution weight of each module. Set to 1; CD is the average of the shortest distances between two point clouds. CD calculation is incomplete point cloud and real point cloud The distance is defined as follows: （8）。

Images