Dual pipeline point cloud completion method based on geometric feature refinement and generative adversarial network

Abstract

A dual-pipeline point cloud completion method based on geometric feature refinement and adversarial generative networks is proposed. The method comprises: 1. Extracting features from both the missing and complete point clouds; 2. Obtaining the complete and missing point cloud distributions based on the patch features obtained from the encoder, and fitting the missing point cloud distribution to the complete point cloud distribution; 3. After sampling the point cloud distributions, passing them through a geometric feature refinement network to reconstruct the path and obtain a reconstructed point cloud, and completing the path to obtain a coarsely completed point cloud; 4. On the reconstruction path, using the reconstructed point cloud as real data input, introducing noise points and using StyleGAN to obtain a generated point cloud; 5. On the completion path, passing the obtained coarsely completed point cloud through a convolutional encoder, combining patch features and upsampling information to obtain a latent space code, and decoding it through a decoder to obtain a finely completed point cloud; 6. Calculating the cosine distance between the generated point cloud and the finely completed point cloud, assigning different weight values, and fusing them to obtain the final output. This invention can efficiently and accurately complete missing point cloud information.

Description

Technical Field

[0001] This invention relates to a dual-pipeline point cloud completion method based on geometric feature refinement and adversarial generative networks. Background Technology

[0002] Point clouds are simple to capture and represent, and have been the subject of extensive research in fields such as robotics, autonomous driving, 3D modeling, and manufacturing. However, due to sensor limitations, object occlusion, and noise, the generated point clouds are often sparse and incomplete. Point cloud completion can estimate the complete point cloud from the missing points, thereby obtaining higher-quality point cloud data to prepare for downstream tasks such as classification and segmentation.

[0003] Traditional point cloud completion methods are generally based on prior information about the natural structure of objects, such as symmetry or semantic class information, and interpolate to complete the original point cloud based on prior inference. However, these methods can only handle missing point clouds with low missing rates and obvious structural features, resulting in low applicability. The development of deep learning has greatly promoted the ability and generalization of point cloud completion. Early point cloud completion (A. Dai, C. Ruizhongtai Qi, and M. Nieβner, “Shape completion using 3d-encoder-predictor cnns and shape synthesis,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 5868–5877) attempted to transfer mature methods from 2D completion tasks to 3D point clouds through voxel localization and 3D convolution; however, the computational cost was very high. With the tremendous success of PointNet (CRQi, H. Su, K. Mo, and LJ Guibas, “Pointnet: Deep learning on point sets for 3D classification and segmentation,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 652–660) and PointNet++ (CRQi, L. Yi, H. Su, and LJ Guibas, “Pointnet++: Deep hierarchical feature learning on point sets in a metric space,” arXiv preprint arXiv:1706.02413, 2017), the direct use of 3D coordinates has become mainstream in point cloud processing, and this technology has been further applied to many pioneering works on point cloud completion. Currently, most point cloud completion methods employ an encoder-decoder approach.PCN (Wentao Yuan, Tejas Khot, David Held, Christoph Mertz, and Martial Hebert. PCN: Point completion network. In 2018 International Conference on 3DVision (3DV), pages 728–737. IEEE, 2018) first learns global features from the missing point cloud, and then completes the point cloud from coarse to fine based on the feature information. Following PCN, TopNet (Lyne P Tchapmi, Vineet Kosaraju, Hamid Rezatofifighi, Ian Reid, and Silvio Savarese. Topnet: Structural point cloud decoder. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 383–392, 2019) proposed a tree-structured decoder to predict the complete shape. However, existing deep learning-based methods still have the following problems:

[0004] (1) Focusing too much on the overall features of an object while neglecting the fine local details;

[0005] (2) The structural information of man-made objects was ignored;

[0006] (3) The completed point cloud obtained from training loses the individual features of objects and only retains the common features of a certain type of objects.

[0007] (4) Sensitive to noise and has poor robustness;

[0008] (5) Spatial and semantic information of adjacent points is missing. Summary of the Invention

[0009] To overcome the problems in existing technologies, the purpose of this invention is to provide a dual-pipeline point cloud completion method based on geometric feature refinement and adversarial generative networks. It includes one reconstruction path and one completion path, enabling efficient and accurate completion of missing point cloud information.

[0010] To achieve the above objectives, the present invention adopts the following technical solution:

[0011] A dual-pipeline point cloud completion method based on geometric feature refinement and adversarial generative networks includes the following steps:

[0012] Step 1: Extract features from the missing point cloud and the corresponding complete point cloud respectively, where the encoder of the upper and lower dual pipeline architecture shares weights;

[0013] Step 2: Based on the patch features obtained from the encoder, obtain the complete point cloud distribution q(φ) and the missing point cloud distribution. Distribute the missing point cloud The complete point cloud distribution q(φ) is fitted, and KL divergence is used as the loss function;

[0014] Step 3: After sampling the point cloud distribution, the point cloud is processed by a geometric feature refinement network. The reconstructed path is used to obtain the reconstructed point cloud, and the completed path is used to obtain the coarsely completed point cloud. The geometric feature refinement network aggregates features from local regions of neighboring points based on the information obtained from the encoder, and represents the generation process of new points as the upsampled weighted average of features in the local region. The loss of geometric feature refinement is represented by the single-sided chamfer distance.

[0015] Step 4: On the reconstruction path, the reconstructed point cloud is used as the real data input, and noise points are introduced to obtain the generated point cloud Y through StyleGAN. g The accuracy loss during reconstruction is represented by the distance of the double-sided chamfer.

[0016] Step 5: Along the completion path, the obtained coarsely completed point cloud is passed through a convolutional encoder, combining patch features and upsampling information to obtain a latent space code. This latent space code is then decoded by a decoder to obtain the finely completed point cloud Y. v ;

[0017] Step 6: Calculate and generate point cloud Y g And complete point cloud Y v The cosine distance is assigned to Y based on the result. g With Y v Different weight values ​​are combined to obtain the final output Y. out .

[0018] As an alternative implementation, latent space encoding can also be used as generator parameters with known weights in StyleGAN to generate the point cloud Y. g Finally, the cosine distance is calculated to obtain the final output Y. out .

[0019] As an alternative implementation, the geometric feature refinement network can obtain fine local details and structural features, and effectively combine the spatial and semantic relationships between adjacent points. Specific steps include:

[0020] Step 1): The initial encoder generates global features, patch features, and patch center coordinates of the point cloud. After passing through the upsample transformer, a set of feature tensors that fuse global structural features and local fine features is obtained.

[0021] Step 2): After the feature tensor passes through the fully connected layer, it uses a self-attention mechanism to remove false data and obtain higher quality relational data.

[0022] Step 3): Use an upsampling transformer to complete the feature structure into a complete but coarse model.

[0023] The innovation of this invention lies in its ability to refine point cloud local details and structured features through a geometric feature refinement network. It can combine the features of the input point cloud to construct a higher-quality model.

[0024] A further innovation of this invention lies in the fact that the dual-pipeline design allows the missing point cloud to fit the complete point cloud in terms of distribution, providing a uniform latent space encoding for the subsequent generator. The point cloud completion accuracy obtained by this invention is far superior to completion networks such as PCN and TopNet.

[0025] The further innovation of this invention lies in fusing the point cloud generated by the reconstructed path with the point cloud obtained by the completed path, and assigning different weights to the two through a cosine distance function. This method can greatly improve the robustness of the invention and reduce its sensitivity to noise.

[0026] A further innovation of this invention lies in the improvement of the network's loss function. The entire network of this invention is trained end-to-end, and the training loss includes the KL divergence loss for distribution fitting, the accuracy loss for reconstruction (represented by bilateral chamfer distance), and the loss for geometric feature refinement (represented by unilateral chamfer distance), with different weights assigned to each.

[0027] Compared with the prior art, the present invention has the following advantages:

[0028] This invention utilizes deep learning to directly process 3D point clouds, offering strong scalability. It employs a dual-pipeline point cloud completion method based on geometric feature refinement and adversarial generative networks, enabling efficient and accurate completion of missing point clouds. The design of the geometric feature refinement network also allows structural details to be presented in the completed point cloud. The model used in this invention is based on StyleGAN, transformers, and convolutional modules, making it insensitive to noise points and discrete values, significantly improving the network's robustness. The coarse completion of the point cloud is constructed through an upsampling transformer, combining semantic and spatial information from neighboring points, playing a crucial role in the two-stage point cloud completion task and greatly improving registration accuracy. Furthermore, the accurate completion results can provide higher-quality point cloud data for downstream point cloud tasks. Attached Figure Description

[0029] Figure 1 This is a flowchart of a dual-pipeline point cloud completion method based on geometric feature refinement and adversarial generative networks.

[0030] Figure 2 The structure diagram of the geometric feature refinement network. Detailed Implementation

[0031] The present invention will now be described in detail with reference to the accompanying drawings.

[0032] like Figure 1 As shown, the present invention provides a dual-pipeline point cloud completion method based on geometric feature refinement and adversarial generative networks, comprising the following steps:

[0033] a. This invention prepares a training dataset (62400, 2048, 3) and a test dataset (41600, 2048, 3). For the dual-branch network, the missing point cloud X and the complete point cloud Y are used as inputs, represented as P = {p i |i=1,2,...,N}∈R N×3 Where N is the total number of points, P i Let represent the i-th point in the point cloud, where each point has (x, y, z) coordinates. A dual-path encoder applies a point transformer and sets up abstraction layers to extract features from the missing point cloud, with the number of points decreasing progressively in each layer. Finally, the global feature F is obtained. g Patch characteristics and the corresponding patch center coordinates They represent structural information of the missing point cloud, where the encoders share weights.

[0034] b. The global feature F g As input to subsequent modules, the Gaussian distribution of the point cloud is obtained after passing through a linear layer. The network is divided into an upper reconstruction path for the complete point cloud Y and a lower completion path for the missing point cloud X. The goal of the completion path is to reconstruct a complete shape from an incomplete input point cloud X. First, the global features and latent distribution of the complete point cloud Y are encoded, and then the distribution of the missing point cloud is used. We fit a complete point cloud distribution q(φ) to complete the structural features of the missing point cloud. The completed path and the reconstructed path have similar structures, so we use KL divergence as the loss function for this part.

[0035] c. Combined with appendix Figure 2 Distribution of missing point clouds After fitting the complete point cloud distribution q(φ) and upsampling, new feature point pairs are obtained, which are then combined with the patch features F from the initial encoder. p And the center coordinates P of the patch pThe geometric feature refinement network is designed to generate a coarse yet complete point cloud. It captures regional information of local patterns through an upsampling transformer, combines the acquired local features with global features, and generates a novel feature tensor after passing through a fully connected layer. A self-attention mechanism is employed to extract semantic relevance from the tensor. An upsampling layer is used in the coarse-to-fine generation process, ultimately yielding the coarsely completed point cloud X. c The loss function for this part is represented by the one-sided chamfered distance.

[0036] d. The reconstructed point cloud Y we will obtain in the reconstruction path re As real data, random noise points W are also introduced. These two are used as input to StyleGAN. Through adversarial training between the generator and discriminator, the generator's generation capability is continuously strengthened. After each forward computation and backpropagation, the resulting real point cloud and virtual point cloud are concatenated to obtain a new generated point cloud Y. g We use the bilateral chamfer distance to represent the accuracy loss in the reconstruction.

[0037] e. Along the completion path, we will obtain the coarsely completed point cloud X c Through a convolutional encoder, combined with patch features F p The latent space code Z is obtained by fusing the upsampled information. The decoder is designed to enhance structural relationships to recover local shape details. Inspired by image recognition networks, this invention uses a point self-attention kernel to adaptively aggregate learned features of neighboring points. The decoder can observe that different relational structures have different scales. By fusing structural relationships learned from different scales, a new, finely completed point cloud Y is generated based on the latent space code Z. v .

[0038] f. Y g and Y v Dividing by their respective norms yields the regularization term. A score S is obtained by calculating the cosine distance between the two terms. Based on the constraint range of scores S from largest to smallest, Y is assigned... g and Y v Different weighting coefficients are combined to obtain the final output Y. out .

[0039] It is worth mentioning that all the above experimental processes were end-to-end. Figure 1 The modules connected by the dashed lines shown are only used during training. Alternatively, the latent space code Z can be used as input parameters, and the generated point cloud Y can be obtained using a generator with shared weights. g The coarse-completed full point cloud X obtained by sampling from the geometric feature refinement network c A new, finely completed point cloud Y can also be generated after passing through a generator that shares weights with StyleGAN.v Following the same principle, Y g and Y v Divide each by its respective norm to obtain the regularization term, calculate the cosine distance between the two to obtain a score S, assign different weight coefficients to them according to the constraint range from largest to smallest score, and finally combine them to obtain the final Y. out .

[0040] The training loss of this invention comprises three parts: L kl (KL divergence loss), L re (reconstruction accuracy loss), and L p (Geometric refining loss). The KL divergence loss is defined as:

[0041]

[0042] Considering training efficiency, the bilateral chamfer distance is chosen as the reconstruction loss:

[0043]

[0044] Here, x and y represent points belonging to two point clouds, respectively.

[0045] In this invention, the loss in geometric feature refinement is represented by the distance of a single-sided chamfer:

[0046]

[0047] in, and They represent Y respectively out

[0048] Therefore, the total loss function L t This can be expressed as follows: the three losses are combined with balancing parameters λ1 and λ2 to jointly contribute to learning a superior feature representation:

[0049] L t =L kl +λ1L cd +λ2L p

[0050] The inputs and outputs of this invention are visualized. In addition to validation on existing datasets, it also shows excellent shape completion results on other missing point cloud datasets (16,384 points).

Claims

1. A dual-pipeline point cloud completion method based on geometric feature refinement and adversarial generative networks, characterized in that: Includes the following steps: Step 1: Extract features from the missing point cloud and the corresponding complete point cloud respectively, where the encoder of the upper and lower dual pipeline architecture shares weights; Step 2: Based on the patch features obtained from the encoder, obtain the complete point cloud distribution q(φ) and the missing point cloud distribution. Distribute the missing point cloud The complete point cloud distribution q(φ) is fitted, and KL divergence is used as the loss function; Step 3: After sampling the point cloud distribution, the point cloud is processed by a geometric feature refinement network. The reconstructed path is used to obtain the reconstructed point cloud, and the completed path is used to obtain the coarsely completed point cloud. The geometric feature refinement network aggregates features from local regions of neighboring points based on the information obtained from the encoder, and represents the generation process of new points as the upsampled weighted average of features in the local region. The loss of geometric feature refinement is represented by the single-sided chamfer distance. Step 4: On the reconstruction path, the reconstructed point cloud is used as the real data input, and noise points are introduced to obtain the generated point cloud Y through StyleGAN. g The accuracy loss during reconstruction is represented by the distance of the double-sided chamfer. Step 5: Along the completion path, the obtained coarsely completed point cloud is passed through a convolutional encoder, combining patch features and upsampling information to obtain a latent space code. This latent space code is then decoded by a decoder to obtain the finely completed point cloud Y. v ; Step 6: Calculate and generate point cloud Y g And complete point cloud Y v The cosine distance is assigned to Y based on the result. g With Y v Different weight values ​​are combined to obtain the final output Y. out ; Geometric feature refinement networks can obtain fine local details and structural features, and effectively combine the spatial and semantic relationships between adjacent points. Specific steps include: Step 1): The initial encoder generates global features, patch features, and patch center coordinates of the point cloud. After passing through the upsample transformer, a set of feature tensors that fuse global structural features and local fine features is obtained. Step 2): After the feature tensor passes through the fully connected layer, it uses a self-attention mechanism to remove false data and obtain higher quality relational data. Step 3): Use an upsampling transformer to complete the feature structure into a complete but coarse model.

2. The dual-pipeline point cloud completion method based on geometric feature refinement and adversarial generative networks as described in claim 1, characterized in that: Latent space encoding can also be used as generator parameters with known weights in StyleGAN to generate point cloud Y. g Finally, the cosine distance is calculated to obtain the final output Y. out .

Images