A point cloud data restoration method integrating shape structure recovery and detail compensation

Abstract

The present invention relates to a point cloud data repair method that integrates shape structure recovery and detail compensation. The present invention includes a point cloud repair network consisting of a global structure recovery branch and a local detail compensation branch to repair point cloud data; the global structure recovery branch adopts an encoder-decoder structure to perform a coarse repair on the input missing point cloud data to obtain a coarse repair point cloud with complete structural information; the local detail compensation branch obtains a reshaped point cloud based on learning the geometric details of the input missing point cloud; finally, the outputs of the two branches are spliced and fused and iteratively sampled at the farthest point to generate a shape fine repair result with detail information preserved and uniform sampling point distribution. In view of the limitation that it is difficult to effectively maintain the original shape detail structure information in three-dimensional point cloud shape repair, the present invention adopts a dual-branch point cloud repair network to effectively restore the shape details of the original point cloud while repairing the overall structural information of the original point cloud shape and generate a complete three-dimensional point cloud model with uniform sampling point distribution.

Description

Technical Field

[0001] The present invention belongs to the technical field of point cloud data restoration, and relates to a point cloud data restoration method integrating shape structure restoration and detail compensation. Background Art

[0002] Since the scanning data obtained using 3D scanners, depth cameras, depth sensors, etc. is presented as discrete point cloud data, practical applications such as robot recognition and grasping, autonomous driving path planning, and real-time positioning and map construction for mobile platforms require the intelligent agent to interact directly with the surrounding environment, which requires the use of complete 3D point cloud shape data. However, in the process of acquiring point cloud scanning data, due to the single perspective of the camera scanning device, the occlusion of the scene objects themselves, and low scanning accuracy, the obtained discrete point cloud data is usually sparsely distributed, partially scanned, or incomplete. In practical applications, incomplete point cloud scanning data often leads to defects such as inaccurate recognition, increased detection difficulty, and low reconstruction accuracy, so the missing point cloud data needs to be repaired and supplemented.

[0003] Most existing 3D point cloud restoration networks use a traditional encoder-decoder structure to repair and complete point cloud models. They focus on repairing and completing the global structural information of the point cloud shape, but are insufficient in recovering the detailed information of the point cloud shape. Therefore, traditional point cloud completion networks have difficulty synthesizing the fine detail features of point cloud objects when repairing the point cloud shape. In the encoder-decoder structure of the point cloud restoration network, based on the input missing point cloud data, the encoder first extracts features from the point cloud data and compresses it into a low-dimensional global feature vector, and the decoder decodes the compressed feature codewords and converts them into complete point cloud data. However, the traditional encoder-decoder structure does not fully utilize the shape structure and detailed information of the original point cloud model, so it weakens the detailed features of the point cloud during the point cloud restoration and completion process. It has certain limitations in recovering the shape structure, and it is difficult for the repaired point cloud to maintain a uniform distribution. Summary of the Invention

[0004] The purpose of this invention is to provide a point cloud data restoration method that integrates shape structure recovery and detail compensation. To address the limitations of effectively preserving the detailed structural information of the original shape during 3D point cloud restoration, a dual-branch point cloud restoration network is proposed that integrates global structure restoration and local detail compensation, leveraging the geometric learning mechanism of neural networks. This network not only restores the overall structural information of the original point cloud shape, but also effectively restores its shape details and generates a complete 3D point cloud model with a uniform distribution of sampling points.

[0005] The present invention repairs point cloud data based on a point cloud repair network including a global structure recovery branch and a local detail compensation branch; the global structure recovery branch adopts an encoder-decoder structure, firstly performs a coarse repair on the input missing point cloud data to obtain a coarse repair point cloud with complete structural information, but it lacks local geometric details of the shape; the local detail compensation branch obtains a reshaped point cloud based on learning the geometric details of the input missing point cloud; finally, the outputs of the two branches are spliced and fused and subjected to iterative farthest point sampling (IFPS) to generate a shape fine repair result with detail information preserved and uniform sampling point distribution.

[0006] The specific steps include:

[0007] 1) The original point cloud data is input into the encoder, and the maximum pooling operation is used to aggregate global features to solve the rotation and disorder problems of the original input point cloud. The feature codeword with global information is obtained through multi-layer perceptron encoding. The decoder performs two folding operations on the input feature codeword using four 2D grids to fit the rough restoration result of the point cloud shape.

[0008] 2) The local detail compensation branch uses the input point cloud model obtained in the encoder to perform multi-granularity feature learning on each sampling point from local to global features at various scales, thereby expanding the set of sampling points in the feature space and restoring the detail information of the original point cloud through detail reshaping.

[0009] The feature information in the global structure recovery branch encoder is input. This feature information can reflect the hierarchical feature information of each level in the encoder and includes the local geometric information of the input point cloud. The information extracted at different levels facilitates the provision of feature information of different dimensions. The hierarchical intermediate features are respectively input into the corresponding hierarchical feature learning modules. During the feature learning process, the geometric features of the point cloud are extracted and fused through multi-level features. The features at different scales are then combined and output to obtain the upsampled point cloud, thereby effectively retaining the local detailed structural information of the original point cloud.

[0010] 3) The sampling points reshaped by the local detail compensation branch are spliced and fused with the sampling points obtained by the rough restoration to obtain the complete point cloud shape, and the shape fine restoration result with detail information preserved and uniform sampling point distribution is generated through iterative farthest point sampling (IFPS).

[0011] The Chamfer distance and Earth Mover distance loss functions are used to train a dual-branch point cloud restoration network to measure the error difference between the point cloud model restoration result and the real point cloud.

[0012] The loss function includes: the difference between the rough restoration result of the point cloud shape and the true point cloud shape, the error between the reshaped point cloud obtained by detail feature learning and the true point cloud shape, and the error between the final fine restoration point cloud and the true point cloud shape.

[0013] The point cloud restoration network proposed in this invention adopts a dual-branch structure. During the restoration process, not only the completion of the global structural information of the shape is considered, but also the learning of local detail features and the recovery of shape details are considered. The sampling points of the complete point cloud shape after restoration are evenly distributed, and the restoration method is robust to model noise and different degrees of model missing. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 Schematic diagram of the dual-branch point cloud restoration network structure;

[0015] Figure 2 This is an example diagram of the effect of repairing and completing the point cloud data of the car model in the embodiment;

[0016] Figure 3 This is an example diagram of the effect of repairing and completing the point cloud data of the ship model in the embodiment. DETAILED DESCRIPTION

[0017] The technical method and three-dimensional point cloud restoration effect of the present invention are further described and illustrated below with reference to the accompanying drawings.

[0018] like Figure 1 As shown in FIG, a point cloud data restoration method that integrates shape structure recovery and detail compensation is described as follows:

[0019] Point cloud data is repaired based on a point cloud repair network consisting of a global structure recovery branch and a local detail compensation branch. The global structure recovery branch adopts an encoder-decoder structure.

[0020] Step 1: Input the original point cloud data into the encoder, use the maximum pooling operation to aggregate global features to solve the rotation and disorder problems of the original input point cloud, and obtain feature codewords with global information through multi-layer perceptron encoding;

[0021] In the shape encoding stage, the encoder input in the global structure recovery branch is an M×3 matrix with coordinate information of M discrete sampling points; to solve the rotation problem of the original point cloud, two joint T-Net transformations are used, where the 3×3 input transformation can adjust the point cloud shape posture, and the 64×64 feature transformation can align the point cloud features trained by the shared multi-layer perceptron. After two transformations, the 64-dimensional point cloud features are obtained through a 3-layer MLP perceptron (64, 128, 1024) to obtain the M×1024 point cloud shape global feature information; in order to overcome the point cloud disorder problem, the maximum pooling operation is used to aggregate the global features to obtain a 1×1024 feature vector; finally, the global features of the point cloud are encoded through a 2-layer MLP perceptron to obtain a 1×512-dimensional feature codeword with global information of the input point cloud.

[0022] The decoder performs a two-fold operation on the input feature codeword using four 2D grids to fit the coarse restoration result of the point cloud shape. In the feature decoding stage, the decoder recovers the point cloud shape from the 1×512-dimensional feature codeword encoded by the encoder. The decoder obtains the coarse restoration point cloud by performing a two-fold operation on the feature codeword using four 2D grids.

[0023] Specifically, the decoder first replicates the feature codeword generated by the encoder K times to obtain a K×512 matrix. Next, the network initializes four square 2D grids of dimension K×2 and performs a folding operation to fit the 3D shape of the point cloud. The K×512 feature codeword matrix is concatenated with the four 2D grids to obtain four matrices of size K×514. These are processed by three layers of perceptrons to obtain the first folding result and output four K×3 matrices. Next, the K-folded global shape feature codeword is concatenated with the K×3 matrix of the first folding result to obtain four K×515 matrices. These are then processed by three layers of perceptrons for a second folding operation to obtain four K×3 matrices. These four matrices are then concatenated to obtain a 4K×3 matrix, completing the second folding in 3D space and obtaining a coarsely repaired point cloud.

[0024] In step 2, the local detail compensation branch uses the input point cloud model obtained in the encoder to perform multi-granular feature learning on each sampling point, from local to global, based on features at various scales. This expands the set of sampling points in the feature space and restores the detail information of the original point cloud through detail reconstruction. This information is fed into the encoder of the global structure recovery branch. This feature information reflects the hierarchical feature information at each level of the encoder and includes the local geometric information of the input point cloud. Extracting information at different levels can conveniently provide feature information of different dimensions.

[0025] Specifically: first, the hierarchical feature information encoded by the global structure recovery branch encoder is input into the corresponding hierarchical feature learning module respectively. During the feature learning process, the geometric features of the point cloud can be effectively extracted and fused through multi-level features. Then, the features at different scales are combined and output to obtain the upsampled point cloud, thereby effectively retaining the local detailed structural information of the original point cloud.

[0026] That is: for the input The hierarchical feature information is processed by each level feature learning module and is obtained Features, and then the obtained Features are concatenated to obtain Shape tensor, and finally the shape tensor is transformed into Point cloud coordinate information and output upsampled point cloud.

[0027] Step 3: The sampling points reshaped by the local detail compensation branch are spliced and fused with the sampling points of the rough restoration result to obtain a three-dimensional complete point cloud shape, and the fusion result is subjected to iterative farthest point sampling (IFPS) to generate a shape fine restoration result with detail information preserved and uniform sampling point distribution.

[0028] In order to effectively train the dual-branch point cloud restoration network proposed in this paper that integrates shape structure recovery and detail compensation, the network's loss function uses Chamfer distance (CD) and Earth Mover distance (EMD) to measure the error difference between the point cloud model restoration result and the real point cloud.

[0029] The loss function includes: the difference between the rough restoration result of the point cloud shape and the true point cloud shape, the error between the reshaped point cloud obtained by detail feature learning and the true point cloud shape, and the error between the final fine restoration point cloud and the true point cloud shape.

[0030] Attachment Figure 2 This is an example diagram of the effect of repairing and completing the point cloud data of the car model according to the present invention. Figure 2 a is the incomplete 3D shape point cloud data input by the user, Figure 2 b is the result of rough repair of the incomplete 3D point cloud shape through global structure recovery. Figure 2 c is the shape of the 3D point cloud obtained by splicing and fusion of the upsampled point cloud obtained by the local detail compensation branch and the rough restoration result. Figure 2 d gives the corresponding complete point cloud real data; Appendix Figure 3 This is an example diagram of the effect of repairing and completing the point cloud data of a ship model according to the present invention. Figure 3 a is the incomplete 3D shape point cloud data input by the user, Figure 3 b is the result of rough repair of the incomplete 3D point cloud shape through global structure recovery. Figure 3 c is the shape of the 3D point cloud obtained by splicing and fusion of the upsampled point cloud obtained by the local detail compensation branch and the rough restoration result. Figure 3 d gives the corresponding complete point cloud real data.

[0031] By the attached Figure 2 and attached Figure 3 It can be seen that although the shape global structure recovery branch repairs the complete structure of the input point cloud model, it lacks effective recovery of the shape details of the point cloud model. For example, the front tires of the car are not completely preserved, only the overall outline of the ship can be restored without edge details, and the poop at the rear of the ship is not completely preserved. The repair results of the dual-branch point cloud repair network can better restore its shape detail information. Experiments show that the local detail compensation branch in the present invention can well restore the shape details of the original point cloud.

Claims

1. A point cloud data restoration method integrating shape structure recovery and detail compensation, characterized by: Point cloud data is repaired based on a point cloud restoration network consisting of a global structure restoration branch and a local detail compensation branch. The global structure restoration branch adopts an encoder-decoder structure. First, the missing point cloud data is roughly repaired to obtain a coarse repair point cloud with complete structural information, but it lacks local geometric details of the shape. The local detail compensation branch reshapes the point cloud based on the geometric details of the missing point cloud input. Finally, the outputs of the two branches are spliced and fused, and the farthest point sampling is iterated to generate a fine shape repair result with uniform sampling point distribution and detail information preservation. The specific steps include: 1) The original point cloud data is input into the encoder; in the feature encoding stage, the encoder input in the global structure recovery branch is an M×3 matrix with M discrete sampling point coordinate information, and two joint T-Net transformations are used. The 3×3 input transformation can adjust the point cloud shape posture, and the 64×64 feature transformation can align the point cloud features trained by the shared multi-layer perceptron. After two transformations, a 64-dimensional point cloud feature is obtained. The M×1024 point cloud shape global feature information is obtained through a 3-layer MLP perceptron; the maximum pooling operation is used to aggregate global features to solve the rotation and disorder problems of the original input point cloud, and a 1×1024 feature vector is obtained; finally, the global features of the point cloud are encoded through a 2-layer MLP perceptron to obtain a 1×512-dimensional feature codeword with global information of the input point cloud; in the feature decoding stage, the decoder recovers the point cloud shape from the 1×512-dimensional feature codeword encoded by the encoder. The decoder obtains a coarse repaired point cloud by performing two folding operations on the feature codeword using 4 2D grids; 2) The local detail compensation branch uses the input point cloud model obtained in the encoder to perform multi-granular feature learning on each sampling point from local to global features at various scales, thereby expanding the set of sampling points in the feature space and restoring the detail information of the original point cloud through detail reconstruction; The feature information in the global structure recovery branch encoder is input. This feature information can reflect the hierarchical feature information of each level in the encoder and includes the local geometric information of the input point cloud. The information extracted at different levels facilitates providing feature information of different dimensions. The hierarchical intermediate features are respectively input into the corresponding hierarchical feature learning module. During the feature learning process, the geometric features of the point cloud are extracted and fused through multi-level features. The features at different scales are then combined and output to obtain the upsampled point cloud, thereby effectively preserving the local detailed structural information of the original point cloud. That is: for the input The hierarchical feature information is processed by each level feature learning module and is obtained Features, and then the obtained Features are concatenated to obtain Shape tensor, and finally the shape tensor is transformed into Point cloud coordinate information and output upsampled point cloud; 3) The sampling points reshaped by the local detail compensation branch are spliced and fused with the sampling points obtained by the rough restoration to obtain the complete point cloud shape, and the shape fine restoration result with detail information preserved and uniform sampling point distribution is generated through iterative farthest point sampling (IFPS).

2. The point cloud data restoration method integrating shape structure recovery and detail compensation according to claim 1, characterized in that: The Chamfer distance and Earth Mover distance loss functions are used to train a dual-branch point cloud restoration network to measure the error difference between the point cloud model restoration result and the real point cloud; The loss function includes: the difference between the rough restoration result of the point cloud shape and the true point cloud shape, the error between the reshaped point cloud obtained by detail feature learning and the true point cloud shape, and the error between the final fine restoration point cloud and the true point cloud shape.

3. The point cloud data restoration method integrating shape structure recovery and detail compensation according to claim 1, characterized in that: The decoder first copies the feature codeword generated by the encoder K times to obtain a K×512 matrix; secondly, the network initializes four square 2D grids of dimension K×2 and performs a folding operation to fit the 3D shape of the point cloud. The feature codeword K×512 matrix is spliced with the four 2D grids to obtain four matrices of size K×514, which are processed by a three-layer perceptron to obtain the first folding result and output four K×3 matrices; then, the shape global information feature codeword copied K times is spliced with the K×3 matrix of the first folding result to obtain four K×515 matrices, which are processed by a three-layer perceptron for a second folding operation to obtain four K×3 matrices; the four matrices are then spliced to obtain a 4K×3 matrix, thus completing the second folding in the 3D space and obtaining a coarse repaired point cloud.

Images