A Transformer-based method for missing-part sensitive 3D defect cloud completion

Abstract

This invention discloses a Transformer-based method for missing-part-sensitive 3D point cloud completion. First, the true missing point cloud is separated from its corresponding ground truth values ​​based on the coordinates of the input missing point cloud. Second, a feature and location information extractor is used to extract information from the missing point cloud and the true missing part, generating corresponding point proxies. Then, a feature generator is used to generate features for the missing part and incorporate random location information as missing point proxies. The missing point proxies and the missing point proxies are then fed into a missing-part-sensitive Transformer to generate predicted missing-part proxies. Finally, the predicted proxies are corrected for errors using the ground truth proxies, and these corrections are used as features in subsequent networks for refined point cloud generation. This invention completes incomplete 3D point clouds in an end-to-end manner, obtaining complete point clouds with rich details and good visual effects, which can meet the needs of real-time application scenarios such as autonomous driving.

Description

Technical Field

[0001] This invention belongs to the field of 3D point cloud processing, specifically relating to a 3D point cloud completion method based on Transformer that is sensitive to missing parts. Background Technology

[0002] Three-dimensional information describes the shape and structure of objects and scenes, involving multi-dimensional indicators such as shape, position, depth, pose, and volume. It is an important component of visual information. Compared to 2D images, 3D information can provide a more three-dimensional, comprehensive, and structured description of the real world. Point clouds are a common form of 3D information storage, and with the rise of virtual reality (VR), mixed reality (MR), and autonomous driving, point clouds are receiving increasing attention.

[0003] Specifically, point clouds are the raw data format acquired by many scanning devices. A point cloud is a collection of points in three-dimensional space, where each point contains (x, y, z) coordinates and optional attributes (such as color, normal vector, etc.). Therefore, processing data in the form of point clouds can preserve the original information to the greatest extent. However, whether using RGBD sensors, LiDAR to capture point clouds, or reconstructing point clouds from images, there are limitations in device resolution and susceptibility to occlusion, directly resulting in sparse or incomplete point clouds, greatly limiting their application scope. Therefore, how to predict complete point clouds with reasonable shapes, rich details, and good visual effects from incomplete point clouds has attracted the attention of many researchers. Summary of the Invention

[0004] Purpose of the invention: This invention provides a missing part sensitive 3D point cloud completion method based on Transformer, which can extract features from incomplete 3D point clouds with limited information and predict complete point clouds with reasonable shape, rich details and good visual effect using less computing resources and memory usage.

[0005] Technical solution: This invention provides a method for completing 3D defect clouds that is sensitive to missing parts based on Transformer, including the following steps:

[0006] (1) Randomly select a viewpoint to delete the point cloud within a spherical range of the complete point cloud. The remaining part is used as the incomplete point cloud, and the deleted part is used as the lost point cloud. This forms a "incomplete-lost-complete" data pair, which is used as the training data for the neural network model.

[0007] (2) A point cloud feature extraction module is designed using the farthest point sampling algorithm and vector attention mechanism to extract high-dimensional features of the point cloud point by point. A point cloud position extraction module is designed using the K-nearest neighbor algorithm and the subtraction of coordinate and feature dimensions to extract the geometric position information of the point cloud point by point. The point cloud features and point cloud position information are added together to generate existing point surrogates. and real loss point agent ;

[0008] (3) Use the feature generation module to generate features of the lost point cloud, and construct a lost point cloud proxy prediction module through attention mechanism, residual connection and feedforward neural network to obtain the predicted lost point proxy. ;

[0009] (4) Use real loss point proxy Agent for predicting lost points The correction is performed and then fed into the point cloud refinement module for refined point cloud prediction.

[0010] Furthermore, the construction process of the "incomplete-lost-complete" data pair described in step (1) is as follows:

[0011] Input a complete point cloud, randomly select a viewpoint from 8 visible viewpoints, and randomly select a point point under that viewpoint. Specify the radius of the spherical deletion, so that the resulting sphere overlaps with the complete point cloud in space. Points contained within the sphere are deleted, and points contained outside the sphere are retained. The retained part corresponds to the incomplete part, and the deleted part corresponds to the missing part.

[0012] Furthermore, the neural network model in step (1) includes a point cloud feature extraction module, a point cloud location extraction module, a point cloud feature generation module, a lost point cloud proxy prediction module, and a point cloud refinement module.

[0013] Furthermore, the implementation process of step (2) is as follows:

[0014] For the input point cloud, the furthest point sampling algorithm is used to perform downsampling twice, transforming the number of points in the point cloud into the input value. and After each downsampling operation, a vector attention calculation is added to obtain the final point cloud feature information. This feature information is then input into a multilayer perceptron to predict a coarse loss of point cloud data. A K-nearest neighbor query is performed on the point cloud after the last downsampling operation to find the K nearest neighbors for each point in the cloud, using the formula... Subtracting them along the coordinate dimensions, where, Represents each point in the point cloud The three-dimensional coordinates of the K nearest neighbors; similarly, using the formula Subtract along the feature dimension, where Represents each point in the point cloud The features of the K nearest neighbors are used to construct a feature extraction module and a location extraction module to extract information from the point cloud, which is then used to generate a representation of existing points. and real loss point agent .

[0015] Furthermore, the implementation process of step (3) is as follows:

[0016] Point cloud feature information is divided into After applying a convolutional layer with shared parameters to each group of features, the shape of the feature matrix is ​​changed using a reshape operation and used as a feature. This feature is then added to a random initial position that satisfies a normal distribution to obtain a missing point proxy. The existing point proxy generated in step (2) is used as K and V of the Transformer, and the missing point proxy is used as Q of the Transformer. The missing point proxy is then predicted using the formula:

[0017]

[0018] in, Presentation layer standardization; This indicates multi-head attention calculation; Indicates a feedforward network; This represents the result of attention calculation; This indicates the predicted loss point agent.

[0019] Furthermore, the implementation process of step (4) is as follows:

[0020] Using the real loss point proxy obtained in step (2) The predicted loss point proxy obtained in step (3) Alignment constraints are applied to control the error values; the added alignment constraints are formulated as follows:

[0021]

[0022] in, This represents the alignment constraint loss; The mean squared error is represented; the predicted lost point proxies are used as features and fed into the point cloud refinement network FoldingNet to perform upsampling on the coarse lost point cloud obtained in step (2); the neural network model is regarded as a mapping function F, and the network prediction block is minimized. and corresponding real tags Losses between Learn network parameters We use a weighted sum of the chamfer distance and alignment constraint losses as the loss function:

[0023]

[0024] in, This represents the chamfer distance between the predicted coarse-grained lost point cloud and the actual lost point cloud; This represents the chamfer distance between the predicted complete point cloud and the true complete point cloud. This represents the alignment constraint loss; the chamfer distance is calculated using the following formula:

[0025]

[0026] in, and Point cloud and The point in the middle.

[0027] Beneficial Effects: Compared with the prior art, the beneficial effects of this invention are as follows: This invention divides point clouds into incomplete and missing parts, and proposes a missing part-sensitive 3D point cloud completion method based on Transformer. To improve the efficiency of 3D point cloud completion, the traditional encoder-decoder design in Transformer is abandoned, and only the encoder is used to reduce the number of network parameters and computational load. To achieve better 3D point cloud completion results, neighbor coordinate and neighbor feature aggregation operations are used, and special features are generated for the missing parts. A missing part-sensitive Transformer is designed to generate missing point proxies, and the real missing proxies are used to align the predicted missing proxies, making the final point cloud completion result more reasonable in shape, richer in detail, and better in visual effect. Attached Figure Description

[0028] Figure 1 This is a flowchart of the present invention;

[0029] Figure 2 A schematic diagram illustrating the training data generated for training the network in this invention;

[0030] Figure 3 This is a schematic diagram of the feature and location extractor proposed in this invention;

[0031] Figure 4 This is a schematic diagram illustrating the working process of the location extractor;

[0032] Figure 5 This is a schematic diagram of the feature generator proposed in this invention.

[0033] Figure 6 This is a schematic diagram of the structure of the lost point cloud proxy prediction module proposed in this invention.

[0034] Figure 7 A schematic diagram of the residual cloud for input;

[0035] Figure 8 This is a schematic diagram illustrating the results of PCN's incomplete cloud completion.

[0036] Figure 9 This is a schematic diagram of the results of GRNet's completion of incomplete cloud structures;

[0037] Figure 10 This is a schematic diagram of the result of PoinTr completing the incomplete cloud.

[0038] Figure 11 This is a schematic diagram showing the result of the method of the present invention for completing incomplete cloud structures. Detailed Implementation

[0039] The present invention will now be described in further detail with reference to the accompanying drawings.

[0040] This invention provides a Transformer-based method for missing-part-sensitive 3D defect cloud completion. First, training data is constructed, and existing point proxies and true missing point proxies are obtained through a designed feature and location extractor. Simultaneously, the features of the existing point proxies are used by a feature generator to generate features for the missing point proxies, and random location information is fused to generate the missing point proxies. Next, the existing point proxies and the missing point proxies are input into a missing-part-sensitive Transformer to generate predicted missing point proxies. Finally, these proxies are fed into a point cloud refinement network, FoldingNet, to obtain a refined missing point cloud, such as... Figure 1 As shown, it includes the following steps:

[0041] Step 1: Randomly select a viewpoint and delete points within a spherical range from the complete point cloud. The remaining part is called the incomplete point cloud, and the deleted part is called the lost point cloud, forming a "incomplete-lost-complete" data pair, which is used as training data for the neural network model.

[0042] Input a complete point cloud. Randomly select one viewpoint from eight visible viewpoints, and then randomly select a point within that viewpoint. Specify the radius of the spherical deletion sphere. The resulting sphere spatially overlaps with the complete point cloud. Points contained within the sphere are deleted, while points outside the sphere are retained. The retained portions correspond to the incomplete parts, and the deleted portions correspond to the missing parts. For example... Figure 2 As shown, the complete point cloud is labeled as the network's true label ( For a complete point cloud, a viewpoint is randomly selected as the initial point. A sphere with a fixed radius r is drawn. Points contained within the sphere are considered the true missing parts of the point cloud, while points outside or on the sphere are considered the existing parts. The true missing parts and the existing parts are used as the input to the network. , Therefore, ( This method converts a complete point cloud into a training dataset, serving as a training sample for the network model.

[0043] Step 2: Design a point cloud feature extraction module using the farthest point sampling algorithm and vector attention mechanism, and design a point cloud location extraction module using the K-nearest neighbor algorithm and coordinate and feature dimension subtraction operation.

[0044] For the input point cloud, the furthest point sampling algorithm is used to perform downsampling twice, transforming the number of points in the point cloud into the input value. and After each downsampling operation, a vector attention calculation is added to obtain the final point cloud feature information. This feature information is then input into a multilayer perceptron to predict a coarse loss of point cloud data. After this, a K-nearest neighbor query is performed on the point cloud after the last downsampling operation to find the K nearest neighbors for each point in the cloud, using the formula... Subtracting along the coordinate dimensions, where , representing each point in the point cloud The three-dimensional coordinates of the K nearest neighbors; similarly, using the formula Subtract along the feature dimension, where , representing each point in the point cloud The features of the K nearest neighbors; these two calculations constitute a feature and location extractor, used to extract information from the point cloud in step 1, which can be used to generate a replacement of existing points ( ) and real loss point proxy ( ).

[0045] Design of point cloud feature and location extractor, such as Figure 3 As shown, one dimension is The input point cloud, after undergoing two farthest point sampling (FPS) algorithms, yields a dimension of... Similarly, after two vector attention calculations, the features are transformed from 3D to the center point cloud. And it is transformed into the final required feature dimensions by a shared multilayer perceptron.

[0046] For the location information extraction process, using methods such as Figure 4 The method shown aggregates the coordinates and features of neighboring points. The coordinates and features of the extracted center point are used as input. Here we obtain... coordinate information and The feature information is then concatenated, and a shared multilayer perceptron is used to extract the features from... Convert to That is, the transition characteristics between each point and its neighboring points. In obtaining Next, we use an attention mechanism to learn a unique attention score for the point features of each channel, and then use the formula... Aggregate it, where It is a set of transition features of K adjacent points. It has learnable weights The shared function is a pointwise multilayer perceptron.

[0047] Step 3: Use a feature generator to generate features of the lost point cloud. Through attention mechanism, residual connection and feedforward neural network, a Transformer sensitive to missing parts, i.e., a lost point cloud proxy prediction module, is constructed.

[0048] The point cloud feature information is divided into For each group of features, a convolutional layer with shared parameters is applied, and the shape of the feature matrix is ​​changed using a reshape operation. This reshape matrix is ​​then used as a feature, and this feature is added to a random initial position that satisfies a normal distribution to obtain a missing point proxy. After this, the existing point proxies generated in step 2 are used as K and V of the Transformer, and the missing point proxies are used as Q of the Transformer. The missing point proxies are then predicted using the following formula:

[0049]

[0050] in, Presentation layer standardization; This indicates multi-head attention calculation; Indicates a feedforward network; This represents the result of attention calculation; This indicates the predicted loss point agent.

[0051] Design of a point cloud feature generator, such as Figure 5 As shown, the dimension is Using existing point features as input, the C-dimensional channel is divided into... Group. Then, the variation dimension of the convolution is determined by the number of points M of the predicted coarsely missing part, which means we will group each Convert to Finally, the transition features are transformed using the Reshape operation. The predicted missing point features are obtained. All channel groups use convolutional layers with shared parameters, reducing the number of parameters and computational cost.

[0052] The design of a Transformer that is sensitive to missing parts, such as Figure 6 As shown, existing point proxies and lost point proxies are fed into this Transformer, and then... Perform multi-head attention calculation, where This represents the input and output corresponding to each attention head. This represents the normalized exponential function. After calculating the attention, a residual connection and a normalization operation are added, and the result is fed into the feedforward network. Finally, another residual connection is added to output the predicted missing point proxy.

[0053] Step 4: After generating the prediction agent, correct it with the real agent and send it to the point cloud refinement module for refined point cloud prediction.

[0054] Using the real loss point proxy obtained in step 2 The predicted loss point proxy obtained in step 3 Alignment constraints are applied to control the error values; the added alignment constraints are formulated as follows:

[0055]

[0056] in, This represents the alignment constraint loss; This represents the mean square error.

[0057] Following this, the predicted lost point proxies are used as features and fed into the point cloud refinement network FoldingNet to upsample the coarsely lost point cloud; the entire neural network is treated as a mapping function F, and the network prediction blocks are minimized. and corresponding real tags Losses between Learn network parameters We use a weighted sum of the chamfer distance and alignment constraint losses as the loss function:

[0058]

[0059] in, This represents the chamfer distance between the predicted coarse-grained lost point cloud and the actual lost point cloud; This represents the chamfer distance between the predicted complete point cloud and the true complete point cloud. This represents the alignment constraint loss; the chamfer distance is calculated using the following formula:

[0060]

[0061] in and Point cloud and The point in the middle.

[0062] The method proposed in this invention uses point clouds from the ShapeNet-55 / 34 dataset as training data. By generating 80% of the point clouds in this dataset according to the training data generation method of this invention, the existing parts and the truly missing parts are divided, thus creating a training dataset containing 41,952 point cloud data points across 55 classes. For the network model parameter settings, we initially set the network learning rate to 10. −4 The learning rate was adjusted at the same intervals. Furthermore, the network used the Adam optimizer with a batch size of 64. After nearly 200 epochs of training, the training loss gradually converged. Chamfer distance (CD) and density-sensitive chamfer distance (DCD) were used as metrics for evaluating the completion performance.

[0063] This invention sets the number of incomplete point cloud points to 2048, the predicted coarse number of lost point cloud points to 96, the predicted fine number of lost point cloud points to 6144, and the complete number of point cloud points to 8192. After feature and location extraction, the point proxies are: 128 existing point proxies and 96 lost point proxies, both with a feature dimension of 384.

[0064] To verify the effectiveness of this invention, comparative experiments were conducted with three other 3D residual point cloud completion network models. These three models are the currently popular PCN (Point Completion Network), GRNet (Gridding Residual Network for Dense Point Cloud Completion), and PoinTr (Diverse Point Cloud Completion with Geometry-Aware Transformers). All models were trained using the same method. Figure 7 The image shown is a residual cloud from the input of the ShapeNet-55 / 34 dataset. Figures 8 to 11 The diagram shows the complete point clouds completed by various methods. Here, PCN, GRNet, PoinTr, and the refined complete point cloud generated by this invention are used. The complete point clouds generated by different methods are compared with the real complete point cloud computing CD and DCD to compare the differences between each scheme and the real values. Experimental observations show that the network model proposed in this invention has superior completion performance and the fastest inference speed.

[0065] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0066] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style of the specification is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A method for completing 3D defect clouds sensitive to missing parts based on Transformer, characterized in that, Includes the following steps: (1) Randomly select a viewpoint to delete the point cloud within a spherical range of the complete point cloud. The remaining part is used as the incomplete point cloud, and the deleted part is used as the lost point cloud. This forms a "incomplete-lost-complete" data pair, which is used as the training data for the neural network model. (2) A point cloud feature extraction module is designed using the farthest point sampling algorithm and vector attention mechanism to extract high-dimensional features of the point cloud point by point. A point cloud position extraction module is designed using the K-nearest neighbor algorithm and the subtraction of coordinate and feature dimensions to extract the geometric position information of the point cloud point by point. The point cloud features and point cloud position information are added together to generate existing point surrogates. and real loss point agent ; (3) Use the feature generation module to generate features of the lost point cloud, and construct a lost point cloud proxy prediction module through attention mechanism, residual connection and feedforward neural network to obtain the predicted lost point proxy. ; (4) Use real loss point proxy Agent for predicting lost points The correction is performed and then fed into the point cloud refinement module for refined point cloud prediction; The implementation process of step (4) is as follows: Using the real loss point proxy obtained in step (2) The predicted loss point proxy obtained in step (3) Alignment constraints are applied to control the error values; the added alignment constraints are formulated as follows: ； in, Indicates the alignment constraint loss; The mean squared error is represented; the predicted lost point proxies are used as features and fed into the point cloud refinement network FoldingNet to perform upsampling on the coarse lost point cloud obtained in step (2); the neural network model is regarded as a mapping function F, and the network prediction block is minimized. and corresponding real tags Losses between Learn network parameters We use a weighted sum of the chamfer distance and alignment constraint losses as the loss function: ； in, This represents the chamfer distance between the predicted coarse-grained lost point cloud and the actual lost point cloud; This represents the chamfer distance between the predicted complete point cloud and the true complete point cloud. The chamfer distance represents the alignment constraint loss; it is calculated using the following formula: ； in, and Point cloud and The point in the middle.

2. The method for completing 3D defect clouds based on Transformer with sensitivity to missing parts as described in claim 1, characterized in that, The process of constructing the "incomplete-lost-complete" data pair in step (1) is as follows: Input a complete point cloud, randomly select a viewpoint from 8 visible viewpoints, and randomly select a point point under that viewpoint. Specify the radius of the spherical deletion, so that the resulting sphere overlaps with the complete point cloud in space. Points contained within the sphere are deleted, and points contained outside the sphere are retained. The retained part corresponds to the incomplete part, and the deleted part corresponds to the missing part.

3. The method for completing 3D defect clouds based on Transformer with sensitivity to missing parts according to claim 1, characterized in that, The neural network model in step (1) includes a point cloud feature extraction module, a point cloud location extraction module, a point cloud feature generation module, a lost point cloud proxy prediction module, and a point cloud refinement module.

4. The method for completing 3D defect clouds sensitive to missing parts based on Transformer according to claim 1, characterized in that, The implementation process of step (2) is as follows: For the input point cloud, the furthest point sampling algorithm is used to perform downsampling twice, transforming the number of points in the point cloud into the input value. and After each downsampling operation, a vector attention calculation is added to obtain the final point cloud feature information. This feature information is then input into a multilayer perceptron to predict a coarse loss of point cloud data. A K-nearest neighbor query is performed on the point cloud after the last downsampling operation to find the K nearest neighbors for each point in the cloud, using the formula... Subtracting them along the coordinate dimensions, where, Represents each point in the point cloud The three-dimensional coordinates of the K nearest neighbors; similarly, using the formula Subtract along the feature dimension, where Represents each point in the point cloud The features of the K nearest neighbors are used to construct a feature extraction module and a location extraction module to extract information from the point cloud, which is then used to generate a representation of existing points. and real loss point agent .

5. The method for completing 3D defect clouds based on Transformer with sensitivity to missing parts according to claim 1, characterized in that, The implementation process of step (3) is as follows: Point cloud feature information is divided into After applying a convolutional layer with shared parameters to each group of features, the shape of the feature matrix is ​​changed using a reshape operation and used as a feature. This feature is then added to a random initial position that satisfies a normal distribution to obtain a missing point proxy. The existing point proxy generated in step (2) is used as K and V of the Transformer, and the missing point proxy is used as Q of the Transformer. The missing point proxy is then predicted using the formula: ； ； in, Presentation layer standardization; This indicates multi-head attention calculation; Indicates a feedforward network; This represents the result of attention calculation; This indicates the predicted loss point agent.

Images