A point cloud completion method based on double feature fusion and neighborhood perception upsampling

Abstract

The application discloses a point cloud completion method based on double-feature fusion and neighborhood perception upsampling, which firstly builds and trains a point cloud completion network comprising an encoder module, a double-feature fusion module, a decoder module and a multi-level upsampling module; in the completion process, the incomplete point cloud and the corresponding single-view image are firstly changed into corresponding feature vectors through the encoder module, and the feature vectors with stronger representation ability are obtained after the double-feature fusion module, and then the sparse and rough complete point cloud is obtained through decoding, the point cloud only has a global shape and lacks fine geometric details; then, the multi-level upsampling module is used for denoising and upsampling the sparse and rough complete point cloud to obtain a dense and complete low-noise point cloud. Compared with the prior art, the point cloud completion method can recover the global shape of the point cloud while retaining the detailed geometric structure possessed by the incomplete point cloud, and the completed missing part has good uniformity.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of three-dimensional data reconstruction of computer vision, and particularly relates to a point cloud completion method based on double-feature fusion and neighborhood perception upsampling. BACKGROUND

[0002] Point cloud is a collection of points sampled from the surface of an object in three-dimensional space, which has rich geometric information and spatial information. Each point can also contain other attributes such as color information (RGB value), normal vector, etc. In the field of three-dimensional vision and computer graphics, geometric shape representation has various forms such as voxel, polygonal mesh, depth map, oriented distance function, implicit potential field, etc. Point cloud is widely used due to its small data volume, easy storage, easy acquisition, strong expression ability and other advantages.

[0003] Common point cloud data scanning devices include laser scanners, binocular cameras, structured light sensors and laser radars, etc. However, in the real world, due to the limited resolution of point cloud scanning devices, occlusion of the viewing angle, distance limitation of the sensor and other problems, the obtained point cloud is often sparse and incomplete, that is, the point cloud has holes or missing parts. Directly using such point cloud data for downstream tasks will cause serious performance loss. Therefore, point cloud completion, that is, inferring a complete point cloud from a sparse and incomplete point cloud, is urgently needed for downstream tasks.

[0004] Existing point cloud completion methods can be roughly divided into two categories: traditional methods and deep learning-based methods. Traditional methods such as symmetry driving, surface reconstruction and retrieval matching usually require the object to have good symmetry, and can only handle incomplete point clouds with small holes, and are very sensitive to noise. In addition, the retrieval matching method needs to be optimized in the retrieval process, and has high time complexity, which is not suitable for online applications. Early deep learning-based methods usually adopt an autoencoder architecture, that is, an encoder extracts global features from an incomplete point cloud, and a decoder decodes a complete point cloud based on the global features. However, this kind of method can only generate a complete point cloud with global shape, and lacks fine geometric details. In recent years, popular deep learning-based methods usually adopt a coarse-to-fine architecture, that is, an additional refinement network is introduced based on the original autoencoder to optimize the generated point cloud, and the refinement network is usually an upsampling network. However, the upsampling network in the existing method lacks reasonable use of neighborhood information of the point cloud, resulting in a lack of fine geometric details in the completed point cloud.

[0005] Compared to 3D point cloud data, the corresponding 2D image data is obviously easier to obtain. Furthermore, humans can easily infer the 3D shape of an object from a 2D image, which intuitively demonstrates the important role 2D images can play in 3D shape completion tasks. Specifically, an image contains global shape information of an object, while a fragmented point cloud contains rich local details. These two complementary pieces of information can enhance the representational power of the extracted features, enabling the decoder to generate a more reasonable shape. However, existing multimodal point cloud completion methods typically rely primarily on fragmented point cloud features to generate a complete point cloud, lacking proper utilization of image features. Moreover, existing multimodal point cloud completion methods directly mix the complete point cloud obtained by decoding global features with the fragmented point cloud and perform farthest point sampling to obtain the final output. While such a point cloud has a complete shape, the number of points in the originally fragmented parts is very small, resulting in a lot of noise. Summary of the Invention

[0006] The purpose of this invention is to address the aforementioned deficiencies in the prior art by providing a point cloud completion method based on dual feature fusion and neighborhood-aware upsampling.

[0007] The objective of this invention can be achieved by adopting the following technical solutions:

[0008] A point cloud completion method based on dual feature fusion and neighborhood-aware upsampling, the point cloud completion method comprising the following steps:

[0009] S1. Construct a point cloud completion network based on dual feature fusion and neighborhood-aware upsampling. The network includes an encoder module, a dual feature fusion module, a decoder module, and a multi-level upsampling module connected in sequence. The multi-level upsampling module consists of two neighborhood-aware upsampling Transformer modules connected in sequence.

[0010] S2. Input the residual point cloud P and the corresponding single-view image I. Use the encoder module to perform 3D encoding on the residual point cloud to obtain the point cloud features F. P The image features F are obtained by performing two-dimensional encoding on a single-view image through an encoder module. I ;

[0011] S3, using the dual-feature fusion module to process point cloud features F P and image features F I The fusion is performed to obtain the fusion feature F;

[0012] S4. The fused feature F is decoded by the decoder module to obtain a sparse, coarse, complete point cloud P0.

[0013] S5. After mixing the sparse coarse complete point cloud P0 with the residual point cloud P, downsample the point cloud obtained by downsampling. Input the downsampled point cloud into the multi-level upsampling module. The multi-level upsampling module performs two consecutive upsampling operations on the downsampled point cloud to obtain the dense complete point cloud C.

[0014] S6. Train a point cloud completion network based on dual feature fusion and neighborhood-aware upsampling using a point cloud completion dataset. Iterate the training in an end-to-end manner by minimizing the loss function. Calculate the loss function value during each iteration and use backpropagation to update the parameters in the network until the network converges. Input the incomplete point cloud and images from the dataset into the trained network to obtain the complete point cloud.

[0015] Furthermore, the encoder module operates as follows:

[0016] Given a residual cloud Let P be a point in the point cloud, and N be the number of points in the residual point cloud. The encoder module performs 3D encoding on the residual point cloud P, as follows: A first dynamic graph convolution operation is performed on the residual point cloud P to obtain the point-by-point feature F. graph1 For point-by-point features F graph1 A second dynamic graph convolution operation is performed to obtain the pointwise features F. graph2 The first self-attention pooling operation is performed on the point-by-point features to obtain the point-by-point features F′. graph2 For point-by-point features F′ graph2 The third dynamic graph convolution operation is performed to obtain the pointwise feature F. graph3 For point-by-point features F graph3 A second self-attention pooling operation is performed to obtain the pointwise feature F′. graph3 For point-by-point features F′ graph3 The fourth dynamic graph convolution operation is performed to obtain pointwise features. Where N P It is the number of points, c P It is the number of feature channels;

[0017] Dynamic graph convolution operations are used to extract local detail information contained in the residual defect cloud. The process is as follows: for each point in the residual defect cloud P, a neighborhood is found to obtain the neighborhood tensor. k0 is the size of the neighborhood; simultaneously, the residual point cloud P is expanded to a new dimension, and the point coordinates of the point cloud are copied k0 times along this dimension to form the original coordinate tensor. The relative coordinates ΔP = N are constructed by combining the neighborhood tensor and the original coordinate tensor. geo -O geo The geometric context C is obtained by concatenating relative coordinates and the original coordinate tensor. geo =concat[O geo;△P], concat[;] represents the tensor concatenation operation along a certain dimension; using a multilayer perceptron to analyze C geo After feature encoding, max pooling is performed on the neighborhood to obtain point-by-point features. max(·) represents performing max pooling on a certain dimension. It is a multilayer perceptron, f i For the i-th point p in the defective cloud P i The corresponding features are: c is the number of channels for the point-by-point feature; the multilayer perceptron, proposed by Qi et al. in the paper "PointNet: Deep learning on point sets for 3D classification and segmentation", consists of a one-dimensional convolutional layer, a ReLU activation function layer, and a batch normalization layer, and is used to extract point-by-point features from point clouds and to reduce or increase the dimensionality of point-by-point features; the dynamic graph convolution, proposed by Wang et al. in the paper "Dynamic Graph CNN for Learning on Point Clouds", can be applied to residual point clouds to construct point-by-point features, or applied to point-by-point features to construct new point-by-point features;

[0018] The self-attention pooling operation is as follows: For F... graph2 The feature f of each point i Find the neighborhood and obtain the neighborhood tensor. k1 is the size of the neighborhood; simultaneously, the point-by-point feature F graph2 Expand to a new dimension and copy the features of each point along that dimension k1 times to obtain the original feature tensor. The relative feature tensor ΔF = N is obtained by comparing the feature neighborhood tensor with the original feature tensor. f -O f The feature context C is obtained by concatenating the relative feature tensor and the original feature tensor. f =concat[O f Then, a multilayer perceptron and neighborhood max pooling are used to obtain the score for each point. φ is a multilayer perceptron; the features of the N′ points with the highest scores are concatenated to obtain the downsampled point-by-point features. c′ is the number of feature channels per point after downsampling; Self-Attention Graph Pooling was proposed by Lee et al. in the paper "Self-Attention Graph Pooling", which can downsample the feature per point to reduce the number of points and thus reduce the computational complexity.

[0019] Given a single-view RGB image corresponding to a defective cloud. m∈{1, 2,...,H}, n∈{1, 2,...,W}, Let I be the pixel in the m-th row and n-th column of the image. The encoder module performs two-dimensional encoding on image I to extract the global shape information contained in the image. The process is as follows: pixel-by-pixel features of the image are obtained through the ResNet18 image feature extraction network. Its N l It is the number of pixels, c I It refers to the number of feature channels; ResNet18 was proposed by He et al. in the paper "Deep Residual Learning for Image Recognition". It solves the problem of gradient vanishing or exploding while increasing the number of neural network layers by using residual connections.

[0020] Furthermore, in the field of multimodal point cloud completion, existing modal fusion methods can be divided into two types. One is explicit fusion, which involves explicitly converting the image into a point cloud using a pre-trained autoencoder, and then stitching it with the incomplete point cloud after viewpoint alignment to achieve modal fusion. This requires additional training of the autoencoder, and viewpoint alignment is highly dependent on the camera's internal parameters; even slight differences in viewpoint can generate a lot of noise. The other is feature-level fusion, which involves first extracting features from the incomplete point cloud and the image using independent encoders, and then fusing the two features through style transfer, cross-attention, etc., to achieve modal fusion. Existing works usually prioritize the features of the incomplete point cloud for modal fusion, while this invention gives equal importance to both modalities when performing feature fusion, thus proposing a dual-feature fusion module.

[0021] Furthermore, the dual-feature fusion module operates as follows:

[0022] Given the pointwise features of a residual cloud Pixel-wise features of single-view images Global features of residual defect clouds were obtained using max pooling. and global features of the image G P Copy N I After that, with F I Features obtained by splicing G I Copy N P After that, with F P spliced ​​together repeat(·,·) is a copy operation performed after expanding one dimension;

[0023] The attention weight matrix is ​​calculated using two concatenated features, as follows: For F′ IThe attention weight matrix is ​​calculated using a two-layer multilayer perceptron and the Softmax function. and They are two independent multilayer perceptrons; for F′ P The attention weight matrix is ​​calculated using a two-layer multilayer perceptron and the Softmax function. and These are two independent multilayer perceptrons; F′ is given by matrix dot product. I Weighted summation yields image features enhanced from point cloud features. Use matrix dot product to give F′ P Weighted summation yields point cloud features enhanced by image features. The two enhanced features are concatenated and then max pooled to obtain the fused feature F = max(concat[F PI ;F IP ]).

[0024] Furthermore, the working process of the decoder module is as follows:

[0025] The fused feature F is a vector, while the sparse, coarse, complete point cloud is a matrix. The decoder's goal is to process this vector into a matrix representing the point cloud. First, the global feature F is expanded using a 1D transpose convolution to obtain the feature vector. N0 is the feature number, which also corresponds to the number of points in the sparse, coarse, complete point cloud P0, c t This represents the number of channels in the expanded feature. Then, the fused feature F is repeated N0 times and then combined with F... G Features are obtained by concatenating them along the channel dimension. Finally, feature extraction is performed using a two-layer residual multilayer perceptron, and the number of extracted feature channels is reduced to 3 using the multilayer perceptron, thus obtaining a sparse, coarse point cloud. The residual multilayer perceptron, proposed by Xiang et al. in the paper "SnowflakeNet: Point Cloud Completion by Snowflake PointDeconvolution with Skip-transformer", consists of two multilayer perceptrons and residual connections, and is used for point-by-point feature extraction or enhancement of point clouds.

[0026] Furthermore, in the extraction of neighborhood information from point clouds, two types of contexts are mainly considered. The first type is geometric context, which is the context reflected by the spatial position of a point and its neighborhood. The second type is feature context, which is the coding space formed after preliminary feature extraction, containing rich potential representations for visual analysis. Here, the two types of contexts are combined to improve the representational ability of point cloud features through neighborhood information.

[0027] Furthermore, the upsampling process of the neighborhood-aware point cloud upsampling Transformer module is as follows:

[0028] The sparse, coarse point cloud P0 is mixed with the residual point cloud P and downsampled using the farthest point sampling method to obtain P′0; features of the mixed point cloud P′0 are extracted. N′0 is the number of points in the point cloud P′0, and D0 is the number of channels for this feature; for each point p in P0′ j Find the neighborhood j∈{1, 2, ..., N′0}, k∈{1, 2, ..., K}, K is the neighborhood size, p j It is the j-th point in P0′, P jk It is p j Find the k-th point in the neighborhood; construct new local coordinates and encode them using a multilayer perceptron to obtain the point-by-point features in the neighborhood. Ω represents a multilayer perceptron, and D represents the number of feature channels per point; for mixed point cloud features The feature f of each point j Find the neighborhood f j f is the feature corresponding to the j-th point in P0′. jk It is p j The features corresponding to the k-th point in the neighborhood are identified; new local features are constructed and encoded using a multilayer perceptron to obtain the features. Θ is a multilayer perceptron; R jk With r jk After concatenating along the channel dimension, the maximum value is taken among the K neighborhoods to obtain the neighborhood features. The fused feature F is adjusted to have 2D feature channels using two multilayer perceptrons and repeated N′0 times before being combined with feature F. neighbor The features of the point cloud to be upsampled are obtained by stitching them together along the channel dimension. c1 is the number of channels for this point-by-point feature;

[0029] Feature enhancement is performed on the point-by-point features. Feature enhancement relies on the self-attention module, which enhances the point-by-point features F by... downThree consecutive, sequentially connected self-attention modules are fed in. A single self-attention module can be represented as:

[0030] Q = F down W Q K = F down W K V = F down W V

[0031] F=Norm(Q+MultiHead(Q,K,V))

[0032] SAM(F down )=F+FFN(F)

[0033] W Q W K W V These are all transformation matrices, typically implemented using a multilayer perceptron. MultiHead(·) is a multi-head attention mechanism, Norm(·) is a normalization operation, FFN(·) is a feedforward neural network, and SAM(·) is a self-attention module. MultiHead(·), Norm(·), and FFN(·) were proposed by Wang et al. in their paper "Attention Is All You Need." The self-attention module can achieve information exchange between point-by-point features by performing multi-head attention on itself on the point cloud features. After three consecutively connected self-attention modules, enhanced point-by-point features are obtained. c2 represents the number of channels for the enhanced pointwise feature;

[0034] Based on F down and F down The feature expansion operation aims to expand the number of features by the upsampling factor to construct the features of the upsampled point cloud. To increase the number of features, F is first... down Features are obtained by expanding using 1D transposed convolution. r is the upsampling factor. Let F be the number of channels for this feature; the kernel size and stride of the transposed convolution are both r, which can generate r new point features from one point feature. This operation is called partition expansion; partition expansion was proposed by Xiang et al. in "SnowflakeNet: Point Cloud Completion by Snowflake Point Deconvolution with Skip-transformer"; then F down Repeating point by point r times and then connecting with F t Feature F is obtained by splicing the channels. upThis feature is the expanded feature. The purpose of this operation is to limit the partitioning process so that the points are not partitioned too far apart.

[0035] Based on the extended feature F up Coordinate reconstruction is performed to complete the upsampling operation, aiming to reconstruct the 3D coordinates of the point cloud from the expanded point-by-point features. A common method is to reduce the dimensionality of the point-by-point features to 3 using Shared MLP to obtain the coordinates. However, this method has low performance and is prone to generating noisy points. To improve network performance and to deal with residual noise points in the denoised point cloud P1, the expanded features are processed into displacements, i.e., F is first processed using Shared MLP. up Dimensionality reduced to 3 and processed into point-by-point displacement The input sparse point cloud is processed point-by-point repeatedly r times, and the displacements of each point are summed to generate the final upsampled point cloud. That is, P up =ΔP+repeat(O) down ,r);N up For upsampled point cloud P up The number of midpoints, and there are N up =rN down .

[0036] Furthermore, the loss function is defined as follows:

[0037]

[0038] in, For the overall loss function, To reconstruct the loss term, This is the first-level upsampling loss term. This is the second-level upsampling loss term;

[0039] Reconstructing the loss term Calculated using the following formula:

[0040]

[0041] Where Y0 is from the reference point cloud P GT The point cloud obtained by downsampling has the same number of points as the sparse, coarse, complete point cloud P0. The chamfer distance is calculated using the following formula:

[0042]

[0043] Where S1 is the predicted point cloud, S2 is the reference point cloud corresponding to S1, x is any point in S1, y is any point in S2, |·| represents the number of points in a point cloud, and ||·||2 represents the Euclidean distance.

[0044] The first-level upsampling loss term is calculated using the following formula:

[0045]

[0046] Where Y1 is from the reference point cloud P GT The point cloud obtained by downsampling has the same number of points as the first-level upsampled point cloud P1.

[0047] The second-level upsampling loss term is calculated using the following formula:

[0048]

[0049] Among them, P GT For reference point cloud, the number of points is the same as that of dense complete point cloud C.

[0050] Furthermore, the defect cloud and image data in the dataset come from scanning devices including laser scanners, binocular cameras, structured light sensors, and lidar.

[0051] The present invention has the following advantages and effects compared with the prior art:

[0052] 1. Strong representational ability of extracted features. Compared with existing multimodal point cloud completion methods, the dual-feature modality fusion strategy proposed in this invention gives equal importance to both the point cloud and the image modality. It can better utilize the complementarity between the global shape information of the image and the rich local detail information of the incomplete point cloud to provide guidance for point cloud completion. The fused features have better representational ability and the network has strong interpretability.

[0053] 2. Improved uniformity in point cloud completion. The neighborhood-aware point cloud upsampling Transformer proposed in this invention aggregates neighborhood features for each point and uses an attention mechanism to enable spatial and semantic interactions between these features. It also expands features using transposed convolution and feature concatenation, improving the flexibility of upsampling. Compared to point clouds obtained by mixing coarse and incomplete point clouds and then sampling from the farthest point, the upsampled point cloud exhibits better uniformity in the originally incomplete areas, with less or almost no noise. Attached Figure Description

[0054] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0055] Figure 1 This is a flowchart of the point cloud completion method disclosed in this invention, which includes a training process and a testing process;

[0056] Figure 2 This is a schematic diagram of the three-dimensional encoding process of the encoder module in the point cloud completion method disclosed in this invention;

[0057] Figure 3 This is a schematic diagram of the composition of the multi-level upsampling module in the point cloud completion method disclosed in this invention;

[0058] Figure 4 This is a comparison of the point cloud completion method disclosed in this invention with existing methods at upsampling magnifications of 2 and 2;

[0059] Figure 5 This is a comparison of the point cloud completion method disclosed in this invention with existing methods at upsampling magnifications of 1 and 4. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0061] Example 1

[0062] This embodiment discloses a point cloud completion method based on dual feature fusion and neighborhood-aware upsampling, which specifically includes the following steps:

[0063] S1. Construct a point cloud completion network based on dual feature fusion and neighborhood-aware upsampling. The network includes an encoder module, a dual feature fusion module, a decoder module, and a multi-level upsampling module connected in sequence. The multi-level upsampling module consists of two neighborhood-aware upsampling Transformer modules connected in sequence.

[0064] The first neighborhood-aware upsampling Transformer module has an upsampling factor of 2, and the second neighborhood-aware upsampling Transformer module also has an upsampling factor of 2.

[0065] S2. Input the residual point cloud P and the corresponding single-view image I. Use the encoder module to perform 3D encoding on the residual point cloud to obtain the point cloud features F. P The image features F are obtained by performing two-dimensional encoding on a single-view image through an encoder module. I ;

[0066] Among them, such as Figure 2As shown, the KNN algorithm used in the dynamic graph convolution operation in 3D encoding has a neighborhood size of 20, and the KNN algorithm used in the two self-attention pooling operations has neighborhood sizes of 16 and 6 respectively. The downsampling ratios of the two pooling operations are 4 and 4 respectively; the point cloud feature F P With image features F I All are 512-dimensional feature vectors.

[0067] S3, using the dual-feature fusion module to process point cloud features F P and image features F I The fusion is performed to obtain the fusion feature F;

[0068] The dimension of the fusion feature F is 512.

[0069] S4. The fused feature F is decoded by the decoder module to obtain a sparse, coarse, complete point cloud P0.

[0070] The number of points P0 in the sparse, coarse, complete point cloud obtained by decoding is 256.

[0071] S5. After mixing the sparse, coarse, complete point cloud P0 with the residual point cloud P, downsample the resulting point cloud. Input the downsampled point cloud into the multi-level upsampling module. The multi-level upsampling module performs two consecutive upsampling operations on the downsampled point cloud to obtain the dense, complete point cloud C.

[0072] The downsampling yielded 512 point cloud points, and the neighborhood size of the KNN algorithm used in the multi-level upsampling module was 16.

[0073] S6. Train a point cloud completion network based on dual feature fusion and neighborhood-aware upsampling using a point cloud completion dataset. Iterate the training in an end-to-end manner by minimizing the loss function. Calculate the loss function value during each iteration and use backpropagation to update the parameters in the network until the network converges. Input the incomplete point cloud and images from the dataset into the trained network to obtain the complete point cloud.

[0074] The point cloud completion dataset used is the ShapeNet-ViPC dataset proposed by Zhang et al. in the paper "View-Guided Point Cloud Completion". The dataset contains 2048 points in the residual point cloud, the single-view image resolution is 224*224*3, and the reference point cloud contains 2048 points, for a total of 38328 training reference point clouds. Each reference point cloud corresponds to one residual point cloud and 24 images from different viewpoints.

[0075] The loss function is defined as follows:

[0076]

[0077] in, For the overall loss function, To reconstruct the loss term, This is the first-level upsampling loss term. This is the second-level upsampling loss term;

[0078] Reconstructing the loss term Calculated using the following formula:

[0079]

[0080] Where Y0 is from the reference point cloud P GT The point cloud obtained by downsampling has the same number of points as the sparse, coarse, complete point cloud P0, both being 256. The chamfer distance is calculated using the following formula:

[0081]

[0082] Where S1 is the predicted point cloud, S2 is the reference point cloud corresponding to S1, x is any point in S1, y is any point in S2, |·| represents the number of points in a point cloud, and ||·||2 represents the Euclidean distance.

[0083] The first-level upsampling loss term is calculated using the following formula:

[0084]

[0085] Where Y1 is from the reference point cloud P GT The point cloud obtained by downsampling has the same number of points as the first-level upsampled point cloud P1, both being 1024.

[0086] The second-level upsampling loss term is calculated using the following formula:

[0087]

[0088] Among them, P GT For reference point cloud, the number of points is the same as that of dense complete point cloud C, which is 2048.

[0089] During training, the initial learning rate was 0.0001, and it decreased to 0.7 times its original value every 20 rounds, for a total of 120 rounds.

[0090] The entire network training process is as follows: Figure 1 As shown in Table 1, the hardware parameters of the training platform used are shown in Table 2, and the software parameters are shown in Table 3. After the network training is completed, refer to... Figure 1 The testing process in the paper takes a partial point cloud and a single-view image as input and performs point cloud completion to obtain a dense and complete point cloud.

[0091] Table 1. Hardware Environment Parameters

[0092]

[0093] Table 2. Software Environment Parameters

[0094]

[0095]

[0096] Table 3 shows the quantitative comparison results of the currently popular methods ViPC, CSDN, and XMFnet with the present invention on the test set of the ShapeNet-ViPC dataset, covering eight categories. The evaluation metric is the chamfer distance, with a smaller value being better. As can be seen from the table, the method disclosed in this invention outperforms existing methods in all eight categories, demonstrating the effectiveness of the present invention.

[0097] Table 3. Comparison of quantitative results between the point cloud completion method disclosed in this invention and existing methods (×10) -3 )

[0098]

[0099] like Figure 4 As shown, the first column is a single-view image, the second column is the point cloud of the incomplete point cloud, the third column is the point cloud of the complete point cloud obtained by the ViPC method, the fourth column is the point cloud of the complete point cloud obtained by the XMFnet method, the fifth column is the dense and complete point cloud obtained in step S5, and the sixth column is the reference point cloud. It can be seen from the figure that, for the aircraft in the first row, the wings and tail of the aircraft completed by this invention are closer to the aircraft wings and tail of the aircraft in the reference point cloud, and have better uniformity. The aircraft completed by the ViPC method has a lot of noise, while the wings and tail of the aircraft completed by the XMFnet method are too sparse. For the seats in the second row, the seat back and legs of the seat completed by this invention have intact shapes, while the seats completed by other methods have large deformations and many noise points.

[0100] Example 2

[0101] This embodiment discloses a point cloud completion method based on dual feature fusion and neighborhood-aware upsampling, which specifically includes the following steps:

[0102] S1. Construct a point cloud completion network based on dual feature fusion and neighborhood-aware upsampling. The network includes an encoder module, a dual feature fusion module, a decoder module, and a multi-level upsampling module connected in sequence. The multi-level upsampling module consists of two neighborhood-aware upsampling Transformer modules connected in sequence.

[0103] The first neighborhood-aware upsampling Transformer module has an upsampling factor of 1, while the second neighborhood-aware upsampling Transformer module has an upsampling factor of 4.

[0104] S2. Input the residual point cloud P and the corresponding single-view image I. Use the encoder module to perform 3D encoding on the residual point cloud to obtain the point cloud features F. P The image features F are obtained by performing two-dimensional encoding on a single-view image through an encoder module. I ;

[0105] Among them, such as Figure 2 As shown, the KNN algorithm used in the dynamic graph convolution operation in 3D encoding has a neighborhood size of 20, and the KNN algorithm used in the two self-attention pooling operations has neighborhood sizes of 16 and 6 respectively. The downsampling ratios of the two pooling operations are 4 and 4 respectively; the point cloud feature F P With image features F I All are 512-dimensional feature vectors.

[0106] S3, using the dual-feature fusion module to process point cloud features F P and image features F I The fusion is performed to obtain the fusion feature F;

[0107] The dimension of the fusion feature F is 512.

[0108] S4. The fused feature F is decoded by the decoder module to obtain a sparse, coarse, complete point cloud P0.

[0109] The number of points P0 in the sparse, coarse, complete point cloud obtained by decoding is 256.

[0110] S5. After mixing the sparse, coarse, complete point cloud P0 with the residual point cloud P, downsample the resulting point cloud. Input the downsampled point cloud into the multi-level upsampling module. The multi-level upsampling module performs two consecutive upsampling operations on the downsampled point cloud to obtain the dense, complete point cloud C.

[0111] The downsampling yielded 512 point cloud points, and the neighborhood size of the KNN algorithm used in the multi-level upsampling module was 16.

[0112] S6. Train a point cloud completion network based on dual feature fusion and neighborhood-aware upsampling using a point cloud completion dataset. Iterate the training in an end-to-end manner by minimizing the loss function. Calculate the loss function value during each iteration and use backpropagation to update the parameters in the network until the network converges. Input the incomplete point cloud and images from the dataset into the trained network to obtain the complete point cloud.

[0113] The point cloud completion dataset used is the ShapeNet-ViPC dataset proposed by Zhang et al. in the paper "View-Guided Point Cloud Completion". The dataset contains 2048 points in the residual point cloud, the single-view image resolution is 224*224*3, and the reference point cloud contains 2048 points, for a total of 38328 training reference point clouds. Each reference point cloud corresponds to one residual point cloud and 24 images from different viewpoints.

[0114] The loss function is defined as follows:

[0115]

[0116] in, For the overall loss function, To reconstruct the loss term, This is the first-level upsampling loss term. This is the second-level upsampling loss term;

[0117] Reconstructing the loss term Calculated using the following formula:

[0118]

[0119] Where Y0 is from the reference point cloud P GT The point cloud obtained by downsampling has the same number of points as the sparse, coarse, complete point cloud P0, both being 256. The chamfer distance is calculated using the following formula:

[0120]

[0121] Where S1 is the predicted point cloud, S2 is the reference point cloud corresponding to S1, x is any point in S1, y is any point in S2, |·| represents the number of points in a point cloud, and ||·||2 represents the Euclidean distance.

[0122] The first-level upsampling loss term is calculated using the following formula:

[0123]

[0124] Where Y1 is from the reference point cloud P GT The point cloud obtained by downsampling has the same number of points as the first-level upsampled point cloud P1, both being 512.

[0125] The second-level upsampling loss term is calculated using the following formula:

[0126]

[0127] Among them, P GTFor reference point cloud, the number of points is the same as that of dense complete point cloud C, which is 2048.

[0128] During training, the initial learning rate was 0.0001, and the learning rate decreased to 0.5 times the original value every 10 rounds, for a total of 60 rounds of training.

[0129] Table 4 shows the quantitative comparison results of the currently popular methods ViPC, CSDN, XMFnet, and the present invention on the test set of the ShapeNet-ViPC dataset, covering eight categories. The evaluation metric is the chamfer distance, with a smaller value being better. As can be seen from the table, the method disclosed in this invention outperforms existing methods in all eight categories, demonstrating that the present invention has good tolerance for different upsampling ratios.

[0130] Table 4. Comparison of quantitative results between the point cloud completion method disclosed in this invention and existing methods (×10) -3 )

[0131]

[0132] like Figure 5 As shown, the first column is a single-view image, the second column is the point cloud of the missing parts, the third column is the point cloud of the complete point cloud obtained by the ViPC method, the fourth column is the point cloud of the complete point cloud obtained by the XMFnet method, the fifth column is the dense and complete point cloud obtained in step S5, and the sixth column is the reference point cloud. It can be seen from the figure that for the boat in the first row, the stern and part of the hull are severely missing. This invention can restore the overall shape of the boat while maintaining good uniformity. The boat completed by the ViPC method has a lot of noise, while the stern and part of the hull completed by the XMFnet method are relatively sparse, and the shape of the bow is not well restored. For the car in the second row, the car completed by this invention restores the local detailed structure of the tires and rearview mirrors. The cars completed by other methods lack local details and have poor uniformity.

[0133] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A point cloud completion method based on dual feature fusion and neighborhood-aware upsampling, characterized in that, The point cloud completion method includes the following steps: S1. Construct a point cloud completion network based on dual feature fusion and neighborhood-aware upsampling. The point cloud completion network includes an encoder module, a dual feature fusion module, a decoder module, and a multi-level upsampling module connected in sequence. The multi-level upsampling module consists of two neighborhood-aware upsampling Transformer modules connected in sequence. S2. Input the residual point cloud P and the corresponding single-view image I. Use the encoder module to perform 3D encoding on the residual point cloud to obtain the point cloud features F. P The image features F are obtained by performing two-dimensional encoding on a single-view image through an encoder module. I ; S3, using the dual-feature fusion module to process point cloud features F P and image features F I The fusion process yields the fused feature F. The working process of the dual-feature fusion module is as follows: Given the pointwise features of a residual cloud Pixel-wise features of single-view images Global features of residual defect clouds were obtained using max pooling. and global features of the image G P Copy N I After that, with F I Features obtained by splicing G I Copy N P After that, with F P spliced ​​together `repeat(·,·)` performs a copy operation after expanding one dimension. The attention weight matrix is ​​calculated using two concatenated features, as follows: For F′ I The attention weight matrix is ​​calculated using a two-layer multilayer perceptron and the Softmax function. and They are two independent multilayer perceptrons; for F′ P The attention weight matrix is ​​calculated using a two-layer multilayer perceptron and the Softmax function. and These are two independent multilayer perceptrons; F′ is given by matrix dot product. I Weighted summation yields image features enhanced from point cloud features. Use matrix dot product to give F′ P Weighted summation yields point cloud features enhanced by image features. The two enhanced features are concatenated and then max pooled to obtain the fused feature F = max(concat[F PI ;F IP ]); S4. The fused feature F is decoded by the decoder module to obtain a sparse, coarse, complete point cloud P0. S5. After mixing the sparse coarse complete point cloud P0 with the residual point cloud P, downsample the point cloud obtained by downsampling. Input the downsampled point cloud into the multi-level upsampling module. The multi-level upsampling module performs two consecutive upsampling operations on the downsampled point cloud to obtain the dense complete point cloud C. S6. Train a point cloud completion network based on dual feature fusion and neighborhood-aware upsampling using a point cloud completion dataset. Iterate the training in an end-to-end manner by minimizing the loss function. Calculate the loss function value during each iteration and use backpropagation to update the parameters in the network until the network converges. Input the incomplete point cloud and images from the dataset into the trained network to obtain the complete point cloud.

2. The point cloud completion method based on dual feature fusion and neighborhood-aware upsampling according to claim 1, characterized in that, The encoder module operates as follows: Given a residual cloud i∈{1,2,…,N}, Let N be the i-th point in the point cloud, and N be the number of points in the residual point cloud. The encoder module performs 3D encoding on the residual point cloud P, as follows: Perform the first dynamic graph convolution operation on the residual point cloud P to obtain the point-by-point feature F. graph1 For point-by-point features F graph1 A second dynamic graph convolution operation is performed to obtain the pointwise features F. graph2 The first self-attention pooling operation is performed on the point-by-point features to obtain the point-by-point features F′. graph2 For point-by-point features F′ graph2 The third dynamic graph convolution operation is performed to obtain the pointwise feature F. graph3 For point-by-point features F graph3 A second self-attention pooling operation is performed to obtain the pointwise feature F′. graph3 For point-by-point features F′ graph3 The fourth dynamic graph convolution operation is performed to obtain pointwise features. Where N P It is the number of points, c P It is the number of feature channels; The dynamic graph convolution operation is as follows: for each point in the residual defect cloud P, a neighborhood is found to obtain the neighborhood tensor. k0 is the size of the neighborhood; simultaneously, the residual point cloud P is expanded to a new dimension, and the point coordinates of the point cloud are copied k0 times along this dimension to form the original coordinate tensor. The relative coordinates ΔP = N are constructed by combining the neighborhood tensor and the original coordinate tensor. geo -O geo The geometric context C is obtained by concatenating relative coordinates and the original coordinate tensor. geo =concat[O geo ;△P], concat[;] represents the tensor concatenation operation along a certain dimension; using a multilayer perceptron to analyze C geo After feature encoding, max pooling is performed on the neighborhood to obtain point-by-point features. This indicates that max pooling is performed on a specific dimension. It is a multilayer perceptron, where f is the i-th point p in the defect cloud P. i The corresponding feature, c, is the number of channels for the pointwise feature; dynamic graph convolution operations can also be applied to pointwise features F. graph1 Constructing new pointwise features F graph2 ; The self-attention pooling operation is as follows: For F graph2 The feature f of each point i Find the neighborhood and obtain the neighborhood tensor. k1 is the size of the neighborhood; simultaneously, the point-by-point feature F graph2 Expand to a new dimension and copy the features of each point along that dimension k1 times to obtain the original feature tensor. The relative feature tensor ΔF = N is obtained by comparing the feature neighborhood tensor with the original feature tensor. f -O f The feature context C is obtained by concatenating the relative feature tensor and the original feature tensor. f =concat[O f Then, a multilayer perceptron and neighborhood max pooling are used to obtain the score for each point. φ is a multilayer perceptron; the features of the N′ points with the highest scores are concatenated to obtain the downsampled point-by-point features. c′ is the number of feature channels per point after downsampling; Given a single-view RGB image corresponding to a defective cloud. m∈{1,2,…,H}, n∈{1,2,…,W}, Let I be the pixel in the m-th row and n-th column of the image. The encoder module performs two-dimensional encoding on image I, as follows: Pixel-by-pixel features of the image are obtained through the ResNet18 image feature extraction network. Its N I It is the number of pixels, c I It is the number of feature channels.

3. The point cloud completion method based on dual feature fusion and neighborhood-aware upsampling according to claim 1, characterized in that, The decoder module operates as follows: The fused feature F is expanded using a 1D transposed convolution to obtain the feature. N0 is the number of features after expansion, c t The number of channels in the expanded feature; the feature matrix formed by stacking the fused feature F after copying it N0 times, and F G Features are obtained by concatenating them along the channel dimension. Finally, through feature extraction and dimensionality reduction of the extracted feature channels to 3, a sparse coarse point cloud is obtained.

4. The point cloud completion method based on dual feature fusion and neighborhood-aware upsampling according to claim 1, characterized in that, The working process of the multi-level upsampling module is as follows: The sparse, coarse point cloud P0 is mixed with the residual point cloud P and downsampled to obtain P0′; P0′ is then upsampled for the first time using a neighborhood-aware upsampling Transformer module to obtain a first-level upsampled point cloud P1; P1 is then upsampled a second time using a neighborhood-aware upsampling Transformer module to obtain a dense, complete point cloud C; the working process of the neighborhood-aware upsampling Transformer module is as follows: Given a mixed point cloud Extracting features from the hybrid point cloud P′0 N′0 is the number of points in point cloud P′0, and D0 is the number of channels for this feature; For each point p in P0′ j Find the neighborhood j∈{1, 2, ..., N′0}, k∈{1, 2, ..., K}, k is the neighborhood size, p j It is the j-th point in P0′, p jk It is p j Find the k-th point in the neighborhood; construct new local coordinates and encode them using a multilayer perceptron to obtain the point-by-point features in the neighborhood. Ω represents a multilayer perceptron, and D represents the number of feature channels per point; for mixed point cloud features The feature f of each point j Find the neighborhood f j f is the feature corresponding to the j-th point in P0′. jk It is p j The features corresponding to the k-th point in the neighborhood are identified; new local features are constructed and encoded using a multilayer perceptron to obtain the features. Θ is a multilayer perceptron; R jk With r jk After concatenating along the channel dimension, the maximum value is taken among the K neighborhoods to obtain the neighborhood features. The fused feature F is adjusted to have 2D feature channels using two multilayer perceptrons and repeated N′0 times before being combined with feature F. neighbor The features of the point cloud to be upsampled are obtained by stitching them together along the channel dimension. c1 is the number of channels for this point-by-point feature; the point-by-point feature F down Enhanced pointwise features are obtained by performing three self-attention operations. c2 represents the number of channels for the enhanced pointwise feature; For F down Features are obtained by expanding using 1D transposed convolution. r is the upsampling factor. The number of channels for this feature is r, and the kernel size and stride of the transposed convolution are both r; F′ down Repeating point by point r times and then connecting with F t Feature F is obtained by splicing the channels. up ; For feature F up The upsampling operation is completed by reconstructing the coordinates to obtain the upsampled point cloud. The workflow is as follows: First, the F-axis is processed by a multilayer perceptron. up Dimensionality reduced to 3 and processed into point-by-point displacement Then, by repeating P′0 point by point r times and summing the displacements point by point, we obtain the upsampled point cloud. That is, P up =ΔP + repeat(P) down ,r);N up For upsampled point cloud P up The number of midpoints, and there are N up =rN down .

5. The point cloud completion method based on dual feature fusion and neighborhood-aware upsampling according to claim 1, characterized in that, The loss function is defined as follows: in, For the overall loss function, To reconstruct the loss term, This is the first-level upsampling loss term. This is the second-level upsampling loss term; Reconstructing the loss term Calculated using the following formula: Where Y0 is from the reference point cloud P GT The point cloud obtained by downsampling has the same number of points as the sparse, coarse, complete point cloud P0. The chamfer distance is calculated using the following formula: Where S1 is the predicted point cloud, S2 is the reference point cloud corresponding to S1, x is any point in S1, y is any point in S2, |·| represents the number of points in a point cloud, and ‖·‖2 represents the Euclidean distance. The first-level upsampling loss term is calculated using the following formula: Where Y1 is from the reference point cloud P GT The point cloud obtained by downsampling has the same number of points as the first-level upsampled point cloud P1. The second-level upsampling loss term is calculated using the following formula: Among them, P GT For reference point cloud, the number of points is the same as that of dense complete point cloud C.

6. The point cloud completion method based on dual feature fusion and neighborhood-aware upsampling according to claim 1, characterized in that, The defect cloud and image data in the dataset come from scanning devices including laser scanners, binocular cameras, structured light sensors, and lidar.

Images