Construction and completion of three-dimensional point cloud completion model based on multi-modal hierarchical features

Abstract

The invention discloses construction and completion of a three-dimensional point cloud completion model based on multi-modal hierarchical features, and the method comprises the steps: carrying out the projection of an input point cloud, and obtaining a multi-view depth map; constructing a three-dimensional point cloud completion model; the input point cloud and the multi-view depth map serve as input, the complete point cloud serves as output, Chamfer Distance serves as a loss function, the constructed three-dimensional point cloud completion model is trained, and a trained three-dimensional point cloud completion model is obtained; multi-view depth map semantic features are extracted through a ResNet18 network and are effectively fused with point cloud geometric features extracted by a DGCNN through a multi-head cross attention mechanism in an encoding stage, and meanwhile, an encoder is introduced to capture local geometric details in a shallow layer and explicitly fuse semantic information in a deep layer and keep feature consistency, so that the accuracy of the multi-view depth map is improved. And the decoder reconstructs the missing region on the basis of the fused features, so that high-quality point cloud completion is realized, and the technical problems of insufficient geometric detail recovery and weak missing region completion capability in the existing point cloud completion method are solved.

Description

Technical Field

[0001] This invention belongs to the field of 3D computer vision and deep learning technology, and relates to 3D point cloud completion, specifically the construction and completion of a 3D point cloud completion model based on multimodal hierarchical features. Background Technology

[0002] With the widespread application of 3D vision technology in fields such as computer graphics, autonomous driving, and robot perception, 3D point cloud completion, as a core task for restoring missing geometric structures, has received high attention from academia and industry. Existing deep learning-based point cloud completion methods generally adopt an encoder-decoder architecture, which reconstructs missing regions by encoding and decoding the geometric features of the point cloud. However, the inherent disorder and sparsity of point cloud data lead to significant limitations in traditional methods when capturing complex geometric details (such as sharp edges and changes in surface curvature), easily resulting in problems such as coarse completion results and structural discontinuities.

[0003] Furthermore, the application of single-modal features further limits the performance of point cloud completion. Traditional methods typically rely solely on the point cloud data itself, lacking effective fusion of multimodal geometric information (such as surface geometric constraints implicit in depth maps and image texture semantics). For example, in occluded regions or low-density point cloud scenes, single-modal point clouds lack local structural context, making it difficult to infer reasonable geometric shapes, resulting in significant deviations between the completed results and the actual structure at the detail level. Although some studies have attempted to introduce multimodal data, existing fusion strategies often remain at the feature level with simple stitching, failing to fully explore the hierarchical semantic relationships and geometric complementarities between different modalities, making it difficult to meet the point cloud completion needs in complex scenes. Therefore, how to overcome the inherent defects of point cloud disorder and sparsity, construct an efficient multimodal hierarchical feature fusion mechanism, and achieve accurate modeling of complex geometric details has become a key problem that current point cloud completion technology urgently needs to solve. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the present invention aims to provide a method for constructing and completing a 3D point cloud completion model based on multimodal hierarchical features, thereby solving the technical problems of insufficient geometric detail recovery and weak ability to complete missing regions in existing point cloud completion methods.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0006] A method for constructing a 3D point cloud completion model based on multimodal hierarchical features, specifically including the following steps:

[0007] Step 1: Project the input point cloud to obtain a multi-view depth map;

[0008] The input point cloud is a set of three-dimensional coordinates of the object surface;

[0009] Step 2: Construct a 3D point cloud completion model;

[0010] The 3D point cloud completion model includes DGCNN, ResNet18 network, encoder and decoder;

[0011] The DGCNN is used to extract local geometric features of the input point cloud and progressively downsample the local geometric features to obtain the first point cloud features;

[0012] The ResNet18 network is used to extract high-level features from multi-view depth maps and compress the feature dimension of these high-level features to obtain depth map features.

[0013] The encoder includes a six-layer encoder and a cross-modal attention fusion module. The first five layers of encoder are used to process the first point cloud features to generate the second point cloud features. The cross-modal attention fusion module is used to fuse the second point cloud features and the depth map features to obtain fused features. The sixth layer of encoder is used to process the fused features to obtain hybrid features.

[0014] The decoder decodes the mixed features to generate a complete point cloud;

[0015] Step 3: Using the input point cloud and multi-view depth map from Step 1 as input and the complete point cloud as output, train the 3D point cloud completion model constructed in Step 2 using Chamfer Distance as the loss function to obtain the trained 3D point cloud completion model.

[0016] This invention also includes the following technical features:

[0017] The ResNet18 network includes a 7×7 convolutional layer, a 3×3 max pooling layer, four convolutional modules, an adaptive global average pooling layer, and a fully connected layer arranged sequentially.

[0018] Each convolutional module includes two residual blocks, and each residual block includes two 3×3 convolutional layers, batch normalization (BatchNorm), and ReLU activation arranged sequentially.

[0019] A 3D point cloud completion method based on multimodal hierarchical features, specifically including the following steps:

[0020] Step 1: Project the point cloud to be completed to obtain a multi-view depth map;

[0021] Step 2: Input the point cloud to be completed and the multi-view depth map obtained in Step 1 into the trained 3D point cloud completion model obtained in the 3D point cloud completion model construction method based on multimodal hierarchical features, and output the complete point cloud of the point cloud to be completed.

[0022] Compared with the prior art, the beneficial technical effects of this invention are:

[0023] This invention extracts semantic features from multi-view depth maps using a ResNet18 network and effectively fuses them with geometric features from point clouds extracted by DGCNN during the encoding stage through a multi-head cross-attention mechanism. Simultaneously, an encoder is introduced to capture local geometric details in the shallow layer and explicitly fuse semantic information in the deep layer while maintaining feature consistency. The decoder reconstructs the missing regions based on the fused features, achieving high-quality point cloud completion. This solves the technical problems of insufficient geometric detail recovery and weak missing region completion capabilities in existing point cloud completion methods. Attached Figure Description

[0024] Figure 1 This is a structural diagram of the present invention;

[0025] Figure 2 This is a structural diagram of the encoder.

[0026] The specific content of the present invention will be further explained in detail below with reference to the embodiments. Detailed Implementation

[0027] It should be noted that, unless otherwise specified, all components in this invention are those known in the art.

[0028] The following are specific embodiments of the present invention. It should be noted that the present invention is not limited to the following specific embodiments. All equivalent modifications made based on the technical solutions of this application fall within the protection scope of the present invention.

[0029] This invention presents a method for constructing a 3D point cloud completion model based on multimodal hierarchical features, specifically including the following steps:

[0030] Step 1: Project the input point cloud to obtain a multi-view depth map;

[0031] The input point cloud is a set of three-dimensional coordinates of the object's surface;

[0032] Step 2: Construct a 3D point cloud completion model;

[0033] The 3D point cloud completion model includes DGCNN, ResNet18 network, encoder and decoder;

[0034] DGCNN is used to extract local geometric features from the input point cloud and progressively downsamples these local geometric features to obtain the first point cloud features.

[0035] The ResNet18 network is used to extract high-level features from multi-view depth maps and compress the feature dimension of these high-level features to obtain depth map features.

[0036] The encoder consists of a six-layer encoder and a cross-modal attention fusion module. The first five layers of encoder are used to process the first point cloud features to generate the second point cloud features. The cross-modal attention fusion module is used to fuse the second point cloud features and the depth map features to obtain the fused features. The sixth layer of encoder is used to process the fused features to obtain the hybrid features.

[0037] The decoder decodes the mixed features to generate a complete point cloud;

[0038] Step 3: Using the input point cloud and multi-view depth map from Step 1 as input and the complete point cloud as output, train the 3D point cloud completion model constructed in Step 2 using Chamfer Distance as the loss function to obtain the trained 3D point cloud completion model.

[0039] In the above technical solution, semantic features of multi-view depth maps are extracted by ResNet18 network and effectively fused with point cloud geometric features extracted by DGCNN during the encoding stage through a multi-head cross-attention mechanism. At the same time, an encoder is introduced to capture local geometric details in the shallow layer and explicitly fuse semantic information in the deep layer while maintaining feature consistency. The decoder reconstructs the missing region based on the fused features, achieving high-quality point cloud completion. This solves the technical problems of insufficient geometric detail recovery and weak missing region completion ability in existing point cloud completion methods.

[0040] In step 1, the input point cloud is projected to obtain a multi-view depth map, specifically as follows:

[0041] (1) Definition of multi-view parameters: Three viewing angles are defined by the preset three-dimensional Euclidean transformation group (SE(3)) parameters, and each viewing angle is defined by the rotation matrix R. i ∈SO(3) and translation vector t i ∈R 3 This is used to construct a multi-view camera coordinate system;

[0042] (2) Coordinate system transformation: For the input point cloud P = {p j |p j ∈R 3 Perform a geometric transformation on each point in} Project it onto the camera coordinate system of the i-th viewpoint, and calculate its two-dimensional projection coordinates (u) using the pinhole camera model. ij ,vij );

[0043] (3) Depth value fusion: For the projection plane of each viewpoint, the depth values ​​{z} of multiple points mapped to the same pixel (u,v) are fused. ij The weighted average depth, calculated using the weighted least squares method, reflects the local geometric consistency of the point cloud.

[0044] (4) Multi-scale optimization: Minimize projection error by introducing consistency constraints between viewpoints;

[0045] (5) Output depth map: The optimized depth value z uv Mapped to the target resolution 2D tensor D∈R H×W This forms the final depth image.

[0046] By fusing multi-view geometric constraints with adaptive weights, projection accuracy is significantly improved, especially in complex scenes where noise interference is effectively suppressed, meeting engineering needs in fields such as high-precision 3D reconstruction and robot perception.

[0047] In step 2, local geometric features of the input point cloud are extracted using DGCNN, and progressive downsampling is performed as follows: First, a 1×1 convolution is used to map the original point cloud coordinates (3 channels) to a low-dimensional feature space (8 channels). Then, four convolutional modules are used to progressively increase the feature dimension. Each layer combines Group Normalization (GroupNorm) and LeakyReLU activation functions to enhance gradient propagation efficiency. Simultaneously, a hierarchical downsampling strategy is adopted, using the far-point sampling (FPS) algorithm to expand the point cloud size from the initial 128 points to 512 points and then compress it back to 128 points in stages. After each round of sampling, a dynamic neighborhood graph is constructed using K-nearest neighbors (KNN, k=16), and the neighborhood feature difference tensor is extracted for channel cascade fusion. Finally, multi-layer max pooling operations are used to aggregate local features under geometric constraints, and combined with graph structure feature propagation to form a multi-scale contextual representation. This method significantly improves the local geometric feature representation capability of unstructured point clouds through dynamic graph modeling and a multi-level feature pyramid architecture.

[0048] In the encoder stage: In the first knn_layer of the encoder, geometric constraint features are extracted from the neighborhood graph constructed based on KNN. After linear mapping and max pooling, these features are concatenated with the self-attention output. A multi-scale geometric-semantic joint representation is generated through a fusion mapping layer. In the last layer of the encoder, an explicit fusion of depth map features and point cloud features is introduced (F = CrossAttention(X)). p ,X d Furthermore, gradient stability is ensured through residual connections and layer normalization, thereby reducing cross-modal feature alignment errors in point cloud completion tasks.

[0049] In the decoder stage, a complete point cloud is generated based on the fused features. In the first knn_layer of the decoder, dynamic neighborhood features are extracted using new_knn_index and cross_knn_index to enhance the geometric constraint information of the features. The FoldingNet module is used to convert the fused features into relative coordinates and add them to the coarse point cloud to obtain the reconstructed point cloud. The reconstructed point cloud is then concatenated with the input point cloud to obtain the final complete point cloud.

[0050] Preferably, the input point cloud can be a publicly available point cloud dataset (ShapeNet55, PCN).

[0051] The ResNet18 network consists of a 7×7 convolutional layer, a 3×3 max pooling layer, four convolutional modules, an adaptive global average pooling layer, and a fully connected layer arranged sequentially.

[0052] Each convolutional module includes two residual blocks, and each residual block includes two 3×3 convolutional layers, batch normalization (BatchNorm), and ReLU activation arranged sequentially.

[0053] In the above technical solution, the ResNet18 network extracts high-level features from multi-view depth maps and compresses the feature dimensions of these high-level features to obtain depth map features. Specifically, it includes: using a single-channel input ResNet18 architecture (input size B×1×H×W), first performing initial feature encoding through 7×7 convolutions (stride 2) and 3×3 max pooling (stride 2), then sequentially constructing four convolutional modules, each containing two residual blocks. Each residual block consists of two cascaded 3×3 convolutional layers, with batch normalization (BatchNorm) and ReLU activation embedded in between, and gradient stabilization achieved through cross-layer residual connections; at the end of the network, adaptive global average pooling is used to compress the features to a B×512 dimension, which is then mapped to a 384-dimensional feature vector through a fully connected layer, and a three-dimensional feature tensor f is generated through dimension recombination. v ∈R 3×384 These correspond to the spatial location, shape semantics, and structural constraint features in the point cloud completion task, respectively.

[0054] This invention also provides a 3D point cloud completion method based on multimodal hierarchical features. The method for constructing a 3D point cloud completion model based on multimodal hierarchical features specifically includes the following steps:

[0055] Step 1: Project the point cloud to be completed to obtain a multi-view depth map;

[0056] Step 2: Input the point cloud to be completed and the multi-view depth map obtained in Step 1 into the trained 3D point cloud completion model obtained by the 3D point cloud completion model construction method based on multimodal hierarchical features, and output the complete point cloud of the point cloud to be completed.

[0057] Example:

[0058] This embodiment presents a method for evaluating model completion performance using mainstream distance metrics in point cloud reconstruction tasks, including Chamfer Distance (CD), Enhanced Chamfer Distance (ECD), CD-L1, and CD-L2. These metrics effectively measure the geometric similarity and completeness between the completed point cloud and the ground truth point cloud, and are universally accepted performance standards in 3D point cloud completion tasks. Chamfer Distance (CD) is the most commonly used distance metric in point cloud reconstruction tasks. It measures the similarity between the completed and ground truth point clouds by calculating the sum of their nearest neighbor Euclidean distances. The mathematical definition is as follows: Where P represents the completed point cloud and Q represents the true point cloud. A lower CD value indicates that the point cloud generated by the model is closer to the real structure. Enhanced Chamfer Distance (ECD) is an improvement on traditional CD, designed to better penalize shape shifts and density inhomogeneities. Compared to ordinary CD, ECD typically introduces point-to-point density weights or multi-scale distance distribution alignment mechanisms, enhancing constraints on local geometric details while preserving global similarity, making it suitable for evaluating the performance of completed points in sparse regions. CD-L1 and CD-L2 are two variants of Chamfer Distance, using the L1 norm and L2 norm to calculate nearest neighbor distances, respectively. CD-L1 uses the L1 distance (i.e., absolute difference) to evaluate errors between point clouds, offering strong fault tolerance and being more sensitive to sharp structures; CD-L2 uses the L2 distance (i.e., squared difference) to penalize point cloud errors, emphasizing larger deviations and suitable for fine reconstruction tasks. Both are calculated similarly to CD, but their metrics differ. CD-L1 is more robust and resistant to outliers; CD-L2 focuses more on the accuracy evaluation of overall structural shifts. Table 1 below shows the performance of this model on the PCN dataset.

[0059] Table 1

[0060] <![CDATA[CD-l1]]> Average Air Cab Car Cha Lam Sofa Tab Wat PCN 9.64 5.50 22.70 10.63 8.70 11.00 11.34 11.68 8.59 GRNet 8.83 6.45 10.37 9.45 9.41 7.96 10.51 8.44 8.04 PointTr 8.38 4.75 10.47 8.68 9.39 7.75 10.93 7.78 7.29 Ours 7.57 4.39 9.67 8.28 8.24 6.56 9.68 7.03 6.74

[0061] Table 2 below shows the performance of the model on the ShapeNet55 dataset, where CD-S, CD-M, and CD-H represent the CD values ​​for easy, medium, and hard difficulty levels, respectively.

[0062] Table 2

[0063] Methods CD-S CD-M CD-H CD-Avg F1 PCN 1.94 1.96 4.08 2.66 0.133 GRNet 1.35 1.71 2.85 1.97 0.238 PointTr 0.58 0.88 1.79 1.09 0.464 Ours 0.51 0.77 1.58 0.95 0.540

[0064] As can be seen from the above, the completion accuracy of this method is better; compared with the PoinTr model, the Chamfer Distance (CD) is reduced by 12% on the ShapeNet55 dataset and by 9.6% on the PCN dataset.

Claims

1. A method for constructing a 3D point cloud completion model based on multimodal hierarchical features, characterized in that, Specifically, the following steps are included: Step 1: Project the input point cloud to obtain a multi-view depth map; The input point cloud is a set of three-dimensional coordinates of the object surface; Step 2: Construct a 3D point cloud completion model; The 3D point cloud completion model includes DGCNN, ResNet18 network, encoder and decoder; The DGCNN is used to extract local geometric features of the input point cloud and progressively downsample the local geometric features to obtain the first point cloud features; The ResNet18 network is used to extract high-level features from multi-view depth maps and compress the feature dimension of these high-level features to obtain depth map features. The encoder includes a six-layer encoder and a cross-modal attention fusion module. The first five layers of encoder are used to process the first point cloud features and generate the second point cloud features. The cross-modal attention fusion module is used to fuse the second point cloud features and depth map features to obtain fused features; The sixth layer encoder is used to process the fused features to obtain hybrid features; The decoder decodes the mixed features to generate a complete point cloud; Step 3: Using the input point cloud and multi-view depth map from Step 1 as input and the complete point cloud as output, train the 3D point cloud completion model constructed in Step 2 using Chamfer Distance as the loss function to obtain the trained 3D point cloud completion model.

2. The method for constructing a 3D point cloud completion model based on multimodal hierarchical features as described in claim 1, characterized in that, The ResNet18 network includes a 7×7 convolutional layer, a 3×3 max pooling layer, four convolutional modules, an adaptive global average pooling layer, and a fully connected layer arranged sequentially. Each convolutional module includes two residual blocks, and each residual block includes two 3×3 convolutional layers, batch normalization (BatchNorm), and ReLU activation arranged sequentially.

3. A method for constructing a 3D point cloud completion model based on multimodal hierarchical features, characterized in that, The method for constructing a 3D point cloud completion model based on multimodal hierarchical features as described in claim 1 or 2 specifically includes the following steps: Step 1: Project the point cloud to be completed to obtain a multi-view depth map; Step 2: Input the point cloud to be completed and the multi-view depth map obtained in Step 1 into the trained 3D point cloud completion model obtained in the 3D point cloud completion model construction method based on multimodal hierarchical features, and output the complete point cloud of the point cloud to be completed.

Images