Single-view guided 3D point cloud inpainting method and device

Abstract

The application provides a single-view guided three-dimensional point cloud repairing method and device, and belongs to the technical field of computer vision. The method comprises the following steps: receiving partial point cloud and a single-view image of the same object; extracting a point cloud feature sequence and an image feature sequence from the partial point cloud and the single-view image respectively; performing modal alignment on the point cloud feature sequence and the image feature sequence to obtain first point cloud alignment features and first image alignment features; migrating the first image alignment features to the first point cloud alignment features to obtain refined point cloud features; and obtaining complete point cloud based on the refined point cloud features and the partial point cloud. The application eliminates the feature difference between the image and the point cloud through modal alignment of the point cloud and the image, ensures accurate feature migration and point cloud completion in the subsequent process, and avoids "image-point cloud misplacement" of the completion result.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a single-view guided method and apparatus for repairing 3D point clouds. Background Technology

[0002] Point clouds, as the core representation of 3D shapes, are widely used in 3D computer vision tasks due to their advantages of compact storage and complete preservation of geometric details. However, point clouds acquired in real-world scenes by devices such as LiDAR and depth cameras often have missing data due to issues such as self-occlusion, viewpoint limitations, and surface reflections, leading to performance degradation in downstream tasks (such as 3D object detection and scene segmentation). Therefore, point cloud completion has become a critical preprocessing step.

[0003] Existing point cloud completion methods mainly fall into two categories: unimodal methods and multimodal methods. Unimodal methods rely only on partial shape priors of the point cloud, generating a complete point cloud through encoder-decoder architectures (such as Point Completion Network (PCN) and PoinTr), Transformers (such as SeedFormer), or hierarchical decoding (such as TopNet). Multimodal methods introduce single-view images as supplements, utilizing visual and structural cues from the images to improve completion accuracy. Representative methods include the View-guided Point Completion framework (ViPC), the Explicitly Guided Information Interaction Network (EGIInet), and the Cross-modal Fusion Network (XMFnet).

[0004] Existing multimodal methods struggle to eliminate feature differences between 2D images and 3D point clouds, and the completion results are prone to "image-point cloud misalignment". Summary of the Invention

[0005] This application provides a single-view guided 3D point cloud repair method and apparatus, which aims to solve the problem that existing multimodal methods are difficult to eliminate the feature differences between 2D images and 3D point clouds, and the completion results are prone to "image-point cloud misalignment".

[0006] Firstly, this application provides a single-view guided method for repairing 3D point clouds, including:

[0007] Receive partial point cloud and single-view images of the same object;

[0008] Point cloud feature sequences and image feature sequences are extracted from partial point cloud images and single-view images, respectively.

[0009] Modal alignment is performed between the point cloud feature sequence and the image feature sequence to obtain the first point cloud alignment feature and the first image alignment feature.

[0010] The first image alignment features are transferred to the first point cloud alignment features to obtain the refined point cloud features.

[0011] A complete point cloud is obtained based on the refined point cloud features and partial point cloud data.

[0012] In one embodiment, after obtaining the refined point cloud features, the method further includes:

[0013] The fused point cloud features are obtained based on the refined point cloud features, which combine the global and local features of the object.

[0014] Furthermore, a complete point cloud is obtained based on the fused point cloud features and partial point cloud data.

[0015] In one embodiment, extracting a point cloud feature sequence from a partial point cloud specifically includes:

[0016] Multi-stage downsampling is performed on a portion of the point cloud to obtain multiple first initial anchor points. In each stage, neighborhood points of the first initial anchor points are aggregated to obtain the aggregated features of the first initial anchor points.

[0017] Combine the aggregated features of all the first initial anchor points into a point cloud feature sequence.

[0018] In one embodiment, modal alignment is performed between the point cloud feature sequence and the image feature sequence to obtain a first point cloud alignment feature and a first image alignment feature, specifically including:

[0019] The point cloud feature sequence and the image feature sequence are input into the visual converter to obtain the first point cloud alignment feature and the first image alignment feature.

[0020] Visual transformers are used to map point cloud feature sequences and image feature sequences into latent space for alignment.

[0021] In one embodiment, transferring the first image alignment features to the first point cloud alignment features to obtain refined point cloud features specifically includes:

[0022] The first point cloud alignment feature is used as the first query content, the first image alignment feature is used as the first key, and the value of the first image alignment feature is used as the first key value.

[0023] The image-point cloud cross-modal attention weights in the first image alignment feature are determined based on the correspondence between the first query content, the first key, and the first key value.

[0024] Based on the image-point cloud cross-modal attention weight, the image information in the first image alignment feature is transferred to the first point cloud alignment feature to obtain the refined point cloud feature.

[0025] In one embodiment, obtaining fused point cloud features based on the refined point cloud features specifically includes:

[0026] The refined point cloud features are normalized, and the normalized point cloud features are used simultaneously as the second query content, the second key, and the second key value.

[0027] Global features are obtained based on the correspondence between the second query content, the second key, and the second key value;

[0028] Obtain the neighborhood features of each point in the refined point cloud features, and obtain local geometric features based on the neighborhood features;

[0029] Global features and local geometric features are fused to obtain fused point cloud features.

[0030] In one embodiment, global features and local geometric features are fused to obtain fused point cloud features, specifically including:

[0031] By combining global features and local geometric features, the combined features are obtained.

[0032] Based on the refined point cloud features, a spatial variant mask is generated;

[0033] Based on spatial variant masks, interpolation fusion is performed on the refined point cloud features and the stitched features to obtain interpolated features;

[0034] The interpolation features are optimized to obtain optimized features;

[0035] Based on interpolation features and optimization features, the fused point cloud features are obtained.

[0036] In one embodiment, obtaining a complete point cloud based on fused point cloud features and partial point cloud data specifically includes:

[0037] The fused point cloud features are input into the first decoder, which upsamples the fused point cloud features to obtain multiple missing sub-point clouds;

[0038] A portion of the point cloud is downsampled to obtain a downsampled point cloud;

[0039] Multiple missing sub-point clouds are stitched together with the downsampled point cloud to obtain a stitched point cloud;

[0040] By adjusting the position of point clouds in the stitched point cloud using a multilayer perceptron, a complete point cloud can be obtained.

[0041] In one embodiment, the single-view guided 3D point cloud repair method further includes:

[0042] The 3D point cloud restoration model is trained. During the training process, the total training loss includes the complete loss based on the complete point cloud.

[0043] In one embodiment, the total training loss also includes reconstruction loss;

[0044] The losses incurred in reconstruction include:

[0045] Construct a second decoder with the same architecture as the first decoder;

[0046] The second image alignment features obtained during training are input into the second decoder to obtain the reconstructed point cloud.

[0047] Using the ground truth of the complete point cloud corresponding to the alignment features of the second image in the training set as supervision, the distance between the reconstructed point cloud and the ground truth of the complete point cloud is determined as the reconstruction loss.

[0048] In one embodiment, the total training loss also includes anchor point optimization loss;

[0049] The anchor point optimization loss is obtained, specifically including:

[0050] Global shape features are obtained based on the second point cloud alignment features acquired during training.

[0051] The anchor point update position is obtained based on the global shape features, the second point cloud alignment features, and the coordinates of the second initial anchor point corresponding to the second point cloud alignment features;

[0052] Using the ground truth of the complete point cloud corresponding to the alignment feature of the second point cloud in the training set as supervision, the distance between the anchor point update position and the ground truth of the complete point cloud is determined and used as the anchor point optimization loss.

[0053] Secondly, this application provides a single-view guided 3D point cloud repair device, comprising:

[0054] The receiving module is used to receive partial point cloud and single-view images of the same object;

[0055] The feature extraction module is used to extract point cloud feature sequences and image feature sequences from partial point cloud and single-view images, respectively.

[0056] The alignment module is used to perform modal alignment between the point cloud feature sequence and the image feature sequence to obtain the first point cloud alignment feature and the first image alignment feature.

[0057] The migration module is used to migrate the first image alignment features to the first point cloud alignment features to obtain the refined point cloud features.

[0058] The complete point cloud acquisition module is used to obtain a complete point cloud based on the refined point cloud features and partial point cloud data.

[0059] Thirdly, this application also provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the above-described single-view guided 3D point cloud repair methods.

[0060] Fourthly, this application also provides a non-transitory computer-readable storage medium storing a computer program that, when executed by a processor, implements any of the above-described single-view guided 3D point cloud repair methods.

[0061] Fifthly, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements any of the above-described single-view guided 3D point cloud repair methods. Attached Figure Description

[0062] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0063] Figure 1 This is one of the flowcharts of the single-view guided 3D point cloud repair method provided in this application;

[0064] Figure 2 This is the second flowchart of the single-view guided 3D point cloud repair method provided in this application;

[0065] Figure 3 This is one of the flowcharts for obtaining refined point cloud features provided in this application;

[0066] Figure 4 This is one of the flowcharts provided in this application for obtaining fused point cloud features based on refined point cloud features;

[0067] Figure 5 This is one of the comparison charts of verification results provided in this application;

[0068] Figure 6 This is the second comparison chart of the verification results provided in this application;

[0069] Figure 7 This is one of the structural schematic diagrams of the single-view guided 3D point cloud repair device provided in this application;

[0070] Figure 8 This is a schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0072] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0073] The terms "first," "second," etc., used in this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects; for example, a first object can be one or more. Furthermore, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects have an "or" relationship.

[0074] It should be noted that this application covers various scenarios that require point cloud repair or point cloud completion, such as autonomous driving (e.g., LiDAR point cloud repair), 3D scene understanding (e.g., indoor and outdoor scene reconstruction), and industrial 3D modeling (e.g., part defect repair).

[0075] The following is combined Figures 1 to 8 This application describes the single-view guided 3D point cloud repair method and apparatus provided in this application.

[0076] It should be noted that the single-view guided 3D point cloud repair method provided in this application embodiment is implemented based on a single-view guided 3D point cloud repair device. The single-view guided 3D point cloud repair method can eliminate feature differences between the image and the point cloud through modal alignment, avoiding "image-point cloud misalignment" in the completion result.

[0077] This application describes the single-view guided three-dimensional point cloud repair method using a single-view guided three-dimensional point cloud repair device as the execution subject.

[0078] Figure 1 This is one of the flowcharts of the single-view guided 3D point cloud repair method provided in this application. Figure 2 This is the second flowchart of the single-view guided 3D point cloud repair method provided in this application.

[0079] Please combine Figure 1 and 2 The single-view guided 3D point cloud repair method provided in this application includes:

[0080] S110: Receive partial point cloud and single-view images of the same object.

[0081] Before 3D point cloud restoration, the same object (e.g.) was obtained. Figure 2 An image of an airplane and a portion of its point cloud.

[0082] S120: Extract point cloud feature sequences and image feature sequences from partial point cloud and single-view images, respectively.

[0083] In one possible implementation, such as Figure 2 As shown, point cloud feature sequences and image feature sequences are extracted through tokenization.

[0084] In one possible implementation, for a single-view image, the single-view image I (e.g., 224×224 resolution) is divided into 16×16 non-overlapping image blocks, and each image block is flattened into a vector (e.g., 768-dimensional) to obtain the image feature sequence F. I (e.g., 196 tokens), image feature sequence F I Add the location embedding corresponding to the image patch to preserve the image spatial information.

[0085] S130: Modal alignment is performed between the point cloud feature sequence and the image feature sequence to obtain the first point cloud alignment feature and the first image alignment feature.

[0086] Modality alignment is the mapping or transformation of features from different data sources (i.e., different modalities, such as point clouds and images) into a shared, semantically consistent feature representation space, so that information describing the same or similar parts in different modalities can be aligned, matched and complemented at the feature level.

[0087] S140: Transfer the first image alignment features to the first point cloud alignment features to obtain refined point cloud features. These refined point cloud features combine image information (such as structural and semantic information) and can more accurately represent the features of the object.

[0088] S150: Obtain a complete point cloud based on the refined point cloud features and partial point cloud data.

[0089] Specifically, the missing sub-point clouds of an object are predicted using the refined point cloud features. Then, the missing sub-point clouds are stitched together with the partial point cloud to form the complete point cloud of the object, which is used to represent the overall information of the object.

[0090] This application's embodiments eliminate feature differences between images and point clouds by modal alignment of point clouds and images, ensuring accurate subsequent feature transfer and point cloud completion, and avoiding "image-point cloud misalignment" in the completion results.

[0091] Existing point cloud inpainting methods rely on the explicit category labels of objects (e.g., "typical shape" or local geometric patterns, such as the four legs of a chair) for their attention mechanisms. Objects of each known category have a uniform category label, which causes the model to fail for unknown shapes.

[0092] In one possible implementation, such as Figure 2 As shown, after obtaining the refined point cloud features (step S140), the process also includes:

[0093] The fused point cloud features are obtained based on the refined point cloud features, which combine the global and local features of the object.

[0094] In this embodiment, in step S150, a complete point cloud is obtained based on the fused point cloud features and partial point cloud.

[0095] In this embodiment, since the refined point cloud features transfer image information to the point cloud, the model learns the relationships between points of different modalities. It automatically understands the general structural information of objects through the visual content of the image and uses this to guide point cloud completion, rather than relying on fixed geometric templates or category priors. The global features obtained in this way supplement the features of the missing parts of the original point cloud data. By fusing global features and local features, the entire structure of the object can be expressed, thereby accurately obtaining the complete point cloud of the object and improving the generalization ability for unknown objects.

[0096] In one possible implementation, step S120, extracting a point cloud feature sequence from a portion of the point cloud, specifically includes:

[0097] S1201: Perform multi-stage downsampling on a portion of the point cloud to obtain multiple first initial anchor points, and aggregate neighborhood points of the obtained first initial anchor points in each stage to obtain the aggregated features of the first initial anchor points.

[0098] In one possible implementation, the Farthest Point Sampling (FPS) method is used for multi-stage downsampling. A group (e.g., 128) of representative points with good spatial coverage are selected from a portion of the point cloud P (e.g., 1024 points) as the first initial anchor points. All first initial anchor points The set of anchor points formed.

[0099] In each stage, a ball query is used to search for multiple local neighborhood points of the first initial anchor point obtained by downsampling in each stage. Geometric features such as coordinate differences and normal vectors of each neighborhood point are aggregated to generate an aggregated feature (e.g., 256-dimensional) for each first initial anchor point. This aggregated feature encodes the geometric structure information of the neighborhood around the first initial anchor point.

[0100] S1202: Combine the aggregated features of all first initial anchor points into a point cloud feature sequence.

[0101] By combining the aggregated features of all the first initial anchor points, a point cloud feature sequence F of fixed length (e.g., [128*256]) is finally obtained. P Point cloud feature sequence F P Includes spatial location embedding, coordinates of the anchor set, and point cloud feature sequence F. P The spatial location embedding corresponds one-to-one and is used to represent local regions of the point cloud.

[0102] This application embodiment obtains representative points with good spatial coverage and their neighborhood features by combining farthest point sampling and ball query, providing a good data foundation for subsequent point cloud repair.

[0103] In one possible implementation, please refer to Figure 2 In step S130, the point cloud feature sequence and the image feature sequence are modally aligned to obtain the first point cloud alignment feature and the first image alignment feature, specifically including:

[0104] The point cloud feature sequence and the image feature sequence are input into the Vision Transformer to obtain the first point cloud alignment feature and the first image alignment feature.

[0105] The Vision Transformer is used to map point cloud feature sequences and image feature sequences to latent space for alignment.

[0106] In one possible implementation, the Vision Transformer comprises a 6-layer encoder, each with 8 self-attention heads. The first point cloud is aligned with the feature F. PAlignment features F with the first image I After inputting the shared Vision Transformer, the sequence modeling capability of the Transformer is used to map the features of the two modalities to a unified, semantically aligned representation space. In the shared representation space, the point cloud feature sequence and the image feature sequence undergo the same attention computation in the same sequence, thereby obtaining the first point cloud aligned feature. Alignment features with the first image This eliminates the feature differences between point clouds and images.

[0107] This application embodiment uses a shared visual converter to ensure that the point cloud feature sequence and the image feature sequence undergo the same attention calculation in the same sequence, thereby eliminating the feature differences between the two and avoiding "image-point cloud misalignment".

[0108] In one possible implementation, such as Figure 3 As shown, in step S140, the first image alignment features are transferred to the first point cloud alignment features to obtain refined point cloud features, specifically including:

[0109] S1401: Align features with the first point cloud The first query content is aligned with the features of the first image. As the first key, align features with the first image. The value is the first key value.

[0110] S1402: Determine the image-point cloud cross-modal attention weights in the first image alignment feature based on the correspondence between the first query content, the first key, and the first key value.

[0111] In one possible implementation, a multi-head attention (MHA) mechanism is used to calculate image-point cloud cross-modal attention weights based on the correspondence between query content, keys, and key-value pairs.

[0112] S1403: Based on image-point cloud cross-modal attention weights, image information (such as object contours, textures, and other structural information, as well as semantic information) in the first image alignment features is transferred to the first point cloud alignment features to obtain refined point cloud features. The formula is as follows:

[0113] (1).

[0114] This application embodiment explicitly injects image information into point cloud features through a cross-modal attention mechanism, and uses image information to refine the point cloud features. This refines the point cloud features, which not only contain geometric information, but also incorporate high-level semantics from the image, thereby enhancing cross-modal features and improving feature representation capabilities. This transfer mechanism also helps to capture the structural and semantic features of unknown categories of objects, improving the model's generalization ability to unknown categories of objects.

[0115] Existing point cloud restoration methods employ either global uniform attention (such as XMFnet) or local graph networks. Global attention ignores the irregular distribution of point clouds, while local networks struggle to capture long-distance structural relationships. As a result, such methods cannot balance global shape coherence with local geometric details, leading to incomplete restoration of complex areas of objects (such as airplane wings, car fronts, etc.).

[0116] Based on these considerations, in one possible implementation, such as Figure 4 As shown, the fused point cloud features are obtained based on the refined point cloud features, specifically including:

[0117] P1: Normalize the refined point cloud features, and use the normalized point cloud features as the second query content Q, the second key K, and the second key value V.

[0118] Specifically, the refined point cloud features Input layer normalization ( The refined point cloud features are normalized.

[0119] Subsequently, the normalized point cloud features are used simultaneously as the second query content Q, the second key K, and the second key value V.

[0120] P2: Obtain global features based on the correspondence between the second query content, the second key, and the second key value.

[0121] In one possible implementation, a global multi-head self-attention mechanism is used to capture long-distance structural associations (such as the positional relationship between a car body and its wheels) by leveraging the correspondence between the second query content Q, the second key K, and the second key value V, thereby obtaining global features. :

[0122] (2).

[0123] P3: Obtain the neighborhood features of each point in the refined point cloud features, and obtain local geometric features based on the neighborhood features.

[0124] In one possible implementation, the K-Nearest Neighbor (kNN) algorithm is used to refine the point cloud features. For each point, search for its k neighboring points (e.g., k=16) and extract the neighborhood features.

[0125] After extracting neighborhood features, the normalized point cloud features are used as the third query content. Using neighborhood features as the third key The neighborhood feature value is used as the third key value V1 and input into a multilayer perceptron (MLP). The output is a 256-dimensional feature, which is then aggregated through max pooling to obtain local geometric features. (Such as the curvature details of the chair armrests):

[0126] (3);

[0127] in, , These are the coordinates of the points corresponding to the third key and the third query content, respectively.

[0128] P4: Fuse global features and local geometric features to obtain fused point cloud features.

[0129] This application embodiment adaptively obtains global features through a global attention mechanism and inputs neighborhood features into a multilayer perceptron, which is then aggregated to obtain local geometric features. By fusing global features and local geometric features, the global structure (such as the overall outline of a car) and local details (such as door handles) are balanced, so that the model can take into account both global and local information and avoid information loss or excessive smoothing.

[0130] In one possible implementation, such as Figure 4 As shown, in step P4, global features and local geometric features are fused to obtain fused point cloud features, specifically including:

[0131] P41: Combine global features and local geometric features to obtain the combined features. .

[0132] P42: Generate a spatial variant mask based on the refined point cloud features.

[0133] In one possible implementation, the refined point cloud features are first used... Input a multilayer perceptron (MLP), then use the output features of the MLP and an activation function (Sigmoid function) to generate a spatial variant mask M:

[0134] (4);

[0135] In the spatial variant mask M, each point has a weight of 0 or 1, which is used to distinguish between "valid regions" and "missing regions".

[0136] P43: Based on the spatial variant mask M, the refined point cloud features Interpolation fusion is performed with the concatenated features Fsk to obtain interpolated features. .

[0137] Specifically, a spatial variant mask M is used to guide the refinement of point cloud features. splicing features Interpolation fusion to obtain interpolation features :

[0138] (5).

[0139] P44: Interpolation Features Optimize to obtain optimized features .

[0140] In one possible implementation, by interpolating features Perform layer normalization The normalized features are input into a multilayer perceptron (MLP) and subjected to a nonlinear transformation to obtain optimized features. :

[0141] (6).

[0142] P45: Based on interpolation features and optimization features Obtain fused point cloud features .

[0143] In one possible implementation, the optimized features will be... Interpolation features Perform residual connections to obtain fused point cloud features. :

[0144] (7).

[0145] The embodiments of this application achieve adaptive feature fusion through dynamic weight masking. Combined with structure guidance and residual optimization, it not only significantly improves the geometric accuracy and rationality of point cloud restoration, but also enhances the model's adaptability to complex structures and unknown scenes.

[0146] In one possible implementation, step S150, obtaining a complete point cloud based on the fused point cloud features and partial point cloud, specifically includes:

[0147] S1501: Integrating point cloud features The input is to the first decoder, which upsamples the fused point cloud features to obtain multiple missing sub-point clouds.

[0148] In one possible implementation, the first decoder sets up multiple parallel branches, each branch fusing point cloud features through 4 layers of transposed convolutions (upsampling factor 2). (128 points) were upsampled to 1024 points, and each branch was responsible for predicting different missing regions (such as the left wing, right wing, and tail of the aircraft) to obtain multiple missing sub-point clouds.

[0149] S1502: Downsample a portion of the point cloud to obtain a downsampled point cloud (1024 points).

[0150] S1503: Combine multiple missing sub-point clouds with downsampled point clouds to obtain a stitched point cloud.

[0151] S1504: Adjust the position of point clouds in the stitched point cloud based on the multilayer perceptron to obtain a complete point cloud.

[0152] In one possible implementation, the position of the points in the stitched point cloud is adjusted by a multilayer perceptron (MLP) that outputs 3D coordinates to obtain a complete point cloud Po (4096 points).

[0153] This application embodiment inputs the fused point cloud features into the decoder to predict the sub-point cloud of the missing region, and then fuses it with the original partial point cloud to generate a complete point cloud. Through fine-grained control and end-to-end optimization, it not only preserves the original geometric structure, but also achieves high-precision and high-robust point cloud completion.

[0154] Based on the above, in one possible implementation, this application uses a three-dimensional point cloud repair model to implement a single-view guided three-dimensional point cloud repair method.

[0155] In one possible implementation, the training steps for training the 3D point cloud repair model are the same as the inference steps described above.

[0156] In one possible implementation, the training process uses the Adam optimizer (β1=0.9, β2=0.999) with an initial learning rate of The learning rate decreases by 10 times every 32 epochs, and the total number of training epochs is 320.

[0157] During training, please combine Figure 2 The total training loss includes the full loss based on the complete point cloud.

[0158] In one possible implementation, the full loss is calculated based on the L2 Chamfer distance. The formula for calculating the full loss Lcomp is as follows:

[0159] (8);

[0160] in, It is the complete point cloud ground truth, which is a complete, realistic, and dense 3D point cloud provided in the dataset that corresponds to a partial point cloud. It usually comes from CAD models or high-precision sensors and serves as a target supervision signal for the reconstruction task. , They represent point clouds respectively. , The number of midpoints The distance is Euclidean.

[0161] In one possible implementation, please combine Figure 2 Total training loss also includes reconstruction loss.

[0162] The losses incurred in reconstruction include:

[0163] Q1: Construct a second decoder with the same architecture as the first decoder.

[0164] Q2: Input the second image alignment features obtained during training (see step S130 for the acquisition process) into the second decoder to obtain the reconstructed point cloud. .

[0165] Q3: Using the ground truth of the complete point cloud corresponding to the alignment features of the second image in the training set as supervision, determine the distance between the reconstructed point cloud and the ground truth of the complete point cloud as the reconstruction loss.

[0166] In one possible implementation, the point cloud will be reconstructed. With the true value of the complete point cloud The L2 Chamfer distance between them is used as the reconstruction loss Lre:

[0167] (9);

[0168] in, , They represent point clouds respectively. , The number of midpoints.

[0169] This application embodiment uses a reconstruction loss-supervised reconstruction process to force image features to be accurately mapped to 3D geometry, further enhancing modal alignment and ensuring that image features have 3D expressive capabilities.

[0170] In one possible implementation, please combine Figure 2 The total training loss also includes anchor point optimization loss. In this embodiment, the total training loss for:

[0171] (10);

[0172] Where α and β are the weight parameters for reconstruction loss and anchor point optimization loss, respectively.

[0173] In one possible implementation, the anchor point optimization loss is obtained, specifically including:

[0174] R1: Obtain global shape features based on the second point cloud alignment features obtained during training (see step S130 for the acquisition process).

[0175] In one possible implementation, the second point cloud alignment feature is... Perform 1D convolution (Conv1D convolution kernel size is 3, output channels are 256-dimensional), then perform max pooling on the convolution result (MaxPool pooling kernel size is 2) to obtain global shape features. (256 dimensions):

[0176] (11).

[0177] R2: The anchor point update position is obtained based on the global shape features, the second point cloud alignment features, and the coordinates of the second initial anchor point corresponding to the second point cloud alignment features.

[0178] In one possible implementation, global shape features are used. Second point cloud alignment features With the second initial anchor point The coordinates are spliced ​​along the channel dimension. (Total dimensions: 256 + 256 + 3 = 515), then input into a two-layer multilayer perceptron (MLP) (the first layer outputs 256 dimensions and performs ReLU activation; the second layer outputs 3-dimensional coordinates) to update the anchor point position, thus obtaining the updated anchor point position. :

[0179] (12).

[0180] Multilayer perceptron (MLP) learns a "position offset" or "new position representation", which enables it to reasonably predict or optimize the 3D coordinates of each anchor point based on the geometric context, structural priors and feature semantics of the current point cloud, thereby updating the anchor point position. This ensures that the image is not only aligned with the point cloud in the feature space, but also aligned with the final 3D output, which is a stronger form of cross-modal supervision.

[0181] R3: Using the ground truth of the complete point cloud corresponding to the feature aligned with the second point cloud in the training set as supervision, determine the distance between the anchor update position and the ground truth of the complete point cloud, and use it as the anchor optimization loss.

[0182] For the same dataset, the ground truth of the complete point cloud here is the same as the ground truth of the complete point cloud in step Q3 above.

[0183] In one possible implementation, the complete point cloud truth value is used. For monitoring, calculation and The L2Chamfer distance is used as the anchor point to optimize the loss. :

[0184] (13);

[0185] in, Point cloud The number of midpoints.

[0186] This application embodiment uses anchor point supervision loss for explicit 3D geometric supervision, forcing the model to learn, thereby inferring more accurate and reasonable 3D anchor point positions from multimodal features. This enables these anchor points to more accurately reflect the geometric structure of the complete object, thereby improving the quality of overall point cloud representation, completion, and reconstruction.

[0187] This application employs a single-stage end-to-end architecture, achieving a single-GPU inference speed of 30fps, and can be directly integrated into engineering systems. The solution is applicable to the ShapeNet-ViPC (ShapeNet-Visual Physical Properties) synthetic dataset and the KITTI real-world LiDAR point cloud dataset. During validation, point cloud completion was achieved for eight categories of known objects, including airplanes, cars, and furniture, and four categories of unknown objects, including benches and monitors. In validation based on the ShapeNet-ViPC dataset, completion accuracy was improved by 10%-15%.

[0188] The following example is given to help illustrate the beneficial effects of this application:

[0189] 1. Experimental Environment and Parameter Settings

[0190] (1) Hardware environment: NVIDIA A6000 GPU (48GB video memory), Intel Xeon Platinum 8375C CPU, 128GB memory.

[0191] (2) Software environment: PyTorch 1.9.0, CUDA 11.4, Python 3.8, Open3D 0.15.1 (point cloud visualization).

[0192] (3) Training parameters: batch size=64, initial learning rate=1e-3, learning rate decay strategy is "0.1 every 32 epochs", total training rounds=320, weight decay=1e-5.

[0193] (4) Dataset:

[0194] Synthetic dataset: ShapeNet-ViPC (containing 13 object classes, 38,328 samples, each sample containing partial point clouds and corresponding images from 24 viewpoints);

[0195] Real dataset: KITTI 3D object detection dataset (extracted LiDAR point clouds for automotive applications, totaling 1458 samples, with some point clouds having a missing rate of 30%-50% due to occlusion).

[0196] 2. Module validity verification (ablation experiment)

[0197] The function of the core module was verified in the "Chair" and "Table Lamp" categories of ShapeNet-ViPC, and the results are shown in Table 1 below:

[0198] Table 1: Ablation test results (L2-CD×103 ↓ , Mean F-Score@0.001↑)

[0199]

[0200] The conclusions are as follows:

[0201] (1) Anchor point removal optimization: CD increased by 5.0%-7.2% (anchor points could not cover the missing area);

[0202] (2) Removing geometric enhancements: F-Score decreased by 2.1%-2.3% (unable to balance global and local features);

[0203] (3) Removal of cross-modal reconstruction: CD increased by 2.7%-4.6% (weak modal alignment).

[0204] 3. Compare experimental results

[0205] The following tables 2 and 3 compare ShapeNet-ViPC with existing mainstream methods on 8 known object classes:

[0206] Table 2: Comparison Results of Known Categories L2-CD×103

[0207]

[0208] Table 3: Comparison results of Mean F-Score @ 0.001 for known categories

[0209]

[0210] The qualitative conclusions are as follows:

[0211] (1) ShapeNet-ViPC: XMFnet cannot repair the tail of an aircraft, and EGIInet results in a blurred front of a car. This application can restore the complete global shape and clear local details (such as the curvature of a chair armrest). Please refer to [reference needed]. Figure 5 ;

[0212] (2) KITTI dataset: Existing point cloud restoration methods are prone to "front dents" in severely occluded cars. The front structure generated in this application is more coherent and has higher consistency with the real shape. Please refer to [reference needed]. Figure 6 .

[0213] 4. Explanation of the feasibility of the method

[0214] Those skilled in the art can reproduce this by following these steps:

[0215] (1) Dataset preparation: Download ShapeNet-ViPC and KITTI datasets, and preprocess some point clouds (downsampled to 1024 points) and images (adjusted resolution to 224×224).

[0216] (2) Model construction, including PyTorch-based implementation of lexicalization, shared visual converter, anchor optimization, feature transfer, geometric enhancement, decoder module, etc.

[0217] (3) Training process: Initialize model parameters, build data loader, train for 320 rounds according to the above parameters, and save the optimal model;

[0218] (4) Reasoning process: Input partial point cloud and single-view image, process them through each module to generate complete point cloud, and visualize it through Open3D.

[0219] Based on the above, this application also provides a single-view guided 3D point cloud repair device. The single-view guided 3D point cloud repair device and the aforementioned single-view guided 3D point cloud repair method can be referred to and correspond to each other.

[0220] As an example, such as Figure 7 As shown, the single-view guided 3D point cloud repair device provided in this application includes:

[0221] The receiving module 710 is used to receive partial point cloud and single-view images of the same object;

[0222] The feature extraction module 720 is used to extract point cloud feature sequences and image feature sequences from partial point cloud and single-view images, respectively.

[0223] Alignment module 730 is used to perform modal alignment between point cloud feature sequence and image feature sequence to obtain first point cloud alignment feature and first image alignment feature;

[0224] The migration module 740 is used to migrate the first image alignment features to the first point cloud alignment features to obtain the refined point cloud features.

[0225] The complete point cloud acquisition module 750 is used to obtain a complete point cloud based on the refined point cloud features and partial point cloud.

[0226] This application's embodiments eliminate feature differences between images and point clouds by modal alignment of point clouds and images, ensuring accurate subsequent feature transfer and point cloud completion, and avoiding "image-point cloud misalignment" in the completion results.

[0227] In one possible implementation, such as Figure 7 As shown, the single-view guided 3D point cloud repair device provided in this application also includes:

[0228] The fusion module 760 is used to obtain fused point cloud features based on the refined point cloud features. The fused point cloud features integrate the global and local features of the object.

[0229] In this embodiment, the complete point cloud acquisition module 750 is used to obtain a complete point cloud based on the fused point cloud features and the partial point cloud.

[0230] In this embodiment, since the refined point cloud features transfer image information to the point cloud, the refined point cloud features break through the limitations of explicit category labels on object information. The global features obtained thereby supplement the missing features of the original point cloud data. By fusing global features and local features, the entire structure of the object can be expressed, thereby accurately obtaining the complete point cloud of the object and improving the generalization ability for unknown objects.

[0231] In one possible implementation, the single-view guided 3D point cloud repair device provided in this application further includes:

[0232] Training module 770 is used to train the 3D point cloud repair model.

[0233] In one possible implementation, during training, the total training loss includes the full loss based on the complete point cloud.

[0234] Based on the above, in one possible implementation, the total training loss also includes reconstruction loss.

[0235] This application embodiment supervises the reconstruction process by reconstruction loss, forcing image features to be accurately mapped to 3D geometry, thereby further enhancing modal alignment.

[0236] Based on the above, in one possible implementation, the total training loss also includes anchor point optimization loss.

[0237] This application embodiment uses anchor point supervision loss for explicit 3D geometric supervision, forcing the model to learn, thereby inferring more accurate and reasonable 3D anchor point positions from multimodal features. This enables these anchor points to more accurately reflect the geometric structure of the complete object, thereby improving the quality of overall point cloud representation, completion, and reconstruction.

[0238] Figure 8 This is a schematic diagram of the structure of the electronic device provided in this application, such as... Figure 8 As shown, the electronic device may include: a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute a single-view guided 3D point cloud repair method, which includes:

[0239] Receive partial point cloud and single-view images of the same object;

[0240] Point cloud feature sequences and image feature sequences are extracted from partial point cloud images and single-view images, respectively.

[0241] Modal alignment is performed between the point cloud feature sequence and the image feature sequence to obtain the first point cloud alignment feature and the first image alignment feature.

[0242] The first image alignment features are transferred to the first point cloud alignment features to obtain the refined point cloud features.

[0243] A complete point cloud is obtained based on the refined point cloud features and partial point cloud data.

[0244] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0245] On the other hand, this application also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer is able to execute the single-view guided 3D point cloud repair method provided in the above embodiments. The method includes:

[0246] Receive partial point cloud and single-view images of the same object;

[0247] Point cloud feature sequences and image feature sequences are extracted from partial point cloud images and single-view images, respectively.

[0248] Modal alignment is performed between the point cloud feature sequence and the image feature sequence to obtain the first point cloud alignment feature and the first image alignment feature.

[0249] The first image alignment features are transferred to the first point cloud alignment features to obtain the refined point cloud features.

[0250] A complete point cloud is obtained based on the refined point cloud features and partial point cloud data.

[0251] In another aspect, this application also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the single-view guided 3D point cloud restoration method provided in the above embodiments, the method comprising:

[0252] Receive partial point cloud and single-view images of the same object;

[0253] Point cloud feature sequences and image feature sequences are extracted from partial point cloud images and single-view images, respectively.

[0254] Modal alignment is performed between the point cloud feature sequence and the image feature sequence to obtain the first point cloud alignment feature and the first image alignment feature.

[0255] The first image alignment features are transferred to the first point cloud alignment features to obtain the refined point cloud features.

[0256] A complete point cloud is obtained based on the refined point cloud features and partial point cloud data.

[0257] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0258] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0259] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A single-view guided method for repairing 3D point clouds, characterized in that, include: Receive partial point cloud and single-view images of the same object; Point cloud feature sequences and image feature sequences are extracted from the partial point cloud and the single-view image, respectively. The point cloud feature sequence and the image feature sequence are modally aligned to obtain a first point cloud alignment feature and a first image alignment feature. The first image alignment feature is transferred to the first point cloud alignment feature to obtain the refined point cloud feature. A complete point cloud is obtained based on the refined point cloud features and the partial point cloud; Extracting point cloud feature sequences from the aforementioned partial point cloud specifically includes: The point cloud is downsampled in multiple stages to obtain multiple first initial anchor points, and the neighborhood points of the obtained first initial anchor points are aggregated in each stage to obtain the aggregated features of the first initial anchor points. The aggregated features of all the first initial anchor points are combined into the point cloud feature sequence; The three-dimensional point cloud restoration method is implemented using a three-dimensional point cloud restoration model. The three-dimensional point cloud restoration method further includes: training the three-dimensional point cloud restoration model. During the training process, the total training loss includes the complete loss based on the complete point cloud. The total training loss also includes anchor point optimization loss; The anchor point optimization loss is obtained specifically by: Global shape features are obtained based on the second point cloud alignment features acquired during training. The anchor point update position is obtained based on the global shape feature, the second point cloud alignment feature, and the coordinates of the second initial anchor point corresponding to the second point cloud alignment feature; Using the ground truth of the complete point cloud corresponding to the alignment feature of the second point cloud in the training set as supervision, the distance between the anchor point update position and the ground truth of the complete point cloud is determined and used as the anchor point optimization loss.

2. The single-view guided 3D point cloud repair method according to claim 1, characterized in that, After obtaining the refined point cloud features, the process also includes: Based on the refined point cloud features, a fused point cloud feature is obtained, which integrates the global and local features of the object. Furthermore, a complete point cloud is obtained based on the fused point cloud features and the partial point cloud.

3. The single-view guided 3D point cloud repair method according to claim 1, characterized in that, The step of modally aligning the point cloud feature sequence with the image feature sequence to obtain a first point cloud alignment feature and a first image alignment feature specifically includes: The point cloud feature sequence and the image feature sequence are input into a visual converter to obtain the first point cloud alignment feature and the first image alignment feature. The visual converter is used to map the point cloud feature sequence and the image feature sequence into the latent space for alignment.

4. The single-view guided 3D point cloud repair method according to claim 1, characterized in that, The step of transferring the first image alignment features to the first point cloud alignment features to obtain refined point cloud features specifically includes: The first point cloud alignment feature is used as the first query content, the first image alignment feature is used as the first key, and the value of the first image alignment feature is used as the first key value. Based on the correspondence between the first query content, the first key, and the first key value, determine the image-point cloud cross-modal attention weight in the first image alignment feature; Based on the image-point cloud cross-modal attention weights, the image information in the first image alignment feature is transferred to the first point cloud alignment feature to obtain the refined point cloud feature.

5. The single-view guided 3D point cloud repair method according to claim 2, characterized in that, The process of obtaining fused point cloud features based on the refined point cloud features specifically includes: The refined point cloud features are normalized, and the normalized point cloud features are used simultaneously as the second query content, the second key, and the second key value. Global features are obtained based on the correspondence between the second query content, the second key, and the second key value; Obtain the neighborhood features of each point in the refined point cloud features, and obtain local geometric features based on the neighborhood features; The global features and the local geometric features are fused to obtain the fused point cloud features.

6. The single-view guided 3D point cloud repair method according to claim 5, characterized in that, The step of fusing the global features and the local geometric features to obtain the fused point cloud features specifically includes: By concatenating the global features and the local geometric features, a concatenated feature is obtained; Based on the refined point cloud features, a spatial variant mask is generated; Based on the spatial variant mask, the refined point cloud features and the stitched features are interpolated and fused to obtain interpolated features; The interpolation features are optimized to obtain optimized features; The fused point cloud features are obtained based on the interpolation features and the optimization features.

7. The single-view guided 3D point cloud repair method according to claim 2, characterized in that, The process of obtaining a complete point cloud based on the fused point cloud features and the partial point cloud specifically includes: The fused point cloud features are input into the first decoder, and the first decoder upsamples the fused point cloud features to obtain multiple missing sub-point clouds; The aforementioned point cloud is downsampled to obtain a downsampled point cloud; The multiple missing sub-point clouds are stitched together with the downsampled point cloud to obtain a stitched point cloud; The complete point cloud is obtained by adjusting the position of the point cloud in the stitched point cloud based on the multilayer perceptron.

8. The single-view guided 3D point cloud repair method according to claim 7, characterized in that, The total training loss also includes reconstruction loss; Obtaining the reconstruction loss specifically includes: Construct a second decoder, the architecture of which is the same as that of the first decoder; The second image alignment features obtained during training are input into the second decoder to obtain the reconstructed point cloud; Using the ground truth value of the complete point cloud corresponding to the alignment feature of the second image in the training set as supervision, the distance between the reconstructed point cloud and the ground truth value of the complete point cloud is determined as the reconstruction loss.

9. A single-view guided 3D point cloud repair device, characterized in that, The apparatus for implementing the single-view guided 3D point cloud repair method according to any one of claims 1 to 8 includes: The receiving module is used to receive partial point cloud and single-view images of the same object; The feature extraction module is used to extract point cloud feature sequences and image feature sequences from the partial point cloud and the single-view image, respectively. An alignment module is used to perform modal alignment between the point cloud feature sequence and the image feature sequence to obtain a first point cloud alignment feature and a first image alignment feature. The migration module is used to migrate the first image alignment features to the first point cloud alignment features to obtain refined point cloud features. The complete point cloud acquisition module is used to obtain a complete point cloud based on the refined point cloud features and the partial point cloud.

Images