A new perspective local point cloud completion method and device

Abstract

The application provides a new perspective local point cloud completion method and device, steps comprising: inputting an RGB image into a monocular depth estimation network to obtain a dense depth map, and combining a camera intrinsic matrix to generate a new perspective point cloud through back projection; through geometric feature extraction, cross-point cloud feature interaction and coarse-to-fine point cloud matching, estimating a rigid body transformation between the new perspective point cloud and the point cloud to be completed, and obtaining a pre-aligned point cloud; through joint point cloud construction, point type embedding, K nearest neighbor local feature enhancement and prediction point position residual estimation, and performing local geometric correction, and fusing the pre-aligned point cloud and the point cloud to be completed to obtain a completed point cloud under a target perspective. The application can complete cross-perspective point cloud coordinate alignment by using structure prior provided by a monocular image, reduce the influence of depth estimation error and registration residual error on the completion result through local residual correction, and improve the spatial consistency and local geometric quality of the completed point cloud.

Description

Technical Field

[0001] This invention relates to the fields of 3D vision, point cloud completion, monocular depth estimation, and cross-viewpoint point cloud registration, specifically a novel method and apparatus for local point cloud completion from a new perspective. Background Technology

[0002] With the development of technologies such as 3D reconstruction, robot perception, indoor scene understanding, augmented reality, and intelligent measurement, point cloud, as a data form that can directly represent the 3D geometric structure of objects or scenes, is widely used in tasks such as target recognition, pose estimation, spatial measurement, path planning, and scene reconstruction.

[0003] In actual data acquisition, the point clouds obtained often suffer from localized missing points or incomplete structures due to factors such as sensor observation range, camera angle, target occlusion, low-texture areas, reflective surfaces, sensor noise, and the invisibility of the target surface. Missing point clouds reduce the accuracy of subsequent 3D recognition, geometric measurement, and scene reconstruction; therefore, it is necessary to restore the 3D structure of the missing areas based on existing observations.

[0004] Existing point cloud completion methods typically infer missing regions based on the spatial distribution and local geometric relationships of the point cloud to be completed. When the missing region is large, the scene structure is complex, or the point cloud to be completed lacks continuous geometric cues, relying solely on incomplete point clouds can easily lead to structural ambiguity, resulting in local distortions, missing details, and spatial position shifts in the completion results.

[0005] Image-guided point cloud completion methods can utilize contour, texture, and semantic structure information in RGB images to provide additional prior knowledge for missing region recovery. However, existing image-guided methods typically require the input image and the point cloud to be completed to be from the same viewpoint, or require a known pixel-level or point-level correspondence between them. When the RGB image and the point cloud to be completed come from different viewpoints, the 3D structure recovered from a single image is located in the image coordinate system and cannot be directly fused with the point cloud to be completed from the target viewpoint.

[0006] Monocular depth estimation networks can predict dense depth maps from a single RGB image and generate new viewpoint point clouds through backprojection using camera intrinsic parameters, providing image structure priors for cross-viewpoint point cloud completion. However, monocular depth estimation suffers from depth prediction errors, local geometric errors, and sampling errors, and the generated new viewpoint point clouds often exhibit coordinate system deviations and local structural deviations compared to the target point clouds.

[0007] Point cloud registration networks can estimate the rigid body transformation between two sets of point clouds through point cloud feature matching, solving the overall spatial alignment problem between the two sets of point clouds. However, rigid body transformation can only describe overall rotation and translation, and cannot completely eliminate local geometric deviations caused by monocular depth estimation errors, point cloud sampling errors, and registration residual errors. Therefore, after completing cross-view registration, a mechanism is needed to locally correct the predicted points using the observed local structure from the target viewpoint. Summary of the Invention

[0008] To address the problems in existing technologies, such as inconsistencies in coordinate systems between point clouds generated from monocular images and point clouds to be completed from the target viewpoint, difficulties in correcting local geometric errors through rigid body registration, and the lack of final geometric constraints in the completed results, this invention proposes a novel method and apparatus for local point cloud completion from a new viewpoint. The novel local point cloud completion process employs a monocular depth estimation network, a point cloud registration network, and a fusion residual correction module. The technical solution provided by this invention is as follows:

[0009] Firstly, a novel approach to local point cloud completion includes the following steps:

[0010] S1. Obtain a single RGB image I, the camera intrinsic parameter matrix K corresponding to the RGB image, and the point cloud Q to be completed under the target view. The point cloud Q to be completed includes the observed 3D points under the target view. The RGB image I and the point cloud Q to be completed correspond to the same target object or the same scene and have at least partially matchable geometric overlap regions.

[0011] S2. Input the RGB image I into the monocular depth estimation network to obtain the dense depth map D. Then, back-project the effective pixel positions, effective depth values ​​and camera intrinsic matrix K in the dense depth map D to generate a new viewpoint point cloud P located in the RGB image viewpoint coordinate system.

[0012] S3. Input the new viewpoint point cloud P and the point cloud Q to be completed into the point cloud registration network, extract the geometric features and contextual features of the two sets of point clouds, and obtain point cloud features for cross-viewpoint matching.

[0013] S4. Perform coarse-to-fine point cloud matching on the new perspective point cloud P and the point cloud Q to be completed, obtain candidate corresponding point sets and matching scores, and estimate the optimal rigid body transformation from the coordinate system of the new perspective point cloud P to the coordinate system of the point cloud Q to be completed. ;

[0014] S5. Utilizing optimal rigid body transformation Transform the new perspective point cloud P to the coordinate system of the point cloud Q to be completed to obtain the pre-aligned point cloud. ; pre-align point cloud The residual correction module is fused with the input of the point cloud Q to be completed, using the observed points in the point cloud Q as local geometric references to pre-align the point cloud. The predicted points in the data are locally corrected to obtain the predicted point cloud after residual correction. And keep the positions of the observed points in the point cloud Q to be completed unchanged;

[0015] S6. Modify the predicted point cloud By concatenating the point cloud Q to be completed, the completed point cloud from the target view is obtained. And retain the observed points in the point cloud Q that is yet to be completed.

[0016] Preferably, the monocular depth estimation network includes an image encoder and a depth prediction head; the image encoder is used to extract image features from the RGB image I, and the depth prediction head is used to output a dense depth map D based on the image features; for effective pixel locations in the dense depth map D, corresponding 3D spatial points are generated based on the effective depth value of the pixel location and the camera intrinsic parameter matrix K. The camera intrinsic parameter matrix K is represented as:

[0017]

[0018] in, and This represents the focal length of the camera in two image coordinate directions. and Represents the principal point coordinates of the camera; a point in three-dimensional space. The new perspective point cloud P, composed of the three-dimensional spatial points corresponding to all valid pixels, is represented as:

[0019]

[0020]

[0021] in, Represents the pixel coordinates of the image. This represents the predicted depth value of the corresponding pixel, where N is the number of new viewpoints obtained by backprojection of the effective pixels, and the effective pixels satisfy the preset depth range and effective pixel mask conditions.

[0022] Preferably, the point cloud registration network includes a point cloud encoding unit, a geometric structure feature extraction unit, a cross-point cloud feature interaction unit, a coarse-grained matching unit, and a fine-grained matching unit, which respectively perform hierarchical point cloud encoding, node partitioning, and local neighborhood construction on the new perspective point cloud P and the point cloud Q to be completed; construct geometric features based on the distance relationship between points and the angle relationship of the local neighborhood; enhance the context representation within the same group of point clouds through a self-attention mechanism; and establish the cross-point cloud feature correspondence between the new perspective point cloud P and the point cloud Q to be completed through a cross-attention mechanism.

[0023] Preferably, step S4 specifically involves: obtaining candidate node correspondences through coarse-grained node matching, and performing fine-grained matching within the local neighborhood of the candidate node correspondences to obtain a set of candidate corresponding points and a logarithmic matching score; wherein, coarse-grained node matching calculates the matching score and selects candidate nodes based on normalized feature distance, and fine-grained matching constructs a matching score matrix based on local point feature similarity, and normalizes or optimizes the matching score using Sinkhorn optimal transmission with effective point masks.

[0024] Preferred, pre-aligned point cloud satisfy:

[0025]

[0026]

[0027] in, This represents the i-th predicted point in the new perspective point cloud P. This represents the pre-alignment point after optimal rigid body transformation. For the optimal rotation matrix and The optimal translation vector is N, which represents the number of predicted points in the new viewpoint point cloud P.

[0028] Preferably, the fusion residual correction module includes a joint point cloud construction unit, an input feature mapping unit, a point type embedding unit, a local neighborhood feature enhancement unit, a feature fusion unit, and a location residual prediction unit; the joint point cloud construction unit is used to assemble the pre-aligned point cloud... The point cloud Q to be completed is concatenated to form a joint point cloud S, and the joint point cloud S is centered and scaled. The input feature mapping unit is used to map the three-dimensional coordinates in the joint point cloud S to a preset feature dimension. The point type embedding unit is used to pre-align the point cloud. Different point type identifiers are assigned to the predicted points and the observed points in the point cloud Q to distinguish the functions of the two types of points in the joint point cloud; the local neighborhood feature enhancement unit is used to construct K nearest neighbor relationships based on the joint point cloud S, generate position codes according to the relative positional relationship between the center point and neighboring points, and aggregate local geometric context features; the feature fusion unit is used to fuse the point features output by the input feature mapping unit with the local enhancement features output by the local neighborhood feature enhancement unit; the position residual prediction unit is used to predict the pre-aligned point cloud based on the fused features. The position residuals of the predicted points are calculated and then superimposed onto the corresponding pre-aligned point positions to obtain the predicted point cloud after residual correction. .

[0029] Preferably, during the training phase, the target depth map, point cloud registration transformation annotation, and complete target point cloud G corresponding to the input sample are obtained to construct a joint training target. Joint training objectives Including depth estimation loss Point cloud registration loss and complete Chamfer distance loss , is represented as:

[0030]

[0031] in, , and These represent the weighting coefficients of the corresponding loss terms.

[0032] Secondly, a novel method and apparatus for local point cloud completion includes:

[0033] The data acquisition module is used to acquire a single RGB image I, the camera intrinsic parameter matrix K, the point cloud to be completed Q from the target viewpoint, and the supervision data corresponding to the training phase.

[0034] The monocular depth estimation and point cloud generation module is used to predict the dense depth map D based on the RGB image I, and generate a new viewpoint point cloud P by combining the camera intrinsic parameter matrix K.

[0035] The cross-viewpoint point cloud registration module is used to perform multi-level geometric feature extraction, cross-point cloud feature interaction, coarse-grained matching, and fine-grained matching on the new viewpoint point cloud P and the point cloud to be completed Q, and to estimate the optimal rigid body transformation. And generate a pre-aligned point cloud. ;

[0036] The fusion residual correction module is used to integrate the pre-aligned point cloud. The point cloud Q to be completed is constructed into a joint point cloud. Based on point type embedding and K-nearest neighbor local geometric features, the position residuals of the pre-aligned predicted points are predicted, and the predicted point cloud after residual correction is obtained. And merge it with the point cloud Q to obtain the completed point cloud. ;

[0037] The joint training objective construction module is used to construct a joint training objective consisting of depth estimation loss, point cloud registration loss, and Chamfer distance completion loss based on depth supervision data, registration supervision data, and the complete point cloud of the target G during the training phase.

[0038] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0039] 1. This invention recovers dense depth information from a single RGB image using a monocular depth estimation network, and combines the camera intrinsic parameter matrix to convert the two-dimensional image structure into a three-dimensional new perspective point cloud, providing additional structural priors for the region to be completed.

[0040] 2. This invention uses a point cloud registration network to extract geometric features and perform cross-view matching on the new viewpoint point cloud and the point cloud to be completed, estimates the rigid body transformation between the two sets of point clouds, and transforms the new viewpoint point cloud in the image viewpoint to the target point cloud coordinate system, thereby reducing the overall spatial misalignment in cross-viewpoint point cloud completion.

[0041] 3. This invention uses a fusion residual correction module to jointly model the pre-aligned prediction points and the observed points under the target view, and uses the K-nearest neighbor local geometric context to predict the position residual of the prediction points, which can reduce the local geometric deviations caused by monocular depth error, point cloud sampling error and rigid body registration residual error.

[0042] 4. This invention introduces the completion Chamfer distance loss during the training phase, so that the final completed point cloud is subject to bidirectional geometric constraints of the target complete point cloud, which is beneficial to improving the overall spatial consistency and local geometric quality of the completion result.

[0043] 5. This invention integrates monocular depth estimation, cross-viewpoint point cloud registration, and fusion residual correction into a continuous processing flow, which is applicable to application scenarios such as indoor scene reconstruction, robot environmental perception, augmented reality, intelligent measurement, and 3D target reconstruction. Attached Figure Description

[0044] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0045] Figure 1 This is a flowchart of the main process of a novel local point cloud completion method according to the present invention;

[0046] Figure 2 This is a schematic diagram of a novel local point cloud completion device according to the present invention. Detailed Implementation

[0047] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0048] To make the above-mentioned objectives, features and effects of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0049] Example 1: A novel local point cloud completion method, such as... Figure 1 As shown, it includes the following steps:

[0050] S1. Data Acquisition: Acquire a single RGB image I, the camera intrinsic parameter matrix K corresponding to the RGB image, and the point cloud Q to be completed from the target viewpoint. The point cloud Q to be completed includes the observed 3D points from the target viewpoint. The viewpoint corresponding to the RGB image I is different from the target viewpoint corresponding to the point cloud Q to be completed. The RGB image I and the point cloud Q to be completed correspond to the same target object or the same scene and have at least partially matching geometric overlap regions so that the point cloud registration network can establish a correspondence.

[0051] S2. New perspective point cloud generation: Input the RGB image I into the monocular depth estimation network to obtain a dense depth map D. Then, back-project the effective pixel positions, effective depth values ​​and camera intrinsic parameter matrix K in the dense depth map D to generate a new perspective point cloud P located in the RGB image perspective coordinate system.

[0052] The monocular depth estimation network includes an image encoder and a depth prediction head; the image encoder is used to extract image features from the RGB image I, and the depth prediction head is used to output a dense depth map D based on the image features; for the effective pixel positions in the dense depth map D... Based on the effective depth value of the pixel location and the camera intrinsic parameter matrix K, the corresponding 3D spatial point is generated. Valid pixels are pixels that meet the preset depth range and do not belong to the invalid depth region. The effective depth value is the depth value of the corresponding valid pixel mask that is within the preset depth range.

[0053] The camera intrinsic parameter matrix K is represented as:

[0054]

[0055] in, and This represents the focal length of the camera in two image coordinate directions. and This indicates the coordinates of the camera's principal point.

[0056] The three-dimensional space point The new perspective point cloud P, composed of the three-dimensional spatial points corresponding to all valid pixels, is represented as:

[0057]

[0058]

[0059] in, Represents the pixel coordinates of the image. This represents the predicted depth value of the corresponding pixel, where N is the number of new viewpoints obtained by backprojection of the effective pixels, and the effective pixels satisfy the preset depth range and effective pixel mask conditions.

[0060] When generating a new perspective point cloud, invalid depth points can be removed according to a preset depth range, and valid 3D points can be sampled according to a preset number of points or spatial sampling rules to form a point cloud input for subsequent registration.

[0061] In this embodiment, an RGB image I is input into the DepthAnythingV2 depth estimation network. The DepthAnythingV2 depth estimation network encodes the image features of the RGB image I and outputs a dense depth map D through a depth prediction head. For the effective pixel locations in the depth map D... According to the corresponding depth value By performing a 3D back projection with the camera intrinsic parameter matrix K, we obtain the 3D points in the new viewpoint point cloud P. .

[0062] This embodiment adopts the above-mentioned depth back projection relationship, and the camera coordinate axis is defined as follows: the depth value is along the Z-axis direction of the camera coordinate system, and the X-axis and Y-axis coordinates are determined by the offset of the pixel coordinates relative to the principal point, the depth value, and the focal length.

[0063] In one implementation, valid depth points are selected based on preset minimum and maximum depth values, and the number of point cloud points is controlled by rule sampling, random sampling, or voxel sampling to obtain a new perspective point cloud P suitable for subsequent point cloud registration.

[0064] S3. Cross-view feature extraction: Input the new view point cloud P and the point cloud Q to be completed into the point cloud registration network, extract the geometric features and contextual features of the two sets of point clouds, and obtain point cloud features for cross-view matching.

[0065] The point cloud registration network includes a point cloud encoding unit, a geometric structure feature extraction unit, a cross-point cloud feature interaction unit, a coarse-grained matching unit, and a fine-grained matching unit. These units perform hierarchical point cloud encoding, node partitioning, and local neighborhood construction on the new perspective point cloud P and the point cloud Q to be completed, respectively. Geometric features are constructed based on the distance relationship between points and the angle relationship of the local neighborhood. The context representation within the same set of point clouds is enhanced through a self-attention mechanism, and the cross-point cloud feature correspondence between the new perspective point cloud P and the point cloud Q to be completed is established through a cross-attention mechanism.

[0066] In this embodiment, the new perspective point cloud P and the point cloud Q to be completed are input into the GeoTransformer point cloud registration network. The GeoTransformer point cloud registration network performs hierarchical point cloud encoding on the two sets of point clouds to obtain coarse-grained point cloud nodes and fine-grained point cloud features; constructs geometric structure features based on the distance and angle relationships between each point and its local neighbors; enhances the context representation within a single set of point clouds through a self-attention mechanism; and establishes the feature correspondence between the new perspective point cloud P and the point cloud Q to be completed through a cross-attention mechanism.

[0067] This embodiment obtains the cross-view point cloud correspondence based on the geometric feature extraction of GeoTransformer and the coarse-to-fine matching process.

[0068] S4. Cross-viewpoint registration: Perform coarse-to-fine point cloud matching on the new viewpoint point cloud P and the point cloud Q to be completed, obtain candidate corresponding point sets and matching scores, and estimate the optimal rigid body transformation from the coordinate system of the new viewpoint point cloud P to the coordinate system of the point cloud Q to be completed.

[0069] Candidate node correspondences are obtained through coarse-grained node matching, and fine-grained matching is performed within the local neighborhood of these candidate node correspondences to obtain a set of candidate corresponding points and a logarithmic matching score. Specifically, coarse-grained node matching calculates the matching score and selects candidate nodes based on normalized feature distance, while fine-grained matching constructs a matching score matrix based on local point feature similarity and uses Sinkhorn optimal transmission with effective point masks to normalize or optimize the matching scores. Furthermore, the optimal rigid body transformation is estimated based on the candidate corresponding point set and corresponding matching weights. Optimal rigid body transformation Including the optimal rotation matrix and optimal translation vector Optimal rotation matrix and optimal translation vector The rigid body transformation T is estimated by performing a weighted rigid body transformation on the candidate corresponding point set. The rigid body transformation T consists of a rotation matrix R and a translation vector t, and is expressed as:

[0070]

[0071] in, Represents a set of three-dimensional rotation matrices. Optimal rigid body transformation. Represented as: .

[0072] This embodiment estimates the optimal rigid body transformation based on the candidate corresponding point set and the matching score.

[0073] S5. Fusion Residual Correction: Utilizing Optimal Rigid Body Transformation Transform the new perspective point cloud P to the coordinate system of the point cloud Q to be completed to obtain the pre-aligned point cloud. ; pre-align point cloud The residual correction module is fused with the input of the point cloud Q to be completed, using the observed points in the point cloud Q as local geometric references to pre-align the point cloud. The predicted points in the cloud are locally corrected, while the positions of the observed points in the point cloud Q to be completed remain unchanged. (Pre-aligned point cloud) satisfy:

[0074]

[0075]

[0076] in, This represents the i-th predicted point in the new perspective point cloud P. This represents the pre-aligned point after optimal rigid body transformation. N represents the number of predicted points in the new viewpoint point cloud P.

[0077] The fusion residual correction module includes a joint point cloud construction unit, an input feature mapping unit, a point type embedding unit, a local neighborhood feature enhancement unit, a feature fusion unit, and a location residual prediction unit; the joint point cloud construction unit is used to assemble the pre-aligned point cloud... The point cloud Q to be completed is concatenated to form a joint point cloud S, and the joint point cloud S is centered and scaled. The input feature mapping unit is used to map the three-dimensional coordinates in the joint point cloud S to a preset feature dimension. The point type embedding unit is used to pre-align the point cloud. Different point type identifiers are assigned to the predicted points and the observed points in the point cloud Q to distinguish the functions of the two types of points in the joint point cloud; the local neighborhood feature enhancement unit is used to construct K nearest neighbor relationships based on the joint point cloud S, generate position codes according to the relative positional relationship between the center point and neighboring points, and aggregate local geometric context features; the feature fusion unit is used to fuse the point features output by the input feature mapping unit with the local enhancement features output by the local neighborhood feature enhancement unit; the position residual prediction unit is used to predict the pre-aligned point cloud based on the fused features. The position residuals of the predicted points are calculated and then superimposed onto the corresponding pre-aligned point positions to obtain the predicted point cloud after residual correction. .

[0078] The joint point cloud S is composed of a pre-aligned point cloud The result is obtained by concatenating the point Q to be completed along the point dimension, and is represented as:

[0079]

[0080] Among them, symbols This indicates stitching along the point dimension. Centering the joint point cloud S yields the center coordinates of the joint point cloud. :

[0081]

[0082] in, This represents the number of points in the joint point cloud S. Let represent the i-th point in the joint point cloud S. Based on the joint point cloud S and its center coordinates... Determine the scale factor :

[0083]

[0084] in, Denotes the L2 norm. Based on the central coordinates... and scale factor Normalize the point coordinates in the joint point cloud to obtain normalized point coordinates. and the normalized point set consisting of all normalized point coordinates :

[0085]

[0086]

[0087] in, Let represent the normalized coordinates of the i-th point in the joint point cloud S. Input the normalized point coordinates into a point coordinate feature mapping network and overlay point type embeddings to obtain the initial point features. The initial feature set consists of all the features of the initial points. :

[0088]

[0089] in, Represents a point coordinate feature mapping network. For point type identifier, This represents the point type embedding corresponding to the point type identifier. The initial feature set, normalized point set, and K-nearest neighbor structure are input into the local feature enhancement unit to obtain the enhanced local geometric features H:

[0090]

[0091] in, It is a K-nearest neighbor structure. This represents a local feature enhancement unit. Based on the enhanced local geometric features H, the position residuals of the pre-aligned prediction points are obtained through the residual prediction head. :

[0092]

[0093] in, Indicates the residual prediction head. This indicates the upper limit of the residual amplitude. The position residual... Overlay to the corresponding pre-aligned point The point coordinates after residual correction are obtained. The residual-corrected point coordinates constitute the predicted point cloud, which is represented as follows:

[0094]

[0095]

[0096] Among them, the position residual is only applied to the pre-aligned prediction point, and the observed points in the point cloud Q to be completed retain their original coordinates.

[0097] In this embodiment, the FRM fusion residual correction module outputs a 3D position residual based on the local geometric context features corresponding to the predicted point. This position residual is applied only to the pre-aligned point cloud. The predicted points in the cloud are not applied to the observed points in the point cloud Q to be completed.

[0098] This embodiment centers and normalizes the scale of the joint point cloud, and constructs input features, point type embeddings, and local relative position features to obtain fused features for residual prediction.

[0099] In this embodiment, a bounded position residual is output for the pre-aligned prediction point, and the position residual is added to the coordinates of the corresponding pre-aligned point.

[0100] In one implementation, the position residual is bounded so that the magnitude of each coordinate component or residual vector of the position residual does not exceed a preset threshold, in order to avoid excessive point position offset caused by local correction.

[0101] S6. Point Cloud Fusion: Fusion of the predicted point cloud after residual correction. By concatenating the point cloud Q to be completed, the completed point cloud from the target view is obtained. And retain the observed points in the point cloud Q to be completed. Complete the point cloud. Represented as:

[0102]

[0103] During the training phase, the target depth map, point cloud registration and transformation annotations, and the complete target point cloud G corresponding to the input samples are obtained to construct a joint training target. Joint training objectives Including depth estimation loss Point cloud registration loss and complete Chamfer distance loss , represented as:

[0104]

[0105] in, , and These represent the weighting coefficients of the corresponding loss terms.

[0106] In this joint training objective, the depth estimation loss is used to constrain the depth difference between the predicted depth map D and the target depth map; the point cloud registration loss includes coarse-grained matching loss and fine-grained matching loss, which are used to constrain the point cloud registration network to establish correct cross-view correspondences; and the completion Chamfer distance loss is used to constrain the final completed point cloud. The bidirectional geometric distance between the target complete point cloud G and the target point cloud G.

[0107] Point cloud registration loss Due to coarse-grained matching loss and fine-grained matching loss Composition, represented as:

[0108]

[0109] in, The weighted circular loss is calculated based on the matching mask for positive and negative nodes, the node feature distance, and the overlap of positive samples. For fine-grained matching loss, and These represent the weight coefficients of the corresponding loss terms. Fine-grained matching loss. Represented as:

[0110]

[0111] Where Y is the set of effective fine-grained matching tags. The corresponding matching score is obtained from the optimal transmission of the Sinkhorn. The depth estimation loss... Represented as:

[0112]

[0113] Where M is the number of effective pixels. Preset coefficients, depth error term Represented as:

[0114]

[0115] in, This represents the predicted depth value of the i-th valid pixel. This represents the target depth value of the i-th valid pixel. Complete the Chamfer distance loss. Represented as:

[0116]

[0117] Here, G represents the target complete point cloud, and a preset upper limit can be set for the nearest neighbor distance during training. This embodiment calculates the symmetric Chamfer distance loss; the code implementation first calculates the pairwise distances between the two sets of point clouds, then obtains the nearest neighbor distances in both directions and averages them, setting an upper limit for the nearest neighbor distance if necessary. The target complete point cloud G is only used during the training phase to calculate the completion Chamfer distance loss; during the inference phase, the completed point cloud can be obtained based on the RGB image, the camera intrinsic parameter matrix, and the point cloud to be completed. .

[0118] In practice, during the training phase, a target depth map, point cloud registration transformation annotations, and a complete target point cloud G are configured for each set of input samples. The depth estimation loss is calculated based on the predicted depth map and the target depth map. The point cloud registration loss is calculated based on the cross-view point cloud registration results and registration annotations, and the complete point cloud is then used for further calculations. The Chamfer distance loss is calculated by combining the target complete point cloud G with the target complete point cloud; the losses are then combined into a joint training objective according to preset weights. The target complete point cloud G is only used for loss calculation during the training phase and is not used as input during the inference phase.

[0119] During the evaluation phase, the completed point cloud will be recorded as... Let G be the complete point cloud of the target corresponding to it. For the predicted point... The nearest neighbor points in the target point cloud G are used. Calculate the distance from the predicted point to the target point cloud; for the target point... Using point cloud completion nearest neighbor in Calculate the distance from the target point cloud to the completed point cloud. In this embodiment, the distance determination threshold is used in the evaluation phase. Complete the result The value is represented as:

[0120]

[0121] in, This is the harmonic mean of Precision and Recall. The Precision of the completed result is expressed as:

[0122]

[0123] in, This is an indicator function; it takes the value 1 when the condition within the parentheses is met, and 0 otherwise. The recall expression for the completed result is:

[0124]

[0125] in, This represents the number of points in the target complete point cloud G. The CDL2 representation of the completion result is:

[0126]

[0127] CDL2 is used to evaluate the squared nearest neighbor distance error between the completed point cloud and the target complete point cloud. The CDL1 representation of the completion result is as follows:

[0128]

[0129] in, Indicates completion of point cloud The number of points in the point cloud; CDL1 is used to evaluate the nearest neighbor distance error between the completed point cloud and the target complete point cloud.

[0130] The higher the values ​​of Precision and Recall, the better the completion result; the lower the values ​​of CDL1 and CDL2, the smaller the geometric error between the completed point cloud and the target complete point cloud.

[0131] In this embodiment, the nearest neighbor search of the evaluation metrics is respectively based on G represents the query set and the reference set, respectively. The evaluation metrics are used to evaluate the completion results without changing the processing flow of depth estimation, point cloud registration, fusion residual correction, and point cloud fusion.

[0132] Example 2: A novel local point cloud completion device, such as Figure 2 As shown, it includes:

[0133] The data acquisition module is used to acquire a single RGB image I, the camera intrinsic parameter matrix K, the point cloud to be completed Q from the target viewpoint, and the supervision data corresponding to the training phase.

[0134] The monocular depth estimation and point cloud generation module is used to predict the dense depth map D based on the RGB image I, and generate a new viewpoint point cloud P by combining the camera intrinsic parameter matrix K.

[0135] The cross-viewpoint point cloud registration module is used to perform multi-level geometric feature extraction, cross-point cloud feature interaction, coarse-grained matching, and fine-grained matching on the new viewpoint point cloud P and the point cloud to be completed Q, and to estimate the optimal rigid body transformation. And generate a pre-aligned point cloud. ;

[0136] The fusion residual correction module is used to integrate the pre-aligned point cloud. The point cloud Q to be completed is constructed into a joint point cloud. Based on point type embedding and K-nearest neighbor local geometric features, the position residuals of the pre-aligned predicted points are predicted, and the predicted point cloud after residual correction is obtained. And merge it with the point cloud Q to obtain the completed point cloud. ;

[0137] The joint training objective construction module is used to construct a joint training objective consisting of depth estimation loss, point cloud registration loss, and Chamfer distance completion loss based on depth supervision data, registration supervision data, and the complete point cloud of the target G during the training phase.

[0138] Example 3: The computer-readable storage medium of this example stores a computer program that, when executed by a processor, implements the steps of the novel perspective local point cloud completion method in Example 1.

[0139] The computer-readable storage medium in this embodiment can be an internal storage unit of the terminal, such as the terminal's hard disk or memory; the computer-readable storage medium in this embodiment can also be an external storage device of the terminal, such as a plug-in hard disk, smart memory card, secure digital card, flash memory card, etc. equipped on the terminal; furthermore, the computer-readable storage medium can include both the terminal's internal storage unit and external storage devices.

[0140] The computer-readable storage medium of this embodiment is used to store computer programs and other programs and data required by the terminal. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0141] Example 4: The computer device of this example includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the novel perspective local point cloud completion method of Example 1.

[0142] In this embodiment, the processor can be a central processing unit, or other general-purpose processors, digital signal processors, application-specific integrated circuits, off-the-shelf programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. The memory can include read-only memory and random access memory, and provides instructions and data to the processor. A portion of the memory can also include non-volatile random access memory. For example, the memory can also store device type information.

[0143] Those skilled in the art will clearly understand that each implementation can be achieved using software plus the necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0144] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A novel method for local point cloud completion, characterized in that, Includes the following steps: S1. Acquire a single RGB image I, the camera intrinsic parameter matrix K corresponding to the RGB image, and the point cloud Q to be completed under the target view. The point cloud Q to be completed includes the three-dimensional points observed under the target view. The RGB image I and the point cloud Q to be completed correspond to the same target object or the same scene, and have at least partially matching geometrically overlapping regions; S2. Input the RGB image I into the monocular depth estimation network to obtain the dense depth map D. Then, back-project the effective pixel positions, effective depth values ​​and camera intrinsic matrix K in the dense depth map D to generate a new viewpoint point cloud P located in the RGB image viewpoint coordinate system. S3. Input the new viewpoint point cloud P and the point cloud Q to be completed into the point cloud registration network, extract the geometric features and contextual features of the two sets of point clouds, and obtain point cloud features for cross-viewpoint matching. S4. Perform coarse-to-fine point cloud matching on the new perspective point cloud P and the point cloud Q to be completed, obtain candidate corresponding point sets and matching scores, and estimate the optimal rigid body transformation from the coordinate system of the new perspective point cloud P to the coordinate system of the point cloud Q to be completed. ; S5. Utilizing optimal rigid body transformation Transform the new perspective point cloud P to the coordinate system of the point cloud Q to be completed to obtain the pre-aligned point cloud. ; Pre-aligned point cloud The residual correction module is fused with the input of the point cloud Q to be completed, using the observed points in the point cloud Q as local geometric references to pre-align the point cloud. The predicted points in the data are locally corrected to obtain the predicted point cloud after residual correction. And keep the positions of the observed points in the point cloud Q to be completed unchanged; S6. Modify the predicted point cloud By concatenating the point cloud Q to be completed, the completed point cloud from the target view is obtained. And retain the observed points in the point cloud Q that is yet to be completed.

2. The novel perspective local point cloud completion method according to claim 1, characterized in that, The monocular depth estimation network includes an image encoder and a depth prediction head. The image encoder extracts image features from the RGB image I, and the depth prediction head outputs a dense depth map D based on the image features. For valid pixel locations in the dense depth map D, corresponding 3D spatial points are generated based on the valid depth value of the pixel location and the camera intrinsic parameter matrix K. The camera intrinsic parameter matrix K is represented as: ； in, and This represents the focal length of the camera in two image coordinate directions. and Represents the principal point coordinates of the camera; a point in three-dimensional space. The new perspective point cloud P, composed of the three-dimensional spatial points corresponding to all valid pixels, is represented as: ； ； in, Represents the pixel coordinates of the image. This represents the predicted depth value of the corresponding pixel, where N is the number of new viewpoints obtained by backprojection of the effective pixels, and the effective pixels satisfy the preset depth range and effective pixel mask conditions.

3. The novel perspective local point cloud completion method according to claim 1, characterized in that, The point cloud registration network includes a point cloud encoding unit, a geometric structure feature extraction unit, a cross-point cloud feature interaction unit, a coarse-grained matching unit, and a fine-grained matching unit, which respectively perform hierarchical point cloud encoding, node partitioning, and local neighborhood construction on the new perspective point cloud P and the point cloud Q to be completed; and construct geometric features based on the distance relationship between points and the angle relationship of the local neighborhood. The contextual representation within the same set of point clouds is enhanced by a self-attention mechanism, and the cross-point cloud feature correspondence between the new perspective point cloud P and the point cloud Q to be completed is established by a cross-attention mechanism.

4. The novel perspective local point cloud completion method according to claim 1, characterized in that, Specifically, S4 involves obtaining candidate node correspondences through coarse-grained node matching, and performing fine-grained matching within the local neighborhood of the candidate node correspondences to obtain a set of candidate corresponding points and a logarithmic matching score. Specifically, coarse-grained node matching calculates the matching score based on the normalized feature distance and selects candidate nodes, while fine-grained matching constructs a matching score matrix based on the local point feature similarity and uses Sinkhorn optimal transmission with an effective point mask to normalize or optimize the matching score.

5. The novel perspective local point cloud completion method according to claim 4, characterized in that, Pre-aligned point cloud satisfy: ； ； in, This represents the i-th predicted point in the new perspective point cloud P. This represents the pre-alignment point after optimal rigid body transformation. For the optimal rotation matrix and The optimal translation vector is N, which represents the number of predicted points in the new viewpoint point cloud P.

6. The novel perspective local point cloud completion method according to claim 5, characterized in that, The fusion residual correction module includes a joint point cloud construction unit, an input feature mapping unit, a point type embedding unit, a local neighborhood feature enhancement unit, a feature fusion unit, and a location residual prediction unit; the joint point cloud construction unit is used to assemble the pre-aligned point cloud... The point cloud Q to be completed is stitched together to form a joint point cloud S, and the joint point cloud S is centered and scale normalized; the input feature mapping unit is used to map the three-dimensional coordinates in the joint point cloud S to a preset feature dimension. The point type embedding unit is used for pre-aligning point clouds. Different point type identifiers are assigned to the predicted points and the observed points in the point cloud Q to be completed, so as to distinguish the functions of the two types of points in the joint point cloud; the local neighborhood feature enhancement unit is used to construct K nearest neighbor relationships based on the joint point cloud S, generate position codes according to the relative positional relationship between the center point and the neighboring points, and aggregate local geometric context features. The feature fusion unit is used to fuse the point features output by the input feature mapping unit with the local enhanced features output by the local neighborhood feature enhancement unit; the position residual prediction unit is used to predict the pre-aligned point cloud based on the fused features. The position residuals of the predicted points are calculated and then superimposed onto the corresponding pre-aligned point positions to obtain the predicted point cloud after residual correction. .

7. The novel perspective local point cloud completion method according to claim 1, characterized in that, During the training phase, the target depth map, point cloud registration and transformation annotations, and the complete target point cloud G corresponding to the input samples are obtained to construct a joint training target. Joint training objectives Including depth estimation loss Point cloud registration loss and complete Chamfer distance loss , is represented as: ； in, , and These represent the weighting coefficients of the corresponding loss terms.

8. A novel method and apparatus for local point cloud completion, characterized in that, include: The data acquisition module is used to acquire a single RGB image I, the camera intrinsic parameter matrix K, the point cloud to be completed Q from the target viewpoint, and the supervision data corresponding to the training phase. The monocular depth estimation and point cloud generation module is used to predict the dense depth map D based on the RGB image I, and generate a new viewpoint point cloud P by combining the camera intrinsic parameter matrix K. The cross-viewpoint point cloud registration module is used to perform multi-level geometric feature extraction, cross-point cloud feature interaction, coarse-grained matching, and fine-grained matching on the new viewpoint point cloud P and the point cloud to be completed Q, and to estimate the optimal rigid body transformation. And generate a pre-aligned point cloud. ; The fusion residual correction module is used to integrate the pre-aligned point cloud. The point cloud Q to be completed is constructed into a joint point cloud. Based on point type embedding and K-nearest neighbor local geometric features, the position residuals of the pre-aligned predicted points are predicted, and the predicted point cloud after residual correction is obtained. And merge it with the point cloud Q to obtain the completed point cloud. ; The joint training objective construction module is used to construct a joint training objective consisting of depth estimation loss, point cloud registration loss, and Chamfer distance completion loss based on depth supervision data, registration supervision data, and the complete point cloud of the target G during the training phase.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the novel perspective local point cloud completion method as described in any one of claims 1-7.

10. A computer device comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the novel perspective local point cloud completion method as described in any one of claims 1-7.

Images