Point cloud completion method and system based on self-supervised conditional diffusion model

Abstract

The invention belongs to the technical field of computer vision, and provides a point cloud completion method and system based on a self-supervised conditional diffusion model, and the method comprises the steps: obtaining a target image; segmenting the obtained target image, and mapping segmentation information of the target image to a point cloud space to obtain a preliminary incomplete point cloud; performing point cloud denoising processing on the obtained preliminary incomplete point cloud; and according to the denoised preliminary incomplete point cloud and a self-supervised conditional diffusion model adopting a multi-scale feature fusion mechanism, capturing point cloud space geometric features, speculating and complementing missing point cloud, generating completed point cloud data, and completing point cloud complementing based on the self-supervised conditional diffusion model.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a point cloud completion method and system based on a self-supervised conditional diffusion model. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] With the rapid development of 3D sensor technology, 3D point cloud, as an important data format, has been widely used in many fields due to its advantages of convenient acquisition and high storage efficiency. However, the scanning process in reality is often limited by the observation angle and self-occlusion, resulting in incompleteness of the generated point cloud data, which brings significant challenges to tasks such as 3D reconstruction, augmented reality, and scene understanding. Therefore, effective solutions are urgently needed to achieve high-quality point cloud completion.

[0004] In response to the problem of incomplete point cloud data, researchers have proposed a variety of deep learning-based methods, usually using Chamfer Distance (CD) or Earth Mover's Distance (EMD) as loss functions to reduce the difference between the generated point cloud and the real point cloud. However, although CD loss performs well in point cloud matching, it is not sensitive enough to the density distribution of the overall point cloud, which can easily lead to uneven distribution of the generated point cloud. On the other hand, EMD performs well in fine structure alignment and discrimination, but its high computational cost limits its application in complex scenes and large-scale data processing.

[0005] Existing shape completion methods can be roughly divided into two categories: deterministic methods and probabilistic methods. Deterministic methods rely on fixed mapping functions and can achieve good performance in point cloud completion accuracy through fully supervised learning, but they are prone to artifacts or unnatural phenomena due to overfitting; while probabilistic methods generate more reasonable completion results through autoencoders or adversarial generative networks, but they still face the dual challenges of accuracy and stability when dealing with noise-contaminated shapes. Summary of the invention

[0006] In order to solve the above problems, the present invention proposes a point cloud completion method and system based on a self-supervised conditional diffusion model, which generates uniform and high-quality point clouds through the self-supervised conditional diffusion model, so as to achieve efficient and high-quality training when using a simple mean square error loss to replace the traditional Chamfer distance and earth mover distance losses, and significantly improve the control ability of the point cloud completion process by capturing the ability of geometric details; combined with the self-supervised learning mechanism, it is possible to improve the point cloud completion effect by comparing with the point cloud generated by itself in the absence of a large amount of labeled data.

[0007] According to some embodiments, a first solution of the present invention provides a point cloud completion method based on a self-supervised conditional diffusion model, which adopts the following technical solutions:

[0008] A point cloud completion method based on a self-supervised conditional diffusion model, comprising:

[0009] Get the target image;

[0010] Segment the acquired target image, map the segmentation information of the target image into the point cloud space, and obtain a preliminary residual point cloud;

[0011] Perform point cloud denoising on the obtained preliminary residual point cloud;

[0012] According to the preliminary residual point cloud after denoising and the self-supervised conditional diffusion model with multi-scale feature fusion mechanism, the spatial geometric features of the point cloud are captured, the missing point cloud is inferred to be completed, the completed point cloud data is generated, and the point cloud completion based on the self-supervised conditional diffusion model is completed.

[0013] As a further technical limitation, the target object is photographed to obtain a single-view target image; after preprocessing the target image, SAM is used to perform image segmentation on the target image to generate segmentation information of the target image; based on the correspondence between pixels and point clouds, the obtained segmentation information is mapped to the point cloud space to obtain a preliminary residual point cloud.

[0014] As a further technical limitation, a denoising method based on density estimation is used to denoise the preliminary residual point cloud. By removing redundant points or abnormal points, point cloud data that can reflect the geometric detail information of the target object is retained to obtain the denoised preliminary residual point cloud.

[0015] As a further technical limitation, through the multi-scale feature fusion mechanism and feature consistency strategy, the self-supervised conditional diffusion model captures the local and global spatial geometric features in the preliminary incomplete point cloud after denoising, combines the multi-scale feature fusion mechanism, denoising branch and conditional branch, integrates incomplete shapes of different sizes, infers and completes the missing point cloud, and generates complete point cloud data.

[0016] Furthermore, based on the conditional branch, more features are extracted, and the extracted features are projected back into a tensor in the same format as the original residual point cloud to obtain the conditional branch point cloud; based on the obtained conditional branch point cloud and the original residual point cloud, a point cloud loss function is obtained, and the Chamfer distance between the generated point cloud and the residual point cloud is calculated according to the obtained point cloud loss function to capture the geometric information of the residual point cloud, and the quality of the generated point cloud is improved by optimizing the obtained point cloud loss function.

[0017] As a further technical limitation, the self-supervised conditional diffusion model adopts a denoising diffusion probability network, compensates for the sparsity of the original point cloud through voxelization processing, enhances the generalization ability of multi-level feature fusion and geometric information capture, gradually adds and removes Gaussian noise, and generates high-quality point clouds from random noise.

[0018] According to some embodiments, a second solution of the present invention provides a point cloud completion system based on a self-supervised conditional diffusion model, which adopts the following technical solutions:

[0019] A point cloud completion system based on a self-supervised conditional diffusion model, comprising:

[0020] An image-point cloud acquisition module, which is configured to acquire a target image and a target point cloud;

[0021] An image-point cloud mapping module is configured to segment the acquired target image, map the segmentation information of the target image to the point cloud space, and obtain a preliminary residual point cloud;

[0022] A point cloud denoising module, configured to perform point cloud denoising on the obtained preliminary residual point cloud;

[0023] The point cloud completion module is configured to capture the spatial geometric features of the point cloud based on the denoised preliminary residual point cloud and the self-supervised conditional diffusion model using a multi-scale feature fusion mechanism, infer and complete the missing point cloud, generate completed point cloud data, and complete the point cloud completion based on the self-supervised conditional diffusion model.

[0024] According to some embodiments, a third solution of the present invention provides a computer-readable storage medium, which adopts the following technical solution:

[0025] A computer-readable storage medium stores a program, which, when executed by a processor, implements the steps in the point cloud completion method based on a self-supervised conditional diffusion model as described in the first solution of the present invention.

[0026] According to some embodiments, a fourth solution of the present invention provides an electronic device, which adopts the following technical solution:

[0027] An electronic device comprises a memory, a processor and a program stored in the memory and running on the processor, wherein when the processor executes the program, the steps in the point cloud completion method based on the self-supervised conditional diffusion model as described in the first solution of the present invention are implemented.

[0028] According to some embodiments, a fifth solution of the present invention provides a computer program product, which adopts the following technical solution:

[0029] A computer program product includes software codes, wherein the program in the software codes executes the steps in the point cloud completion method based on the self-supervised conditional diffusion model as described in the first solution of the present invention.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] The present invention generates uniform and high-quality point clouds through a self-supervised conditional diffusion model to achieve efficient and high-quality training when using a simple mean square error loss instead of the traditional Chamfer distance and earth mover distance losses, and significantly improves the control ability of the point cloud completion process by capturing geometric details; combined with a self-supervised learning mechanism, it can improve the point cloud completion effect by comparing with the point cloud generated by itself in the absence of a large amount of labeled data. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The drawings in the specification that constitute a part of this embodiment are used to provide a further understanding of this embodiment. The schematic embodiments of this embodiment and their descriptions are used to explain this embodiment and do not constitute improper limitations on this embodiment.

[0033] Figure 1 4 is an architecture diagram of a point cloud completion method based on a self-supervised conditional diffusion model in Embodiment 1 of the present invention;

[0034] Figure 2 Schematic diagram of a point cloud completion method based on a self-supervised conditional diffusion model in Embodiment 1 of the present invention;

[0035] Figure 3 : is an overall structural diagram of the self-supervised conditional diffusion model in the first embodiment of the present invention;

[0036] Figure 4 This is a structural block diagram of a point cloud completion system based on a self-supervised conditional diffusion model in Embodiment 2 of the present invention. DETAILED DESCRIPTION

[0037] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0038] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0039] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0040] In the present invention, terms such as "upper", "lower", "left", "right", "front", "back", "vertical", "horizontal", "side", "bottom", etc. indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings. They are relational words determined only for the convenience of describing the structural relationships of the various parts or elements of the present invention, and do not specifically refer to any part or element in the present invention and should not be understood as limitations on the present invention.

[0041] In the present invention, terms such as "fixed connection", "connected", "connection", etc. should be understood in a broad sense, indicating that it can be fixedly connected, integrally connected or detachably connected; it can be directly connected or indirectly connected through an intermediate medium. For relevant scientific research or technical personnel in this field, the specific meanings of the above terms in the present invention can be determined according to specific circumstances, and they cannot be understood as limitations on the present invention.

[0042] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0043] Embodiment 1

[0044] Embodiment 1 of the present invention introduces a point cloud completion method based on a self-supervised conditional diffusion model.

[0045] This embodiment proposes a new type of diffusion model, namely the self-supervised conditional diffusion model SCDNet, to generate high-fidelity and natural point cloud completion results. The denoising diffusion probability model (Denoising Diffusion Probabilistic Models, DDPM) is used to gradually remove Gaussian noise to generate a uniform and high-quality point cloud, so as to achieve efficient and high-quality training when using a simple mean square error (MSE) loss instead of the traditional Chamfer distance and EMD loss; through a multi-scale feature fusion mechanism and a feature consistency strategy, the incomplete shape inputs from different scales are effectively integrated to maintain spatial consistency, enhance the model's ability to capture geometric details, and significantly improve the control ability of the completion process; a self-supervised learning mechanism is introduced to continuously improve the completion effect by comparing with the point cloud generated by itself in the absence of a large amount of labeled data.

[0046] like Figure 1As shown, this embodiment uses a RealSense depth camera to shoot the target object to obtain a single-view image and its corresponding point cloud data. In order to extract the residual point cloud of the target object, the Segment Anything Model (SAM) is used to segment the captured image to generate an accurate segmentation result of the object; the segmentation information in the image is mapped to the point cloud space using the correspondence between pixels and point clouds, thereby obtaining a preliminary residual point cloud.

[0047] In this embodiment, the acquired images may be preprocessed and data enhanced, such as rotation, scaling, translation, etc., to increase the diversity of training data and improve the adaptability of the images to different scenes.

[0048] Since the initially acquired point cloud may contain noise data (such as interference from other background objects or measurement errors of the depth camera), this embodiment introduces a denoising algorithm to process the residual point cloud. The denoising algorithm removes redundant points or abnormal points to ensure that the retained point cloud data can more accurately reflect the geometric shape of the target object, thereby obtaining a cleaner residual point cloud; this embodiment can use a denoising method based on density estimation, which can intelligently detect and remove noise points while retaining important geometric detail information to maximize the quality of the completed point cloud.

[0049] This embodiment inputs the denoised residual point cloud into the preset SCDNet for completion, effectively capturing the local and global spatial geometric features in the point cloud. The self-supervised conditional diffusion model in this embodiment adopts a multi-scale feature fusion mechanism, combined with denoising branches and conditional branches, to extract spatial information from the residual point cloud, infer and complete the missing parts, and generate complete point cloud data; using a self-supervised strategy, through reconstruction conditions, the model can better extract the features of the residual point cloud and improve the completion effect and accuracy; the self-supervised learning mechanism ensures that the completed point cloud has both overall consistency and retains rich details.

[0050] The point cloud data completed by SCDNet in this embodiment has a complete geometric form, which can provide basic data support for subsequent modeling, simulation and robot operation tasks. The generated complete point cloud can not only be used for 3D model reconstruction, but also for sophisticated robot operation tasks such as path planning and object grasping.

[0051] This solution effectively improves the integrity and accuracy of point cloud data by combining depth cameras, image segmentation and point cloud completion technology. It is suitable for point cloud completion tasks in complex scenes and provides a reliable data foundation for a variety of robot applications.

[0052] The point cloud completion process used in this embodiment is as follows: Figure 2As shown in the figure, the adopted diffusion denoising probabilistic model (DDPM) uses a Markov process to gradually add Gaussian noise to the data until the distribution of the original data is close to the prior distribution of the target; the diffusion model is applied to the point cloud completion task, the incomplete point cloud is used as conditional input, and the complete point cloud is predicted by the generative model.

[0053] Assume that the complete point cloud x in the dataset i The distribution of p data , x i The corresponding local point cloud is c i , the shape completion task is expressed as i Generate x for the condition i The generation task adopts the conditional diffusion model. Specifically, the conditional diffusion model consists of two Markov chains, which are respectively called forward diffusion process and reverse diffusion process. Each process has the same time step T, and in this embodiment, T=1000.

[0054] The forward diffusion process is to gradually add Gaussian noise to p data Until p data Close to standard normal distribution formal, For the sake of brevity, the number i is omitted in the following discussion of this embodiment. 0 to x T Defined as:

[0055]

[0056]

[0057] Among them, the hyperparameter β t ∈[0,1] is a small predefined constant used to control the noise level at each step.

[0058] The reverse process is a Markov process, which is used to predict and eliminate the noise introduced by the diffusion process, and to generate the reverse based on the conditional variables (i.e., part of the observed point cloud data c); from the latent variable x T Start by gradually restoring the clean point cloud data x 0 ; The reverse process starts from x T to x 0 The conversion is defined as:

[0059]

[0060] Among them, p θ (x t-1 |x t ,c) is Gaussian distribution:

[0061]

[0062] Mean μ θ (x t ,c,t) is parameterized by a neural network, and the variance is a constant related to the time step t. The process of generating a sample is to first sample from the standard normal distribution Then, according to the conditional probability of the reverse process, we start from x T-1 ,x T-2 ,…,x 0 Sampling in the middle, the final output is x 0 .

[0063] Through the forward and backward process in this embodiment, the complex distribution characteristics of the point cloud can be effectively learned, and finally a high-quality complete point cloud can be generated.

[0064] This embodiment adopts Figure 3 The self-supervised conditional diffusion model SCDNet (Self-supervised Conditional Diffusion Network) ∈ θ shown in the figure takes a noise point cloud x as input. t , incomplete point cloud c, and diffusion time step t. Intuitively, ∈ θ The output is understood as each point in x t and x t-1 At the same time, the self-supervised conditional diffusion model needs to effectively fuse the multi-level and multi-scale features from the conditional information c to generate fine details that meet the conditional constraints while inferring the overall shape.

[0065] As one or more implementations, point clouds are different from images because their points in three-dimensional space have no fixed order, and this disorder increases the difficulty of modeling. When training the conditional diffusion model (SCDNet), since the points in the 3D embedding space are sparsely distributed, it is difficult to capture the structural features of the data by directly optimizing the point coordinates. Therefore, this embodiment adopts a voxelization processing strategy to convert the point cloud into a dense representation, so that the diffusion model can better extract information from the voxel features;

[0066] Specifically, for each point cloud containing N points Its point coordinates are x, y, z, first voxelize it as input Where V is the size of the voxel grid; each voxel contains a three-dimensional vector, which represents the average coordinates or geometric features within the voxel. Under this representation, the diffusion transformer can more effectively capture the local structure and global geometric information of the point cloud, improving the processing ability in sparse 3D space.

[0067] In addition, to avoid the shortcomings of the traditional loss function, this embodiment adopts the denoising diffusion probability model (DDPM). Although Chamfer distance (CD) and earth mover distance (EMD) can measure the difference of point clouds, they each have their own defects; CD is sensitive to local matching, but ignores global density and easily generates unevenly distributed point clouds; while EMD is excellent in structural alignment, but has high computational complexity, i.e. These problems make it difficult to train the model efficiently on large-scale data. DDPM generates high-quality point clouds from random noise by gradually adding and removing Gaussian noise, avoiding the direct use of CD and EMD. Given the initial point cloud x0 and noise ∈, the process of generating a noisy point cloud is: Among them, α t Control the noise intensity. During training, the model minimizes the following noise prediction error, namely Here, ∈ θ is the model’s prediction of the noise. Through this step-by-step denoising, the model can generate a uniform and detailed point cloud without relying on CD and EMD.

[0068] The combination of voxelization and diffusion transformer further improves the performance of SCDNet. Voxel representation makes up for the sparsity of the original point cloud, making the model more generalizable in multi-level feature fusion and geometric information capture. Only one voxelized point cloud and its noise need to be sampled at each time step, greatly simplifying the training process.

[0069] SCDNet uses the DDPM model to achieve efficient and high-quality point cloud generation through voxelization and denoising diffusion mechanism. Compared with traditional losses, DDPM improves training efficiency with a simpler mean square error (MSE) optimization method and effectively captures the structural features in sparse 3D data, making it perform well in complex scenes.

[0070] This embodiment adopts a two-layer feature fusion architecture to effectively integrate the multi-scale features of the denoising branch and the conditional branch, aiming to improve the performance of the point cloud denoising task. In this architecture, the denoising branch is mainly responsible for processing the feature extraction of the complete point cloud, while the conditional branch extracts multi-scale conditional features from the residual point cloud. Although the network structures of the two branches are similar, they do not share parameters to better adapt to different input characteristics. In order to make full use of the contextual information provided by the conditional branch, this embodiment adopts a layer-by-layer fusion mechanism to integrate the features of the denoising branch and the conditional branch at each network level.

[0071] The denoising branch receives the input point cloud x t , through the preprocessing module ∈ x (·) Map it to a high-dimensional feature space, that is, Multi-scale features are extracted layer by layer through N SCD-3D encoding blocks, and the features of each layer are represented as At the same time, the conditional branch receives the residual defect cloud c, which is also processed by the preprocessing module ∈ c (·) Map it to a high-dimensional feature space The conditional branch also passes through N encoding blocks to extract multi-scale features

[0072] In order to effectively integrate the features of the denoising branch and the conditional branch in each layer, this embodiment performs layer-by-layer fusion at multiple network levels. Specifically, at the i-th layer, the features of the denoising branch are and the characteristics of conditional branches Add element by element to get the fused features for This fusion mechanism can effectively integrate the local and global features extracted by the denoising branch from the complete point cloud and the contextual information obtained by the conditional branch from the residual point cloud, thereby enhancing the modeling ability of the denoising branch at multiple scales.

[0073] The fused features As the input of the next layer of the denoising branch, the features are further extracted through the SCD-3D encoding block of the next layer. This mechanism of transferring fused features layer by layer ensures that conditional information can be effectively injected into each layer of the denoising branch, improving the denoising effect of the network on point cloud data. In addition, in order to adapt to the dynamic changes in time in the diffusion model, this embodiment introduces time step embedding, converts the time step t in the diffusion process into an embedding vector, and processes it through a multi-layer perceptron (MLP). The time embedding is added to the features of each layer to ensure that the network can dynamically adjust according to the input features at different time steps.

[0074] The fused multi-scale features are reconstructed through the decoder of the denoising branch. The decoder of the denoising branch restores the high-dimensional features to the noise distribution of the original point cloud, and finally matches the output of the decoder with the real Gaussian noise through the devoxelization module. This denoising process not only utilizes the complete point cloud information extracted by the denoising branch, but also combines the contextual features provided by the conditional branch, enhancing the network's ability to recover complex point cloud data.

[0075] The dual-layer feature fusion strategy in this embodiment achieves effective coordination of conditional information and denoising information by adding the features of the denoising branch and the conditional branch layer by layer. This embodiment can improve the denoising branch's ability to understand the complete point cloud at different levels by integrating the contextual information of the conditional branch in the denoising branch. In particular, when processing residual point clouds, the model can better restore detailed features. In addition, the introduction of time step embedding enables the model to adapt to dynamic changes at different time steps during the diffusion process.

[0076] In order to improve the completion effect of the model when processing incomplete point clouds, this embodiment introduces a self-supervised learning mechanism, so that the model can improve the feature extraction ability of the incomplete point cloud through self-supervision without the need for a large amount of labeled data; the core idea of ​​this innovation is to allow the conditional branch to learn the characteristics of the incomplete point cloud by comparing it with the point cloud generated by itself, thereby enhancing the performance of point cloud completion.

[0077] Specifically, the multi-scale features extracted in the conditional branch Afterwards, this embodiment designs an inverse voxelization module to map these features back to a tensor with the same format as the original residual defect cloud c; the high-dimensional features of the conditional branch are converted into a spatial distribution consistent with the residual defect cloud, denoted as Right now in, It is the final output feature of the conditional branch at layer N.

[0078] In order to make the conditional branch better learn and retain the characteristics of the residual point cloud, the point cloud generated by the conditional branch The Chamfer Distance Loss (CDLoss) is introduced between the real residual cloud c, and its expression is:

[0079] This loss function calculates the Chamfer distance between the generated point cloud and the residual defect cloud to ensure that the features extracted by the conditional branch can effectively capture the geometric information of the residual defect cloud. By optimizing CDLoss, the conditional branch gradually improves the quality of the generated point cloud, thereby improving the modeling ability of the residual defect cloud.

[0080] The self-supervision mechanism in this embodiment not only continuously improves the feature extraction capability of the conditional branch by comparing the self-generated point cloud with the residual point cloud without the need for a large amount of labeled data; combined with the two-layer architecture, it can also ensure that the information of the residual point cloud is fully retained during the feature fusion process; through CDLoss optimization, the multi-scale conditional information provided by the conditional branch is more accurate, so that the denoising branch can further improve the point cloud completion effect during the fusion process.

[0081] By introducing a self-supervised learning mechanism, the model significantly enhances its learning ability in the absence of labeled data, enabling the conditional branch to adaptively optimize the feature extraction of the residual point cloud. At the same time, the addition of CDLoss ensures that the geometric error between the point cloud generated by the conditional branch and the true residual point cloud is smaller. Combined with the two-layer feature fusion architecture, this mechanism further improves the overall effect of point cloud completion. Experimental results show that there are significant improvements in reconstructing the details of the residual point cloud and retaining local features.

[0082] Case Analysis

[0083] This embodiment conducts point cloud completion experiments on the MVP dataset; the MVP dataset contains 62,400 pairs of partially complete point clouds for training and 41,600 pairs of test point clouds (sampled from ShapeNet); each partial point cloud contains 2048 points. In particular, the MVP dataset provides real point clouds of different resolutions, including 2048, 4096, 8192, and 16384 points. In addition, the MVP dataset consists of partial and complete shapes of 16 shapes (airplane, cabinet, car, chair, lamp, sofa, table, boat, bed, bench, bookshelf, bus, guitar, motorcycle, toy, and skateboard) for training and testing. By using more shape categories, it becomes more challenging to train and evaluate the network on the MVP dataset.

[0084] This embodiment uses Chamfer distance (CD) and F1 score to evaluate the quality of the generated point cloud. The experimental results are shown in Table 1. To compensate for the problem that CD loss is sensitive to outliers, the F1 score is used to explicitly evaluate the distance between object surfaces. The F1 score is defined as the harmonic mean of precision LP(ρ) and recall LR(ρ), that is, in, And ρ is a predefined distance threshold. In this embodiment, the MVP is set to ρ=10 -4 .

[0085] Table 1 Experimental results

[0086]

[0087] This embodiment generates a uniform and high-quality point cloud through a self-supervised conditional diffusion model to achieve efficient and high-quality training when using a simple mean square error loss instead of the traditional Chamfer distance and earth mover distance loss, and significantly improves the control ability of the point cloud completion process by capturing the ability of geometric details; combined with the self-supervised learning mechanism, it can improve the point cloud completion effect by comparing with the point cloud generated by itself in the absence of a large amount of labeled data.

[0088] Embodiment 2

[0089] Embodiment 2 of the present invention introduces a point cloud completion system based on a self-supervised conditional diffusion model.

[0090] like Figure 4 A point cloud completion system based on a self-supervised conditional diffusion model is shown, comprising:

[0091] An image-point cloud acquisition module, which is configured to acquire a target image and a target point cloud;

[0092] An image-point cloud mapping module is configured to segment the acquired target image, map the segmentation information of the target image to the point cloud space, and obtain a preliminary residual point cloud;

[0093] A point cloud denoising module, configured to perform point cloud denoising on the obtained preliminary residual point cloud;

[0094] The point cloud completion module is configured to capture the spatial geometric features of the point cloud based on the denoised preliminary residual point cloud and the self-supervised conditional diffusion model using a multi-scale feature fusion mechanism, infer and complete the missing point cloud, generate completed point cloud data, and complete the point cloud completion based on the self-supervised conditional diffusion model.

[0095] The detailed steps are the same as those of the point cloud completion method based on the self-supervised conditional diffusion model provided in Example 1, and will not be repeated here.

[0096] Embodiment 3

[0097] Embodiment 3 of the present invention provides a computer-readable storage medium.

[0098] A computer-readable storage medium stores a program, which, when executed by a processor, implements the steps in the point cloud completion method based on a self-supervised conditional diffusion model as described in Embodiment 1 of the present invention.

[0099] The detailed steps are the same as those of the point cloud completion method based on the self-supervised conditional diffusion model provided in Example 1, and will not be repeated here.

[0100] Embodiment 4

[0101] A fourth embodiment of the present invention provides an electronic device.

[0102] An electronic device comprises a memory, a processor and a program stored in the memory and running on the processor, wherein when the processor executes the program, the steps in the point cloud completion method based on the self-supervised conditional diffusion model as described in the first embodiment of the present invention are implemented.

[0103] The detailed steps are the same as those of the point cloud completion method based on the self-supervised conditional diffusion model provided in Example 1, and will not be repeated here.

[0104] Embodiment 5

[0105] Embodiment 5 of the present invention provides a computer program product.

[0106] A computer program product includes software code, wherein the program in the software code executes the steps in the point cloud completion method based on the self-supervised conditional diffusion model as described in the first embodiment of the present invention.

[0107] The detailed steps are the same as those of the point cloud completion method based on the self-supervised conditional diffusion model provided in Example 1, and will not be repeated here.

[0108] The above description is only a preferred embodiment of the present embodiment and is not intended to limit the present embodiment. For those skilled in the art, the present embodiment may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present embodiment shall be included in the protection scope of the present embodiment.

Claims

1. A point cloud completion method based on a self-supervised conditional diffusion model, characterized in that: include: Get the target image; Segment the acquired target image, map the segmentation information of the target image into the point cloud space, and obtain a preliminary residual point cloud; Perform point cloud denoising on the obtained preliminary residual point cloud; According to the preliminary residual point cloud after denoising and the self-supervised conditional diffusion model with multi-scale feature fusion mechanism, the spatial geometric features of the point cloud are captured, the missing point cloud is inferred to be completed, the completed point cloud data is generated, and the point cloud completion based on the self-supervised conditional diffusion model is completed.

2. A point cloud completion method based on a self-supervised conditional diffusion model as claimed in claim 1, characterized in that: The target object is photographed to obtain a single-view target image; after preprocessing the target image, SAM is used to segment the target image to generate segmentation information of the target image; Combined with the correspondence between pixels and point clouds, the obtained segmentation information is mapped to the point cloud space to obtain a preliminary residual point cloud.

3. A point cloud completion method based on a self-supervised conditional diffusion model as claimed in claim 1, characterized in that: The denoising method based on density estimation is used to denoise the preliminary residual point cloud. By removing redundant points or abnormal points, the point cloud data that can reflect the geometric details of the target object is retained to obtain the denoised preliminary residual point cloud.

4. A point cloud completion method based on a self-supervised conditional diffusion model as claimed in claim 1, characterized in that: Through the multi-scale feature fusion mechanism and feature consistency strategy, the self-supervised conditional diffusion model captures the local and global spatial geometric features in the preliminary incomplete point cloud after denoising, combines the multi-scale feature fusion mechanism, denoising branch and conditional branch, integrates incomplete shapes of different sizes, infers and completes the missing point cloud, and generates complete point cloud data.

5. A point cloud completion method based on a self-supervised conditional diffusion model as claimed in claim 4, characterized in that: Based on the conditional branch, multiple features are extracted, and the extracted features are projected back into a tensor of the same format as the original residual point cloud to obtain the conditional branch point cloud; based on the obtained conditional branch point cloud and the original residual point cloud, a point cloud loss function is obtained, and the Chamfer distance between the generated point cloud and the residual point cloud is calculated according to the obtained point cloud loss function to capture the geometric information of the residual point cloud, and the quality of the generated point cloud is improved by optimizing the obtained point cloud loss function.

6. A point cloud completion method based on a self-supervised conditional diffusion model as claimed in claim 1, characterized in that: The self-supervised conditional diffusion model adopts a denoising diffusion probability network, compensates for the sparsity of the original point cloud through voxelization processing, enhances the generalization ability of multi-level feature fusion and geometric information capture, gradually adds and removes Gaussian noise, and generates high-quality point clouds from random noise.

7. A point cloud completion system based on a self-supervised conditional diffusion model, characterized in that: include: An image-point cloud acquisition module, which is configured to acquire a target image and a target point cloud; An image-point cloud mapping module is configured to segment the acquired target image, map the segmentation information of the target image to the point cloud space, and obtain a preliminary residual point cloud; A point cloud denoising module, configured to perform point cloud denoising on the obtained preliminary residual point cloud; The point cloud completion module is configured to capture the spatial geometric features of the point cloud based on the denoised preliminary residual point cloud and the self-supervised conditional diffusion model using a multi-scale feature fusion mechanism, infer and complete the missing point cloud, generate completed point cloud data, and complete the point cloud completion based on the self-supervised conditional diffusion model.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the point cloud completion method based on the self-supervised conditional diffusion model as described in any one of claims 1 to 6 are implemented.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the program, the steps of the point cloud completion method based on the self-supervised conditional diffusion model as described in any one of claims 1-6 are implemented.

10. A computer program product comprising software code, characterized in that The program in the software code executes the steps of the point cloud completion method based on the self-supervised conditional diffusion model as described in any one of claims 1 to 6.

Images