Three-dimensional point cloud completion method based on three-dimensional Gaussian spot dyeing and fractional distillation sampling

Abstract

The invention discloses a three-dimensional point cloud completion method and system based on fractional distillation sampling and three-dimensional Gaussian dyeing, and relates to the field of three-dimensional reconstruction and computer vision. The method comprises the following steps: inputting incomplete point cloud, enhancing initial point cloud density through point densification operation, and constructing a high-quality three-dimensional Gaussian model; rendering a multi-view image and optimizing Gaussian parameters through differentiable Gaussian rasterization by utilizing a fractional distillation sampling technology and taking a pre-trained two-dimensional diffusion model as guidance; dynamically adjusting a viewpoint range by combining a progressive camera sampling strategy, and ensuring the consistency of an image and a model; and finally generating a complete point cloud. The method does not need to depend on a paired data set, solves the problems that a traditional supervised method is poor in generalization and a non-supervised method is low in efficiency, remarkably improves the precision and efficiency of point cloud completion in a complex real scene, and provides reliable support for tasks such as object recognition and autonomous navigation.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and three-dimensional reconstruction, and specifically relates to a three-dimensional point cloud completion method and system based on fractional distillation sampling and three-dimensional Gaussian dot rendering. Background Art

[0002] In the field of three-dimensional point cloud data processing, completing incomplete point clouds caused by limited sensor viewing angles, object occlusions, or insufficient resolution is a core challenge to improving the performance of downstream tasks such as object recognition, scene understanding, and autonomous navigation. Existing technical solutions mainly rely on supervised learning paradigms. For example, training models based on virtual data sets requires input of paired complete point clouds and incomplete point cloud data. However, it is expensive to obtain such strictly paired data in real scenes, and the virtual incomplete point clouds generated by algorithmic simulation are difficult to accurately reproduce complex real-world conditions (such as multiple occlusions, noise interference, and non-uniform sampling), resulting in a significant decrease in the generalization performance of the model in practical applications. In addition, unsupervised methods (such as technologies based on generative adversarial networks or latent space mapping) generally have low completion accuracy and efficiency due to the lack of point-by-point supervision signals.

[0003] In recent years, fractional distillation sampling (SDS) technology has become a research hotspot by providing cross-modal guidance for 3D generation tasks through the strong prior knowledge of pre-trained 2D diffusion models. However, traditional SDS frameworks rely on volume rendering technologies such as neural radiance fields, which have high computational complexity and are difficult to efficiently process large-scale point cloud data. 3D Gaussian stippling (3D-GS) technology uses Gaussian distribution to model the spatial distribution of point clouds and combines it with differentiable rasterization to achieve real-time rendering, significantly improving efficiency. However, existing methods still have obvious defects: sparse input point clouds lead to insufficient initialization density of the Gaussian model, and key geometric details are easily lost during the optimization process; at the same time, the 2D rendered images generated by the random camera sampling strategy do not match the 3D spatial distribution of the target object, and cannot effectively guide the model to repair the missing areas, causing noise accumulation and optimization direction deviation. Summary of the Invention

[0004] One of the purposes of the present invention is to provide a three-dimensional point cloud completion method based on three-dimensional Gaussian stippling and fractional distillation sampling to solve the problems of loss of geometric details and decreased completion accuracy caused by sparse input, low rendering efficiency and insufficient optimization robustness in the existing technology; the second purpose is to provide a three-dimensional point cloud completion system that is adapted to complex real scenes, and through unsupervised optimization mechanism and efficient rendering technology, provide high-quality and complete point cloud data support for tasks such as object recognition, autonomous navigation and industrial inspection.

[0005] In order to solve the above technical problems, the technical solutions of the present invention are as follows:

[0006] Acquire incomplete three-dimensional point cloud data of the object to be completed, perform point densification processing on the point cloud data to generate a high-density point cloud, and use the high-density point cloud to initialize a Gaussian parameter set to generate an initial three-dimensional Gaussian model;

[0007] A progressive camera sampling strategy is used to dynamically adjust the viewing angle parameter range, render and generate a 2D image, calculate the fractional distillation sampling loss using a pre-trained 2D diffusion model, and optimize the 3D Gaussian parameters in combination with opacity constraints.

[0008] The Gaussian distribution density is adjusted through the adaptive density control mechanism to iteratively update the model geometric features;

[0009] Convert the optimized 3D Gaussian model into a 3D point cloud and output it.

[0010] Preferably, the obtaining of incomplete three-dimensional point cloud data of the object to be completed includes:

[0011] The original point cloud of the object is collected by 3D laser radar, and the outlier noise points are removed based on the radius filtering method to complete the data preprocessing.

[0012] Preferably, the point densification process includes:

[0013] Randomly sample candidate points within the minimum bounding box of the input point cloud, remove the deviation points whose Euclidean distance to the original point cloud exceeds a first threshold, form a first point set, and merge the input point cloud and the first point set into the second point set;

[0014] For the points in the second point set whose nearest neighbor distance is less than the second threshold, randomly generate new points in their neighborhood to form the third point set;

[0015] Performing multiple copy operations on points in the second point set whose nearest neighbor distances are greater than a third threshold value to form a fourth point set;

[0016] Finally, the second point set, the third point set and the fourth point set are merged into a high-density point cloud.

[0017] Preferably, the generating of the initial three-dimensional Gaussian model by initializing the Gaussian parameter set using the high-density point cloud comprises:

[0018] The three-dimensional coordinates of each point in the high-density point cloud are used as the Gaussian center position parameters;

[0019] The distance between adjacent points of each point in the high-density point cloud is used as the covariance matrix parameter;

[0020] Assign randomly generated color parameters and preset initial opacity parameters to each Gaussian point;

[0021] The position parameters, covariance matrix parameters, color parameters and opacity parameters are fused into a three-dimensional Gaussian parameter set to form a three-dimensional Gaussian model with complete geometric representation.

[0022] Preferably, the progressive camera sampling strategy includes:

[0023] Extract the initial camera pose parameters based on the geometric center of the input point cloud, dynamically expand the sampling range through linear interpolation based on the initial values ​​of the direction angle, elevation angle, distance and field of view;

[0024] In the early stages of optimization iterations, based on the ratio of the current iteration step to the total iteration step number, the azimuth angle fluctuation amplitude, elevation angle offset range, camera distance span, and field of view coverage area are gradually relaxed to ensure that the rendering viewpoint gradually extends from the initial perspective to the global space.

[0025] At the same time, the upper limit of the absolute value of the elevation angle and the reasonable range of the camera distance are constrained to avoid geometric distortion caused by extreme viewing angles;

[0026] In the later stage of optimization, we switch to global random sampling mode to evenly cover all potential perspectives in the three-dimensional space. Through high-frequency multi-angle rendering, we refine the geometric details and surface continuity of the Gaussian model, and ultimately achieve a balanced optimization between spatial consistency and local accuracy of the completion results.

[0027] Preferably, the rendering to generate a two-dimensional image includes:

[0028] A virtual viewpoint is constructed based on the dynamically sampled camera direction angle, elevation angle, distance and field of view parameters, and the 3D Gaussian model is projected onto the imaging plane using differentiable Gaussian rasterization technology.

[0029] According to the location parameters, covariance matrix parameters and color parameters of the Gaussian points, the cumulative opacity and weighted color value of the Gaussian distribution in the coverage area of ​​each pixel are calculated to generate a rendered image.

[0030] Preferably, calculating the fractional distillation sampling loss using a pre-trained two-dimensional diffusion model comprises:

[0031] A three-dimensional Gaussian model is projected onto a two-dimensional imaging plane using differentiable rendering technology to generate a noise image, and Gaussian noise of a preset noise level is injected into the rendered image to form a disturbance input;

[0032] Based on the textual prompts corresponding to the noise image and the input point cloud categories, the noise distribution is predicted using a U-Net network with a pre-trained 2D diffusion model. The fractional distillation sampling loss function is constructed by matching the mean squared error between the predicted noise and the injected noise.

[0033] The gradient truncation strategy is used to skip the back propagation path of the U-Net network parameters and only retain the gradient information of the noisy prediction output.

[0034] Preferably, the combining of fractional distillation sampling loss and opacity constraint to optimize three-dimensional Gaussian parameters comprises:

[0035] The difference between the rendered image and the semantic prior of the 2D diffusion model is quantified through the fractional distillation sampling loss, driving the position, covariance and color parameters of the 3D Gaussian model to evolve towards a complete geometric structure.

[0036] At the same time, the opacity constraint loss is introduced to force the opacity mean of the Gaussian points to maintain the preset threshold during the optimization process, avoiding the loss of key features due to excessive sparsification of the model;

[0037] Through back-propagation, the joint loss gradient is injected into the three-dimensional Gaussian parameter set to achieve collaborative optimization of geometric completion accuracy and rendering consistency, and finally generate a complete three-dimensional point cloud with multi-view consistency, surface continuity and semantic rationality.

[0038] Preferably, the adaptive density control mechanism includes:

[0039] The point distribution is dynamically adjusted based on the gradient amplitude and neighborhood density of the three-dimensional Gaussian parameters, and the gradient amplitude and nearest neighbor distance of each Gaussian point are periodically detected during the iteration process;

[0040] When the gradient amplitude of a Gaussian point exceeds the preset gradient threshold and the neighborhood distance is greater than the spatial distribution threshold, a splitting operation is performed to randomly generate multiple sub-Gaussian points in its neighborhood to enhance the local detail depiction;

[0041] When the gradient amplitude exceeds the threshold but the neighborhood distance is within a reasonable range, a cloning operation is performed to generate multiple Gaussian points with the same parameters to improve surface continuity;

[0042] In the late optimization stage, the cloning operation is frozen and the split range is limited, and the proliferation of redundant points is suppressed by the distance threshold constraint;

[0043] At the same time, the Gaussian parameters of the initial point cloud are retained as geometric anchor points to prevent the degradation of key features during the optimization process, and ultimately achieve dynamic balanced optimization of the three-dimensional Gaussian model between geometric accuracy, detail density and computational efficiency.

[0044] Preferably, converting the three-dimensional Gaussian model into complete point cloud data includes:

[0045] Extract the center coordinates of each Gaussian point in the optimized three-dimensional Gaussian model as the spatial coordinates of the completed point cloud;

[0046] Low opacity Gaussian points are filtered by threshold to remove redundant noise data and retain the Gaussian point set with significant geometric contribution;

[0047] Convert the color parameters based on spherical harmonic function encoding into RGB color parameters;

[0048] Finally, the filtered spatial coordinates and color parameters are used to generate a complete three-dimensional point cloud, which is then stored in a standardized format for direct application in downstream tasks.

[0049] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0050] The present invention gets rid of the dependence on large-scale paired data sets, and achieves more generalized point cloud completion through pre-trained diffusion models and unsupervised optimization; combining the efficient rendering of 3D-GS and the precise guidance of SDS, it significantly improves the completion speed and is suitable for large-scale complex point clouds; point densification operations and progressive sampling strategies effectively alleviate the loss of geometric features caused by sparse input, ensuring the detail integrity and structural consistency of the completion results; it can handle various incomplete point clouds caused by sensor acquisition, occlusion or low resolution, and provide high-quality three-dimensional data support for tasks such as object recognition and autonomous navigation.

[0051] Contents of attached figure

[0052] Figure 1 This is a flowchart of a three-dimensional point cloud completion method based on three-dimensional Gaussian stippling and fractional distillation sampling as described in Example 1;

[0053] Figure 2 This is a diagram showing the completion results of Example 2;

[0054] Figure 3 Detailed display of the completion result of Example 2 Specific embodiments

[0055] The accompanying drawings are for illustrative purposes only and are not to be construed as limiting this patent;

[0056] In order to better illustrate this embodiment, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product size;

[0057] It is understandable to those skilled in the art that some well-known structures and descriptions thereof may be omitted in the drawings.

[0058] The technical solution of the present invention is further described below with reference to the accompanying drawings and embodiments.

[0059] Example 1

[0060] This embodiment provides a 3D point cloud completion method based on 3D Gaussian dot coloring and fractional distillation sampling, such as Figure 1 As shown, including:

[0061] S1: Acquire incomplete three-dimensional point cloud data of the object to be completed, perform point densification processing on the point cloud data to generate a high-density point cloud, and use the high-density point cloud to initialize a Gaussian parameter set to generate an initial three-dimensional Gaussian model;

[0062] S2: Use a progressive camera sampling strategy to dynamically adjust the viewing angle parameter range, render and generate a 2D image, use a pre-trained 2D diffusion model to calculate the fractional distillation sampling loss, and optimize the 3D Gaussian parameters in combination with opacity constraints;

[0063] S3: Adjust the Gaussian distribution density through the adaptive density control mechanism and iteratively update the model geometric features;

[0064] S4: Convert the optimized 3D Gaussian model into a 3D point cloud and output it.

[0065] In the specific implementation process, this embodiment first performs density enhancement on incomplete three-dimensional point cloud data through point densification processing to generate a high-density point cloud, and then initializes the Gaussian parameter set based on the spatial distribution of the high-density point cloud to construct a three-dimensional Gaussian model covering the geometric features of the object surface; then adopts a progressive camera sampling strategy to expand the azimuth and elevation angle ranges of the camera viewing angle in stages, generates a multi-view two-dimensional image through differentiable rendering, and then uses a pre-trained two-dimensional diffusion model to perform fractional distillation sampling on the rendered image to calculate the geometric optimization gradient, and combines the opacity constraints of the Gaussian points to jointly optimize the three-dimensional Gaussian parameters; during the optimization process, an adaptive density control mechanism is used to analyze the gradient amplitude and neighborhood distribution characteristics of the Gaussian points, and the splitting, cloning or merging operations of the Gaussian points are dynamically adjusted to optimize the model density distribution, thereby iteratively improving the ability to express geometric details; finally, the distribution center point of the optimized three-dimensional Gaussian model is extracted as the completed point cloud output, and finally a complete and high-fidelity three-dimensional point cloud model is output.

[0066] Example 2

[0067] This embodiment provides a 3D point cloud completion method based on 3D Gaussian stippling and fractional distillation sampling, including:

[0068] S1: Acquire incomplete three-dimensional point cloud data of the object to be completed, perform point densification processing on the point cloud data to generate a high-density point cloud, and use the high-density point cloud to initialize a Gaussian parameter set to generate an initial three-dimensional Gaussian model;

[0069] The obtaining of incomplete three-dimensional point cloud data of the object to be completed includes:

[0070] S111: Scan the surface of the target object with a 3D laser radar to obtain raw point cloud data, and record the direction angle, elevation angle, and distance of the 3D laser radar relative to the target object as posture information;

[0071] S112: Normalize the original point cloud data to eliminate scale differences;

[0072] S113: Use statistical outlier filtering with a radius of 0.03 to remove isolated noise points and retain effective surface features.

[0073] The point densification process includes:

[0074] S121: Calculate the minimum bounding box for the incomplete 3D point cloud data;

[0075] S122: randomly generate multiple new points within the minimum bounding box, calculate the nearest neighbor distance between each candidate point and the original point cloud through KD-Tree acceleration, remove abnormal points with a distance greater than 0.01 to form a first point set, and fuse the preprocessed point cloud with the first point set to form a second point set;

[0076] S123: performing neighborhood density analysis on each point in the second point set based on the octree spatial index. When it is detected that the distance between a point and its nearest neighbor is greater than 0.02, generating 10 new points in a spherical area around the point with the nearest neighbor distance as the radius to form a third point set.

[0077] S124: Establish an octree neighborhood query structure, and when the nearest neighbor distance in the second point set is less than 0.002, copy the point to form a fourth point set;

[0078] S125: Merge the second point set, the third point set, and the fourth point set to form a high-density point cloud.

[0079] The method of initializing a Gaussian parameter set using a high-density point cloud to form an initial three-dimensional Gaussian model includes:

[0080] S131: Use the high-density point cloud coordinates as the Gaussian center position parameter μ, construct the covariance matrix Σ with the nearest neighbor distance of each point, use the randomly generated color parameter c, and set the initial opacity α to 0.5 to form a three-dimensional Gaussian model θ(μ, Σ, c, α).

[0081] S2: Use a progressive camera sampling strategy to dynamically adjust the sampling range of the viewing angle parameters, render and generate a 2D image, use a pre-trained 2D diffusion model to calculate the fractional distillation sampling loss, and optimize the 3D Gaussian parameters in combination with opacity constraints;

[0082] The method of dynamically adjusting the sampling range of the viewing angle parameter by using a progressive camera sampling strategy and rendering to generate a two-dimensional image includes:

[0083] S211: Use the posture information obtained in S101 as the initial direction angle a s , elevation angle e s , camera distance d s , and preset the camera field of view f s is 70;

[0084] S212: According to the current iteration step s and the preset starting iteration step s s and the end step number s e , calculate the current iteration step proportional coefficient

[0085] S213: According to the preset threshold k a , calculate the direction angle sampling range R a :

[0086]

[0087] And from R a The direction angle a is randomly sampled and then normalized to the range of [-180,180];

[0088] S214: According to the preset threshold k e , calculate the elevation sampling range R e :

[0089]

[0090] And from R e The elevation angle e is randomly sampled and then normalized to the range of [-90,90];

[0091] S215: Final sampling range [d e1 ,d e2 ], calculate the camera distance sampling range R d :

[0092] R d =[(1-r)·d s +r·d e1 ,(1-r)·d s +r·d e2 ]

[0093] And from R d Randomly sample the camera distance d;

[0094] S216: Final sampling range [f e1 ,f e2 ], calculate the camera distance sampling range R f :

[0095] R f =[(1-r)·f s +r·f e1 ,(1-r)·f s +r·f e2 ]

[0096] And from R f Randomly sample the camera field of view f;

[0097] S217: Generate a two-dimensional image I by rendering based on the camera perspective parameters (a, e, d, f) generated by sampling and using differentiable rasterization technology.

[0098] The method uses a pre-trained 2D diffusion model to calculate the fractional distillation sampling loss and optimizes the 3D Gaussian parameters in combination with opacity constraints, including:

[0099] S221: Input the rendered image I generated in step S217 and the corresponding category text prompt τ into the pre-trained two-dimensional diffusion model, and perform forward reasoning to predict the noise ∈ θ (I,τ);

[0100] S222: Calculate the gradient signal of each rendered image according to the definition of fractional distillation sampling:

[0101]

[0102] Where ∈ is the Gaussian noise added to the rendered image;

[0103] S223: Calculate the opacity parameter mean of the three-dimensional Gaussian points in the three-dimensional Gaussian model as the opacity loss:

[0104]

[0105] To ensure that the Gaussian points always maintain moderate visibility during the rendering process;

[0106] S224: According to total loss

[0107] L=L SDS +L opacity

[0108] The Adam optimizer with adaptive learning rate is used to update the three-dimensional Gaussian model θ(μ,Σ,c,α).

[0109] S3: Adjust the Gaussian distribution density through the adaptive density control mechanism and iteratively update the model geometric features;

[0110] The adjusting of the Gaussian distribution density by the adaptive density control mechanism includes:

[0111] S311: every preset number of iteration steps T, counting the gradient g of the three-dimensional Gaussian point in the current three-dimensional Gaussian model and the distance s to its nearest neighbor point;

[0112] S312: Based on the preset ladder threshold k g , nearest neighbor distance threshold k s and the nearest neighbor threshold interval [k c1 ,k c2 ], to control the three-dimensional Gaussian density, including:

[0113] If g>k g And s>k s , then in this three-dimensional Gaussian radius ss Randomly generate n s coordinates, and copy the Σ, c, α parameters of the original 3D Gaussian to complete the splitting operation;

[0114] If g>k g And s∈[k c1 ,k c2 ], then copy n c Three-dimensional Gaussian coordinates to complete the cloning operation;

[0115] S4: Convert the optimized 3D Gaussian model into a 3D point cloud and output it.

[0116] Converting the optimized three-dimensional Gaussian model into a three-dimensional point cloud comprises:

[0117] The center coordinate parameters of the three-dimensional Gaussian set are used as the coordinate information of the three-dimensional point cloud, and the color parameters of the three-dimensional Gaussian set are converted into the color information of the three-dimensional point cloud to generate a three-dimensional point cloud.

[0118] Experiments show that this embodiment can correctly complete the three-dimensional point cloud on the ShapeNet-ViPC data subset, such as Figure 2 and Figure 3 shown.

[0119] Experiments show that, in this embodiment, on the RedWood-3DScan data subset, the average L1-CD index is 20.5, as shown in the following table.

[0120] Object ShapeFormer PoinTr cGAN ShapeInversion SDS-complete ACL-SPC Ours Upholstered Chair 23.2 34.1 33.2 36.7 19.3 26.7 33.7 Outdoor Chairs 25.9 29.6 42.8 28.7 22.6 35.7 18.7 One-legged table 39.7 21.6 99.4 24.9 20.3 49.8 11.4 office chair 33.6 43.9 207.0 20.6 23.7 39.5 18.2 mean 30.6 32.3 95.8 27.7 21.5 37.9 20.5

[0121] The terms used in the drawings to describe positional relationships are for illustrative purposes only and should not be construed as limiting this patent;

[0122] Obviously, the above embodiments of the present invention are merely examples for the purpose of clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the claims of the present invention.

Claims

1. A 3D point cloud completion method based on fractional distillation sampling and 3D Gaussian dot coloring, characterized by: The following steps are involved: S1: Input the incomplete 3D point cloud data to be completed, enhance the initial point cloud density through point densification operation, and generate a high-density point cloud containing surface features; S2: constructing a three-dimensional Gaussian model based on the high-density point cloud, wherein each three-dimensional Gaussian is defined by position, covariance, color, and opacity parameters; S3: Using fractional distillation sampling technology combined with a pre-trained 2D diffusion model, rendering multi-view 2D images through differentiable Gaussian rasterization, and calculating the fractional distillation loss to optimize the parameters of the 3D Gaussian model; S4: During the optimization process, a progressive camera sampling strategy is used to dynamically adjust the camera viewpoint’s direction angle, elevation angle, distance, and field of view to ensure spatial consistency between the rendered image and the 3D model. S5: Iteratively execute steps S3 to S4 to convert the optimized 3D Gaussian model into complete 3D point cloud data.

2. The method according to claim 1, characterized in that The point densification operation includes: Randomly sample the minimum bounding box of the initial incomplete point cloud to generate a noise point set, filter out the points in the noise point set that deviate from the original point cloud, and combine the noise point set with the initial point cloud to generate a densified point set; The densified point set is split or cloned based on the neighborhood distance threshold to generate a high-density point cloud.

3. The method according to claim 2, characterized in that The splitting operation is: when the distance between a point in the point set and its nearest neighbor exceeds a threshold, multiple new points are randomly generated in its neighborhood; the cloning operation is: when the distance is lower than the threshold, the point is copied to increase the local density.

4. The method according to claim 1, wherein The progressive camera sampling strategy includes: The initial sampling range is set according to the camera pose when the initial incomplete point cloud is obtained, and gradually expanded to the final sampling range as the number of iteration steps increases; Dynamically adjust the sampling interval of direction angle, elevation angle, distance and field of view parameters through linear interpolation; In the later stage of optimization, we switch to global random sampling to expand the sampling range and refine the model details.

5. The method according to claim 1, wherein The optimization process of the three-dimensional Gaussian model also includes adaptive density control: The Gaussian points are periodically split or cloned according to the gradient statistics, and the cloning is stopped after the number of iterations exceeds a threshold to limit the model complexity.

6. The method according to claim 1, characterized in that The calculation of the fractional distillation loss includes: The rendered image is fed into the pre-trained diffusion model, noise is predicted and compared with Gaussian noise to generate the loss; Incorporating an opacity constraint loss ensures that the Gaussian model maintains visibility during optimization.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 6 are implemented.

Images