A controllable point cloud completion method and system based on large model prior

Abstract

The present invention discloses a controllable point cloud completion method and system based on large model priors, including orthophoto depth projection of residual point cloud; redrawing of point cloud depth projection based on image generation model; three-dimensional representation guidance based on multi-view diffusion model and feedforward Gaussian sputtering generation model; feature-level fusion control and result sampling based on three-plane expression. The present invention introduces text and image control on the basis of the point cloud completion framework, and realizes the speculation of unknown parts through text-controlled depth image redrawing and multi-view generation; and realizes point cloud completion based on the reasoning result of the large model by unifying the results of the feedforward three-dimensional Gaussian sputtering generation model and the residual point cloud into three-plane expression for feature-level control and fusion. The present invention realizes a point cloud completion framework with artificially adjustable results, strong robustness and versatility, which can be applied to application scenarios where there is a need for point cloud completion but a lack of true values for targeted training.

Description

Technical Field

[0001] The present invention belongs to the intersection of computer vision and laser scanning data processing, and in particular relates to point cloud data processing, point cloud quality enhancement, reconstruction content, and automated technology for laser point cloud measurement data enhancement. Background Art

[0002] Laser point cloud is a type of data obtained by laser radar that expresses the surface position information of an object and is composed of a series of three-dimensional coordinates. As an easily accessible 3D discrete representation, laser point cloud can be used to model urban objects and provide an accurate depiction of the physical world. It is one of the main data sources for building a high-precision digital base. Point clouds have also received widespread attention in many fields of computer vision, such as 3D reconstruction, scene understanding, and autonomous driving. However, due to the inherent limitations of the scanning principle, the original point cloud obtained by laser radar inevitably has defects caused by occlusion, reflection, and other reasons. These defects will seriously affect the accuracy of downstream tasks such as segmentation and reconstruction. Therefore, repairing the missing parts is a key step in enhancing geometric quality.

[0003] Traditional point cloud completion methods, based on artificial priors, can only repair missing surface shapes. Deep learning-based point cloud completion methods are limited by training data and over-engineered for uniformly distributed synthetic data, making them incapable of handling situations outside the dataset distribution, uneven point cloud distribution, and the presence of noise. In practice, due to a lack of real-world data for training, deep learning-based methods struggle to achieve the same results on synthetic data. These issues limit the practical application of various current point cloud completion methods.

[0004] Driven by the need to generate and reconstruct a wide range of 3D assets, 3D generative models based on image generation models or large 3D datasets have emerged in recent years, providing important prior knowledge for point clouds. However, optimization algorithms based on image generation models are computationally expensive, making them unacceptable for point cloud completion tasks that require batch processing of objects. While the use of priors from image generation models in point cloud completion remains limited to the results level, the use of priors from feedforward 3D models remains a promising area. Feedforward 3D generative models outperform optimization-based methods in terms of generation speed and are increasingly capable of generating target-level objects. The development and utilization of their prior knowledge is of great significance for point cloud completion tasks that require inferring unknown parts. Summary of the Invention

[0005] In order to overcome the shortcomings of the above-mentioned existing technologies, the present invention provides a controllable point cloud completion method and system based on large model priors. By introducing text and image control on the basis of the point cloud completion framework, the inference of unknown parts is realized through text-controlled depth image redrawing and multi-perspective generation; by unifying the results of the feedforward three-dimensional Gaussian sputtering generation model and the residual point cloud into a three-plane expression for feature-level control and fusion, point cloud completion based on the large model reasoning results is realized.

[0006] According to one aspect of the present invention, a controllable point cloud completion method based on a large model prior is provided, comprising:

[0007] Perform orthographic depth projection based on the acquired target-level residual cloud to obtain an orthographic projection depth map;

[0008] Performing large model reasoning based on the orthographic projection depth map to obtain terminal features;

[0009] Inputting the terminal features and the target-level residual point cloud into a trained point cloud completion model, and outputting a completed complete point cloud; wherein the training of the point cloud completion model includes:

[0010] Construct a target-level residual cloud dataset;

[0011] Performing orthophoto depth projection and large model inference on the residual defect clouds in the target cascade residual defect cloud dataset in sequence to obtain terminal features corresponding to the residual defect clouds;

[0012] Inputting the terminal features and the residual defect cloud into a point cloud completion model, the point cloud completion model performs three-plane expression encoding on the terminal features and the residual defect cloud to obtain a three-plane expression of a complete point cloud, and performing complete point cloud sampling based on the three-plane expression of the complete point cloud to obtain a complete point cloud;

[0013] A loss function is constructed to supervise the training process of the point cloud completion model, and a trained point cloud completion model is output.

[0014] As a further technical solution, large model reasoning is performed based on the orthographic projection depth map, including:

[0015] generating a false color image based on the orthographic depth map using a text-conditioned image editing model;

[0016] Based on the false color image, a surround view image is generated using a multi-view diffusion model;

[0017] Based on the surround view image, a feed-forward 3D generative model is used to generate a 3D Gaussian result.

[0018] As a further technical solution, after generating the 3D Gaussian result, it also includes:

[0019] The terminal features output by the feedforward 3D generation model are used as the prior of the large model, and the 3D Gaussian point coordinates output by the feedforward 3D generation model are filtered according to occupancy as the point cloud representation of the inference result of the large model.

[0020] As a further technical solution, after inputting the terminal features and the residual defect cloud into the point cloud completion model, the method further includes:

[0021] The terminal features are transformed into a multi-channel image, input into a neural network based on a Transformer architecture for feature extraction, and a three-plane expression of the terminal features is obtained;

[0022] The residual defect cloud is input into a neural network based on PointNet and residual connection structure to extract point-by-point features and project them onto three coordinate planes. The coordinate planes projected with point-by-point features are then subjected to feature extraction using a neural network based on U-Net to obtain a three-plane representation of the residual defect cloud.

[0023] The three-planar expression of the terminal feature and the three-planar expression of the residual point cloud are input into the feature fusion network based on cross attention to obtain the three-planar expression of the fused complete point cloud.

[0024] As a further technical solution, after obtaining the three-plane expression of the fused complete point cloud, it also includes:

[0025] A complete point cloud is sampled from the three-planar representation using a neural network based on a multi-layer perceptron.

[0026] As a further technical solution, a loss function is constructed, including:

[0027] A one-way chamfer distance loss is performed between the residual point cloud and the output complete point cloud, a chamfer distance loss is performed between the Gaussian point cloud generated by the feedforward 3D generative model and the output complete point cloud, and the density penalty loss of the output point cloud itself is calculated. The three losses are weighted and summed to construct a loss function.

[0028] According to one aspect of the present invention, a controllable point cloud completion system based on a large model prior is provided, which is used to implement the controllable point cloud completion method based on a large model prior.

[0029] As a further technical solution, the system includes:

[0030] The first main module is used to perform orthographic depth projection based on the acquired target-level residual cloud to obtain an orthographic projection depth map;

[0031] The second main module is used to perform large model reasoning based on the orthographic projection depth map to obtain terminal features;

[0032] The third main module is configured to input the terminal features and the target-level residual point cloud into the trained point cloud completion model and output the completed complete point cloud. The training of the point cloud completion model includes:

[0033] Construct a target-level residual cloud dataset;

[0034] Performing orthophoto depth projection and large model inference on the residual defect clouds in the target cascade residual defect cloud dataset in sequence to obtain terminal features corresponding to the residual defect clouds;

[0035] Inputting the terminal features and the residual defect cloud into a point cloud completion model, the point cloud completion model performs three-plane expression encoding on the terminal features and the residual defect cloud to obtain a three-plane expression of a complete point cloud, and performing complete point cloud sampling based on the three-plane expression of the complete point cloud to obtain a complete point cloud;

[0036] A loss function is constructed to supervise the training process of the point cloud completion model, and a trained point cloud completion model is output.

[0037] According to one aspect of the present invention, a controllable point cloud completion system based on large model prior is provided, comprising a processor and a memory, wherein the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute the controllable point cloud completion method based on large model prior.

[0038] According to one aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed, the controllable point cloud completion method based on large model prior is implemented.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] 1. This paper introduces text and image control on the basis of the point cloud completion framework, and realizes the inference of unknown parts through text-controlled depth image redrawing and multi-view generation. By unifying the results of the feedforward 3D Gaussian sputtering generation model and the residual point cloud into a three-plane expression for feature-level control and fusion, point cloud completion based on the inference results of the large model is realized.

[0041] 2. This method can combine large model priors and the original incomplete point cloud geometry to complete the incomplete point cloud. It is highly robust to target object type, point cloud density, and uneven point cloud distribution. It can also achieve good results in situations where there is a lack of true value or type characteristics. It can be applied to raw point cloud data enhancement and shape completion of various 3D point cloud assets. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, a brief introduction will be given below to the drawings used in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0043] Figure 1 This is the overall technical process provided by the embodiment of the present invention;

[0044] Figure 2 This is a schematic diagram of a residual defect cloud provided by an embodiment of the present invention;

[0045] Figure 3 (a) is a schematic diagram of a dense orthophoto depth acquisition method provided by an embodiment of the present invention. Figure 3 (b) is a schematic diagram of a large model inference pipeline provided by an embodiment of the present invention;

[0046] Figure 4 This is a point cloud completion result diagram provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0047] Aiming at the target-level residual point cloud, the present invention proposes a controllable point cloud completion method based on large model prior, including projecting the residual point cloud into a dense orthophoto depth image; generating a false color image, a surround view image and a 3D Gaussian expression based on the depth image; encoding the residual point cloud and the 3D Gaussian expression inferred from the large model into a three-plane space and fusing them; and sampling a complete point cloud from the fused three-plane expression.

[0048] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. In addition, the technical features in the various embodiments or single embodiments provided by the present invention are arbitrarily combined with each other to form a new technical solution. This combination is not restricted by the sequence of steps and / or structural composition mode, but must be based on the ability of ordinary technicians in this field to implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that this combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0049] See also Figure 1, the controllable point cloud completion method based on large model prior proposed in an embodiment of the present invention first projects the residual point cloud into a dense orthophoto depth image, then uses the image generation large model conditioned on the depth image and text to generate a false color image from the dense orthophoto depth image, uses a multi-view diffusion model to generate four surround images from a single false color image, and uses a feedforward three-dimensional generation large model to generate a three-dimensional Gaussian expression of the complete shape obtained by the large model prior reasoning from the four surround images. The present invention uses the terminal features of the large model reasoning process as a priori guidance, encodes the terminal features and the residual point cloud into a three-plane expression space, and then fuses them on the three-plane expression. Finally, the present invention samples a complete point cloud from the fused three-plane expression containing the complete shape three-dimensional expression. The specific implementation process includes the following steps:

[0050] Step 1: Obtain the depth image of the target-level residual cloud data through orthographic depth projection.

[0051] The embodiment of the present invention further proposes that the implementation of step 1 is as follows:

[0052] First, the object-level residual point cloud is rotated to axis-aligned and normalized to the unit sphere, with its geometric center at the origin of the 3D coordinate system. An orthographic depth projection is then performed from the front side to obtain a point cloud depth image. A fill-in-fast algorithm based on dilated convolution is used to fill the sparse point cloud depth image into a dense point cloud depth image.

[0053] Step 1 in the embodiment is specifically implemented as follows:

[0054] 1.1) Example, rotate to axis alignment and normalize:

[0055] Assume that the residual defect cloud is , Represents the coordinates of each point in the residual point cloud, N represents the number of points, Indicates a point First, construct a three-dimensional rectangular coordinate system so that the positive direction of the z axis is vertically upward. The coordinates of its geometric center point are calculated as follows: :

[0056]

[0057] As shown in formula (2), the residual cloud is moved until its geometric center is at the origin:

[0058]

[0059] As shown in formula (3), the distance from the point farthest from the origin to the origin in the residual defect cloud is calculated, and the residual defect cloud is normalized to the unit sphere space based on this distance:

[0060]

[0061] The residual cloud is artificially rotated so that it is in the axis-aligned direction and the projection obtained when observed from the y-axis direction is relatively complete.

[0062] 1.2) Example, orthographic depth projection:

[0063] Orthographic depth projection is performed by aligning the point cloud coordinates along the y-axis. The mapping relationship between the point cloud coordinates and the depth image pixel coordinates and grayscale is shown in Equations (4), (5), and (6). Here, H and W are the height and width of the depth image. The depth image consists of three RGB channels, and the color values are assigned to 8-bit integers. In this embodiment, all channels are assigned the same value to achieve a grayscale image.

[0064]

[0065] Among them, the subscript The subscript "part" represents the pixel coordinates projected onto the image. The subscript "part" represents the xyz coordinates of the point cloud, and "R", "G", and "B" represent the three color channels of the image. During projection, the depth is used to determine whether the current point is occluded by other points.

[0066] 1.3) In the embodiment, the orthographic depth projection gap is filled:

[0067] After projecting the image into a depth image, the gaps in the depth image are filled using a gap-filling algorithm. This gap-filling algorithm is a fill-in-fast algorithm based on dilated convolution.

[0068] The process and intermediate results of step 1 are as follows Figure 3 As shown in (a) in .

[0069] In step 2, the depth image is rendered as a false color image using a large text-conditioned image editing model (e.g., ControlNet). The dense orthophoto depth projection obtained in step 1 and the text hint are used as input to the large depth map-conditioned image editing model to obtain a false color image, which is then de-backgrounded.

[0070] Step 2 in the embodiment is specifically implemented as follows:

[0071] The orthophoto depth image obtained in step 1 is used as the conditional input for the image editing model. In this embodiment, the image editing model is a ControlNet based on the depth map condition. The control text is a manual conditional input and should include a detailed description of the target point cloud, including its type, name, and features.

[0072] In step 3, based on the false color image, a multi-view diffusion model (MVDream) is used to generate four surround view images from a single false color image.

[0073] Step 3 in the embodiment is specifically implemented as follows:

[0074] The false color image obtained in step 3 is used as input to a large multi-view image generation model. In this embodiment, the large multi-view image generation model is MVDream. The control text is a manual condition input and should include a detailed description of the target point cloud, including type, name, features, etc.

[0075] In step 4, a feedforward 3D generative model (LGM) is used to generate corresponding 3D Gaussian results from the four surround-view images. The four surround-view images obtained in step 3 are used as input to the feedforward 3D generative model (LGM), which outputs the 3D generated results (Gaussian sputtering). The positional attributes of the Gaussian sputtering are used as 3D coordinates and filtered according to occupancy to obtain a Gaussian point cloud for subsequent supervision. The final layer output (terminal features) of the main neural network module (U-Net) in the 3D generative model is used as the large model inference prior for subsequent unified 3D representation and feature-level fusion control.

[0076] Step 4 in the embodiment is specifically implemented as follows:

[0077] The four surround view images obtained in step 4 are used as inputs to a feed-forward 3D Gaussian generative model. In the embodiment, the feed-forward 3D Gaussian generative model is LGM. After passing through U-Net, the terminal features are obtained After the terminal features pass through the projection layer and the corresponding Gaussian attribute activation function, they are transformed into 3D Gaussian expressions. By setting a threshold on the Gaussian occupancy attribute opacity Filter and retain its position attributes as point cloud coordinates to obtain Gaussian point cloud .

[0078] Steps 2, 3, and 4 are the process of the large model inference pipeline provided by the embodiment of the present invention, that is, the process of the second main module, the process diagram and the intermediate result example are as follows: Figure 3 As shown in (b) in .

[0079] In summary, in steps 1 to 4 of this embodiment of the present invention, the object-level residual point cloud is first rotated to an axis-aligned orientation and normalized to the unit sphere space, with its geometric center at the origin of the 3D coordinate system. An orthographic depth projection is then performed on the relatively complete observation surface to obtain a point cloud depth image. A fill-in-fast algorithm is used to fill the sparse point cloud depth image into a dense point cloud depth image. The dense orthographic depth projection and textual hints are used as input to a depth-map-conditioned image editing model (ControlNet) to obtain a false-color image. This image is then subjected to background removal. The background-free false-color image is used as input to a multi-view generative model (MVDream) to obtain four multi-view consistent surround images. The four surround images are then used as input to a feed-forward 3D generative model (LGM) to obtain a 3D generation result (Gaussian sputtering). The positional attributes of the Gaussian sputtering are used as 3D coordinates and filtered according to occupancy to obtain a Gaussian point cloud for subsequent supervision. The last layer output (end features) of the main neural network module (U-Net) in the 3D generation model is used as the large model inference prior for subsequent unified 3D representation and feature-level fusion control.

[0080] In step 5, the large model prior is encoded into a three-plane representation using a Transformer-based three-plane encoder. The terminal features obtained in step 4 are transformed into a multi-channel image format, which is input into a Transformer-based neural network for feature extraction. Then, the image is transformed into a three-plane feature format through an upsampling layer.

[0081] Step 5 in the embodiment is specifically implemented as follows:

[0082] The specific implementation of the terminal feature three-plane encoder can refer to the image three-plane encoder in LRM. This encoder is based on the Transformer architecture and converts the initial three-plane embedding into a three-plane representation containing image geometric information based on the input image. After deformation and a layer of convolution, the features of the input Transformer are obtained The output features of the Transformer are expressed in Gaussian three planes after deformation and deconvolution layers. .

[0083] In step 6, the residual defect cloud is encoded into a three-plane representation using a point cloud three-plane encoder based on PointNet and projection relationships. First, the residual defect cloud is input into a neural network based on PointNet and a residual connection structure to extract point-by-point features. The residual defect cloud is then projected onto the three coordinate planes according to coordinates, and point-by-point features are assigned to the corresponding pixels on the coordinate planes. The features of the three planes are combined to obtain the projection of the point cloud in the three-plane space. Finally, a U-Net-based neural network is used to extract features from this three-plane representation to obtain the final three-plane representation of the point cloud.

[0084] Step 6 in the embodiment is specifically implemented as follows:

[0085] The architecture of the point cloud three-plane encoder can be used as a reference for the point cloud three-plane encoder in GenSDF. The encoder extracts point-by-point features of the input point cloud and projects them onto three coordinate planes. It then extracts features from the projected coordinate planes to obtain a three-plane representation containing the point cloud geometry information. Encoded as point cloud three-plane expression .

[0086] In step 7, a cross-attention-based neural network is used to fuse the prior three-planar representation of the large model with the three-planar representation of the residual point cloud to obtain a three-planar representation of the final complete shape. The three-planar representation of the terminal features obtained in step 5 and the three-planar representation of the point cloud obtained in step 6 are combined to the same size through a projection layer and then input into the cross-attention-based feature fusion network to obtain the fused three-planar representation.

[0087] Step 7 in the embodiment is specifically implemented as follows:

[0088] First, the three-plane representation of the terminal feature and the three-plane representation of the point cloud are transformed into , and then these two features are fused through stacked cross-attention and self-attention layers. The fusion method of the cross-attention layer is shown in Equation (7), the calculation process of the self-attention layer is shown in Equation (8), and the overall fusion process is shown in Equations (9) and (10). Among them, MHA is the abbreviation of MultiHeadAttention, and the three inputs of this network layer are query, key, and value, respectively.

[0089]

[0090] The fused features After deformation and deconvolution layers, it is transformed into a fused three-plane representation .

[0091] In step 8, a complete point cloud is sampled from the three-plane representation using a multi-layer perceptron-based neural network. First, an initial point cloud is randomly initialized. The three-plane representation obtained in step 7 is used to query the corresponding three-plane features for each point in this initial point cloud. The retrieved features are then fed into the multi-layer perceptron-based neural network, where a coordinate offset is calculated. This offset is then added to the coordinates of the query point to obtain the coordinates of the final point in the complete point cloud. By repeatedly querying all points in the initial point cloud, a complete point cloud is obtained.

[0092] Step 8 in the embodiment is specifically implemented as follows:

[0093] First, randomly initialize the point cloud ,right For each point in , use its three-dimensional coordinates in Query the corresponding features on the three corresponding coordinate planes to obtain the features of the three corresponding pixels. Combine the features obtained by the query to obtain the point-by-point features of N points ,Will Input the point cloud three-plane feature decoder based on the multi-layer perceptron to decode the corresponding point-by-point offset The point-by-point offset is added to the initial random point cloud to obtain the point-by-point coordinates of the complete point cloud, i.e., Equation (11).

[0094]

[0095] Step 9, the training phase. First, set the number of points used to sample the complete point cloud in step 8 to the same number of points as the input residual point cloud. Calculate the mean of the point-by-point chamfer distances from the residual point cloud to the complete point cloud as loss 1. Then, downsample the Gaussian point cloud obtained in step 4 to the same number of points as the complete point cloud, and calculate the mean of the point-by-point chamfer distances from the Gaussian point cloud to the complete point cloud, and from the complete point cloud to the Gaussian point cloud as loss 2. Furthermore, calculate the density penalty loss for the complete point cloud as loss 3. The total loss is the weighted sum of losses 1, 2, and 3, and is used to supervise the training of the neural network involved in steps 5 through 8.

[0096] Step 9 in the embodiment is specifically implemented as follows:

[0097] One-way chamfer distance loss The calculation method of is shown in formula (12), the chamfer distance loss The calculation method is shown in formula (13). Where X and Y are two point clouds, and N(X) represents the number of points in point cloud X. Density penalty loss The calculation method of is shown in formula (14), where STD is the standard deviation function. When calculating the density penalty loss, first calculate the point-to-point distance , select the 16 largest values and calculate their standard deviation as the loss. The overall loss of the present invention The calculation method is shown in formula (15).

[0098]

[0099] In an embodiment, the controllable point cloud completion framework based on large model priors constructed by the present invention is trained for 160 rounds on eight known categories of the ShapeNet-ViPC dataset without relying on ground truth.

[0100] The point cloud completion results obtained through the above steps can be applied to the original point cloud data enhancement and the shape completion of various 3D point cloud assets. Figure 2 and Figure 4 shown.

[0101] In summary, in the implementation of steps 5 to 9 of the present embodiment, the terminal features obtained in step 4 are first transformed into a multi-channel image and input into a neural network based on the Transformer architecture for feature extraction. They are then transformed into a three-plane feature format through an upsampling layer. The residual defect cloud is input into a neural network based on PointNet and a residual connection structure to extract point-by-point features. The residual defect cloud is then projected onto three coordinate planes according to coordinates, and point-by-point features are assigned to the corresponding pixels on the coordinate planes. The features of the three planes are combined to obtain the projection of the point cloud in the three-plane space. Finally, a U-Net-based neural network is used to extract features from this three-plane representation to obtain the final three-plane representation of the point cloud. The three-plane representation of the terminal features and the three-plane representation of the point cloud are integrated to the same size through a projection layer. They are then input into a feature fusion network based on cross-attention to obtain the fused three-plane representation. A random initial point cloud is initialized. The three-plane representation obtained in step 7 is used to query the corresponding three-plane features for each point in the initial point cloud. The queried features are input into a neural network based on a multi-layer perceptron. The coordinate offset is calculated and added to the coordinates of the query point to obtain the coordinates of the point in the final complete point cloud. After looping through all points in the initial point cloud, the complete point cloud is obtained. The mean of the point-by-point chamfer distances from the residual point cloud to the complete point cloud is calculated as loss 1. Next, the Gaussian point cloud obtained in step 4 is downsampled to the same number of points as the complete point cloud. The mean of the point-by-point chamfer distances from the Gaussian point cloud to the complete point cloud and vice versa is calculated as loss 2. Furthermore, a density-penalized loss is calculated for the complete point cloud as loss 3. The total loss is the weighted sum of losses 1, 2, and 3, and is used to supervise the training of the neural network involved in steps 5 through 8.

[0102] In specific implementation, the method proposed in the technical solution of the present invention can be automatically run by those skilled in the art using computer software technology. System devices that implement the method, such as computer-readable storage media that store the corresponding computer program of the technical solution of the present invention and computer equipment that runs the corresponding computer program, should also be within the scope of protection of the present invention.

[0103] In some possible embodiments, a controllable point cloud completion system based on a large model prior is provided, comprising the following modules:

[0104] The first main module, the orthographic depth projection module, is used to perform orthographic depth projection based on the acquired target-level residual defect cloud to obtain an orthographic projection depth map; this module includes point cloud axis alignment and normalization, orthographic depth projection and gap filling, and is used to project the residual defect cloud into a dense orthographic depth image.

[0105] The second main module, the large model inference pipeline module, is used to perform large model inference based on the orthographic projection depth map to obtain terminal features; this module includes an image editing large model, a multi-view generation large model and a three-dimensional generation large model, which are used to obtain a reasonable complete three-dimensional shape from the large model prior knowledge.

[0106] The third main module, the three-plane encoding network, is used to input the terminal features and the target-level residual defect cloud into the trained point cloud completion model, and output the completed complete point cloud; this module includes a point cloud three-plane encoder, a Gaussian terminal feature three-plane encoder, a three-plane feature fusion network and a point cloud sampling network, which are used to unify the inference results and the residual defect cloud of the large model into the same three-dimensional expression, realize feature-level fusion control, and obtain a point cloud expression from the fusion result that conforms to both the original shape and the complete shape inferred by the large model.

[0107] In some possible embodiments, a controllable point cloud completion system based on large model prior is provided, including a processor and a memory, the memory being used to store program instructions, and the processor being used to call the stored instructions in the memory to execute a controllable point cloud completion method based on large model prior as described above.

[0108] In some possible embodiments, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed, the controllable point cloud completion method based on large model prior is implemented.

[0109] In summary, the present invention provides a controllable point cloud completion method based on generating a large model, including orthophoto depth projection of residual point cloud; redrawing of point cloud depth projection based on image generation model; three-dimensional representation guidance based on multi-view diffusion model and feedforward Gaussian sputtering generation model; feature-level fusion control and result sampling based on three-plane expression. The present invention introduces text and image control on the basis of the point cloud completion framework, and realizes the speculation of unknown parts through text-controlled depth image redrawing and multi-view generation; and realizes point cloud completion based on the reasoning result of the large model by unifying the result of the feedforward three-dimensional Gaussian sputtering generation model and the residual point cloud into a three-plane expression for feature-level control and fusion. In the model training stage, the present invention can not use the true value, but only rely on the residual point cloud and the reasoning result of the large model for training. By combining the implicit expression level with the large model's inference of the incomplete parts of the point cloud and the shape of the original incomplete point cloud, the present invention realizes a point cloud completion framework with adjustable results, strong robustness, and strong versatility. It can be applied to application scenarios where there is a need for point cloud completion but a lack of true values for targeted training, such as the 3D reconstruction of various infrastructure and assets.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.

Claims

1. A controllable point cloud completion method based on large model priors, characterized by: include: Perform orthographic depth projection based on the acquired target-level residual cloud to obtain an orthographic projection depth map; Performing large model reasoning based on the orthographic projection depth map to obtain terminal features; Performing large-scale model reasoning based on the orthographic projection depth map includes: generating a false color image based on the orthographic projection depth map using a text-conditioned image editing large-scale model; generating a surround view image based on the false color image using a multi-view diffusion model; and generating a 3D Gaussian result based on the surround view image using a feed-forward 3D generative model; Inputting the terminal features and the target-level residual point cloud into a trained point cloud completion model, and outputting a completed complete point cloud; wherein the training of the point cloud completion model includes: Construct a target-level residual cloud dataset; Performing orthophoto depth projection and large model inference on the residual defect clouds in the target cascade residual defect cloud dataset in sequence to obtain terminal features corresponding to the residual defect clouds; Inputting the terminal features and the residual defect cloud into a point cloud completion model, the point cloud completion model performs three-plane expression encoding on the terminal features and the residual defect cloud to obtain a three-plane expression of a complete point cloud, and performing complete point cloud sampling based on the three-plane expression of the complete point cloud to obtain a complete point cloud; A loss function is constructed to supervise the training process of the point cloud completion model, and a trained point cloud completion model is output.

2. The controllable point cloud completion method based on large model prior according to claim 1, characterized in that: After generating the 3D Gaussian result, it also includes: The terminal features output by the feedforward 3D generation model are used as the prior of the large model, and the 3D Gaussian point coordinates output by the feedforward 3D generation model are filtered according to occupancy as the point cloud representation of the inference result of the large model.

3. The controllable point cloud completion method based on large model prior according to claim 1, characterized in that: After inputting the terminal features and residual defect cloud into the point cloud completion model, the following steps are also included: The terminal features are transformed into a multi-channel image, input into a neural network based on a Transformer architecture for feature extraction, and a three-plane expression of the terminal features is obtained; The residual defect cloud is input into a neural network based on PointNet and residual connection structure to extract point-by-point features and project them onto three coordinate planes. The coordinate planes projected with point-by-point features are then subjected to feature extraction using a neural network based on U-Net to obtain a three-plane representation of the residual defect cloud. The three-planar expression of the terminal feature and the three-planar expression of the residual point cloud are input into the feature fusion network based on cross attention to obtain the three-planar expression of the fused complete point cloud.

4. The controllable point cloud completion method based on large model prior according to claim 3, characterized in that: After obtaining the three-plane expression of the complete fused point cloud, it also includes: A complete point cloud is sampled from the three-planar representation using a neural network based on a multi-layer perceptron.

5. The controllable point cloud completion method based on large model prior according to claim 2, characterized in that: Construct a loss function, including: A one-way chamfer distance loss is performed between the residual point cloud and the output complete point cloud, a chamfer distance loss is performed between the Gaussian point cloud generated by the feedforward 3D generative model and the output complete point cloud, and the density penalty loss of the output point cloud itself is calculated. The three losses are weighted and summed to construct a loss function.

6. A controllable point cloud completion system based on large model priors, characterized by: Used to implement a controllable point cloud completion method based on large model prior as described in any one of claims 1-5.

7. The controllable point cloud completion system based on large model prior according to claim 6, characterized in that: The system comprises: The first main module is used to perform orthographic depth projection based on the acquired target-level residual cloud to obtain an orthographic projection depth map; The second main module is used to perform large model reasoning based on the orthographic projection depth map to obtain terminal features; The third main module is configured to input the terminal features and the target-level residual point cloud into the trained point cloud completion model and output the completed complete point cloud. The training of the point cloud completion model includes: Construct a target-level residual cloud dataset; Performing orthophoto depth projection and large model inference on the residual defect clouds in the target cascade residual defect cloud dataset in sequence to obtain terminal features corresponding to the residual defect clouds; Inputting the terminal features and the residual defect cloud into a point cloud completion model, the point cloud completion model performs three-plane expression encoding on the terminal features and the residual defect cloud to obtain a three-plane expression of a complete point cloud, and performing complete point cloud sampling based on the three-plane expression of the complete point cloud to obtain a complete point cloud; A loss function is constructed to supervise the training process of the point cloud completion model, and a trained point cloud completion model is output.

8. A controllable point cloud completion system based on large model priors, characterized by: It includes a processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a controllable point cloud completion method based on large model prior as described in any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed, it implements the controllable point cloud completion method based on large model prior as described in any one of claims 1 to 5.

Images