A point cloud completion method and device based on shape prior and deep learning

Abstract

The present application relates to the field of computer vision, in particular to a point cloud completion method and device based on shape prior and deep learning. The method comprises the following steps: firstly, feature extraction is performed on the input incomplete point cloud data; then, the multi-scale features are refined through the prefix embedding tree to obtain refined features; next, the multi-scale features and the refined features are connected through the semi-connected U-Net to obtain the preliminary completed seed point cloud; subsequently, the preliminary completed seed point cloud is up-sampled to generate the point cloud reaching the target resolution; finally, the generated point cloud is constrained by using a composite loss function, and the composite loss function comprises a chamfer distance, a double orthogonal constraint loss and an elastic potential energy loss, so as to obtain the complete point cloud with balanced density. Through the method and device, the technical effect of balanced density, geometric accuracy and complete detail recovery in the point cloud completion process is realized, and the problems of local density imbalance and geometric information loss are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, and in particular to a point cloud completion method and device based on shape prior and deep learning. BACKGROUND

[0002] In three-dimensional data processing, point cloud is a commonly used data representation form. Since point cloud data can represent three-dimensional structures at any resolution, and has the advantages of high geometric accuracy and flexible storage, it has been widely used in industrial detection, autonomous driving, building surveying, cultural heritage protection and other fields. However, in the actual point cloud data acquisition process, due to problems such as occlusion of view angle, incomplete coverage, high reflectivity, etc., data loss often occurs; in addition, due to the resolution limitation of the acquisition device, the original point cloud data collected usually has sparsity and incompleteness. These defects have a serious impact on downstream tasks that rely on complete three-dimensional data (such as segmentation, target detection, three-dimensional reconstruction, etc.), making point cloud completion and repair a very important task in computer vision.

[0003] With the development of technology, existing point cloud completion methods can be roughly divided into two categories: traditional methods and learning-based completion methods.

[0004] Traditional point cloud completion methods mainly restore missing data through interpolation, shape matching or based on shape database. This kind of method has good effect when dealing with regular shapes or small range missing objects, but for point cloud data with large range missing or complex shape, the effect is obviously insufficient. The reason is that these methods lack understanding of the global shape of the object and cannot infer the complex geometric structure of the missing part.

[0005] With the rapid development of deep learning, learning-based point cloud completion methods have been widely used. Compared with traditional methods, deep learning methods not only can be trained through incomplete point clouds (i.e. sparse point cloud data as input), but also can extract more rich features using large-scale data sets. Therefore, deep learning has stronger completion ability when dealing with point clouds with more missing data.

[0006] Most of the current common learning-based point cloud completion schemes adopt a two-stage completion framework. First, the incomplete point cloud is input into the model to generate representative seed point clouds (i.e. sparse complete point clouds). These seed point clouds generate preliminary completion results through feature extraction. Then, an upsampling network is used to further refine the seed point clouds to generate complete point clouds with more balanced density. However, the existing technology still has the following technical problems in point cloud completion:

[0007] Lack of detailed geometric features: Since the features extracted during the encoding stage only represent partial information, the generated seed point cloud is difficult to accurately express the detailed geometric features of the missing parts, resulting in unsatisfactory final completion results.

[0008] Insufficient introduction of shape priors: Existing methods usually ignore how to effectively utilize the shape priors of objects. Shape priors can provide guidance for geometric understanding of different object categories, but current deep learning models lack effective mechanisms to fully utilize these prior information, which limits the completion ability of complex shape objects.

[0009] Uneven density of seed point cloud: In the traditional U-Net structure, the features of the incomplete point cloud are directly concatenated with the up-sampled features. This approach works well in 2D image processing because the feature distribution of the image is relatively uniform. However, in point cloud completion, the features extracted by the encoder only come from the incomplete point cloud, lacking global geometric information. The features generated by the decoder represent the features of the global point cloud. Directly connecting the two will cause the explicit features of the incomplete point cloud to be more significant than the abstract features of the global point cloud, resulting in an imbalance of features, and thus the generated seed point cloud has uneven density distribution.

[0010] Local point cloud density problem: Existing up-sampling methods often result in the generated points being too concentrated around the seed point cloud, causing local point cloud density distribution problems. Using traditional CD-l1 and CD-l2 indicators cannot well reflect the point cloud density distribution problem. Excessive local point cloud not only reduces the expression ability of the point cloud, but also causes memory redundancy.

[0011] To solve the above technical problems, the present application provides a point cloud completion method and device based on shape prior and deep learning. SUMMARY

[0012] The purpose of the present application is to overcome the problems of uneven point cloud density distribution and insufficient completion of missing geometric features in the prior art, and to provide a point cloud completion method and device based on shape prior and deep learning. This method not only effectively introduces shape prior when generating seed point cloud, improving the inference ability of complex geometric structure, but also solves the problems of uneven point cloud density distribution and local density by introducing semi-connected U-Net structure and elastic potential loss. This method can generate more complete and balanced density point cloud, and is suitable for large-scale missing and complex shape completion tasks.

[0013] To achieve the above invention purposes, the present application provides the following technical solutions:

[0014] A point cloud completion method based on shape prior and deep learning, comprising the following steps:

[0015] S1, feature extraction is performed on the input incomplete point cloud data to obtain multi-scale features of the incomplete point cloud;

[0016] S2, the features extracted in S1 are refined using a prefix embedding tree to obtain refined features;

[0017] S3, the multi-scale features in S1 and the refined features in S2 are processed by a semi-connected U-Net, and after multi-layer perceptron regression, a preliminary completed seed point cloud is obtained;

[0018] S4, the preliminary completed seed point cloud in S3 is upsampled to obtain a generated point cloud reaching a target resolution;

[0019] S5, the complete point cloud in step S4 is constrained by using a composite loss function to obtain a complete point cloud with balanced density.

[0020] The method gradually generates a complete point cloud with balanced density by processing incomplete point cloud data in multiple steps, combining shape priors of a prefix embedding tree and deep learning techniques. Multi-scale feature extraction can capture global and local geometric information, and the prefix embedding tree dictionary refines geometric details based on shape priors. The semi-connected U-Net connection mode ensures uniform generation of seed point clouds. Finally, the composite loss function constraint ensures that the generated point cloud has a uniform density distribution.

[0021] Preferably, step S1 specifically includes using an abstract set module and a Point Transformer to extract multi-scale features of the incomplete point cloud at different resolutions; the multi-scale features include global geometric features of the incomplete point cloud. By introducing an abstract set module and a Point Transformer for multi-scale feature extraction, the ability to capture and process geometric details of the point cloud during completion can be improved.

[0022] Preferably, the prefix embedding tree dictionary in step S2 includes a root dictionary and a leaf dictionary. The root dictionary is used to match the global geometric features to generate refined root features, and the leaf dictionary is used to refine local geometric features. Through the prefix embedding tree dictionary, the overall structure and local details of the generated point cloud can be accurately ensured, thereby improving the geometric integrity of the completion effect.

[0023] Preferably, the semi-connected U-Net in step S3 processes the features by encoding and decoding the multi-scale features in S1 and the refined features in S2, and concatenating them; in the decoding stage, only the low-level global features of the multi-scale features in the encoder are connected to the upsampled layers of the decoder. Through this improvement, the global geometric features can be maintained while ensuring that the generated point cloud has a uniform density distribution.

[0024] Preferably, the upsampling in step S4 is performed by sequentially upsampling the preliminary completed seed point cloud through an Upsample Transformer module, gradually refining, and obtaining a generated point cloud reaching the target resolution. In this way, the resolution (number) of the point cloud can be gradually improved, ensuring the density balance and geometric fineness of the generated point cloud during the upsampling process, and finally obtaining a generated point cloud with a target resolution.

[0025] Preferably, the composite loss in step S5 includes functions including chamfer distance, double orthogonal constraint loss, and elastic potential energy loss. The introduction of the composite loss function, including chamfer distance, double orthogonal constraint loss, and elastic potential energy loss, can comprehensively constrain the generated point cloud from the aspects of geometric accuracy, feature independence, and density balance, thereby improving the overall quality of point cloud completion and ensuring that the generated point cloud is more accurate in geometry, more effective in feature learning, and more uniform in point cloud distribution.

[0026] Preferably, the formula of the composite constraint function in step S5 is as follows:

[0027]

[0028] wherein, represents the composite loss function value; CD sparse represents the chamfer distance between the seed point cloud and the real point cloud, CD dense represents the chamfer distance between the final generated point cloud and the real point cloud; and are double orthogonal constraint losses of root dictionary and leaf dictionary, respectively; is the elastic potential energy loss; and a, b, and g are hyperparameters. By constraining the generated point cloud through the composite loss function, the chamfer distances between the seed point cloud and the final generated point cloud and the real point cloud are considered, and the double orthogonal constraint loss and the elastic potential energy loss are introduced, thereby ensuring the geometric accuracy and density balance of the generated point cloud. The combination of these loss functions can significantly improve the quality and consistency of the generated point cloud.

[0029] Preferably, the elastic potential energy loss in step S5 is obtained by simulating the seed point cloud as a spring system, and the original length of the i-th spring is defined as the average distance between the i-th down-sampled point and its k nearest neighbor points. The formula for calculating the spring length is as follows:

[0030]

[0031] wherein, L i is the length of the i-th spring, x n is the coordinate of the n-th nearest neighbor point, and x iis the coordinate of the i th down-sampling point, and M is the total number of down-sampling points. The point cloud is simulated as a spring system by the elastic potential energy loss function, the density distribution between the neighboring points of the point cloud is controlled, and the phenomenon of local over-dense or over-dilute is avoided. In this way, the local and global density uniform distribution of the generated point cloud can be effectively ensured, and the geometric fineness and density balance of the point cloud are improved.

[0032] Compared with the prior art, the present application has the following advantages:

[0033] The present application provides a point cloud completion method based on shape prior and deep learning, which significantly solves the problems of uneven density distribution, missing of geometric details and the like in the process of point cloud completion compared with the prior art. The point cloud completion method in the prior art mostly adopts traditional interpolation or rule matching, which cannot effectively handle point clouds with large-scale missing or complex shape, resulting in poor geometric accuracy and density consistency of the completion result. However, the present application realizes a technical breakthrough through the following innovative steps:

[0034] 1. Feature extraction and semi-connected U-Net: The prior art lacks balanced extraction of global features and local geometric features when extracting incomplete point cloud features. The present application generates seed point clouds by multi-scale feature extraction and semi-connected U-Net encoding and decoding of the features, avoiding the uneven density problem caused by excessive explicit local geometric features in the prior art, thereby realizing uniform distribution of point clouds and complete preservation of global geometric features.

[0035] 2. Prefix embedding tree dictionary refinement: Compared with the prior art, the present application introduces a prefix embedding tree dictionary, which can effectively complete the geometric information of missing point clouds, especially complex-shaped objects, by using shape prior knowledge. Through hierarchical processing of the root dictionary and the leaf dictionary, the present application solves the shortcomings of the prior art in local geometric detail inference and improves the geometric detail recovery capability of the completion result.

[0036] 3. Compound loss function and density control: The loss function of the prior art usually only considers the geometric distance of the point cloud, and it is difficult to balance the density and geometric accuracy at the same time. The present application uses a compound loss function, including chamfer distance, double orthogonal constraint loss and elastic potential energy loss, to comprehensively consider the geometric accuracy and density distribution of the point cloud, solve the problem of uneven local density in the prior art, and generate point clouds with better density consistency in the global and local.

[0037] Through the above innovative steps, the present application has a significant improvement in multi-scale feature extraction, geometric information refinement and compound constraint of the loss function compared with the prior art, and can generate point clouds with balanced density and geometric accuracy, which is particularly suitable for point cloud completion tasks with large-scale missing or complex shape. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1A method flowchart of the point cloud completion method based on shape prior and deep learning of embodiment 1 of the present application is shown in FIG. 1.

[0039] Figure 2 A detailed algorithm flowchart of the point cloud completion method based on shape prior and deep learning of embodiment 1 of the present application is shown in FIG. 2.

[0040] Figure 3 A detailed implementation process diagram of the point cloud completion method based on shape prior and deep learning of embodiment 2 of the present application is shown in FIG. 3.

[0041] Figure 4 A prefix embedding tree dictionary structure diagram of embodiment 2 of the present application is shown in FIG. 4.

[0042] Figure 5 A schematic diagram of the elastic potential energy loss function principle of embodiment 2 of the present application is shown in FIG. 5.

[0043] Figure 6 A point cloud completion effect comparison diagram on the PCN dataset of embodiment 3 of the present application is shown in FIG. 6.

[0044] Figure 7 A point cloud completion effect comparison diagram on the KITTI dataset of embodiment 3 of the present application is shown in FIG. 7. DETAILED DESCRIPTION

[0045] The point cloud completion method based on shape prior and deep learning provided by the present application will be further described in detail below in combination with the accompanying drawings and specific embodiments. However, this should not be understood as limiting the scope of the above-mentioned subject matter of the present application to the following embodiments, and any technology realized based on the content of the present application falls within the scope of the present application. The advantages and features of the present application will be clearer in combination with the following description. It should be noted that the drawings are very simplified and use non-precise proportions, and are only used to facilitate and clarify the purpose of assisting in the description of the embodiments of the present application.

[0046] Embodiment 1

[0047] Embodiment 1 gives a specific implementation of the point cloud completion method based on shape prior and deep learning described in the present application, as shown in FIG. 1, which includes the following steps: Figure 1

[0048] S1, feature extraction is performed on the input incomplete point cloud data to obtain multi-scale features of the incomplete point cloud;

[0049] S2, the features extracted in S1 are refined using a prefix embedding tree to obtain refined features;

[0050] S3, the multi-scale features described in S1 and the refined features described in S2 are processed through a semi-connected U-Net, and after multi-layer perceptron regression, a preliminary completed seed point cloud is obtained; ​

[0051] S4, upsampling the preliminary completed seed point cloud in S3 to obtain a generated point cloud reaching a target resolution;

[0052] S5, using a compound loss function to constrain the generated point cloud in S4 to obtain a complete point cloud with balanced density.

[0053] The method gradually generates a complete point cloud with balanced density by processing the incomplete point cloud data through multiple steps, combining shape priors of prefix embedding trees and deep learning techniques. Multi-scale feature extraction can capture global and local geometric information, and the prefix embedding tree dictionary refines geometric details based on shape priors. The semi-connected U-Net connection mode ensures the uniform generation of seed point clouds. Finally, the compound loss function constraint ensures that the generated point cloud has a uniform density distribution.

[0054] Preferably, step S1 specifically includes using an abstract set module and a Point Transformer to extract multi-scale features of the incomplete point cloud at different resolutions; the multi-scale features include global geometric features of the incomplete point cloud. In this way, the multi-scale feature extraction by introducing the abstract set module and the Point Transformer can improve the ability to capture and process geometric details of the point cloud during the completion process.

[0055] Preferably, the prefix embedding tree dictionary in step S2 includes a root dictionary and a leaf dictionary. The root dictionary is used to match the global geometric features to generate refined root features, and the leaf dictionary is used to refine local geometric features. Through the prefix embedding tree dictionary, the overall structure and local details of the generated point cloud can be ensured to be accurate, thereby improving the geometric integrity of the completion effect.

[0056] Preferably, the semi-connected U-Net in step S3 includes: encoding and decoding the multi-scale features in S1 and the refined features in S2, and splicing them; in the decoding stage, only the low-level global features of the multi-scale features in the encoder are connected to the up-sampling layer of the decoder by jumping. Through this improvement, the global geometric features can be maintained while ensuring that the generated point cloud has a uniform density distribution.

[0057] Preferably, the upsampling in step S4 is performed by an Upsample Transformer module to gradually refine the preliminary completed seed point cloud, thereby obtaining a generated point cloud reaching a target resolution. In this way, the resolution of the point cloud can be gradually improved to ensure the density balance and geometric precision of the generated point cloud during the upsampling process, and finally a generated point cloud reaching a target resolution is obtained.

[0058] Preferably, the composite loss in step S5 includes functions including chamfer distance, double orthogonal constraint loss, and elastic potential energy loss. The introduction of the composite loss function, including chamfer distance, double orthogonal constraint loss, and elastic potential energy loss, can comprehensively constrain the generated point cloud from the aspects of geometric accuracy, feature independence, and density balance, thereby improving the overall quality of point cloud completion and ensuring that the generated point cloud is more accurate in geometry, more effective in feature learning, and more uniform in point cloud distribution.

[0059] Preferably, the formula of the composite constraint function in step S5 is as follows:

[0060]

[0061] wherein, represents the composite loss function value; CD sparse represents the chamfer distance between the seed point cloud and the real point cloud, CD dense represents the chamfer distance between the final generated point cloud and the real point cloud; and are double orthogonal constraint losses of root dictionary and leaf dictionary, respectively; is the elastic potential energy loss; and α, β, and γ are hyperparameters. By using the composite loss function to constrain the generated point cloud, the geometric chamfer distance between the seed point cloud and the final generated point cloud and the real point cloud is comprehensively considered, and the double orthogonal constraint loss and the elastic potential energy loss are introduced, thereby ensuring the geometric accuracy and density balance of the generated point cloud. The combination of these loss functions can significantly improve the quality of the generated point cloud.

[0062] Preferably, the elastic potential energy loss in step S5 is defined by simulating the seed point cloud as a spring system, and the original length of the i-th spring is defined as the average distance between the down-sampled point and its k nearest neighbor points. The formula for calculating the spring length is as follows:

[0063]

[0064] wherein, L i is the length of the i-th spring, x n is the coordinate of the n-th nearest neighbor point, x i is the coordinate of the i-th down-sampled point, and M is the total number of down-sampled points. By simulating the point cloud as a spring system through the elastic potential energy loss function, the density distribution between the neighboring points of the point cloud is controlled, and the phenomenon of local over-dense or over-diluted is avoided. In this way, the local and global density uniform distribution of the generated point cloud can be effectively ensured, and the geometric fineness and density balance of the point cloud are improved.

[0065] Embodiment 2

[0066] As an optimization of the foregoing embodiment, the present embodiment provides a detailed implementation process of a point cloud completion method based on shape prior and deep learning, which combines multi-scale feature extraction, prefix embedding tree dictionary, semi-connected U-Net and composite loss function to solve the density imbalance and missing problem in point cloud data.

[0067] Figure 2 The algorithm flowchart of the present application is shown, which outlines the overall steps of the point cloud completion method based on shape prior and deep learning. First, multi-scale feature extraction is performed on the input incomplete point cloud data. Second, the multi-scale features are refined using the prefix embedding tree dictionary to obtain refined features. Then, the multi-scale features and refined features are spliced using semi-connected U-Net, and the preliminary seed point cloud is obtained using multi-layer perceptron (MLP) regression. Subsequently, the preliminary completed seed point cloud is upsampled to generate the generated point cloud of the target resolution. Finally, the generated point cloud is constrained by the composite loss function (including chamfer distance, double orthogonal constraint loss and elastic potential energy loss) to ensure its density balance and geometric accuracy.

[0068] Figure 3 The specific implementation process diagram of the method of the present application is shown, which details the operation details of each step. First, the input incomplete point cloud is feature extracted, and global and local multi-scale features are extracted through the abstract set module and Point Transformer module. Then, the geometric features are refined using the prefix embedding tree dictionary, and the multi-scale features and refined features are processed by semi-connected U-Net. Afterwards, Upsample Transformer is used for upsampling in the implementation process to gradually improve the resolution of the point cloud. Finally, the generated point cloud is density balanced and accuracy optimized by combining the composite loss function, generating a high-quality complete point cloud.

[0069] The detailed implementation process of the point cloud completion method based on shape prior and deep learning described in the present embodiment is as follows:

[0070] S1, feature extraction

[0071] As shown in Figure 3 , the feature extraction module includes a structure combining the abstract set module (Set Abstract Module, SAM) and the Point Transformer. The incomplete point cloud is input into the feature extraction module, and multi-scale features of the incomplete point cloud at different resolutions are extracted. Feature extraction is divided into multiple levels:

[0072] (1) SAM down-samples the incomplete point cloud to obtain point cloud features at different resolutions;

[0073] (2)Point Transformer is based on self-attention mechanism, including a self-attention layer (SAL), which enhances the geometric features of each point by designing a self-attention layer for point clouds, and extracts global and local geometric features of the point cloud.

[0074] Specifically, the multi-scale feature and the point cloud coordinates P i+1 is obtained through the nested calculation of two functions. Wherein, i represents the scale corresponding to the current feature and the incomplete point cloud coordinates. The multi-scale feature extraction module in the application has three scales, and i takes the value range of {0, 1, 2}. The parameter selection of SAM and SAL under each scale corresponds to the feature dimension under the corresponding scale and the number of incomplete point clouds, which is expressed as follows:

[0075]

[0076] It should be noted that in the first step SAM, i.e. i = 0, By stacking the abstract set module and the point cloud self-attention layer, multi-scale features and global features for the bottom layer of the semi-connected U-Net can be obtained.

[0077] S2, point cloud completion

[0078] After obtaining the multi-scale features of the incomplete point cloud, the preliminary seed point cloud is obtained by using the prefix embedding tree dictionary and the semi-connected U-Net. The prefix embedding tree shape prior and the semi-connected U-Net are complementary to each other. The prefix embedding tree shape prior is responsible for refining the multi-scale features, and uses the shape prior to make reasonable geometric inference on the missing part, thereby generating refined features. Then, the semi-connected U-Net further generates a more complete and balanced density point cloud by upsampling processing and introducing a semi-connected mechanism to solve the problem of uneven density of the seed point cloud.

[0079] S201, introducing the shape prior of the prefix embedding tree

[0080] The shape prior of the prefix embedding tree is mainly used to help the model learn global and local geometric features from the input incomplete point cloud. This process is based on the global shape prior of the point cloud and the existing geometric knowledge, and the multi-scale features are refined by the shape prior to obtain features with missing geometric information, which lays a foundation for the subsequent MLP regression to obtain seed point clouds, and ensures that the generated point cloud can reasonably infer the geometric structure of the missing part.

[0081] To ensure the shape prior learned from the prefix embedding tree dictionary is inherited in the decoding process of the subsequent half-connection U-Net, a learnable embedding dictionary of the prefix embedding tree shape is designed, which includes two layers: root dictionary and leaf dictionary. Specifically:

[0082] (1) Root dictionary: The root dictionary combines the input global feature with the rough geometric shape through the cross-attention mechanism, which is used to refine the overall geometric structure of the point cloud. The input rough root feature is matched with the shape information in the root dictionary, and the similarity between them is calculated according to the consistency graph, thereby generating refined root features. The higher the similarity, the closer the shape is to the geometric features in the incomplete point cloud, and the more accurate the generated root features are.

[0083] (2) Leaf dictionary: The refined root feature is further upsampled to become the input feature of the leaf dictionary. The leaf dictionary further refines the input rough leaf feature to ultimately complete the detailed geometric features of the point cloud. These refined features combine multi-scale information and can better restore the local geometric shape of the point cloud.

[0084] The features refined by the prefix embedding tree dictionary are concatenated with the multi-scale features obtained in step S1 to obtain complete shape features.

[0085] As shown in FIG. 3, the global feature extracted based on the abstract set module and the Point Transformer Figure 3 is first upsampled to a rough root feature through one-dimensional deconvolution. The rough root feature is refined by the root dictionary to obtain a refined root feature . The refined root feature is concatenated with the previously extracted middle-resolution point cloud feature , which is to retain the original features at the global semantic level. The concatenated feature is again upsampled through one-dimensional deconvolution to obtain a rough leaf feature , which is the input of the leaf dictionary. The rough leaf feature is output after being refined by the leaf dictionary to obtain a refined leaf feature . The refined leaf feature is no longer connected with the original point cloud feature , but directly obtains the complete point cloud P seed through an MLP regression.

[0086] The refinement process of the prefix-embedded tree dictionary is divided into two stages. Firstly, the root dictionary combines the global features of the incomplete point cloud with the pre-stored geometric shapes in the dictionary through cross-attention mechanism, refines and generates preliminary completed geometric features. Subsequently, these refined root features are further up-sampled and input into the leaf dictionary, combined with local detailed features for further fine processing to generate the final completed geometric features. The whole process ensures the balance between global and local geometric features.

[0087] As shown in Figure 4 , the prefix-embedded tree dictionary includes a root dictionary and a leaf dictionary, wherein Figure 4 (a) is the refinement process of the root dictionary, Figure 4 (b) is the refinement process of the leaf dictionary. Through the prefix-embedded tree dictionary, the shape completion process can obtain multi-scale inheritable prior knowledge guidance to generate more accurate results. In the root dictionary refinement process, the global features are obtained through one-dimensional deconvolution to obtain coarse root features The coarse root features will use the root dictionary to generate refined root features The refinement process of the root dictionary is shown in Figure 4 (a), and the input coarse root features are first cross-attention operated with the root dictionary and the root consistency map is obtained This step is to calculate the vectors similar to the coarse root features in the root dictionary and the corresponding weights. The higher the similarity, the greater the attention weight obtained by the vector, indicating that the vector has high similarity with the features corresponding to the incomplete point cloud and has guiding significance. The weighted root embedding is concatenated with the input coarse root features, and the final output is the refined root feature. The cross-attention operation function is defined as:

[0088]

[0089] wherein Q and K represent feature matrices, d K is the dimension of K. The calculation process of the consistency map ConsMap r can be represented as:

[0090]

[0091] wherein and represent two one-dimensional convolution layers. Based on the consistency map, the root dictionary is weighted to obtain the refined root features

[0092]

[0093] Wherein, MatMul represents matrix multiplication, and ψ is an equalization factor, which is set as 0.5 as a hyperparameter in the embodiment. The reason is that adding two kinds of features will cause the values in the matrix to be quite different from other parts of the module, and through the equalization factor, the values in the matrix can be restored to the original level, preventing the occurrence of abnormal values.

[0094] Refined root features output by the root dictionary Firstly, the mid-level resolution point cloud features are concatenated Then, the one-dimensional deconvolution layer is input to upsample to obtain the coarse leaf features Coarse leaf features As the input of the leaf dictionary refinement process. The leaf dictionary refinement process is as shown in Figure 4 (b), the input coarse leaf features are first calculated by a multi-layer perception to obtain a leaf index n. The leaf index points to the target leaf dictionary of the current feature. The refinement process of the coarse leaf features and the leaf dictionary is consistent with the refinement process of the coarse root features and the root dictionary. In order to improve the representation ability of the vectors in the dictionary, a double orthogonal constraint loss function is introduced in the embodiment, and the calculation method can be represented as:

[0095]

[0096] Wherein, I r and I l represent the unit matrix corresponding to the root dictionary and the leaf dictionary, and ε r represents the root dictionary matrix, represents the i-th leaf dictionary matrix. In the entire prefix embedding tree dictionary refinement process, through the calculation of the loss and the back propagation, the model will optimize the parameters according to the gradient, and this optimization process is the fitting process of the prefix embedding tree dictionary to the complete geometric features in the training set.

[0097] S202, improvement of the semi-connected U-Net

[0098] In order to solve the problem that the traditional U-Net lacks accurate modeling of the missing point cloud geometric features in the point cloud completion task, and the generated seed point cloud has uneven density distribution, the application proposes an improved semi-connected U-Net structure, which cancels the skip connection of the uppermost layer of the traditional U-Net, and only retains the connection of the low-level global features. By weakening the skip connection of the high-level explicit features, the semi-connected U-Net can avoid the phenomenon of excessively high local density in the point cloud completion process, so as to generate point clouds with more balanced density while maintaining the global geometric features. This method effectively alleviates the problem that the point-by-point features of the incomplete point cloud are too explicit in the upsampling process, and ensures that the generated point cloud has a uniform density distribution in the local and global.

[0099] The decoding process of the semi-connected U-Net generates multi-scale features by stacking multiple one-dimensional deconvolution layers with different convolution kernel sizes and steps, which are used to capture the local geometric information of the incomplete point cloud at multiple scales.

[0100] After extracting the multi-scale features in step S1, step S201 performs deconvolution upsampling and prefix embedding tree dictionary shape prior on the multi-scale features to obtain refined features. Then, the processing stage of the semi-connected U-Net is entered. The semi-connected U-Net encodes and decodes the multi-scale features obtained in step S1 and the refined features obtained in step S201, and performs splicing. Then, an MLP regression is used to generate the coordinates and point-by-point features of the preliminary completed seed point cloud. The generation and upsampling of the seed point cloud are realized through a direct functional relationship, i.e., the seed point cloud is passed as a parameter to the upsampling function for processing.

[0101] S3, upsampling

[0102] The seed point cloud is upsampled by the Upsample Transformer. The upsampling ratio is set to [1, 4, 4], and the seed point cloud is gradually upscaled to the target resolution.

[0103] Specifically, the seed point cloud generated in step S2 contains sparse and uniformly distributed point cloud coordinates. These seed point clouds are then input into the Upsample Transformer module, which gradually refines the seed point cloud into the generated point cloud of the target resolution through successive upsampling. In each upsampling operation, the Upsample Transformer processes the geometric features of the seed point cloud and combines the shape prior information in the prefix embedding tree dictionary.

[0104] S4, loss function and optimization

[0105] To ensure that the generated point cloud has high density balance and geometric precision, the present embodiment adopts multiple loss functions, including the classic chamfer distance, double orthogonal constraint loss, and the elastic potential energy loss designed by the present embodiment, which are used to constrain the training process. The specific content is as follows:

[0106] (1) Chamfer Distance (CD): Chamfer Distance is used to measure the geometric chamfer distance between the completed point cloud and the real point cloud. By calculating the distance between each point of the completed point cloud and the target point cloud, it ensures that the completed point cloud can be as close to the target shape as possible. CD sparse is used to measure the chamfer distance between the seed point cloud and the real point cloud. CD dense represents the chamfer distance between the final completed point cloud and the real point cloud.

[0107] (2) Dual orthogonal constraint loss: In order to ensure that the features in the prefix embedding tree dictionary are not redundant, the dual orthogonal constraint loss optimizes the multi-scale feature vectors in the root dictionary and the leaf dictionary. and are the dual orthogonal constraint losses of the root dictionary and the leaf dictionary respectively; specifically, is used to constrain the feature representation in the root dictionary, ensuring that each feature has sufficient discrimination and there is no redundant information; is used to constrain the feature representation in the leaf dictionary, ensuring that the multi-level geometric refinement features in the leaf dictionary are not repeated, improving the efficiency and accuracy of dictionary learning.

[0108] (3) Elastic potential energy loss:

[0109] In the upsampling process, the generated point cloud may be too concentrated around the seed point cloud, resulting in uneven local density. To solve this problem, the elastic potential energy loss is designed, which simulates the generated point cloud as a spring system to constrain the position of the point cloud and ensure its uniform distribution locally and globally. As shown in Figure 5 , the specific process is as follows:

[0110] First, use FPS (farthest point sampling) to downsample the generated complete point cloud to M points. Then, find the k-nearest neighbors of each downsampled point (k is twice the number of upsampled points). The downsampled point and its k-nearest neighbors are regarded as a spring system, and the original length of the i-th spring is defined as the average distance between the i-th downsampled point and its k-nearest neighbors, which can control the density distribution of the completed point cloud of the generated point cloud. The formula for calculating the spring length is as follows:

[0111]

[0112] where L i is the length of the i-th spring, x n is the coordinate of the n-th nearest neighbor, and x i is the coordinate of the i-th downsampled point; M is the total number of downsampled points, and each downsampled point corresponds to a spring. The downsampled points come from the completed complete point cloud. Based on Hooke's law, the deformation of the spring and the corresponding elastic potential energy are calculated. Through this spring system model, the position of the generated point cloud is effectively constrained, which can prevent the problem of local over-density and ensure the uniform distribution of the point cloud globally and locally. The elastic potential energy is proportional to the square of the deformation, and the formula is:

[0113]

[0114] where E irepresents the elastic potential energy of the i-th spring, and ξ is the 0.5 times the elastic coefficient. The constant part can be used as a hyperparameter, so ξ can be directly regarded as a hyperparameter, and the elastic coefficient is set to 10 in all experiments.

[0115] Next, the sum of the potential energy of all springs is set as the elastic potential energy loss function, and the formula is:

[0116]

[0117] wherein, is the elastic potential energy loss. By introducing this constraint, the problem of the generated point cloud being too concentrated around the seed point cloud can be effectively alleviated, ensuring that the local and global densities of the generated point cloud are uniform.

[0118] Finally, the composite loss function is established by combining the chamfer distance, double orthogonal constraint loss and elastic potential energy loss as follows:

[0119]

[0120] wherein, represents the composite loss function value; CD sparse represents the chamfer distance between the seed point cloud and the real point cloud, CD dense represents the chamfer distance between the final generated point cloud and the real point cloud; and are the double orthogonal constraint losses of the root dictionary and the leaf dictionary, respectively; is the elastic potential energy loss; α, β and γ are hyperparameters, which are set to 1, 1 and 10 respectively in all experiments.

[0121] Through the design and optimization of the above loss function, it is ensured that the generated point cloud is close to the real point cloud in geometric shape, and at the same time, the local and global densities of the completed point cloud are balanced.

[0122] In Figure 4 , in order to reflect the density of the point cloud, the low-density area is represented by cyan, and the high-density area is represented by magenta. When the density of the point cloud is relatively uniform, the point cloud will overall present the same color, such as magenta or cyan.

[0123] Embodiment 3

[0124] In order to verify the effectiveness of the method, this embodiment is verified on the PCN dataset. The PCN dataset is a subset of the ShapeNet dataset, containing 8 categories of objects, a total of 30974 point clouds. Based on the experience of previous research, the dataset is divided into: 28,974 training sets, 1,200 test sets, and 800 validation sets.

[0125] To ensure the consistency of the incomplete point cloud data, the incomplete point cloud is resampled to 2048 points of standard size. The up-sampling ratio is consistent with the previous method, and the up-sampling scale is set to {1, 4, 4}. In addition, the following parameters are set in the experiment: the dimension of the root dictionary is set to 256, and the dimension of the leaf dictionary is set to 128; the root dictionary node number N is set to 10; the training round is set to 400, and the batch size is set to 12; the Adam optimizer is used for training, and the learning rate is set to 4x10 -4 , which is attenuated by 0.8 times every 20 rounds; the experimental environment is Nvidia GeForce RTX 4090 graphics card (24 GB GPU).

[0126] The present method is compared with nine existing methods. The other methods in the comparative experiment are represented by the original English names. Table 1 shows the quantitative results of the PCN dataset comparative experiment, and it can be seen that the present method ranks second on CDL1 and ranks first on DCD and F1 value. In particular, the present method has a 1.1x10 -2 improvement over ODGNet on DCD. This shows that the present method has achieved significant results in considering the density distribution of the point cloud.

[0127] Table 1: Quantitative results of PCN dataset comparative experiment (CD-L1x10 -3 )

[0128]

[0129] Wherein, the smaller the values of CD and DCD and the higher the F1 score, the better the effect of the method in completing the point cloud in terms of density distribution.

[0130] Figure 6 The visualization results of several methods are shown, which are Snowflakenet, SeedFormer, ODGNet and the present method. The colors in the figure are used to represent the density of the point cloud, and the cyan color represents the area with lower density, and the magenta color represents the area with higher density. When the density of the point cloud is uniform, the color of the entire point cloud will present a consistent state, that is, all areas are cyan or magenta. It can be seen that the point cloud generated by the present method is more uniform in density distribution, which shows that in the area not containing the input points, the present method can better restore the geometric information of the missing part. In particular, in the part of the point cloud not contained in the input points, the shape completion effect of the present method is obviously more complete than other methods. In contrast, other methods have the problem of uneven density, which is specifically manifested as the polarization phenomenon (i.e. the mixture of cyan and magenta) in the point cloud of the same object, which shows that the density distribution of the point cloud is uneven, and is mainly concentrated around the input points, while the density of the area to be completed is low.

[0131] Through experimental results and visual comparison, the method is significantly superior to other existing methods in point cloud completion tasks, especially in density balancing and shape restoration. This shows that the method has made breakthrough progress in handling the density distribution of point clouds.

[0132] Embodiment 4

[0133] In order to verify the performance of the method in real-world datasets, especially the effect in practical applications such as cultural relic restoration, building surveying and mapping, and autonomous driving. This embodiment is further optimized on the basis of the foregoing embodiments, and uses the KITTI dataset for testing. Compared with the virtual point cloud data in the ShapeNet dataset, the KITTI dataset contains point clouds in a real scanning environment, which provides a basis for testing the effectiveness of the point cloud completion method in real-world applications.

[0134] In this experiment, only objects labeled as Cars in the KITTI dataset are considered, and point cloud data in their bounding boxes are extracted for completion. A total of 2483 incomplete point clouds are tested, which have no ground truth, so the data in the PCN dataset that also belong to the Cars category are used to pretrain the model. Then, the following two indicators are used to evaluate the completion effect of the model:

[0135] Fidelity: used to evaluate the similarity between the completed point cloud and the original incomplete point cloud.

[0136] MMD (Maximum Mean Discrepancy): used to measure the difference between the generated point cloud and the real point cloud distribution.

[0137] In the experiment, the dimension of the root dictionary is set to 128, the dimension of the leaf dictionary is set to 64, and the number of root dictionary nodes N is set to 8.

[0138] In order to verify the effectiveness of the method, it is compared with 8 existing point cloud completion methods. The other methods in the comparison experiment are represented by their original English names. The quantitative experimental results of each method are shown in Table 2. The method achieves the optimal result (SOTA) in the MMD indicator, which is 14% higher than other methods, indicating that the method generates a point cloud distribution closer to the real distribution when processing real data.

[0139] Table 2: KITTI dataset comparison experiment quantitative results (FD × 10 3 , MMD × 10 3 )

[0140]

[0141] Among them, the smaller the values of FD and MMD, the better the point cloud completion effect achieved by the method.

[0142] Figure 7 The visualization comparison results of the present method and the PoinTr method on the KITTI dataset are shown. As can be seen from the figure, the present method can more smoothly complete the point cloud, and the generated noise is less. In contrast, the PoinTr generated point cloud has more noise, and the shape of the completed point cloud is relatively rough. The present method not only generates a point cloud with more uniform density, but also has a more complete shape, indicating that it has a strong advantage in processing real-world point cloud data.

[0143] In summary, through the test of the KITTI dataset, the present method not only performs well on virtual point cloud data (such as ShapeNet), but also can achieve a smooth and less noisy high-quality completion effect in real point cloud data. This proves that the present method has practical application value in the fields of cultural relic restoration, building surveying and mapping, and autonomous driving, and has a significant advantage compared with the prior art.

[0144] Since the ShapeNet dataset is a virtual generated point cloud, there may be differences with the real-world scanned point cloud dataset. In order to test the effect of the completion method on the real dataset and its application in autonomous driving, the present method is further tested using the KITTI dataset. Following the previous method, only the points in the bounding box of the object marked as a car are focused on, and there are a total of 2483 incomplete point clouds, and there is no ground truth. The model is pre-trained using the Cars category in the PCN dataset, and the model is evaluated using Fidelity and MMD. The dimension of the root dictionary is set to 128, and the dimension of the leaf dictionary is set to 64. For the root dictionary node number N, it is set to 8. The present method is compared with the existing 8 methods. As shown in Table 2, the present method achieves the SOTA result on MMD, with a relative improvement of 14%. As shown in FIG. 6, the present method can also complete the real-world point cloud. Compared with PoinTr, the present method is smoother and has less noise. Figure 7

[0145] Embodiment 5

[0146] As a preferred embodiment of the foregoing embodiments, the present embodiment provides an apparatus for performing a point cloud completion method based on shape prior and deep learning.

[0147] The apparatus includes at least one processor, and a memory connected in communication with the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform a point cloud completion method based on shape prior and deep learning.

[0148] The apparatus includes a plurality of functional modules, each module corresponding to a different step in the method. Specifically, the following components are included:​

[0149] Feature extraction module: This module is used to extract multi-scale features from the input incomplete point cloud. It uses an abstract set module and a Point Transformer module to capture multi-scale features of the point cloud, thereby generating preliminary global and local geometric features.

[0150] Prefix embedding tree module: This module uses a prefix embedding tree dictionary to further refine the multi-scale features. Through the cooperation of the root dictionary and the leaf dictionary, it ensures that the refined features include the geometric structure of the missing part of the input point cloud.

[0151] Semi-connected U-Net processing module: This module processes the multi-scale features generated by the feature extraction module and the refined features generated by the prefix embedding tree module based on a semi-connected U-Net, generating a preliminary seed point cloud. By only retaining low-level global feature skip connections, it avoids the influence of high-level local geometric features on the density distribution of the seed point cloud, improving the balance of the generated point cloud.

[0152] Up-sampling module: This module uses an Upsample Transformer to perform successive up-sampling on the refined seed point cloud, gradually refining it to obtain a generated point cloud that reaches the target resolution. The up-sampling module can maintain the balance of the point cloud density and gradually improve the point cloud resolution.

[0153] Composite loss constraint module: used to constrain the generated point cloud through a composite loss function, which includes chamfer distance, double orthogonal constraint loss, and elastic potential energy loss. It combines chamfer distance, double orthogonal constraint loss, and elastic potential energy loss to ensure that the generated point cloud is uniformly distributed in global and local density, improving geometric accuracy.

[0154] The device can automatically complete the point cloud completion task and generate a complete point cloud with geometric accuracy and balanced density through the coordinated operation of the modules.

[0155] Those skilled in the art should understand that embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects.

[0156] The technical features of the above-described embodiments can be combined in any manner. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described, but as long as the combinations of the technical features do not contradict, they should be considered within the scope of the present disclosure.

[0157] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be noted that for ordinary skilled persons in the art, without departing from the concept of the present application, several modifications and improvements can be made, which are within the scope of protection of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A point cloud completion method based on shape prior and deep learning, characterized in that, The method comprises the following steps: S1, feature extraction is performed on the input incomplete point cloud data, wherein the point cloud data is a point cloud in a KITTI dataset, and multi-scale features of the incomplete point cloud are obtained; S2, the features extracted in S1 are refined using a prefix embedding tree, and refined features are obtained; In S2, the prefix embedding tree dictionary comprises a root dictionary and a leaf dictionary, the global geometric features are matched using the root dictionary to generate refined root features, and the local geometric features are generated by refining the leaf dictionary; S3, the multi-scale features in S1 and the refined features in S2 are processed by a semi-connected U-Net, and a preliminary completed seed point cloud is obtained through multi-layer perception regression; In S3, the semi-connected U-Net processes the features by encoding and decoding the multi-scale features in S1 and the refined features in S2, and performing splicing; in the decoding stage, only the low-level global features of the multi-scale features in the encoder are connected to the up-sampling layer of the decoder by jumping connection; S4, the preliminary completed seed point cloud is up-sampled to obtain a generated point cloud reaching a target resolution; S5, a composite loss function is used to constrain the generated point cloud in S4 to obtain a complete point cloud with balanced density; The formula of the composite loss function in S5 is as follows: wherein, represents a composite loss function value; represents a chamfer distance between the seed point cloud and the real point cloud, represents a chamfer distance between the final generated point cloud and the real point cloud; and are double orthogonal constraint losses of the root dictionary and the leaf dictionary, respectively; is an elastic potential energy loss; , and is a hyperparameter; The double orthogonal constraint loss calculation method is as follows: wherein, and denote the identity matrix corresponding to the root dictionary and the leaf dictionary, denotes the root dictionary matrix, denotes the matrix of the th leaf dictionary. S5 said elastic potential energy loss by simulating the seed point cloud as a spring system, define the original length of the first spring is the average distance between the first down-sampling points to its nearest neighbor points; the formula for calculating the spring length is as follows: wherein, is the length of the first spring, is the coordinate of the first nearest neighbor point, is the coordinate of the first downsampled point, is the total number of downsampled points.

2. The point cloud completion method based on shape prior and deep learning according to claim 1, characterized in that, Step S1 specifically comprises using an abstract set module and a Point Transformer to extract multi-scale features of the incomplete point cloud at different resolutions; the multi-scale features comprise global geometric features of the incomplete point cloud.

3. The point cloud completion method based on shape prior and deep learning according to claim 1, characterized in that, The up-sampling in S4 is performed by an Upsample Transformer module to gradually up-sample the preliminary completed seed point cloud, gradually refine, and obtain a generated point cloud reaching a target resolution.

4. The point cloud completion method based on shape prior and deep learning according to claim 3, characterized in that, The composite loss in S5 comprises a chamfer distance, a double orthogonal constraint loss, and an elastic potential energy loss.

5. An apparatus for a point cloud completion method based on shape priors and deep learning, the apparatus comprising: The device comprises at least one processor and a memory connected to the at least one processor in communication; the memory stores instructions executable by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor can execute the method in any one of claims 1-4.

6. An apparatus for a point cloud completion method based on shape priors and deep learning, the apparatus comprising: The device comprises: A feature extraction module is configured to extract multi-scale features from an input incomplete point cloud, wherein the point cloud data is a point cloud in a KITTI dataset, and multi-scale features of the incomplete point cloud are obtained; A prefix embedding tree module is configured to further refine the geometric information of the multi-scale features using a prefix embedding tree dictionary; the prefix embedding tree dictionary comprises a root dictionary and a leaf dictionary, the global geometric features are matched using the root dictionary to generate refined root features, and the local geometric features are generated by refining the leaf dictionary; The semi-connected U-Net processing module is configured to process the multi-scale features generated by the feature extraction module and the refined features generated by the prefix embedding tree module based on a semi-connected U-Net to generate preliminary seed point clouds. The semi-connected U-Net processing of the features includes encoding and decoding the multi-scale features and the refined features and splicing them. In the decoding stage, only the low-level global features of the multi-scale features in the encoder are connected to the up-sampling layer of the decoder by a skip connection. The up-sampling module is configured to perform successive up-sampling on the refined seed point clouds by an UpsampleTransformer to gradually refine the seed point clouds and obtain generated point clouds that reach a target resolution. The composite loss constraint module is configured to constrain the generated point clouds by a composite loss function. The composite loss includes a chamfer distance, a double orthogonal constraint loss, and an elastic potential energy loss. The formula of the composite loss function is as follows: wherein, represents a composite loss function value; represents a chamfer distance between the seed point cloud and the real point cloud, represents a chamfer distance between the final generated point cloud and the real point cloud; and are double orthogonal constraint losses of the root dictionary and the leaf dictionary, respectively; is an elastic potential energy loss; , and is a hyperparameter; The double orthogonal constraint loss is calculated in the following manner: wherein, and denote the identity matrix corresponding to the root dictionary and the leaf dictionary, denotes the root dictionary matrix, denotes the matrix of the th leaf dictionary. The elastic potential energy loss is defined as the first original length of the first down-sampled point to its nearest neighbor point; the formula for the spring length is as follows: wherein, is the length of the th spring, is the coordinate of the th nearest point, is the coordinate of the th down-sampled point, is the total number of down-sampled points.

Images