Point cloud completion method and device based on shape prior and deep learning

Abstract

The invention relates to the field of computer vision, in particular to a point cloud completion method and device based on shape prior and deep learning. The method comprises the following steps: firstly, carrying out feature extraction on input incomplete point cloud data; then, the multi-scale features are refined through a prefix embedding tree, and refined features are obtained; thirdly, the multi-scale features and the refined features are connected through semi-connection U-Net, and a preliminarily completed seed point cloud is obtained; then, performing up-sampling on the preliminarily completed seed point cloud to generate a point cloud reaching a target resolution; and finally, the generated point cloud is constrained by using a composite loss function, and the composite loss function comprises a chamfering distance, dual orthogonal constraint loss and elastic potential energy loss, so that a complete point cloud with balanced density can be obtained. According to the method and the device, the technical effects of density balance, geometric accuracy and complete detail recovery in the whole point cloud complementing process are realized, and the problems of non-uniform local density and missing geometric information are solved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a point cloud completion method and device based on shape prior and deep learning. Background Art

[0002] In three-dimensional data processing, point cloud is a commonly used data representation. Since point cloud data can represent three-dimensional structures at any resolution and has the advantages of high geometric accuracy and flexible storage, it has been widely used in industrial inspection, autonomous driving, architectural surveying and mapping, cultural heritage protection and other fields. However, in the actual point cloud data acquisition process, data is often missing due to problems such as view occlusion, incomplete coverage, and high reflection; coupled with the resolution limitation of the acquisition equipment, the collected raw point cloud data is usually sparse and incomplete. These defects have a serious impact on downstream tasks that rely on complete three-dimensional data (such as segmentation, target detection, three-dimensional reconstruction, etc.), making the completion and repair of point cloud data a very important task in computer vision.

[0003] With the development of technology, existing point cloud completion methods can be roughly divided into two categories: traditional methods and deep learning-based completion methods.

[0004] Traditional point cloud completion methods mainly recover missing data through interpolation, shape matching or based on shape databases. These methods work well when dealing with objects with regular shapes or small-scale missing objects, but they are obviously ineffective for point cloud data with large-scale missing objects or complex shapes. The reason is that these methods lack an understanding of the global shape of the object and cannot infer the complex geometric structure of the missing part.

[0005] With the rapid development of deep learning, learning-based point cloud completion methods have begun to be widely used. Compared with traditional methods, deep learning methods can not only be trained through incomplete point clouds (i.e., sparse point cloud data input), but also extract richer features using large-scale data sets. Therefore, deep learning has a stronger completion ability when dealing with point clouds with more missing points.

[0006] Most of the common learning-based point cloud completion solutions currently use a two-stage completion framework. First, the incomplete point cloud is input into the model to generate a representative seed point cloud (i.e., a sparse complete point cloud). These seed point clouds generate preliminary completion results through feature extraction. Then, the seed point cloud is further refined using an upsampling network to generate a complete point cloud with a more balanced density. However, the existing technology still has the following technical problems in point cloud completion:

[0007] Lack of detailed geometric features: Since the features extracted in the encoding stage can only represent partial information, the generated seed point cloud is difficult to accurately express the detailed geometric features of the missing part, resulting in unsatisfactory final completion effect.

[0008] Insufficient introduction of shape priors: Existing methods often ignore how to effectively utilize the shape priors of objects. Shape priors can provide guidance for geometric understanding of different object categories, but current deep learning models lack an effective mechanism to fully utilize this prior information, which limits the ability to complete objects with complex shapes.

[0009] Uneven density of seed point clouds: In the traditional U-Net structure, the features of the incomplete point cloud are directly concatenated with the upsampled features. This works well in 2D image processing because the features of the image are relatively evenly distributed. However, in point cloud completion, the features extracted by the encoder come only from the incomplete point cloud and lack global geometric information. The features generated by the decoder represent the features of the global point cloud. Directly connecting the two will cause the explicit features corresponding to the incomplete point cloud to be larger than the abstract features of the global point cloud, resulting in an imbalance in the features and an uneven density distribution of the generated seed point cloud.

[0010] The problem of local point cloud density: When completing the point cloud, the existing upsampling method often causes the generated points to be too concentrated near the seed point cloud, resulting in the problem of local point cloud density distribution. The traditional CD-l1 and CD-l2 indicators cannot well reflect the problem of point cloud density distribution. Too many local point clouds will not only reduce the expression ability of the point cloud, but also cause memory redundancy.

[0011] In order to address the above technical problems, the present invention proposes a point cloud completion method and device based on shape prior and deep learning. Summary of the invention

[0012] The purpose of the present invention is to overcome the problems of uneven density distribution of point clouds and insufficient completion of missing geometric features in the prior art, and propose a point cloud completion method and device based on shape prior and deep learning. This method not only effectively introduces shape prior when generating seed point clouds, improving the ability to infer complex geometric structures, but also solves the problems of uneven density distribution and local density of point clouds by introducing semi-connected U-Net structure and elastic potential energy loss. This method can generate more complete and density-balanced point clouds, which is suitable for completion tasks of large-scale missing and complex shapes.

[0013] In order to achieve the above-mentioned object of the invention, the present invention provides the following technical solutions:

[0014] A point cloud completion method based on shape prior and deep learning includes the following steps:

[0015] S1. Extract features from the input incomplete point cloud data to obtain multi-scale features of the incomplete point cloud;

[0016] S2, use the prefix embedding tree to refine the features extracted in S1 to obtain refined features;

[0017] S3, processing the multi-scale features described in S1 and the refined features described in S2 through a semi-connected U-Net, and obtaining a preliminary completed seed point cloud through multi-layer perceptron regression;

[0018] S4, upsampling the seed point cloud initially completed in S3 to obtain a generated point cloud reaching a target resolution;

[0019] S5. Constrain the complete point cloud described in step S4 using a composite loss function to obtain a complete point cloud with balanced density.

[0020] This method processes incomplete point cloud data in multiple steps, combines the shape prior of prefix embedding tree and deep learning technology, and gradually generates a density-balanced and complete point cloud. Multi-scale feature extraction can capture global and local geometric information, the prefix embedding tree dictionary refines the geometric details based on the shape prior, and the connection method of semi-connected U-Net can ensure the uniform generation of seed point clouds. Finally, through the constraints of the composite loss function, it is ensured that the generated point cloud has a uniform density distribution.

[0021] Preferably, step S1 specifically includes using an abstract set module and Point Transformer to extract multi-scale features of incomplete point clouds at different resolutions; the multi-scale features include global geometric features of the incomplete point clouds. By introducing the abstract set module and Point Transformer to extract multi-scale features, the ability to capture and process point cloud geometric details during the completion process can be improved.

[0022] Preferably, the prefix embedding tree dictionary in step S2 includes a root dictionary and a leaf dictionary. The root dictionary is first used to match the global geometric features to generate refined root features, and then the leaf dictionary is used to refine and generate local geometric features. By embedding the prefix into the tree dictionary, the overall structure of the generated point cloud and the accuracy of local details can be guaranteed, thereby improving the geometric integrity of the completion effect.

[0023] Preferably, the semi-connected U-Net in step S3 processes the features by encoding and decoding the multi-scale features in S1 and the refined features in S2, and concatenating them; in the decoding stage, only the low-level global features of the multi-scale features in the encoder are jump-connected to the upsampling layer of the decoder. Through this improvement, the generated point cloud can be guaranteed to have a uniform density distribution while maintaining the global geometric features.

[0024] Preferably, the upsampling in step S4 is to upsample the initially completed seed point cloud one by one through the Upsample Transformer module, and gradually refine it to obtain a generated point cloud with a target resolution. In this way, the resolution (number) of the point cloud can be gradually improved, the density balance and geometric refinement of the generated point cloud during the upsampling process can be ensured, and finally the generated point cloud with the target resolution can be obtained.

[0025] Preferably, the composite loss in step S5 includes functions including chamfer distance, dual orthogonal constraint loss, and elastic potential energy loss. The introduction of a composite loss function, including chamfer distance, dual orthogonal constraint loss, and elastic potential energy loss, can comprehensively constrain the generated point cloud from three aspects: geometric accuracy, feature independence, and density balance, thereby improving the overall quality of point cloud completion, ensuring that the generated point cloud is more geometrically accurate, feature learning is more effective, and point cloud distribution is more uniform.

[0026] Preferably, the formula of the composite constraint function in step S5 is as follows:

[0027]

[0028] in, Represents the value of the composite loss function; CD sparse Represents the chamfer distance between the seed point cloud and the real point cloud, CD dense Indicates the chamfer distance between the final generated point cloud and the real point cloud; and are the dual orthogonal constraint losses for the root dictionary and the leaf dictionary respectively; is the elastic potential energy loss; α, β and γ are hyperparameters. The generated point cloud is constrained by a composite loss function, which comprehensively considers the chamfer distance between the seed point cloud and the final generated point cloud and the real point cloud. At the same time, dual orthogonal constraint loss and elastic potential energy loss are introduced to ensure the geometric accuracy and density balance of the generated point cloud. The combined optimization of these loss functions can significantly improve the quality and consistency of the generated point cloud.

[0029] Preferably, the elastic potential energy loss in step S5 is calculated by simulating the seed point cloud as a spring system, defining the original length of the i-th spring as the average distance between the i-th down-sampling point and its k nearest neighbor points; the formula for calculating the spring length is as follows:

[0030]

[0031] Among them, L i is the length of the ith spring, x n is the coordinate of the nth nearest neighbor, x iis the coordinate of the ith downsampling point, and M is the total number of downsampling points. The elastic potential energy loss function is used to simulate the point cloud as a spring system, and the density distribution between adjacent points of the point cloud is controlled to avoid local over-density or over-sparseness. In this way, the local and global density of the generated point cloud can be effectively guaranteed to be uniformly distributed, and the geometric precision and density balance of the point cloud can be improved.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] The present invention provides a point cloud completion method based on shape prior and deep learning. Compared with the prior art, it significantly solves the common problems of uneven density distribution and missing geometric details in the point cloud completion process. Most point cloud completion methods in the prior art use traditional interpolation or rule matching, which cannot effectively handle point clouds with large-scale missing or complex shapes, resulting in poor geometric accuracy and density consistency of the completion results. The present invention achieves a technological breakthrough through the following innovative steps:

[0034] 1. Feature extraction and semi-connected U-Net: When extracting incomplete point cloud features, existing methods lack balanced extraction of global features and local geometric features. The present invention generates seed point clouds by extracting multi-scale features and encoding and decoding features in combination with semi-connected U-Net, thus avoiding the density imbalance problem caused by excessive explicitness of local geometric features in the prior art, thereby achieving uniform distribution of point clouds and complete retention of global geometric features.

[0035] 2. Prefix embedding tree dictionary refinement: Compared with the existing technology, the present invention can effectively complete the geometric information of missing point clouds, especially objects with complex shapes, by introducing prefix embedding tree dictionaries. Through the hierarchical processing of root dictionaries and leaf dictionaries, the shortcomings of the existing technology in inferring local geometric details are solved, and the geometric detail recovery ability of the completion results is improved.

[0036] 3. Composite loss function and density control: The loss function of existing methods often only considers the geometric distance of point clouds, and it is difficult to balance density and geometric accuracy at the same time. The present invention uses a composite loss function, including chamfer distance, dual orthogonal constraint loss and elastic potential energy loss, to comprehensively consider the geometric accuracy and density distribution of point clouds, solving the problem of local density unevenness in existing methods. The generated point cloud shows better density consistency both globally and locally.

[0037] Through the above-mentioned innovative steps, the present invention has significant improvements over the prior art in multi-scale feature extraction, geometric information refinement and composite constraints of loss functions, and can generate density-balanced and geometrically accurate point clouds, which are particularly suitable for point cloud completion tasks with large-scale missing or complex shapes. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1This is a method flow chart of a point cloud completion method based on shape prior and deep learning according to Embodiment 1 of the present invention;

[0039] Figure 2 is an algorithm flow chart of a point cloud completion method based on shape prior and deep learning according to Embodiment 1 of the present invention;

[0040] Figure 3 is a detailed implementation process diagram of the point cloud completion method based on shape prior and deep learning according to Embodiment 2 of the present invention;

[0041] Figure 4 2 is a schematic diagram of the prefix embedding tree dictionary structure of embodiment 2 of the present invention;

[0042] Figure 5 Schematic diagram of elastic potential energy loss function principle of embodiment 2 of the present invention;

[0043] Figure 6 is a comparison diagram of point cloud completion effects on the PCN dataset of Example 3 of the present invention;

[0044] Figure 7 This is a comparison chart of the point cloud completion effect on the KITTI dataset of Example 3 of the present invention. DETAILED DESCRIPTION

[0045] The point cloud completion method based on shape prior and deep learning provided by the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. However, this should not be understood as the scope of the above-mentioned subject matter of the present invention being limited to the following embodiments. All technologies implemented based on the content of the present invention belong to the scope of the present invention. Combined with the following description, the advantages and features of the present invention will become clearer. It should be noted that the drawings are all in a very simplified form and use non-precise proportions, which are only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention.

[0046] Example 1

[0047] Example 1 provides a specific implementation of a point cloud completion method based on shape prior and deep learning according to the present invention, such as Figure 1 As shown, the following steps are included:

[0048] S1. Extract features from the input incomplete point cloud data to obtain multi-scale features of the incomplete point cloud;

[0049] S2, use the prefix embedding tree to refine the features extracted in S1 to obtain refined features;

[0050] S3, processing the multi-scale features described in S1 and the refined features described in S2 through a semi-connected U-Net, and obtaining a preliminary completed seed point cloud through multi-layer perceptron regression;

[0051] S4, upsampling the seed point cloud initially completed in S3 to obtain a generated point cloud reaching a target resolution;

[0052] S5. Use a composite loss function to constrain the point cloud generated in step S4 to obtain a complete point cloud with balanced density.

[0053] This method processes incomplete point cloud data in multiple steps, combines the shape prior of prefix embedding tree and deep learning technology, and gradually generates a density-balanced and complete point cloud. Multi-scale feature extraction can capture global and local geometric information, the prefix embedding tree dictionary refines the geometric details based on the shape prior, and the connection method of semi-connected U-Net can ensure the uniform generation of seed point clouds. Finally, through the constraints of the composite loss function, it is ensured that the generated point cloud has a uniform density distribution.

[0054] Preferably, step S1 specifically includes using an abstract set module and a Point Transformer to extract multi-scale features of incomplete point clouds at different resolutions; the multi-scale features include global geometric features of the incomplete point clouds. In this way, by introducing an abstract set module and a Point Transformer to extract multi-scale features, the ability to capture and process point cloud geometric details during the completion process can be improved.

[0055] Preferably, the prefix embedding tree dictionary in step S2 includes a root dictionary and a leaf dictionary. The root dictionary is first used to match the global geometric features to generate refined root features, and then the leaf dictionary is used to refine and generate local geometric features. By embedding the prefix into the tree dictionary, the overall structure of the generated point cloud and the accuracy of local details can be guaranteed, thereby improving the geometric integrity of the completion effect.

[0056] Preferably, the semi-connected U-Net in step S3 processes the features by encoding and decoding the multi-scale features in S1 and the refined features in S2, and concatenating them; in the decoding stage, only the low-level global features of the multi-scale features in the encoder are jump-connected to the upsampling layer of the decoder. Through this improvement, the generated point cloud can be guaranteed to have a uniform density distribution while maintaining the global geometric features.

[0057] Preferably, the upsampling in step S4 is to upsample the seed point cloud that is initially completed one by one through the Upsample Transformer module, and gradually refine it to obtain a generated point cloud that reaches the target resolution. In this way, the resolution of the point cloud can be gradually improved, the density balance and geometric precision of the generated point cloud during the upsampling process can be ensured, and finally the generated point cloud of the target resolution can be obtained.

[0058] Preferably, the composite loss in step S5 includes functions including chamfer distance, dual orthogonal constraint loss, and elastic potential energy loss. The introduction of a composite loss function, including chamfer distance, dual orthogonal constraint loss, and elastic potential energy loss, can comprehensively constrain the generated point cloud from three aspects: geometric accuracy, feature independence, and density balance, thereby improving the overall quality of point cloud completion, ensuring that the generated point cloud is more geometrically accurate, feature learning is more effective, and point cloud distribution is more uniform.

[0059] Preferably, the formula of the composite constraint function in step S5 is as follows:

[0060]

[0061] in, Represents the value of the composite loss function; CD sparse Represents the chamfer distance between the seed point cloud and the real point cloud, CD dense Indicates the chamfer distance between the final generated point cloud and the real point cloud; and are the dual orthogonal constraint losses for the root dictionary and the leaf dictionary respectively; is the elastic potential energy loss; α, β and γ are hyperparameters. The generated point cloud is constrained by the composite loss function, and the geometric chamfer distance between the seed point cloud and the final generated point cloud and the real point cloud is comprehensively considered. At the same time, the dual orthogonal constraint loss and elastic potential energy loss are introduced to ensure the geometric accuracy and density balance of the generated point cloud. The combined optimization of these loss functions can significantly improve the quality of the generated point cloud.

[0062] Preferably, the elastic potential energy loss in step S5 is calculated by simulating the seed point cloud as a spring system, defining the original length of the i-th spring as the average distance between the down-sampled point and its k nearest neighbor points; the formula for calculating the spring length is as follows:

[0063]

[0064] Among them, L i is the length of the ith spring, x n is the coordinate of the nth nearest neighbor, x i is the coordinate of the ith downsampling point, and M is the total number of downsampling points. The elastic potential energy loss function is used to simulate the point cloud as a spring system, and the density distribution between adjacent points of the point cloud is controlled to avoid local over-density or over-sparseness. In this way, the local and global density of the generated point cloud can be effectively guaranteed to be uniformly distributed, and the geometric precision and density balance of the point cloud can be improved.

[0065] Example 2

[0066] As an optimization of the aforementioned embodiment, this embodiment provides a detailed implementation process of a point cloud completion method based on shape prior and deep learning, combining multi-scale feature extraction, prefix embedding tree dictionary, semi-connected U-Net and composite loss function to solve the problems of uneven density and missing points in point cloud data.

[0067] Figure 2 The algorithm flow chart of the present invention is shown, and the overall steps of the point cloud completion method based on shape prior and deep learning are summarized. First, multi-scale feature extraction is performed on the input incomplete point cloud data. Secondly, the multi-scale features are refined using the prefix embedding tree dictionary to obtain refined features. Next, the multi-scale features and refined features are spliced ​​using a semi-connected U-Net, and a preliminary seed point cloud is obtained using multi-layer perceptron (MLP) regression. Subsequently, the preliminary completed seed point cloud is upsampled to generate a generated point cloud of the target resolution. Finally, the generated point cloud is constrained by a composite loss function (including chamfer distance, dual orthogonal constraint loss, and elastic potential energy loss) to ensure its density balance and geometric accuracy.

[0068] Figure 3 The specific implementation process diagram of the method described in the present invention is shown, and the operational details of each step are described in detail. First, the input incomplete point cloud is subjected to feature extraction, and global and local multi-scale features are extracted through the abstract set module and the Point Transformer module. Then, the geometric features are refined using the prefix embedding tree dictionary, and the multi-scale features and refined features are processed through the semi-connected U-Net. After that, the Upsample Transformer is used for upsampling during the implementation process to gradually improve the resolution of the point cloud. Finally, the generated point cloud is density-balanced and precision-optimized in combination with the composite loss function to generate a high-quality complete point cloud.

[0069] The detailed implementation process of the point cloud completion method based on shape prior and deep learning described in this embodiment includes the following specific steps:

[0070] S1. Feature extraction

[0071] like Figure 3 As shown in Figure 1, the feature extraction module includes a structure that combines the Set Abstract Module (SAM) and the Point Transformer. The incomplete point cloud is input into the feature extraction module, and multi-scale features of the incomplete point cloud at different resolutions are extracted. Feature extraction is divided into multiple levels:

[0072] (1) SAM downsamples the incomplete point cloud to obtain point cloud features at different resolutions;

[0073] (2) Point Transformer is based on the self-attention mechanism, including the Self Attention Layer (SAL). By designing a self-attention layer for point clouds, the geometric features of each point are enhanced to extract the global and local geometric features of the point cloud.

[0074] Specifically, multi-scale features and point cloud coordinates P i+1 It is obtained by nested calculation of two functions. Wherein, i represents the scale corresponding to the current feature and the incomplete point cloud coordinates. The multi-scale feature extraction module in the present invention has three scales, and the value range of i is {0, 1, 2}. The parameter selection of SAM and SAL at each scale corresponds to the feature dimension and the number of incomplete point clouds at the corresponding scale, and the formula is expressed as follows:

[0075]

[0076] It should be noted that in the first step of SAM, when i=0, By stacking abstract set modules and point cloud self-attention layers, multi-scale features can be obtained, as well as global features for the bottom layer of the semi-connected U-Net.

[0077] S2. Point cloud completion

[0078] After obtaining the multi-scale features of the incomplete point cloud, the prefix embedding tree dictionary and semi-connected U-Net are used to obtain the preliminary seed point cloud. Among them, the prefix embedding tree shape prior and the semi-connected U-Net are complementary. The prefix embedding tree shape prior is responsible for refining the multi-scale features, and uses the shape prior to make reasonable geometric inferences on the missing parts to generate refined features. Then, the semi-connected U-Net uses upsampling processing on this basis and introduces a semi-connected mechanism to solve the problem of uneven density of the seed point cloud, and further generate a more complete point cloud with balanced density.

[0079] S201. Introducing shape priors for prefix embedding trees

[0080] The shape prior of the prefix embedding tree is mainly used to help the model learn global and local geometric features from the input incomplete point cloud. This process is based on the global shape prior of the point cloud and the existing geometric knowledge. The multi-scale features are refined through the shape prior to obtain features with missing geometric information, laying the foundation for subsequent MLP regression to obtain seed point clouds, ensuring that the generated point cloud can reasonably infer the geometric structure of the missing part.

[0081] In order to ensure the inheritance of the shape prior learned from the prefix embedding tree dictionary in the subsequent decoding process of the semi-connected U-Net, a learnable embedding dictionary of the prefix embedding tree shape is designed, which includes two layers: the root dictionary and the leaf dictionary. Specifically:

[0082] (1) Root dictionary: The root dictionary combines the input global features with the rough geometric shapes through the cross-attention mechanism to refine the overall geometric structure of the point cloud. The input coarse root features are matched with the shape information in the root dictionary, and the similarity between the two is calculated based on the consistency graph to generate refined root features. The higher the similarity, the closer its shape is to the geometric features in the incomplete point cloud, and the more accurate the generated root features are.

[0083] (2) Leaf dictionary: The refined root features are further upsampled and become the input features of the leaf dictionary. The leaf dictionary is further refined based on the input coarse leaf features, and finally completes the detailed geometric features of the point cloud. These refined features combine multi-scale information and can better restore the local geometric shape of the point cloud.

[0084] The features refined by the prefix embedding tree dictionary will be cascaded with the multi-scale features obtained in step S1 to finally obtain the complete shape features.

[0085] like Figure 3 As shown, the global features extracted based on the abstract set module and Point Transformer First, upsample to coarse root features through one-dimensional deconvolution First, the root dictionary is refined to obtain the refined root features. Refined root features and previously extracted mid-resolution point cloud features The cascade is performed to preserve the original features at the global semantic level. The cascaded features are upsampled again through one-dimensional deconvolution to obtain coarse leaf features. That is, the input of the leaf dictionary. Coarse leaf features After the leaf dictionary is refined, the refined leaf features are output Refine leaf features No longer related to the original point cloud feature Instead of connecting, we directly obtain the complete point cloud P through MLP regression. seed .

[0086] The refinement process of the prefix embedding tree dictionary is divided into two stages. The first is the root dictionary, which combines the global features of the incomplete point cloud with the pre-stored geometric shapes in the dictionary through the cross-attention mechanism to refine and generate preliminary completed geometric features. Subsequently, these refined root features are further upsampled and input into the leaf dictionary, where they are further refined in combination with local detail features to generate the final completed geometric features. The entire process ensures a balance between global and local geometric features.

[0087] like Figure 4 As shown, the prefix embedding tree dictionary contains a root dictionary and a leaf dictionary, where Figure 4 (a) is the refinement process of the root dictionary. Figure 4 (b) is the refinement process of the leaf dictionary. By embedding the prefix into the tree dictionary, the shape completion process can obtain multi-scale inheritable prior knowledge guidance, thereby generating more accurate results. In the root dictionary refinement process, the global features Obtain coarse root features through one-dimensional deconvolution The coarse root feature will use the root dictionary Additional shape information in the generated refined root features The refinement process of the root dictionary is as follows Figure 4 As shown in (a), the input coarse root feature First, do a cross-attention operation with the root dictionary and obtain the root consistency map This step is to calculate the vectors and corresponding weights that are similar to the coarse root features in the root dictionary. The higher the similarity, the greater the attention weight obtained by the vector, which indicates that the vector has a high similarity with the feature corresponding to the incomplete point cloud, which is of guiding significance. The weighted root embedding is cascaded with the input coarse root features to finally output the refined root features. The cross attention operation function is defined as:

[0088]

[0089] Among them, Q and K represent feature matrices, d K Represents the dimension of K. Consistency Map ConsMap r The calculation process can be expressed as:

[0090]

[0091] in, and Represents two one-dimensional convolutional layers. By weighting the root dictionary based on the consistency graph, the refined root features can be obtained.

[0092]

[0093] Among them, MatMul represents matrix multiplication, and ψ is an equalization factor, which is set to 0.5 as a hyperparameter in this embodiment. The reason is that the addition of two features will cause the values ​​in the matrix to differ greatly from other parts of the module. The equalization factor can restore the values ​​in the matrix to the original level to prevent abnormal values.

[0094] Refined root features output from the root dictionary First, with the mid-resolution point cloud feature Cascade, then input the one-dimensional deconvolution layer for upsampling to obtain coarse leaf features Coarse leaf characteristics As the input of the leaf dictionary refinement process. The leaf dictionary refinement process is as follows Figure 4 As shown in (b), the input coarse leaf feature is first calculated by a multi-layer perceptron to obtain a leaf index n. The leaf index points to the target leaf dictionary of the current feature. The refinement process of the coarse leaf feature and the leaf dictionary is consistent with the refinement process of the coarse root feature and the root dictionary. In order to improve the representation ability of the vectors in the dictionary, this embodiment introduces a dual orthogonal constraint loss function, which can be calculated as follows:

[0095]

[0096] Among them, I r and I l represents the identity matrix corresponding to the root dictionary and the leaf dictionary, ε r represents the root dictionary matrix, Represents the i-th leaf dictionary matrix. During the entire prefix embedding tree dictionary refinement process, by calculating the loss and back propagation, the model will optimize the parameters according to the gradient. This optimization process is the fitting process of the prefix embedding tree dictionary to the complete geometric features in the training set.

[0097] S202, Improvements to Semi-Connected U-Net

[0098] In order to solve the problem of the lack of accurate modeling of missing point cloud geometric features and uneven density distribution of generated seed point clouds in traditional U-Net in point cloud completion tasks, the present invention proposes an improved semi-connected U-Net structure, which cancels the jump connection of the top layer of the traditional U-Net and only retains the connection of low-level global features. By weakening the jump connection of high-level explicit features, the semi-connected U-Net can avoid the phenomenon of excessive local density in the point cloud completion process, thereby generating a point cloud with more balanced density while maintaining global geometric features. This method effectively alleviates the problem of excessive explicitness of point-by-point features of incomplete point clouds during upsampling, ensuring that the generated point cloud has a uniform density distribution both locally and globally.

[0099] The decoding process of the semi-connected U-Net generates multi-scale features by stacking multiple one-dimensional deconvolution layers with different convolution kernel sizes and strides. These features are used to capture the local geometric information of incomplete point clouds at multiple scales.

[0100] After extracting multi-scale features in step S1, step S201 performs deconvolution upsampling and prefix embedding tree dictionary shape prior on the multi-scale features to obtain refined features. Then, enter the processing stage of semi-connected U-Net. Semi-connected U-Net encodes and decodes the multi-scale features obtained in step S1 and the refined features obtained in step S201, and splices them. Then, MLP regression is used to generate the coordinates and point-by-point features of the preliminary completed seed point cloud. The generation and upsampling of the seed point cloud are realized through a direct functional relationship, that is, the seed point cloud is passed as a parameter to the upsampling function for processing.

[0101] S3, upsampling

[0102] Upsample the seed point cloud through Upsample Transformer. Set the upsampling ratio to [1,4,4] and gradually upsample the seed point cloud to the target resolution.

[0103] Specifically, the seed point cloud generated in step S2 contains sparse, evenly distributed point cloud coordinates. These seed point clouds are then input into the Upsample Transformer module, which gradually refines the seed point cloud into a generated point cloud of the target resolution through successive upsampling. In each upsampling operation, the Upsample Transformer processes the geometric features of the seed point cloud and combines the shape prior information in the prefix embedding tree dictionary.

[0104] S4. Loss Function and Optimization

[0105] In order to ensure that the generated point cloud has high density balance and geometric accuracy, this embodiment uses a variety of loss functions, including the classic chamfer distance, dual orthogonal constraint loss, and elastic potential energy loss designed in this embodiment, to constrain the training process. The specific contents are as follows:

[0106] (1) Chamfer Distance (CD): Chamfer distance is used to measure the geometric chamfer distance between the completed point cloud and the real point cloud. By calculating the distance between each point of the completed point cloud and the target point cloud, it ensures that the completed point cloud can be as close to the target shape as possible. sparse Used to measure the chamfer distance between the seed point cloud and the real point cloud. dense Indicates the chamfer distance between the final completed point cloud and the real point cloud.

[0107] (2) Doubly orthogonal constraint loss: To ensure that the features in the prefix embedding tree dictionary are not redundant, the doubly orthogonal constraint loss optimizes the multi-scale feature vectors in the root dictionary and the leaf dictionary. and are the dual orthogonal constraint losses of the root dictionary and the leaf dictionary respectively; specifically, It is used to constrain the feature representation in the root dictionary to ensure that each feature has sufficient discrimination and no redundant information exists; It is used to constrain the feature representation in the leaf dictionary, ensure that the multi-level geometric refinement features in the leaf dictionary are not repeated, and improve the efficiency and accuracy of dictionary learning.

[0108] (3) Elastic potential energy loss:

[0109] During the upsampling process, the generated point cloud may be too concentrated around the seed point cloud, resulting in uneven local density. To solve this problem, this embodiment designs elastic potential energy loss, which constrains the position of the point cloud by simulating the generated point cloud as a spring system to ensure its uniform distribution locally and globally. Figure 5 As shown, the specific process is as follows:

[0110] First, use FPS (Furthest Distance Sampling) to downsample the generated complete point cloud to M points. Then, find the k nearest neighbors of each downsampled point (k is twice the number of upsampled points). The downsampled point and its k nearest neighbors are regarded as a spring system, and the original length of the i-th spring is defined as the average distance between the i-th downsampled point and its K nearest neighbors. In this way, the density distribution of the point cloud generated for point cloud completion can be controlled. The formula for calculating the spring length is as follows:

[0111]

[0112] Among them, L i is the length of the ith spring, x n is the coordinate of the nth nearest neighbor, x i is the coordinate of the ith downsampling point; M is the total number of downsampling points, and each downsampling point corresponds to a spring. The downsampling points come from the completed point cloud. Based on Hooke's law, the deformation of the spring and the corresponding elastic potential energy are calculated. Through this spring system model, the position of the generated point cloud can be effectively constrained, which can prevent the problem of local over-density and ensure the uniform distribution of the point cloud globally and locally. The elastic potential energy is proportional to the square of the deformation, and the formula is:

[0113]

[0114] Among them, E irepresents the elastic potential energy of the i-th spring, and ξ is 0.5 times the elastic coefficient. The constant part can be used as a hyperparameter, so ξ can be directly regarded as a hyperparameter. In all experiments, the elastic coefficient is set to 10.

[0115] Next, the sum of the potential energies of all springs is set as the elastic potential energy loss function, which is:

[0116]

[0117] in, By introducing this constraint, the problem of the generated point cloud being too concentrated around the seed point cloud can be effectively alleviated, ensuring that the local and global density of the generated point cloud is uniform.

[0118] Finally, the composite loss function is established by integrating the chamfer distance, dual orthogonal constraint loss and elastic potential energy loss as follows:

[0119]

[0120] in, Represents the value of the composite loss function; CD sparse Represents the chamfer distance between the seed point cloud and the real point cloud, CD dense Indicates the chamfer distance between the final generated point cloud and the real point cloud; and are the dual orthogonal constraint losses for the root dictionary and the leaf dictionary, respectively; is the elastic potential energy loss; α, β and γ are hyperparameters, which are set to 1, 1, and 10 respectively in all experiments.

[0121] Through the design and optimization of the above loss function, it is ensured that the generated point cloud is close to the real point cloud in geometry, while ensuring that the local and global density of the completed point cloud is balanced.

[0122] exist Figure 4 In order to reflect the density of the point cloud, cyan is used to represent areas with lower density and magenta is used to represent areas with higher density. When the density of the point cloud is relatively uniform, the point cloud will appear in the same color as a whole, such as magenta or cyan at the same time.

[0123] Example 3

[0124] In order to verify the effectiveness of this method, this embodiment is verified on the PCN dataset. The PCN dataset is a subset of the ShapeNet dataset, containing 8 types of objects and a total of 30,974 point clouds. Based on the experience of previous studies, the dataset is divided into: a training set of 28,974 items, a test set of 1,200 items, and a validation set of 800 items.

[0125] In order to ensure the consistency of the input residual cloud data, the residual cloud is resampled to a standard size of 2048 points. The upsampling ratio is consistent with the previous method, and the upsampling scale is set to {1, 4, 4}. In addition, the following parameters are set in the experiment: the dimension of the root dictionary is set to 256, the dimension of the leaf dictionary is set to 128; the number of root dictionary nodes N is set to 10; the training round is set to 400, the batch size is set to 12; the Adam optimizer is used for training, and the learning rate is set to 4×10 -4 , decaying by 0.8 times every 20 rounds; the experimental environment is Nvidia GeForce RTX 4090 graphics card (24GB GPU).

[0126] The proposed method is compared with nine existing methods. The other methods in the comparative experiment are represented by their original English names. Table 1 shows the quantitative results of the comparative experiment on the PCN dataset. It can be seen that the proposed method ranks second in CDL1 and ranks first in DCD and F1 value. In particular, the proposed method is 1.1×10 higher than ODGNet in DCD. -2 This shows that this method has achieved significant results when considering the density distribution of point clouds.

[0127] Table 1: Quantitative results of PCN dataset comparison test (CD-L1×10 -3 )

[0128]

[0129] Among them, the smaller the values ​​of CD and DCD and the higher the F1 score, the better the effect of the point cloud completed by this method in density distribution.

[0130] Figure 6 The visualization results of several methods are shown, including Snowflakenet, SeedFormer, ODGNet and this method. The colors in the figure are used to represent the density of the point cloud, cyan represents areas with lower density, and magenta represents areas with higher density. When the density of the point cloud is uniform, the color of the entire point cloud will be consistent, that is, all areas are cyan or magenta. It can be seen that the point cloud generated by this method is more uniform in density distribution, indicating that in areas that do not contain input points, this method can better restore the geometric information of the missing part. Especially in the part of the point cloud that is not included in the input point, the shape completion effect of this method is significantly more complete than that of other methods. In contrast, other methods have the problem of uneven density, which is specifically manifested in the phenomenon of density polarization (i.e., cyan and magenta are mixed) in the point cloud of the same object, which indicates that the density distribution of the point cloud is uneven, mostly concentrated around the input point, while the density of the area to be completed is low.

[0131] Through experimental results and visual comparison, this method significantly outperforms other existing methods in point cloud completion tasks, especially in density balance and shape restoration. This shows that this method has made a breakthrough in processing the density distribution of point clouds.

[0132] Example 4

[0133] In order to verify the performance of this method in real-world datasets, especially in practical applications such as cultural relics restoration, architectural surveying and mapping, and autonomous driving. This embodiment is further optimized on the basis of the previous embodiment and uses the KITTI dataset for testing. Compared with the virtual point cloud data in the ShapeNet dataset, the KITTI dataset contains point clouds in a real scanning environment, which provides a basis for testing the effectiveness of the point cloud completion method in real-world applications.

[0134] In this experiment, we only focus on objects marked as cars in the KITTI dataset and extract the point cloud data in their bounding boxes for completion. A total of 2483 incomplete point clouds were tested, which have no ground truth. Therefore, the model was pre-trained using data from the PCN dataset that also belongs to the Cars category. Then, the following two indicators are used to evaluate the completion effect of the model:

[0135] Fidelity: used to evaluate the similarity between the completed point cloud and the original incomplete point cloud.

[0136] MMD (Maximum Mean Difference): It is used to measure the difference between the generated point cloud and the real point cloud distribution.

[0137] In the experiment, the dimension of the root dictionary is set to 128, the dimension of the leaf dictionary is set to 64, and the number of root dictionary nodes N is set to 8.

[0138] In order to verify the effectiveness of this method, it is compared with 8 existing point cloud completion methods. The other methods in the comparative experiment are represented by their original English names. Table 2 shows the quantitative experimental results of each method. This method achieves the best result (SOTA) on the MMD indicator, which is 14% higher than other methods, indicating that when processing real data, the generated point cloud distribution of this method is closer to the real distribution.

[0139] Table 2: Quantization results of KITTI dataset comparison experiment (FD×10 3 ,MMD×10 3 )

[0140]

[0141] Among them, the smaller the values ​​of FD and MMD, the better the point cloud completion effect achieved by this method.

[0142] Figure 7 The visualization comparison results of this method and the PoinTr method on the KITTI dataset are shown. As can be seen from the figure, this method can complete the point cloud more smoothly and generate less noise. In contrast, the point cloud generated by PoinTr has more noise points and the completed shape is also rough. This method not only generates a more uniform point cloud density, but also a more complete shape, indicating that it has a strong advantage in processing real-world point cloud data.

[0143] In general, through the test of KITTI dataset, this method not only performs well on virtual point cloud data (such as ShapeNet), but also can achieve smooth, low-noise high-quality completion effect in real point cloud data. This proves that this method has practical application value in the fields of cultural relics restoration, architectural surveying and mapping, and autonomous driving, and has significant advantages over existing technologies.

[0144] Since the ShapeNet dataset is a virtually generated point cloud, it may differ from the real-world scanned point cloud dataset. In order to test the effect of the completion method on real datasets and its application in autonomous driving, the KITTI dataset was further used to test this method. Following the previous method, we only focus on the points in the bounding box of objects marked as cars. There are a total of 2483 incomplete point clouds and no ground truth. The model was pre-trained using the Cars category in the PCN dataset, and evaluated using Fidelity and MMD. The dimension of the root dictionary is set to 128, and the dimension of the leaf dictionary is set to 64. For the number of root dictionary nodes N, it is set to 8. This method is compared with 8 existing methods. As shown in Table 2, this method achieved SOTA results on MMD, with a relative improvement of 14%. Figure 7 As shown in Figure 3, this method can also complete the point cloud of the real world. Compared with PoinTr, this method is smoother and has less noise.

[0145] Example 5

[0146] As a preferred embodiment of the aforementioned embodiments, this embodiment provides a device for executing a point cloud completion method based on shape prior and deep learning.

[0147] The device includes at least one processor and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a point cloud completion method based on shape prior and deep learning.

[0148] The device includes multiple functional modules, each of which corresponds to a different step in the method. Specifically, it includes the following components:

[0149] Feature extraction module: This module is used to extract multi-scale features from the input incomplete point cloud. It uses the abstraction set module and the Point Transformer module to capture the multi-scale features of the point cloud, thereby generating preliminary global and local geometric features.

[0150] Prefix Embedding Tree Module: This module further refines the multi-scale features using the prefix embedding tree dictionary. By coordinating the root dictionary with the leaf dictionary, it ensures that the refined features include the geometric structure of the missing parts of the input point cloud.

[0151] Semi-connected U-Net processing module: This module processes the multi-scale features generated by the feature extraction module and the refined features generated by the prefix embedding tree module based on the semi-connected U-Net to generate a preliminary seed point cloud. By retaining only low-level global feature jump connections, the influence of high-level local geometric features on the density distribution of the seed point cloud is avoided, thus improving the balance of point cloud generation.

[0152] Upsampling module: This module uses the Upsample Transformer to upsample the refined seed point cloud one by one, gradually refines it, and obtains the generated point cloud that reaches the target resolution. The upsampling module can maintain the balance of point cloud density and gradually improve the point cloud resolution.

[0153] Composite loss constraint module: It is used to constrain the generated point cloud through a composite loss function, which includes chamfer distance, dual orthogonal constraint loss and elastic potential energy loss. It combines chamfer distance, dual orthogonal constraint loss and elastic potential energy loss to ensure that the generated point cloud is evenly distributed in global and local density, improving geometric accuracy.

[0154] Through the coordinated operation of various modules, the device can automatically complete the point cloud completion task and generate a complete point cloud with accurate geometry and balanced density.

[0155] Those skilled in the art will appreciate that the embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware.

[0156] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0157] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A point cloud completion method based on shape prior and deep learning, characterized in that: The following steps are involved: S1. Extract features from the input incomplete point cloud data to obtain multi-scale features of the incomplete point cloud; S2, use the prefix embedding tree to refine the features extracted in S1 to obtain refined features; S3, processing the multi-scale features described in S1 and the refined features described in S2 through a semi-connected U-Net, and obtaining a preliminary completed seed point cloud through multi-layer perceptron regression; S4, upsampling the initially completed seed point cloud to obtain a generated point cloud reaching a target resolution; S5. Use a composite loss function to constrain the point cloud generated in step S4 to obtain a complete point cloud with balanced density.

2. The point cloud completion method based on shape prior and deep learning according to claim 1, characterized in that: Step S1 specifically includes using an abstract set module and a Point Transformer to extract multi-scale features of an incomplete point cloud at different resolutions; the multi-scale features include global geometric features of the incomplete point cloud.

3. The point cloud completion method based on shape prior and deep learning according to claim 2, characterized in that: The prefix embedding tree dictionary in step S2 includes a root dictionary and a leaf dictionary. The root dictionary is first used to match the global geometric features to generate refined root features, and then the leaf dictionary is used to refine and generate local geometric features.

4. The point cloud completion method based on shape prior and deep learning according to claim 3, characterized in that: The semi-connected U-Net in step S3 processes the features by encoding and decoding the multi-scale features in S1 and the refined features in S2, and concatenating them; in the decoding stage, only the low-level global features of the multi-scale features in the encoder are jump-connected to the upsampling layer of the decoder.

5. The point cloud completion method based on shape prior and deep learning according to claim 4, characterized in that: The upsampling in step S4 is to upsample the initially completed seed point cloud one by one through the UpsampleTransformer module, and gradually refine it to obtain a generated point cloud that reaches the target resolution.

6. The point cloud completion method based on shape prior and deep learning according to claim 5, characterized in that: The composite loss in step S5 includes functions including chamfer distance, dual orthogonal constraint loss, and elastic potential energy loss.

7. The point cloud completion method based on shape prior and deep learning according to claim 6, characterized in that: The formula of the composite constraint function in step S5 is as follows: in, Represents the composite loss function value; CD sparse Represents the chamfer distance between the seed point cloud and the real point cloud, CD dense Indicates the chamfer distance between the final generated point cloud and the real point cloud; and are the dual orthogonal constraint losses for the root dictionary and the leaf dictionary respectively; is the elastic potential energy loss; α, β and γ are hyperparameters.

8. The point cloud completion method based on shape prior and deep learning according to claim 6, characterized in that: The elastic potential energy loss in step S5 is calculated by simulating the seed point cloud as a spring system, defining the original length of the i-th spring as the average distance between the i-th down-sampled point and its k nearest neighbor points; the formula for calculating the spring length is as follows: Among them, L i is the length of the ith spring, x n is the coordinate of the nth nearest neighbor, x i is the coordinate of the i-th downsampling point, and M is the total number of downsampling points.

9. A device for a point cloud completion method based on shape prior and deep learning, characterized in that: The device includes at least one processor and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor is enabled to execute the method described in claims 1-8.

10. A device for a point cloud completion method based on shape prior and deep learning, characterized in that: The device comprises: A feature extraction module is used to extract multi-scale features from the input incomplete point cloud; The prefix embedding tree module uses the prefix embedding tree dictionary to further refine the geometric information of multi-scale features; A semi-connected U-Net processing module is used to process the multi-scale features generated by the feature extraction module and the refined features generated by the prefix embedding tree module based on the semi-connected U-Net to generate a preliminary seed point cloud; The upsampling module uses the Upsample Transformer to upsample the refined seed point cloud one by one, and gradually refines it to obtain the generated point cloud that reaches the target resolution; The composite loss constraint module is used to constrain the generated point cloud through a composite loss function. The composite loss includes chamfer distance, dual orthogonal constraint loss and elastic potential energy loss.

Images