A point cloud completion method based on stacked grid transformation

Abstract

The application discloses a point cloud completion method based on stacked grid transformation and belongs to the technical field of laser scanning data processing. The application comprises the following steps: acquiring an original point cloud, fixing the point cloud shape through a sampling algorithm, obtaining feature mapping in a high-dimensional feature space after a series of high-dimensional space transformations, and down-sampling the feature mapping into feature coding; and then acquiring a rough point cloud by using a stacked grid transformation decoder; and performing a three-dimensional space expansion operation on the rough point cloud, averaging the point density of the point cloud sampling space, and obtaining a complete point cloud result. The application can obtain the set features of the three-dimensional space through edge convolution, cover higher latitude coupling context information by using style coding, and enhance the robustness of feature extraction in processing complex scenes; and the stacked point cloud reconstruction based on the coding-decoding structure of the deep grid transformation can obtain higher-precision information and geometric details in the process of completing the point cloud.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of laser scanning data processing, and particularly relates to a point cloud completion method based on stacked grid transformation. BACKGROUND

[0002] With the development of software and hardware revolution, point cloud data has become an indispensable element for building a future intelligent society. A series of point data-based datasets have emerged, such as ScanNet which samples RGB-D video data, and ModelNet and Kitti which sample raw point data to help researchers conduct meaningful experiments. Due to these datasets, various point cloud algorithms have also been implemented. However, during the point cloud collection process, some factors such as uneven distribution and local shape caused by limited view, sensor resolution, etc., and the negligence of staff inevitably lead to accuracy loss when using these data for classification or segmentation. Therefore, post-processing methods such as completion, denoising and resampling need to be proposed in the upstream task.

[0003] While deep learning has made breakthroughs in the field of 3D processing, many deep learning-based methods, such as voxel-based and pointnet-based encoding-decoding architecture methods, can achieve certain results in the point cloud completion task. However, these methods cannot extract reasonable features and can only reconstruct shape details. The fundamental reason is that when constructing a complete point cloud, the model needs to have the following necessary conditions: first, it must be able to infer the incomplete part of the point cloud. Secondly, detailed shape needs to be completed during the reconstruction process. And the model must be as small and effective as possible and be directly deployed on the collection device. The bad news is that current work cannot completely solve these problems, either completing rough shape details or requiring a large amount of computing resources. Therefore, a method that can take into account and solve the above problems is needed. SUMMARY

[0004] Therefore, the application provides a point cloud completion method based on stacked grid transformation, which aims to solve the problem that the prior art cannot complete the high-precision point cloud completion task through effective feature extraction.

[0005] The technical scheme adopted by the application is as follows:

[0006] A point cloud completion method based on stacked grid transformation, the method comprising the following steps:

[0007] Step S1, acquiring point cloud data and performing data preprocessing, including normalization (reshaping the point cloud data to the target input size, such as N), formatting and other operations;

[0008] Step S2, training an encoder, which obtains a feature mapping in a high-dimensional feature space through a high-dimensional space transformation of a batch of point cloud data, and down-samples the feature mapping to obtain a feature code;

[0009] Step S3, performing affine transformation on the feature code, and performing parallel up-sampling on the feature code based on a stacked grid transformation decoder to obtain a completed rough point cloud;

[0010] Step S4, performing a three-dimensional space expansion operation on the rough point cloud through an optimization module to average the point density of the point cloud sampling space, and obtaining a complete point cloud result.

[0011] Preferably, in step S1, when the obtained point cloud format does not conform to the input format, a point cloud sampling algorithm is used to ensure consistency.

[0012] Preferably, in step S2, the high-dimensional space transformation includes: a one-dimensional convolution-based point feature space transformation, a two-dimensional convolution-based image feature space transformation, and a pooling operation based on aggregated features, and the point cloud is subjected to point dimension and image dimension feature extraction according to the high-dimensional space transformation to obtain highly aggregated feature codes.

[0013] Preferably, step S2 specifically includes:

[0014] The one-dimensional convolution is performed on the initial point cloud data with a dimension of N*3 in the point number direction, so that the projection in the point coordinate direction is used as a representation of the spatial relationship to obtain intermediate features;

[0015] A KNN clustering algorithm (K-Nearest Neighbor) is used to perform point clustering processing on the intermediate features to obtain a clustering mapping set of point number size, and a two-dimensional convolution filter is used to process the clustering mapping set to obtain a high-dimensional feature relationship;

[0016] The high-dimensional feature relationship is subjected to a dimension reduction pooling operation to map the point coordinate space to a linear space to obtain low-dimensional and highly aggregated feature codes.

[0017] Preferably, in step S3, the affine transformation of the feature code is specifically:

[0018] The feature code is subjected to a high-dimensional mapping in a multi-scale feature space, and the dimension size of the high-dimensional mapping is matched with the size of the input point cloud after the channel stacking of the up-sampling.

[0019] The stacked grid transformation decoder includes: an identity mapping transformation based on one-dimensional convolution and feature compression based on an attention mechanism.

[0020] Preferably, in step S4, the optimization module channel splices the missing point cloud with the coarse point cloud, and the point overlap problem caused in this process can be solved by the Minimal Density Sampling algorithm. Secondly, channel division is performed, and the point vector set after channel division is used as the basis for multi-scale feature mapping. Multi-scale feature mapping is performed in the autoencoder based on one-dimensional convolution, the mapped multi-scale coordinate point set is obtained, and finally, reverse channel division is performed to stack the point sets one by one in the spatial dimension to obtain the final complete point cloud result.

[0021] That is, the optimization module channel splices the missing point cloud with the coarse point cloud; then performs channel division (obtains multi-scale feature vectors through channel-level segmentation), uses the point vector set after channel division as the basis for multi-scale feature mapping, performs multi-scale feature mapping in the autoencoder based on one-dimensional convolution, obtains the mapped multi-scale coordinate point set, and finally performs reverse channel division to stack the point sets one by one in the spatial dimension to obtain the final complete point cloud result.

[0022] Preferably, the encoder algorithm is:

[0023]

[0024] wherein C() represents a convolution module constructed based on edge convolution, MLP() represents a high-dimensional feature space mapping based on affine transformation, B() represents a convolution module constructed based on residual convolution, R represents an integrated encoding block, the subscript i represents the identification of the encoding block for constructing the encoder, i.e., the layer number of the encoder, f i,j represents the jth point feature of the ith layer, N represents the number of point features, f i,N represents the N point features of the ith layer.

[0025] Preferably, when training the encoder, the stacked grid transformation decoder, and the optimization module, the loss function used is:

[0026]

[0027] l rec = λ r D emd (p refine ,p gt )+ λ c D emd (p coarse ,p gt )

[0028] wherein S1 and S2 represent two three-dimensional point cloud sets, Φ represents the mapping relationship between S1 and S2, Φ(x) represents the mapping result of any point x in the point cloud set S1, l rec represents the overall loss, and λr , λ c respectively represent the weights of loss D emd (p refine , p gt ) and D emd (p coarse , p gt ), p c o arse represents the coarse point cloud output by the decoder, p refine represents the point cloud output by the optimization module, and p gt represents the real data point cloud.

[0029] The technical solutions provided by the present application at least bring the following beneficial effects:

[0030] (1) By edge convolution, the set features of the three-dimensional space are obtained, and higher latitude coupled context information is covered by using style encoding, thereby enhancing the robustness of feature extraction in processing complex scenes.

[0031] (2) Based on the stacked point cloud reconstruction, the encoding-decoding structure of the depth grid transformation can obtain higher precision information and geometric details in the process of completing the point cloud. BRIEF DESCRIPTION OF DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0033] Figure 1 A flowchart of a point cloud completion method based on stacked grid transformation is provided for the embodiments of the present application.

[0034] Figure 2 A functional module schematic diagram of the first embodiment of the point cloud completion method, device and medium based on stacked style encoding and edge convolution of the present application.

[0035] Figure 3 A flowchart of one of the embodiments of the point cloud completion method, device and medium based on stacked style encoding and edge convolution of the present application. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0037] The embodiment of the application provides a point cloud completion method based on stacked grid transformation, and the encoder is improved to enhance feature extraction, the feature extractor of PointNet is applied to deep residual connection, and deep potential features can be efficiently extracted and reserved; in the feature extraction process, the KNN nearest point sampling algorithm is used to sample the point cloud feature space, the point cloud feature space is regarded as a feature, and a two-dimensional image convolution algorithm similar to an attention mechanism is used to extract the point cloud feature; the output of the encoder is decoded into a stacked grid transformation network based on the stylegan (style encoding) feature; the output bottleneck of the traditional FoldingNet encoder is improved for point cloud feature representation; finally, a refinement module is used to make the results more evenly distributed.

[0038] As a possible implementation manner, the embodiment of the application provides a point cloud completion method based on stacked grid transformation, which comprises the following steps:

[0039] Step S10: acquiring an original point cloud and fixing the point cloud shape through a sampling algorithm.

[0040] It should be understood that the point cloud is a mass point set for expressing the target space distribution and the target surface characteristics in the same space reference system, the point cloud obtained according to the laser measurement principle includes three-dimensional coordinates and laser reflection intensity, the point cloud obtained according to the photographic measurement principle includes three-dimensional coordinates and color information, and the point cloud obtained by combining the laser measurement and the photographic measurement principle includes three-dimensional coordinates, laser reflection intensity and color information. After obtaining the space coordinates of each sampling point of the object surface, a point set is obtained, which is called a point cloud.

[0041] Step S20: training an encoder to obtain a feature mapping in a high-dimensional feature space after a series of high-dimensional space transformations, and down-sampling the feature mapping to obtain a feature code. Through affine transformation of the feature code, the decoder is used to up-sample the feature code, and a rough point cloud is obtained. That is, the up-sampled feature code obtained based on the decoder is the rough point cloud.

[0042] In the embodiment of the application, the encoding algorithm of the encoder is represented as:

[0043]

[0044] Wherein, C represents a convolution module constructed based on edge convolution, MLP represents a multi-layer perception machine, that is, a high-dimensional feature space mapping based on affine transformation, B represents a convolution module constructed based on residual convolution, R represents an integrated encoding block, i represents an encoding block identifier for constructing the encoder, f i,N represents N-point features of the i-th layer.

[0045] Step S30: Perform an inflation operation in the three-dimensional space on the rough point cloud, average the point density of the point cloud sampling space, and obtain a complete point cloud result.

[0046] In the embodiment of the present application, the algorithm of the loss function of the network is represented as:

[0047]

[0048] l rec =λ r D emd (p refine ,p gt ) + λ c D emd (p coarse ,p gt )

[0049] wherein S1 and S2 represent two three-dimensional point cloud sets, represents the mapping relationship between S1 and S2, represents the mapping result of any point x in the point cloud set S1, l rec represents the overall loss, represents the weight value predefined by the loss function, and λ r , λ c represent the weights of the losses D emd (p refine ,p gt ) and D emd (p coarse ,p gt ), respectively. Usually, the sum of λ r and λ c is 1, p c o arse represents the rough point cloud output by the decoder, p refine represents the optimized point cloud, and p gt represents the real data point cloud.

[0050] In the embodiment, first, the point cloud completion training based on the point cloud data of the training set is performed on the three-dimensional point cloud reconstruction network built based on residual convolution and edge convolution; second, the rough point cloud after reconstruction is inflated by using a sampling algorithm. Finally, the missing point cloud is reconstructed at the pixel level based on the trained model.

[0051] In the embodiment, the edge convolution is used to obtain the set features of the three-dimensional space, and the style encoding is used to cover higher latitude coupling context information, thereby enhancing the robustness of feature extraction in processing complex scenes. Through the encoding-decoding structure of the deep grid transformation, the stacked point cloud reconstruction can obtain higher precision information and geometric details in the process of completing the point cloud.

[0052] As Figure 2As shown, as a possible implementation, the overall network model of the point cloud completion method based on stacked mesh transformation provided in this embodiment of the invention includes: an encoder (Res-E), a decoder (stacked stylefolding), and an optimization module (Stacked Refine Block). First, the missing point cloud (Partial Cloud) undergoes data processing, transforming it into a point set vector of size Nx3 (N represents the number of points in the point cloud), which is then fed into the encoder called Res-E. This encoder focuses on cross-channel feature processing through residual connections and geometric feature processing using edge convolution. That is, the Res-E encoder includes multiple encoding blocks (Res-E Blocks), thus forming a multi-layered integrated encoding block. In each integrated encoding block, the input data passes through the encoding block (Res-E Block) to obtain the output of the encoding block (i.e., C). i (f i,N The output of the MLP is then fed into a multilayer perceptron (MLP), and the input of each MLP also includes the input of the currently integrated coding block. The output of the MLP is added to the output of the (Res-E Block) to obtain the output of the currently integrated coding block (i.e., R). i The Res-E Block is a convolutional module constructed based on edge convolutions. For example, this Res-E Block includes two branches: one branch includes one edge convolution, and the other branch includes three cascaded edge convolutions. Multiple integrated encoding blocks are fed into a Shared MLP (to reduce the network's training parameters and achieve a weight-sharing mechanism similar to CNNs), and then pooled to obtain the encoder's output (feature encoding). That is, features are pooled and compressed to obtain feature encodings. Next, the feature encodings are fed into a decoder called Stacked Style Folding for decoding. The feature encodings are decoded through stacked folding operations based on grid transformation and style encoding. Finally, the obtained coarse point cloud is further optimized by a module called Stacked Refine Block. The coarse points output by the decoder and the refined points output by the optimization module are then used to calculate the loss function on the ground truth data to back-update the model parameters, thereby training the model.

[0053] Furthermore, embodiments of the present invention also propose a point cloud completion device based on stacked mesh transformation, such as... Figure 3 As shown, the point cloud completion device includes: a point cloud acquisition and processing module, an encoder module for style encoding and edge convolution, a decoder module for stacked mesh transformation, and an optimization module for stacked high-precision reconstruction.

[0054] The point cloud acquisition and processing module is configured to read in the point cloud and call a series of processing operations to normalize the point cloud.

[0055] The style encoding and edge convolution encoder module is configured to perform high-dimensional space transformation on the point cloud, acquire feature mapping, and convert the feature mapping into feature encoding.

[0056] The stacked grid transformation decoder module is configured to perform parallel upsampling on the feature encoding to obtain a coarse but complete point cloud.

[0057] The stacked high-precision reconstruction optimization module is configured to perform density normalization on the coarse point cloud in three-dimensional space to obtain a complete point cloud result.

[0058] It should be noted that the device provided in the above embodiments is only used to illustrate the division of the above functional modules, and in actual applications, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the above-described functions. In addition, the device and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process is described in detail in the method embodiments, which will not be described here.

[0059] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

[0060] The above only describes some embodiments of the present application. For those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of protection of the present application.

Claims

1. A point cloud completion method based on stacked mesh transformation, characterized in that, Includes the following steps: Step S1: Acquire point cloud data and perform data preprocessing; Step S2: Train an encoder that takes a batch of point cloud data as input, transforms it in a high-dimensional space to obtain a feature map in a high-dimensional feature space, and downsamples it into a feature code. Step S3: Perform affine transformation on the feature encoding, and upsample the feature encoding in parallel based on the stacked grid transformation decoder to obtain the completed coarse point cloud; Step S4: The coarse point cloud is expanded in three-dimensional space by the optimization module to average the point density of the point cloud sampling space and obtain the complete point cloud result. In step S2, the high-dimensional space transformation includes: point feature space transformation based on one-dimensional convolution, image feature space transformation based on two-dimensional convolution, pooling operation based on aggregated features, and feature extraction of point dimension and image dimension of point cloud according to the high-dimensional space transformation to obtain highly aggregated feature encoding. Step S2 specifically includes: For the initial point cloud data of dimension N×3, a one-dimensional convolution is performed in the direction of the number of points, and the projection of the point onto the direction of the point coordinates is used as a representation of the spatial relationship to obtain intermediate features; The KNN clustering algorithm is used to perform point clustering on the intermediate features to obtain a clustering mapping set of a certain number of points. Then, a two-dimensional convolutional filter is applied to the clustering mapping set to obtain high-dimensional feature relationships. The obtained high-dimensional feature relationships are subjected to a dimensionality reduction pooling operation to map the point coordinate space into a linear space, thereby obtaining a low-dimensional, highly aggregated feature encoding. In step S3, the affine transformation of the feature encoding is specifically performed as follows: The feature encoding is mapped in a high-dimensional space at multiple scales. The dimension of this high-dimensional mapping is matched with the size of the input point cloud after upsampling and channel stacking. The stacked grid transform decoder includes: identity mapping transformation based on one-dimensional convolution and feature compression based on attention mechanism; The encoder is: Where C() represents a convolutional module constructed based on edge convolution, MLP() represents a high-dimensional feature space mapping based on affine transformation, B() represents a convolutional module constructed based on residual convolution, R represents an ensemble coding block, and the subscript i represents the coding block identifier used to construct the encoder, i.e., the layer number of the encoder. Let N represent the feature of the j-th point in the i-th layer, and let N represent the number of feature points. This represents the N-point feature of the i-th layer.

2. The method as described in claim 1, characterized in that, In step S1, when the obtained point cloud format does not conform to the input format, the input format is normalized by the point cloud sampling algorithm.

3. The method as described in claim 1, characterized in that, The loss function used when training the encoder, the stacked mesh transform decoder, and the optimization module is: l rec λ r D emd ( p refine ,p gt )+λ c D emd ( p coarse ,p gt ) Where S1 and S2 represent two 3D point cloud sets, Φ represents the mapping relationship between S1 and S2, and Φ(x) represents the mapping result of any point x in point cloud set S1. rec λ represents the overall loss. r , λ c Representing the loss D respectively emd (p refine ,p gt ) and D emd (p coarse ,p gt The weight of p) coarse p represents the coarse point cloud output by the decoder. refine p represents the point cloud output by the optimization module. gt This represents a real point cloud.

Images