A density-preserving point cloud compression method based on deep learning
Through deep learning-based encoder and decoder, combining entropy coding and sub-point convolution, the problem of density information loss in point cloud compression is solved, and high-quality point cloud reconstruction and performance improvement is achieved.
Patent Information
- Application Number
- CN202111597535.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-24
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2041-12-24
AI Technical Summary
The existing point cloud compression algorithm fails to effectively retain the density information of the original point cloud during the compression process, resulting in a large difference between the decompressed point cloud density and the original point cloud density, affecting the performance of downstream tasks such as semantic segmentation.
Using a deep learning-based method, point cloud features are extracted by constructing encoder, entropy encoder and decoder, density embedding, position embedding and ancestor embedding, and compression and subpoint convolution are performed through entropy encoder for upsampling, to construct The loss function is used to train the point cloud compression model to ensure that density information is retained during compression.
It realizes the preservation of original point cloud density information during compression, improves the reconstruction quality, avoids the problem of excessive differences in density information, and improves the overall performance of point cloud compression.
Smart Images

Figure CN114286103B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of three-dimensional computer vision, and in particular relates to a point cloud compression method based on deep learning that can retain the original point cloud density. Background Art
[0002] Point cloud is widely used in fields such as autonomous driving, robotics, and physical simulation. It is one of the most important and widespread 3D representations. With the rapid development of 3D scanning technology, people can now effectively capture complex geometric information through fine-grained point clouds. Therefore, compression algorithms are crucial for the storage and transmission of point cloud data. In particular, in order to obtain a higher compression rate, researchers have been focusing on lossy point cloud compression algorithms and thinking about a key question: Which features of point clouds should be retained under limited bit rate conditions?
[0003] In addition to the global geometric information of the point cloud, attention should also be paid to the preservation of local density. The reasons are as follows: First, preserving density usually reconstructs fewer outliers, resulting in smaller reconstruction errors. Second, the point cloud obtained from the actual scan rarely presents a uniform distribution, so losing local density means losing important original features, such as scanning resolution. Then, the point cloud is usually processed or simplified to have complex geometric shapes in the area of interest, such as faces, hands, etc., and preserving density during compression means that these areas will be better reconstructed. Finally, if the density of the decompressed point cloud is very different from the original point cloud density, downstream tasks such as semantic segmentation may be affected.
[0004] Traditional point cloud compression algorithms usually rely on octree or KD tree structures to improve storage efficiency. Inspired by the great success of deep learning technology in point cloud processing and image compression, researchers have begun to focus on lossy point cloud compression algorithms based on deep learning.
[0005] In order to solve the problem of unstructured, irregular and disordered point clouds, some point cloud compression algorithms usually voxelize the point cloud first and then apply 3D convolution for processing, which consumes a lot of computing resources. In contrast, other compression algorithms act directly on the original point cloud. For example, [1] builds an autoencoder for point cloud compression based on the PointNet structure of [2], while [3] replaces the PointNet structure of [1] with the PointNet++ structure of [4]. In addition, [5] proposes to downsample the point cloud during encoding and upsample the point cloud during decoding. However, these methods do not consider retaining the density information of the original point cloud during compression.
[0006] The above references are:
[0007] [1]Wei Yan,Shan Liu,Thomas H Li,Zhu Li,Ge Li,et al.Deep autoencoder-based lossy geometry compression for point clouds.arXiv preprint arXiv:1905.03691,2019.
[0008] [2]Charles R Qi,Hao Su,Kaichun Mo,and Leonidas J Guibas.Pointnet:Deeplearning on point sets for 3d classification and segmentation.In Proceedingsof the IEEE conference on computer vision and pattern recognition,pages 652–660,2017.
[0009] [3]Tianxin Huang and Yong Liu.3d point cloud geometry compression ondeep learning.In Proceedings of the 27th ACM International Conference onMultimedia,pages 890–898,2019.
[0010] [4]Charles R Qi,Li Yi,Hao Su,and Leonidas J Guibas.Pointnet++:Deephierarchical feature learning on point sets in a metric space.arXiv preprintarXiv:1706.02413,2017.
[0011] [5] Louis Wiesmann, Andres Milioto, Xieyuanli Chen, Cyrill Stachniss, and Jens Behley. Deep compression for dense point cloud maps. IEEE Robotics and Automation Letters, 6(2):2060–2067, 2021. Summary of the invention
[0012] In order to solve the above problems, a point cloud compression method is provided which retains density information during the compression process and obtains higher reconstruction quality. The present invention adopts the following technical solutions:
[0013] The present invention provides a density-preserving point cloud compression method based on deep learning, which is used to preserve the density of the original point cloud during the point cloud compression process to generate a high-quality reconstructed point cloud, and is characterized in that it includes the following steps: step S1, preprocessing the pre-acquired point cloud data to obtain training samples; step S2, constructing an encoder for extracting geometric information and density information of the point cloud data, and obtaining point cloud features based on the encoder; step S3, using an entropy encoder to compress the point cloud features; step S4, constructing a decoder for upsampling the point cloud features; step S5, constructing a loss function; step S6, based on the training samples, training a point cloud compression model composed of an encoder, an entropy encoder and a decoder using a loss function; step S7, inputting the point cloud data into the trained point cloud compression model to achieve point cloud compression.
[0014] The present invention provides a density-preserving point cloud compression method based on deep learning, which may also have such a technical feature, wherein the point cloud data is the three-dimensional point coordinates of the point cloud, and the preprocessing is to normalize the point cloud data and divide the normalized point cloud into non-overlapping small blocks.
[0015] The present invention provides a density-preserving point cloud compression method based on deep learning, which may also have such technical features, wherein step S2 includes the following sub-steps: step S2-1, using farthest point sampling to perform multi-stage downsampling of the point cloud, and for each point in the point cloud after downsampling in each stage, finding the corresponding contraction point set for each stage; step S2-2, using density embedding and position embedding to respectively characterize the density information and geometric information of the contraction point set in the current stage, and using ancestor embedding to characterize the density information and geometric information of the contraction point set in the previous stage; step S2-3, embedding and fusing density embedding, position embedding and ancestor embedding to obtain point cloud features.
[0016] The present invention provides a point cloud compression method for preserving density based on deep learning, which may also have such a technical feature, wherein the process of finding a contraction point set is as follows: for each downsampling stage, each discarded point will be shrunk to the nearest retained downsampling point, so that each downsampling point will form a contraction point set.
[0017] The present invention provides a density-preserving point cloud compression method based on deep learning, which may also have the following technical features: density embedding is to calculate the number of points in the contraction point set for the down-sampling points in each stage; position embedding is to calculate the direction and distance from the points in the contraction point set to the down-sampling points in each stage; the direction and distance information in the contraction point set are fused based on the attention mechanism; ancestor embedding is the density information and geometric information extracted in the previous stage of each stage.
[0018] The present invention provides a density-preserving point cloud compression method based on deep learning, which may also have such a technical feature, wherein the entropy encoder includes a quantization operation and arithmetic coding, and during the training process, the quantization operation is replaced by adding random noise to realize the back propagation of the gradient.
[0019] The density-preserving point cloud compression method based on deep learning provided by the present invention may also have such a technical feature, wherein step S4 includes the following sub-steps: step S4-1, based on the predefined maximum upsampling multiple U, constructing a scale-adaptive upsampling module to upsample the downsampled point cloud to obtain candidate point clouds and candidate features; step S4-2, based on the upsampling multiple prediction branch, predicting the actual upsampling multiple, and selecting candidate point clouds and candidate features based on this, thereby forming the final upsampling result.
[0020] The present invention provides a density-preserving point cloud compression method based on deep learning, which may also have the following technical features: an upsampling module uses sub-point convolution to perform point coordinate upsampling and feature upsampling; the sub-point convolution is based on a predefined maximum upsampling multiple U, and the input features are grouped in the channel dimension; an independent convolution is used for each group of features, and the result after the convolution is deformed to the required size; the point coordinate upsampling is as follows: uniformly sampling the surface of a unit sphere to obtain a predetermined number of candidate directions; weights of all candidate directions are obtained based on sub-point convolution; the actual movement direction of the weighted sum and the scale of the actual movement direction are obtained according to the weights; the actual offset is obtained based on the actual movement direction multiplied by the scale, and the offset is added to the point cloud feature to obtain a candidate point cloud; the feature upsampling is as follows: upsampling the point cloud features based on sub-point convolution, and obtaining the candidate features using a residual connection method.
[0021] The present invention provides a density-preserving point cloud compression method based on deep learning, which may also have such a technical feature, wherein a loss function is constructed based on a three-dimensional supervisory signal, and the loss function includes chamfer loss, density loss and bit rate loss. The density loss is obtained by calculating the difference in the number of points between the shrunken point set at a certain stage in downsampling and the shrunken point set at the corresponding stage in upsampling, as well as the difference in offset from the center point. The bit rate loss is the number of bits after entropy coding.
[0022] The present invention provides a density-preserving point cloud compression method based on deep learning, which may also have such a technical feature, wherein the point cloud compression may also perform attribute compression on the point cloud, the input and output of the attribute compression are six-dimensional coordinates, and the loss function is L2 loss.
[0023] Function and Effect of the Invention
[0024] According to a density-preserving point cloud compression method based on deep learning of the present invention, a point cloud compression model is constructed based on an encoder, a decoder and an entropy encoder. The point cloud compression model takes point cloud data as input, and the encoder uses density embedding, position embedding and ancestor embedding for downsampling to extract the density and geometric information of the point cloud to obtain point cloud features. The downsampled point cloud features are further compressed by the entropy encoder. Finally, adaptive learning of different geometric patterns is achieved through the scale-adaptive upsampling module of the sub-point convolution of the decoder, so that a high-quality point cloud with preserved density is reconstructed. The density-preserving point cloud compression method based on deep learning of the present invention retains the density information of the input original point cloud during compression, avoiding the defect that the density information of the decompressed point cloud is significantly different from the original point cloud density information. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 is a flow chart of a density-preserving point cloud compression method based on deep learning in an embodiment of the present invention;
[0026] Figure 2 is a schematic diagram of a downsampling module of an encoder in an embodiment of the present invention;
[0027] Figure 3 is a schematic diagram of an upsampling module of a decoder in an embodiment of the present invention;
[0028] Figure 4 is a schematic diagram of sub-point convolution in an embodiment of the present invention;
[0029] Figure 5 is a schematic diagram of the structure of a point cloud compression model in an embodiment of the present invention;
[0030] Figure 6 2 is a comparison chart of the experimental results in the embodiments of the present invention. DETAILED DESCRIPTION
[0031] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the density-preserving point cloud compression method based on deep learning of the present invention is specifically described below in combination with embodiments and drawings.
[0032] <Example>
[0033] Figure 1 4 is a flow chart of a density-preserving point cloud compression method based on deep learning in an embodiment of the present invention.
[0034] like Figure 1 As shown, the density-preserving point cloud compression method based on deep learning includes the following steps:
[0035] Step S1, preprocessing the pre-acquired point cloud data to obtain training samples.
[0036] In this embodiment, the point cloud data is the three-dimensional point coordinates of the point cloud. The point cloud data uses the publicly available SemanticKITTI dataset, which contains 22 point cloud sequences 00-21, of which sequences 00-10 are used as training sets and sequence 08 is used as a validation set.
[0037] In this step S1, the preprocessing process is: first normalize the point cloud data to between 0 and 100, and then divide the normalized point cloud into non-overlapping cubic blocks, and the size of each block is 12.
[0038] Step S2, constructing an encoder for extracting geometric information and density information of point cloud data, and obtaining point cloud features based on the encoder.
[0039] This step S2 includes the following sub-steps:
[0040] In step S2-1, the input point cloud block is downsampled in three stages using the farthest point sampling (FPS), and the downsampling ratio of each downsampling stage is 1 / 3. For each point in the point cloud after downsampling in each stage, the corresponding contraction point set is found for each stage.
[0041] Figure 2 Schematic diagram of a downsampling module of an encoder in an embodiment of the present invention.
[0042] like Figure 2 As shown, the process of finding the contraction point set is as follows: for each downsampling stage, each discarded point will be contracted to the nearest retained downsampling point, so that each downsampling point will form a contraction point set.
[0043] In step S2-2, density embedding and position embedding are used to respectively represent the density information and geometric information of the contracted point set in the current stage, and ancestor embedding is used to represent the density information and geometric information of the contracted point set in the previous stage.
[0044] The number of points in the shrunken point set is used as density embedding, the aggregation direction and distance from the points in the shrunken point set to the downsampled points are used as position embedding, and the density information and geometric information extracted in the previous stage of each stage are used as ancestral embedding. The dimension of all embeddings is 8.
[0045] Step S2-3, density embedding, position embedding and ancestor embedding are fused to obtain the final feature, namely the point cloud feature.
[0046] Step S3: Use an entropy encoder to compress the point cloud features.
[0047] In this embodiment, the entropy encoder includes quantization operation and arithmetic coding. Due to the non-differentiability of quantization, the quantization operation is replaced by adding random noise to achieve gradient back propagation during the training process, while the test process is direct quantization.
[0048] Step S4, constructing a decoder for upsampling point cloud features.
[0049] In this embodiment, the decoder includes three upsampling stages corresponding to the encoder. The upsampling process is as follows:
[0050] Step S4-1, for each upsampling stage, based on the predefined maximum upsampling multiple U, a scale-adaptive upsampling module is constructed to upsample the downsampled point cloud by U times to obtain candidate point clouds and candidate features, where the maximum upsampling multiple U is 8.
[0051] Step S4-2, predicting the actual upsampling multiple based on the upsampling multiple prediction branch, and selecting candidate point clouds and candidate features to form the final upsampling result.
[0052] Among them, the sampling multiple prediction branch is as follows: first, a multi-layer perceptron (MLP) is constructed. For each point in the input point cloud, its features are used to regress a floating-point number between 0 and 1 for each point, and then multiplied by the predefined maximum upsampling multiple of 8 and rounded to get the actual upsampling multiple.
[0053] Figure 3 Schematic diagram of an upsampling module of a decoder in an embodiment of the present invention.
[0054] In this embodiment, the upsampling module uses sub-point convolution to perform feature upsampling and point coordinate upsampling. Figure 3As shown in the figure, N represents the number of points, d represents the feature channel dimension, M represents the number of candidate directions, and U represents the predefined maximum upsampling multiple.
[0055] Among them, feature upsampling is to upsample the point cloud features based on sub-point convolution, and use residual connection to obtain candidate features.
[0056] The process of sampling point coordinates is as follows:
[0057] Firstly, 43 candidate directions are obtained by uniformly sampling the surface of the unit sphere, and the weights of all candidate directions are obtained based on sub-point convolution.
[0058] Then, the actual moving direction of the weighted sum and the scale of the actual moving direction are obtained according to the weights.
[0059] Finally, the actual offset is obtained based on the actual motion direction multiplied by the scale, and added to the point cloud features to obtain the candidate point cloud.
[0060] Figure 4 Schematic diagram of sub-point convolution in an embodiment of the present invention.
[0061] like Figure 4 As shown, the sub-point convolution in this embodiment is based on a predefined maximum upsampling multiple of 8, and the input features are divided into 8 groups in the channel dimension. An independent convolution is used for each group of features, and finally the convolution result is deformed to the required size. in ,d out They represent the input and output channel dimensions of the feature, N represents the number of points, and U represents the predefined maximum upsampling multiple.
[0062] Step S5, constructing a loss function using the three-dimensional supervisory signal.
[0063] In this embodiment, the loss function includes chamfer loss, density loss and bit rate loss. The chamfer loss makes the coordinates of the reconstructed point close to the real coordinates, the density loss makes the local density of the reconstructed point cloud close to the real density, and the bit rate loss is the number of bits after entropy coding, which is used to control the compression rate.
[0064] The specific calculation process of the density loss is as follows: for the real point cloud after downsampling at a certain stage in the encoder and the reconstructed point cloud after upsampling at the corresponding stage in the decoder, the difference in the number of points in their contraction point sets and the difference in offset from the center point are calculated.
[0065] Step S6: Based on the training samples, the point cloud compression model composed of the encoder, the entropy encoder and the decoder is trained using a loss function.
[0066] After preparing the training data through the above steps, we start to build the point cloud compression model. Figure 5As shown in Figure 1, the point cloud compression model includes an encoder, an entropy encoder, and a decoder. In the figure, P represents the real point cloud, F represents the real feature, Represents the reconstructed point cloud, Represents the reconstruction feature.
[0067] In this embodiment, the back propagation algorithm and the gradient descent algorithm are used to optimize the weight parameters in the point cloud compression model to complete the training. The Adam optimizer is used to optimize the network during training, with coefficient betas = (0.9, 0.999) and an initial learning rate of 1e-3. The network is trained for a total of about 50 rounds until convergence, and the learning rate decays by 0.5 every 15 rounds.
[0068] Step S7, inputting the point cloud data into the trained point cloud compression model to achieve point cloud compression.
[0069] In this embodiment, after the point cloud compression model is trained, given point cloud data, the point cloud compression model can achieve density-preserving point cloud compression.
[0070] Figure 6 2 is a comparison chart of the experimental results in the embodiments of the present invention.
[0071] Figure 6 The horizontal axis represents the average number of bits per point, and the vertical axis represents different measurement indicators. The smaller the Symmetric Chamfer Distance and the Density Metric, the better, while the larger the PSNR indicator, the better. Figure 6 As shown, by evaluating the RD curve of the compression results, the performance of the point cloud compression model of this embodiment exceeds the existing traditional point cloud compression methods: G-PCC, Draco, MPEGanchor and deep learning-based point cloud compression methods: Depoco, PCGC.
[0072] In summary, the density-preserving point cloud compression method based on deep learning in this embodiment undoubtedly has better results than previous methods.
[0073] Example Function and Effect
[0074] According to the density-preserving point cloud compression method based on deep learning provided in this embodiment, a point cloud compression model is constructed based on an encoder, a decoder and an entropy encoder. The point cloud compression model takes point cloud data as input, and the encoder uses density embedding, position embedding and ancestor embedding for downsampling to extract the density and geometric information of the point cloud to obtain point cloud features. The downsampled point cloud features are further compressed by the entropy encoder. Finally, the scale-adaptive upsampling module of the sub-point convolution of the decoder realizes adaptive learning of different geometric patterns, so a high-quality point cloud with preserved density is reconstructed. The density-preserving point cloud compression method based on deep learning in this embodiment retains the density information of the input original point cloud during compression, avoiding the defect that the density information of the decompressed point cloud is significantly different from the original point cloud density information.
[0075] The above embodiments are only used to illustrate specific implementation modes of the present invention, and the present invention is not limited to the description scope of the above embodiments.
[0076] In an embodiment, a density-preserving point cloud compression method based on deep learning performs coordinate compression on three-dimensional coordinate data of a point cloud. In other schemes of the present invention, the point cloud compression method can also be extended to attribute compression of point clouds. Taking normal compression as an example, it is only necessary to replace the input and output of this instance with the original three-dimensional coordinate data by six-dimensional coordinate plus normal data, and then add an L2 loss to the loss function of this instance to constrain the reconstructed normal.
Claims
1. A density-preserving point cloud compression method based on deep learning, which is used to preserve the density of the original point cloud during the point cloud compression process to generate a high-quality reconstructed point cloud, characterized in that: The steps include: Step S1, preprocessing the pre-acquired point cloud data to obtain training samples; Step S2, constructing an encoder for extracting geometric information and density information of the point cloud data, and acquiring point cloud features based on the encoder; Step S3, using an entropy encoder to compress the point cloud features; Step S4, constructing a decoder for upsampling the point cloud features; Step S5, constructing a loss function; Step S6, training the point cloud compression model composed of the encoder, the entropy encoder and the decoder using the loss function based on the training samples; Step S7, inputting the point cloud data into the trained point cloud compression model to achieve point cloud compression, Wherein, the step S2 includes the following sub-steps: Step S2-1, performing multi-stage downsampling on the point cloud using farthest point sampling, and for each point in the point cloud after downsampling in each stage, finding a contraction point set corresponding to each stage; Step S2-2, using density embedding and position embedding to respectively represent the density information and geometric information of the contracted point set in the current stage, and using ancestor embedding to represent the density information and geometric information of the contracted point set in the previous stage; Step S2-3, embedding and fusing the density embedding, the position embedding and the ancestor to obtain the point cloud feature, The step S4 comprises the following sub-steps: Step S4-1, based on a predefined maximum upsampling multiple U, construct a scale-adaptive upsampling module to upsample the downsampled point cloud to obtain a candidate point cloud and candidate features; Step S4-2, predicting the actual upsampling multiple based on the upsampling multiple prediction branch, and selecting the candidate point cloud and the candidate feature to form the final upsampling result, The sampling multiple prediction branch is as follows: first, a multi-layer perceptron (MLP) is constructed. For each point in the input point cloud, its features are used to regress a floating-point number between 0 and 1 for each point. The number is then multiplied by the predefined maximum upsampling multiple of 8 and rounded to get the actual upsampling multiple.
2. The method for preserving density of point cloud based on deep learning according to claim 1, characterized in that: in, The point cloud data is the three-dimensional point coordinates of the point cloud. The preprocessing is to normalize the point cloud data and divide the normalized point cloud into small non-overlapping blocks.
3. The method for preserving density of point cloud based on deep learning according to claim 1, characterized in that: in, The process of searching for the contraction point set is: For each downsampling stage, each discarded point is shrunk to its nearest retained downsampled point, so that each downsampled point forms a shrunk point set.
4. The method for preserving density of point cloud based on deep learning according to claim 1, characterized in that: in, The density embedding is to calculate the number of points in the contraction point set for each stage of down-sampling points. The position embedding is to calculate the direction and distance from the point in the contraction point set to the downsampled point in each stage, and fuse the direction and distance information in the contraction point set based on the attention mechanism. The ancestral embedding is the density information and geometric information extracted at the previous stage in each stage.
5. The method for preserving density of point cloud based on deep learning according to claim 1, characterized in that: in, The entropy encoder includes quantization operation and arithmetic coding, During training, the quantization operation is replaced by adding random noise to achieve gradient backpropagation.
6. The density-preserving point cloud compression method based on deep learning according to claim 1, characterized in that: in, The upsampling module uses sub-point convolution to perform point coordinate upsampling and feature upsampling. The sub-point convolution is based on the predefined maximum upsampling multiple U, groups the input features in the channel dimension, uses independent convolution for each group of features, and deforms the convolution result into the required size. The point coordinates are sampled as follows: uniformly sampling the surface of the unit sphere to obtain a predetermined number of candidate directions, obtaining weights of all the candidate directions based on the sub-point convolution, and obtaining weighted summed actual motion directions and scales of the actual motion directions according to the weights, The actual offset is obtained by multiplying the scale by the actual motion direction, and added to the point cloud feature to obtain the candidate point cloud. The feature upsampling is: The point cloud features are upsampled based on the sub-point convolution, and the candidate features are obtained by adopting a residual connection method.
7. The density-preserving point cloud compression method based on deep learning according to claim 1, characterized in that: in, The loss function is constructed based on the three-dimensional supervisory signal, and the loss function includes chamfer loss, density loss and bit rate loss. The density loss is obtained by calculating the difference in the number of points between the shrunken point set at a certain stage in downsampling and the shrunken point set at the corresponding stage in upsampling, as well as the difference in the offset from the center point. The bit rate loss is the number of bits after entropy coding.
8. The method for preserving density of point cloud based on deep learning according to claim 1, characterized in that: in, The point cloud compression may also perform attribute compression on the point cloud, the input and output of the attribute compression are six-dimensional coordinates, and the loss function is L2 loss.
Citation Information
Patent Citations
Method for classifying and segmenting sparse point clouds by utilizing point cloud density and rotation information
CN111832582A
Semantic segmentation method for point cloud in outdoor large scene
CN112560865A
