A device and method for enhancing the quality of compressed point cloud reconstruction
By calculating density information before point cloud compression and using a multi-scale feature extraction module and a quality enhancement module, different types of point clouds can be reconstructed in a targeted manner, which solves the problem of poor point cloud reconstruction in existing technologies and achieves higher quality point cloud reconstruction.
Patent Information
- Application Number
- CN202211312662.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-10-25
- Publication Date
- 2025-09-09
- Estimated Expiration
- 2042-10-25
AI Technical Summary
Existing point cloud compression and reconstruction methods fail to fully utilize the original information of the point cloud when enhancing quality, resulting in poor generalization and limited results, especially performing well on the training set point cloud type but not so good on other point cloud types.
The density information of the point cloud is calculated before compression, and targeted quality enhancement is performed based on the density information during the reconstruction process. Through the point cloud density calculation module, multi-scale feature extraction module and point cloud quality enhancement module, different reconstruction methods are used for solid, dense and sparse point clouds to improve the reconstruction effect.
The reconstruction quality of compressed point clouds is significantly improved, making the reconstructed point clouds closer to the original point clouds. It is suitable for point clouds of different compression rates and types, and is in line with actual application scenarios.
Smart Images
Figure CN115661166B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of point cloud geometry, and in particular relates to a device and method for enhancing the quality of compressed point cloud reconstruction. Background Art
[0002] Point clouds, a collection of data representing the shape, color, and reflectivity of three-dimensional objects, have become a mainstream method for describing their shape and structure, and are widely used in technologies such as autonomous driving and virtual reality. Point clouds used in real-world applications often contain a large number of points and exhibit significant redundancy. Directly storing and transmitting uncompressed point clouds consumes significant amounts of memory and bandwidth. Point cloud compression effectively addresses this problem.
[0003] Among the traditional point cloud compression methods, the more representative ones are the Geometry-based Point Cloud Compression (G-PCC) and Video-based Point Cloud Compression (V-PCC) methods proposed by the Moving Picture Experts Group (MPEG). Among them, G-PCC is usually used for the compression of static point clouds, while V-PCC is used for the compression of dynamic point clouds. These two methods use traditional means such as octree decomposition, two-dimensional to three-dimensional projection, and triangular surface models to efficiently complete lossless and lossy compression of point clouds. However, the visual effect of the compressed point cloud is not good. Upsampling the compressed point cloud can improve the visual effect of the point cloud and objective evaluation indicators, making the reconstructed point cloud closer to the original point cloud.
[0004] Currently, methods for point cloud reconstruction can be divided into traditional and deep learning-based methods. Traditional methods mainly use table lookup methods: after voxelizing the point cloud, a lookup table is constructed using the voxel information of the point cloud at multiple scales. The point cloud is reconstructed according to this table, which can improve the quality of the reconstructed point cloud. However, the time required to build the table is too long and the improvement effect is limited. Deep learning-based methods include: (1) point-based methods, which use multi-layer perceptrons to extract feature information between points and reconstruct the point cloud. However, this method is only suitable for point clouds with a small number of points and has high complexity; (2) voxel-based methods: After voxelizing the point cloud, three-dimensional convolution is used to extract point cloud feature information and then reconstruct the point cloud. This method is suitable for point clouds with a large number of points and is suitable for practical applications.
[0005] However, existing point cloud compression reconstruction methods do not fully utilize the original information of the point cloud when enhancing the quality of the reconstructed point cloud. They can usually only enhance the quality of specific types of point clouds, especially the point cloud types in the training set. However, their generalization is poor, and the effect of point cloud enhancement is also limited. Summary of the Invention
[0006] In view of the deficiencies of the prior art, the present invention proposes a device and method for enhancing the reconstruction quality of compressed point clouds. The density information of the point cloud is calculated before compression, and then targeted quality enhancement is performed according to the density information during the reconstruction process, overcoming the disadvantage of poor quality of the enhancement methods in the prior art.
[0007] A device for enhancing the reconstruction quality of compressed point clouds includes a point cloud density calculation module, a multi-scale feature extraction module, and a point cloud quality enhancement module.
[0008] The point cloud density calculation module calculates the global density information of the original point cloud. Set the sphere radius as R. Sequentially take each point in the original point cloud as the center of the sphere, calculate the quotient of the number of point elements covered by the sphere divided by the volume of the sphere as the local density of this point, and then take the median of the local densities of all points in the point cloud as the global density ρ of this point cloud. Where R is a positive integer.
[0009] The multi-scale feature extraction module includes multiple feature extraction branches, performs downsampling operations on the input point cloud at different scales, uses convolution to extract features for the point cloud at each scale, and finally fuses the multi-scale features to obtain the multi-scale features of the point cloud.
[0010] The point cloud quality enhancement module compares the global density ρ of the point cloud with the set thresholds T1 and T2 to classify the point cloud. When ρ≥T1, the current point cloud is a Solid point cloud, and the coordinate recovery module is used to enhance the point cloud quality; when T2<ρ<T1, the current point cloud is a Dense point cloud, first use the coordinate recovery module, and then use the coordinate refinement module to enhance the point cloud quality; when ρ≤T2, the current point cloud is a Sparse point cloud, and the coordinate refinement module is used to enhance the point cloud quality.
[0011] The coordinate recovery module performs deconvolution on the coordinates of the compressed point cloud to generate new points around each point. Then calculate the probability corresponding to each point according to the multi-scale features of the original point cloud, and then retain the k points with the highest probability to obtain the reconstructed point cloud. k is the number of points in the original point cloud.
[0012] The coordinate refinement module performs sparse convolution on the multi-scale features of the original point cloud, and then obtains the coordinate offset through 3 consecutive fully connected operations. Add the offset to the coordinates of the compressed point cloud to obtain the reconstructed point cloud.
[0013] Preferably, it further includes a point cloud segmentation module for segmenting the original point cloud into M sub-point clouds, and each sub-point cloud is respectively compressed and quality enhanced. Where M is a positive integer.
[0014] A method for enhancing the quality of compressed point cloud reconstruction, specifically including the following steps:
[0015] Step 1, Original point cloud segmentation
[0016] Segment the original point cloud to be compressed into M sub-point clouds, where M is a positive integer.
[0017] Step 2, Sub-point cloud density calculation
[0018] Set the sphere radius as R, and successively take each point in the sub-point cloud as the sphere center, calculate the quotient of the number of point elements covered by the sphere divided by the volume of the sphere as the local density of this point, and then take the median of the local densities of all points in the sub-point cloud as the global density ρ of this sub-point cloud. Among them, R is a positive integer.
[0019] Step 3, Feature extraction and compression
[0020] Perform downsampling and convolution operations on the original point cloud at different scales through multiple branches to obtain features of the original point cloud at different scales. Finally, fuse the output results of each branch to obtain the multi-scale features of the original point cloud. Finally, compress the geometric information of the original point cloud.
[0021] Step 4, Point cloud quality enhancement
[0022] Compare the global density ρ of each sub-point cloud before compression with the set thresholds T1 and T2, classify the point clouds, and adopt different quality enhancement methods for point clouds of different categories:
[0023] s4.1, When ρ≥T1, then this sub-point cloud is a Solid point cloud. Generate new points around each point by performing deconvolution on the compressed point cloud coordinates. Then, according to the multi-scale features extracted in Step 3, calculate the probability of each point and retain the k points with the highest probability to obtain the reconstructed point cloud. k is the number of points in the original point cloud.
[0024] s4.2, When T2<ρ<T1, then this sub-point cloud is a Dense point cloud. First, generate new points around each point by performing deconvolution on the compressed point cloud coordinates. Then, according to the multi-scale features extracted in Step 3, calculate the probability of each point and retain the k points with the highest probability. Then perform sparse convolution and 3 fully connected operations on the multi-scale features to obtain the coordinate offset. After adding the offset to the coordinates of these k points, obtain the reconstructed point cloud.
[0025] s4.3, When ρ≤T2, then this sub-point cloud is a Sparse point cloud. Perform sparse convolution and 3 fully connected operations on the multi-scale features to obtain the coordinate offset. After adding the offset to the coordinates of the compressed point cloud, obtain the reconstructed point cloud.
[0026] The present invention has the following beneficial effects:
[0027] 1. After segmenting the original point cloud, the sub-point clouds are classified. This serves as density information for enhancing the quality of the compressed point cloud. Selecting an appropriate reconstruction method can significantly improve the reconstruction of the compressed point cloud, making the reconstructed point cloud closer to the original point cloud. This reconstruction method can handle point clouds of varying compression rates and types, making it suitable for practical applications.
[0028] 2. During point cloud quality enhancement, a multi-scale feature extraction module is used. This module downsamples the input point cloud one or more times, extracting features using convolution at each scale. Finally, the multi-scale features are fused to obtain multi-scale features for subsequent point cloud reconstruction. This fully extracts feature information from the point cloud and enhances point cloud reconstruction quality. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Figure 1 Schematic diagram of a device structure for enhancing the quality of compressed point cloud reconstruction in an embodiment;
[0030] Figure 2 Schematic diagram of point cloud segmentation in the embodiment;
[0031] Figure 3 Schematic diagram of a point multi-scale feature extraction module in an embodiment;
[0032] Figure 4 Schematic diagram of the point cloud quality enhancement module in the embodiment
[0033] Figure 5 Schematic diagram of a coordinate recovery module in an embodiment;
[0034] Figure 6 Schematic diagram of the coordinate refinement module in the embodiment;
[0035] Figure 7 These are comparative experimental results for different data in the examples. DETAILED DESCRIPTION
[0036] The present invention will be further explained below with reference to the accompanying drawings;
[0037] like Figure 1 As shown, a device for enhancing the quality of compressed point cloud reconstruction includes a point cloud feature extraction module 100 and a point cloud quality enhancement module 200. The point cloud feature extraction module 100 includes a point cloud segmentation module 101, a multi-scale feature extraction module 102, a point cloud classification module 103 and a point cloud density calculation module 104.
[0038] like Figure 2As shown, the point cloud segmentation module 101 is used to segment the original point cloud into M sub-point clouds, and perform subsequent density calculation, feature extraction, compression reconstruction and other operations on the sub-point clouds, which can reduce computational complexity and processing time.
[0039] The point cloud density calculation module 104 calculates the global density information of the sub-point clouds segmented by the point cloud segmentation module 101 .
[0040] like Figure 3 As shown, the multi-scale feature extraction module 102 includes multiple branches. In each branch, features of sub-point clouds at different scales are extracted through downsampling and convolution operations at different scales. Finally, the multi-scale features of the sub-point clouds are obtained by fusion, which are used for subsequent quality enhancement of the reconstructed point clouds.
[0041] The point cloud classification module 103 classifies the sub-point cloud into a solid point cloud, a dense point cloud or a sparse point cloud according to the comparison result between the sub-point cloud global density ρ calculated by the point cloud density calculation module 104 and the threshold values T1 and T2.
[0042] like Figure 4 As shown, the point cloud quality enhancement module 200 uses the coordinate recovery module to enhance the point cloud quality for Solid point cloud; for Dense point cloud, first uses the coordinate recovery module and then uses the coordinate refinement module to enhance the point cloud quality; for Sparse point cloud, uses the coordinate refinement module to enhance the point cloud quality.
[0043] like Figure 5 As shown in Figure 1, the coordinate recovery module performs deconvolution on the compressed point cloud coordinates, generating new points around each point. The probability of each point being a point is then calculated based on the multi-scale features of the original point cloud. The k points with the highest probabilities are retained to form the reconstructed point cloud. k is the number of points in the original point cloud.
[0044] like Figure 6 As shown in the figure, the coordinate refinement module performs sparse convolution on the multi-scale features of the original point cloud, and then obtains the offset of the coordinates through three consecutive fully connected operations. The offset is added to the coordinates of the compressed point cloud to obtain the reconstructed point cloud.
[0045] A method for improving the quality of compressed point cloud reconstruction, comprising the following steps:
[0046] Step 1: Original point cloud segmentation
[0047] The original point cloud to be compressed is divided into M sub-point clouds, where M is a positive integer.
[0048] Step 2: Sub-point cloud density calculation
[0049] Set the radius of the sphere as R. Taking each point in the sub-point cloud as the center of the sphere in turn, calculate the quotient of the number of point elements covered by the sphere divided by the volume of the sphere as the local density of this point. Then take the median of the local densities of all points in the sub-point cloud as the global density ρ of this sub-point cloud. Here, R is a positive integer.
[0050] Step 3: Feature extraction and compression
[0051] Perform downsampling operations on the original point cloud at different scales through multiple branches to obtain features of the original point cloud at different scales. Finally, fuse the output results of each branch to obtain the multi-scale features of the original point cloud. Finally, compress the geometric information of the original point cloud.
[0052] Step 4: Point cloud quality enhancement
[0053] Compare the global density ρ of each sub-point cloud before compression with the set thresholds T1 and T2. The threshold T1 can be set to 1e-4, and T2 can be set to 1e-5. Classify the point clouds and adopt different quality enhancement methods for different types of point clouds:
[0054] s4.1: When ρ≥T1, then this sub-point cloud is a Solid point cloud. Generate new points around each point by performing deconvolution on the coordinates of the compressed point cloud. Then, according to the multi-scale features extracted in Step 3, calculate the probability of each point and retain the k points with the highest probability to obtain the reconstructed point cloud. k is the number of points in the original point cloud.
[0055] s4.2: When T2<ρ<T1, then this sub-point cloud is a Dense point cloud. First, generate new points around each point by performing deconvolution on the coordinates of the compressed point cloud. Then, according to the multi-scale features extracted in Step 3, calculate the probability of each point and retain the k points with the highest probability. Then perform sparse convolution and 3 fully connected operations on the multi-scale features to obtain the offset of the coordinates. After adding the offset to the coordinates of these k points, obtain the reconstructed point cloud.
[0056] s4.3: When ρ≤T2, then this sub-point cloud is a Sparse point cloud. Perform sparse convolution and 3 fully connected operations on the multi-scale features to obtain the offset of the coordinates. After adding the offset to the coordinates of the compressed point cloud, obtain the reconstructed point cloud.
[0057] To illustrate the effectiveness of this method, use the same dataset to train the comparative method PU-Dense and test it on the MPEG sequence. The experimental results are as Figure 7As shown in the figure, the horizontal axis Bpp represents the compression rate, and the vertical axis D1 PSNR is a commonly used objective indicator for evaluating point cloud quality, with the dimension of dB. GPCC+upsampling is the method of the present invention, which uses the density information of the original point cloud to select a suitable point cloud quality enhancement method. PU-Dense is a voxel-based point cloud quality enhancement method that does not use the density information of the original point cloud. LUT is a lookup table upsampling method, and GPCC (octree) and GPCC (trisoup) are lossy compression methods. It can be seen from the direction of the lines in the figure that the method of the present invention performs better than other point cloud quality enhancement methods on multiple data sets.
[0058] The basic principles, main features, and advantages of the present invention are shown and described above. Those skilled in the art should understand that the present invention is not limited to the foregoing embodiments. The foregoing embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.
Claims
1. A device for enhancing the quality of compressed point cloud reconstruction, characterized by: It includes a point cloud density calculation module, a multi-scale feature extraction module, and a point cloud quality enhancement module; The point cloud density calculation module is used to calculate the global density information of the original point cloud; The multi-scale feature extraction module extracts multi-scale features of the point cloud by performing downsampling and convolution operations on the input point cloud at different scales; The point cloud quality enhancement module compares the global density ρ of the point cloud with the set thresholds T1 and T2, classifies the point cloud, and then selects different quality enhancement methods; when ρ≥T1, the coordinate recovery module is used to enhance the quality of the current point cloud; when T2<ρ<T1, the coordinate recovery module is first used for the current point cloud, and then the coordinate refinement module is used to enhance the quality of the point cloud; when ρ≤T2, the coordinate refinement module is used to enhance the quality of the current point cloud; The coordinate recovery module performs deconvolution on the compressed point cloud coordinates to generate new points around each point; then calculates the probability corresponding to each point according to the multi-scale features of the original point cloud, and then retains the k points with the highest probability, where k is the number of points in the original point cloud; The coordinate refinement module performs sparse convolution on the multi-scale features of the original point cloud, and then obtains the coordinate offset through 3 consecutive fully connected operations, and adds the offset to the coordinates of the compressed point cloud.
2. The apparatus for enhancing the quality of compressed point cloud reconstruction according to claim 1, wherein: In the point cloud density calculation module, the sphere radius is set to R. Taking each point in the original point cloud as the sphere center in turn, calculate the quotient of the number of point elements covered by the sphere divided by the volume of the sphere as the local density of this point, and then take the median of the local densities of all points in the point cloud as the global density ρ of this point cloud; where R is a positive integer.
3. The apparatus for enhancing the quality of compressed point cloud reconstruction according to claim 1, wherein: It also includes a point cloud segmentation module, which is used to segment the original point cloud into M sub-point clouds, and compress and enhance the quality of each sub-point cloud respectively; where M is a positive integer.
4. The apparatus for enhancing the quality of compressed point cloud reconstruction according to claim 1, wherein: The multi-scale feature extraction module includes 3 branches. The first branch directly performs a convolution operation on the input point cloud. After the other two features perform downsampling on the input point cloud for different times, they perform a convolution operation, and then are restored to the original size through upsampling and fused with the result of the first branch to obtain the multi-scale features of the input point cloud.
5. The device for enhancing the quality of compressed point cloud reconstruction according to claim 1, wherein: Set the thresholds T1 = 1e-4 and T2 = 1e-5.
6. A method for enhancing the quality of compressed point cloud reconstruction, characterized by: Specifically, it includes the following steps: Step 1, Original point cloud segmentation The original point cloud to be compressed is segmented into M blocks of sub-point clouds, where M is a positive integer; Step 2, Sub-point cloud density calculation Set the sphere radius to R. Taking each point in the sub-point cloud as the sphere center in turn, calculate the quotient of the number of point elements covered by the sphere divided by the volume of the sphere as the local density of this point, and then take the median of the local densities of all points in the sub-point cloud as the global density ρ of this sub-point cloud; where R is a positive integer; Step 3, Feature extraction and compression Perform downsampling and convolution operations on the sub-point cloud at different scales through multiple branches to obtain multi-scale features of the sub-point cloud, and finally fuse the output results of each branch to obtain the multi-scale features of the sub-point cloud; finally, compress the geometric information of the original point cloud; Step 4, Point cloud quality enhancement Compare the global density ρ of each sub-point cloud before compression with the set thresholds T1 and T2, classify the point cloud, and adopt different quality enhancement methods for point clouds of different categories: s4.
1. When ρ ≥ T1, generate new points around each point by deconvolving the coordinates of the compressed point cloud; then, according to the multi-scale features extracted in step 3, calculate the probability of each point and retain the k points with the highest probability, obtaining the reconstructed point cloud; k is the number of points in the original point cloud; s4.
2. When T2 < ρ < T1, first generate new points around each point by deconvolving the coordinates of the compressed point cloud; then, according to the multi-scale features extracted in step 3, calculate the probability of each point and retain the k points with the highest probability; then perform sparse convolution and three fully connected operations on the multi-scale features to obtain the coordinate offset, and after adding the offset to the coordinates of these k points, obtain the reconstructed point cloud; s4.
3. When ρ ≤ T2, perform sparse convolution and three fully connected operations on the multi-scale features of the point cloud to obtain the coordinate offset, and after adding the offset to the coordinates of the compressed point cloud, obtain the reconstructed point cloud.
7. A computer-readable storage medium, on which a computer program is stored, and when the computer program is executed on a computer, it causes the computer to execute the method according to claim 6.