A point cloud geometry compression method based on block partitioning and deep learning

Abstract

The present invention discloses a point cloud geometry compression method based on block partitioning and deep learning, comprising the following steps: (1) using the farthest point sampling FPS and K nearest neighbor KNN operation to divide the point cloud into blocks; (2) using the S point cloud blocks obtained in step (1) as the input of the encoder in the autoencoder in turn, and obtaining S low-dimensional feature vectors after the S blocks are compressed by the encoder, and the low-dimensional feature vectors are quantized and spliced ​​with the coordinates of the S structural points sampled in step (1) to form the final hidden layer representation of the point cloud; (3) performing entropy encoding on the final hidden layer representation of the point cloud to form a final bit stream, and transmitting it to the decoding end; (4) the decoding end obtains the transmitted bit stream and entropy decodes it into S low-dimensional feature vectors and S structural point coordinates. The present invention can achieve a better point cloud compression effect.

Description

Technical Field

[0001] The present invention relates to the field of multimedia compression technology, and in particular to a point cloud geometry compression method based on block division and deep learning. Background Art

[0002] 3D point cloud is an important representation of objects in 3D space. With the rise of virtual reality and mixed reality technologies, point cloud is receiving more and more attention. Point cloud is a set of points in 3D space, each point is specified by (x, y, z) coordinates and optional attributes (such as color, normal vector, etc.). The amount of point cloud data is usually large, which puts high demands on the performance of point cloud compression methods.

[0003] At present, the research on point cloud compression can be roughly divided into three categories: LiDAR point cloud compression, point cloud geometric information compression, and video-based point cloud compression. However, traditional lossy point cloud compression methods usually perform poorly in low bit rate environments. For example, the number of points generated by the octree-based compression method will decrease sharply as the tree depth decreases, and it will also produce a mosaic-like block effect.

[0004] Autoencoders are a new data compression model based on machine learning, which can automatically learn the analysis and synthesis transformations of point cloud data. However, most of the existing point cloud compression autoencoders are based on voxelization and 3D convolution, which have problems such as low memory usage efficiency and high time complexity. In addition, voxelization and 3D convolution cannot effectively handle irregular or sparse point clouds. Recently proposed deep neural network structures such as PointNet and PointNet++ can directly extract point cloud features from point sets without voxelization. PointNet uses a symmetric function to extract features of disordered point clouds, but it lacks the process of obtaining local features. PointNet++, as an improved version of PointNet, uses multiple sampling, grouping and PointNet processes to obtain local features of point clouds at different scales, similar to multiple convolution operations in two-dimensional images, and has achieved good results in point cloud classification and segmentation. Although there are methods that use hierarchical structures similar to PointNet++ to compress point clouds, they are all only for sparse point clouds. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a point cloud geometry compression method based on block division and deep learning, which can achieve better point cloud compression effect.

[0006] In order to solve the above technical problems, the present invention provides a point cloud geometry compression method based on block partitioning and deep learning, comprising the following steps:

[0007] (1) Use the farthest point sampling FPS and K nearest neighbor KNN operation to divide the point cloud into blocks; use the farthest point sampling to sample S structural points in the original point cloud, and then use the K nearest neighbor method to obtain K adjacent points in the original point cloud with each structural point as the center, that is, divide the original point cloud into S blocks, each block has K points;

[0008] (2) The S point cloud blocks obtained in step (1) are sequentially used as the input of the encoder in the autoencoder. After the S blocks are compressed by the encoder, S low-dimensional feature vectors are obtained. After the low-dimensional feature vectors are quantized, they are concatenated with the coordinates of the S structural points sampled in step (1) to form the final hidden layer representation of the point cloud.

[0009] (3) Entropy encoding the final hidden layer representation of the point cloud to form a final bit stream and transmit it to the decoding end;

[0010] (4) The decoding end obtains the transmitted bit stream and entropy decodes it into S low-dimensional feature vectors and S structural point coordinates; the decoder of the autoencoder decodes the S low-dimensional feature vectors into S point cloud blocks. Finally, the S point cloud blocks are added and merged with the corresponding S structural point coordinates to obtain the final point cloud reconstruction result.

[0011] Preferably, in step (1), the point cloud is divided into blocks using the farthest point sampling FPS and K nearest neighbor KNN operations as follows: for an original point cloud with N points, S structural points are sampled from the original point cloud using the farthest point sampling; for each structural point, KNN is used to find the K nearest neighbor points of the structural point from the original point cloud, and the coordinates of the sampling point are subtracted from the coordinates of the K nearest neighbor points, that is, the coordinates of the K points are moved to a coordinate system with the structural point as the origin, so as to obtain a block; for S structural points, S blocks can be generated, each block has K points, and each block can be regarded as a "mini" point cloud.

[0012] Preferably, in step (2) and step (4), the autoencoder is an end-to-end trained deep neural network, the input and output of the network are the original point cloud block and the reconstruction result of the point cloud block respectively; the autoencoder is divided into three parts: an analysis transform encoder, a quantization and a synthesis transform decoder, and the chamfer distance is used to constrain the error between the reconstructed block and the input block. The loss function of the autoencoder is defined as:

[0013] Loss = D CD +λR

[0014] Where D CD is the chamfer distance between the input block and the reconstructed block, R represents the bit rate estimated according to the quantized hidden layer features, and λ represents the Lagrange multiplier.

[0015] The beneficial effects of the present invention are as follows: the present invention proposes a method for dividing a point cloud into small blocks, wherein the points in the point cloud are independent of each other and there is no regular voxel spatial correlation; the farthest point sampling and K nearest neighbor algorithm are used to divide a complex point cloud into a number of small blocks with simple shapes, so that the local structural information of the point cloud can be better captured in the subsequent reconstruction process; the point cloud is compressed based on a block-based autoencoder structure, and this autoencoder can obtain better results in the reconstruction process. The autoencoder can also freely adjust the number of points of the reconstructed point cloud, which is also of great reference significance for solving other problems involving point cloud reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a schematic diagram of the compression process of the present invention.

[0017] Figure 2 Schematic diagram of the autoencoder structure of the present invention.

[0018] FIG3( a ) is a schematic diagram of an input point cloud including 8192 points according to the present invention.

[0019] FIG3( b ) is a schematic diagram of 32 structural points sampled from the input point cloud using farthest point sampling according to the present invention.

[0020] FIG3( c ) is a schematic diagram of the block division result when the present invention uses K nearest neighbors (K=256) for 32 structural points.

[0021] FIG3( d ) is a schematic diagram of the block division result when the present invention uses K nearest neighbors (K=512) for 32 structural points.

[0022] Figure 4 It is a visualization diagram of the training process of the present invention.

[0023] FIG5( a ) is a schematic diagram of the “aircraft” point cloud input by the present invention.

[0024] FIG5( b ) is a schematic diagram of the “aircraft” point cloud reconstructed by the present invention.

[0025] FIG5( c ) is a schematic diagram of the “bathtub” point cloud input by the present invention.

[0026] FIG5( d ) is a schematic diagram of the “bathtub” point cloud reconstructed by the present invention. DETAILED DESCRIPTION

[0027] A point cloud geometry compression method based on block partitioning and deep learning comprises the following steps:

[0028] (1) Use the farthest point sampling FPS and K nearest neighbor KNN operation to divide the point cloud into blocks; use the farthest point sampling to sample S structural points in the original point cloud, and then use the K nearest neighbor method to obtain K adjacent points in the original point cloud with each structural point as the center, that is, divide the original point cloud into S blocks, each block has K points;

[0029] (2) The S point cloud blocks obtained in step (1) are sequentially used as the input of the encoder in the autoencoder. After the S blocks are compressed by the encoder, S low-dimensional feature vectors are obtained. After the low-dimensional feature vectors are quantized, they are concatenated with the coordinates of the S structural points sampled in step (1) to form the final hidden layer representation of the point cloud.

[0030] (3) Entropy encoding the final hidden layer representation of the point cloud to form a final bit stream and transmit it to the decoding end;

[0031] (4) The decoding end obtains the transmitted bit stream and entropy decodes it into S low-dimensional feature vectors and S structural point coordinates; the decoder of the autoencoder decodes the S low-dimensional feature vectors into S point cloud blocks. Finally, the S point cloud blocks are added and merged with the corresponding S structural point coordinates to obtain the final point cloud reconstruction result.

[0032] like Figure 1 As shown in the figure, the point cloud is first divided into S blocks, each with K points. During the encoding process, the encoding result of the autoencoder and the S sampling points are connected to form a point cloud hidden layer representation of the final (S, 3+d) matrix size. During the decoding process, the point cloud hidden layer representation is split into two parts that are the same as the encoder output. Finally, the decoded blocks are assembled to generate a point cloud reconstruction result with a matrix size of (S×k, 3), where k represents the number of points in the reconstructed block point cloud. It should be noted that the number of points in the input block point cloud and the number of points in the reconstructed block point cloud can be set to be unequal, that is, K can not be equal to k.

[0033] Among them, different from the block division method of two-dimensional images, the present invention adopts the farthest point sampling (FPS) and K-nearest neighbors (KNN) method to divide the point cloud into several blocks with the same resolution. The specific block division process is described as follows.

[0034] First, for an original point cloud with N points, S structure points are sampled from the original point cloud using farthest point sampling.

[0035] Next, for each structure point, use KNN to find the K nearest neighbor points of the structure point from the original point cloud. Subtract the coordinates of the sampling point from the coordinates of the K nearest neighbor points, that is, move the coordinates of these K points to the coordinate system with the structure point as the origin, and you can get a block. For S structure points, S blocks can be generated, each with K points. Each block can be regarded as a "mini" point cloud.

[0036] Figure 3(a)-(d) is an illustration of the size of S and K in block segmentation. Figure 3(a) is a point cloud of 8192 points, Figure 3(b) is the FPS sampling result when S is 32, Figure 3(c) is the block result when S is 32 and K is 256, and Figure 3(d) is the block result when S is 32 and K is 512. It can be seen that even when the product of S and K is exactly the number of input point clouds, the point cloud structure information cannot be fully captured. It is necessary to use S×K=αN(α>1) to avoid the situation where some points in the original point cloud are omitted. In the specific implementation, we use α=2.

[0037] When the number of blocks S is large and the resolution K of the blocks is small, the autoencoder is more suitable for high-quality reconstruction of point clouds at high bit rates; when the number of blocks S is small and the resolution K of the blocks is large, it is more suitable for compressing point clouds to smaller volumes at low bit rates.

[0038] The present invention designs an autoencoder based on PointNet to realize the transformation and compression of point cloud blocks. The autoencoder includes an analysis transformation (f a ), quantization function (Q) and synthesis transform (f s ). We use the analysis transform to extract hidden features from a simple block; use the quantization function to quantize the hidden features for further compression; and use the synthesis transform to reconstruct the quantized features into a block.

[0039] The analysis transform first uses a set abstraction (SA) to extract a local feature for each point in the local area, and then uses PointNet to extract higher-level features on a global scale. For the quantization process, during training, we superimpose a uniformly distributed noise between -0.5 and 0.5 on each element of the hidden layer features extracted by the analysis transform. During testing, we round the hidden layer features to achieve subsequent entropy coding. The synthesis transform is a multi-layer perceptron composed of several fully connected layers. The last step is to deform the output of the multi-layer perceptron into a two-dimensional matrix of block geometry information.

[0040] The autoencoder uses the chamfer distance (D CD ) to constrain the error between the reconstruction result and the input patch. Figure 2As shown, the final loss function of the autoencoder is L = D CD +λR, where R represents the bit rate estimated by the probability distribution of the hidden layer features. It should be noted that the effect of λ on the compression effect is not significant. We mainly adjust the compression ratio by changing the size of the bottleneck dimension of the autoencoder.

[0041] Example

[0042] The present invention is further described in detail below in conjunction with a specific embodiment. For ease of description and without loss of generality, the following assumptions are made:

[0043] The method proposed in the present invention is intended to be trained and tested using the ModelNet40 dataset. The ModelNet40 dataset contains 9835 training point clouds and 2467 test point clouds, where the point cloud data is created by uniformly sampling each 3D model in ModelNet40, and each point cloud contains 8192 points.

[0044] This embodiment controls the compression ratio of the input point cloud by S, K and the size of the bottleneck dimension d of the autoencoder. Experiments show that for the point cloud of 8192 points used in this embodiment, when the bottleneck dimension size is between 8 and 16, the compression performance is better.

[0045] Taking the reconstruction of point cloud at a low bit rate as an example, S=32, K=512, d=16 are set here, that is, α=2 here, that is, S×K=2N, to more completely capture the overall structure of the point cloud, where N is the number of points in the input point cloud, that is, 8192. In this embodiment, the point cloud in each training set is first divided into 32 blocks, each block contains 512 points, that is, 9835 point clouds are converted into 9835×32 blocks. We use these 9835×32 blocks as training data for the autoencoder. Such a large amount of data can significantly avoid the overfitting problem often encountered in model training. At the same time, since the shape of each block is relatively simple, the autoencoder does not require a large amount of parameters to obtain good reconstruction effects. The parameters of the autoencoder in this embodiment are as follows:

[0046] SA(K=512,8,[32,64,128])→PN(64,32,d=16)→Quantization→FC(d=16,128)

[0047] →ReLU→FC(128,256)→ReLU→FC(256,256×3)

[0048] The parameters of each operation SA (representing the set feature extraction layer), PN (representing the PointNet layer), and FC (representing the fully connected layer) are defined as:

[0049] Aggregate feature extraction layer: SA (number of points in blocks, number of points in groups, shared multi-layer perceptron size)

[0050] PointNet layer: PN (shared multi-layer perceptron size)

[0051] Fully connected layer: FC (input feature dimension, output feature dimension)

[0052] After the autoencoder is trained in the above way, it is applied to the compression process we proposed. The reconstruction results of the autoencoder after different iterations are as follows: Figure 4 shown.

[0053] During the compression process, we divide the input point cloud into 32 blocks using FPS and KNN to input the trained autoencoder. The autoencoder converts the 32 blocks into 32 hidden layer representations, which are concatenated with the sampled 32 structure point coordinates into a (32, 16+3) two-dimensional matrix as the final representation of the input point cloud. This two-dimensional matrix is ​​entropy encoded and transmitted to the decoder.

[0054] During the decoding process, the two-dimensional matrix (32, 16+3) is first split into a set of (32, 16) block latent representations and 32 structural point coordinate information with a matrix size of (32, 3). The 32 16-dimensional block latent representations are reconstructed into 32 blocks by the decoder of the autoencoder, and the matrix size of each block is (256, 3). Then the 32 blocks are added to the corresponding structural point coordinates, that is, the 32 blocks are relocated to their original positions in the input point cloud, and their union is taken to obtain a two-dimensional matrix coordinate set with a size of (32×256, 3), which completes the overall reconstruction of the original point cloud. Examples of the results of three-dimensional point cloud reconstruction are shown in Figure 5(a)-(d).

Claims

1. A point cloud geometry compression method based on block partitioning and deep learning, characterized in that: The steps include: (1) Use the farthest point sampling (FPS) and K nearest neighbor (KNN) operations to divide the point cloud into blocks; use the farthest point sampling to sample S structural points in the original point cloud, and then use the K nearest neighbor method to obtain K adjacent points in the original point cloud with each structural point as the center, that is, divide the original point cloud into S blocks, each block has K points; specifically: for an original point cloud with N points, use the farthest point sampling to sample S structural points from the original point cloud; for each structural point, use KNN to find the K nearest neighbor points of the structural point from the original point cloud, subtract the coordinates of the structural point from the coordinates of the K nearest neighbor points, that is, move the coordinates of these K points to the coordinate system with the structural point as the origin, and then get a block; for S structural points, generate S blocks, each block has K points, and each block is regarded as a "mini" point cloud; (2) The S point cloud blocks obtained in step (1) are sequentially used as the input of the encoder in the autoencoder. After the S blocks are compressed by the encoder, S low-dimensional feature vectors are obtained. After the low-dimensional feature vectors are quantized, they are concatenated with the coordinates of the S structural points sampled in step (1) to form the final hidden layer representation of the point cloud. (3) Entropy encoding the final hidden layer representation of the point cloud to form a final bit stream and transmit it to the decoding end; (4) The decoding end obtains the transmitted bit stream and entropy decodes it into S low-dimensional feature vectors and S structural point coordinates; the decoder of the autoencoder decodes the S low-dimensional feature vectors into S point cloud blocks. Finally, the S point cloud blocks are added and merged with the corresponding S structural point coordinates to obtain the final point cloud reconstruction result; the autoencoder is an end-to-end trained deep neural network, the input and output of which are the original point cloud block and the reconstruction result of the point cloud block respectively; the autoencoder is divided into three parts: analysis transform encoder, quantization and synthesis transform decoder, and uses the chamfer distance to constrain the error between the reconstructed block and the input block. The loss function of the autoencoder is defined as: Loss=D CD +λR Where D CD is the chamfer distance between the input block and the reconstructed block, R represents the bit rate estimated according to the quantized hidden layer features, and λ represents the Lagrange multiplier.

Images