A point cloud geometry compression method based on a sparse convolutional neural network

Abstract

This invention discloses a point cloud geometric compression method based on a sparse convolutional neural network. The method includes: acquiring a scene point cloud; processing the sparse tensor of the point cloud using a sparse convolutional neural network to obtain latent variables and sparse features, and decoding the sparse features to obtain sparse prediction features; obtaining reconstructed features by sparsely convolving and upsampling the latent variables; extracting features from the dense tensor of the point cloud to obtain dense features; calculating sparse-dense residual features based on the sparse prediction features and dense features; fusing the sparse-dense residual features and the reconstructed features to obtain fused features; and sparsely convolving and upsampling the fused features to obtain the reconstructed point cloud. This invention reduces the coding redundancy in point cloud geometric compression, significantly improving coding performance and the quality of the compressed point cloud.

Description

Technical Field

[0001] This invention belongs to the field of point cloud compression coding technology, and particularly relates to a point cloud geometric compression method based on sparse convolutional neural networks. Background Technology

[0002] Point clouds are data sets composed of multiple three-dimensional points, often used to represent the geometric structure of three-dimensional objects. Because point cloud data is typically high-dimensional and non-uniform, directly processing this data consumes significant computational resources. Therefore, point clouds need to be compressed. Point Cloud Geometry Compression (PCG) is an important technique in computer vision and graphics processing. Its purpose is to reduce redundancy in point cloud data through algorithms, thereby lowering its storage and transmission costs, while ensuring that the decompressed point cloud data can restore the original geometric structure as much as possible.

[0003] Point cloud compression techniques mainly include geometric information compression and attribute information compression. Geometric information compression typically uses tree or block structures to divide and encode point clouds, such as octree-based methods. Attribute information compression reduces redundancy between point cloud attributes through prediction, transform coding, and other techniques.

[0004] With the advancement of deep learning technology, learning-based PCG compression methods have emerged. Most of these methods inherit the network architecture learned from 2D image compression but utilize 3D convolutions. Compared to traditional convolutional neural networks (CNNs), sparse convolutional neural networks (Sparse CNNs) consider the sparsity of data and only compute convolution operations on non-zero elements, making them more efficient at handling sparse data, particularly suitable for processing data formats such as point clouds and sparse images.

[0005] The kernel of sparse convolution is the same as that of traditional convolution, but its output is quite different. There are two output definitions for sparse convolution. One is the regular output definition, which, like ordinary convolution, computes the output point as long as the kernel covers an input point. The other is called the submanifold output definition. In this definition, the convolution output is only computed when the center of the kernel covers an input point.

[0006] In terms of computational implementation, Sparse CNN first establishes an index-coordinate hash table for the input and output tensors, mapping the pixel coordinates of the input and output tensors to their corresponding indices. Then, a Rulebook is constructed to establish the mapping relationship between input indices and output indices, a key step in implementing sparse convolution. The implementation of sparse convolution involves querying the Rulebook to match and calculate the convolution kernel weights with the input pixel values, placing the results in the corresponding positions of the output tensors. This parallel implementation using GPUs improves computational efficiency.

[0007] Data tensors use a set of coordinates C = {(x i ,y i ,z i )} and related features F={f i The term} indicates that convolution only gathers features that are currently occupying the coordinates.

[0008] The computational definition of sparse convolution is:

[0009] For u∈C out C out Indicates the output coordinates, C in Indicates the input coordinates; This represents the output feature vector at coordinate u. W represents the input feature vector at coordinate u. i This represents the kernel value of a 3D convolution kernel.

[0010] Define a 3D convolution kernel: N 3 (u,C in )={i|u+i∈C in ,i∈N 3};

[0011] It covers a set centered at u, in C in The position with offset i. This sparse convolution uses the sparsity of the point cloud to reduce complexity, applying computation only to the occupied voxels.

[0012] Recent studies have shown that Sparse Convolutional Neural Networks (SCNNs) can handle large-scale point clouds and perform very well in semantic segmentation and object detection, but PCGs are still lacking in point cloud geometric compression. Summary of the Invention

[0013] To address the aforementioned technical problems, this invention proposes a point cloud geometric compression method based on sparse convolutional neural networks, which includes:

[0014] S1: Obtain scene point cloud;

[0015] S2: Use a sparse convolutional neural network to process the sparse tensor of the point cloud to obtain latent variables and sparse features. After decoding the sparse features, obtain sparse prediction features.

[0016] S3: Extract features from the dense tensor of the point cloud to obtain dense features;

[0017] S4: Calculate the sparse and dense residual features based on the sparse prediction features and dense features;

[0018] S5: Reconstructed features are obtained by sparse convolution upsampling the latent variables;

[0019] S6: The sparse-dense residual features and the reconstructed features are fused to obtain the fused features;

[0020] S7: Perform sparse convolution upsampling on the fused features to obtain the reconstructed point cloud.

[0021] The beneficial effects of this invention are as follows: By utilizing sparse convolutional neural networks to extract sparse features and latent variables of point clouds, and combining dense feature extraction with sparse-dense residual feature fusion, this invention effectively improves the detail richness, integrity, and geometric accuracy of the reconstructed point cloud, thereby significantly improving image quality. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating an embodiment of the present invention;

[0023] Figure 2 This is a flowchart illustrating the steps of an embodiment of the present invention;

[0024] Figure 3 This is a schematic diagram of the sparse convolutional downsampling module in an embodiment of the present invention;

[0025] Figure 4 This is a schematic diagram of the structure of the initial residual network module IRN in an embodiment of the present invention;

[0026] Figure 5 This is a schematic diagram of the sparse convolution upsampling module in an embodiment of the present invention;

[0027] Figure 6 PSNR curves for PCGC encoding and testing of the Soldier test sequence according to this invention;

[0028] Figure 7 PSNR curves for PCGC encoding and testing of the longdress test sequence according to this invention. Detailed Implementation

[0029] The terms "first," "second," "third," etc., used in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application.

[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] This invention investigates how to combine point cloud encoding with Sparse Convolutional Neural Networks (SCNNs) to leverage the feature extraction and compression capabilities of neural networks for a more efficient encoding and decoding process. This invention employs sparse convolutions for low-complexity tensor processing. Simultaneously, it utilizes Inception-Residual Network (IRN) units for effective feature extraction. For downscaling on the encoder side, convolutions are applied with a stride of 2, halving the scale of each geometric dimension at each step.

[0032] This invention proposes a point cloud geometric compression method based on sparse convolutional neural networks, referring to... Figure 1 , 2 As shown, the method includes:

[0033] S1: Obtain the scene point cloud.

[0034] In one alternative embodiment, the point cloud data used is obtained from the open-source dataset 8i Voxelized FullBodies (8iVFB v2).

[0035] For example, to obtain the scene point cloud, its sparse tensor representation is: {C X F X}, where C X This represents the coordinate information of each point in the point cloud X, usually a three-dimensional vector containing the point's position in space, such as (x, y, z) coordinates. X This represents the feature information of each point in the point cloud X. This feature information can be various attributes related to the point, such as color, intensity, and normal. The feature information can be a multi-dimensional vector, the dimension of which depends on the types and number of features included. This embodiment of the invention mainly focuses on the coordinate information portion (C) of each point in the point cloud. X ).

[0036] S2: Use a sparse convolutional neural network to process the sparse tensor of the point cloud to obtain latent variables and sparse features. After decoding the sparse features, obtain sparse prediction features.

[0037] In a preferred embodiment, a sparse neural network is used to perform sparse convolution downsampling and residual connection on the sparse tensors of the point cloud to obtain latent variables and sparse features, respectively. After decoding the sparse features, sparse prediction features are obtained.

[0038] In a preferred embodiment, a sparse convolutional neural network is used to process the sparse tensor of the point cloud. The specific processing steps include:

[0039] S201: Using the sparse convolution downsampling module in the Sparse Neural Network (Sparse CNN), the sparse tensor of the point cloud is downsampled to obtain the first sparse feature map.

[0040] Figure 3 This is a schematic diagram of the sparse convolution downsampling module in an embodiment of the present invention.

[0041] Reference Figure 3 As shown, the network structure of the sparse convolution downsampling module includes: a 16-channel 3d sparse convolutional layer with a kernel of 3*3*3, a first ReLU activation function layer, a 32-channel 3d sparse convolutional layer with a kernel of 3*3*3 / 2↓, and a second ReLU activation function layer.

[0042] Specifically, refer to Figure 3 As shown, the sparse tensor of the point cloud is input into the sparse neural network Sparse CNN, and downsampled by the sparse convolution downsampling module to obtain the first sparse feature map.

[0043] S202: Use the initial residual network IRN to perform a reversible bijective transformation on the first sparse feature map to obtain the latent variable Y, and obtain the sparse feature p based on the latent variable Y.

[0044] Figure 4 This is a schematic diagram of the structure of the initial residual network IRN in an embodiment of the present invention.

[0045] Reference Figure 4As shown, the network structure of the initial residual network IRN includes: N input channels, a first branch and a second branch. The output of the first branch is added to the output of the second branch, and then added to the N input channels to obtain the output of the initial residual network IRN. The first branch includes: a 3d sparse convolutional layer with N / 4 channels and a 3*3*3 kernel, a third ReLU activation function layer, a 3d sparse convolutional layer with N / 2 channels and a 3*3*3 kernel, and a fourth ReLU activation function layer. The second branch includes: a 3d sparse convolutional layer with N / 4 channels and a 1*1*1 kernel, a fifth ReLU activation function layer, a 3d sparse convolutional layer with N / 4 channels and a 3*3*3 kernel, a sixth ReLU activation function layer, and a 3d sparse convolutional layer with N / 2 channels and a 1*1*1 kernel.

[0046] IRN is used for sampling, and sparse features and latent variables in the sparse feature map are obtained through IRN.

[0047] For example, refer to Figure 4 As shown, x represents the input feature map with dimensions H×W×C; the scaling factor is s and the model f. θ,s The model outputs a low-resolution image p with a size of [size missing]. Through model f θ,s The algorithm calculates two results (p, Y) on the input image x, where p is the low-resolution image, i.e., the sparse feature, and Y is the latent variable (i.e., the hidden representation); finally, it returns the sparse feature p.

[0048] S203: Use a sparse neural network (Sparse CNN) to decode the sparse feature p and obtain the sparse prediction feature P'.

[0049] S3: Reconstructed features are obtained by sparse convolution upsampling the latent variable Y.

[0050] Figure 4 This is a schematic diagram of the sparse convolution upsampling module in an embodiment of the present invention.

[0051] Reference Figure 4 As shown, the network structure of the sparse convolution upsampling module includes: a 3d sparse convolutional layer with 32 channels and a kernel size of 3*3*3 / 2↑, a seventh ReLU activation function layer, a 3d sparse convolutional layer with 32 channels and a kernel size of 3*3*3, and an eighth ReLU activation function layer.

[0052] The reconstruction feature F r The expression is:

[0053] in, Represents the reconstructed feature F r The coordinates.

[0054] The specific steps of upsampling are as follows: the input is a sparse feature p with dimensions H×W×C, and it also includes a scaling factor s and a model f. θ,s The model output is a high-resolution image x with dimensions sH×sW×C.

[0055] Specifically: First, obtain the prior distribution p(z) through the latent variable Y. Then, randomly sample z from the prior distribution p(z), with dimensions H×W×(s). 2 -1). Then, through the inverse transformation f of the model. θ,s -1 The high-resolution image x is calculated. Finally, the generated high-resolution image x is returned, which is based on the reconstructed features F. r The obtained feature map.

[0056] S4: Extract features from the dense tensor of the point cloud to obtain dense features.

[0057] In one embodiment, dense features are obtained by performing coordinate transformation, quantization, and removal of duplicate points on the point cloud dense tensor.

[0058] Specifically: In the dense tensor of a point cloud, the position information Xn|n=1,…,N of the point cloud is usually represented by floating-point numbers and is located in the world coordinate system.

[0059] First, translate all points in the point cloud X to the origin to convert it to the object coordinate system, and then perform quantization. The expression is as follows:

[0060] X n '=(X n -T) / q;

[0061] T=(min(x n ),min(y n ),min(z n ))|n=1,…,N;

[0062] In the formula, X n ' represents the nth point in the point cloud X after quantization. n Let X represent the nth point in the point cloud X, T represent the minimum value of the point cloud X in each coordinate axis direction, and q represent the quantization step size, which is set by the user. The parameters T and q ensure that all coordinates are within the range [0, 2^d), where d is a non-negative integer, and its calculation formula is:

[0063] d = Ceil(Log2(max(x)) n ,y n ,z n (n = 1, ..., N) + 1);

[0064] In the formula, Ceil(.) represents the floor function, specifically rounding up to the smallest integer greater than or equal to (.), and max(.) represents the maximum value function, (x n ,y n ,z n ) represents the coordinates of the nth point in the point cloud X, and N represents the number of points in the point cloud.

[0065] To facilitate partitioning, the quantized position information needs to be rounded down, converting the floating-point number to an integer, as shown in the following formula:

[0066] Int(X n ')=Round(X n '),

[0067] In the formula, Int(X) n ') indicates that the position information of the midpoint of Xn is rounded, and Round(.) indicates the rounding function.

[0068] After quantization and rounding, multiple points with the same geometric position may appear. It is usually necessary to delete the duplicate points so that there is only one point with the same geometric position, so as to facilitate partitioning. After removing duplicate points, dense feature P is obtained.

[0069] S5: Calculate the sparse and dense residual features based on the sparse prediction feature P' and the dense feature P'".

[0070] The sparse-dense residual feature F c The expression is:

[0071]

[0072] In the formula, Coordinates representing the sparse-dense residual characteristics.

[0073] Performing residual calculations on sparse predictive features P' and dense features P” can effectively preserve feature information, and the residuals are smaller, making the features easier to learn, especially when the input and output features are similar.

[0074] S6: The sparse and dense residual features and the reconstructed features are fused to obtain the fused features.

[0075] Specifically, the sparse-dense residual feature F c and reconstruction features F r The data is then stitched together and processed through a multilayer perceptron (MLP) to obtain fused features.

[0076] The fused features are quantized to remove duplicate points, and then upsampled using sparse convolution to obtain the reconstructed point cloud.

[0077] A hierarchical reconstruction based on binary classification is used to reconstruct the point cloud. Binary classification is used to classify whether the generated voxels are occupied. Sparse convolutional layers are used to generate the probability of voxels being occupied after continuous convolution.

[0078] The loss function is set during training, and its expression is:

[0079]

[0080] Where, x i p represents the voxel label that is actually occupied (1) or empty (0). i This represents the probability of an occupied voxel, activated by the sigmoid function.

[0081] Experimental verification:

[0082] Simulation platform or software: A Linux server with two 3090 graphics cards, and the software includes VS Code, CMake, and Git.

[0083] Data source for simulation: The point cloud data used in this invention are all obtained from the open-source dataset 8i VoxelizedFull Bodies (8iVFB v2).

[0084] Evaluation Indicators Explanation:

[0085] BD Rate is a metric used to evaluate the efficiency of video coding, especially when comparing different video encoders or coding parameters. BD Rate measures the bit rate variation required by one coding method relative to another while maintaining the same quality level (such as PSNR or SSIM). A lower BD Rate value indicates a more efficient coding method.

[0086] PSNR (Peak Signal-to-Noise Ratio) is a metric used to measure the quality of an image or video. It assesses quality by comparing the difference between the original image and the compressed or processed image. A higher PSNR value indicates better image or video quality, meaning less distortion or noise. PSNR is usually expressed in decibels (dB).

[0087] Simulation results:

[0088] According to Table 1, the present invention significantly reduces the BD rate and improves the coding efficiency.

[0089] Table 1 compares the octree coding, triangular soup coding, PCGC coding, and the BD-Rate of this invention.

[0090]

[0091] In Table 1, PointCloud represents point cloud, G-PCC(octree) represents point cloud geometric octree coding, G-PCC(trisoup) represents point cloud geometric trisoup coding, PCGC represents point cloud geometric coding, soldier represents point cloud sample soldier, longdress represents point cloud sample longdress, loot represents point cloud sample loot, and Average represents average bdrate.

[0092] Figure 6 PSNR curves for PCGC encoding and testing of the Soldier test sequence according to this invention. Figure 6 The "ourmethod" parameter refers to the PSNR curve obtained during the simulation test of the Soldier test sequence in this invention. Figure 6 As can be seen, for the Soldier test sequence, the present invention has a higher PSNR and better image quality at various resolutions.

[0093] Figure 7 PSNR curves for PCGC encoding and testing of the longdress test sequence according to this invention. Figure 7 The `ourmethod` parameter refers to the PSNR curve obtained during simulation testing of the long dress test sequence using this invention. Figure 7 As can be seen, for the longdress test sequence, the present invention has a higher PSNR and better image quality at various resolutions.

[0094] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include ROM, RAM, disk, or optical disk, etc.

[0095] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for point cloud geometry compression based on sparse convolutional neural network, characterized in that, The method comprises: obtaining a scene point cloud; down-sampling a sparse tensor of the point cloud by using a sparse convolution down-sampling module in a sparse neural network Sparse CNN to obtain a first sparse feature map; performing reversible bijective transformation on the first sparse feature map by using an initial residual network IRN to obtain a latent variable, and obtaining a sparse feature according to the latent variable; decoding the sparse feature by using the sparse neural network Sparse CNN to obtain a sparse predicted feature; performing sparse convolution up-sampling on the latent variable by using a sparse convolution up-sampling module to obtain a reconstructed feature; extracting features from a point cloud dense tensor to obtain a dense feature; performing residual calculation on the sparse predicted feature and the dense feature to obtain a sparse-dense residual feature; fusing the sparse-dense residual feature and the reconstructed feature to obtain a fused feature; performing sparse convolution up-sampling on the fused feature by using the sparse convolution up-sampling module to obtain a reconstructed point cloud.

2. The method of claim 1, wherein, The network structure of the sparse convolution down-sampling module comprises: a 3d sparse convolution layer with 16 channels and a convolution kernel of 3*3*3, a first ReLU activation function layer, a 3d sparse convolution layer with 32 channels and a convolution kernel of 3*3*3 / 2, and a second ReLU activation function layer.

3. The method of claim 1, wherein, The network structure of the initial residual network IRN comprises: N input channels, a first branch and a second branch, the output results of the first branch and the second branch are added, and then added with the N input channels to obtain the output results of the initial residual network IRN; the first branch comprises: a 3d sparse convolution layer with N / 4 channels and a convolution kernel of 3*3*3, a third ReLU activation function layer, a 3d sparse convolution layer with N / 2 channels and a convolution kernel of 3*3*3, and a fourth ReLU activation function layer; the second branch comprises: a 3d sparse convolution layer with N / 4 channels and a convolution kernel of 1*1*1, a fifth ReLU activation function layer, a 3d sparse convolution layer with N / 4 channels and a convolution kernel of 3*3*3, a sixth ReLU activation function layer, and a 3d sparse convolution layer with N / 2 channels and a convolution kernel of 1*1*1.

4. The method of claim 1, wherein, The network structure of the sparse convolution up-sampling module comprises: a 3d sparse convolution layer with 32 channels and a convolution kernel of 3*3*3 / 2, a seventh ReLU activation function layer, a 3d sparse convolution layer with 32 channels and a convolution kernel of 3*3*3, and an eighth ReLU activation function layer.

5. The method of claim 1, wherein, After coordinate transformation, quantization processing and removal of duplicate points are performed on the point cloud dense tensor, the dense feature is obtained.

Images