Method and device for compressing and decompressing geometric information of point cloud and method and device for encoding and decoding point cloud video
Patent Information
- Application Number
- CN202280094989.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-24
- Publication Date
- 2024-12-06
AI Technical Summary
Existing technologies perform poorly in the compression and decompression process of point cloud data, especially on sparse point clouds, resulting in low compression performance and difficulty in effectively saving storage and transmission bandwidth.
A probabilistic prediction model based on sparse convolution and attention mechanism is used to perform voxel downsampling and entropy coding on point clouds. The occupancy probability of voxels is predicted through feature extraction and upsampling to achieve lossless point cloud geometric data compression and decompression.
It significantly improves the compression performance of sparse point clouds and saves code rate and bandwidth. Compared with traditional methods, it saves 12.8% and 24.8% of code rate at 1mm and 2cm accuracy respectively.
Smart Images

Figure CN119096544A_ABST
Abstract
Description
Compression and decompression of point cloud geometric information and point cloud video encoding and decoding method and device Technical Field
[0001] The embodiments of the present disclosure relate to, but are not limited to, point cloud compression technology, and more specifically, to a method, device, and system for compressing and decompressing point cloud geometric information. Background Art
[0002] A point cloud is a collection of irregularly distributed discrete points in space that represent the spatial structure and surface properties of a three-dimensional object or scene. A point cloud is a type of three-dimensional data, a collection of vectors in a three-dimensional coordinate system. These vectors can represent (x, y, z) coordinates as well as attributes such as color and reflectivity. With the rapid development of emerging technologies such as augmented reality, virtual reality, autonomous driving, and robotics, point cloud data has become a primary data format due to its concise representation of three-dimensional space. However, point cloud data is massive, and directly storing it consumes a large amount of memory, making it difficult to transmit. Therefore, continuous improvement in point cloud compression performance is needed.
[0003] SUMMARY OF THE INVENTION
[0004] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.
[0005] An embodiment of the present disclosure further provides a method for compressing point cloud geometric information, including:
[0006] Downsampling the geometric data of the first-scale point cloud to obtain geometric data of the second-scale point cloud, where the geometric data includes information about occupancy signs of voxels in the point cloud;
[0007] The geometric data of the second-scale point cloud is input into the probability prediction model for feature extraction, upsampling, and probability prediction, and the occupancy probability of the voxels in the first-scale point cloud is output. The feature extraction is implemented based on sparse convolution and attention mechanism.
[0008] Entropy coding is performed on the occupancy symbols of the voxels in the first-scale point cloud according to the occupancy probabilities of the voxels in the first-scale point cloud, thereby achieving lossless compression of the geometric data of the first-scale point cloud.
[0009] An embodiment of the present disclosure provides a method for decompressing point cloud geometric information, comprising:
[0010] Decoding the geometric code stream of the point cloud to obtain lossless geometric data of the second-scale point cloud and the first encoded data, where the geometric data includes information about occupancy symbols of voxels in the point cloud;
[0011] The geometric data of the second-scale point cloud is input into the probability prediction model for feature extraction, upsampling, and probability prediction, and the occupancy probability of the voxels in the first-scale point cloud is output. The feature extraction is implemented based on sparse convolution and attention mechanism.
[0012] Entropy decoding is performed based on the occupancy probabilities of voxels in the first-scale point cloud and the first encoded data to obtain occupancy symbols of the voxels in the first-scale point cloud to reconstruct geometric data of the first-scale point cloud; wherein the first encoded data is obtained by entropy encoding the occupancy symbols of the voxels in the first-scale point cloud based on the occupancy probabilities of the voxels in the first-scale point cloud.
[0013] An embodiment of the present disclosure further provides a point cloud video encoding method, including:
[0014] Performing coordinate transformation and voxelization on the geometric information of the first-scale point cloud to obtain geometric data of the first-scale point cloud;
[0015] The geometric data of the first-scale point cloud is compressed and encoded according to the point cloud geometric information compression method of any embodiment of the present disclosure to obtain a geometric code stream.
[0016] An embodiment of the present disclosure further provides a point cloud video decoding method, including:
[0017] Obtaining geometric data of a reconstructed first-scale point cloud according to the decompression method for point cloud geometric information according to any embodiment of the present disclosure;
[0018] Perform inverse coordinate transformation on the geometric data of the first-scale point cloud to obtain the geometric information of the first-scale point cloud.
[0019] An embodiment of the present disclosure further provides a video code stream, which includes first encoded data obtained by compressing point cloud geometric information from the outside to the inside according to any implementation of the present disclosure, and encoded geometric data of at least one scale point cloud.
[0020] An embodiment of the present disclosure further provides a device for compressing point cloud geometric information, comprising a processor and a memory storing a computer program, wherein the processor, when executing the computer program, can implement a method for compressing point cloud geometric information as in any embodiment of the present disclosure.
[0021] An embodiment of the present disclosure further provides a device for decompressing point cloud geometric information, comprising a processor and a memory storing a computer program, wherein the processor, when executing the computer program, can implement a method for decompressing point cloud geometric information as in any embodiment of the present disclosure.
[0022] An embodiment of the present disclosure further provides a point cloud video encoding device, comprising a processor and a memory storing a computer program, wherein the processor can implement a point cloud video encoding method as described in any embodiment of the present disclosure when executing the computer program.
[0023] An embodiment of the present disclosure further provides a point cloud video decoding device, comprising a processor and a memory storing a computer program, wherein the processor can implement a point cloud video decoding method as described in any embodiment of the present disclosure when executing the computer program.
[0024] An embodiment of the present disclosure further provides a video encoding and decoding system, which includes a point cloud video encoding device as in any embodiment of the present disclosure, and a point cloud video decoding device as in any embodiment of the present disclosure.
[0025] An embodiment of the present disclosure further provides a non-transitory computer-readable storage medium, which stores a computer program, wherein the computer program, when executed by a processor, implements a method for decompressing point cloud geometric information as in any embodiment of the present disclosure, or a method for compressing point cloud geometric information as in any embodiment of the present disclosure.
[0026] Still other aspects will become apparent upon reading and understanding the accompanying drawings and detailed description.
[0027] Summary of the Figures
[0028] The accompanying drawings are used to provide an understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the technical solutions of the present disclosure and do not constitute a limitation to the technical solutions of the present disclosure.
[0029] FIG1 is a flow chart of G-PCC coding;
[0030] FIG2 is a flow chart of G-PCC decoding;
[0031] FIG3 is a flow chart of a method for compressing point cloud geometric information according to an embodiment of the present disclosure;
[0032] FIG4 is a schematic diagram of the probability prediction, entropy coding, and entropy decoding processes associated with adjacent low-scale point clouds and high-scale point clouds according to an embodiment of the present disclosure;
[0033] 5A , 5B , and 5C are diagrams showing occupancy symbols of voxels in the high-scale point cloud, the low-scale point cloud, and the high-scale point cloud, respectively, as examples in FIG. 4 ;
[0034] FIG6 is a schematic diagram of a compression process of point cloud geometric information after secondary pixel downsampling according to an embodiment of the present disclosure;
[0035] FIG7 is a network architecture diagram of a probability prediction model according to an embodiment of the present disclosure;
[0036] FIG8A , FIG8B , and FIG8C are schematic diagrams of a residual network, a self-attention network, and a probability generation network, respectively, according to an embodiment of the present disclosure;
[0037] FIG9 is a schematic diagram of feature extraction performed by a self-attention layer according to an embodiment of the present disclosure;
[0038] FIG10 is a flowchart of a method for decompressing point cloud geometric information according to an embodiment of the present disclosure;
[0039] FIG11 is a schematic diagram of a decompression process of point cloud geometric information after secondary pixel downsampling according to an embodiment of the present disclosure;
[0040] FIG12 is a flowchart of a point cloud video decoding method according to an embodiment of the present disclosure;
[0041] FIG13 is a flowchart of a point cloud video encoding method according to an embodiment of the present disclosure;
[0042] FIG14 is a schematic diagram of a device for decompressing point cloud geometric information according to an embodiment of the present disclosure.
[0043] Details
[0044] The present disclosure describes multiple embodiments, but the description is exemplary rather than restrictive, and it is obvious to those skilled in the art that there may be more embodiments and implementations within the scope of the embodiments described in the present disclosure.
[0045] In the description of the present disclosure, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment described as "exemplary" or "for example" in the present disclosure should not be interpreted as being more preferred or advantageous than other embodiments. "And / or" in this article is a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. "Multiple" refers to two or more than two. In addition, in order to facilitate the clear description of the technical solutions of the embodiments of the present disclosure, words such as "first" and "second" are used to distinguish between identical or similar items with basically the same functions and effects. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.
[0046] When describing representative exemplary embodiments, the specification may have presented the method and / or process as a specific sequence of steps. However, to the extent that the method or process does not rely on the specific order of the steps described herein, the method or process should not be limited to the steps in the specific order described. As will be understood by those skilled in the art, other sequences of steps are also possible. Therefore, the specific sequence of the steps set forth in the specification should not be interpreted as a limitation to the claims. In addition, the claims for the method and / or process should not be limited to the steps performed in the order written, and those skilled in the art can readily understand that these sequences can vary and still remain within the spirit and scope of the disclosed embodiments.
[0047] Point cloud compression algorithms include two schemes developed by the Moving Picture Experts Group (MPEG): Video-based Point Cloud Compression (V-PCC) and Geometry-based Point Cloud Compression (G-PCC). G-PCC primarily implements geometry compression using an octree model and / or a triangular surface model. V-PCC primarily uses 3D-to-2D projection and video compression.
[0048] In order to facilitate the understanding of the technical solutions provided by the embodiments of the present disclosure, a flowchart of G-PCC encoding and a flowchart of G-PCC decoding are first provided. It should be noted that the flowchart of G-PCC encoding and the flowchart of G-PCC decoding described in the embodiments of the present disclosure are only for more clearly illustrating the technical solutions of the embodiments of the present disclosure, and do not constitute a limitation on the technical solutions provided by the embodiments of the present disclosure. It is known to those skilled in the art that with the evolution of point cloud compression technology and the emergence of new business scenarios, the technical solutions provided by the embodiments of the present disclosure are also applicable to point cloud compression architectures similar to G-PCC. The point cloud compressed by the embodiments of the present disclosure can be a point cloud in a video, but is not limited to this.
[0049] In the point cloud G-PCC encoder framework, the point cloud of the input 3D image model is sliced and each slice is encoded independently.
[0050] As shown in the flow chart of G-PCC encoding in Figure 1, as applied to a point cloud encoder, the point cloud data to be encoded is first divided into multiple slices using slicing. Within each slice, the geometric and attribute information of the point cloud are encoded separately. During the geometric encoding process, the geometric information undergoes coordinate transformation so that the entire point cloud is contained within a bounding box. Quantization then occurs. Quantization primarily serves a scaling purpose. Due to quantization rounding, the geometric information of some point clouds remains the same, allowing for parameter-based decision-making regarding whether to remove duplicate points. This process of quantization and removing duplicate points is also known as voxelization. The bounding box is then partitioned into an octree. In the octree-based geometric information encoding process, the bounding box is divided into eight equal parts. The sub-cubes that are not empty (contain points in the point cloud) are further divided into eight equal parts until the resulting leaf nodes are 1x1x1 unit cubes. The points in the leaf nodes are then arithmetic-coded to generate a binary geometric bitstream, also known as the geometry codestream. In the process of geometric information encoding based on triangle soup (trisoup), octree partitioning must also be performed first. However, unlike the geometric information encoding based on octree, the trisoup does not need to divide the point cloud into unit cubes with a side length of 1x1x1 step by step. Instead, the division stops when the sub-block (block) has a side length of W. Based on the surface formed by the distribution of the point cloud in each block, the surface and the twelve edges of the block are obtained. The vertices are arithmetically encoded (surface fitting is performed based on the intersections) to generate a binary geometric bit stream, that is, a geometric code stream. Vertex is also used to implement the process of geometric reconstruction, and the reconstructed geometric information is used when encoding the attributes of the point cloud.
[0051] During the attribute encoding process, color conversion is performed to convert the color information (i.e., attribute information) from the RGB color space to the YUV color space. The reconstructed geometric information is then used to recolor the point cloud so that the unencoded attribute information corresponds to the reconstructed geometric information. There are two main transformation methods for color information encoding: a distance-based lifting transform that relies on level of detail (LOD) partitioning, and a direct region-adaptive hierarchical transform (RAHT) transformation. Both methods convert color information from the spatial domain to the frequency domain, obtaining high-frequency and low-frequency coefficients. These coefficients are then quantized (i.e., quantized coefficients). Finally, the geometrically encoded data (octree partitioning and surface fitting) is sliced and synthesized with the attribute encoded data (quantized coefficients). The vertex coordinates of each block are then encoded (i.e., arithmetic encoding) to generate a binary attribute bitstream, i.e., the attribute codestream.
[0052] The flow chart of G-PCC decoding shown in Figure 2 is applied to the point cloud decoder. The decoder obtains the binary code stream and independently decodes the geometric bit stream (i.e., geometric code stream) and attribute bit stream in the binary code stream. When decoding the geometric bit stream, the geometric information of the point cloud is obtained through arithmetic decoding-octree synthesis-surface fitting-reconstruction geometry-inverse coordinate transformation; when decoding the attribute bit stream, the attribute information of the point cloud is obtained through arithmetic decoding-inverse quantization-LOD-based inverse lifting or RAHT-based inverse transformation-inverse color conversion, and the three-dimensional image model of the point cloud data to be encoded is restored based on the geometric information and attribute information.
[0053] The compression method of the point cloud geometric information of the embodiment of the present disclosure can be applied to the geometric information encoding process of G-PCC as shown in Figure 1. After the voxelization is completed, it is used to replace the octree allocation and surface fitting processing therein to obtain data that needs to be entropy encoded. The entropy encoding of the embodiment of the present disclosure can adopt the arithmetic coding method in Figure 1. The decompression method of the point cloud geometric information of the embodiment of the present disclosure can be applied to the geometric information decoding process of G-PCC as shown in Figure 2. The entropy decoding of the embodiment of the present disclosure can adopt the arithmetic decoding method in Figure 2. After arithmetic decoding, it is used to replace the octree synthesis and surface fitting processing therein to obtain reconstructed geometric data. However, the present disclosure is not limited to this. The compression and decompression method of the point cloud geometric information of the embodiment of the present disclosure can also be used in other point cloud encoding and decoding processes other than G-PCC.
[0054] Neural networks and deep learning techniques can also be applied to point cloud geometry compression. Examples include volumetric model compression techniques based on 3D convolutional neural networks (3D CNNs), compression techniques that directly apply point coordinate sets to neural networks based on multi-layer perceptrons (MLPs), compression techniques that use MLPs or 3D CNNs for probability estimation and entropy coding of octree node symbols, and compression techniques based on 3D sparse convolutional neural networks. Point clouds can be divided into sparse and dense point clouds based on point density. Sparse point clouds have a large representation range and sparse distribution in 3D space, making them suitable for representing a scene; dense point clouds, on the other hand, have a small representation range and dense distribution, making them suitable for representing an object. The compression performance of these compression techniques on these two types of point clouds often differs significantly, with better performance on dense point clouds and worse performance on sparse point clouds.
[0055] In order to improve the compression performance of neural network-based compression algorithms on sparse point clouds, the present disclosure combines an attention mechanism with a sparse convolutional neural network to provide a lossless compression method for point cloud geometric information, as shown in FIG3 . The method includes:
[0056] Step 110 , performing voxel downsampling on the geometric data of the first-scale point cloud to obtain geometric data of the second-scale point cloud, wherein the geometric data includes information on occupancy signs of voxels in the point cloud;
[0057] Step 120: Input the geometric data of the second-scale point cloud into a probability prediction model for feature extraction, upsampling, and probability prediction, and output the occupancy probability of voxels in the first-scale point cloud; wherein the feature extraction is implemented based on sparse convolution and attention mechanism;
[0058] Step 130 : performing entropy coding on the occupancy symbols of the voxels in the first-scale point cloud according to the occupancy probabilities of the voxels in the first-scale point cloud, thereby achieving lossless compression of the geometric data of the first-scale point cloud.
[0059] In the above step 110, before the geometric data of the first-scale point cloud is voxel-downsampled, the geometric information of the point cloud needs to be voxelized. After voxelization, the point cloud is presented in the form of a voxel grid. Voxel is the smallest unit in the voxel grid. A point in the point cloud corresponds to an occupied voxel (i.e., a non-empty voxel), and an unoccupied voxel (i.e., an empty voxel) indicates that there is no point at that position. The geometric data of the point cloud can be represented by the occupancy symbols of the voxels in the point cloud. For example, occupied voxels are marked as 1 and unoccupied voxels are marked as 0, resulting in a binary occupancy symbol sequence. In order to save overhead, the point cloud can be first divided into an octree, and then the occupancy symbol sequence is used to represent the position of the point in the point cloud. The geometric data of the point cloud can also be represented by the coordinate data of all points in the point cloud. The coordinate data of all points in the point cloud and the occupancy symbols of the voxels in the point cloud can be converted into each other, so the coordinate data of the points in the point cloud also contains the information of the occupancy symbols of the voxels in the point cloud.
[0060] In this step, voxel downsampling can be achieved through simple pooling. For example, a max pooling layer with a stride of 2×2×2 is used to merge 8 voxels in the first-scale point cloud into 1 voxel in the second-scale point cloud. Each downsampling reduces the size of the point cloud in all three dimensions to half. The first-scale point cloud can be called a high-scale point cloud relative to the second-scale point cloud, and the second-scale point cloud can be called a low-scale point cloud relative to the first-scale point cloud.
[0061] Please refer to Figure 4. As an example of a high-scale point cloud, the figure shows a point cloud consisting of 4×4×2 voxels. After a single voxel downsampling, the resulting low-scale point cloud consists of 2×2×1 voxels. Occupied voxels in the high-scale point cloud are represented by solid cubes, representing the locations of points in the point cloud. Unoccupied voxels in the point cloud are represented by empty cubes, representing locations where no points are located in the point cloud. As shown in the figure, 6 voxels are occupied on the side of the high-scale point cloud facing the paper, and the corresponding occupancy symbols are shown in Figure 5A. The occupancy symbol of the voxel is 1, indicating that the voxel is occupied, and 0, indicating that the voxel is unoccupied. After pixel downsampling, 3 of the 4 voxels included in the low-scale point cloud are occupied, and 1 voxel is unoccupied. The corresponding occupancy symbol is shown in Figure 5B. However, the point cloud in Figure 4 is merely exemplary, and actual point clouds usually include more voxels. There is a certain degree of correlation between the geometric data of the low-scale point cloud and the geometric data of the high-scale point cloud. For example, if an occupied voxel in the low-scale point cloud is surrounded by occupied voxels (such as when the voxel is located in the middle of an object), then after the voxel is decomposed into multiple voxels in the high-scale point cloud, the multiple voxels obtained by decomposition have a high probability of also being occupied voxels.
[0062] In the above step 120, the probability prediction model is a trained deep learning network for performing feature extraction, upsampling and probability prediction processing. Among them, feature extraction is implemented based on sparse convolution and attention mechanism. Under sparse convolution, it is not necessary to traverse all voxels in the point cloud, only the occupied voxels need to be convolved. For point clouds, which are very sparsely distributed in space, the processing amount can be greatly reduced. Combining the self-attention mechanism with the sparse convolutional network can more efficiently extract the neighborhood context information of the point cloud space in the sparse point cloud, achieve more accurate probability prediction, and higher performance lossless compression.
[0063] Taking the low-scale point cloud shown in Figure 4 as an example of the second-scale point cloud, the sequence of occupancy symbols shown in Figure 5B can be input into the probability prediction model for feature extraction, upsampling and probability prediction. For the unoccupied voxels in the low-scale point cloud, the multiple voxels in the high-scale point cloud obtained by decomposition are also unoccupied, and no probability prediction is required. However, the multiple voxels in the high-scale point cloud obtained by decomposing the occupied voxels in the low-scale point cloud need to be predicted. One voxel in Figure 5B will be decomposed into 8 voxels (only 4 voxels facing the side of the paper are shown in Figure 5C). Whether these 8 voxels are occupied is uncertain, and all require probability prediction. Figure 5C shows the probability prediction results. If the occupancy probability predicted for the actually occupied voxels is closer to 1, and the occupancy probability predicted for the actually unoccupied voxels is closer to 0, the prediction is more accurate. The more accurate the prediction, the less data is generated when entropy encoding the occupancy symbols of voxels in the high-scale point cloud based on their occupancy probabilities. In other words, the more accurate the probability prediction, the better the compression performance of the point cloud geometric information. The predicted probabilities in Figure 5C are for illustrative purposes only and should not be interpreted as the actual results of the calculation.
[0064] In step 130, the occupancy probability of voxels in the high-scale point cloud can be represented by the sequence of occupancy probabilities predicted for each voxel in FIG. 5C . For voxels for which no occupancy probability prediction is performed, the occupancy probability can be directly set to 0. The occupancy symbols of the voxels in the high-scale point cloud can be represented by the sequence of occupancy symbols for each voxel in FIG. 5A . These two sequences are input into an entropy encoder for encoding, and the encoding result is recorded as the first encoded data. This achieves lossless compression of the geometric data of the high-scale point cloud. The entropy encoding here can employ, but is not limited to, the context-based adaptive binary arithmetic coding (CABAC) algorithm. According to the principle of entropy coding, the more accurate the prediction of the occupancy probability, the lower the information entropy, and the greater the actual bit rate and bandwidth savings. As shown in FIG. 4 , at the decoding end, the first encoded data and the geometric data of the low-scale point cloud (which can be directly encoded) are extracted and fed into the entropy decoder, thereby reconstructing the lossless geometric data of the high-scale point cloud.
[0065] Figure 4 uses a single pixel downsampling as an example. In actual compression, more pixel downsampling can be performed, such as 2, 3, 4, or more times. When more than 2 pixel downsamplings are performed, the two point clouds before and after each pixel downsampling can be used as high-scale point clouds and low-scale point clouds, respectively, and probability prediction and entropy coding can be performed in a similar manner.
[0066] In an example of this embodiment, the method for compressing point cloud geometric information further includes:
[0067] Performing at least K voxel downsampling on the geometric data of the second-scale point cloud to obtain geometric data of the third-scale point cloud to the K+2-th scale point cloud;
[0068] Directly perform entropy coding on the geometric data of the K+2th scale point cloud;
[0069] For the geometric data of the k-th scale point cloud, the geometric data of the k+1-th scale point cloud is input into the probability prediction model for feature extraction, upsampling and probability prediction, and the occupancy probability of the voxels in the k-th scale point cloud is output; and the occupancy symbols of the voxels in the k-th scale point cloud are entropy encoded according to the occupancy probability of the voxels in the k-th scale point cloud to achieve lossless compression of the geometric data of the k-th scale point cloud, wherein k=2,3,…,K+1, and K is a positive integer.
[0070] In an exemplary embodiment of the present disclosure, a method for compressing point cloud geometric information is provided. The compression process is illustrated in Figure 6. This embodiment uses the example of an encoder performing secondary pixel downsampling on the geometric data of a first-scale point cloud to obtain geometric data of a second-scale point cloud and a third-scale point cloud at smaller scales. In this embodiment, the first-scale point cloud is the original point cloud after voxelization, but the present disclosure is not limited to this. For example, when the decoder does not need to display the original point cloud, the first-scale point cloud can also be a point cloud obtained by pixel downsampling the original-scale point cloud.
[0071] As shown in FIG6 , the compression method of point cloud geometric information in this embodiment includes a pixel downsampling process and three encoding processes. The three encoding processes are relatively independent and can be executed in parallel in the overall encoding process of the point cloud, even in the encoding process of a slice.
[0072] During the downsampling process, the geometric data of the largest-scale point cloud, i.e., the first point cloud, is subjected to a second pixel downsampling. After the first pixel downsampling, the geometric data of the second-scale point cloud is obtained. After the second pixel downsampling, the geometric data of the third-scale point cloud is obtained. In this embodiment, voxel downsampling is achieved using a simple pooling method (e.g., a maximum pooling layer with a stride of 2×2×2).
[0073] In the first encoding process, the geometric data of the second-scale point cloud obtained by the first pixel downsampling is input into the probability prediction model for a first probability prediction. The prediction process includes feature extraction, upsampling, and probability prediction, and outputs the occupancy probabilities of the voxels in the first-scale point cloud. Then, the occupancy symbols and occupancy probabilities of the voxels in the first-scale point cloud are input into the entropy encoder for the first entropy encoding process, including obtaining the first encoded data and writing it into the bitstream.
[0074] In the second encoding process, the geometric data of the third-scale point cloud obtained by the second pixel downsampling is input into the probability prediction model for a second probability prediction. The prediction process includes feature extraction, upsampling, and probability prediction, and outputs the occupancy probability of the voxels in the second-scale point cloud. Then, the occupancy symbols of the voxels in the second-scale point cloud and the occupancy probabilities of the voxels in the second-scale point cloud are input into the entropy encoder for the second entropy encoding process, including obtaining the second encoded data and writing it into the bitstream.
[0075] The feature extraction performed by the probabilistic prediction model in the first and second encoding processes is based on sparse convolution and an attention mechanism. Therefore, this probabilistic prediction model can also be referred to as a "probabilistic prediction model based on an attention mechanism and sparse convolution." Both the first and second entropy encoding processes employ the CABAC algorithm.
[0076] In the third encoding process, the geometric data of the third-scale point cloud obtained by the second pixel downsampling is directly subjected to a third entropy encoding to obtain encoded geometric data of the third-scale point cloud. When encoding the geometric data of the third-scale point cloud, the geometric data of the third-scale point cloud can be first partitioned into an octree to obtain a corresponding binary representation before performing entropy encoding. The specific algorithm of the third entropy encoding can be different from the specific algorithms of the first and second entropy encoding, and can, for example, adopt a run-length encoding method or a Huffman encoding method.
[0077] The above three encoding processes are all lossless encoding compression processes.
[0078] While the compression method of this embodiment is illustrated above using the example of a second pixel downsampling of the geometric data of a first-scale point cloud at the encoder end, the disclosure provides a clear understanding of how to compress the geometric information of a point cloud when downsampling is performed three or more times. Regardless of the number of downsampling steps, the overall compression process involves the compression of the geometric data of point clouds at adjacent scales (including probability prediction and entropy coding), as shown in Figure 4. This compression process for geometric data of point clouds at adjacent scales can be repeated across multiple adjacent scales, with the compression process for each set of adjacent scales being relatively independent, enabling flexible, scalable encoding and decoding.
[0079] The encoder of this embodiment uses simple voxel downsampling to progressively generate multiple smaller-scale point cloud data based on the geometric data of the first-scale point cloud. Between each two adjacent scales, a probabilistic prediction model is used to predict the occupancy probability (also known as the occupied probability, occupation probability, or probability of being occupied) of each voxel in the higher-scale point cloud based on the geometric data of the lower-scale point cloud. The voxel occupancy symbol is then entropy-encoded, such as by arithmetic coding, based on the predicted occupancy probability. The first encoded data is then output and written into the geometric bitstream, thereby achieving lossless compression. This method of this embodiment can achieve good performance when used to compress sparse point cloud geometric information.
[0080] In an exemplary embodiment of the present disclosure, the feature extraction is performed before and / or after the upsampling, and the feature extraction includes:
[0081] Extract features from the geometric data or feature data of the input point cloud based on one or more layers of sparse convolutional networks, and output feature data of the point cloud containing the extracted features;
[0082] Based on one or more layers of self-attention networks, features of the neighborhood context in the point cloud space are extracted from the feature data of the input point cloud, and feature data of the point cloud containing the extracted features is output.
[0083] In this embodiment, each layer of the sparse convolutional network includes a residual network constructed based on the sparse convolutional layer; each layer of the self-attention network includes the self-attention layer, the first normalization layer, the linear layer and the second normalization layer connected in sequence, the output data and the input data of the self-attention layer are added and then input into the first normalization layer, and the output data and the input data of the linear layer are added and then input into the second normalization layer.
[0084] In an exemplary embodiment of the present disclosure, the upsampling is implemented based on a transposed sparse convolutional network; the probability prediction is implemented based on a probability generation network, and the probability generation network includes a sparse convolutional layer and an activation function layer, and the last layer is an activation function layer implemented by a sigmoid function.
[0085] The structure of the probability prediction model of an embodiment of the present disclosure is shown in Figure 7. The probability prediction model consists of four networks cascaded in sequence: a first feature extraction network, an upsampling network, a second feature extraction network, and a probability generation network, which respectively perform the first feature extraction, upsampling, second feature extraction, and probability generation processes.
[0086] The first feature extraction network of this embodiment includes a cascade of three layers of residual network (ResNet×3) and three layers of self-attention network (Transformer×3), which is used to extract features from the geometric data of the input point cloud before upsampling.
[0087] An exemplary structure of the residual network (ResNet) is shown in Figure 8A, which consists of two sparse convolutional layers (SConv K3 3 ,S1 3 ,C32) and an activation function (ReLU) layer are connected through residuals, but different numbers of layers and structures can also be used. The parameters K, S, and C of the sparse convolution layer represent the kernel size, stride, and number of feature channels, respectively. In the example in the figure, they are 3 3 , 1 3 and 32, indicating that the size of the convolution kernel in all three dimensions is 3, and the convolution step size in all three dimensions is 1. The sparse convolution layer can play a role in feature aggregation.
[0088] The self-attention network (Transformer) uses a deep self-attention grid. An exemplary structure is shown in Figure 8B, which consists of a self-attention layer, a linear layer, and a normalization layer consisting of two normalization functions (norm) connected through residual connections. That is, it includes a self-attention layer, a first normalization layer, a linear layer, and a second normalization layer connected in sequence. The output data of the self-attention layer is added to the input data and then input into the first normalization layer. The output data of the linear layer is added to the input data and then input into the second normalization layer. The structure of the self-attention layer will be explained in detail below.
[0089] The feature extraction module of this embodiment can extract a wide range of deep contextual features by stacking multiple layers of neural networks. The combination of residual networks and deep self-attention networks improves the ability to extract features of domain context in space.
[0090] The upsampling network in this embodiment includes a transposed sparse convolutional network (TSConv K2 3 ,2 3 ,C32), whose convolution kernel (K) and stride (S) are both 2 3 ,After pixel upsampling, a voxel can be divided into 2×2×2 small voxels, and the low-scale point cloud can be converted into a high-scale point cloud.
[0091] The structure of the second feature extraction network can be the same as that of the first feature extraction network, and is used to extract features from the feature data of the input point cloud after upsampling.
[0092] The probability generation network is the output layer (Output Layers). An exemplary structure is shown in Figure 8C, which includes three sparse convolutional layers (SConv K1 3 ,S1 3 ,C32), two activation function layers implemented by ReLU functions, and one activation function layer implemented by Sigmod function. The two activation function layers implemented by ReLU functions are located after the first and second sparse convolution layers respectively. The activation function layer implemented by Sigmod function is the last layer of the output layer, and can also be the last layer of the entire probability prediction model. The probability generation network maps the input high-dimensional features (32 dimensions) to 1 dimension and limits the numerical range to between 0 and 1 to obtain the occupancy probability of each voxel decomposed by the occupied voxels.
[0093] In this paper, the feature data of a point cloud is the collection of feature data of all points in the point cloud. The data features of a point are also referred to as point features. The features of each point are bound to the coordinate data of the point. The feature data of a point cloud can be obtained by sparse convolution of the occupancy symbol (occupied or unoccupied) of the point cloud. The feature data of a point will continue to change after subsequent processing such as sparse convolution and self-attention mechanism. In the process of probability prediction, before upsampling, the occupied voxels in the low-scale point cloud, that is, the points in the point cloud, are determined. After upsampling, the occupied voxels in a low-scale point cloud are decomposed into N voxels in the high-scale point cloud (such as N=8). Whether these voxels decomposed from the occupied voxels are occupied is uncertain. When performing the second feature extraction, these multiple voxels are regarded as points in the high-scale point cloud for convolution, neighbor point search, attention weight calculation and other processing.
[0094] The network structure of the probability prediction module described above is merely exemplary. For example, in other embodiments, the number of layers in the residual network and self-attention network included in the feature extraction module can vary, and the residual network can be replaced with other sparse convolutional neural networks that can be used for feature extraction. Furthermore, the probability prediction module can also be supplemented with other networks, and so on.
[0095] When training the probability prediction module of this embodiment, point cloud samples commonly used in deep neural network training can be used, and the training loss function can be set to BCE (Binary Cross Entropy) loss, that is, the cross entropy of the predicted probability and the occupied symbol.
[0096] In an exemplary embodiment of the present disclosure, extracting features of neighborhood context in point cloud space from feature data of an input point cloud based on one or more layers of self-attention networks includes: performing the following processing by the self-attention layer in each layer of the self-attention network:
[0097] For each point in the point cloud, find the neighboring points of the point based on the coordinate data of the point, and perform linear transformation on the distance information from the point to the neighboring points to obtain the position feature. The position feature is added to the features of the neighboring points to obtain the aggregated feature after position encoding;
[0098] Performing a first linear transformation on the input feature data to obtain a first vector, performing matrix multiplication on the first vector and a second vector obtained by performing a second linear transformation on the aggregated features, and activating the result to obtain an attention weight for each point in the point cloud relative to its neighboring points;
[0099] The attention weight is multiplied by a third vector to obtain feature data of the output point cloud, where the third vector is obtained by performing a third linear transformation on the aggregated feature.
[0100] In an example of this embodiment, finding the neighbor points of the point based on the coordinate data of the point in the point cloud includes: using the K nearest neighbor algorithm to find a set number of neighbor points closest to the point based on the coordinate data of the point in the point cloud.
[0101] In one example of this embodiment, the processing performed by the self-attention layer in the feature extraction network is shown in FIG9 . The input point cloud data (input) includes feature data and coordinate C in ∈R n×3 , coordinate C in ∈R n×3 Used to find neighbor points. n is the total number of points in the point cloud, d in is the dimension of the input feature data.
[0102] As shown in the figure, the feature extraction process performed by the self-attention layer includes:
[0103] K-nearest neighbor (KNN) search:
[0104] For each point p in the point cloud i , use the K nearest neighbor search algorithm (k nearest neighbor, KNN) to find the k nearest neighbor points {p i1 ,p i2 ,…p ik}, and gather the coordinates C of k neighbor points knn ∈R n×k×3 and features In some algorithms, the point may be counted as one of the k neighboring points of the point. In other algorithms, the point is not counted as a neighboring point of the point. This disclosure does not limit this.
[0105] Positional encoding:
[0106] For each point p i As the center point, find p i k neighbor points {p i1 ,p i2 ,…p ik} and the center point p i The relative distance {dist i1 ,dist i2 ,…dist ik}, get the relative distance information dist knn ∈R n×k×1 . Then pass through the linear layer W dist dist knn The dimension is mapped from 1 to d in Dimension, the relative position feature and feature F knn Add (i.e. add to feature Fknn After that, implement position encoding:
[0107] F′ knn =dist knn W dist +F knn
[0108] in, F′ knn is the aggregated feature after position encoding, Through position encoding, the features are given the perceptual information of the relative positions between corresponding points, and the features of each neighbor point have spatial position information.
[0109] QKV vector generation:
[0110] The input feature data F in Through the linear layer W Q Transform to obtain the Q vector, and transform the position-encoded aggregate feature F′ knn Pass through the linear layer W respectively K and linear layer W V Transform to get K vector and V vector, namely:
[0111] Q=F in W q ,(K,V)=F′ knn ·(W k ,W v )
[0112] in, and Represents three different linear transformations. The Q vector represents the query vector (Query), the K vector represents the vector of the relevance between the queried information and other information (Key), and the V vector represents the vector of the queried information (Value). The above dimension parameter d a and d out Can be equal to d in , such as setting them to 32. a and d out It may not be equal to d in That is, dimension transformation can be performed.
[0113] Attention weight generation and attention-based feature aggregation:
[0114] After obtaining the Q vector, K vector, and V vector, perform matrix multiplication on the Q vector and the K vector. The result is activated by the Softmax activation function, and the attention weight A of each point relative to its neighboring points when it is used as the center point is output. Finally, the attention weight A is multiplied by the V vector to obtain the feature data F of the output point cloud. out .Right now:
[0115]
[0116] Before activation, this example multiplies the result of matrix multiplication of the Q vector and the K vector by the scaling factor.
[0117] Compared with the neural network based solely on sparse convolution, the above-mentioned embodiment of the present disclosure can enhance the spatial modeling capability on sparse point clouds by introducing an attention mechanism network. Because the convolution network with a fixed convolution kernel size is difficult to extract effective neighbor features (i.e., features of the neighborhood context) on a sparsely distributed point cloud, the above-mentioned embodiment of the present disclosure introduces a network based on an attention mechanism, which directly obtains k points around the center point based on the k-nearest neighbor algorithm on the point set, and then obtains the attention weight of the center point for other points through the attention mechanism, which can more effectively extract the feature information of the neighborhood context and improve the compression performance on the sparse point cloud.
[0118] An embodiment of the present disclosure further provides a method for decompressing point cloud geometric information, as shown in FIG10 , including:
[0119] Step 210: Decode the geometric code stream of the point cloud to obtain lossless geometric data of the second-scale point cloud and the first encoded data, wherein the geometric data includes information on occupancy symbols of voxels in the point cloud.
[0120] Step 220: Input the geometric data of the second-scale point cloud into a probability prediction model for feature extraction, upsampling, and probability prediction, and output the occupancy probability of voxels in the first-scale point cloud; wherein the feature extraction is implemented based on sparse convolution and attention mechanism;
[0121] Step 230: performing entropy decoding based on the occupancy probabilities of the voxels in the first-scale point cloud and the first encoded data to obtain occupancy symbols of the voxels in the first-scale point cloud to reconstruct geometric data of the first-scale point cloud;
[0122] The first coded data is obtained by entropy coding the occupancy symbols of the voxels in the first-scale point cloud according to the occupancy probabilities of the voxels in the first-scale point cloud.
[0123] In step 210, the first encoded data can be directly extracted from the point cloud geometry stream, and the geometric data of the second-scale point cloud can be obtained by entropy decoding the encoded geometric data of the second-scale point cloud in the geometry stream. The decoded geometric data can include occupancy symbols of voxels in the point cloud, or can be converted into occupancy symbols of voxels in the point cloud.
[0124] The processing in the above step 220 may be the same as the processing in step 120 performed by the encoding end.
[0125] In step 230, the occupancy symbols of the voxels in the first-scale point cloud can be directly used as the geometric data of the first-scale point cloud, or the geometric data of the first-scale point cloud can be converted into geometric data of another format. After reconstructing the geometric data of the point cloud, an inverse coordinate transformation is performed to obtain the geometric information of the original point cloud, as shown in Figure 2.
[0126] In an example of this embodiment, the encoder performs multiple pixel downsampling on the geometric data of the first-scale point cloud, and the corresponding decompression method further includes:
[0127] Decoding the geometric code stream of the point cloud to obtain lossless geometric data of the K+2th scale point cloud and i-th coded data obtained by entropy encoding the occupancy symbols of the voxels in the i-th scale point cloud according to the occupancy probabilities of the voxels in the i-th scale point cloud, wherein the i-th scale point cloud is obtained by performing a single voxel downsampling of the i-1th scale point cloud, where i = K+2, K+1, ..., 3, and K is a positive integer;
[0128] Starting from i=K+2, perform the following processing on each value of i in descending order until the lossless geometric data of the second-scale point cloud is obtained:
[0129] Input the geometric data of the i-th scale point cloud into the probability prediction model for feature extraction, upsampling and probability prediction, and output the occupancy probability of the voxels in the i-1-th scale point cloud; wherein the feature extraction is implemented based on sparse convolution and attention mechanism;
[0130] Entropy decoding is performed according to the occupancy probability of the voxels in the i-1th scale point cloud and the i-1th encoded data to obtain the occupancy symbols of the voxels in the i-1th scale point cloud to reconstruct the lossless geometric data of the i-1th scale point cloud.
[0131] The following example illustrates the decompression method for point cloud geometric information in this embodiment, using the case where the encoder performs two pixel downsamplings on the geometric data of a first-scale point cloud. The decompression process is shown in Figure 11. In this example, the geometric data of the first-scale point cloud is downsampled twice to obtain geometric data of a second-scale point cloud and a third-scale point cloud of smaller scales. The decoder then sequentially decodes the geometric data to obtain lossless geometric data of the third-scale point cloud, lossless geometric data of the second-scale point cloud, and lossless geometric data of the first-scale point cloud.
[0132] The corresponding encoding end processing of this example is shown in Figure 6. As shown in Figure 11, the decompression process of this example includes three decoding processes performed in sequence, where:
[0133] In the first decoding process, the encoded geometric data of the third-scale point cloud is extracted from the geometric code stream of the point cloud and third entropy decoding is performed (the algorithm of the third entropy decoding corresponds to the algorithm of the third entropy coding at the encoding end) to obtain lossless geometric data of the third-scale point cloud.
[0134] In the second decoding process, the decoded geometric data of the third-scale point cloud is input into the probability prediction model for a second probability prediction. The second probability prediction at the decoding end is the same as the second probability prediction at the encoding end, and the output is the same occupancy probability of the voxels in the second-scale point cloud as the encoding end. Then, the occupancy probability of the voxels in the second-scale point cloud and the second encoded data extracted from the geometric bitstream are input into the entropy encoder for a second entropy decoding process, obtaining the occupancy symbols of the voxels in the second-scale point cloud. This occupancy symbol can be directly used as the geometric data of the second-scale point cloud or further converted into geometric data of the second-scale point cloud in another format, such as the coordinate data of the points in the second-scale point cloud, to reconstruct the geometric data of the second-scale point cloud.
[0135] In the third decoding process, the decoded geometric data of the second-scale point cloud is input into the probability prediction model for a first probability prediction. The first probability prediction at the decoding end is the same as the first probability prediction at the encoding end, and the output is the same occupancy probability of the voxels in the first-scale point cloud as that at the encoding end. Then, the occupancy probability of the voxels in the first-scale point cloud and the first encoded data extracted from the geometric bitstream are input into the entropy encoder for a first entropy decoding process, resulting in occupancy symbols for the voxels in the first-scale point cloud. This occupancy symbol can be directly used as the geometric data of the first-scale point cloud or further converted into geometric data of the first-scale point cloud in another format, thereby reconstructing the geometric data of the first-scale point cloud.
[0136] Through the above three decoding processes, the geometric data of the first-scale point cloud can be decoded and obtained without loss. The feature extraction performed in the second and third decoding processes is based on sparse convolution and attention mechanisms. The first and second entropy decoding can both use adaptive binary arithmetic decoding methods, while the specific algorithm of the third entropy decoding can be different. During the decoding process, it may not be necessary to reconstruct the geometric data of the point cloud at all scales. In this case, the last one or more decoding processes can be ignored, and decoding can be terminated after obtaining the geometric data of the point cloud at the corresponding scale.
[0137] While the decompression method of this embodiment is illustrated above using the example of a second pixel downsampling of the geometric data of a first-scale point cloud at the encoder end, the disclosure provides a clear understanding of how to decompress the geometric information of a point cloud when downsampling occurs three or more times. Regardless of the number of downsampling steps, the overall decompression process includes a fundamental step: decompressing the geometric data of point clouds at adjacent scales, as shown in Figure 4 and the accompanying description. This decompression process can be repeated across multiple adjacent scales, with the compression and decompression processes for each set of adjacent scales being relatively independent, enabling flexible, scale-scalable encoding and decoding.
[0138] The probability prediction process must be performed on both the encoder and decoder. The geometric data of the low-scale point cloud is input into the probability prediction model for feature extraction, upsampling, and probability prediction to obtain the occupancy probability of voxels in the high-scale point cloud. The probability prediction performed on the decoder must produce the same prediction results as the encoder. The decoder can use the same probability prediction model as the encoder.
[0139] The decoding end of this embodiment uses multiple decoding processes, starting with smaller-scale point clouds whose geometric data can be directly decoded, and then reconstructing the geometric data of larger-scale point clouds. Between each two adjacent scales, a probabilistic prediction model is used to predict the occupancy probability of each voxel in the higher-scale point cloud based on the geometric data of the lower-scale point cloud. Entropy decoding is then performed based on the predicted occupancy probabilities and the corresponding encoded data extracted from the bitstream to obtain the occupancy symbols of the voxels in the higher-scale point cloud, thereby reconstructing the geometric data of the lossless higher-scale point cloud.
[0140] In an exemplary embodiment of the present disclosure, the feature extraction is performed before and / or after the upsampling, and the feature extraction includes:
[0141] Extract features from the geometric data or feature data of the input point cloud based on one or more layers of sparse convolutional networks, and output feature data of the point cloud containing the extracted features;
[0142] Based on one or more layers of self-attention networks, features of the neighborhood context in the point cloud space are extracted from the feature data of the input point cloud, and feature data of the point cloud containing the extracted features is output.
[0143] In an exemplary embodiment of the present disclosure, extracting features of neighborhood context in point cloud space from feature data of an input point cloud based on one or more layers of self-attention networks includes: performing the following processing by the self-attention layer in each layer of the self-attention network:
[0144] For each point in the point cloud, find the neighboring points of the point based on the coordinate data of the point, and perform linear transformation on the distance information from the point to the neighboring points to obtain the position feature. The position feature is added to the features of the neighboring points to obtain the aggregated feature after position encoding;
[0145] Performing a first linear transformation on the input feature data to obtain a first vector, performing matrix multiplication on the first vector and a second vector obtained by performing a second linear transformation on the aggregated features, and activating the result to obtain an attention weight for each point in the point cloud relative to its neighboring points;
[0146] The attention weight is multiplied by a third vector to obtain feature data of the output point cloud, where the third vector is obtained by performing a third linear transformation on the aggregated feature.
[0147] In an exemplary embodiment of the present disclosure, finding neighbor points of the point based on the coordinate data of the point in the point cloud includes: using a K-nearest neighbor algorithm to find a set number of neighbor points closest to the point based on the coordinate data of the point in the point cloud.
[0148] In an exemplary embodiment of the present disclosure, each layer of the sparse convolutional network includes a residual network constructed based on the sparse convolutional layer; each layer of the self-attention network includes the self-attention layer, the first normalization layer, the linear layer and the second normalization layer connected in sequence, the output data and the input data of the self-attention layer are added and then input into the first normalization layer, and the output data and the input data of the linear layer are added and then input into the second normalization layer.
[0149] In an exemplary embodiment of the present disclosure, the upsampling is implemented based on a transposed sparse convolutional network; the probability prediction is implemented based on a probability generation network, the probability generation network includes a sparse convolutional layer and an activation function layer, the last layer is an activation function layer implemented by a sigmoid function, and the entropy coding adopts a context-based adaptive binary arithmetic decoding algorithm.
[0150] The probability prediction performed at the decoding end in this embodiment is the same as that performed at the encoding end, and reference can be made to the detailed description of the embodiment above.
[0151] Although the entropy coding and entropy decoding in the above embodiment are based on the CABAC algorithm and the corresponding decoding algorithm as an example, the present disclosure is not limited thereto. In other embodiments, the occupancy symbol of each point can be predicted based on the occupancy probability of the point.
[0152] For example, when entropy encoding the occupancy symbols of voxels in the first-scale point cloud based on the occupancy probabilities of voxels in the first-scale point cloud, the occupancy symbols of the voxels in the first-scale point cloud can also be first predicted based on a binary classification method. In this case, at the encoding end, if the occupancy probability of a point is greater than or equal to a set threshold (e.g., 0.5), the predicted point symbol for the point is determined to be 1; if the occupancy probability of a point is less than the set threshold (e.g., 0.5), the predicted point symbol for the point is determined to be 0. The predicted occupancy symbols of the voxels in the high-scale point cloud are compared with the occupancy symbols of the voxels in the known high-scale point cloud, and incorrectly predicted points are marked. The corresponding marked data is then entropy encoded to obtain encoded data for entropy decoding. At the decoding end, after probabilistic prediction based on the geometric data of the low-scale point cloud, the occupancy probabilities of the voxels in the high-scale point cloud are similarly compared with the threshold to obtain the predicted occupancy symbols of the voxels in the high-scale point cloud. The incorrectly predicted points are then corrected based on the decoded marked data to obtain the correct occupancy symbols of the voxels in the high-scale point cloud. When using this approach, the probabilistic prediction model is trained using a loss function based on the accuracy of the occupancy signs of voxels in the predicted high-scale point cloud.
[0153] The above-mentioned embodiments of the present disclosure combine the self-attention mechanism with a deep sparse convolutional network, which can more efficiently extract the neighborhood context information of the point cloud space in the sparse point cloud, achieve more accurate probability prediction, and higher performance lossless compression.
[0154] The above-mentioned embodiment of the present disclosure uses the K-nearest neighbor algorithm to extract neighbor points and calculate local neighborhood attention in the deep self-attention network. Compared with the existing global attention mechanism, it can save running memory and time and is more efficiently applied to the compression of large-scale point clouds.
[0155] The above-mentioned embodiment of the present disclosure is lossless compression, which requires lossless entropy coding of the occupancy state of each voxel in the high-scale point cloud. The entropy coding performance can be improved by improving the prediction accuracy of the voxel occupancy probability, thereby reducing the bit rate and saving bandwidth.
[0156] The applicant compared the compression scheme of the disclosed embodiment with other point cloud geometry compression schemes. The point cloud lossless compression comparison metric used was bitrate (bits per input point, bpp). The test data was the commonly used SemanticKITTI dataset, and the experimental results are as follows.
[0157] KITTI_q1mm20.17bpp19.66bpp17.57bppKITTI_q2cm7.62bpp7.06bpp5.73bpp
[0158] As shown in the table above, compared with the MPEG G-PCC standard solution, this solution achieves 12.8% and 24.8% bitrate savings at 1mm and 2cm accuracy, respectively. Compared with the solution based only on sparse convolution, this solution achieves 10.6% and 18.8% bitrate savings at 1mm and 2cm accuracy, respectively.
[0159] An embodiment of the present disclosure further provides a point cloud video decoding method, as shown in FIG12 , comprising:
[0160] Step 310 , obtaining geometric data of a reconstructed first-scale point cloud according to the point cloud geometric information decompression method described in any embodiment of the present disclosure;
[0161] Step 320 : Perform inverse coordinate transformation on the geometric data of the first-scale point cloud to obtain geometric information of the first-scale point cloud.
[0162] Referring to Figure 2 , the point cloud geometric information decompression method of the above-mentioned embodiment of the present disclosure can be applied to the geometric information decoding process of the G-PCC as shown in Figure 2 . Specifically, the entropy decoding in the decompression method of the embodiment of the present disclosure can adopt the arithmetic decoding method in Figure 2 , replacing the octree synthesis and surface fitting processes therein to obtain reconstructed geometric data. However, the present disclosure is not limited to the G-PCC decoding framework. The point cloud geometric information decompression method of the embodiment of the present disclosure can also be used in other point decoding frameworks besides G-PCC.
[0163] An embodiment of the present disclosure further provides a point cloud video encoding method, as shown in FIG13 , comprising:
[0164] Step 410 , performing coordinate transformation and voxelization on the geometric information of the first-scale point cloud to obtain geometric data of the first-scale point cloud;
[0165] Step 420 : compress and encode the geometric data of the first-scale point cloud according to the point cloud geometric information compression method described in any embodiment of the present disclosure to obtain a geometric code stream.
[0166] The point cloud geometric information compression method of the disclosed embodiment can be applied to the geometric information encoding process of G-PCC as shown in Figure 1. After voxelization is completed, it is used to replace the octree allocation and surface fitting processing to obtain data that needs to be entropy encoded. The entropy encoding of the disclosed embodiment can adopt the arithmetic coding method shown in Figure 1. However, the present disclosure is not limited to this. The point cloud geometric information compression method (also referred to as a compression coding method) of the disclosed embodiment can also be used in other point cloud coding frameworks besides G-PCC.
[0167] An embodiment of the present disclosure further provides a video code stream, which includes first encoded data obtained by the point cloud geometric information compression method described from the outside to the inside according to any embodiment of the present disclosure, and encoded geometric data of at least one scale point cloud.
[0168] An embodiment of the present disclosure also provides a decompression device for point cloud geometric information, as shown in Figure 15, comprising a processor 5 and a memory 6 storing a computer program, wherein the processor 5 can implement the decompression method for point cloud geometric information as described in any embodiment of the present disclosure when executing the computer program.
[0169] An embodiment of the present disclosure further provides a device for compressing point cloud geometric information, as shown in FIG5 , which includes a processor and a memory storing a computer program, wherein the processor, when executing the computer program, can implement the method for compressing point cloud geometric information as described in any embodiment of the present disclosure.
[0170] An embodiment of the present disclosure further provides a point cloud video decoding device, as shown in FIG5 , comprising a processor and a memory storing a computer program, wherein the processor can implement the point cloud video decoding method as described in any embodiment of the present disclosure when executing the computer program.
[0171] An embodiment of the present disclosure further provides a point cloud video encoding device, as shown in Figure 5, which includes a processor and a memory storing a computer program, wherein the processor can implement the point cloud video encoding method described in any embodiment of the present disclosure when executing the computer program.
[0172] An embodiment of the present disclosure further provides a video encoding and decoding system, which includes a point cloud video encoding device as described in any embodiment of the present disclosure, and a point cloud video decoding device as described in any embodiment of the present disclosure.
[0173] The apparatus and system of the above-mentioned embodiments of the present disclosure can be implemented based on computing devices such as terminals or servers. The terminals may include mobile terminals such as mobile phones, tablet computers, laptop computers, PDAs, portable media players (PMPs), navigation devices, wearable devices, smart bracelets, pedometers, etc., as well as fixed terminals such as digital TVs and desktop computers.
[0174] An embodiment of the present disclosure further provides a non-transitory computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the decompression method of point cloud geometric information as described in any embodiment of the present disclosure, or the compression method of point cloud geometric information as described in any embodiment of the present disclosure.
[0175] In one or more exemplary embodiments, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions or codes on or transmitted via a computer-readable medium and executed by a hardware-based processing unit. The computer-readable medium may include a computer-readable storage medium corresponding to a tangible medium such as a data storage medium, or a communication medium that facilitates the transfer of a computer program from one place to another, such as according to a communication protocol. In this manner, a computer-readable medium may generally correspond to a non-transitory tangible computer-readable storage medium or a communication medium such as a signal or carrier wave. The data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures for implementing the techniques described in this disclosure. A computer program product may include a computer-readable medium.
[0176] By way of example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Furthermore, any connection may also be referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient (transient) media, but rather refer to non-transient tangible storage media. As used herein, disk and optical disk include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, or Blu-ray disc, among others, where disks typically reproduce data magnetically, while optical discs use lasers to reproduce data optically. Combinations of the above should also be included within the scope of computer-readable media.
[0177] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Thus, the term "processor," as used herein, may refer to any of the aforementioned structures or any other structure suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described herein may be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into a combined codec. Furthermore, the techniques may be fully implemented in one or more circuits or logic elements.
[0178] The technical solutions of the embodiments of the present disclosure can be implemented in a wide variety of devices or equipment, including wireless mobile phones, integrated circuits (ICs), or a group of ICs (e.g., chipsets). Various components, modules, or units are described in the embodiments of the present disclosure to emphasize the functional aspects of the devices configured to perform the described techniques, but they do not necessarily need to be implemented by different hardware units. Instead, as described above, the various units can be combined in a codec hardware unit or provided by a collection of interoperable hardware units (including one or more processors as described above) in combination with appropriate software and / or firmware.
Claims
1. A method for compressing point cloud geometric information, comprising: Downsampling the geometric data of the first-scale point cloud to obtain geometric data of the second-scale point cloud, wherein the geometric data includes information of occupancy signs of voxels in the point cloud; Inputting the geometric data of the second-scale point cloud into a probability prediction model for feature extraction, upsampling, and probability prediction, and outputting the occupancy probability of voxels in the first-scale point cloud; wherein the feature extraction is implemented based on sparse convolution and attention mechanism; Entropy coding is performed on occupancy symbols of voxels in the first-scale point cloud according to the occupancy probabilities of the voxels in the first-scale point cloud, thereby achieving lossless compression of geometric data of the first-scale point cloud.
2. The method according to claim 1, wherein Also includes: Performing at least K voxel downsampling on the geometric data of the second-scale point cloud to obtain geometric data of the third-scale point cloud to the K+2-th scale point cloud; Directly perform entropy coding on the geometric data of the K+2th scale point cloud; For the geometric data of the k-th scale point cloud, the geometric data of the k+1-th scale point cloud is input into the probability prediction model for feature extraction, upsampling and probability prediction, and the occupancy probability of the voxels in the k-th scale point cloud is output; and the occupancy symbols of the voxels in the k-th scale point cloud are entropy encoded according to the occupancy probability of the voxels in the k-th scale point cloud to achieve lossless compression of the geometric data of the k-th scale point cloud, wherein k=2,3,…,K+1, and K is a positive integer.
3. The method according to claim 1 or 2, wherein: The feature extraction is performed before and / or after the upsampling, and the feature extraction includes: Extract features from the geometric data or feature data of the input point cloud based on one or more layers of sparse convolutional networks, and output feature data of the point cloud containing the extracted features; Based on one or more layers of self-attention networks, features of the neighborhood context in the point cloud space are extracted from the feature data of the input point cloud, and feature data of the point cloud containing the extracted features is output.
4. The method of claim 3, wherein: The method of extracting features of neighborhood context in point cloud space from feature data of input point cloud based on one or more layers of self-attention network includes: performing the following processing by the self-attention layer in each layer of the self-attention network: For each point in the point cloud, find the neighboring points of the point based on the coordinate data of the point, and perform linear transformation on the distance information from the point to the neighboring points to obtain the position feature. The position feature is added to the features of the neighboring points to obtain the aggregated feature after position encoding; Performing a first linear transformation on the input feature data to obtain a first vector, performing matrix multiplication on the first vector and a second vector obtained by performing a second linear transformation on the aggregated features, and activating the result to obtain an attention weight for each point in the point cloud relative to its neighboring points; The attention weight is multiplied by a third vector to obtain feature data of the output point cloud, where the third vector is obtained by performing a third linear transformation on the aggregated feature.
5. The method of claim 4, wherein: Each layer of the sparse convolutional network includes a residual network constructed based on the sparse convolutional layer; Each layer of the self-attention network includes the self-attention layer, the first normalization layer, the linear layer and the second normalization layer connected in sequence, the output data and the input data of the self-attention layer are added and then input into the first normalization layer, and the output data and the input data of the linear layer are added and then input into the second normalization layer; The finding of neighbor points of the point based on the coordinate data of the point in the point cloud includes: finding a set number of neighbor points closest to the point using a K-nearest neighbor algorithm based on the coordinate data of the point in the point cloud.
6. The method of claim 1, wherein: The upsampling is implemented based on a transposed sparse convolutional network; The probability prediction is implemented based on a probability generation network, which includes a sparse convolution layer and an activation function layer, and the last layer is an activation function layer implemented by a sigmoid function; The entropy coding adopts a context-based adaptive binary arithmetic coding algorithm.
7. A method for decompressing point cloud geometric information, comprising: Decoding the geometric code stream of the point cloud to obtain lossless geometric data of the second-scale point cloud and the first encoded data, wherein the geometric data includes information on occupancy symbols of voxels in the point cloud; Inputting the geometric data of the second-scale point cloud into a probability prediction model for feature extraction, upsampling, and probability prediction, and outputting the occupancy probability of voxels in the first-scale point cloud; wherein the feature extraction is implemented based on sparse convolution and attention mechanism; Entropy decoding is performed based on the occupancy probabilities of voxels in the first-scale point cloud and the first encoded data to obtain occupancy symbols of the voxels in the first-scale point cloud to reconstruct geometric data of the first-scale point cloud; wherein the first encoded data is obtained by entropy encoding the occupancy symbols of the voxels in the first-scale point cloud based on the occupancy probabilities of the voxels in the first-scale point cloud.
8. The method of claim 7, wherein: Also includes: Decoding the geometric code stream of the point cloud to obtain lossless geometric data of a K+2th scale point cloud and i-th coded data obtained by entropy encoding the occupancy symbols of the voxels in the i-th scale point cloud according to the occupancy probabilities of the voxels in the i-th scale point cloud, wherein the i-th scale point cloud is obtained by performing a single voxel downsampling of the i-1th scale point cloud, where i=K+2, K+1, ..., 3, and K is a positive integer; Starting from i=K+2, perform the following processing on each value of i in descending order until the lossless geometric data of the second-scale point cloud is obtained: Input the geometric data of the i-th scale point cloud into the probability prediction model for feature extraction, upsampling and probability prediction, and output the occupancy probability of the voxels in the i-1-th scale point cloud; wherein the feature extraction is implemented based on sparse convolution and attention mechanism; Entropy decoding is performed according to the occupancy probability of the voxels in the i-1th scale point cloud and the i-1th encoded data to obtain the occupancy symbols of the voxels in the i-1th scale point cloud to reconstruct the lossless geometric data of the i-1th scale point cloud.
9. The method according to claim 7 or 8, wherein: The feature extraction is performed before and / or after the upsampling, and the feature extraction includes: Extract features from the geometric data or feature data of the input point cloud based on one or more layers of sparse convolutional networks, and output feature data of the point cloud containing the extracted features; Based on one or more layers of self-attention networks, features of the neighborhood context in the point cloud space are extracted from the feature data of the input point cloud, and feature data of the point cloud containing the extracted features is output.
10. The method of claim 9, wherein: The method of extracting features of neighborhood context in point cloud space from feature data of input point cloud based on one or more layers of self-attention network includes: performing the following processing by the self-attention layer in each layer of the self-attention network: For each point in the point cloud, find the neighboring points of the point based on the coordinate data of the point, and perform linear transformation on the distance information from the point to the neighboring points to obtain the position feature. The position feature is added to the features of the neighboring points to obtain the aggregated feature after position encoding; Performing a first linear transformation on the input feature data to obtain a first vector, performing matrix multiplication on the first vector and a second vector obtained by performing a second linear transformation on the aggregated features, and activating the result to obtain an attention weight for each point in the point cloud relative to its neighboring points; The attention weight is multiplied by a third vector to obtain feature data of the output point cloud, where the third vector is obtained by performing a third linear transformation on the aggregated feature.
11. The method of claim 10, wherein: The finding of neighbor points of the point based on the coordinate data of the point in the point cloud includes: finding a set number of neighbor points closest to the point using a K-nearest neighbor algorithm based on the coordinate data of the point in the point cloud.
12. The method of claim 10, wherein: Each layer of the sparse convolutional network includes a residual network constructed based on the sparse convolutional layer; Each layer of the self-attention network includes the self-attention layer, the first normalization layer, the linear layer and the second normalization layer connected in sequence. The output data and input data of the self-attention layer are added and then input into the first normalization layer, and the output data and input data of the linear layer are added and then input into the second normalization layer.
13. The method of claim 7, wherein: The upsampling is implemented based on a transposed sparse convolutional network; The probability prediction is implemented based on a probability generation network, which includes a sparse convolution layer and an activation function layer, and the last layer is an activation function layer implemented by a sigmoid function; The entropy decoding adopts a context-based adaptive binary arithmetic decoding algorithm.
14. A point cloud video encoding method, comprising: Performing coordinate transformation and voxelization on the geometric information of the first-scale point cloud to obtain geometric data of the first-scale point cloud; The geometric data of the first-scale point cloud is compressed and encoded according to the point cloud geometric information compression method according to any one of claims 1 to 6 to obtain a geometric code stream.
15. A point cloud video decoding method, comprising: Obtaining geometric data of a reconstructed first-scale point cloud according to the decompression method for point cloud geometric information according to any one of claims 7 to 13; An inverse coordinate transformation is performed on the geometric data of the first-scale point cloud to obtain geometric information of the first-scale point cloud.
16. A video stream, wherein: The method comprises first coded data obtained according to the method for compressing point cloud geometric information as claimed in any one of claims 1 to 6, and coded geometric data of at least one scale point cloud.
17. A device for compressing point cloud geometric information, comprising a processor and a memory storing a computer program, wherein: When the processor executes the computer program, it is capable of implementing the method for compressing point cloud geometric information as described in any one of claims 1 to 6.
18. A device for decompressing point cloud geometric information, comprising a processor and a memory storing a computer program, wherein: When the processor executes the computer program, it is capable of implementing the method for decompressing point cloud geometric information as described in any one of claims 7 to 13.
19. A point cloud video encoding device, comprising a processor and a memory storing a computer program, wherein: When the processor executes the computer program, it can implement the point cloud video encoding method as claimed in claim 14.
20. A point cloud video decoding device, comprising a processor and a memory storing a computer program, wherein: When the processor executes the computer program, it can implement the point cloud video decoding method as described in claim 15.
21. A video encoding and decoding system, wherein: It includes the point cloud video encoding device as described in claim 19 and the point cloud video decoding device as described in claim 20.
22. A non-transitory computer-readable storage medium storing a computer program, wherein: When the computer program is executed by a processor, the method for compressing point cloud geometric information according to any one of claims 1 to 6 or the method for decompressing point cloud geometric information according to claims 7 to 13 is implemented.
