Point cloud geometric data enhancement, coding and decoding method, device and system

Abstract

The invention discloses a point cloud geometric data enhancement, coding and decoding method, device and system. According to the method, multiple times of pixel down-sampling and feature extraction are carried out on geometric data of the Nth scale at a coding end, and feature data used for enhancing geometric data of point cloud of the (N + 1) th scale are obtained; and at a decoding end, after pixel up-sampling and feature reasoning are carried out on the feature data, the output feature data are spliced with geometric data of the (N + 1) th scale point cloud to obtain enhanced geometric data of the (N + 1) th scale point cloud, and then the enhanced geometric data are used for subsequent decoding processing to obtain reconstructed geometric features of the Nth scale point cloud. According to the embodiment of the invention, the point cloud coding performance can be improved, and the used auto-encoder model is convenient to train and use.

Description

A point cloud geometric data enhancement, encoding and decoding method, device and system Technical Field

[0001] The embodiments of the present disclosure relate to, but are not limited to, point cloud compression technology, and more specifically, to a method for enhancing point cloud geometric data, a coding and decoding method, an apparatus, and a system. Background Art

[0002] A point cloud is a collection of irregularly distributed discrete points in space that represent the spatial structure and surface properties of a three-dimensional object or scene. A point cloud is a type of three-dimensional data, a collection of vectors in a three-dimensional coordinate system. These vectors can represent (x, y, z) coordinates as well as attributes such as color and reflectivity. With the rapid development of emerging technologies such as augmented reality, virtual reality, autonomous driving, and robotics, point cloud data has become a primary data format due to its concise representation of three-dimensional space. However, point cloud data is massive, and directly storing it consumes a large amount of memory, making it difficult to transmit. Therefore, continuous improvement in point cloud compression performance is needed.

[0003] SUMMARY OF THE INVENTION

[0004] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0005] An embodiment of the present disclosure provides a point cloud geometric data enhancement method, which is applied to a point cloud decoder, including:

[0006] Parse the code stream to obtain feature data used to enhance the geometric data of the point cloud at the i+1th scale;

[0007] The feature data is processed by the partial decoder of the i-th decoder network M i -1 voxel upsampling and feature inference, the output feature data is spliced ​​with the geometric data to be enhanced of the i+1 scale point cloud to obtain the enhanced geometric data of the i+1 scale point cloud;

[0008] Where i is an integer greater than or equal to 1, M i is an integer greater than or equal to 2.

[0009] An embodiment of the present disclosure further provides a method for decoding point cloud geometric data, which is applied to a point cloud decoder and includes:

[0010] Parse the bitstream, obtain the geometric data of the N+1th scale point cloud as the geometric data to be enhanced, and perform data enhancement according to the point cloud geometric data enhancement method described in any embodiment of the present disclosure to obtain enhanced geometric data of the N+1th scale point cloud, where N≥1;

[0011] The remaining decoders of the Nth decoder network perform voxel upsampling and feature inference on the geometric data after the N+1th scale point cloud enhancement. The output data is then subjected to probability prediction and point cloud cropping to obtain the reconstructed geometric data of the Nth scale point cloud.

[0012] An embodiment of the present disclosure further provides a method for encoding point cloud geometric data, which is applied to a point cloud encoder and includes:

[0013] Perform N voxel downsampling on the geometric data of the first-scale point cloud to obtain the geometric data of the second-scale point cloud to the N+1-th scale point cloud, where N ≥ 1;

[0014] The geometric data of the N-th scale point cloud is input into the N-th encoder network of the N-th autoencoder model for M N Secondary voxel downsampling and feature extraction, outputting feature data for enhancing the geometric data of the point cloud at the N+1 scale, M N ≥2;

[0015] Entropy encoding is performed on the geometric data of the N+1th scale point cloud and the feature data output by the Nth encoder network.

[0016] An embodiment of the present disclosure further provides a point cloud geometry code stream, wherein the geometry code stream is obtained according to the encoding method of point cloud geometry data described in any embodiment of the present disclosure, and includes the geometry data of the N+1th scale point cloud and the feature data output by the Nth encoder network.

[0017] An embodiment of the present disclosure further provides a point cloud geometric data enhancement device, comprising a processor and a memory storing a computer program, wherein when the processor executes the computer program, it can implement the point cloud geometric data enhancement method as described in any embodiment of the present disclosure.

[0018] An embodiment of the present disclosure further provides a point cloud decoder, comprising a processor and a memory storing a computer program, wherein when the processor executes the computer program, it is capable of implementing the method for decoding point cloud geometric data as described in any embodiment of the present disclosure.

[0019] An embodiment of the present disclosure further provides a point cloud encoder, comprising a processor and a memory storing a computer program, wherein when the processor executes the computer program, it is capable of implementing the method for encoding point cloud geometric data as described in any embodiment of the present disclosure.

[0020] An embodiment of the present disclosure further provides a point cloud encoding and decoding system, which includes the point cloud encoder described in any embodiment of the present disclosure and the point cloud decoder described in any embodiment of the present disclosure.

[0021] An embodiment of the present disclosure further provides a non-transitory computer-readable storage medium, which stores a computer program, wherein the computer program, when executed by a processor, can implement the point cloud geometric data enhancement method described in any embodiment of the present disclosure, or can implement the point cloud geometric information decoding method described in any embodiment of the present disclosure, or can implement the point cloud geometric information encoding method described in any embodiment of the present disclosure.

[0022] An embodiment of the present disclosure further provides a point cloud clipping method, which is applied to a point cloud decoder and includes:

[0023] Parse the code stream to obtain the number K of occupied voxels in the point cloud to be cropped;

[0024] Determining the occupancy probability of voxels in the point cloud to be cropped;

[0025] The M voxels decomposed from the same voxel in the point cloud to be cropped are divided into a group, and the occupation probability of the m voxels with the highest occupation probability in each group is set to 1. Then, the occupation probabilities of all voxels in the point cloud to be cropped are sorted, and the K voxels with the highest occupation probability are determined as the occupied voxels in the point cloud to be cropped, 1≤m <M<K。

[0026] Still other aspects will become apparent upon reading and understanding the accompanying drawings and detailed description.

[0027] Summary of the Figures

[0028] The accompanying drawings are used to provide an understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the technical solutions of the present disclosure and do not constitute a limitation to the technical solutions of the present disclosure.

[0029] FIG1 is a flow chart of G-PCC coding;

[0030] FIG2 is a flow chart of G-PCC decoding;

[0031] FIG3 is a schematic diagram of a method for encoding and decoding point cloud geometric information according to an embodiment of the present disclosure;

[0032] FIG4 is a schematic diagram of a method for encoding and decoding point cloud geometric data according to another embodiment of the present disclosure;

[0033] FIG5 is a flowchart of a method for encoding point cloud geometric data according to an embodiment of the present disclosure;

[0034] FIG6 is a schematic diagram of the network structure of the residual layer according to an embodiment of the present disclosure;

[0035] FIG7 is a schematic diagram of a network structure of an encoder according to an embodiment of the present disclosure;

[0036] FIG8 is a schematic diagram of the network structure of the self-attention layer according to an embodiment of the present disclosure;

[0037] FIG9 is a schematic diagram of a process in which a point cloud neighborhood self-attention layer obtains neighborhood context features in a point cloud space from input data according to an embodiment of the present disclosure;

[0038] FIG10 is a flowchart of a method for enhancing point cloud geometric data according to an embodiment of the present disclosure;

[0039] FIG11 is a flowchart of a method for decoding point cloud geometric data according to an embodiment of the present disclosure;

[0040] FIG12 is a schematic diagram of a network structure of a probability predictor according to an embodiment of the present disclosure;

[0041] FIG13A is a schematic diagram of voxel occupancy in a second-scale point cloud according to an embodiment of the present disclosure;

[0042] FIG13B is a schematic diagram of voxel occupancy in a third-scale point cloud according to an embodiment of the present disclosure;

[0043] FIG13C is a schematic diagram of the occupancy probability of voxels in the second-scale point cloud obtained after probability prediction according to an embodiment of the present disclosure;

[0044] FIG14 is a schematic diagram of a network structure of a decoder according to an embodiment of the present disclosure;

[0045] FIG15 is a schematic diagram of a point cloud geometric data enhancement device according to an embodiment of the present disclosure.

[0046] Details

[0047] The present disclosure describes multiple embodiments, but the description is exemplary rather than restrictive, and it is obvious to those skilled in the art that there may be more embodiments and implementations within the scope of the embodiments described in the present disclosure.

[0048] In the description of the present disclosure, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment described as "exemplary" or "for example" in the present disclosure should not be interpreted as being more preferred or advantageous than other embodiments. "And / or" in this article is a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. "Multiple" refers to two or more than two. In addition, in order to facilitate the clear description of the technical solutions of the embodiments of the present disclosure, words such as "first" and "second" are used to distinguish between identical or similar items with basically the same functions and effects. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.

[0049] When describing representative exemplary embodiments, the specification may have presented the method and / or process as a specific sequence of steps. However, to the extent that the method or process does not rely on the specific order of the steps described herein, the method or process should not be limited to the steps in the specific order described. As will be understood by those skilled in the art, other sequences of steps are also possible. Therefore, the specific sequence of the steps set forth in the specification should not be interpreted as a limitation to the claims. In addition, the claims for the method and / or process should not be limited to the steps performed in the order written, and those skilled in the art can readily understand that these sequences can vary and still remain within the spirit and scope of the disclosed embodiments.

[0050] Point cloud compression algorithms include geometry-based point cloud compression (G-PCC). Geometry compression in G-PCC is mainly implemented through an octree model and / or a triangle surface model.

[0051] In order to facilitate the understanding of the technical solutions provided by the embodiments of the present disclosure, a flowchart of G-PCC encoding and a flowchart of G-PCC decoding are first provided. It should be noted that the flowchart of G-PCC encoding and the flowchart of G-PCC decoding described in the embodiments of the present disclosure are only for the purpose of more clearly illustrating the technical solutions of the embodiments of the present disclosure, and do not constitute a limitation on the embodiments of the present disclosure. It is known to those skilled in the art that with the evolution of point cloud compression technology and the emergence of new business scenarios, the technical solutions provided by the embodiments of the present disclosure are also applicable to point cloud compression architectures similar to G-PCC. The point cloud compressed by the embodiments of the present disclosure can be a point cloud in a video, but is not limited to this.

[0052] In the point cloud G-PCC encoder framework, the point cloud of the input 3D image model is sliced ​​and each slice is encoded independently.

[0053] As shown in the flow chart of G-PCC encoding in Figure 1, as applied to a point cloud encoder, the point cloud data to be encoded is first divided into multiple slices using slicing. Within each slice, the geometric and attribute information of the point cloud are encoded separately. During the geometric encoding process, the geometric information undergoes coordinate transformation so that the entire point cloud is contained within a bounding box. Quantization then occurs. Quantization primarily serves a scaling purpose. Due to quantization rounding, the geometric information of some point clouds remains the same, allowing for parameter-based decision-making regarding whether to remove duplicate points. This process of quantization and removing duplicate points is also known as voxelization. The bounding box is then partitioned into an octree. In the octree-based geometric information encoding process, the bounding box is divided into eight equal parts. The sub-cubes that are not empty (contain points in the point cloud) are further divided into eight equal parts until the resulting leaf nodes are 1x1x1 unit cubes. The points in the leaf nodes are then arithmetic-coded to generate a binary geometric bitstream, also known as the geometry codestream. In the process of geometric information encoding based on triangle soup (trisoup), octree partitioning must also be performed first. However, unlike the geometric information encoding based on octree, the trisoup does not need to divide the point cloud into unit cubes with a side length of 1x1x1 step by step. Instead, the division stops when the sub-block (block) has a side length of W. Based on the surface formed by the distribution of the point cloud in each block, the surface and the twelve edges of the block are obtained. The vertices are arithmetically encoded (surface fitting is performed based on the intersections) to generate a binary geometric bit stream (i.e., geometric code stream). Vertex is also used to implement the process of geometric reconstruction, and the reconstructed geometric information is used when encoding the attributes of the point cloud.

[0054] During the attribute encoding process, color conversion is performed to convert the color information (i.e., attribute information) from the RGB color space to the YUV color space. The reconstructed geometric information is then used to recolor the point cloud so that the unencoded attribute information corresponds to the reconstructed geometric information. There are two main transformation methods for color information encoding: a distance-based lifting transform that relies on level of detail (LOD) partitioning, and a direct region-adaptive hierarchical transform (RAHT) transformation. Both methods convert color information from the spatial domain to the frequency domain, obtaining high-frequency and low-frequency coefficients. These coefficients are then quantized (i.e., quantized coefficients). Finally, the geometrically encoded data (octree partitioning and surface fitting) is sliced ​​and synthesized with the attribute encoded data (quantized coefficients). The vertex coordinates of each block are then encoded (i.e., arithmetic encoding) to generate a binary attribute bitstream, i.e., the attribute codestream.

[0055] The flow chart of G-PCC decoding shown in Figure 2 is applied to the point cloud decoder. The decoder obtains a binary code stream and independently decodes the geometric bit stream (i.e., geometric code stream) and attribute bit stream in the binary code stream. When decoding the geometric bit stream, the geometric information of the point cloud is obtained through arithmetic decoding, octree synthesis, surface fitting, geometry reconstruction, and inverse coordinate transformation; when decoding the attribute bit stream, the attribute information of the point cloud is obtained through arithmetic decoding, inverse quantization, LOD-based inverse lifting or RAHT-based inverse transformation, and inverse color conversion. The three-dimensional image model of the point cloud data is restored based on the geometric information and attribute information.

[0056] Neural networks and deep learning techniques can also be applied to point cloud geometry compression. Examples include volumetric model compression techniques based on 3D convolutional neural networks (3D CNNs), compression techniques that directly apply point coordinate sets to neural networks based on multi-layer perceptrons (MLPs), compression techniques that use probability estimation and entropy coding of octree node symbols using MLPs or 3D CNNs, and compression techniques based on 3D sparse convolutional neural networks. Point clouds can be divided into sparse and dense point clouds based on point density. Sparse point clouds have a large representation range and sparse distribution in 3D space, making them suitable for representing a scene; dense point clouds have a small representation range and dense distribution, making them suitable for representing an object. The compression performance of these compression techniques on these two types of point clouds often differs significantly, with some performing better on dense point clouds and some performing worse on sparse point clouds.

[0057] In order to improve the performance of neural network-based encoding and decoding methods on sparse point clouds, the embodiments of the present disclosure provide a point cloud geometry encoding and decoding method based on an autoencoder model, which can achieve lossy compression of point clouds.

[0058] The encoding method of the point cloud geometric data of the embodiment of the present disclosure can be applied to the geometric information encoding process of G-PCC as shown in Figure 1, replacing the encoding processing after voxelization is completed (such as octree partitioning, surface fitting, etc.) to obtain a geometric code stream. The decoding method of the point cloud geometric data of the embodiment of the present disclosure can be applied to the geometric information decoding process of G-PCC as shown in Figure 2, replacing the decoding processing of the geometric code stream before the inverse coordinate transformation (such as octree synthesis, surface fitting, etc.) to obtain the reconstructed geometric data of the point cloud. The entropy coding in the encoding method of the embodiment of the present disclosure can use the arithmetic coding method in Figure 1, and the entropy decoding in the decoding method of the embodiment of the present disclosure can use the arithmetic decoding method in Figure 2. However, the encoding and decoding method of the point cloud geometric data of the embodiment of the present disclosure can also be used in other point cloud encoding and decoding processes other than G-PCC.

[0059] A schematic diagram of a method for encoding and decoding point cloud geometric data according to an embodiment of the present disclosure is shown in FIG3 . At the encoding end, the geometric data of the first-scale point cloud is subjected to two voxel downsamplings. The first-scale point cloud may be the original-scale point cloud to be encoded. After the geometric data of the first-scale point cloud is subjected to one voxel downsampling, the geometric data of the second-scale point cloud is obtained; and after the geometric data of the second-scale point cloud is subjected to another voxel downsampling, the geometric data of the third-scale point cloud is obtained. The geometric data of the third-scale point cloud is entropy encoded to generate a geometric code stream. The decoding end can obtain lossless geometric data of the third-scale point cloud through entropy decoding, and it is necessary to obtain reconstructed geometric data of a higher scale (such as the second-scale point cloud and the first-scale point cloud) based on the geometric data of the third-scale point cloud.

[0060] In order to improve the accuracy of reconstructed geometric data at higher scales, the disclosed embodiment enhances the geometric data of low-scale point clouds through an autoencoder model. Specifically, at the encoding end, this embodiment performs voxel downsampling and feature extraction on the geometric data of the second-scale point cloud through the encoder network at least twice to obtain feature data for enhancing the geometric data of the third-scale point cloud. The figure shows that two encoders perform voxel downsampling (step size 2×2×2) and feature extraction once respectively to extract feature data that is truly helpful for reconstruction and reduce the amount of data to be transmitted. The feature data extracted by the neural network is referred to as latent feature data in this article. The feature data output by the encoder network is quantized and entropy encoded and written into the bitstream, or it can be directly written into the bitstream through entropy encoding.

[0061] At the decoding end, entropy decoding yields lossless geometric data of the third-scale point cloud and feature data used to enhance the third-scale point cloud's geometric data. This lossless geometric data serves as the geometric data to be enhanced for the third-scale point cloud. A decoder in the decoder network performs voxel upsampling and feature inference on this feature data. The output feature data is then concatenated with the geometric data to be enhanced for the third-scale point cloud, yielding the enhanced geometric data for the third-scale point cloud, represented in the figure as the third-scale point cloud's geometric data + feature data.

[0062] As shown in Figure 3, after obtaining the enhanced geometric data of the third-scale point cloud, another decoder in the decoder network performs voxel upsampling and feature inference on this enhanced geometric data. The decoder outputs the data through probability prediction and point cloud cropping to obtain the reconstructed geometric data of the second-scale point cloud. After feature enhancement, the reconstructed geometric data of the second-scale point cloud obtained by decoding the geometric data of the third-scale point cloud is closer to the original geometric data of the second-scale point cloud, which can significantly improve decoding performance. The encoder and decoder networks described above belong to the same autoencoder model, and their network parameters are jointly trained.

[0063] The reconstructed geometric data from the second-scale point cloud can be fed into the probabilistic prediction model for voxel upsampling and feature inference, as well as probabilistic prediction and point cloud cropping, to obtain the reconstructed geometric data from the first-scale point cloud. The decoder used for voxel upsampling and feature inference can use the same structure as the decoder in the decoder network, or it can be designed separately.

[0064] The illustrated embodiment does not enhance the reconstructed geometric data of the second-scale point cloud. However, in other embodiments, similar enhancement methods can be used to obtain enhanced geometric data of the second-scale point cloud, which can then be fed into the probabilistic prediction model to obtain reconstructed geometric data of the first-scale point cloud. Whether enhancement is necessary can be determined based on the required cost and the degree of performance improvement, and this disclosure is not limited to this.

[0065] During the encoding and decoding process of point cloud geometric data, the disclosed embodiments can flexibly enhance the geometric data of point clouds of different scales. The training of the autoencoder model can be implemented based on the geometric data of point clouds of two adjacent scales. Unlike other methods, there is no need to design encoding networks and decoding networks for point clouds of all scales and then train them together. This is simple, convenient and has good portability.

[0066] A schematic diagram of a method for encoding and decoding point cloud geometric data according to an embodiment of the present disclosure is shown in FIG4 . In this embodiment, the number of times i of voxel downsampling of the geometric data of the first-scale point cloud is greater than or equal to 3 times, which is more than the embodiment shown in FIG3 . The i+1-th scale point cloud shown in the figure is a point cloud of the minimum scale, and the geometric data of the point cloud is losslessly compressed by entropy coding. In addition to enhancing the geometric data of the minimum scale, i.e., the i+1-th scale point cloud, this embodiment also enhances the reconstructed geometric data of the next smallest scale, i.e., the i-th scale point cloud. The method for enhancing the geometric data of the i+1-th scale point cloud can refer to the method for enhancing the geometric data of the third-scale point cloud in the embodiment shown in FIG3 . The two are the same, and only the numbering of the point cloud and the encoding and decoding network is different. The i-th encoder network (including two encoders) and the i-th decoder network (including two decoders) used therein belong to the i-th autoencoder model.

[0067] As shown in the figure, in order to enhance the reconstructed geometric data of the i-th scale point cloud, the i-1th encoder network (including two encoders) is used at the encoding end to perform M on the geometric data of the i-1th scale point cloud. i-1 The number of times voxel downsampling and feature extraction is performed on the geometric data of the i-th scale point cloud is M, and feature data is obtained for enhancing the geometric data of the i-th scale point cloud. The structures of the i-1th encoder network and the i-th encoder network used here can be the same or different and can be trained separately. i-1 and the number of times M that voxel downsampling and feature extraction are performed on the geometric data of the i-th scale point cloud i The feature data used to enhance the geometric data of the point cloud at the i-th scale is quantized and entropy-coded and written into the geometric code stream, or is entropy-coded and written into the geometric code stream.

[0068] At the decoding end, entropy decoding is performed to obtain the feature data used to enhance the geometric data of the i-th scale point cloud. Simultaneously, voxel upsampling and feature inference are performed on the feature-enhanced geometric data of the i+1-th scale point cloud, along with probability prediction and point cloud cropping. The reconstructed geometric data of the i-th scale point cloud, or the geometric data to be enhanced, is obtained. A decoder in the i-1-th decoder network performs voxel upsampling and feature inference on the feature data used to enhance the geometric data of the i-th scale point cloud. The output feature data is then concatenated with the geometric data to be enhanced at the i-th scale point cloud, yielding the enhanced geometric data of the i-th scale point cloud.

[0069] As shown in Figure 4, after obtaining the geometric data after the i-th scale point cloud enhancement, the geometric data after the i-th scale point cloud enhancement is subjected to voxel upsampling and feature inference, as well as probability prediction and point cloud cropping, through the probability prediction model, and the reconstructed geometric data of the i-1-th scale point cloud is output. In this probability prediction model, another decoder of the i-1-th decoder network is used to perform voxel upsampling and feature inference on the geometric data after the i-th scale point cloud enhancement. The i-1-th decoder network and the i-1-th encoder network belong to the i-1-th autoencoder model. In the illustrated example, the i-1-th decoder network includes two encoders, and the i-1-th encoder network includes two decoders. However, in other embodiments, more encoders can be used to achieve more times of voxel downsampling and feature extraction, and more decoders can be used to achieve more times of voxel upsampling and feature inference.

[0070] This embodiment enhances the geometric data of the minimum-scale point cloud and the next-smallest-scale point cloud. This is merely exemplary. In other embodiments, the geometric data of point clouds at more scales can be enhanced, or the geometric data of the minimum-scale point cloud and point clouds at scales other than the next-smallest scale can be enhanced. The implementation methods are similar and will not be repeated here. The geometric data of the point clouds at which the enhancement is performed can be determined based on the required cost and the degree of performance improvement.

[0071] Taking Figure 3 as an example, when training the autoencoder model, the encoder network and the decoder network are connected as shown in the figure, but the entropy coding and entropy decoding in the figure can be cancelled. The point cloud samples commonly used for deep neural network training can be used, and the training loss function can be set to BCE (Binary Cross Entropy) loss, that is, the cross entropy of the occupancy probability of the voxel in the third-scale point cloud obtained by probability prediction and the actual occupancy symbol of the voxel in the third-scale point cloud.

[0072] The present disclosure also provides a method for encoding point cloud geometric data, as shown in FIG5 , including:

[0073] Step 110 , performing N voxel downsampling on the geometric data of the first-scale point cloud to obtain geometric data of the second-scale point cloud to the N+1-th scale point cloud, where N≥1;

[0074] Step 120: Input the geometric data of the N-th scale point cloud into the N-th encoder network of the N-th autoencoder model for M N Secondary voxel downsampling and feature extraction, outputting feature data for enhancing the geometric data of the point cloud at the N+1 scale, M N ≥2;

[0075] Step 130 : Perform entropy coding on the geometric data of the N+1th scale point cloud and the feature data output by the Nth encoder network.

[0076] In the above step 110, before performing voxel downsampling on the geometric data of the first-scale point cloud, it is necessary to complete voxelization of the point cloud geometric information. After voxelization, the point cloud is presented in the form of a voxel grid. Voxel is the smallest unit in the voxel grid. A point in the point cloud corresponds to an occupied voxel (i.e., a non-empty voxel), and an unoccupied voxel (i.e., an empty voxel) indicates that there is no point at that position. The geometric data of the point cloud can be represented in different ways. For example, the geometric data of the point cloud can be represented by the occupation symbol of the voxel in the point cloud (also known as a placeholder symbol, placeholder, etc.), marking the occupied voxel as 1 and the unoccupied voxel as 0, to obtain a binary symbol sequence. For another example, the geometric data of the point cloud can also be represented in the form of a sparse tensor, arranging the coordinate data of all points in the point cloud in an agreed order. Different representation methods can be converted to each other.

[0077] The encoding method of the point cloud geometric data of the embodiment of the present disclosure is as follows: at the encoding end, the geometric data of the N-th scale point cloud is input into the N-th encoder network of the N-th autoencoder model for M N The process performs voxel downsampling and feature extraction, and outputs feature data used to enhance the geometric data of the N+1th scale point cloud. After entropy coding, this feature data is transmitted to the decoder along with the geometric code stream. This feature data is extracted using the geometric data of the Nth scale point cloud as input, and contains implicit geometric information of higher-scale point clouds that is not covered by the geometric data of the N+1th scale point cloud. This can help the decoder enhance the geometric data of the N+1th scale point cloud, thereby obtaining more accurate reconstructed geometric data of the Nth scale point cloud. This improves the quality of the reconstructed point cloud, and since this feature data has undergone multiple voxel downsampling, the amount of data required to be transmitted is small, which can improve the efficiency of point cloud compression.

[0078] In an exemplary embodiment of the present disclosure, when voxel downsampling is performed on the geometric data of the first-scale point cloud, it can be achieved through a simple pooling method. For example, a maximum pooling layer with a step size of 2×2×2 is used to merge the 8 voxels of the first-scale point cloud into 1 voxel in the second-scale point cloud, thereby achieving one-time voxel downsampling. Each downsampling reduces the size of the point cloud in three dimensions to half of the original size. Between two scale point clouds, the larger one can be called a high-scale point cloud, and the smaller one can be called a low-scale point cloud. Among the point clouds obtained by N times of voxel downsampling, the N+1th scale point cloud is the smallest point cloud with the least amount of data, which can be written into the code stream through entropy coding.

[0079] Please refer to Figure 3. The third-scale point cloud in the figure includes 2×2×1 voxels, the second-scale point cloud includes 4×4×2 voxels, and the first-scale point cloud includes 8×8×4 voxels. The figure only uses solid cubes to show the occupied voxels in the point clouds of each scale. The point cloud shown in Figure 3 is only exemplary, and the actual point cloud usually includes more voxels. There is a certain degree of correlation between the geometric data of the low-scale point cloud and the geometric data of the high-scale point cloud. For example, if an occupied voxel in the low-scale point cloud is surrounded by occupied voxels (such as when the voxel is located in the middle of an object), then after the voxel is decomposed into multiple voxels in the high-scale point cloud, the multiple voxels obtained by decomposition have a greater probability of being occupied voxels. These correlations can be reflected by the features extracted by the neural network.

[0080] In an exemplary embodiment of the present disclosure, before entropy encoding the feature data, the method further includes: quantizing the feature data. Quantization can reduce the codewords required to transmit the feature data, but it also causes certain losses.

[0081] In an exemplary embodiment of the present disclosure, the method further includes: calculating the number K of occupied voxels in the N-th scale point cloud. N Perform entropy coding. The number K N After being entropy-coded and written into the geometric code stream, it can be used for point cloud cropping at the decoding end to improve the accuracy of point cloud cropping.

[0082] In an exemplary embodiment of the present disclosure, the method further includes:

[0083] When N≥2, the geometric data of the j-th scale point cloud is input into the j-th encoder network of the j-th autoencoder model for M j Secondary voxel downsampling and feature extraction, outputting feature data for enhancing the point cloud geometric data of the j+1th scale point cloud;

[0084] Quantize and entropy encode the feature data output by the j-th encoder network, M j ≥2, the value of j is any one or more of {1,2,…,N-1}.

[0085] That is, in this embodiment, when there are point clouds of more than three scales, not only the geometric data of the smallest scale point cloud is enhanced, but also the geometric data of one or more scale point clouds other than the first scale point cloud are enhanced. Please refer to the encoding process shown in Figure 4. For example, when N=4, there are point clouds of five scales, from the first scale to the fifth scale. In addition to enhancing the geometric data of the smallest scale, i.e., the fifth scale point cloud, when the value of j is 3, it means that the geometric data of the fourth scale point cloud is also enhanced. When the value of j is 2, it means that the geometric data of the third scale point cloud is also enhanced. When the values ​​of j are 2 and 3, it means that the geometric data of the third scale point cloud and the fourth scale point cloud are also enhanced, and so on. When the value of j is different, M j The values ​​can be the same or different.

[0086] The encoder network described in this article performs voxel downsampling and feature extraction, which does not mean that the encoder network first performs voxel downsampling and then performs feature extraction. Voxel downsampling can be performed before feature extraction, or after feature extraction, or between multiple feature extractions, and the present disclosure does not impose any limitations on this. Similarly, the decoder network described in this article performs voxel upsampling and feature inference, which does not mean that the decoder network first performs voxel upsampling and then performs feature inference. Voxel upsampling can be performed before feature inference, or after feature inference, or between multiple feature inferences, and the present disclosure does not impose any limitations on this.

[0087] In an exemplary embodiment of the present disclosure, each voxel downsampling and feature extraction includes:

[0088] Performing feature extraction on input data through at least one of a first residual network based on sparse convolution and a first self-attention network;

[0089] Perform a voxel downsampling on the data output by the first residual network or the first self-attention network through a sparse convolutional layer with a stride of 2×2×2;

[0090] Feature extraction is performed on the data output by the sparse convolution layer through at least one of a second residual network and a second self-attention network based on sparse convolution.

[0091] Each time the encoder network of this embodiment performs voxel downsampling and feature extraction, it is performed in a manner of feature extraction, voxel downsampling, and feature extraction.

[0092] In one example of this embodiment, the first residual network and the second residual network include one or more residual layers based on sparse convolution. Each residual layer, as shown in Figure 6, includes three or more branches. Branch one directly outputs the input data, and the other branches perform feature inference on the input data through different numbers of sparse convolution layers. The outputs of the other branches are concatenated and then added to the output of branch one to obtain the output of the residual layer. Figure 6 shows three branches: branch two includes two sparse convolution layers, and branch three includes three sparse convolution layers. Activation functions are provided between adjacent sparse convolution layers.

[0093] In an example of this embodiment, each time the voxel downsampling and feature extraction is implemented by an encoder based on a neural network. As shown in Figure 7, the encoder includes, in sequence: a first sparse convolutional network, a first self-attention network, a first residual network, a sparse convolutional layer with a step size of 2×2×2, a second residual network, a second self-attention network, and a second sparse convolutional network; an activation function is provided between the first sparse convolutional network and the first self-attention network, and between the first residual network and the sparse convolutional layer, and the first sparse convolutional network and the second sparse convolutional network include one or more sparse convolutional layers.

[0094] In an exemplary embodiment of the present disclosure, the first self-attention network and / or the second self-attention network includes one or more self-attention layers, and the processing performed by each self-attention layer includes:

[0095] For each point in the point cloud, find the neighboring points of the point based on the coordinate data of the point, and perform linear transformation on the distance information from the point to the neighboring points to obtain the position feature. The position feature is added to the features of the neighboring points to obtain the aggregated feature after position encoding;

[0096] Performing a first linear transformation on the input feature data to obtain a first vector, performing matrix multiplication on the first vector and a second vector obtained by performing a second linear transformation on the aggregated features, and activating the result to obtain an attention weight for each point in the point cloud relative to its neighboring points;

[0097] The attention weight is multiplied by a third vector to obtain data containing the neighborhood context feature, where the third vector is obtained by performing a third linear transformation on the aggregated feature.

[0098] In an example of this embodiment, as shown in Figure 8, the self-attention layer includes a point cloud neighborhood self-attention layer, a first normalization layer, a linear layer and a second normalization layer connected in sequence. The point cloud neighborhood self-attention layer is used to obtain neighborhood context features in the point cloud space from the input data. The output data of the point cloud neighborhood self-attention layer is added to the input data and then input into the first normalization layer for batch normalization. The result is then input into the linear layer for linear transformation. The output data of the linear layer is added to the input data and then input into the second normalization layer for batch normalization to obtain the output of the self-attention layer.

[0099] In an example of this embodiment, as shown in FIG9 , the point cloud data (input) of the input point cloud neighborhood self-attention layer includes feature data and coordinate C in ∈R n×3 , coordinate C in ∈R n×3 Used to find neighbor points. n is the total number of points in the point cloud, d in is the dimension of the input feature data.

[0100] As shown in the figure, the processing performed by the point cloud neighborhood self-attention layer includes:

[0101] K-nearest neighbor (KNN) search:

[0102] For each point p in the point cloud i , use the K nearest neighbor search algorithm (k nearest neighbor, KNN) to find the k nearest neighbor points {p i1 ,p i2 ,…p ik}, and gather the coordinates C of k neighbor points knn ∈R n×k×3 and features In some algorithms, the point may be counted as one of the k neighboring points of the point. In other algorithms, the point is not counted as a neighboring point of the point. This disclosure does not limit this.

[0103] Positional encoding:

[0104] For each point p i As the center point, find p i k neighbor points {p i1 ,p i2 ,…p ik} and the center point p i The relative distance {dist i1 ,dist i2 ,…dist ik}, get the relative distance information dist knn ∈Rn×k×1 . Then pass through the linear layer W dist dist knn The dimension is mapped from 1 to d in Dimension, the relative position feature and feature F knn Add (i.e. add to feature F knn After that, implement position encoding:

[0105] F′ knn =dist knn W dist +F knn

[0106] in, F′ knn is the aggregated feature after position encoding, Through position encoding, the features are given the perceptual information of the relative positions between corresponding points, and the features of each neighbor point have spatial position information.

[0107] QKV vector generation:

[0108] The input feature data F in Through the linear layer W Q Transform to obtain the Q vector, and transform the position-encoded aggregate feature F′ knn Pass through the linear layer W respectively K and linear layer W V Transform to get K vector and V vector, namely:

[0109] Q=F in W q ,(K,V)=F′ knn ·(W k ,W v )

[0110] in, and Represents three different linear transformations. The Q vector represents the query vector (Query), the K vector represents the vector of the relevance between the queried information and other information (Key), and the V vector represents the vector of the queried information (Value). The above dimension parameter d a and d out Can be equal to d in , such as setting them to 32. a and d out It may not be equal to d in That is, dimension transformation can be performed.

[0111] Attention weight generation and attention-based feature aggregation:

[0112] After obtaining the Q vector, K vector, and V vector, perform matrix multiplication on the Q vector and the K vector. The result is activated by the Softmax activation function, and the attention weight A of each point relative to its neighboring points when it is used as the center point is output. Finally, the attention weight A is multiplied by the V vector to obtain the feature data F of the output point cloud. out .Right now:

[0113]

[0114] Before activating this example, you can also multiply the result of matrix multiplication of the Q vector and the K vector by the scaling factor.

[0115] Compared with the neural network based solely on sparse convolution, the above-mentioned embodiment of the present disclosure can enhance the spatial modeling capability on sparse point clouds by introducing an attention mechanism network. Because the convolution network with a fixed convolution kernel size is difficult to extract effective neighbor features (i.e., features of the neighborhood context) on a sparsely distributed point cloud, the above-mentioned embodiment of the present disclosure introduces a network based on an attention mechanism, which directly obtains k points around the center point based on the k-nearest neighbor algorithm on the point set, and then obtains the attention weight of the center point for other points through the attention mechanism, which can more effectively extract the feature information of the neighborhood context and improve the compression performance on the sparse point cloud.

[0116] An embodiment of the present disclosure provides a method for enhancing point cloud geometric data, as shown in FIG10 , the method comprising:

[0117] Step 210: parse the code stream to obtain feature data for enhancing the geometric data of the i+1 scale point cloud; the feature data is obtained by performing M on the geometric data of the i scale point cloud through the i encoder network. i Subvoxel downsampling and feature extraction, i≥1, M i ≥2;

[0118] The i-th encoder network of this embodiment can be set to cascade M i The encoders are configured to perform one voxel downsampling and feature extraction on the input data. However, in other embodiments, the number of encoders is variable, and a single encoder can also perform multiple voxel downsampling and feature extraction.

[0119] Step 220: The feature data is processed by a partial decoder of the i-th decoder network. i -1 voxel upsampling and feature inference, the output feature data is spliced ​​with the geometric data to be enhanced of the i+1 scale point cloud to obtain the enhanced geometric data of the i+1 scale point cloud;

[0120] The i-th encoder network and the i-th decoder network belong to the i-th autoencoder model.

[0121] In an example of this embodiment, the output feature data includes L i+1 feature data, the reconstructed geometric data of the i+1th scale point cloud includes L i+1 The coordinate data of the points; the splicing is to combine the L i+1 feature data and L i+1 The coordinate data of each point are spliced ​​one by one to obtain L i+1 The coordinates and feature data of each point, L i+1 is the number of points in the i+1th scale point cloud. When the encoder performs voxel downsampling and feature extraction on the geometric data of the i-th scale point cloud, the resulting feature data (e.g., eigenvalues) and geometric data (e.g., point coordinates) are sequentially mapped. By concatenating the two at the decoder, we can obtain the coordinates and feature data for each point in the point cloud. In other words, we can obtain the eigenvalues ​​of each occupied voxel in the point cloud.

[0122] In one example of this embodiment, the i-th encoder network implements voxel downsampling through sparse convolution with a stride of 2×2×2; the i-th decoder network implements voxel upsampling through transposed sparse convolution with a stride of 2×2×2. Voxel downsampling is implemented through sparse convolution in the encoder network, and voxel upsampling is implemented through transposed sparse convolution in the decoder network. The parameters of both sparse convolution and transposed sparse convolution are learnable, which helps improve compression coding performance.

[0123] An embodiment of the present disclosure further provides a method for decoding point cloud geometric data, as shown in FIG11 , comprising:

[0124] Step 310: Parse the bitstream, obtain the geometric data of the N+1th scale point cloud as the geometric data to be enhanced, and perform enhancement according to the point cloud geometric data enhancement method described in any embodiment of the present disclosure to obtain enhanced geometric data of the N+1th scale point cloud, where N≥1.

[0125] In step 320 , the remaining decoders of the Nth decoder network perform voxel upsampling and feature inference on the geometric data of the enhanced point cloud at the N+1th scale. The output data is then subjected to probability prediction and point cloud cropping to obtain the reconstructed geometric data of the Nth scale point cloud.

[0126] The remaining decoders of the Nth decoder network, along with the network for performing probability prediction and point cloud cropping, constitute the Nth probability prediction model. The output of the Nth probability prediction model is the reconstructed geometric data of the Nth scale point cloud. Probability prediction can be achieved by the probability predictor, and point cloud cropping can be achieved by the point cloud cropper.

[0127] The decoding method of the point cloud geometric data of the embodiment of the present disclosure uses the feature data obtained by decoding to enhance the geometric data of the N+1 scale point cloud, and then performs voxel upsampling and feature inference, as well as probability prediction and point cloud cropping based on the enhanced geometric data of the N+1 scale point cloud to obtain the reconstructed geometric data of the N scale point cloud. The feature data is the geometric data of the N scale point cloud. N The data obtained through multiple voxel downsampling and feature extraction contains implicit feature information of the N-th scale point cloud, which can help the decoder reconstruct the geometric data of the N-th scale point cloud more accurately. This improves the quality of the reconstructed point cloud. Furthermore, this feature data undergoes multiple voxel downsampling, requiring less data to be transmitted, which can improve the efficiency of point cloud compression.

[0128] In an exemplary embodiment of the present disclosure, the method further includes:

[0129] When N ≥ 2, the reconstructed geometric data of the N-th scale point cloud is input into the cascaded N-1 probability prediction models, and voxel upsampling and feature inference, probability prediction and point cloud cropping are performed in each of the probability prediction models, and the reconstructed geometric data of the corresponding scale point cloud is output;

[0130] The reconstructed geometric data of the first-scale point cloud is obtained from the output of the last probabilistic prediction model.

[0131] In this embodiment, when the first-scale point cloud is downsampled two or more times, feature enhancement is performed only on the geometric data at the smallest scale, and no enhancement is performed on the reconstructed geometric data at other scales. The reconstructed geometric data at the Nth scale is input into a probabilistic prediction model to obtain the reconstructed geometric data for the N-1th scale point cloud. This reconstructed geometric data at the N-1th scale is then input into a probabilistic prediction model to obtain the reconstructed geometric data for the N-2th scale point cloud, and so on until the reconstructed geometric data for the first-scale point cloud is obtained. This process can be seen in Figure 3 and its related description.

[0132] In an exemplary embodiment of the present disclosure, the method further includes:

[0133] When N ≥ 2, the reconstructed geometric data of the j-th scale point cloud or the enhanced geometric data of the j-th scale point cloud is input into the j-1th probability prediction model, and after performing voxel upsampling and feature inference, probability prediction and point cloud cropping in the j-1th probability prediction model, the reconstructed geometric data of the j-1th scale point cloud is output, where j = 2, 3, ..., N;

[0134] Among them, the geometric data after enhancement of the j-th scale point cloud is obtained by taking the reconstructed geometric data of the j-th scale point cloud as the geometric data to be enhanced of the j-th scale point cloud, and enhancing it according to the point cloud geometric data enhancement method described in any embodiment of the present disclosure.

[0135] In addition to enhancing the geometric data of the minimum scale point cloud, this embodiment can also enhance the reconstructed geometric data of one or more scale point clouds other than the first scale point cloud. The encoding and decoding process shown in Figure 4 is an example of this embodiment, and please refer to Figure 4 and related instructions. When the input data is feature-enhanced geometric data, the remaining decoders in the j-1 decoder network should be used in the j-1 probability prediction model to perform voxel upsampling and feature inference once, and the j-1 encoder network belonging to the same autoencoder model as the j-1 decoder network is used to perform multiple voxel downsampling and feature extraction on the j-1 scale point cloud to obtain feature data for enhancing the geometric data of the j-1 scale point cloud. When the input data is unenhanced reconstructed geometric data, the decoder that performs voxel upsampling and feature inference once in the j-1 probability prediction model can be designed separately.

[0136] In an exemplary embodiment of the present disclosure, the probability prediction is implemented by multiple sparse convolutional layers and sigmod functions. In an example of this embodiment, a probability predictor as shown in Figure 12 can be used to implement probability prediction. The probability predictor includes three sparse convolutional layers, two activation functions (such as ReLU functions) provided between adjacent sparse convolutional layers, and a Sigmod function provided in the last layer. The Sigmod function outputs the occupancy probability of the voxels in the point cloud obtained by inference. The numerical range of the occupancy probability can be limited to between 0 and 1. The sparse convolutional layer can use SConv K1 3 ,S1 3 ,C32, the convolution kernel size in three dimensions is 1, the stride is 1, and the number of channels is 32.

[0137] After obtaining the occupancy probability of a point in a certain scale point cloud, a simple binary classification method can be used to determine the occupied voxels in the point cloud. See Figure 3. Figure 13A shows the occupied voxels in the second scale point cloud in Figure 3, while Figure 13B shows the occupied voxels in the third scale point cloud in Figure 3. Figure 13C is the occupancy probability of the voxels in the second scale point cloud obtained by probability prediction (indicating the probability of occupancy). When using binary classification, voxels with an occupancy probability not less than a set threshold (such as 0.5) can be regarded as occupied voxels, and voxels with an occupancy probability less than a set threshold (such as 0.5) can be regarded as unoccupied voxels, thereby obtaining the reconstructed geometric data of the point cloud. However, using binary classification for point cloud cropping is sometimes not accurate enough.

[0138] To improve the accuracy of point cloud cropping, an exemplary embodiment of the present disclosure provides a method for assisting cropping based on the number of points in a point cloud. At the encoder, the number of points in the point cloud to be cropped at one or more scales is entropy encoded, and at the decoder, this number is used to assist in determining occupied voxels.

[0139] The decoding method of this embodiment further includes: parsing the code stream to obtain the number K of occupied voxels in the N-th scale point cloud N , K N It is also the number of points in the N-th scale point cloud; and point cloud clipping is achieved by the following method: the M voxels obtained by decomposing the same voxel in the N-th scale point cloud obtained after probability prediction are grouped together, and the occupancy probability of the m voxels with the highest occupancy probability in each group of voxels is set to 1, and then the occupancy probabilities of all voxels in the N-th scale point cloud are sorted, and the K voxels with the highest occupancy probability are ranked. N The voxel is determined as the occupied voxel of the N-th scale point cloud, 1≤m <M。

[0140] In one example, the eight voxels obtained by decomposing the same voxel can be grouped together, and the occupancy probabilities of the one, two, or three voxels with the highest occupancy probability in each group are set to 1. When decomposing voxels in a low-scale point cloud, unoccupied voxels do not need to be decomposed, so that at least one of the eight decomposed voxels is 1. In other examples, M can also be equal to other values ​​such as 64. When M is larger, the value of m can be increased accordingly.

[0141] In this embodiment, the occupancy probability of at least one voxel with the highest occupancy probability in each group of voxels is set to 1 before unified sorting, and then the K voxels with the highest occupancy probability are selected based on the number of points in the point cloud as a constraint. N Voxels are occupied voxels, which can significantly improve the accuracy of point cloud cropping.

[0142] In an exemplary embodiment of the present disclosure, each voxel upsampling and feature inference includes:

[0143] Performing feature inference on input data through at least one of a first residual network based on sparse convolution and a first self-attention network;

[0144] Perform a voxel upsampling on the data output by the first residual network or the first self-attention network through a transposed sparse convolutional layer with a stride of 2×2×2;

[0145] Feature inference is performed on the data output by the transposed sparse convolution layer through at least one of a second residual network and a second self-attention network based on sparse convolution.

[0146] Each time the decoder network of this embodiment performs voxel upsampling and feature extraction, it performs the operation in a manner of feature extraction, voxel upsampling, and feature extraction.

[0147] In one example of this embodiment, the first residual network and the second residual network include one or more residual layers based on sparse convolution. Each residual layer, as shown in Figure 6, includes three or more branches. Branch one directly outputs the input data, and the other branches perform feature inference on the input data through different numbers of sparse convolution layers. The outputs of the other branches are concatenated and then added to the output of branch one to obtain the output of the residual layer. Figure 6 shows three branches: branch two includes two sparse convolution layers, and branch three includes three sparse convolution layers. Activation functions are provided between adjacent sparse convolution layers.

[0148] In an example of this embodiment, each time the voxel upsampling and feature extraction is implemented by a decoder based on a neural network, as shown in Figure 14, the encoder includes in sequence: a first sparse convolutional network, a first self-attention network, a first residual network, a transposed sparse convolutional layer with a step size of 2×2×2, a second residual network, a second self-attention network, and a second sparse convolutional network; an activation function is provided between the first sparse convolutional network and the first self-attention network, and between the first residual network and the transposed sparse convolutional layer, and the first sparse convolutional network and the second sparse convolutional network include one or more sparse convolutional layers.

[0149] Although the embodiments of the present disclosure and the above embodiments provide a decoder and encoder structure, there are various neural networks that can implement feature extraction and feature inference, and all of them may be used in the present disclosure. Therefore, the present disclosure is not limited to a specific network structure disclosed herein, and any neural network that can implement feature extraction or feature inference based on sparse convolution can be used. In particular, the network structure that implements feature extraction and feature inference can be the same, which is called feature extraction in the encoding network and feature inference in the decoding network.

[0150] In one example of this embodiment, the first self-attention network and / or the second self-attention network includes one or more self-attention layers, and the processing performed by each self-attention layer includes: obtaining neighborhood context features in the point cloud space from the input data by:

[0151] For each point in the point cloud, find the neighboring points of the point based on the coordinate data of the point, and perform linear transformation on the distance information from the point to the neighboring points to obtain the position feature. The position feature is added to the features of the neighboring points to obtain the aggregated feature after position encoding;

[0152] Performing a first linear transformation on the input feature data to obtain a first vector, performing matrix multiplication on the first vector and a second vector obtained by performing a second linear transformation on the aggregated features, and activating the result to obtain an attention weight for each point in the point cloud relative to its neighboring points;

[0153] The attention weight is multiplied by a third vector to obtain data containing the neighborhood context feature, where the third vector is obtained by performing a third linear transformation on the aggregated feature.

[0154] In one example, referring to FIG8 , the self-attention layer includes a point cloud neighborhood self-attention layer, a first normalization layer, a linear layer, and a second normalization layer connected in sequence. The point cloud neighborhood self-attention layer is used to obtain neighborhood context features in the point cloud space from the input data. The output data of the point cloud neighborhood self-attention layer is added to the input data and then input to the first normalization layer for batch normalization. The result is then input to the linear layer for linear transformation. The output data of the linear layer is added to the input data and then input to the second normalization layer for batch normalization to obtain the output of the self-attention layer. The process of the point cloud neighborhood self-attention layer obtaining neighborhood context features in the point cloud space from the input data can be seen in FIG9 and related descriptions, which will not be repeated here.

[0155] The point cloud encoding and decoding methods provided in some embodiments of the present disclosure can achieve lossy compression of point cloud geometry. By combining an attention mechanism with a convolutional neural network, an autoencoder model and a probability prediction model are constructed. Compared to existing convolution-based structures, the attention mechanism improves the model's feature extraction capabilities and compression performance.

[0156] Some embodiments of the present disclosure propose a probability-based point cloud clipping method for the local density of a point cloud, which can improve the model's ability to restore the local density of a point cloud.

[0157] The encoding and decoding method of the above embodiment of the present disclosure can be used between point clouds of multiple scales, and the compression of each scale is independent of each other, which can realize scale-scalable encoding with strong flexibility.

[0158] The point cloud encoding and decoding method for implementing lossy point cloud geometry compression in this disclosure is compared with the G-PCC point cloud compression scheme, using the BD-rate as the comparison metric. The results are as follows:

[0159] BD-rate gain relative to GPCC Example of the present disclosure Arco_Valentino_Dense_vox12-25% Egyptian_mask_vox12-16% Facade_00009_vox12-61% House_without_roof_00057_vox12-56% Shiva_00035_vox12-44% Staue_Klimt_vox12-43% Average-41%

[0160] The "Arco_Valentino_Dense_vox12" data in the table represents 12-bit point cloud data provided by the GPCC public test conditions. As can be seen from the table above, the disclosed method demonstrates advantages over the G-PCC point cloud compression solution at all bitrates, achieving a 41% average BD-rate improvement over MPEG G-PCC. This demonstrates superior compression performance compared to the MPEG G-PCC method.

[0161] An embodiment of the present disclosure further provides a point cloud geometry code stream, wherein the geometry code stream is obtained according to the encoding method of point cloud geometry data described in any embodiment of the present disclosure, and includes the geometry data of the N+1th scale point cloud and the feature data output by the Nth encoder network.

[0162] An embodiment of the present disclosure also provides a point cloud geometric data enhancement device, as shown in Figure 15, comprising a processor 5 and a memory 6 storing a computer program, wherein the processor 5 can implement the point cloud geometric data enhancement method described in any embodiment of the present disclosure when executing the computer program.

[0163] An embodiment of the present disclosure further provides a point cloud decoder, see Figure 15, which includes a processor and a memory storing a computer program, wherein when the processor executes the computer program, it can implement the decoding method of point cloud geometric data as described in any embodiment of the present disclosure.

[0164] An embodiment of the present disclosure further provides a point cloud encoder, see Figure 15, comprising a processor and a memory storing a computer program, wherein when the processor executes the computer program, it can implement the encoding method of point cloud geometric data as described in any embodiment of the present disclosure.

[0165] An embodiment of the present disclosure further provides a point cloud encoding and decoding system, which includes a point cloud encoder as described in any embodiment of the present disclosure, and a point cloud decoder as described in any embodiment of the present disclosure.

[0166] The processors of the above embodiments of the present disclosure may be general-purpose processors, including central processing units (CPUs), network processors (NPs), microprocessors, etc., or other conventional processors; the processors may also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components; or combinations of the above devices. That is, the processors of the above embodiments may be any processing device or combination of devices that implement the various methods, steps, and logic block diagrams disclosed in the embodiments of the present disclosure. If the embodiments of the present disclosure are partially implemented in software, the instructions for the software may be stored in a suitable non-volatile computer-readable storage medium, and one or more processors may be used to execute the instructions in hardware to implement the methods of the embodiments of the present disclosure.

[0167] The apparatus and system of the above-mentioned embodiments of the present disclosure can be implemented based on computing devices such as terminals or servers. The terminals may include mobile terminals such as mobile phones, tablet computers, laptop computers, PDAs, portable media players (PMPs), navigation devices, wearable devices, smart bracelets, pedometers, etc., as well as fixed terminals such as digital TVs and desktop computers.

[0168] An embodiment of the present disclosure further provides a non-transitory computer-readable storage medium, which stores a computer program, wherein the computer program, when executed by a processor, can implement the point cloud geometric data enhancement method as described in any embodiment of the present disclosure, or can implement the point cloud geometric information decoding method as described in any embodiment of the present disclosure, or can implement the point cloud geometric information encoding method as described in any embodiment of the present disclosure.

[0169] An embodiment of the present disclosure further provides a point cloud clipping method, which is applied to a point cloud decoder and includes:

[0170] Parse the code stream to obtain the number K of occupied voxels in the point cloud to be cropped;

[0171] Determining the occupancy probability of voxels in the point cloud to be cropped;

[0172] The M voxels decomposed from the same voxel in the point cloud to be cropped are divided into a group, and the occupation probability of the m voxels with the highest occupation probability in each group is set to 1. Then, the occupation probabilities of all voxels in the point cloud to be cropped are sorted, and the K voxels with the highest occupation probability are determined as the occupied voxels in the point cloud to be cropped, 1≤m <M<K。

[0173] In an example of this embodiment, m=1 or 2 or 3, and M=8. However, the present disclosure is not limited thereto, and for example, M may also be 64. The larger M is, the larger m may be set.

[0174] This embodiment not only obtains the exact number K of occupied voxels in the point cloud to be cropped through decoding, but also, when sorting probabilities, groups the M voxels decomposed from the same voxel into groups, and sets the occupancy probabilities of the m voxels with the highest occupancy probabilities in each group to 1. Because unoccupied voxels are not subject to probability prediction, and at least one of the decomposed voxels is occupied, this method utilizes the laws of point cloud decomposition and can significantly improve the accuracy of point cloud cropping (i.e., determining occupied voxels in a point cloud).

[0175] In one or more exemplary embodiments, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions or codes on or transmitted via a computer-readable medium and executed by a hardware-based processing unit. The computer-readable medium may include a computer-readable storage medium corresponding to a tangible medium such as a data storage medium, or a communication medium that facilitates the transfer of a computer program from one place to another, such as according to a communication protocol. In this manner, a computer-readable medium may generally correspond to a non-transitory tangible computer-readable storage medium or a communication medium such as a signal or carrier wave. The data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures for implementing the techniques described in this disclosure. A computer program product may include a computer-readable medium.

[0176] By way of example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Furthermore, any connection may also be referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient (transient) media, but rather refer to non-transient tangible storage media. As used herein, disk and optical disk include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, or Blu-ray disc, among others, where disks typically reproduce data magnetically, while optical discs use lasers to reproduce data optically. Combinations of the above should also be included within the scope of computer-readable media.

[0177] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Thus, the term "processor," as used herein, may refer to any of the aforementioned structures or any other structure suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described herein may be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into a combined codec. Furthermore, the techniques may be fully implemented in one or more circuits or logic elements.

[0178] The technical solutions of the embodiments of the present disclosure can be implemented in a wide variety of devices or equipment, including wireless mobile phones, integrated circuits (ICs), or a group of ICs (e.g., chipsets). Various components, modules, or units are described in the embodiments of the present disclosure to emphasize the functional aspects of the devices configured to perform the described techniques, but they do not necessarily need to be implemented by different hardware units. Instead, as described above, the various units can be combined in a codec hardware unit or provided by a collection of interoperable hardware units (including one or more processors as described above) in combination with appropriate software and / or firmware.

Claims

1. A point cloud geometric data enhancement method, applied to a point cloud decoder, comprising: Parse the code stream to obtain feature data used to enhance the geometric data of the point cloud at the i+1th scale; The feature data is processed by the partial decoder of the i-th decoder network M i -1 voxel upsampling and feature inference, the output feature data is spliced ​​with the geometric data to be enhanced of the i+1 scale point cloud to obtain the enhanced geometric data of the i+1 scale point cloud; Where i is an integer greater than or equal to 1, M i is an integer greater than or equal to 2.

2. The method of claim 1, wherein: The feature data is obtained by performing M on the geometric data of the i-scale point cloud through the i-th encoder network. i obtained by subvoxel downsampling and feature extraction; The i-th encoder network and the i-th decoder network belong to the i-th autoencoder model.

3. The method of claim 1, wherein: The output feature data includes L i+1 feature data, the reconstructed geometric data of the i+1th scale point cloud includes L i+1 Coordinate data of each point; The splicing is to connect the L i+1 feature data and L i+1 The coordinate data of each point are spliced ​​one by one to obtain L i+1 The coordinates and feature data of each point, L i+1 is the number of points in the i+1th scale point cloud.

4. A method for decoding point cloud geometric data, applied to a point cloud decoder, comprising: Parsing the bitstream, obtaining geometric data of the point cloud at the N+1th scale as geometric data to be enhanced, and performing point cloud geometric data enhancement according to the method of any one of claims 1 to 3 to obtain enhanced geometric data of the point cloud at the N+1th scale, where N is an integer greater than or equal to 1; The remaining decoders of the Nth decoder network perform voxel upsampling and feature inference on the geometric data after the N+1th scale point cloud enhancement. The output data is then subjected to probability prediction and point cloud cropping to obtain the reconstructed geometric data of the Nth scale point cloud.

5. The method according to claim 4, wherein: The method further comprises: When N≥2, the reconstructed geometric data of the point cloud at the Nth scale is input into the cascaded N-1 probability prediction models, voxel upsampling and feature inference, probability prediction and point cloud cropping are performed once in each of the probability prediction models, and the reconstructed geometric data of the point cloud at the corresponding scale is output; The reconstructed geometric data of the first-scale point cloud is obtained from the output of the last probabilistic prediction model.

6. The method of claim 4, wherein: The method further comprises: When N ≥ 2, the reconstructed geometric data of the j-th scale point cloud or the enhanced geometric data of the j-th scale point cloud is input into the j-1th probability prediction model, and after performing voxel upsampling and feature inference, probability prediction and point cloud cropping in the j-th probability prediction model, the reconstructed geometric data of the j-1th scale point cloud is output, where j = 2, 3, ..., N; The enhanced geometric data of the j-th scale point cloud is obtained by using the reconstructed geometric data of the j-th scale point cloud as the geometric data to be enhanced of the j-th scale point cloud, and performing point cloud geometric data enhancement according to any one of the methods described in claims 1 to 3.

7. The method of claim 4, wherein: The method further includes: parsing the code stream to obtain the number K of occupied voxels in the N-th scale point cloud N ; The point cloud clipping is achieved by: grouping the M voxels obtained by decomposing the same voxel in the N-th scale point cloud obtained after probability prediction into a group, setting the occupancy probability of the m voxels with the highest occupancy probability in each group of voxels to 1, and then sorting the occupancy probabilities of all voxels in the N-th scale point cloud, and setting the K voxels with the highest occupancy probability to 1. N The voxel is determined as the occupied voxel of the N-th scale point cloud, 1≤m <M。 8. The method according to any one of claims 4 to 6, wherein: Each voxel upsampling and feature inference process includes: Performing feature inference on input data through at least one of a first residual network based on sparse convolution and a first self-attention network; Perform a voxel upsampling on the data output by the first residual network or the first self-attention network through a transposed sparse convolutional layer with a stride of 2×2×2; Feature inference is performed on the data output by the transposed sparse convolution layer through at least one of a second residual network and a second self-attention network based on sparse convolution.

9. The method of claim 8, wherein: The first residual network and the second residual network include one or more residual layers based on sparse convolution, each residual layer includes more than three branches, branch one directly outputs the input data, and the other branches perform feature inference on the input data through different numbers of sparse convolution layers. The outputs of the other branches are spliced ​​and then added to the output of branch one to obtain the output of the residual layer.

10. The method of claim 8, wherein: The first self-attention network and / or the second self-attention network includes one or more self-attention layers, and the processing performed by each self-attention layer includes: obtaining neighborhood context features in the point cloud space from the input data by: For each point in the point cloud, find the neighboring points of the point based on the coordinate data of the point, and perform linear transformation on the distance information from the point to the neighboring points to obtain the position feature. The position feature is added to the features of the neighboring points to obtain the aggregated feature after position encoding; Performing a first linear transformation on the input feature data to obtain a first vector, performing matrix multiplication on the first vector and a second vector obtained by performing a second linear transformation on the aggregated features, and activating the result to obtain an attention weight for each point in the point cloud relative to its neighboring points; The attention weight is multiplied by a third vector to obtain data containing the neighborhood context feature, where the third vector is obtained by performing a third linear transformation on the aggregated feature.

11. The method of claim 10, wherein: The self-attention layer includes a point cloud neighborhood self-attention layer, a first normalization layer, a linear layer and a second normalization layer connected in sequence. The point cloud neighborhood self-attention layer is used to obtain neighborhood context features in the point cloud space from the input data. The output data of the point cloud neighborhood self-attention layer is added to the input data and then input into the first normalization layer for batch normalization. The result is then input into the linear layer for linear transformation. The output data of the linear layer is added to the input data and then input into the second normalization layer for batch normalization to obtain the output of the self-attention layer.

12. The method of claim 8, wherein: Each time the voxel upsampling and feature inference are implemented through a decoder based on a neural network, and the decoder includes, in sequence: a first sparse convolutional network, a first self-attention network, a first residual network, a transposed sparse convolutional layer with a stride of 2×2×2, a second residual network, a second self-attention network, and a second sparse convolutional network; an activation function is provided between the first sparse convolutional network and the first self-attention network, and between the first residual network and the transposed sparse convolutional layer, and the first sparse convolutional network and the second sparse convolutional network include one or more sparse convolutional layers.

13. The method of claim 4, wherein: The probability prediction is achieved through multiple sparse convolutional layers and sigmoid functions.

14. A method for encoding point cloud geometric data, applied to a point cloud encoder, comprising: Perform N voxel downsampling on the geometric data of the first-scale point cloud to obtain the geometric data of the second-scale point cloud to the N+1-th scale point cloud, where N ≥ 1; The geometric data of the N-th scale point cloud is input into the N-th encoder network of the N-th autoencoder model for M N Secondary voxel downsampling and feature extraction, outputting feature data for enhancing the geometric data of the point cloud at the N+1 scale, M N ≥2; Entropy encoding is performed on the geometric data of the N+1th scale point cloud and the feature data output by the Nth encoder network.

15. The method of claim 14, wherein: Before entropy encoding the feature data, the method further includes: quantizing the feature data.

16. The method of claim 14, wherein: The method further includes: calculating the number K of occupied voxels in the N-th scale point cloud. N Perform entropy coding.

17. The method of claim 14, wherein: The method further comprises: When N≥2, the geometric data of the j-th scale point cloud is input into the j-th encoder network of the j-th autoencoder model for M j Secondary voxel downsampling and feature extraction, outputting feature data for enhancing the point cloud geometric data of the j+1th scale point cloud; Quantize and entropy encode the feature data output by the j-th encoder network, M j ≥2, the value of j is any one or more of {1,2,…,N-1}.

18. The method of claim 14 or 17, wherein: Each time the voxel downsampling and feature extraction includes: Performing feature extraction on input data through at least one of a first residual network based on sparse convolution and a first self-attention network; Perform a voxel downsampling on the data output by the first residual network or the first self-attention network through a sparse convolutional layer with a stride of 2×2×2; Feature extraction is performed on the data output by the sparse convolution layer through at least one of a second residual network and a second self-attention network based on sparse convolution.

19. The method of claim 18, wherein: The first residual network and the second residual network include one or more residual layers based on sparse convolution, each residual layer includes more than three branches, branch one directly outputs the input data, and the other branches perform feature inference on the input data through different numbers of sparse convolution layers. The outputs of the other branches are spliced ​​and then added to the output of branch one to obtain the output of the residual layer.

20. The method of claim 18, wherein: The first self-attention network and / or the second self-attention network includes one or more self-attention layers, and the processing performed by each self-attention layer includes: obtaining neighborhood context features in the point cloud space from the input data by: For each point in the point cloud, find the neighboring points of the point based on the coordinate data of the point, and perform linear transformation on the distance information from the point to the neighboring points to obtain the position feature. The position feature is added to the features of the neighboring points to obtain the aggregated feature after position encoding; Performing a first linear transformation on the input feature data to obtain a first vector, performing matrix multiplication on the first vector and a second vector obtained by performing a second linear transformation on the aggregated features, and activating the result to obtain an attention weight for each point in the point cloud relative to its neighboring points; The attention weight is multiplied by a third vector to obtain data containing the neighborhood context feature, where the third vector is obtained by performing a third linear transformation on the aggregated feature.

21. The method of claim 20, wherein: The self-attention layer includes a point cloud neighborhood self-attention layer, a first normalization layer, a linear layer and a second normalization layer connected in sequence. The point cloud neighborhood self-attention layer is used to obtain neighborhood context features in the point cloud space from the input data. The output data of the point cloud neighborhood self-attention layer is added to the input data and then input into the first normalization layer for batch normalization. The result is then input into the linear layer for linear transformation. The output data of the linear layer is added to the input data and then input into the second normalization layer for batch normalization to obtain the output of the self-attention layer.

22. The method of claim 18, wherein: Each time the voxel downsampling and feature extraction is implemented through an encoder based on a neural network, and the encoder includes, in sequence: a first sparse convolutional network, a first self-attention network, a first residual network, a sparse convolutional layer with a step size of 2×2×2, a second residual network, a second self-attention network, and a second sparse convolutional network; an activation function is provided between the first sparse convolutional network and the first self-attention network, and between the first residual network and the sparse convolutional layer, and the first sparse convolutional network and the second sparse convolutional network include one or more sparse convolutional layers.

23. A point cloud geometry code stream, wherein: The geometric code stream is obtained according to the encoding method of point cloud geometric data as described in any one of claims 14 to 22, including the geometric data of the N+1th scale point cloud and the feature data output by the Nth encoder network.

24. A point cloud geometric data enhancement device, comprising a processor and a memory storing a computer program, wherein: When the processor executes the computer program, it is capable of implementing the point cloud geometric data enhancement method as described in any one of claims 1 to 3.

25. A point cloud decoder comprising a processor and a memory storing a computer program, wherein: When the processor executes the computer program, it is capable of implementing the method for decoding point cloud geometric data as described in any one of claims 4 to 13.

26. A point cloud encoder comprising a processor and a memory storing a computer program, wherein: When the processor executes the computer program, it is capable of implementing the method for encoding point cloud geometric data as described in any one of claims 14 to 22.

27. A point cloud encoding and decoding system, wherein: Comprises the point cloud encoder as claimed in claim 26 and the point cloud decoder as claimed in claim 25.

28. A non-transitory computer-readable storage medium storing a computer program, wherein: When the computer program is executed by a processor, it can implement the point cloud geometric data enhancement method as described in any one of claims 1 to 3, or can implement the point cloud geometric data decoding method as described in any one of claims 4 to 13, or can implement the point cloud geometric data encoding method as described in any one of claims 14 to 22.

29. A point cloud cropping method, applied to a point cloud decoder, comprising: Parse the code stream to obtain the number K of occupied voxels in the point cloud to be cropped; Determining the occupancy probability of voxels in the point cloud to be cropped; The M voxels decomposed from the same voxel in the point cloud to be cropped are divided into a group, and the occupation probability of the m voxels with the highest occupation probability in each group is set to 1. Then, the occupation probabilities of all voxels in the point cloud to be cropped are sorted, and the K voxels with the highest occupation probability are determined as the occupied voxels in the point cloud to be cropped, 1≤m <M<K。 30. The point cloud clipping method according to claim 29, wherein: The m=1 or 2 or 3, and the M=8.

Images