Method, device and readable storage medium for encoding and decoding point cloud information

Abstract

The embodiment of the present application provides a method, device and readable storage medium for encoding and decoding information of a point cloud. The method comprises: obtaining a point cloud including a set of points in a three-dimensional space; dividing the point cloud into a tree structure including multiple nodes at different depths; encoding the geometric information of the node; and encoding the attribute information of the node before the entire point cloud is divided.

Description

[0001] This application claims priority under 35 U.S.C. §119 to U.S. Provisional Patent Application No. 62 / 972,155 filed with the U.S. Patent and Trademark Office on February 10, 2020, and U.S. Patent Application No. 17 / 084,927 filed on October 30, 2020, the disclosures of which are incorporated herein by reference in their entirety. Technical Field

[0002] The present application relates to video coding and decoding technology, and in particular to a method, device and readable storage medium for coding and decoding point cloud information. Background Art

[0003] Point cloud modeling has been widely used in recent years, for example, in autonomous vehicles for object detection and localization, in geographic information systems (GIS) for mapping, and in cultural heritage projects for visualization and archiving of cultural heritage objects and collections.

[0004] A point cloud contains a set of high-dimensional points, usually three-dimensional (3D), each of which includes 3D position information and additional attributes, such as color, reflectivity, etc. High-dimensional points can be captured via Light Detection and Ranging (LIDAR) in various settings using multiple cameras and depth sensors, and can consist of thousands to billions of points, allowing for a realistic representation of the original scene.

[0005] Compression techniques are needed to reduce the amount of data required to represent a point cloud in order to make transmission faster and / or reduce storage. As explained in Non-Patent Document 1 and Non-Patent Document 2, the Moving Picture Experts Group (MPEG), a working group of the International Organization for Standardization (ISO) and the International Electrotechnical Commission (IEC), has created a joint technical committee (JTC 1 / SC 29 / WG 11) and an ad hoc group (MPEG-PCC) to standardize compression techniques for static point clouds and / or dynamic point clouds.

[0006] Non-patent document 1: Application example of point cloud compression, ISO / IEC JTC1 / SC29 WG11, document N16331, Geneva, Switzerland, June 2016.

[0007] Non-Patent Document 2: Requirements for Point Cloud Compression, ISO / IEC JTC1 / SC29 WG11, Document N16330, Geneva, Switzerland, June 2016.

[0008] However, current point cloud compression methods have difficulty achieving low-latency encoding and decoding. Summary of the invention

[0009] The present application relates to point cloud coding (PCC). As described in the present application, a node-based geometry and attribute coding method, device and storage medium for coding and decoding point cloud information are disclosed.

[0010] According to one aspect of the present application, a method for encoding and decoding point cloud information can be executed by at least one processor, and the method includes: obtaining a point cloud including a set of points in three-dimensional space; dividing the point cloud into a tree structure, including multiple nodes at different depths; encoding the geometric information of the node; and encoding the attribute information of the node before the entire point cloud is divided.

[0011] According to an embodiment of the present application, for each node at a given depth among the multiple nodes, the encoding of its geometric information is before the encoding of the attribute information of the node at the given depth, and before the encoding of the geometric information and attribute information of the node at a depth different from the given depth.

[0012] According to an embodiment of the present application, the encoding of geometric information and attribute information may be performed alternately and iteratively.

[0013] According to an embodiment of the present application, the aforementioned method may further include: sending a code stream generated for both the geometric information and the attribute information of the multiple nodes, wherein the generated code stream can be transmitted before the encoding of the geometric information of all nodes among the multiple nodes of the point cloud is completed.

[0014] According to an embodiment of the present application, encoding of at least one of the geometric information of the multiple nodes and the attribute information of the multiple nodes may be performed without considering the respective geometric information and attribute information of the adjacent nodes of the multiple nodes.

[0015] According to an embodiment of the present application, encoding of at least one of the geometric information of the multiple nodes and the attribute information of the multiple nodes may be performed in consideration of each geometric information and attribute information of the adjacent nodes of the multiple nodes.

[0016] According to an embodiment of the present application, before dividing the point cloud into a tree structure, the method may further include determining the sizes of the multiple nodes.

[0017] According to an embodiment of the present application, the encoding of the geometric information of the multiple nodes can be performed using at least one of the following methods: an octree encoding and decoding method with breadth-first traversal, an octree encoding and decoding method with depth-first traversal, a prediction tree-based encoding and decoding method, and a triangle set encoding and decoding method.

[0018] According to an embodiment of the present application, the encoding of the attribute information of the multiple nodes may be performed using at least one of the following methods: a prediction-based attribute encoding and decoding method, a lifting-based attribute encoding and decoding method, and a region-based adaptive hierarchical transform (RAHT) encoding and decoding method.

[0019] According to an embodiment of the present application, at least one of the encoding method of the geometric information and the encoding method of the attribute information can be signaled at a sequence level, a frame level, or a slice level.

[0020] According to an embodiment of the present application, the first encoding code and the second encoding code may be configured to enable the at least one processor to alternately and iteratively encode various geometric information and attribute information.

[0021] According to one aspect of the present application, the computer program code may also include: generating code, which is configured to cause the at least one processor to generate a code stream for both the geometric information and the attribute information of the multiple nodes; and sending code, which is configured to cause the at least one processor to send the generated code stream, wherein the generated code stream can be transmitted before the encoding of the geometric information of all nodes among the multiple nodes of the point cloud is completed.

[0022] According to an embodiment of the present application, the first encoding code and the second encoding code can be configured to enable the at least one processor to encode the respective geometric information and attribute information of the multiple nodes without considering the respective geometric information and attribute information of the adjacent nodes of the multiple nodes.

[0023] According to an embodiment of the present application, the first encoding code and the second encoding code can be configured to enable the at least one processor to encode the respective geometric information and attribute information of the multiple nodes while taking into account the respective geometric information and attribute information of the adjacent nodes of the multiple nodes.

[0024] According to an embodiment of the present application, the computer program code may further include: a determination code configured to cause the at least one processor to determine the sizes of the multiple nodes before the division code is configured to divide the point cloud.

[0025] According to an embodiment of the present application, the first encoding code can be configured to enable the at least one processor to encode the geometric information of the multiple nodes using at least one of the following methods: an octree encoding and decoding method with breadth-first traversal, an octree encoding and decoding method with depth-first traversal, a prediction tree-based encoding and decoding method, and a triangle set encoding and decoding method.

[0026] According to an embodiment of the present application, the second encoding code can be configured to enable the at least one processor to encode the attribute information of the multiple nodes using at least one of the following methods: a prediction-based attribute encoding and decoding method, a lifting-based attribute encoding and decoding method, and a region-based adaptive hierarchical transform (RAHT) encoding and decoding method.

[0027] According to another aspect of the present application, a non-volatile computer-readable storage medium storing instructions may be provided, wherein the instructions may enable at least one processor to execute the aforementioned method.

[0028] Through the embodiments of the present application, the delay of point cloud video encoding and decoding can be reduced, and the compression efficiency can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a diagram of octree partitioning in three-dimensional space according to an embodiment.

[0030] Figure 2 is a diagram of a quadtree partitioning scheme according to an embodiment.

[0031] Figure 3 is a diagram of a binary tree partitioning scheme according to an embodiment.

[0032] Figure 4A is a diagram of a breadth-first traversal scheme according to an embodiment.

[0033] Figure 4B is a diagram of a depth-first traversal scheme according to an embodiment.

[0034] Figure 5 is a diagram of a prediction tree according to an embodiment.

[0035] Figure 6 is an illustration of a direct / forward transformation in a lifting scheme according to an embodiment.

[0036] Figure 7 is a diagram of an inverse transform in a lifting scheme according to an embodiment.

[0037] Fig. 8A is an illustration of a Region Adaptive Hierarchical Transform (RAHT) according to an embodiment.

[0038] Figure 8B is an illustration of a reverse region adaptive hierarchical transform (RAHT) according to an embodiment.

[0039] Fig. 9 is a diagram of an octree partitioning scheme according to an embodiment.

[0040] Fig.10 is a diagram of a node-based geometry and attribute encoding and decoding scheme according to an embodiment.

[0041] Fig.11 is a flow chart of an exemplary process for encoding and decoding information of a point cloud.

[0042] Fig.12 is a diagram of an exemplary environment in which the systems and / or methods described herein may be implemented.

[0043] Fig.13 yes Fig. 8A and Figure 8B A diagram of exemplary components of one or more devices. DETAILED DESCRIPTION

[0044] Point cloud compression

[0045] As described in non-patent document 3 and non-patent document 4, in the Test Model 13 (TMC13) codec released by MPEG, the geometric information and associated attributes (such as color or reflectivity) of the points of the point cloud are compressed separately. The geometric information of the points (e.g., the 3D coordinates of the points) is encoded by octree partitioning, quadtree partitioning, and binary partitioning with occupancy information. After the geometric information is encoded, the attributes of the points are then compressed based on the reconstructed geometric information using prediction, lifting, and region-adaptive hierarchical transformation techniques. For geometric coding and decoding, there are generally two methods: (1) an octree-based method, and (2) a prediction tree-based method.

[0046] Non-patent document 3: G-PCC codec description, ISO / IEC JTC1 / SC29 / WG11, document N18891, October 2019.

[0047] Non-Patent Document 4: Text of ISO / IEC CD 23090-9, Geometry-based Point Cloud Compression, ISO / IEC JTC1 / SC29 WG11, Document N18478, Geneva, July 2019.

[0048] Octree-based geometry encoding and decoding

[0049] As disclosed in the present application, octree-based encoding and decoding is a partitioning scheme method, in which the point cloud can be partitioned by octree partitioning, quadtree partitioning or binary partitioning.

[0050] For example, for a given point cloud with a bounding box B, the bounding box B is not necessarily limited to have the same size in all directions. In fact, the bounding box B can be a cuboid of any size to better fit the shape of the 3D scene or 3D object.

[0051] According to an embodiment of the present application, the size of the bounding box B can be expressed as a power of 2, that is, According to the embodiment of the present application, d x ,d y ,d z Not necessarily equal.

[0052] To partition the bounding box B, at least three types of partitioning methods can be used:

[0053] (1) Figure 1 As shown, octree partitioning, where the x-, y-, and z-dimensions of the box can be divided into two halves, resulting in 8 sub-boxes of the same size;

[0054] (2) Figure 2 As shown, quadtree partitioning, in which two dimensions of three dimensions (e.g., x-dimension, y-dimension, z-dimension) are divided into two halves, resulting in four sub-boxes of the same size;

[0055] (3) Figure 3 As shown, binary tree partitioning, in which only one of the three dimensions (eg, x-dimension, y-dimension, z-dimension) is divided into two halves, resulting in two sub-boxes of the same size.

[0056] According to an embodiment, the point cloud can be represented by a general tree structure, with octree partitioning, quadtree partitioning, and binary tree partitioning. In order to traverse such a tree, a breadth-first method can be adopted, as discussed in MPEG TMC13. Some examples are disclosed in Non-Patent Document 3 and Non-Patent Document 4. On the other hand, according to an embodiment, a depth-first method can also be used. This is Figure 4A , Figure 4B , where shaded circles represent occupied nodes in the tree and open circles represent unoccupied nodes.

[0057] like Figure 4A and Figure 4B As shown, the numbers in the circles can represent the traversal order. Figure 4A , a breadth-first traversal order is shown, where nodes are visited / processed starting with depth 0, followed by depth 1, depth 2, and so on. Figure 4B A depth-first traversal order is shown, where nodes are visited / processed starting with the root node, followed by its first occupied child and that child's own first occupied child, until a leaf node is reached.

[0058] Geometric coding and decoding based on prediction tree

[0059] As disclosed in the present application, prediction tree-based geometric coding can use a prediction tree, for example, a tree spanning all points in a point cloud. To predict a point, all of its ancestors can be used. For example, the position of a point can be predicted based on the position of its parent point, or based on the position of its parent point and its grandparent point. For example, Figure 5 Depicted is the prediction tree across the point cloud of the bunny. Specifically, Figure 5 Depicted is a prediction tree for rabbits, with the enlarged block showing a portion of the tree.

[0060] Tri-Soup-Based Geometry Encoding and Decoding

[0061] The present application discloses a technique for using geometric encoding and decoding based on triangle sets. This is a geometric encoding and decoding method that can represent the surface of an object as a series of triangle meshes (one or more). According to an embodiment, the method can be applied to dense surface point clouds. According to an embodiment, a triangle set decoder can generate a point cloud from a mesh surface at a specified voxel granularity to ensure the density of the reconstructed point cloud. According to the disclosed technology, the triangle set method may introduce distortion to the original point cloud, but the advantage is that the size of the code stream can be reduced.

[0062] Prediction-based attribute encoding and decoding

[0063] The present application discloses a technique that can use prediction-based attribute coding. For simplicity, the technique assumes only one level of detail (LoD), however more detailed levels can be used.

[0064] According to the technology of this application, let (P i ) i＝1…N is the set of locations associated with the points of the point cloud, and let (M i ) i＝1…N is the same as (P i ) i＝1…N Here, the points are first sorted in ascending order according to the Morton codes associated with the points. I may be an array of point indices sorted according to the process. According to an embodiment, the encoder / decoder may compress / decompress the points respectively according to the order defined by I. In each iteration i, a point P may be selected. i . We can analyze P i The distance to its s (e.g., s=64) previous points, and P can be selected i k (e.g., k=3) nearest neighbors of point i are used for prediction. More precisely, a linear interpolation process can be used to predict the attribute value (ai ) i∈0…k-1 .make is the set of k nearest neighbors of the current point i, and let are their decoded / reconstructed attribute values, where is their distance to the current point. Then, the predicted attribute value It can be given by the following formula:

[0065]

[0066] Lifting-Based Attribute Encoding and Decoding

[0067] The present application discloses a technique for lifting-based attribute coding, which can be built on the prediction-based coding method. The difference between lifting-based attribute coding and prediction-based coding is that two additional steps can be introduced in lifting-based attribute coding: (1) updating the operator; and (2) using an adaptive quantization strategy. Figure 6 and Figure 7 The update operation strategy is shown. Specifically, Figure 6 illustrates the direct / forward transformation in the lifting scheme, Figure 7 The inverse transformation in the lifting scheme is illustrated.

[0068] Attribute encoding and decoding based on Region Adaptive Hierarchical Transform (RAHT)

[0069] This application discloses techniques that may use RAHT. Fig. 8A The positive RAHT transformation is shown, Figure 8B The inverse RAHT transformation is shown. Fig. 8A and Figure 8B The following equation can be used for transformation: and Where w0 can be the input coefficient F l+1,2n The sign of the weight, and w1 is the input coefficient F l+1,2n+1 The sign of the weight.

[0070] As described in the aforementioned non-patent document 3 and non-patent document 4, in the Test Model 13 (TMC13) codec released by MPEG, attribute coding and decoding is started after the geometric coding and decoding of the entire point cloud is completed. However, this may make it difficult to achieve low-latency coding and decoding. The following discloses a technology, method, and device that solves this problem and improves latency.

[0071] Non-patent document 3: G-PCC codec description, ISO / IEC JTC1 / SC29 / WG11, document N18891, October 2019.

[0072] Non-Patent Document 4: Text of ISO / IEC CD 23090-9, Geometry-based Point Cloud Compression, ISO / IEC JTC1 / SC29 WG11, Document N18478, Geneva, July 2019

[0073] The methods and devices proposed in this application can be used alone or in combination in any order. Further, each of the method (or embodiment), encoder and decoder can be implemented by a processing circuit (e.g., at least one processor or at least one integrated circuit). In one example, at least one processor executes a program stored in a non-volatile computer-readable medium. Further, the disclosure of this application is not limited to use cases related to TMC13 software or MPEG-PCC standards.

[0074] Node-based geometry and attribute encoding and decoding

[0075] The present application discloses techniques that can use tree-based geometric encoding and decoding. According to the present application, a point cloud can be represented as a general tree structure, for example, including not only octree partitioning, but also quadtree partitioning and binary tree partitioning. The root of the tree can contain the entire volume of the point cloud, and the intermediate nodes of the tree can contain sub-volumes (or subtrees) of the point cloud. According to the present application, the following annotations can be used:

[0076] The root node is at depth 0 in the tree;

[0077] After one level of partitioning, the resulting node can be at depth 1 of the tree;

[0078] After k levels of partitioning, the resulting nodes may be at depth k in the tree until all nodes are unit nodes, eg, their size may be 1 in all three dimensions.

[0079] Fig. 9The diagram illustrates an octree and the corresponding depths of the nodes in the tree. According to the present application, in some embodiments, instead of encoding the attributes after the geometric encoding is completed, the geometric information of the point cloud is first encoded until the depth k is reached, where k is specified by the encoder and sent in the bitstream. For each occupied node at depth k (which can be regarded as a subvolume (or subtree) of the point cloud), the geometric information of all points in the node (subtree) can be encoded first, and then the attributes of all points in the node can be encoded. In the same or another embodiment, the geometric encoding and decoding and attribute encoding and decoding of all nodes in the node (subtree) can be performed in an interlaced manner. In either of the aforementioned two methods, the node (subtree) at depth k can be regarded as a top-level coding unit. This concept can be similar to the largest coding unit (LCU) used in the High Efficiency Video Coding (HEVC) standard. According to an embodiment, each node at depth k forms a separate tree and can be regarded as an LCU, such as Fig.10 shown. Fig.10 The figure shows node-based (LCU-based) geometry and attribute encoding and decoding. Fig.10 In , a quadtree is used (eg, for simplicity of illustration purposes) and k is set equal to 1.

[0080] According to the present application, for example, for simplicity, a node at a depth k may be referred to as an LCU, and the terms may be used interchangeably.

[0081] According to an embodiment, a code stream generated for both the geometric characteristics and attributes of each node can be sent without waiting for the geometric encoding of the entire point cloud to be completed. Also according to an embodiment, a decoder can decode all points in the corresponding node and can display these points without waiting for the decoding of the geometric characteristics of the entire point cloud to be completed. In this way, low-latency encoding and decoding can be achieved.

[0082] According to an embodiment, the occupied nodes at depth k (LCU) may be coded and decoded in Morton order.According to the same or another embodiment, for example, in addition to Morton code, the occupied nodes at depth k may be coded and decoded in other space filling orders.

[0083] According to an embodiment, the coding and decoding of the geometric characteristics and attributes of an LCU may depend on the information of its neighboring LCUs. For example, the coding and decoding of the geometric characteristics and attributes of an LCU does not have to depend on the information of its neighboring LCUs. To this end, prediction / reference across LCU boundaries may be disabled, and context and history information may need to be reinitialized for each LCU. This can achieve maximum parallelism at depth k, for example, LCU-level parallel encoding and decoding.

[0084] In another or the same embodiment, the coding and decoding of the geometric characteristics and attributes of the LCU may rely on the information of the coded and decoded neighboring nodes and the coded and decoded child nodes of the neighboring nodes, so as to achieve better compression efficiency.

[0085] Decisions about node size

[0086] According to the present application, in order to encode and decode based on the geometry and attributes of the node, the point cloud encoder may need to decide the node size. The node size may need to be signaled in the high-level syntax. The decision about the node size can be made on the encoder side based on different information.

[0087] According to an embodiment, the node size may be determined based on the number of points in the node. For example, when the number of points in the node is less than or greater than a threshold, node-based geometry and attribute encoding and decoding may be enabled.

[0088] In the same or another embodiment, node size can be determined by node density, where node density is defined as follows:

[0089]

[0090] The node density may be a ratio of the number of points in a node to the volume of the node. For example, when the node density is less than or greater than a threshold, node-based geometry and attribute encoding and decoding may be enabled.

[0091] Node-based quantization

[0092] According to the present application, quantization can be applied to nodes to reduce the number of points to be encoded or to make the points within a node denser to achieve more effective compression. The details of the node-based quantization scheme and corresponding signaling are disclosed below.

[0093] Decisions for different geometry encoding and decoding modes

[0094] According to the present application, for node-based geometry and attribute encoding and decoding, it may be necessary to decide which geometry encoding and decoding method (or referred to as geometry method) to use from multiple options including but not limited to the following:

[0095] Geometric Approach 1: Octree Encoding and Decoding with Breadth-First Traversal

[0096] Geometric Approach 2: Octree Encoding and Decoding with Depth-First Traversal

[0097] Geometric method 3: Encoding and decoding based on prediction trees

[0098] Geometric Method 4: Encoding and Decoding Triangle Sets

[0099] According to an embodiment, in order to reduce the complexity of the encoder / decoder, only a subset of the above four geometric coding methods may be used, for example, using geometric method 1 and geometric method 3. In the same or another embodiment, the geometric coding method may be fixed to one of the four options, for example, to save selection costs.

[0100] According to an embodiment, the geometry encoding and decoding mode may be determined based on heuristics. For example, for relatively dense point clouds, an octree-based encoding and decoding method generally works well, while for relatively less dense point clouds, such as point clouds generated via LIDAR (e.g., used in autonomous vehicles), a prediction tree-based method works well. Further, a triangle set encoding and decoding method may be more suitable for dense surface point clouds.

[0101] According to the same or another embodiment, rate distortion optimization (RDO) can be used to determine the best geometric coding mode. In the technology disclosed in this application, the code stream size and the resulting distortion can be estimated, for example, based on the measurement of D1-PSNR, and the total cost can be calculated as follows:

[0102] total_cost(g)=distortion(g)+λ×bitstream_size(g)

[0103] Here, λ is a weight factor used to balance distortion and bitstream size, and g is the index of the geometric codec method, g = 1, 2, 3, 4, etc. After calculating the total cost of all geometric codec methods, a geometric codec method with the minimum cost can be selected as the selected codec method, as shown below:

[0104]

[0105] Here, g* is the index of the best geometry codec method for a particular node. It should be noted that octree-based codecs themselves do not necessarily introduce distortions in geometry codecs, whereas triangle set codecs may cause distortions.

[0106] For predictive coding, according to an embodiment, the difference between the position of a point and its predicted value can be found without quantization. In this case, no geometric distortion is introduced. In the same or another embodiment, the difference between the position of a point and its predicted value can be quantized, and the difference can be quantized and encoded. In this case, geometric distortion may be introduced.

[0107] Decisions on different attribute encoding and decoding modes

[0108] According to the present application, in addition to deciding on the geometry encoding and decoding mode, it may also be necessary to decide which attribute encoding and decoding method (or attribute method) to use from multiple options. The options include but are by no means limited to the following:

[0109] Attribute method 1: prediction-based attribute encoding and decoding

[0110] Attribute method 2: promotion-based attribute encoding and decoding

[0111] Attribute method 3: Region-based Adaptive Hierarchical Transform (RAHT)

[0112] Depending on the embodiment, only a subset of the attribute codec methods may be used, for example, only attribute method 1 and attribute method 2. In the same or another embodiment, only a single attribute codec method may be used, for example, to reduce encoder / decoder complexity, although sometimes at the expense of some compression efficiency.

[0113] According to an embodiment, the attribute encoding and decoding method can be selected based on heuristics. For example, the prediction-based attribute encoding and decoding method and the lifting-based attribute encoding and decoding method usually work well in relatively dense point clouds, while RAHT usually works well in sparse point clouds.

[0114] In the same or another embodiment, a rate-distortion optimization (RDO) method can be used to select the best attribute codec method, for example, the best attribute codec method has the smallest RDO cost. In this method, the code stream size and corresponding distortion, for example, as measured by PSNR, can be estimated for all points in the node, and the total cost can be calculated as follows:

[0115] total_cost(a)=distortion(a)+λ×bitstream_size(a)

[0116] Here, λ is a weight factor that can be used to balance distortion and bitstream size, and a is the index of the attribute codec method, a=1, 2, 3, etc. After calculating the total cost of all attribute codec methods, the attribute codec method with the lowest cost can be selected as the optimal codec method, as shown below:

[0117]

[0118] (a* is the index of the best attribute encoding / decoding method to use for this node.)

[0119] Node size signaling

[0120] According to the present application, the size of the node for node-based geometry and attribute coding can be signaled at the sequence level, frame level or slice level. The signaling can be introduced in the sequence parameter set, geometry parameter set or slice header.

[0121] The methods, apparatus and techniques disclosed herein are not limited to the MPEG-PCC standard. However, the MPEG-PCC presentation signaling method may be used as follows.

[0122] According to an embodiment, the size of a node for node-based geometry and attribute coding may be specified by gps_geometry_attribute_coding_nodesize_log2_minus1 in the geometry parameter set in TMC13 as follows:

[0123] Table 1: Node sizes

[0124] geometry_parameter_set(){ Descriptors gps_geom_parameter_set_id ue(v) … … log2_trisoup_node_size ue(v) gps_geometry_attribute_coding_nodesize_log2_minus1 ue(v) … … byte_alignment() }

[0125] Here, gps_geometry_attribute_coding_nodesize_log2_minus1 specifies the size of a node for node-based geometry and attribute encoding as follows: geometry_attribute_coding_node_size=2^(1+gps_geometry_attribute_coding_nodesize_log2_minus1). If geometry_attribute_coding_node_size is equal to the size of the bounding box of the point cloud, then the node can be the entire point cloud.

[0126] In the same or another embodiment, the octree depth may be as follows, and all nodes at that depth may have node-based geometry and attribute encodings:

[0127] Table 2: Node Depth

[0128] geometry_parameter_set(){ Descriptors gps_geom_parameter_set_id ue(v) … … log2_trisoup_node_size ue(v) gps_geometry_attribute_coding_node_depth_minus1 ue(v) … … byte_alignment() }

[0129] Here, (gps_geometry_attribute_coding_node_depth_minus1+1) specifies the octree depth at which all nodes at a particular depth will have node-based geometry and attribute encodings, where M is the total depth of the point cloud.

[0130] In the same or another embodiment, at each octree depth starting from depth 0, a flag geometry_attribute_coding_node_presentflag is specified. The flag is set to 0 until depth = (gps_geometry_attribute_coding_node_depth_minus1+1) is reached, at which the flag is set to 1. For all depths > (gps_geometry_attribute_coding_node_depth_minus1+1), the flag may not be signaled.

[0131] Signaling of Geometry Codec Mode and Attribute Codec Mode

[0132] According to an embodiment, at least one of the geometric coding mode and the attribute coding mode may be signaled at the sequence level, the frame level, or the slice level. The signaling may be introduced in a sequence parameter set, a geometric parameter set, or a slice header. Using the geometric parameter set as an example, the signaling may be shown in Table 3 below:

[0133] Table 3: Signaling

[0134] geometry_parameter_set(){ Descriptors gps_geom_parameter_set_id ue(v) … … log2_trisoup_node_size ue(v) gps_geometry_attribute_coding_nodesize_log2_minus1 ue(v) gps_geometry_coding_mode u(2) gps_attribute_coding_mode… u(2)… byte_alignment() }

[0135] Here, gps_geometry_coding_mode can specify the geometry encoding and decoding method used for node-based geometry and attribute encoding and decoding, as shown in Table 4 below:

[0136] Table 4: Encoding and decoding methods

[0137] gps_geometry_coding_mode Geometric encoding and decoding method 0 Octree encoding and decoding method with breadth-first traversal 1 Octree encoding and decoding method with depth-first traversal 2 Prediction Tree Encoding and Decoding Method 3 Triangle set encoding and decoding method

[0138] Here, gps_attribute_coding_mode specifies the attribute encoding and decoding method used for node-based geometry and attribute encoding and decoding, as shown in Table 5 below:

[0139] Table 5: Encoding and decoding methods

[0140] gps_attribute_coding_mode Attribute encoding and decoding methods 0 Prediction-based coding and decoding method 1 Encoding and decoding method based on lifting 2 RAHT method

[0141] Here, if the signaled geometry coding method and attribute coding method are specified in the sequence parameter set or geometry parameter set, respectively, they are applied to all nodes of the specified node size (e.g., gps_geometry_attribute_coding_nodesize_log2_minus1+1) in the sequence or frame;

[0142] On the other hand, according to an embodiment, if the signaled geometry codec method and attribute codec method are specified in the slice header, they may be used for all nodes of the specified node size.

[0143] Note that according to an embodiment, u(2) represents a fixed length binarization of 2 bits with bypass codec. According to the same or another embodiment, gps_geometry_coding_mode and gps_attribute_coding_mode may be binarized using ue(v) and encoded with bypass codec. When ue(v) is used, the mapping between gps_geometry_coding_mode and its corresponding geometry codec method may be changed to reflect that more likely modes have smaller mode values. Similar principles may be applied to gps_attribute_coding_mode and its corresponding attribute codec method.

[0144] In the same or another embodiment, the geometry codec mode and the attribute codec mode may be specified as follows:

[0145] Table 6: Codec methods

[0146] geometry_parameter_set(){ Descriptors gps_geom_parameter_set_id ue(v) … … log2_trisoup_node_size ue(v) gps_geometry_attribute_coding_node_depth_minus1 ue(v) gps_geometry_coding_mode u(2) gps_attribute_coding_mode u(2) byte_alignment() }

[0147] Here, if the signaled geometry coding method and attribute coding method are specified in the sequence parameter set or the geometry parameter set, respectively, they are applied to all nodes at the specified octree depth (e.g., gps_geometry_attribute_coding_node_depth_minus1+1) in the sequence or frame; on the other hand, according to an embodiment, if they are specified at the stripe header, they can be applied to all nodes at the specified octree depth in the stripe.

[0148] Please note that according to an embodiment, the geometry parameter set and the attribute parameter set may be separate, because the geometry encoding and decoding and the attribute encoding and decoding may be performed separately. As disclosed in the present application, node-based geometry encoding and decoding and attribute encoding and decoding are proposed. According to an embodiment, the two parameter sets of the geometry parameter set and the attribute parameter set may be merged into one set. However, also according to some embodiments, a new syntax element may be used in this case, as follows:

[0149] gps_geometry_attribute_coding_node_depth_minus1;

[0150] gps_geometry_attribute_coding_nodesize_log2_minus1;

[0151] gps_geometry_coding_mode;

[0152] gps_attribute_coding_mode.

[0153] As disclosed in this application, in a geometry parameter set, these syntax elements can be applied to both geometry codec and attribute codec.

[0154] In the same or another embodiment, gps_geometry_coding_mode and gps_attribute_coding_mode may be signaled at a specified octree depth, e.g., gps_geometry_attribute_coding_node_depth_minus1+1, or at a specified octree node with a node size equal to gps_geometry_attribute_coding_nodesize_log2_minus1+1. In this way, node-level geometry coding and attribute coding adaptation may be achieved. If the number of geometry coding methods is limited to only 1, then at least gps_geometry_coding_mode may not be signaled, but may be inferred. Similarly, if the number of attribute coding methods is limited to only 1, then at least gps_attribute_coding_mode may not be signaled, but may be inferred.

[0155] Fig.11 is a flow chart of an exemplary process 1100 for encoding and decoding information of a point cloud. Fig.12 As shown, in some embodiments, Fig.11 At least one process block of may be performed by user device 1210. In some embodiments, Fig.11 At least one process block of may be performed by another device or a group of devices separate from or including the user device 1210, for example, the platform 1220.

[0156] like Fig.11 As shown, process 1100 may include acquiring a point cloud comprising a set of points in three-dimensional space (block 1110 ).

[0157] A point cloud can be a set of points in 3D space, each with associated attributes such as color, material properties, etc.

[0158] like Fig.11As further shown in , process 1100 may include dividing the point cloud into a tree structure, including multiple nodes at different depths (box 1120), encoding geometric information of the nodes (box 1130), and encoding attribute information of the nodes before the entire point cloud is divided (box 1140).

[0159] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in the aforementioned various optional implementations.

[0160] Fig.12 1 is a diagram of an exemplary communication system 1200 in which the systems and / or methods described herein may be implemented. Fig.12 As shown, communication system 1200 may include user equipment 1210, platform 1220, and network 1230. The devices of communication system 1200 may be interconnected via wired connections, wireless connections, or a combination of wired and wireless connections.

[0161] The communication system 1200 may support unidirectional transmission of data. For example, the first user device 1210 may encode video data at a local location to be sent to the second user device 1210 via the network 1230. The second user device 1210 may receive the encoded video data sent by the first user device 1210 from the network 1230, decode the encoded data and display the recovered video data. Unidirectional data transmission is common in media service applications and the like.

[0162] The communication system 1200 may support bidirectional transmission of data. For example, the communication system 1200 may support bidirectional transmission of encoded video, such as may occur during a video conference. For bidirectional transmission of data, each user device 1210 may encode video data captured at a local location for transmission to another user device 1210 via the network 1230. Each user device 1210 may also receive encoded video data sent by another user device 1210, may decode the encoded data, and may display the recovered video data on a local display device.

[0163] User device 1210 includes one or more devices that can receive, generate, store, process and / or provide information related to platform 1220. For example, user device 1210 may include a computing device (e.g., a desktop computer, a laptop computer, a tablet computer, a handheld computer, a smart speaker, a server, etc.), a mobile phone (e.g., a smart phone, a wireless phone, etc.), a wearable device (e.g., smart glasses or a smart watch), or a similar device. In some embodiments, user device 1210 can receive information from platform 1220 and / or send information to platform 1220.

[0164] The platform 1220 includes one or more devices as described elsewhere in this application that can provide information to the user device 1210. In some embodiments, the platform 1220 may include a cloud server or a cloud server group. In some embodiments, the platform 1220 can be designed to be modular so that software components can be swapped in or out according to specific needs. In this way, the platform 1220 can be easily and / or quickly reconfigured to have different uses.

[0165] In some embodiments, as shown, the platform 1220 may be hosted in a cloud computing environment 1222. It is worth noting that although the embodiments described herein describe the platform 1220 as being hosted in a cloud computing environment 1222, in some embodiments, the platform 1220 is not cloud-based (i.e., may be implemented outside of a cloud computing environment) or may be partially cloud-based.

[0166] Cloud computing environment 1222 includes an environment hosting platform 1220. Cloud computing environment 1222 can provide computing, software, data access, storage, and other services that do not require end users (e.g., user devices 1210) to be aware of the physical location and configuration of the systems and / or devices hosting platform 1220. As shown, cloud computing environment 1222 can include a set of computing resources 1224 (collectively, “computing resources 1224” and individually, “computing resource 1224”).

[0167] The computing resources 1224 include one or more personal computers, workstation computers, server devices, or other types of computing and / or communication devices. In some embodiments, the computing resources 1224 can host the platform 1220. Cloud resources can include computing instances executed in the computing resources 1224, storage devices provided in the computing resources 1224, data transmission devices provided by the computing resources 1224, etc. In some embodiments, the computing resources 1224 can communicate with other computing resources 1224 via wired connections, wireless connections, or a combination of wired and wireless connections.

[0168] Further Fig.12 As shown, the computing resources 1224 include a set of cloud resources, such as one or more application programs ("APP") 1224-1, one or more virtual machines ("VM") 1224-2, virtualized storage ("VS") 1224-3, one or more hypervisors ("HYP") 1224-4, etc.

[0169] The applications 1224-1 include one or more software applications that can be provided to or accessed by the user device 1210. The applications 1224-1 do not require the software applications to be installed and executed on the user device 1210. For example, the applications 1224-1 may include software associated with the platform 1220, and / or any other software that can be provided through the cloud computing environment 1222. In some embodiments, one application 1224-1 can send / receive information to or from one or more other applications 1224-1 through the virtual machine 1224-2.

[0170] Virtual machine 1224-2 includes a software implementation of a machine (e.g., a computer) that executes programs, similar to a physical machine. Virtual machine 1224-2 can be a system virtual machine or a process virtual machine, depending on the use and correspondence of virtual machine 1224-2 to any real machine. System virtual machines can provide a complete system platform that supports the execution of a complete operating system ("OS"). Process virtual machines can execute a single program and can support a single process. In some embodiments, virtual machine 1224-2 can execute on behalf of a user (e.g., user device 1210) and can manage the infrastructure of cloud computing environment 1222, such as data management, synchronization, or long-term data transfer.

[0171] Virtualized storage 1224-3 includes one or more storage systems and / or one or more devices that use virtualization technology within the storage system or device of computing resource 1224. In some embodiments, within the context of a storage system, the types of virtualization may include block virtualization and file virtualization. Block virtualization may refer to the abstraction (or separation) of logical storage from physical storage so that the storage system may be accessed without regard to physical storage or heterogeneous structures. Separation may allow administrators of storage systems to flexibly manage storage for end users. File virtualization may eliminate the dependency between data accessed at the file level and the location of the physical storage file. This may optimize the performance of storage usage, server consolidation, and / or non-disruptive file migration.

[0172] Hypervisor 1224-4 may provide hardware virtualization technology that allows multiple operating systems (e.g., "guest operating systems") to execute simultaneously on a host computer such as computing resource 1224. Hypervisor 1224-4 may provide a virtual operating platform to the guest operating systems and may manage the execution of the guest operating systems. Multiple instances of various operating systems may share virtualized hardware resources.

[0173] The network 1230 includes one or more wired and / or wireless networks. For example, the network 1230 may include a cellular network (e.g., a fifth generation (5G) network, a long-term evolution (LTE) network, a third generation (3G) network, a code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., a public switched telephone network (PSTN)), a private network, an ad hoc network, an intranet, the Internet, a fiber-based network, etc., and / or a combination of these or other types of networks.

[0174] Fig.12 The number and arrangement of devices and networks shown are provided as examples. Fig.12 There may be more devices and / or networks, fewer devices and / or networks, different devices and / or networks, or different arrangements of devices and / or networks than those shown. Fig.12Two or more of the devices shown may be implemented in a single device, or Fig.12 The single device shown may be implemented as multiple distributed devices. Additionally or alternatively, one set of devices (eg, one or more devices) of environment 1200 may perform one or more functions described as being performed by another set of devices of environment 1200.

[0175] Fig.13 1300 is a block diagram of example components of device 1300. Device 1300 may correspond to user device 1210 and / or platform 1220. Fig.13 As shown, device 1300 may include a bus 1310 , a processor 1320 , a memory 1330 , a storage component 1340 , an input component 1350 , an output component 1360 , and a communication interface 1370 .

[0176] The bus 1310 includes components that allow communication between components of the device 1300. The processor 1320 is implemented in hardware, firmware, or a combination of hardware and software. The processor 1320 is a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or another type of processing component. In some embodiments, the processor 1320 includes one or more processors that can be programmed to perform functions. The memory 1330 includes a random access memory (RAM), a read-only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, and / or optical memory) that stores information and / or instructions for use by the processor 1320.

[0177] Storage component 1340 stores information and / or software related to the operation and use of device 1300. For example, storage component 1340 may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optical disk, and / or a solid-state disk), a compact disk (CD), a digital versatile disk (DVD), a floppy disk, a cassette, a magnetic tape, and / or another type of non-volatile computer-readable medium, and a corresponding drive.

[0178] Input components 1350 include components that allow device 1300 to receive information, such as through user input, such as a touch screen display, a keyboard, a keypad, a mouse, buttons, switches, and / or a microphone. Additionally or alternatively, input components 1350 may include sensors for sensing information (e.g., global positioning system (GPS) components, accelerometers, gyroscopes, and / or actuators). Output components 1360 include components that provide output information from device 1300, such as a display, a speaker, and / or one or more light emitting diodes (LEDs).

[0179] The communication interface 1370 includes a transceiver-like component (e.g., a transceiver and / or a separate receiver and transmitter) that enables the device 1300 to communicate with other devices, for example, via a wired connection, a wireless connection, or a combination of wired and wireless connections. The communication interface 1370 can allow the device 1300 to receive information from another device and / or provide information to another device. For example, the communication interface 1370 can include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, etc.

[0180] Device 1300 may perform one or more processes described in this application. Device 1300 may perform these processes in response to processor 1320 executing software instructions stored by non-volatile computer-readable media (e.g., memory 1330 and / or storage component 1340). Computer-readable media is defined in this application as non-volatile memory devices. Memory devices include storage space within a single physical storage device or storage space distributed on multiple physical storage devices.

[0181] The software instructions may be read into the memory 1330 and / or storage component 1340 from another computer-readable medium or from another device through the communication interface 1370. When executed, the software instructions stored in the memory 1330 and / or storage component 1340 may cause the processor 1320 to perform one or more processes described in the present application. Additionally or alternatively, hardwired circuitry may be used in place of or in combination with software instructions to perform one or more processes described in the present application. Therefore, the embodiments described in the present application are not limited to any specific combination of hardware circuitry and software.

[0182] Fig.13 The number and arrangement of components shown are provided as examples. Fig.13 The device 1300 may include more components, fewer components, different components, or components arranged differently than the components shown. Additionally or alternatively, one or more components of the device 1300 may perform one or more functions described as being performed by another set of components of the device 1300.

[0183] The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit the embodiments to the precise form disclosed. Modifications and variations are possible in light of the above disclosure or may be acquired from practice of the embodiments.

[0184] As used in this application, the term component is intended to be broadly interpreted as hardware, firmware, or a combination of hardware and software.

[0185] Obviously, the system and / or method described in the present application can be implemented in different forms of hardware, firmware or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods is not limited to these embodiments. Therefore, the operation and behavior of the system and / or method are described in the present application without reference to specific software codes-it should be understood that software and hardware can be designed based on the description in the present application to implement the system and / or method.

[0186] Although particular combinations of features are set forth in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. In fact, many of these features may be combined in ways not specifically recited in the claims and / or disclosed in the specification. Although a dependent claim may be directly dependent on only one claim, the disclosure of possible implementations includes each dependent claim in combination with every other claim in the claim group.

[0187] Elements, actions or instructions used in this application should not be interpreted as critical or necessary unless explicitly described as such. In addition, as used in this application, the articles "one" and "an" are intended to include one or more projects, and can be used interchangeably with "one or more". Further, as used in this application, the term "set" is intended to include one or more projects (e.g., related projects, unrelated projects, combinations of related projects and unrelated projects, etc.), and can be used interchangeably with "one or more". When only one project is meant, the term "one" or similar language is used. In addition, as used in this application, the terms "have", "have", "contain" etc. are intended to be open terms. Further, the phrase "based on" is intended to mean "based at least in part on", unless otherwise explicitly stated.

Claims

1. A method for encoding and decoding point cloud information, characterized in that: The method comprises: Obtaining a point cloud comprising a set of points in a three-dimensional space; Dividing the point cloud into a tree structure including a plurality of nodes at different depths; Encoding the geometric information of the node; and Before the entire point cloud is divided, the attribute information of the node is encoded.

2. The method according to claim 1, characterized in that For each node at a given depth among the plurality of nodes, encoding of its geometry information precedes encoding of attribute information of the node at the given depth, and precedes encoding of geometry information and attribute information of nodes at depths different from the given depth.

3. The method according to claim 1, characterized in that The encoding of the geometric information and the attribute information is performed alternately and iteratively.

4. The method according to any one of claims 1 to 3, characterized in that: Also includes: sending a code stream generated for both the geometric information and the attribute information of the plurality of nodes, Wherein, before the encoding of the geometric information of all the nodes in the plurality of nodes of the point cloud is completed, the generated code stream can be transmitted.

5. The method according to any one of claims 1 to 3, characterized in that: The encoding of at least one of the geometric information of the multiple nodes and the attribute information of the multiple nodes is performed without considering the respective geometric information and attribute information of the adjacent nodes of the multiple nodes.

6. The method according to any one of claims 1 to 3, characterized in that: The encoding of at least one of the geometric information of the plurality of nodes and the attribute information of the plurality of nodes is performed in consideration of each geometric information and attribute information of adjacent nodes of the plurality of nodes.

7. The method according to any one of claims 1 to 3, characterized in that: Also includes: Before dividing the point cloud into a tree structure, the sizes of the plurality of nodes are determined.

8. The method according to any one of claims 1 to 3, characterized in that: The encoding of the geometric information of the multiple nodes uses at least one of the following methods: an octree encoding method with breadth-first traversal, an octree encoding method with depth-first traversal, a prediction tree-based encoding method, and a triangle set encoding method.

9. The method according to any one of claims 1 to 3, characterized in that: The encoding of the attribute information of the plurality of nodes uses at least one of the following methods: a prediction-based attribute encoding method, a lifting-based attribute encoding method, and a region-based adaptive hierarchical transformation encoding method.

10. The method according to any one of claims 1 to 3, characterized in that: At least one of the encoding method of the geometric information and the encoding method of the attribute information is signaled at a sequence level, a frame level, or a slice level.

11. A device for encoding and decoding point cloud information, characterized in that: The device comprises: at least one memory configured to store program code; and at least one processor configured to access the at least one memory and to operate according to a computer program code comprising: acquiring code configured to cause the at least one processor to acquire a point cloud comprising a set of points in a three-dimensional space; partitioning code configured to cause the at least one processor to partition the point cloud into a tree structure including a plurality of nodes at different depths; A first encoding code configured to cause the at least one processor to encode geometric information of a node; and The second encoding code is configured to enable the at least one processor to encode the attribute information of the node before dividing the entire point cloud.

12. The device according to claim 11, characterized in that For each node at a given depth among the multiple nodes, before the second encoding code causes the at least one processor to encode attribute information of the node at the given depth, and before the first encoding code and the second encoding code respectively cause the at least one processor to encode geometric information and attribute information of the node at a depth different from the given depth, the first encoding code causes the at least one processor to first encode the geometric information of the node at the given depth.

13. The device according to claim 11, characterized in that The first encoding code and the second encoding code are configured to cause the at least one processor to respectively encode geometric information and attribute information alternately and iteratively.

14. The device according to any one of claims 11 to 13, characterized in that The computer program code also includes: generating code configured to cause the at least one processor to generate a code stream for both the geometric information and the attribute information of the plurality of nodes, and a sending code configured to cause the at least one processor to send the generated code stream, Wherein, before the encoding of the geometric information of all the nodes in the plurality of nodes of the point cloud is completed, the generated code stream can be transmitted.

15. The device according to any one of claims 11 to 13, characterized in that The first encoding code and the second encoding code are configured to enable the at least one processor to encode respective geometric information and attribute information of the plurality of nodes without considering respective geometric information and attribute information of neighboring nodes of the plurality of nodes.

16. The device according to any one of claims 11 to 13, characterized in that The first encoding code and the second encoding code are configured to cause the at least one processor to encode respective geometric information and attribute information of the plurality of nodes in consideration of respective geometric information and attribute information of neighboring nodes of the plurality of nodes.

17. The device according to any one of claims 11 to 13, characterized in that The computer program code also includes determining code configured to cause the at least one processor to determine a size of the plurality of nodes before the partitioning code causes the at least one processor to partition the point cloud.

18. The device according to any one of claims 11 to 13, characterized in that The first encoding code is configured to enable the at least one processor to encode the geometric information of the multiple nodes using at least one of the following methods: an octree encoding and decoding method with breadth-first traversal, an octree encoding and decoding method with depth-first traversal, a prediction tree-based encoding and decoding method, and a triangle set encoding and decoding method.

19. The device according to any one of claims 11 to 13, characterized in that The second encoding code is configured to enable the at least one processor to encode the attribute information of the multiple nodes using at least one of the following methods: a prediction-based attribute encoding and decoding method, a lifting-based attribute encoding and decoding method, and a region-based adaptive hierarchical transform encoding and decoding method.

20. A non-volatile computer-readable storage medium storing instructions, characterized in that: The instructions enable at least one processor to encode and decode point cloud information through the method of any one of claims 1 to 10.

21. A method for storing or sending a video code stream, characterized in that: The video code stream can be generated according to the method according to any one of claims 1 to 10.

Images