Method, encoder and decoder for encoding and decoding 3d point clouds

Abstract

Provided herein is a method of decoding a geometry of a 3D point cloud by a bitstream, preferably implemented in a decoder, the method comprising: receiving a bitstream, where the bitstream comprises vertex information comprising information about a position of a vertex on an edge of a cube of leaf nodes of an octree structure of the point cloud; obtaining vertex information to determine a position of a vertex, wherein at least one vertex is adjusted towards an average position of the surface points; a point of the point cloud is determined based on at least one adjusted vertex, where the bitstream further includes plane information indicating whether the point cloud data is a plane, the determination of the point of the point cloud is further based on the obtained plane information.

Description

Technical Field

[0001] The present invention relates to a method, an encoder and a decoder for encoding and decoding a 3D point cloud. Background Art

[0002] Point clouds have recently gained attention as a format for 3D data representation due to their versatility in representing all types of 3D objects or scenes. However, for all compression schemes, the reconstruction quality of the points of the point cloud is crucial. Summary of the invention

[0003] Therefore, an object of the present invention is to provide a method for decoding the geometry of a 3D point cloud from a bitstream and encoding the 3D point cloud into a bitstream, which method improves efficiency.

[0004] This problem is solved by a decoding method according to claim 1 , an encoding method according to claim 2 , an encoder according to claim 8 , a decoder according to claim 9 , a bit stream according to claim 10 and software according to claim 11 .

[0005] In a first aspect, a method for decoding the geometry of a 3D point cloud from a bitstream is provided, the method preferably being implemented in a decoder. The method comprises: receiving a bitstream, wherein the bitstream comprises vertex information, the vertex information comprising information about the positions of vertices on the edges of a cube of a leaf node of an octree structure of a point cloud; obtaining the vertex information to determine the positions of the vertices, wherein at least one vertex is adjusted toward the average position of a surface point; determining a point of the point cloud based on at least one adjusted vertex, characterized in that the bitstream further comprises plane information indicating whether the point cloud data is a plane, and the determination of the point of the point cloud is further based on the obtained plane information.

[0006] Preferably, vertex information is obtained to determine the position of the vertex, wherein at least one vertex is adjusted toward the average position of the surface point, including: determining a virtual position based on the position of the vertices of a cube associated with a leaf node of an octree structure; constructing a triangle based on the vertices and the virtual position of a cube; determining a normal vector of the cube based on the constructed triangle; determining a center of mass position based on the normal vector; and adjusting at least one vertex toward the center of mass position.

[0007] Therefore, in a first step, a bitstream is received, and the bitstream may contain information about an octree structure of a volume of a point cloud that can be decoded. Preferably, the geometric structure of the point cloud is GPCC encoded. Therefore, preferably, by decoding from the bitstream, octree information about the volume of the point cloud can be provided. In addition, the bitstream includes vertex information, which includes the positions of vertices on the edges of a cube associated with a leaf node in the octree structure. Preferably, information about the existence of vertices is also included. Therefore, the vertex information can be provided by receiving and / or decoding a bitstream. Wherein, the bitstream is preferably encoded by a TriSoup encoding scheme at the encoder.

[0008] After decoding the vertex information and preferably also decoding the octree information from the bitstream described in the previous step, in the next step for reconstructing the point cloud geometry, triangles can be determined for each cube. In particular, a virtual position can first be determined based on the vertex positions of a cube associated with the leaf nodes of the octree structure, preferably by averaging the positions of the vertices of a cube. Then, based on the vertices and virtual positions of a cube, a set of triangles is constructed, preferably by connecting two consecutive vertices on the edge of the cube and the determined virtual positions clockwise. Therefore, the surface of the first set of triangles is determined by the vertex positions included in the bitstream and the determined virtual positions (i.e., the average positions of the vertices).

[0009] Subsequently, the normal vector of the cube can be determined based on the constructed triangles. Among them, the normal vector is a vector perpendicular to a given object. Therefore, the normal vector of each triangle in the first group of triangles can be determined separately. Then, the normal vector of the cube can be determined based on the sum of the determined normal vectors of the first group of triangles. Then, the center of mass position can be determined based on the virtual position and the normal vector of the cube. Through this adjustment, the determined center of mass position may be closer to the original position of the point cloud in the leaf node. In addition, according to the present invention, at least one vertex can be adjusted toward the corresponding average position of the surface point. Surface points are points that constitute the outer surface of the 3D point cloud, specifically located on the edge, and the edge converges at the vertex in the original point cloud. The original vertex information is established using the surface points derived from the original point cloud. The surface points are a subset of the 3D point cloud. In particular, the vertex can be related to the target surface, that is, different vertices can correspond to different surfaces. Surface points, such as Fig.14 1405 may be a point corresponding to the target surface. In particular, Fig.14, vertex 1401 may correspond to surface point 1405. Wherein, the corresponding surface point is an original point in the point cloud that is close to the edge to which at least one vertex belongs. Preferably, this refers to a subset of the original points in the point cloud that are located within a distance from its edge that is less than 1 / 4 of the leaf node width. More preferably, less than 1 / 8 (of the leaf node width). Preferably, the at least one vertex can be adjusted toward the determined centroid position to obtain a better modeling of the original point cloud surface. In particular, if the point cloud data is planar according to the plane information, when moving along the vector The vertices on the edge of the axis with the maximum value can be adjusted along the edge they belong to to make the reconstructed surface more natural. From C mean To C. More specifically, we can first determine an axis, the vector has the maximum value along this axis. Then, we can find the vertices on the edges of the leaf nodes that are parallel to the determined axis. Finally, these vertices can be based on Then, according to the adjusted vertices and centroids, triangles are constructed in a manner similar to that described above. Whether the point cloud data is a plane can be determined by various methods or a dedicated plane determination algorithm. For example, the plane information can be obtained based on the characteristics of the input point cloud data, or determined at the encoder based on the percentage of leaf nodes with a centroid residual value of 0, or at the encoder based on the normal vector of each leaf node. The direction is determined automatically.

[0010] In order to reconstruct the points of the point cloud from the reconstructed triangles, voxelization is performed, preferably by a ray tracing process, in which rays are emitted along three directions parallel to any of the three axes. Their origin is a point with integer coordinates corresponding to the sampling accuracy required for rendering. The intersection of the ray with one of the reconstructed triangles (if any) is then determined and added to the list of rendered points, i.e., to the points of the point cloud. In the voxelization process, the surface of the reconstructed triangle is sampled with the rays to determine the points of the point cloud.

[0011] Preferably, the at least one vertex is adjusted only when the difference between the virtual position and the centroid position exceeds a threshold, preferably the threshold is based on the leaf node size, preferably a ratio s of the leaf node size, wherein Preferably, S=1 / 128. Therefore, before executing the method according to the present invention, it is first determined whether the prominence of the modeled surface in the leaf node is large enough. For example, by comparing the centroid residual with a threshold. This constraint reduces unnecessary adjustments, thereby improving computational efficiency and reducing complexity.

[0012] Preferably, at least one vertex is adjusted according to a fixed value, preferably, the fixed value is determined based on the type of point cloud and is encoded into or from the bitstream. In particular, the fixed value can be determined based on statistics of an input point cloud sequence that indicate properties, characteristics or features of a sequence of 3D point clouds (e.g., ordinary or non-ordinary). The statistics of the input point cloud sequence can include various metrics and attributes, such as: Descriptive statistics: These include metrics such as mean, median, mode, variance and standard deviation of point coordinates (x, y, z), as well as other attributes, such as color, intensity or normal vectors. Geometric properties: Features such as point density, spatial distribution or point cloud range can be quantified and summarized. Temporal properties: For a point cloud sequence that changes over time, properties such as motion or deformation of the point cloud can be analyzed and summarized. Topological properties: The connectivity or relationships between points in the point cloud can be studied to provide insights into the underlying structure of the data. This approach provides simplicity and consistency between similar point cloud types.

[0013] Preferably, at least one vertex is adjusted based on the difference between the virtual position and the centroid position, preferably further based on the boundary of the edge of the leaf node. As mentioned above, the vertex can be adjusted along certain determined edges towards the centroid position. In addition, the adjustment can be further based on the boundary of the edge of the leaf node, so that it can be ensured that the adjusted vertex does not exceed the leaf node to which it belongs. Therefore, this technique can perform more accurate vertex adjustment, thereby more accurately presenting the point cloud geometry.

[0014] Preferably, the plane information is encoded into a set of geometric parameters GPS.

[0015] In another aspect of the present invention, a method for encoding a 3D point cloud into a bitstream is provided, which method is preferably implemented in an encoder. The method for encoding a 3D point cloud comprises: obtaining vertex information of each cube associated with a leaf node from a point cloud surface, wherein the vertex information comprises position information of vertices on the edges of the cube of the leaf node of an octree structure of the point cloud; encoding the vertex information into a bitstream, wherein at least one vertex is adjusted toward an average position of a surface point; determining a point of the point cloud based on the encoded vertex information;

[0016] Characterized in that the method further comprises determining and encoding plane information indicating whether the point cloud data is planar, and the determination of the points of the point cloud is further based on the encoded plane information.

[0017] Therefore, by the encoding method, vertex information and preferably octree information are generated. This information can be encoded into a bitstream. Subsequently, on the encoder side, a reconstruction step can be performed. In this reconstruction step, the point cloud geometry is reconstructed, wherein the reconstruction step is the same as the step in the above-mentioned decoding method. Then, the attributes of the points of the point cloud (color, reflectivity...) are encoded on the encoder side using the reconstructed geometry of the point cloud, for example, in sequence through RAHT (Region Adaptive Hierarchical Transform), prediction transform or lifting transform to encode the attributes of the points of the point cloud.

[0018] Preferably, the geometric structure of the point cloud is encoded into the bitstream via Geometry-based Point Cloud Compression (G-PCC).

[0019] Preferably, the bitstream is an MPEG G-PCC compliant bitstream.

[0020] Preferably, the encoding method is further constructed according to the features described above in conjunction with the decoding method.

[0021] In another aspect of the present invention, an encoder for encoding a 3D point cloud into a bit stream is provided. The encoder comprises a memory and a processor, wherein instructions are stored in the memory, and when the instructions are executed by the processor, the steps of the aforementioned encoding method are performed.

[0022] In another aspect of the present invention, a decoder for decoding a 3D point cloud from a bitstream is provided. The decoder comprises a memory and a processor, wherein instructions are stored in the memory, and when the instructions are executed by the processor, the steps of the above-mentioned decoding method are performed.

[0023] In another aspect of the present invention, a bit stream is provided, wherein the bit stream is encoded by the steps of the aforementioned encoding method.

[0024] In another aspect of the present invention, a computer-readable storage medium is provided, which includes instructions for executing the steps of the above method for encoding a 3D point cloud into a bitstream.

[0025] In another aspect of the present invention, a computer-readable storage medium is provided, comprising instructions for executing the steps of the above method for decoding a 3D point cloud from a bitstream. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Hereinafter, the present invention is described in more detail with reference to the accompanying drawings.

[0027] The accompanying drawings show:

[0028] Figure 1 A flowchart showing a method for decoding 3D point cloud geometry according to the present invention is shown.

[0029] Figure 2 An example of generating an octree structure is shown,

[0030] Figure 3 Show according to Figure 2 The octree of

[0031] Figure 4 An example for determining the vertices on the edge of a cube is shown,

[0032] Figure 5 An example showing a generated triangle,

[0033] Figure 6 An example showing vertices on the edge of a cube,

[0034] Figure 7 Shows the triangles generated by vertices,

[0035] Figure 8 Show according to Figure 7 Example of determining triangle order,

[0036] Fig. 9 Schematic diagram showing the voxelization step,

[0037] Fig.10 shows an example of reconstructing a triangle using the centroid point C as the pivot point,

[0038] Fig.11 An example showing a normal vector,

[0039] Fig.12 shows an example of a 1D residual along the normal vector,

[0040] Fig.13 shows an example of a smooth convex surface in a leaf node with modeling triangles built from vertices and centroid points,

[0041] Fig.14 shows an example of a refined surface,

[0042] Fig.15 shows an example of planar data modeled using the Trisoup method without vertex refinement,

[0043] Fig.16 shows an example of planar data modeled using the Trisoup method with vertex refinement,

[0044] Fig.17 A decoder or encoder is shown. DETAILED DESCRIPTION

[0045] Point clouds have recently gained attention as a format for 3D data representation due to their versatility in representing all types of 3D objects or scenes.

[0046] Therefore, many use cases can be addressed with point clouds, including

[0047] Film post-production,

[0048] Real-time 3D immersive telepresence or virtual reality (VR) / augmented reality (AR) applications,

[0049] Free viewpoint video (e.g. for watching sports),

[0050] Geographic Information Systems (also known as Cartography),

[0051] Cultural heritage (scans of rare objects stored in digital form),

[0052] Autonomous driving, including 3D mapping of the environment and real-time Lidar data collection

[0053] A point cloud is a set of points located in 3D space, with the option to attach additional values ​​to each point. These additional values ​​are often called point attributes. Therefore, a point cloud is a combination of geometry (the 3D position of each point) and attributes.

[0054] The attributes may be, for example, a three-component color, a material property such as reflectivity, and / or a two-component normal vector of the surface associated with the point.

[0055] Point clouds can be captured by various types of devices, such as camera arrays, depth sensors, Lidar, scanners, or can be generated by computers (for example, in movie post-production). Depending on the use case, for mapping applications, point clouds can have thousands to billions of points.

[0056] The raw representation of point clouds requires a very high number of bits per point, at least twelve bits per spatial component X, Y or Z, and optionally more bits for attributes, e.g., three times 10 bits for color. Practical deployment of point cloud based applications requires compression techniques that can store and distribute point clouds over reasonable storage and transmission infrastructure.

[0057] For distribution to and visualization by end users, for example on AR / VR glasses or any other 3D-enabled device, compression can be lossy (like in video compression). Other use cases do require lossless compression, for example, medical applications or autonomous driving, to avoid altering the decision results obtained from the analysis of the compressed and transmitted point cloud.

[0058] Until recently, point cloud compression (aka PCC) had not been addressed by the mass market and no standardized point cloud codec was available. In 2017, the standardization working group ISO / JCT1 / SC29 / WG11, also known as the Moving Picture Experts Group or MPEG, started a work item on point cloud compression. This resulted in two standards, namely

[0059] MPEG-I Part 5 (ISO / IEC 23090-5) or Video-based Point Cloud Compression (V-PCC)

[0060] MPEG-I Part 9 (ISO / IEC 23090-9) or Geometry-based Point Cloud Compression (G-PCC)

[0061] Both the V-PCC and G-PCC standards completed their first versions at the end of 2020.

[0062] The V-PCC encoding method compresses point clouds by performing multiple projections of 3D objects to obtain 2D patches that are packed into images (or videos when processing mobile point clouds). The acquired images or videos are then compressed using existing image / video codecs, allowing the use of already deployed image and video solutions. By its nature, V-PCC is only effective on dense and continuous point clouds, because image / video codecs cannot compress non-smooth patches, such as those obtained from the projection of sparse geometric data collected by Lidar.

[0063] The G-PCC coding method has two schemes for geometry compression. The first scheme is based on an occupancy tree (octree / quadtree / binary tree) representation of the point cloud geometry. Occupied nodes are split until a certain size is reached, and the occupied leaf nodes provide the location of the points, usually at the center of these nodes. A high level of compression can be obtained for dense point clouds by using neighbor-based prediction techniques. Sparse point clouds are also solved by directly encoding the locations of points within nodes with non-minimum sizes, by stopping tree construction when only isolated points exist in the node; this technique is called Direct Coding Mode (DCM).

[0064] The second scheme is based on prediction trees, where each node represents the 3D position of a point and the relationship between nodes is the spatial prediction from parent to child. This approach can only handle sparse point clouds and has the advantages of lower latency and simpler decoding compared to occupancy trees. However, the compression performance is only slightly better than the first occupancy-based approach, and the encoding is complex, with intensive searching for the best predictor (in a long list of potential predictors) when building the prediction tree.

[0065] In both schemes, the attribute (de)coding is performed after the geometry (de)coding is completed, resulting in a two-pass encoding. Thus, low latency is obtained by using slices that decompose the 3D space into independently coded sub-volumes without prediction between sub-volumes. When many slices are used, the compression performance may be severely affected.

[0066] An important use case is the transmission of dynamic AR / VR point clouds. Dynamic means that the point cloud evolves over time. Moreover, AR / VR point clouds are usually locally 2D, as they represent the surface of objects most of the time. Therefore, AR / VR point clouds are highly connected (or dense), as points are rarely isolated but have many neighbors.

[0067] A dense (or solid) point cloud represents a continuous surface at a resolution such that the volumes (small cubes called voxels) associated with the points touch each other without showing any visible holes in the surface.

[0068] Such point clouds are typically used in AR / VR environments and viewed by end users through devices such as TVs, smartphones or headphones. They are either transmitted to the device or stored locally. Many AR / VR applications use dynamic point clouds, which change over time, rather than static point clouds. Therefore, the amount of data is huge and must be compressed. Today, lossless compression based on an octree representation of the geometry of the point cloud can achieve slightly less than one bit per point (1bpp). This may not be enough for real-time transmission, which may involve several million points per frame and frame rates of up to 50 frames per second (fps), resulting in hundreds of megabits of data per second.

[0069] Therefore, lossy compression can be used, with the usual requirements of maintaining acceptable visual quality, while compressing sufficiently to fit within the bandwidth provided by the transmission channel, while maintaining real-time transmission of frames. In many applications, bit rates as low as 0.1bpp (10 times higher compression than lossless coding compression) have made real-time transmission possible.

[0070] Codecs based on MPEG-I Part 5 (ISO / IEC 23090-5) or Video-based Point Cloud Compression (V-PCC) VPCC can achieve such low bitrates by using lossy compression with a video codec that compresses 2D frames obtained from the projection of a point cloud onto a plane. The geometry is represented by a sequence of projected patches assembled into a frame, each patch being a small local depth map. However, VPCC is not general and is limited to narrow types of point clouds that do not exhibit locally complex geometry (e.g., trees, hair), since the obtained projected depth maps are not smooth enough to be effectively compressed by video codecs.

[0071] Pure 3D compression techniques can handle any type of point cloud. It is still an open question whether 3D compression techniques can compete with VPCC (or any projection + image coding scheme) on dense point clouds. Standardization is still moving towards providing an extension (revision) of GPCC that provides competitive lossy compression that can compress dense point clouds as well as VPCC intra-frame coding, while maintaining the versatility of GPCC to handle any type of point cloud (dense point cloud, Lidar, 3D map). This extension will probably use the so-called TriSoup scheme, which works with octrees and will be described in detail in subsequent sections. TriSoup is being explored by the ISO / IEC standardization working group JTC1 / SC29 / WG7.

[0072] refer to Figure 1 , which shows a schematic diagram of a method for decoding geometric information of a 3D point cloud from a bitstream.

[0073] A method for decoding the geometry of a 3D point cloud from a bitstream, preferably implemented in a decoder, comprises the following steps:

[0074] S11: receiving a bit stream, wherein the bit stream includes vertex information, and the vertex information includes information about vertex positions on cube edges of leaf nodes of an octree structure of the point cloud;

[0075] S12: Acquire vertex information to determine the position of the vertex, wherein at least one vertex is adjusted toward the average position of the surface points;

[0076] S13: Determine the points of the point cloud based on at least one adjusted vertex, characterized in that the bit stream further includes plane information indicating whether the point cloud data is a plane, and the determination of the points of the point cloud is further based on the obtained plane information.

[0077] To determine the octree information, the first step of the geometry encoding process is to construct and encode the octree, such as Figure 2 and Figure 3 As shown. The bounding box is a body 100 that contains all points and is associated with a root node 112 (i.e., a single node at the top of the tree 110). The body 100 is first divided into eight sub-volumes 102 called octants, each represented by a node 114 in the tree 110. Octants 106 are then recursively split out of the sub-volumes 104 until a target level is reached, where the octant 106 is occupied by at least one point. Figure 2 and Figure 3 The target level is the level at which the volume is no longer segmented, and it can be determined based on the compression ratio.

[0078] Each octant (or node) is represented by an occupation byte containing one bit for each sub-octant, where the corresponding bit is set to 1 if the sub-octant is occupied by at least one node, and is set to zero otherwise. The occupation bytes 118 of all octants, e.g., "100010000001000 11000011", are serialized (in breadth-first order) and entropy encoded using a binary arithmetic encoder.

[0079] Figure 4 A block representation of a 3D surface 210 is shown, along with an example of a block 220 in a trisoup. The surface 210 intersects the block 220, which is therefore an occupied block, and the block 220 exists between multiple blocks 200 in the 3D space. Within the block 220, the closed portion of the surface 210 intersects the edge of the block at the six illustrated vertices 232 of the polygon 230, and if the edge of the block 220 contains the vertex, then the edge is said to be selected.

[0080] Figure 5 Block 220 in Trisoup is shown, with surface 210 omitted for clarity, and unselected edge 270, selected edge 260, and i-th edge 250 are shown. To specify vertex v on edge i i , specify a scalar value to indicate the corresponding fraction of the length of edge 250.

[0081] like Figure 4 and Figure 5 As shown, within each octant 220 in the target level of the octree, Trisoup represents the original surface 210 as a set of triangles 245. The surface is encoded and used to obtain the position of the reconstructed (or decoded) points. First, the intersection of the surface represented by the original points with the edges of the octant is estimated by averaging the positions of the points closest to those edges within the octant among the original points used to represent the surface. Secondly, the twelve edges of all octants and their associated intersections (if any) are stored as segments and vertices respectively. Each (unique) segment is then encoded as follows. The first single bit is arithmetically encoded, and the bit is set to 1 if the segment is occupied by a vertex, otherwise it is set to 0. If it is occupied, the relative position of the vertex on the line segment is also arithmetically encoded.

[0082] The vertices 310 of the triangles are encoded along the edges 320 of the volume associated with the leaf nodes 300 of the tree, as Figure 6 As shown. These vertices 310 on the edge 320 are shared between the leaf nodes 300 with the common edge 320. This means that each edge belonging to at least one leaf node encodes at most one vertex. In this way, the continuity of the model is ensured through the leaf nodes.

[0083] As described above, encoding of TriSoup vertices may require two pieces of information per edge: a vertex flag indicating whether a TriSoup vertex is present on the edge, and, if so, the vertex position along the edge.

[0084] Therefore, the encoded data includes Octree data as well as TriSoup data.

[0085] The vertex flags are encoded by an adaptive binary arithmetic encoder that uses a specific context to encode the vertex flags. The positions of vertices on an edge of length n=2^s can be encoded with unit precision by pushing s bits into (bypassing / not entropy encoding) the bitstream.

[0086] Within a leaf node, if there are at least three vertices 310 on an edge 320 of the leaf node 300, a triangle is constructed from the TriSoup vertices. Figure 7 Reconstructed triangles 330 and 340 are depicted in FIG.

[0087] Obviously, other combinations of triangles 330 and 340 are possible. The selection of triangles results from a three-step process

[0088] 1. Determine the dominant direction along one of the three axes

[0089] 2. Sort TriSoup vertices according to dominant direction

[0090] 3. Construct a triangle based on an ordered list of vertices

[0091] Knowledge of the exact position of the triangle in the current leaf node is not required and can be inferred from the vertices.

[0092] Figure 8 will be used to explain this process. Each of the three axes is tested, and the one that maximizes the total surface of the triangle is chosen as the principal axis. For simplicity of the diagram, Figure 8 Only tests on two axes are described.

[0093] The first test (top) along the vertical axis is performed by vertically projecting the cube and TriSoup vertices 310 on the 2D plane. The vertices 310 are then sorted in clockwise order relative to the center of the projection node (square). Then, based on the ordered vertices, triangles 330, 340 are constructed according to a fixed rule. Here, here, when 4 vertices are involved, triangle 123 as well as triangle 134 are systematically constructed. When there are 3 vertices, the only possible triangle is 123. When there are 5 vertices, the fixed rule can be to construct triangles 123, 134 and 451. And so on, up to a maximum of 12 vertices.

[0094] The second test (left side) along the horizontal vertical axis is performed by projecting the cube and trisoup vertices horizontally onto the 2D plane.

[0095] The vertical projection shows the maximum total 2D surface of the triangle, therefore, the principal axis is chosen as the vertical axis, and the constructed TriSoup triangles are obtained in the order of vertical projection, such as Figure 8 As shown, it is located inside the node. It should be noted that taking the horizontal axis as the main axis will result in another construction of the triangle.

[0096] The principal axes are fully selected by maximizing the projection surface, thus achieving continuous reconstruction of point clouds without holes.

[0097] Rendering of TriSoup triangles into points can be performed by ray tracing. The set of all points rendered by ray tracing will form the decoded point cloud.

[0098] for Fig. 9 For the ray tracing shown, rays can be emitted along three directions parallel to the axis. Their origin is a point with integer (voxelized) coordinates of a precision corresponding to the sampling precision required for rendering. The intersection with one of the trisoup triangles (dashed point if any) is then voxelized (= rounded to the nearest point of the required sampling precision) and added to the list of rendered points.

[0099] After applying Trisoup to all leaf nodes, i.e., building triangles and obtaining points preferably by ray tracing, discard all copies of the same point in the list of rendered points (i.e., keep only one voxel among all voxels sharing the same position and volume) to obtain a set of decoded (unique) points.

[0100] Based on the above concept, the Trisoup encoding can be improved. For example, by calculating the centroid point, the coordinates of which are the average coordinates of all (ordered) vertices Vi, see Fig.10 , where the centroid C is depicted using a checkerboard fill. The centroid is used as the pivot point. A triangle consists of ordered vertices (V1, V2, …, V M ) is constructed by pivoting around the centroid point C, constructing the following M triangles: V1V2C, V2V3C, ..., V M-1 V M C, V M V1C.

[0101] The vertices are sorted clockwise, and it does not matter which vertex is chosen as V1.

[0102] This structure preserves the natural symmetry of the model without arbitrarily privileging triangles. In addition, it provides an additional degree of freedom to improve the accuracy of the model, namely, the location of the centroid point C.

[0103] Therefore, the position of the centroid point C can be further improved by encoding the residual position in the bitstream so that the position of the centroid point C is closer to the original point of the point cloud.

[0104] For example, C = C mean +C res .

[0105] Among them, C mean is the average position obtained by averaging the coordinates of all (ordered) vertices, and C res is the coding residual.

[0106] The coded residual can be a 3D residual. However, it is observed that a 3D residual is rarely advantageous because it requires many bits to be coded and such many bits are not fully compensated by the better accuracy of the model. Therefore, it is preferred to encode the 1D residual C res to encode.

[0107] For example, the normal vector can be constructed, such as Fig.11 As shown, the residual can be determined by the following formula:

[0108]

[0109] where α is a 1D signed scalar value encoded in the bitstream, see Fig.12 . Normal vector You can export it in two steps:

[0110] 1.

[0111] 2. Then normalize

[0112] where × is the cross product between two vectors (also called the vector (cross) product), and the side yes:

[0113]

[0114] It is understandable that the vector Can be parallel to the axis to simplify its determination and the calculation of the value α. can be parallel to the principal axis as a good approximation to the vector calculated above.

[0115] The value α is determined by the encoder, encoded into the bitstream, and obtained by the decoder by decoding the bitstream. The value α may be binarized, and each bit may be encoded using a binary entropy encoder such as an arithmetic encoder or a context adaptive binary encoder such as CABAC.

[0116] The value α can be binarized as

[0117] A flag f0 indicating whether α is equal to 0

[0118] A sign indicating whether α>0 or α<0,

[0119] A flag f1 indicating whether |α| is equal to 1,

[0120] The remainder |α|-2 encoded by the expGolomb encoder.

[0121] The value α can be obtained by the encoder by considering all points P in the point cloud belonging to the current leaf node K To determine. For each point P K , which is consistent with the straight line The distance d K From the following formula we get

[0122]

[0123] And if this distance d K Below a predefined threshold th (e.g., th=2), point P is used K Calculate the value α. Point P K Relative to the average point C mean The 1D residual r K Obtained by the scalar product (also called inner product or dot product)

[0124]

[0125] Therefore, the value α is obtained by

[0126]

[0127] Where S is point P K A set of such that their distance d K is below a threshold th, and |S| is the number of points belonging to this set.

[0128] In the current trisoup encoding, as described in conjunction with the previous figures, the centroid point C and the vertices of the leaf nodes (V0, ..., V i ) can be used to construct triangles, and then ray tracing methods can be applied to the constructed triangles to obtain the reconstructed points.

[0129] Point cloud data captured in natural scenes often have smooth surfaces, even in concave or convex areas. However, in the current trisoup encoding, concave or convex areas are modeled with triangles in leaf nodes, and the generated centroid residual C resmay be large (for example, C res can be about half the width of a leaf node), then use the triangles constructed within the leaf node (by using the centroid point C and the vertices (V0, ..., V i The modeled surface will have large protrusions and the generated surface will be far from naturally smooth and will have artifact effects. Fig.13 As shown, point 1301 represents the vertex along the edge in the leaf node, and point 1303 represents the centroid point C in the leaf node, and small point 1305 represents the point on the original surface (for clarity, only one of these small points is marked with label 1305). It can be seen that the modeled surface with sharp protrusions cannot model the original surface well enough, and the reconstructed surface may not maintain the smooth characteristics of the original surface, which may make the reconstructed point cloud have poor visual quality, especially when the leaf node size is relatively large, at a lower compression rate. Therefore, the compression efficiency of trisoup encoding in the current MPEGG-PCC is not optimal.

[0130] Therefore, the problem to be solved by the present invention is to improve the compression efficiency of point clouds by selectively reducing the non-smoothness of the reconstructed surface at the decoder side.

[0131] Therefore, according to the present invention, it is proposed to refine the triangle modeling in the leaf node based on the information indicating whether the point cloud is planar data, so that the surface reconstructed on the decoder side is closer to the original surface. Specifically, for example, if the point cloud is not planar data, after obtaining the decoded vertices and the decoded centroid point C on the decoder side, the proposed method can be used to move the vectors in the leaf nodes toward the convex area or the concave area. The position of the vertices is adjusted in the direction of , so that the proposed method can reduce the non-smoothness of the reconstructed point cloud without increasing the bitstream size.

[0132] In some embodiments, the method according to the present invention follows the following steps:

[0133] On the decoder side, after decoding the vertices of each leaf node from the bitstream, each leaf node may be iterated to construct triangles for obtaining the reconstructed point cloud, preferably by a ray tracing method.

[0134] In detail, for each leaf node, for example, if the point cloud data is plane data

[0135] o If the current (i.e., processed) leaf node has more than 3 vertices,

[0136] First, the centroid point C can be determined to construct a modeling surface, which is composed of vertices and triangles constructed by the centroid point C. Specifically, the vertices (V0, ..., V i ) of the average point C mean, and the unit vector of the centroid residual The modulus value C of the centroid residual can then be decoded from the bitstream res , and finally can be Get the center of mass point C, where From C mean The vector to C can also be named

[0137] Then, the prominence of the modeled surface in the leaf node can be determined. In a preferred embodiment, the centroid residual C res Whether it is large relative to the size of the leaf nodes.

[0138] If the centroid residual C res Larger than the leaf nodes (more likely, based on the current vertex (V0, ..., V i )) and the centroid point C will have sharp protrusion artifacts in the generated modeled surface), then for all vertices V (V1,…,V i ) are refined along vectors in all three axes. The vertices on the edge of the axis with the maximum value, the refinement can be along the edge to which these vertices belong, within the boundary of the edge in the leaf node, in a direction that can make the reconstructed surface more natural. (Detail vertex refinement will be described later)

[0139] Otherwise, all vertices in the leaf nodes (V0,…,V i ) may not be refined.

[0140] Then, all vertices in the leaf node (including the refined vertex V' and other non-refined vertices V) are combined with the centroid point C to construct triangles, as described above with reference to the accompanying drawings, and a ray tracing method can be applied to each triangle to obtain the reconstructed points in the leaf node.

[0141] Otherwise, if the leaf node has 3 vertices, there is no average point C in this processed leaf node mean and centroid point C, and only 1 triangle V1V2V3 can be constructed, and a ray tracing method can be applied on this triangle to obtain the reconstructed point in the leaf node.

[0142] o Otherwise, if a leaf node has less than 3 vertices, there will be no triangles built for the leaf node, and thus no reconstruction points will be generated for the leaf node.

[0143] On the encoder side, after encoding the vertices of each leaf node into the bitstream, each leaf node can be iterated to encode the information of the centroid position C, and for subsequent attribute encoding, it may also be necessary to obtain the reconstructed geometric information of the point cloud by constructing triangles based on the vertices and the centroid position C. And in order to obtain a refined modeled surface in the leaf node, the leaf node can be iteratively processed according to the same steps as described above on the decoder side. Of course, in addition, the information indicating whether the point cloud is planar is also encoded at the encoder.

[0144] In some embodiments, to determine whether the centroid residual C res is large relative to the size of the leaf node, the centroid residual C res can be compared with a threshold Th, which is based on the size N of the leaf node. The threshold Th can be a proportion s of the size of the leaf node, Th = N * s, and the proportion s can be 0 < s ≤ 1 / 8, for example. And in an actual implementation, to improve the precision of division / multiplication operations, C res is multiplied by 64 (which can be achieved by applying a left shift of 6 bits to C res ), so the comparison threshold Th needs to be multiplied by 64 and becomes N / 2. And for example, for a leaf node, if C res > Th, the vertices in the leaf node can be refined according to the proposed method; otherwise, the proposed surface refinement method will not be applied to the leaf node.

[0145] As described above, if the centroid residual C res is large relative to the size of the leaf node, then for all vertices (V0,..., V i ) in the leaf node that are on the edge along the axis among the three axes where the vector has the maximum value, they are refined. The refinement can be carried out along the edges to which these vertices belong, within the boundaries of the edges in the leaf node, and in a direction that can make the reconstructed surface more natural. In a preferred embodiment, the detailed vertex refinement can follow the following steps:

[0146] · First, determine the axis axis_max along which the vector has the maximum value;

[0147] · Then, the vertices (V0,..., V i ) parallel to the axis axis_max in the leaf node can be found.

[0148] · Then, based on the centroid residual vector of the leaf node change the axis_max dimension of these vertices, and the obtained vertex V' i can be described as Where, the offset is relative to V i The offset distance, is a unit vector pointing in the offset direction. In a preferred embodiment, Can be based on and the boundaries of the edges, where represents the centroid residual vector The projection vector along the axis_max direction, the edge boundary is the constraint to keep the vertex from exceeding the leaf node to which it belongs. And in such an embodiment, Can be followed by same direction, and the offset can be based on The modulus value and the edge boundary of the leaf node are obtained. In a preferred embodiment, the offset can be 1 / 3 of the modulus value. Fig.14 14 shows a refined surface generated according to the present invention, the point 1401 on the edge is the vertex V used to model the surface according to the current trisoup method, the arrow line parallel to the X axis represents the projection vector of the centroid residual, and the point 1401a (only one of these points is marked with a number for clarity) is obtained by following the The refined vertex V′ is obtained by moving the vertex V in the direction. It can be seen that the refined surface is closer to the original surface in the leaf node and the sharp protrusion artifacts are reduced.

[0149] According to the present invention, point cloud data can be divided into two categories: planar data and non-planar data, depending on whether the planar area is the majority.

[0150] In planar data, since planar regions are in the majority, the centroid residuals of many leaf nodes encoded by Trisoup are 0, such as Fig.15 As shown, it is a planar point cloud data (such as a building with 4 walls (i.e., 1501)) modeled by the Trisoup method (i.e., 1503). Since the front wall is planar, the centroid residuals of leaf nodes M and N are 0. And it can be imagined that when modeling planar data, there are many leaf nodes whose centroid residuals are 0, so there is no need to refine their vertices. However, if we use the above vertex refinement method in all types of point cloud data, each leaf node will be determined whether its vertices need to be refined, and many results are wrong. Therefore, it is a waste of time to make such a determination for planar point cloud data.

[0151] Similarly, on the edge of the plane data, there is often a right angle corner, such as Fig.15 As shown in the leaf node O in , the centroid residual in such a leaf node is not 0, so according to the above refinement method, the vertices in such a leaf node will be offset and refined along the edge direction, for example, Fig.16 As shown, vertices V1 and V2 will move to Direction refinement is performed and refined vertices V′1 and V′2 are obtained, and then triangles are constructed using the refined vertices V′1 and V′2 and other non-refined vertices. The surface reconstructed by the triangles in the leaf nodes N and O will have gaps with the original front surface of the point cloud. In this case, vertex refinement is used to process planar data, and the reconstructed point cloud cannot maintain the original planar features well.

[0152] As described above, using the vertex refinement method for planar point cloud data will result in a waste of time if the centroid residual is 0 in many leaf nodes, and will also result in poor visual quality if the centroid residual is not 0 in some leaf nodes that model the surface in the corner part of the original point cloud. Therefore, the vertex refinement method is not suitable for planar point cloud data and will result in compression performance loss of planar data. In addition, according to experimental results, the vertex refinement method will result in compression performance loss of planar data.

[0153] However, according to our experimental observation and analysis of the above thinning method, this method can have better compression performance when applied to non-planar data.

[0154] Therefore, when vertex refinement is introduced, the present invention can further improve the overall compression performance of point cloud encoding of planar point cloud data and non-planar point cloud data.

[0155] One possible implementation of the present invention is to introduce a flag (e.g., TrisoupPlaneFlag) indicating whether the point cloud is plane data to enable / disable vertex refinement in Trisoup encoding. If the flag is true (the point cloud data is plane data), the vertex refinement method is disabled; otherwise, the vertex refinement method is enabled. Through the proposed method, the loss effect of the vertex refinement method on plane data can be avoided, and the compression efficiency of the point cloud encoding can be achieved to achieve the overall optimization.

[0156] In some embodiments, a flag TRisoupPlaneFlag indicating whether the point cloud is planar data can be obtained through the characteristics of the input point cloud data, and the flag is written into the GPS (geometry parameter set) and encoded into a bit stream at the encoder, which is decoded from the bit stream and then used to enable / disable the vertex refinement method.

[0157] In some embodiments, on the encoder side, a flag TrisoupPlaneFlag indicating whether the point cloud is plane data can be determined based on the percentage of leaf nodes with a centroid residual value of 0. Specifically, on the encoder side, before constructing triangles in each leaf node, each leaf node is iterated to calculate the percentage p of leaf nodes with a centroid residual value of 0, assuming that the number of leaf nodes with a centroid residual of 0 is N. ctrod_is_0， Ntotal is the total number of leaf nodes of the point cloud, so p can be obtained by

[0158] p=N ctrod_is_0 / N total，

[0159] And compare the percentage p with the threshold Th, if p ≥ Th, the flag TrisoupPlaneFlag is true; otherwise, the flag TrisoupPlaneFlag is false. The threshold Th can be determined by user definition, for example, preferably, it is a number greater than 0.7 and less than 1. Then, the flag is written to the GPS (geometry parameter set) and encoded into the bitstream at the encoder, and at the decoder, it is decoded from the bitstream and then used to enable / disable the vertex refinement method.

[0160] In some embodiments, the flag TrisoupPlaneFlag indicating whether the point cloud is plane data can be based on the normal vector of each leaf node The direction is automatically determined.

[0161] First, iterate each leaf node and calculate the normal vector The number of leaf nodes pointing in the same direction N same ,

[0162] Then after the iteration, the percentage p is calculated by

[0163] p=N same / N total，

[0164] Among them, N total is the total number of leaf nodes of the point cloud, and p can be used to indicate whether the encoded point cloud is plane data. A threshold Th needs to be determined. If p>Th, the encoded point cloud is plane data; otherwise, it is not plane data. The threshold Th can be determined by the user, for example, preferably, it is a number greater than 0.7 and less than 1.

[0165] In some embodiments, the present invention follows the following steps:

[0166] On the decoder side, first, the flag TrisoupPlaneFlag can be decoded from the bitstream and used in the subsequent Trisoup encoding process. In the Trisoup encoding process, after the vertices of each leaf node are decoded from the bitstream, each leaf node can be iterated to construct a triangle for obtaining a reconstructed point cloud by a ray tracing method.

[0167] In detail, for each leaf node,

[0168] o If the leaf node being processed has more than 3 vertices,

[0169] First, the centroid point C can be determined to construct a modeling surface consisting of triangles constructed from vertices and the centroid point C. Specifically, the vertices in the leaf nodes (V0, ..., V i ) of the average point C mean and the unit vector of the centroid residual Then, the magnitude of the centroid residual C can be decoded from the bitstream res Afterwards, you can Get the center of mass point C, where From C mean The vector to C can also be named

[0170] Then, it can be determined whether the current point cloud is plane data based on the flag TrisoupPlaneFlag to decide whether the vertices in the current leaf node need to be refined.

[0171] If the flag TrisoupPlaneFlag is true, the vertex optimization method is disabled;

[0172] Otherwise, enable the vertex optimization method, and then use the vertex optimization method for the vertices in the leaf node. In one embodiment, first, the prominence of the modeling surface in the leaf node can be determined, for example, it can be determined whether the centroid residual Cres is large relative to the size of the leaf node.

[0173] If the centroid residual C res Relative to the size of the leaf node, all vertices V(V0,…,V i ) in the vectors along the three axes Vertices on the edges of the axis with the maximum value are refined along the edges to which they belong, within the boundaries of the edges in the leaf nodes, and in a direction that makes the reconstructed surface more natural.

[0174] Otherwise, all vertices (V0,…,V i ) may not be refined in leaf nodes.

[0175] Then, triangles are constructed using all vertices in the leaf nodes (including the refined vertex V' and other non-refined vertices V) combined with the centroid point C, and a ray tracing method can be applied to each triangle to obtain the reconstructed points in the leaf nodes.

[0176] Otherwise, if the leaf node has 3 vertices, there will be no average point C in the processed leaf node meanand the centroid point C, only one triangle V1V2V3 can be constructed, and the ray tracing method can be applied to this triangle to obtain the reconstructed points in the leaf nodes.

[0177] o Otherwise, if a leaf node has less than 3 vertices, no triangles are built for the leaf node, and thus no reconstruction points are generated for the leaf node.

[0178] On the encoding side, after encoding the vertices of each leaf node into a bitstream, each leaf node can then be iterated to encode the information of the centroid position C, and for subsequent attribute encoding, it may also be necessary to obtain the reconstructed geometric information of the point cloud by constructing triangles based on the vertices and the centroid position C. Before constructing triangles at the encoder, the leaf nodes are iteratively processed following the same steps described above on the decoder side.

[0179] In some embodiments, the flag TrisoupPlaneFlag indicating whether the point cloud is plane data can be based on the normal vector of each leaf node The orientation of is automatically determined as described above. For the case where the point cloud has more than 1 plane, a variant method can be used. In detail,

[0180] First, we can use the normal vector Whether parallel to the x-axis, y-axis or z-axis to set the 3 categories.

[0181] Then, we can iterate over each leaf node to calculate the normal vector separately The number of leaf nodes parallel to the x-axis N x , the number of leaf nodes whose normal vector is parallel to the y-axis N y , and the number of leaf nodes whose normal vectors are parallel to the z-axis N z .

[0182] Then after the iteration, the percentage p is calculated by x

[0183] p x =N x / N total ,

[0184] And calculate the percentage p by the following formula y

[0185] p y =N y / N total ,

[0186] And calculate the percentage p by the following formula z

[0187] p z =N z / N total ,

[0188] Among them, N total is the total number of leaf nodes of the point cloud, and p x , p y and p z It can be used together to indicate whether the encoded point cloud is planar data. And it may be necessary to determine the threshold Th. If p x +p y +p z >Th, the encoded point cloud is plane data; otherwise, it is not plane data. And a threshold Th can be determined, preferably a number greater than 0.7 and less than 1.

[0189] Reference now Fig.17 , which shows a simplified block diagram of an example embodiment of an encoder or decoder 300. The encoder or decoder 300 includes a processor 301 and a memory storage device 303. The memory storage device 303 can store a computer program or application containing instructions that, when executed, cause the processor 301 to perform operations such as those described herein. For example, the instructions can encode and output an encoded bit stream or decode a bit stream and output points of a point cloud according to the methods described herein. It should be understood that the instructions can be stored on a non-transitory computer-readable medium, such as an optical disk, a flash memory device, a random access memory, a hard drive, etc. When the instructions are executed, the processor 301 performs the operations and functions specified in the instructions so as to operate as a dedicated processor that implements the process. In some examples, such a processor may be referred to as a "processor circuit" or "processor circuitry".

[0190] It is understood that decoders and / or encoders according to the present application can be implemented in many computing devices, including but not limited to servers, appropriately programmed general-purpose computers, machine vision systems, and mobile devices. A decoder or encoder can be implemented by software containing instructions for configuring one or more processors to perform the functions described herein. The software instructions can be stored on any suitable non-transient computer-readable memory, including CD, RAM, ROM, flash memory, etc.

[0191] It should be understood that standard computer programming techniques and languages ​​can be used to implement the decoder and / or encoder described herein and to implement the module, routine, process, thread or other software components of the method / process for configuring the encoder or decoder. The application is not limited to specific processors, computer languages, computer programming conventions, data structures, other such implementation details. Those skilled in the art will recognize that the described process can be implemented as a part of a computer executable code stored in a volatile or non-volatile memory, a part of a special integrated chip (ASIC), etc.

[0192] Certain adjustments and modifications may be made to the described embodiments. Therefore, the embodiments discussed above are considered to be illustrative rather than restrictive. In particular, the embodiments may be freely combined with each other.

Claims

1. A method for decoding the geometry of a 3D point cloud from a bitstream, preferably implemented in a decoder, the method comprising: receiving a bitstream, wherein the bitstream includes vertex information, the vertex information including information about positions of vertices on edges of cubes of leaf nodes of an octree structure of a point cloud; acquiring the vertex information to determine positions of vertices, wherein at least one vertex is adjusted toward an average position of the surface points; determining a point of the point cloud based on at least one adjusted vertex, It is characterized in that the bit stream further includes plane information indicating whether the point cloud data is a plane, and the determination of the points of the point cloud is further based on the obtained plane information.

2. A method for encoding a 3D point cloud into a bitstream, preferably implemented in an encoder, the method comprising: Acquire vertex information of each cube associated with a leaf node from the point cloud surface, wherein the vertex information includes information about positions of vertices on edges of the cubes of the leaf nodes of the octree structure of the point cloud; encoding the vertex information into a bitstream, wherein at least one vertex is adjusted toward an average position of surface points; Determining points of the point cloud based on the encoded vertex information; It is characterized in that the method further includes determining and encoding plane information, wherein the plane information indicates whether the point cloud data is planar, and the determination of the points of the point cloud is further based on the encoded plane information.

3. The method according to any one of claims 1 to 2, wherein obtaining the vertex information to determine the position of the vertex, wherein at least one vertex is adjusted toward the average position of the surface points, comprises: determining a virtual position based on positions of vertices of a cube associated with a leaf node of the octree structure; constructing triangles based on the vertices and the virtual positions of a cube; Determining a normal vector of the cube based on the constructed triangle; determining a center of mass position based on the normal vector; At least one of the vertices is adjusted toward the centroid position.

4. The method according to any one of claims 1 to 3, wherein: At least one of the vertices is adjusted only when the difference between the virtual position and the centroid position exceeds a threshold, preferably the threshold is based on the size of the leaf node, preferably a ratio s of the size of the leaf node, wherein Preferably, S=1 / 128.

5. The method according to any one of claims 1 to 4, wherein: At least one of the vertices is adjusted according to a fixed value, preferably, the fixed value is determined based on the type of the point cloud and is encoded into or from a bitstream.

6. The method according to any one of claims 1 to 5, wherein: At least one of the vertices is adjusted based on a difference between the virtual position and the centroid position, preferably further based on a boundary of an edge of the leaf node.

7. The method according to any one of claims 1 to 6, wherein: The plane information is encoded into the geometry parameter set GPS.

8. An encoder for encoding a 3D point cloud into a bitstream, the encoder comprising at least one processor and a memory, wherein: The processor stores instructions which, when executed by the processor, perform the steps of the method according to claim 2 and any one of claims 3 to 7 dependent on claim 2.

9. A decoder for decoding a 3D point cloud from a bitstream, the decoder comprising at least one processor and a memory, wherein: The memory stores instructions, which, when executed by the processor, perform the steps of the method according to claim 1 and any one of claims 3 to 6 dependent on claim 1.

10. A bit stream encoded by the method according to claim 2 and any one of claims 3 to 7 when dependent on claim 2.

11. A computer-readable storage medium comprising instructions, which, when executed by a processor, perform the steps of the method according to any one of claims 1 to 7.

Images