3D point cloud encoding and decoding method, encoder and decoder

Abstract

A method of decoding a 3D point cloud geometry from a bitstream, preferably implemented in a decoder, by decoding information constituting a geometric representation of the geometry, comprising: receiving and decoding the bitstream, where the bitstream comprises octree information comprising information about an octree structure of a point cloud volume and vertex information comprising information about an octree structure of the point cloud volume; the vertex information comprises information about existence of vertexes and position information about the vertexes on the edges of the cube; determining a triangle by connecting vertexes of a cube in the octree structure; the triangles are voxelized to determine points of the point cloud, where at least one triangle extends along at least one side to voxelize based on a sampling distance dsampl of the point cloud.

Description

Technical Field

[0001] The present invention relates to a method for decoding a 3D point cloud from a bitstream. In addition, the object of the present invention is to provide a method for encoding a 3D point cloud into a bitstream. In addition, the object of the present invention is to provide an encoder and a decoder, a bitstream encoded according to the present invention and software. In particular, the object of the present invention is to provide a method for improving the accuracy of the decoding or reconstruction process of a 3D point cloud. Background Art

[0002] As a representation format for 3D data, point clouds have recently gained appeal due to their ability to flexibly represent all types of 3D objects or scenes. Therefore, many use cases can be addressed with point clouds, including:

[0003] Film post-production,

[0004] Real-time 3D immersive telepresence or VR / AR applications,

[0005] Free-viewpoint videos (e.g. watching sports games),

[0006] Geographic Information Systems (also known as Cartography),

[0007] Cultural heritage (scans of rare cultural relics are stored in digital form),

[0008] Autonomous driving, including 3D mapping of the environment and real-time lidar data collection.

[0009] A point cloud is a collection of points located in 3D space, optionally with additional values ​​attached to each point. These additional values ​​are usually called point attributes. Therefore, a point cloud is a combination of geometry (the 3D position of each point) and attributes.

[0010] The attributes may be a three-component color, a material property such as reflectivity, and / or a two-component normal vector of the surface associated with the point.

[0011] Point clouds can be captured by various types of devices, such as camera arrays, depth sensors, LiDAR, scanners, or they can be computer generated (e.g. in movie post-production). Depending on the use case, point clouds can have anywhere from thousands to billions of points for cartography applications.

[0012] The raw representation of a point cloud requires a very high number of bits per point, with at least a dozen bits for each spatial component X, Y or Z. Optionally, more bits are used for attributes, e.g. three times 10 bits are needed for color. Practical deployment of point cloud-based applications requires compression techniques to store and distribute point clouds with reasonable storage and transmission infrastructure.

[0013] The compression can be lossy, such as video compression, for distribution to and visualization by the end user, for example on AR / VR glasses or any other 3D-capable device. Other use cases do require lossless compression, such as medical applications or autonomous driving, to avoid changes in decision making due to analysis of the compressed and transmitted point cloud.

[0014] Until recently, point cloud compression, also known as PCC, has not been implemented for the mass market and no standardized point cloud codec is available. In 2017, the standardization working group ISO / JCT1 / SC29 / WG11, also known as the Moving Picture Experts Group or MPEG, launched a work item on point cloud compression. This resulted in two standards, namely:

[0015] MPEG-I Part 5 ISO / IEC 23090-5 (also known as Video-based Point Cloud Compression V-PCC), and

[0016] MPEG-I Part 9 ISO / IEC 23090-9 (also known as Geometry-based Point Cloud Compression G-PCC).

[0017] Both V-PCC and G-PCC standards completed their first versions at the end of 2020 and will be launched into the market soon.

[0018] The V-PCC coding method compresses point clouds by performing multiple projections of 3D objects to obtain 2D patches that are packed into images (or videos when processing moving point clouds). The obtained images or videos are then compressed using existing image / video codecs, which allows leveraging already deployed image and video solutions. By its nature, V-PCC is only effective on dense and continuous point clouds, since image / video codecs cannot compress non-smooth patches, such as those obtained from the projection of sparse geometry data acquired by LiDAR.

[0019] The G-PCC coding method has two schemes for geometric compression.

[0020] The first scheme is based on an occupancy tree (octree / quadtree / binarytree) representation of the point cloud geometry. Occupied nodes are split until they reach a certain size. Occupied leaf nodes provide the location of the points, usually located in the center of these nodes. By using neighborhood-based prediction techniques, high levels of compression can be obtained for dense point clouds. Sparse point clouds can also be handled by directly encoding the locations of points within non-minimum-sized nodes by stopping the construction of the tree when there are only isolated points in the node; this technique is called Direct Coding Mode (DCM).

[0021] The second scheme is based on prediction trees, where each node represents the 3D position of a point and the relationship between nodes is the spatial prediction from parent to child nodes. This approach can only handle sparse point clouds and has the advantages of lower latency and simpler decoding compared to occupancy trees. However, its compression performance is only slightly better than the first occupancy-based approach, and the encoding is complex, requiring the search for the best predictor in a long list of potential predictors when building the prediction tree.

[0022] In both schemes, the encoding (de)coding of attributes is done after the geometry encoding (de)coding is done, resulting in two-pass encoding. Therefore, by using slices that decompose the 3D space into independently encoded sub-volumes, without the need for prediction between sub-volumes, low latency can be achieved. When many slices are used, this can severely affect compression performance.

[0023] An important use case is the transmission of dynamic AR / VR point clouds. Dynamic means that the point cloud changes over time. In addition, AR / VR point clouds are usually locally 2D, as they represent the surface of objects most of the time. Therefore, AR / VR point clouds are highly connected (or dense), as a point rarely exists in isolation, but has many neighbors.

[0024] A dense (or solid) point cloud represents a continuous surface at a resolution such that the volumes associated with the points (small cubes called voxels) touch each other without any visible holes in the surface.

[0025] Such point clouds are often used in AR / VR environments and viewed by end users on devices such as TVs, smartphones or headsets. They are either transmitted to the device or stored locally. Many AR / VR applications use moving point clouds, rather than static point clouds that change over time. Therefore, the data volume is huge and must be compressed. Today, lossless compression based on an octree representation of the point cloud geometry can achieve slightly less than 1 bit per point (1bpp). This may not be enough for real-time transmission, which may involve millions of points per frame at frame rates of up to 50 frames per second (fps), resulting in hundreds of megabits of data per second.

[0026] Therefore, lossy compression can be used to meet the usual requirements of maintaining acceptable visual quality, compressing sufficiently to fit within the bandwidth provided by the transmission channel, and maintaining real-time transmission of frames. In many applications, bit rates as low as 0.1bpp (10 times the compression of lossless coding) can achieve real-time transmission.

[0027] The codec VPCC based on MPEG-I Part 5 ISO / IEC 23090-5 (or Video-based Point Cloud Compression V-PCC) can achieve such low bitrates by using lossy compression of a video codec, which compresses 2D frames obtained from the projection of the point cloud on a plane. The geometry is represented by a sequence of projection slices assembled into frames, each slice being a small local depth map. However, since the obtained projected depth maps are not smooth enough to be effectively compressed by a video codec, the use of VPCC is not widespread and is limited to narrow types of point clouds that do not show locally complex geometry, such as trees, hair.

[0028] Pure 3D compression techniques can handle any type of point cloud. It is still an open question whether 3D compression techniques can compete with VPCC (or any projection plus image coding scheme) on dense point clouds. Standardization is still in progress and aims to provide extensions (amendments) to GPCC that will provide competitive lossy compression, compressing dense point clouds as good as in VPCC, while maintaining the versatility of GPCC, i.e., being able to handle any type of point cloud (dense, LiDAR, 3D maps). This extension will probably use the so-called TriSoup encoding scheme, which works with octrees. TriSoup is being explored by the ISO / IEC standardization working group JTC1 / SC29 / WG7. TriSoup coding can also be found in "Adaptive multi-level triangle soup for geometry-based point cloud coding" by A.DRICOT et al. at the 2019 IEEE 21st International Symposium on Multimedia Signal Processing MMSP, "Report on triangle soup decoding" by Nakagami O., m52279 in 2020 by ISO / IEC JTC1 / SC29-WG11, and US10,192,353.

[0029] However, for all lossy compression schemes, the reconstruction quality of the points in the point cloud is crucial. Summary of the invention

[0030] It is therefore an object of the present invention to provide a method for decoding 3D point cloud geometry from a bitstream and encoding a 3D point cloud into a bitstream with higher accuracy.

[0031] This problem is solved by a decoding method according to claim 1 , an encoding method according to claim 2 , an encoder according to claim 15 , a decoder according to claim 16 , a bit stream according to claim 17 and software according to claim 18 .

[0032] In a first aspect, a method for decoding 3D point cloud geometry from a bitstream is provided, preferably implemented in a decoder. The method comprises:

[0033] Receive and decode a bitstream, wherein the bitstream includes octree information and vertex information, the octree information includes information about an octree structure of a point cloud volume, and the vertex information includes information about the existence of a vertex and position information about a vertex on an edge of a leaf node cube in the octree structure;

[0034] The triangles are determined by connecting the vertices of a cube associated with the leaf nodes in the octree structure;

[0035] voxelize the triangles to determine the points of the point cloud,

[0036] Wherein, at least one triangle is extended along at least one side to a sampling distance d based on the point cloud. sampl Voxelize.

[0037] Therefore, in a first step, a bitstream is received, and the bitstream contains information about the octree structure of the point cloud volume to be decoded. Preferably, the point cloud geometry is GPCC encoded. Therefore, by decoding from the bitstream, the octree information about the point cloud volume is provided. In addition, the bitstream also includes vertex information, which includes information about the existence of the vertex and information about the position of the vertex on the edge of the cube associated with the leaf node in the octree structure. Therefore, the vertex information is provided by decoding from the bitstream. Preferably, the bitstream is encoded at the encoder using the TriSoup encoding scheme.

[0038] After decoding the octree information and vertex information from the bitstream as described in the previous step, in a further step, in order to reconstruct the point cloud geometry, triangles are determined for each cube by connecting the vertices on the edges of the cube. The surface of the triangle is therefore determined by the positions of the vertices contained in the bitstream. In order to reconstruct the points of the point cloud from the triangles, voxelization is performed by a ray tracing process, in which rays are emitted along three directions parallel to the three axes. Their origin is a point with integer coordinates corresponding to the sampling accuracy required for rendering. The intersection of the ray with one of the triangles (if any) is then determined and added to the list of rendered points, i.e. to the points of the point cloud. During the voxelization process, the rays sample the surfaces of the triangles to determine the points of the point cloud.

[0039] According to the present invention, for voxelization or during voxelization, at least one triangle is extended along at least one side to be based on the sampling distance d of the point cloud. samplExtend the surface of the triangle in at least one direction. The sampling distance is a property of the initial point cloud data. If there are no missing points during data acquisition, the sampling distance is related to the distance between the actual sampling points of the point cloud, in units of sampling resolution. sampl It is set by, for example, a device for acquiring points of a point cloud, such as a laser radar or the like. Therefore, by expanding the triangle during the voxelization process, the accuracy of the voxelization process can be improved, because additional points of the original point cloud can be reliably determined, otherwise these points will be ignored during the voxelization process. Since the triangle is sampled with a certain accuracy and sampling resolution, the triangle is now expanded along at least one side to expand the surface of the triangle, so that the points of the point cloud outside the triangle can be captured. In addition, since the expansion of the triangle is based on the sampling distance of the point cloud, the expansion will adapt to any point cloud regardless of the sampling distance. Preferably, the expansion is proportional to the sampling distance of the point cloud. Thus, if the sampling distance of the point cloud becomes larger, the triangle will also be expanded to a greater extent. As a result, a higher accuracy of reconstructing the 3D point cloud can be achieved and the number of sampling errors in the voxelization process can be reduced. In addition, the complexity of the encoding and / or decoding algorithm remains unchanged.

[0040] Preferably, at least one triangle is expanded on more than one side to further expand the surface of the corresponding triangle. Thus, a triangle can be expanded on one side, two sides, or all three sides to include points of the original point cloud that are just outside the triangle defined by the vertices on the sides of the cube.

[0041] Preferably, if a cube of a leaf node in the octree structure can contain more than one triangle, each triangle in the cube is extended along at least one side for voxelization. Therefore, the extension of the triangular surface can be applied to all triangles in the cube. As an alternative or in addition, in each cube of the octree structure, at least one triangle is extended along at least one side for voxelization. Alternatively, the extension of one or more sides of the triangle will only be applied to a subset of the leaf nodes in the octree structure. Wherein, the subset can be determined by, for example, the application, the density of points in the leaf nodes of the point cloud, or the requirements for accuracy and decoding speed. More preferably, one or more sides of the triangle are extended based on the local sampling distance. Therefore, the triangles of each subset of leaf nodes can be extended in a manner that can achieve local optimal performance.

[0042] Preferably, the expansion is the same for each side. Thus, the triangle is expanded by the same amount in at least two directions to enlarge the surface of the triangle. More preferably, the amount of expansion in all three directions is the same. Alternatively, the expansion in at least two directions is different. Thus, different directions can be treated differently to improve the accuracy of decoding.

[0043] Preferably, the extension of each leaf node of the octree structure is the same or different. If the extension of more than one side or each side of a triangle in a leaf node of the octree structure is different, the extensions in other leaf nodes of the octree structure may be the same or different. The extension may be pre-selected or determined by, for example, the density of points in the leaf nodes of the application, the point cloud, or the requirements for accuracy and decoding speed.

[0044] Preferably, voxelization is performed using -Trumbore algorithm is performed.

[0045] Preferably, -In the Trumbore algorithm, the convex hull is required to be relaxed to -ε a ≤u,v,w, where ε a >0, and u, v, w are the barycentric coordinates of the triangle, where ε a Sampling distance d based on point cloud sampl OK. In the initial -In the Trumbore algorithm, the convex hull requirement is set to 0≤u,v,w. Therefore, by relaxing this requirement to -ε a ≤u,v,w, the surface of the considered triangle is enlarged and voxelization of points of the original point cloud that would not have been considered in the reconstructed point cloud during sampling is now included. In particular, due to ε a Sampling distance d based on point cloud sampl The expansion is thus adapted to any point cloud, regardless of the sampling distance. Preferably, the expansion is proportional to the sampling distance of the point cloud. Thus, if the sampling distance of the point cloud becomes larger, the triangles will also be expanded to a greater extent. Thus, the reconstruction quality and appearance of the final reconstructed point cloud is improved.

[0046] Preferably, the convex hull requirement is set to -ε u_a ≤u, -ε v_a ≤v and -ε w_a ≤w, where ε u_a ,ε v_a ,ε w_a ≥0, u, v, w are the barycentric coordinates of the triangle, where ε u_a , ε v_a , ε w_a At least one of the sampling distances d based on the point cloud sampl Therefore, for different directions, separate convex hull requirements can be provided to individually control the expansion of the considered triangle. u_a ≠ε w_a As an alternative or in addition, ε u_a ≠ε v_a As an alternative or in addition, ε v_a≠ε w_a Thus, the extension in one or more directions may be selected independently of the other directions to determine that extension alone.

[0047] Preferably, the expansion is provided by an adaptive halo parameter. -In the Trumbore algorithm, the adaptive halo parameter is determined by ε a provided, and in the case of different directions by ε u_a , ε v_a and ε w_a Thus, the amount of expansion is determined by an adaptive halo parameter and can be quantified based on the sampling distance of the point cloud.

[0048] Preferably, the adaptive halo parameter is set to be less than d sampl More preferably, the adaptive halo parameter is set to be less than d sampl Therefore, by selecting the adaptive halo parameter, the amount of expansion can be tailored to achieve the best results, where larger values ​​will result in more points being determined during the voxelization process. The preferred range for the adaptive halo parameter is 0 to d sampl If the sampling distance increases, the adaptive halo parameter will also increase, thereby increasing the expansion amount. Therefore, even if the sampling distance changes, the present invention also provides an adaptive solution for expanding the triangle, thereby ensuring that the expanded triangle always covers a reasonable number of points.

[0049] Preferably, the adaptive halo parameters are pre-set. Thus, the encoder and decoder may have agreed on the adaptive halo parameters, so that for each point cloud generated by the encoder and reconstructed by the decoder, the adaptive halo parameters are fixed. Information about the adaptive halo parameters does not need to be encoded into the bitstream.

[0050] Alternatively, the adaptive halo parameters are encoded into the bitstream, preferably, into the geometric parameter set GPS of the bitstream. In the case of setting the adaptive halo parameters for all subsequent point clouds to be decoded, the encoding into the bitstream may be performed once. Alternatively, the adaptive halo parameters or the adaptive halo parameter set may be encoded separately for each point cloud.

[0051] As an alternative, the adaptive halo parameters also depend on the volume size of the cube, i.e. the level of the octree of the current leaf node.

[0052] Preferably, the sampling distance d of the point cloud is sampl Depend on OK, its N leaf is the number of leaf nodes, N totalis the number of points in the point cloud, N is the size of each leaf node cube, or the sampling distance d of the point cloud sampl Determined by a cyclic method. In which, on the encoder side, N total is known to the encoder. And, the number of leaf nodes N leaf It is known on the encoder side. Further, N defines the size of the leaf node in units of the sampling resolution of the original point cloud data acquired by the device. Therefore, d sampl can be determined from the point cloud data before voxelization and depends on the size of the leaf node cube. Thus, as the leaf node size N increases, d sampl will also increase, thereby increasing the adaptive halo parameter. As a supplement or alternative, during the voxelization process, the sampling distance can also be determined by a loop method to select the best sampling distance. Specifically, the loop method starts from 1 to N, tries different integer values ​​to estimate the sampling distance, and increases the sampling distance from this loop by 1 to enter the next loop. In each loop k, by using the sampling distance d of the loop k To estimate the number of points in the reconstructed point cloud generated during the voxelization process, and compare this number with the N of the original point cloud total Compare; if the number of points in the reconstructed point cloud in the i-th cycle is greater than N total , then the loop method ends and the estimated sampling distance for voxelization is equal to d i -1.

[0053] Preferably, at least one triangle is extended along at least one side to be based on the weighted halo parameter ε a_t Voxelization is performed, where the weighted halo parameter ε a_t By ε a_t =ε a *t is determined, ε a is the sampling distance d based on the point cloud sampland providing an extended adaptive halo parameter for at least one triangle, t is a corresponding weight associated with the sampling distance, preferably t is set to 2. In some embodiments, t is selected to be between 1 and 4, more preferably between 1.5 and 2.5. Wherein, a heuristic method can be used to determine the value of t. A heuristic method is an optimization method that attempts to find the global optimal feasible solution to a specific problem under consideration. Heuristic methods are iterative in nature. After each iteration, a feasible solution to a specific problem is determined. When the heuristic method terminates after a certain time or a certain number of iterations, the output solution is the best solution found in any iteration. More preferably, the weight to be tried in each iteration is an integer selected from a range of 1 to 4. Wherein, the adaptive halo parameter is less than 1. If the weight is too large, the overall accuracy of the TriSoup model may be affected. Therefore, the upper limit may be set to 4. For example, if the adaptive halo parameter is 1 / 4 and it is determined that the best result can be obtained by assigning a weight of 2 to the sampling distance, then if the adaptive halo parameter is proportional to the sampling distance, the updated adaptive halo parameter may be 1 / 4*2=1 / 3. Thus, by providing an appropriate weight setting range, the efficiency and accuracy of the overall algorithm may be further improved. It is understandable that different weights may also be determined in different directions of the triangle.

[0054] In another aspect of the present invention, a method for encoding a 3D point cloud into a bit stream is provided, preferably implemented in an encoder. The 3D point cloud encoding method comprises:

[0055] Obtaining octree information, the octree information including an octree structure of a body, the body including a plurality of cubes;

[0056] Acquire vertex information of each cube associated with the leaf node from the surface of the point cloud, wherein the vertex information includes information about the existence of the vertex and information about the position of the vertex on the edge of the cube;

[0057] Encoding the octree information and the vertex information into a bitstream;

[0058] Reconstruct the point cloud geometry data using the octree information and vertex information obtained in the aforementioned encoding process;

[0059] Among them, the reconstructed point cloud geometric data includes:

[0060] The triangles are determined by connecting the vertices of a cube associated with the leaf nodes of the octree structure;

[0061] voxelize the triangles to determine the points of the point cloud;

[0062] Wherein, at least one triangle is extended along at least one side to a sampling distance d based on the point cloud.sampl Voxelize.

[0063] Therefore, through the encoding method, octree information and vertex information are generated. This information is encoded as a bitstream. A reconstruction step is then performed on the encoder side. In this reconstruction step, the point cloud geometry information is reconstructed, wherein the reconstruction step is the same as the reconstruction step in the above decoding method. Then, the point cloud geometry reconstructed on the encoder side is used to encode the attributes of the points of the point cloud (color, reflectivity, etc.), for example, by using a regional adaptive hierarchical transform RAHT, a prediction transform or a lifting transform to encode the attributes of the points of the point cloud.

[0064] Preferably, the point cloud geometry is encoded into the bitstream via geometry-based point cloud compression G-PCC.

[0065] Preferably, the bitstream is an MPEG G-PCC compliant bitstream.

[0066] Preferably, the encoding method is further established according to the features described above in relation to the decoding method.

[0067] In another aspect of the present invention, an encoder for encoding a 3D point cloud into a bit stream is provided. The encoder comprises a memory and a processor, wherein instructions are stored in the memory, and when the processor executes the instructions, the steps of the encoding method described above are performed.

[0068] In another aspect of the present invention, a decoder for decoding a 3D point cloud from a bit stream is provided. The decoder comprises a memory and a processor, wherein instructions are stored in the memory, and when the processor executes the instructions, the steps of the decoding method described above are performed.

[0069] In another aspect of the present invention, a bit stream is provided, wherein the bit stream is encoded by the steps of the encoding method described above.

[0070] In another aspect of the present invention, a computer-readable storage medium is provided, comprising instructions for executing the steps of the above method of encoding a 3D point cloud into a bitstream.

[0071] In another aspect of the present invention, a computer-readable storage medium is provided, comprising instructions to execute the steps of the above method for decoding a 3D point cloud from a bitstream. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Hereinafter, the present invention is described in more detail with reference to the accompanying drawings.

[0073] The attached figure shows:

[0074] Figure 1 A flowchart of a method for decoding 3D point cloud geometry according to the present invention is shown.

[0075] Figure 2 An embodiment of generating an octree structure is shown.

[0076] Figure 3 Shown according to Figure 2 The octree of

[0077] Figure 4 An embodiment of determining the vertices on the edge of a cube is shown,

[0078] Figure 5 An embodiment of generating triangles is shown,

[0079] Figure 6 An example of vertices on the sides of a cube is shown,

[0080] Figure 7 shows the triangles generated by vertices,

[0081] Figure 8 Shown according to Figure 7 An embodiment of determining the triangle order,

[0082] Fig. 9 A schematic diagram showing the voxelization step,

[0083] Fig.10 shows a 2D representation of the triangles in an octree leaf node,

[0084] Fig.11 Shows Fig.10 An example of voxelization of a triangle in,

[0085] Fig.12 Shows Fig.10 The barycentric coordinates of the triangle in and their definitions,

[0086] Fig.13 shows the comparison between the vertices of the triangle and the original point cloud,

[0087] FIG. 14 shows Fig.10 The triangle in is extended in one direction in the barycentric coordinate system,

[0088] FIG. 15 shows Fig.10 The triangle in expands in all three directions,

[0089] Fig.16 The weighted halo parameter ε is shown a_t and the point cloud sampling distance is extended in three directions.

[0090] Fig.17a shows a representation of triangles expanded in three directions based on the point cloud sampling distance,

[0091] Fig.17b shows a representation of a triangle expanded in three directions by a fixed amount, and

[0092] Fig.18 A schematic flow chart of the encoding method is shown. DETAILED DESCRIPTION

[0093] refer to Figure 1 , a schematic diagram showing a method for decoding 3D point cloud geometry information from a bitstream.

[0094] A method for decoding 3D point cloud geometry from a bitstream, preferably implemented in a decoder, comprises the following steps:

[0095] In step S01, a bit stream is received and decoded, wherein the bit stream includes octree information and vertex information, wherein the octree information includes information about an octree structure of a point cloud volume, and the vertex information includes information about the existence of a vertex and position information about the vertex on an edge of a leaf node cube in the octree structure;

[0096] In step S02, a triangle is determined by connecting the vertices of a cube associated with a leaf node in the octree structure;

[0097] In step S03, the triangles are voxelized to determine the points of the point cloud.

[0098] Where the sampling distance d based on the point cloud sampl Prior to voxelization, at least one triangle is expanded along at least one side.

[0099] To determine the octree information, the first step of the geometry encoding process is to build and encode the octree, such as Figure 2 and Figure 3 The bounding box is the volume 100 that contains all the points and is associated with a root node 112 (i.e., the single node at the top of the tree 110). The volume 100 is first divided into eight sub-volumes 102, called octants, each represented by a node 114 in the tree 110. Then, the octants 106 (in Figure 2 and Figure 3 The layer 104 is recursively divided into sub-volumes 104 until the target layer is reached.

[0100] Each octant (or node) is represented by an occupation byte containing one bit per child octant, set to 1 if it is occupied by at least one node, and set to 0 otherwise. The occupation bytes 118 of all octants are serialized (in breadth-first order) and entropy encoded using a binary arithmetic encoder.

[0101] Figure 4A tiled representation of a 3D surface 210 is shown, along with an example of a tile 220 in TriSoup. Surface 210 intersects tile 220, so tile 220 is an occupied tile, and tile 220 exists in multiple tiles 200 in 3D space. Within tile 220, the enclosed portion of surface 210 intersects the edges of the tile at the six shown vertices of polygon 230. If an edge of a tile 220 contains a vertex, it is said to be selected.

[0102] Figure 5 Block 220 in TriSoup is shown, omitting surface 210 for clarity, and showing unselected edge 270, selected edge 260, and i-th edge 250. Assume that i-th edge 250 is selected. To specify vertex vi on the i-th edge, a scalar value is specified to indicate the corresponding proportion of the length of edge 250.

[0103] like Figure 4 and Figure 5 As shown, within each octant 220 in the octree target layer, trisoup represents the original surface 210 as a set of triangles 245. The surface is encoded and used to obtain the position of the reconstructed (or decoded) point. First, the intersection of the surface represented by the original point and the edges of the octant is estimated by averaging the positions of the points closest to these edges within the octant. Secondly, the twelve edges of all octants and their associated intersections (if any) are stored as segments and vertices respectively. Each (unique) segment is then encoded as follows. The first single bit is arithmetically encoded and is set to 1 if the segment is occupied by a vertex, otherwise it is set to 0. If it is occupied, the relative position of the vertex on the segment is also arithmetically encoded.

[0104] The vertices 310 of the triangles are encoded along the edges 320 of the volume associated with the leaf nodes 300 of the tree, as Figure 6 As shown. These vertices 310 on the edge 320 are shared between the leaf nodes 300 with the common edge 320. This means that at most one vertex is encoded for each edge belonging to at least one leaf node. In this way, the continuity of the model is ensured through the leaf nodes.

[0105] As mentioned above, the encoding of TriSoup vertices requires two pieces of information for each edge:

[0106] a vertex flag indicating whether there is a TriSoup vertex on this edge, and

[0107] The positions of vertices along the edges, when present.

[0108] Therefore, the encoded data includes the Octree data plus the TriSoup data.

[0109] The vertex flags are encoded by an adaptive binary arithmetic encoder that uses a specific context to encode the vertex flags. s The vertex positions on the edges of can be encoded with unit precision by pushing s bits into the bitstream (bypassing / not doing entropy coding).

[0110] In a leaf node, if there are at least three vertices 310 on the edge 320 of the leaf node 300, a triangle is constructed from the TriSoup vertices. Figure 7 Reconstructed triangles 330, 340 are depicted in FIG.

[0111] Obviously, other combinations of triangles 330, 340 are possible. The selection of triangles results from a three-step process:

[0112] 1. Determine the dominant direction along one of the three axes

[0113] 2. Sort TriSoup vertices according to the dominant direction

[0114] 3. Construct a triangle based on an ordered list of vertices

[0115] It is not necessary to know the exact position of the triangles within the current leaf, this can be inferred from the vertices.

[0116] Figure 8 will be used to explain this process. Each of the three axes is tested and the one that maximizes the total triangle surface is taken as the dominant axis. To simplify the diagram, Figure 8 Only the test conditions for two axes are depicted.

[0117] The first (top) test along the vertical axis is performed by projecting the cube and trisoup vertex 310 vertically onto the 2D plane. The vertices 310 are then sorted in clockwise order relative to the center of the projection node (square). Triangles 330, 340 are then constructed according to a fixed rule based on ordered vertices. Here, when 4 vertices are involved, triangles 123 and 134 are systematically constructed. When there are 3 vertices, the only possible triangle is 123. When there are 5 vertices, the fixed rule may be to construct triangles 123, 134 and 451. And so on, up to 12 vertices.

[0118] The second (left) test along the horizontal axis is performed by projecting the cube and trisoup vertices horizontally onto the 2D plane.

[0119] The vertical projection shows the largest total triangular 2D surface, so the dominant axis is selected as the vertical axis, and the constructed TriSoup triangles are obtained within the node according to the order of vertical projection, such as Figure 8It is important to note that taking the horizontal axis as the dominant axis leads to an alternative construction of the triangle.

[0120] Proper selection of the dominant axis by maximizing the projection surface allows for continuous reconstruction of the point cloud without holes.

[0121] The process of rendering TriSoup triangles into points is achieved by ray tracing. The set of all points rendered by ray tracing will constitute the decoded point cloud.

[0122] for Fig. 9 The ray tracing shown, the rays are emitted in three directions parallel to the axes. Their origin is a point with integer (voxelized) coordinates, whose precision corresponds to the sampling precision required for rendering. Then, the intersection with one of the trisoup triangles (the dashed point if there is one) is voxelized (= rounded to the point closest to the required sampling precision) and added to the list of rendered points.

[0123] After Trisoup is applied to all leaf nodes, i.e. after triangles are constructed and points are obtained by ray tracing, all copies of the same point in the rendered point lists are discarded (i.e. only one voxel is kept among all voxels sharing the same position and volume) to obtain a set of (unique) decoded points.

[0124] For simplicity, start here. Figures 10 to 16 Instead of 3D volumes (cubes), 2D bodies (squares) associated with leaf nodes will be depicted. However, readers are kindly reminded that all methods described in this invention are applicable to 3D space.

[0125] refer to Fig.10 , shows an example of an N*N*N body, N=2 s = 8. There are at least three vertices V1, V2, and V3 on the edge 410 of the body (depicted as a square in the figure, but actually a cube).

[0126] The edges of the leaves are located at -0.5 and N-0.5 to ensure continuity of the TriSoup model when transferring from one volume to an adjacent volume. In practice, this means that the faces of the cubes are shared between adjacent volumes. Thus, the position of the vertices present on the edges does not depend on which cube the edge belongs to.

[0127] The position p of the vertex along its edges k are the quantized positions 400 and encoded into the bitstream. These positions 400 can be quantized with a unit step size so that p k is an integer in the interval [0, N-1]. Fig.10 In the example, p1=4, p2=2 and p3=2.

[0128] A TriSoup triangle 440 is constructed from vertices V1 , V2 , V3 , and the set of triangles belonging to the volume models the point cloud contained by the volume.

[0129] The process of recovering the points 430 (of the decoded point cloud) from the triangles 440 is called voxelization of the triangles. Fig.11 Shows Fig.10 Voxelization of the TriSoup triangle. Rays are fired along all integer coordinates 420 (white and black dots), and the rays that intersect the triangle point to a portion of the decoded points (black dots). The starting points of the rays have a spacing D, which sets the sampling resolution of the voxelization.

[0130] The intersection of the ray and the triangle is calculated using - obtained by the Trumbore algorithm, which uses the barycentric coordinates to determine the location of the intersection point, such as Fig.12 Depicted.

[0131] Any point P in 3D space can be uniquely represented by its barycentric coordinates relative to any non-degenerated 3D triangle ABC (equivalent to any triangle V1V2V3 in the TriSoup model).

[0132] Any point P in 3D space can be uniquely represented as

[0133] P=uA+vB+wC

[0134] The condition is

[0135] u+v+w=1.

[0136] The points of the triangle correspond to the convex hull. Therefore,

[0137] 0≤u,v,w.

[0138] -Trumbore determined the values ​​of u and v; then w was simply found by v = 1-uv. Fig.12 , the ray follows the direction From point P start To launch. Fig.12 The 3D vectors derived from the 3D points shown are set to the following signs:

[0139] as well as

[0140] The intersection point P between the ray and the unique plane passing through A, B, and C is found by the following calculation.

[0141] ·

[0142] ·

[0143] ·

[0144] ·

[0145] ·

[0146] ·

[0147] ·

[0148] The intersection point P belongs to the triangle if and only if 0≤u,v,w.

[0149] like Fig.13 As shown, there is a slight offset between the position of the TriSoup triangle V1V2V3 determined by the vertex from the bitstream and the natural position 450 of the triangle in the current volume. The natural position is because the vertex V k The position of is what the encoder has inferred from the closest point in the original point cloud (with respect to the edge). k Nearby voxelized points are likely points of the point cloud. These points are natural candidates for building "natural" triangles that model the point cloud.

[0150] This offset is due to the continuity constraints of adjacent volumes. It causes ray tracing to miss some points 460 ( Fig.13 P miss ), because these points do not belong to the TriSoup triangle (with Fig.11 440 as determined by the vertices provided by the bitstream as shown in ). The direct consequence is that the quantitative geometric indicators are degraded and the rate-distortion performance of the scheme is reduced.

[0151] Therefore, an adaptive "halo" is created around the TriSoup triangle by slightly relaxing the convex hull condition 0≤u,v,w. This way, the size of the triangle increases slightly, so that the ray trace will intersect the enlarged triangle and miss fewer points P miss .

[0152] The advantages of the proposed method are:

[0153] The decoded point cloud has less distortion. In fact, the quantitative indicator BDBR shows that it is comparable to non-automatic point clouds.

[0154] Compared with the adaptive method, the method proposed in the present invention can achieve a 2.6% bit rate gain (under the condition of equal quality).

[0155] The complexity remains the same, since the overall algorithm does not change.

[0156] Let ε a >0 as the adaptive halo parameter, which is determined based on the sampling distance of the point cloud. As shown in Figure 14, the condition 0≤u is relaxed to ε a ≤u, where u is the centroid weight associated with point A, the triangle grows in the direction opposite to point A along the side BC indicated by the dashed area 470.

[0157] By changing the convex hull 0≤u,v,w to

[0158] -ε a ≤u,v,w,

[0159] The condition is relaxed to apply to the three centroid weights u, v and w.

[0160] FIG15 shows the obtained halo 480 around the triangle 440. In a first order approximation, the size of the halo is proportional to the adaptive halo parameter ε a Therefore, it is also proportional to the sampling distance of the point cloud.

[0161] In one embodiment, the adaptive halo parameters may also depend on each barycenter weight of the triangle, e.g.

[0162] -ε u_a ≤u, -ε v_a ≤v, and -ε w_a ≤w,

[0163] where ε u_a , ε v_a and ε w_a are three adaptive halo parameters.

[0164] Fig.17a and Fig.17b The effect on voxelization is depicted. Fig.17b In contrast, when the sampling distance becomes larger and the halo parameter remains fixed (suitable for smaller sampling distances, such as Fig.17c As shown), there are many missing points P miss ,exist Fig.17a Since the adaptive halo parameters according to the present invention are applied, several missing points P miss are now part of the halo and, therefore, are decoded as points of the decoded point cloud. Therefore, they are not missed as in the original algorithm.

[0165] refer to Fig.16 ,and Fig.17a Compared to, or even more P missNow they are all part of the halo. A larger halo is provided by the weighted halo parameter. The weighted parameter not only takes into account the sampling distance of the point cloud, but also assigns a weight t to the sampling distance. Fig.16 In , the weight t is set to 2. Thus, the accuracy of the 3D point cloud decoding or reconstruction process is further improved.

[0166] Of course, the adaptive halo parameter ε a (can be replaced by ε u_a , ε v_a and ε w_a ) must be set so that the halo has the appropriate size. a If ε is too small, the halo will be very small and have almost no effect, thus returning to the problem of missed points in the prior art. If ε is too large, the halo will become larger, affecting the overall accuracy of the TriSoup model. In both cases, the distortion of the decoded point cloud is not optimal.

[0167] It is observed that the halo parameter ε a A reasonable value of a ≈d sampl / 4 or ε a ≈d sampl / 8.

[0168] If the sampling distance is fixed, the adaptive halo parameter ε a (can be replaced by ε u_a , ε v_a and ε w_a ) can be a fixed value. In one variant, the halo parameter ε a (can be replaced by ε u_a , ε v_a and ε w_a ) is encoded into the bit stream, for example into the geometry parameter set GPS. In another variant, the halo parameter ε a (can be replaced by ε u_a , ε v_a and ε w_a ) further depends on the size N of the body. In yet another variant, the adaptive halo parameter ε a (can be replaced by ε u_a , ε v_a and ε w_a ) is a local reflection of the collection of volumes representing the point cloud.

[0169] refer to Fig.18 , showing a flow chart of a method for encoding a 3D point cloud into a bit stream according to the present invention. The method comprises:

[0170] In step S11, octree information is determined, the octree information includes an octree structure of a volume, and the volume includes a plurality of cubes;

[0171] In step S12, vertex information of each cube associated with the leaf node is obtained from the surface of the point cloud, wherein the vertex information includes information about the existence of the vertex and position information about the vertex on the edge of the cube;

[0172] In step S13, the octree information and the vertex information are encoded into a bit stream;

[0173] In step S14, the octree information and vertex information obtained in the above encoding process are used to reconstruct the point cloud data, wherein the reconstructed point cloud data includes:

[0174] In step 141, a triangle is determined by connecting the vertices of a cube associated with a leaf node in the octree structure;

[0175] In step 142 , the triangles are voxelized to determine points of the point cloud;

[0176] Wherein, at least one triangle is extended along at least one side to a sampling distance d based on the point cloud. sampl Voxelize.

[0177] Steps S11 to S13 involve known TriSoup coding, such as "Adaptive multi-level triangle soup for geometry-based point cloud coding" by A.DRICOT et al. at the 2019 IEEE 21st International Symposium on Multimedia Signal Processing MMSP, "Report on triangle soup decoding" by Nakagami O., m52279 of ISO / IEC JTC1 / SC29-WG11 in 2020, and US10,192,353. In addition to the usual encoding of the point cloud, the method also includes a reconstruction step, which includes the same as before, especially the reference step. Figure 1 The same or similar steps in the decoding method described above can then be used to interpolate attributes such as color using the reconstructed point cloud, and then encode the attributes of the points of the point cloud based on the reconstructed geometry.

[0178] Other aspects of the invention are described below.

[0179] In one embodiment, a method for decoding 3D point cloud geometry from a bitstream is provided, preferably implemented in a decoder, comprising:

[0180] Receive and decode a bitstream, wherein the bitstream includes octree information and vertex information, wherein the octree information includes information about an octree structure of a point cloud volume, and the vertex information includes information about the existence of a vertex and position information about a vertex on an edge of a leaf node cube in the octree structure;

[0181] The triangles are determined by connecting the vertices of a cube associated with the leaf nodes in the octree structure;

[0182] voxelize the triangles to determine the points of the point cloud,

[0183] Wherein, at least one triangle is extended along at least one side to a sampling distance d based on the point cloud. sampl Voxelize.

[0184] In another embodiment, a method for encoding a 3D point cloud into a bitstream is provided, preferably implemented in an encoder, comprising:

[0185] Obtaining octree information, the octree information including an octree structure of a body, the body including a plurality of cubes;

[0186] Acquire vertex information of each cube associated with the leaf node from the surface of the point cloud, wherein the vertex information includes information about the existence of the vertex and information about the position of the vertex on the edge of the cube;

[0187] Encode octree information and vertex information into bitstream;

[0188] The point cloud geometric data is reconstructed using the octree information and vertex information obtained in the aforementioned encoding process, wherein the point cloud data is reconstructed including:

[0189] The triangles are determined by connecting the vertices of a cube associated with the leaf nodes in the octree structure;

[0190] voxelize the triangles to determine the points of the point cloud;

[0191] Wherein, at least one triangle is extended along at least one side to a sampling distance d based on the point cloud. sampl Voxelize.

[0192] In one embodiment, the encoding is a Trisoup encoding.

[0193] In one embodiment, at least one triangle is expanded on two or three sides for voxelization.

[0194] In one embodiment, each triangle in the cube is expanded, and preferably at least one triangle in each cube having triangles in the point cloud is expanded.

[0195] In one embodiment, the extension is the same for each side or different for at least two sides.

[0196] In a voxelized implementation, use algorithm, and / or obtain voxelization of the points by rounding their coordinates to the nearest integer.

[0197] In one embodiment, the convex hull requirement is -ε≤u,v,w, where ε>0, and u,v,w are the barycentric coordinates of the triangle.

[0198] In one embodiment, the convex hull requirement is

[0199] -ε u ≤u, -ε v ≤v, and -ε w ≤w,

[0200] Among them, ε u , ε v , ε w > 0, and u, v, w are the barycentric coordinates of the triangle, and ε u ≠ε w and / or ε u ≠ε v and / or ε v ≠ε w .

[0201] In one embodiment, the expansion is provided by a halo parameter, and the expanded halo parameter is less than 1 / 4 of the voxelized sampling accuracy, and preferably less than 1 / 8 of the voxelized sampling accuracy.

[0202] In one embodiment, the spread is provided by a halo parameter and the spread is pre-set.

[0203] In one embodiment, the extension is provided by halo parameters and the halo parameters are encoded in the bitstream, preferably into a geometry parameter set.

[0204] In another embodiment, an encoder is provided for encoding a 3D point cloud into a bit stream, the encoder comprising at least one processor and a memory, wherein the memory stores instructions that, when executed by the processor, perform the steps of the encoding method of one of the above embodiments.

[0205] In another embodiment, a decoder is provided for decoding a 3D point cloud from a bitstream, the decoder comprising at least one processor and a memory, wherein the memory stores instructions that, when executed by the processor, perform the steps of the decoding method of one of the above embodiments.

[0206] In another embodiment, a bit stream encoded by the encoding method of one of the above embodiments is provided.

[0207] In another embodiment, a computer-readable storage medium is provided, which includes instructions that, when executed by a processor, perform the steps of the encoding or decoding method of one of the above embodiments.

Claims

1. A method for decoding 3D point cloud geometry from a bitstream, preferably implemented in a decoder, comprising: Receive and decode a bitstream, wherein the bitstream includes octree information and vertex information, the octree information includes information about an octree structure of a point cloud volume, and the vertex information includes information about the existence of a vertex and position information about a vertex on an edge of a leaf node cube in the octree structure; The triangles are determined by connecting the vertices of a cube associated with the leaf nodes in the octree structure; voxelize the triangles to determine the points of the point cloud, Wherein, at least one triangle is extended along at least one side to a sampling distance d based on the point cloud. sampl Voxelize.

2. A method for encoding a 3D point cloud into a bitstream, preferably implemented in an encoder, comprising: Obtaining octree information, the octree information including an octree structure of a body, the body including a plurality of cubes; Acquire vertex information of each cube associated with the leaf node from the surface of the point cloud, wherein the vertex information includes information about the existence of the vertex and information about the position of the vertex on the edge of the cube; Encoding the octree information and the vertex information into a bit stream; The point cloud geometric data is reconstructed using the octree information and vertex information obtained in the aforementioned encoding process, wherein the point cloud data is reconstructed including: The triangles are determined by connecting the vertices of a cube associated with the leaf nodes in the octree structure; voxelizing the triangles to determine points of a point cloud; Wherein, at least one triangle is extended along at least one side to a sampling distance d based on the point cloud. sampl Voxelize.

3. The method of claim 2, wherein the encoding is a Trisoup encoding. 4 . The method according to claim 1 , wherein the at least one triangle is expanded on two or three sides for voxelization. 5 . The method according to claim 1 , wherein each triangle in a cube is expanded, preferably at least one triangle in each cube having triangles of the point cloud is expanded.

6. The method according to any one of claims 1 to 5, wherein the extension is the same for each side or different for at least two sides.

7. The method according to any one of claims 1 to 6, wherein the - voxelization using the Trumbore algorithm, and / or, voxelization of points is obtained by rounding their coordinates to the nearest integer.

8. The method according to claim 7, wherein the convex hull is required to be -ε a ≤u,v,w, where ε a >0, and u, v, w are the barycentric coordinates of the triangle, where ε a Sampling distance d based on point cloud sampl Sure.

9. The method according to claim 7, wherein the convex hull requirement is -e u_a ≤u,-e v_a ≤v, and -e w_a ≤w, ε u_a ,ε v_a ,ε w_a >0, and u, v, w are the barycentric coordinates of the triangle, and ε u_a ≠ε w_a and / or ε u_a ≠ε v_a and / or ε v_a ≠ε w_a , where ε u_a , ε v_a , ε w_a At least one of the sampling distances d of the point cloud is based on sampl Sure.

10. The method according to any one of claims 1 to 9, wherein the expansion is provided by a halo parameter, and the halo parameter of the expansion is less than d sampl 1 / 4 of d, and preferably less than d sampl 1 / 8 of.

11. A method according to any one of claims 1 to 10, wherein the extension is provided by an adaptive halo parameter and the extension is pre-set.

12. The method according to any one of claims 1 to 10, wherein the extension is provided by adaptive halo parameters and the adaptive halo parameters are encoded into the bitstream, preferably into a geometry parameter set.

13. The method according to any one of claims 1 to 12, wherein the sampling distance d of the point cloud is sampl Depend on Determine, where N leaf is the number of leaf nodes, N total is the number of points in the point cloud, N is the size of each leaf node cube, or the sampling distance d of the point cloud sampl Determined by round-robin method.

14. The method according to any one of claims 1 to 13, wherein the at least one triangle is extended along at least one side for voxelization based on a weighted halo parameter ε a_t wherein the weighted halo parameter ε a_t is determined by ε a_t = ε a * t (1 < t < 4), ε a being an adaptive halo parameter based on a sampling distance d of the point cloud sampl and providing an extension for the at least one triangle, and t being a corresponding weight associated with the sampling distance, preferably t is set to 2.

15. An encoder for encoding a 3D point cloud into a bitstream, comprising at least one processor and a memory, wherein: The memory stores instructions which, when executed by the processor, perform the steps of the method according to any one of claims 2 to 14.

16. A decoder for decoding a 3D point cloud from a bitstream, comprising at least one processor and a memory, wherein: The memory stores instructions which, when executed by the processor, perform the steps of the method according to claim 1 and any one of claims 3 to 14 as dependent thereon.

17. A bit stream encoded according to the method of any one of claims 2 to 14.

18. A computer readable storage medium comprising instructions which, when executed by a processor, perform the steps of the method according to any one of claims 1 to 14.

Images