Decoding method, encoding method, decoding device, and encoding device
Patent Information
- Application Number
- MYPI2025001361
- Authority / Receiving Office
- MY · MY
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-04-11
- Filing Date
- 2023-08-28
- Publication Date
- 2024-03-09
AI Technical Summary
Current point cloud compression methods, such as the TriSoup method, face challenges in accurately reproducing the original shape of three-dimensional data due to irreversible compression, leading to discrepancies between the encoded and decoded point clouds, especially when representing ridge lines and curved surfaces across node boundaries.
The proposed method generates additional vertices, including face vertices, on shared surfaces between nodes to create triangles that span across node boundaries, improving the reproducibility of the decoded point cloud by approximating the original shape more accurately, and includes a bitstream to control the generation of these vertices based on specific conditions.
This approach enhances the reproducibility of the original point cloud shape in the decoded data by generating triangles that pass through areas other than node edges, effectively addressing the limitations of existing methods by improving the representation of ridge lines and curved surfaces.
Smart Images

Figure PI2025001361_DRAWING_001 
Figure PI2025001361_DRAWING_002 
Figure PI2025001361_DRAWING_003
Abstract
Description
Decoding method, encoding method, decoding device, and encoding device
[0001] The present disclosure relates to a decoding method, an encoding method, a decoding device, and an encoding device.
[0002] In the future, devices and services that utilize 3D data are expected to become widespread in a wide range of fields, including computer vision for autonomous operation of automobiles or robots, map information, surveillance, infrastructure inspection, video distribution, etc. 3D data can be acquired in a variety of ways, including distance sensors such as range finders, stereo cameras, or a combination of multiple monocular cameras.
[0003] One method of representing three-dimensional data is a representation called a point cloud, which represents the shape of a three-dimensional structure using a group of points in three-dimensional space. In a point cloud, the position and color of the points are stored. Point clouds are expected to become the mainstream method of representing three-dimensional data, but point clouds have a very large amount of data. Therefore, when storing or transmitting three-dimensional data, it is essential to compress the amount of data by encoding, just as with two-dimensional moving images (examples include MPEG-4 AVC or HEVC standardized by MPEG).
[0004] Furthermore, compression of point clouds is partially supported by a public library (Point Cloud Library) that performs processing related to point clouds.
[0005] Furthermore, a technique is known in which three-dimensional map data is used to search for and display facilities located around a vehicle (see, for example, Patent Document 1).
[0006] International Publication No. 2014 / 020663
[0007] In addition, lossy compression may be used as the encoding method. In such cases, the decoded point cloud does not completely match the original point cloud. Therefore, it is desirable to improve the reproducibility of the decoded point cloud.
[0008] An object of the present disclosure is to provide a decoding method, an encoding method, a decoding device, or an encoding device that can improve the reproducibility of a decoded point cloud.
[0009] A decoding method according to one aspect of the present disclosure is a decoding method for decoding a plurality of three-dimensional points, which generates a first vertex on a first face of a first node at a position other than an edge of the first node, generates a plurality of second vertices on a plurality of edges of the first node, generates a third vertex within the first node based on the plurality of second vertices, generates a triangle within the first node defined by the first vertex, a second vertex of the plurality of second vertices, and the third vertex, and generates the plurality of three-dimensional points on the face of the triangle.
[0010] An encoding method according to one aspect of the present disclosure is an encoding method for encoding a plurality of three-dimensional points, the encoding method comprising: generating a first vertex on a first face of a first node at a position other than an edge of the first node; generating a plurality of second vertices on a plurality of edges of the first node; generating a third vertex within the first node based on the plurality of second vertices; storing information regarding the first vertex, the plurality of second vertices, and the third vertex in a bitstream; the first vertex, the plurality of second vertices, and the third vertex being generated so as to generate a triangle within the first node defined by the first vertex, a second vertex of the plurality of second vertices, and the third vertex; and approximating the plurality of three-dimensional points to the triangle.
[0011] The present disclosure can provide a decoding method, an encoding method, a decoding device, or an encoding device that can improve the reproducibility of a decoded point cloud.
[0012] FIG. 1 is a diagram showing an example of an original point cloud according to an embodiment. FIG. 2 is a diagram showing an example of a pruned octree according to an embodiment. FIG. 3 is a diagram showing an example of a two-dimensional display of leaf nodes according to an embodiment. FIG. 4 is a diagram for explaining a method for generating centroid vertices according to an embodiment. FIG. 5 is a diagram for explaining a method for generating centroid vertices according to an embodiment. FIG. 6 is a diagram showing example vertex information according to an embodiment. FIG. 7 is a diagram showing an example of a Trithorpe surface according to an embodiment. FIG. 8 is a diagram for explaining a point cloud restoration process according to an embodiment. FIG. 9 is a diagram showing an example of a point cloud according to an embodiment. FIG. 10 is a diagram showing an example of generating centroid vertices according to an embodiment. FIG. 11 is a diagram showing an example of generating triangles (Trisope surfaces) according to an embodiment. FIG. 12 is a diagram showing a cross section of a node according to an embodiment. FIG. 13 is a diagram showing an example of generating temporary vertices according to an embodiment. FIG. 14 is a diagram showing an example of generating face vertices according to an embodiment. FIG. 15 is a diagram showing example syntax of a GDU according to an embodiment. FIG. 16 is a flowchart of an encoding process according to an embodiment. FIG. 17 is a flowchart of a decoding process according to an embodiment. FIG. 18 is a diagram for explaining a first variation of a method for generating face vertices according to an embodiment. FIG. 19 is a diagram for explaining a first variation of a method for generating face vertices according to an embodiment. FIG. 20 is a diagram for explaining a second variation of a method for generating face vertices according to an embodiment. FIG. 21 is a diagram for explaining a third variation of a method for generating face vertices according to an embodiment. FIG. 22 is a diagram for explaining a third variation of a method for generating face vertices according to an embodiment. FIG. 23 is a diagram for explaining a fourth variation of a method for generating face vertices according to an embodiment. FIG. 24 is a diagram showing an example of a node face existing between two nodes according to an embodiment. FIG. 25 is a diagram showing an example of a weighted average position of a point cloud according to an embodiment. FIG. 26 is a diagram showing an example of a weighted average position of intersections according to an embodiment. FIG. 27 is a diagram showing an example of a point cloud according to an embodiment. FIG. 28 is a diagram showing an example of generating centroid vertices according to an embodiment. FIG. 29 is a diagram showing an example of generating a triangle (Triteorpe surface) according to an embodiment.FIG. 30 is a diagram showing an example of generation of face vertices according to an embodiment. FIG. 31 is a diagram showing an example of a surface for evaluating connectivity and restoring centroid vertices according to an embodiment. FIG. 32 is a flowchart of encoding processing according to an embodiment. FIG. 33 is a flowchart of decoding processing according to an embodiment. FIG. 34 is a flowchart of transmission processing of face vertex information according to an embodiment. FIG. 35 is a flowchart of decoding processing of face vertex information according to an embodiment. FIG. 36 is a diagram showing an example of face vertex candidates according to an embodiment. FIG. 37 is a diagram showing an example of arrangement of vertices according to an embodiment. FIG. 38 is a diagram showing a vector NF according to an embodiment. FIG. 39 is a diagram showing an example of arrangement of vertices according to an embodiment. FIG. 40 is a diagram showing an example of arrangement of vertices according to an embodiment. FIG. 41 is a diagram showing an example of arrangement of vertices according to an embodiment. FIG. 42 is a diagram showing examples of edge vertices and centroid vertices according to an embodiment. FIG. 43 is a diagram showing an example of a correctly generated triangle according to an embodiment. FIG. 44 is a diagram showing an example of an erroneously generated triangle according to an embodiment. FIG. 45 is a diagram showing examples of edge vertices, centroid vertices, and face vertices according to an embodiment. FIG. 46 is a diagram showing an example of triangle generation when simplified ordering according to an embodiment is applied. FIG. 47 is a diagram for explaining ordering processing according to an embodiment. FIG. 48 is a diagram for explaining ordering processing according to an embodiment. FIG. 49 is a diagram for explaining ordering processing according to an embodiment. FIG. 50 is a diagram for explaining ordering processing according to an embodiment. FIG. 51 is a diagram showing an example of syntax of a GDU according to an embodiment. FIG. 52 is a diagram showing a modified example of syntax of a GDU according to an embodiment. FIG. 53 is a diagram showing a modified example of syntax of a GDU according to an embodiment. FIG. 54 is a flowchart of decoding processing according to an embodiment. FIG. 55 is a block diagram of a decoding device according to an embodiment. FIG. 56 is a flowchart of encoding processing according to an embodiment. FIG. 57 is a block diagram of an encoding device according to an embodiment.
[0013] A three-dimensional data decoding method according to one aspect of the present disclosure is a decoding method for decoding a plurality of three-dimensional points, the decoding method including: generating a first vertex on a first surface of a first node at a position other than an edge of the first node; generating a plurality of second vertices on a plurality of edges of the first node; generating a third vertex within the first node based on the plurality of second vertices; generating a triangle within the first node defined by the first vertex, a second vertex of the plurality of second vertices, and the third vertex; and generating the plurality of three-dimensional points on a surface of the triangle. This allows a decoding device to generate a first vertex on a surface of a node at a position other than an edge and generate a triangle using the first vertex. This allows for the generation of a triangle that does not pass through edges of the node, thereby improving the reproducibility of the shape of the original point cloud in the decoded point cloud.
[0014] For example, the first surface may be a surface common to the first node and a second node adjacent to the first node, and the first vertex may represent a plurality of 3D points located around the first surface in the first node and the second node. In this way, by generating a triangle using the first vertex, it is possible to improve the reproducibility of the plurality of 3D points located around the first surface in the decoded point cloud.
[0015] For example, the first vertex may be located on the first surface of the first node based on the position of the third vertex of the first node and the position of the third vertex of a second node adjacent to the first node, thereby improving the reproducibility of the shape of the original point cloud distributed between two second vertices, for example.
[0016] For example, the first surface may be a surface common to the first node and a second node adjacent to the first node, and the decoding method may further include generating a plurality of fourth vertices on a plurality of edges of the second node and generating a fifth vertex within the second node based on the plurality of fourth vertices, and generating the first vertex based on information indicating that the third vertex is connected to the fifth vertex. This can improve the reproducibility of the shape of a ridge line in the decoded point cloud, for example, when a ridge line is formed across a node boundary.
[0017] For example, the information may be provided for each of three mutually orthogonal faces of the first node, thereby making it possible to control whether or not to generate a first vertex for each face.
[0018] For example, the first vertex may be generated based on information provided for each of three mutually orthogonal faces of the first node, the information indicating whether or not a vertex exists other than on the edge on the corresponding face, thereby making it possible to control whether or not to generate a first vertex on each of the three faces of the node.
[0019] For example, the decoding method may further include receiving a bitstream containing the information. This allows the decoding device to determine whether to generate a first vertex using the information contained in the bitstream. Therefore, the decoding device does not need to make the determination based on other conditions, etc., thereby reducing the amount of processing required by the decoding device.
[0020] For example, the information may include position information of the first vertex. This allows the decoding device to generate the first vertex at an arbitrary position based on the information. This improves the degree of freedom in determining the position at which the first vertex is generated.
[0021] For example, the first vertex may represent the plurality of 3D points in the first node and a plurality of other 3D points in another node. In this way, the first vertex can reproduce not only the 3D points in the first node but also the 3D points of the other node. Therefore, the reproducibility of the shape of the original point cloud in the decoded point cloud can be improved.
[0022] For example, the first vertex may be located at a predetermined distance or more from a line connecting the two second vertices on the first surface. This allows a shape that cannot be reproduced using the second vertices to be reproduced using the first vertex. Therefore, the reproducibility of the shape of the original point cloud in the decoded point cloud can be improved.
[0023] For example, the decoding method may further include receiving a bitstream including, for each face of the first node that satisfies a predetermined condition, information indicating whether or not to generate the first vertex on that face, thereby reducing the amount of data stored in the bitstream.
[0024] For example, the first surface may be a surface common to the first node and a second node adjacent to the first node, the decoding method may further include generating a plurality of fourth vertices on a plurality of edges of the second node and generating a fifth vertex in the second node based on the plurality of fourth vertices, the predetermined condition may include at least one of a first condition that the surface includes two or three of the second vertices and a second condition that a first vector, a second vector, and a third vector face in the same direction, the first vector being a vector from a first center of gravity of the plurality of second vertices to the third vertex, the second vector being a vector from a second center of gravity of the plurality of fourth vertices to the fifth vertex, the third vector being a vector from a first line connecting two of the second vertices of the first surface to a tentative first vertex, and the tentative first vertex being located at a position where a second line connecting the third vertex and the fifth vertex intersects with the first surface. This makes it possible to omit transmission of information about the first vertex in cases where generating a first vertex is unlikely to improve reproducibility.
[0025] For example, in generating the triangle, the decoding device may order the first vertex and the second vertices by calculating the arctangent of the first vertex and the second vertices while facing a circular distribution including the first vertex and the second vertices, select two vertices from the first vertex and the second vertices based on the ordering result, and generate the triangle including the two selected vertices and the third vertex. This allows the decoding device to appropriately generate a triangle when the first vertex is used.
[0026] A three-dimensional data encoding method according to one aspect of the present disclosure encodes a plurality of three-dimensional points, the method comprising: generating a first vertex on a first surface of a first node at a position other than an edge of the first node; generating a plurality of second vertices on a plurality of edges of the first node; generating a third vertex within the first node based on the plurality of second vertices; storing information regarding the first vertex, the plurality of second vertices, and the third vertex in a bitstream; generating the first vertex, the plurality of second vertices, and the third vertex in the first node so as to generate a triangle defined by the first vertex, a second vertex of the plurality of second vertices, and the third vertex within the first node; and approximating the plurality of three-dimensional points to the triangle. According to this method, for example, a decoding device can use information included in the bitstream to generate a first vertex on a surface of a node at a position other than an edge and generate a triangle using the first vertex. This allows for the generation of a triangle that does not pass through edges of the node, thereby improving the reproducibility of the shape of the original point cloud in the decoded point cloud.
[0027] For example, the first surface may be a surface common to the first node and a second node adjacent to the first node, and the first vertex may be generated based on the plurality of three-dimensional points in the first node and the plurality of three-dimensional points in the second node. In this way, since the first vertex is generated based on the three-dimensional points in the first node and the second node, it is possible to improve the reproducibility of the shape of the original point cloud in the decoded point cloud.
[0028] For example, the first vertex may be generated based on a plane or a curved surface within the first node and a plane or a curved surface within the second node. In this way, the first vertex is generated based on the distribution of three-dimensional points within the first node and the second node, thereby improving the reproducibility of the shape of the original point cloud in the decoded point cloud.
[0029] For example, the encoding method may further generate a plurality of fourth vertices on a plurality of edges of the second node, generate a fifth vertex in the second node based on the plurality of fourth vertices, and generate the first vertex by shifting the plane or the curved surface in the first node toward the third vertex and shifting the plane or the curved surface in the second node toward the fifth vertex, and generate the first vertex based on the shifted plane or the shifted curved surface in the first node and the shifted plane or the shifted curved surface in the second node. In this way, the first vertex is generated based on the distribution of three-dimensional points in the first node and the second node, thereby improving the reproducibility of the shape of the original point cloud in the decoded point cloud.
[0030] For example, the shifted plane or the shifted curved surface in the first node may pass through the third vertex, and the shifted plane or the shifted curved surface in the second node may pass through the fifth vertex. In this way, the first vertex is generated based on the distribution of three-dimensional points in the first node and the second node, thereby improving the reproducibility of the shape of the original point cloud in the decoded point cloud.
[0031] Furthermore, a decoding device according to one aspect of the present disclosure is a decoding device that decodes a plurality of three-dimensional points, and includes a processor and a memory, wherein the processor uses the memory to generate a first vertex on a first surface of a first node at a position other than an edge of the first node, generate a plurality of second vertices on a plurality of edges of the first node, generate a third vertex within the first node based on the plurality of second vertices, generate a triangle within the first node defined by the first vertex, a second vertex of the plurality of second vertices, and the third vertex, and generate the plurality of three-dimensional points on the surface of the triangle.
[0032] Furthermore, an encoding device according to one aspect of the present disclosure is an encoding device for encoding a plurality of three-dimensional points, comprising a processor and a memory, wherein the processor uses the memory to generate a first vertex on a first face of a first node at a position other than an edge of the first node, generate a plurality of second vertices on a plurality of edges of the first node, generate a third vertex within the first node based on the plurality of second vertices, store information regarding the first vertex, the plurality of second vertices, and the third vertex in a bitstream, wherein the first vertex, the plurality of second vertices, and the third vertex are generated so as to generate a triangle within the first node defined by the first vertex, a second vertex of the plurality of second vertices, and the third vertex, and the plurality of three-dimensional points are approximated by the triangle.
[0033] These comprehensive or specific aspects may be realized as a system, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or may be realized as any combination of a system, a method, an integrated circuit, a computer program, and a recording medium.
[0034] Hereinafter, the embodiments will be described in detail with reference to the drawings. Note that each of the embodiments described below represents a specific example of the present disclosure. The numerical values, shapes, materials, components, component placement and connection configurations, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components not recited in independent claims will be described as optional components.
[0035] (Embodiment) Hereinafter, an encoding device (three-dimensional data encoding device) and a decoding device (three-dimensional data decoding device) according to this embodiment will be described. The encoding device generates a bitstream by encoding three-dimensional data. The decoding device generates three-dimensional data by decoding the bitstream.
[0036] The three-dimensional data is, for example, three-dimensional point cloud data (also called point cloud data). A point cloud is a collection of multiple three-dimensional points and represents the three-dimensional shape of an object. The point cloud data includes position information and attribute information of multiple three-dimensional points. The position information indicates the three-dimensional position of each three-dimensional point. Note that the position information may also be called geometry information. For example, the position information is expressed in a Cartesian coordinate system or a polar coordinate system.
[0037] The attribute information indicates, for example, color information, reflectance, transmittance, infrared information, normal vector, time information, etc. One 3D point may have a single piece of attribute information or may have multiple types of attribute information.
[0038] Although the following mainly describes the encoding and decoding of position information, the encoding device may also encode and decode attribute information.
[0039] [TriSoup Method] The encoding device according to this embodiment encodes position information using the TriSoup (Triangle-Soup) method.
[0040] The Trithorpe method is a lossy compression method that encodes the positional information of point cloud data. The Trithorpe method converts the original point cloud to be processed into a set of triangles, and then approximates the point cloud on that plane. Specifically, the original point cloud is converted into vertex information within a node, and the vertices are connected to generate a set of triangles. The vertex information used to generate the triangles is then stored in a bitstream and sent to a decoding device.
[0041] First, the encoding process using the Trithorpe method will be described. Fig. 1 shows an example of an original point cloud. As shown in Fig. 1, a point cloud 102 of an object is contained in an object space 101 and includes a plurality of points 103.
[0042] First, the encoding device divides the source point cloud into an octree to a predetermined depth. In the octree division, the target space is divided into eight nodes (subspaces), and 8-bit information (occupancy code) indicating whether or not a point cloud is included in each node is generated. Furthermore, the node including the point cloud is further divided into eight nodes, and 8-bit information indicating whether or not a point cloud is included in each of the eight nodes is generated. This process is repeated up to a predetermined hierarchy.
[0043] In normal octree coding, division is repeated until the number of points included in a node is reduced to one or a threshold value. On the other hand, in the Trithorpe method, octree division is performed up to an intermediate layer, but not to layers below that layer. Such an octree up to an intermediate layer is called a pruned octree.
[0044] 2 is a diagram showing an example of a pruned octree. As shown in FIG. 2, a point cloud 102 is divided into a plurality of leaf nodes 104 (nodes at the bottom level) of the pruned octree.
[0045] Next, the encoding device performs the following process on each leaf node 104 of the pruned octree. Note that hereinafter, a leaf node may also be simply referred to as a node. The encoding device generates a vertex on the edge as a representative point of a group of points close to the edge of the node. This vertex is called an edge vertex. For example, an edge vertex is generated for each of multiple edges (e.g., four parallel sides).
[0046] 3 is an example of a two-dimensional display of the leaf node 104, for example, a diagram showing the xy plane as viewed from the z direction shown in Fig. 1. As shown in Fig. 3, an edge vertex 112 is generated on the edge based on the nearest points of the edge among the multiple points 111 in the leaf node 104.
[0047] In Fig. 3, the dotted line around the periphery of the leaf node 104 is the edge. In this example, the edge vertex 112 is generated at the weighted average position of the points (points included in the range 113 in Fig. 3) whose distance from the edge is within 1. The unit of distance is, for example, the resolution of the point cloud, but is not limited to this. In this example, this distance (threshold) is 1, but it may be a value other than 1 or may be variable.
[0048] Next, the encoding device generates a vertex inside the node based on a group of points existing in the normal direction of the plane containing the edge vertices. This vertex is called a centroid vertex.
[0049] 4 and 5 are diagrams illustrating a method for generating a centroid vertex. First, the encoding device selects, for example, four points as representative points from a group of edge vertices. In the example shown in FIG. 4, edge vertices v1 to v4 are selected. Next, the encoding device calculates an approximate plane 121 that passes through the four points. Next, the encoding device calculates a normal n to the approximate plane 121 and an average coordinate M of the four points. Next, the encoding device generates a centroid vertex C at the weighted average coordinate of one or more points (for example, points included in the range 122 shown in FIG. 5) that are close to a half-line extending from the average coordinate M in the direction of the normal n.
[0050] Next, the encoding device entropy-encodes the vertex information, which is information on the edge vertices and centroid vertices, and stores the encoded vertex information in a Geometry Data Unit (GDU) included in the bitstream. Note that the GDU includes information indicating the pruned octree in addition to the vertex information.
[0051] 6 is a diagram showing an example of vertex information. Through the above processing, the point group 102 is converted into vertex information 123 as shown in FIG.
[0052] Next, the decoding process of the bitstream generated above will be described. First, the decoding device decodes the GDU from the bitstream to obtain vertex information. Next, the decoding device connects the vertices to generate a TriSoup-Surface, which is a group of triangles.
[0053] 7 is a diagram showing an example of a Trithorpe surface. In the example shown in FIG. 7, four edge vertices v1 to v4 and a centroid vertex C are generated based on the vertex information. In addition, a triangle 131 (Trithorpe surface) is generated whose vertices are the centroid vertex C and two edge vertices. For example, pairs of two edge vertices on two adjacent edges are selected, and a triangle 131 is generated whose vertices are the selected pairs and the centroid vertex.
[0054] 8 is a diagram for explaining the point cloud restoration process. By performing the above process for each leaf node, a three-dimensional model is generated in which the target object is represented by a plurality of triangles 131, as shown in FIG.
[0055] Next, the decoding device generates points 132 at regular intervals on the surface of the triangle 131 to restore the position information of the point group 133 .
[0056] [Representation of ridge lines of point cloud surfaces] When the curved portions of a point cloud distribution (point cloud surface) are distributed inside a leaf node in a manner that embraces them, the curved portions of the point cloud do not intersect with the edges (sides), and no vertices are created to represent the curved portions, so the surface model created by connecting the vertices may not be able to reproduce the shape of the original point cloud.
[0057] 9 is a diagram showing an example of a point cloud in which a ridge line is formed by distributing the point cloud across nodes 1 and 2. As shown in FIG. 9, an edge vertex 112 is generated based on the point cloud distribution near the edge.
[0058] 10 is a diagram showing an example of generating a centroid vertex in this case. As shown in Fig. 10, a centroid vertex 151 is generated in the normal direction of the approximation plane of the edge vertex group.
[0059] Fig. 11 shows an example of the generation of a triangle 131 (Trithorpe surface) in this case. As shown in Fig. 11, the triangle 131 is generated by connecting multiple vertices (multiple edge vertices and centroid vertices). In this case, as shown in area 152 in Fig. 11, the point cloud near the boundary of the node cannot be reproduced.
[0060] This is because the centroid vertices can sample the original point cloud surface, but the current method cannot create a vertex between two centroid vertices of two adjacent nodes. For example, if a ridge line continues to be distributed within a node along one of the x, y, or z axes, a vertex corresponding to the ridge line will not be generated because the ridge line does not cross an edge. This causes this problem.
[0061] In this embodiment, the encoding device predicts the ridge line of the point cloud surface, and if it determines that two adjacent nodes have the same ridge line, it transmits information for connecting the two centroid vertices of the two adjacent nodes with a line segment to the decoding device. The decoding device uses this information to connect the two centroid vertices and generate a new vertex (face vertex) at the intersection of the resulting line segment and the shared face of the two nodes. By using this new vertex, the decoding device can reproduce the ridge line when generating triangle 131. Note that a shared face is a face shared by two nodes, that is, a face where two nodes are adjacent.
[0062] The above method can reproduce the point cloud surface near the node boundary, resulting in a decoded point cloud that is closer to the original point cloud. Note that in the above explanation, the point cloud surface is used only to explain the issues related to ridge lines, and the ridge lines do not need to be actually obtained.
[0063] [Method for Generating Face Vertices] A method for generating face vertices, which are new vertices for expressing a ridge line, will be described below. Fig. 12 is a diagram showing a cross section of node 1 and node 2 shown in Fig. 10, and is a diagram showing a cross section on the yz plane.
[0064] First, the encoding device generates an approximation plane using the positions of the edge vertices, and then generates a prediction plane by translating the approximation plane so that it includes the centroid vertex. Note that G1 in Figure 12 is the center of gravity of the edge vertices of node 1, and G2 is the center of gravity of the edge vertices of node 2. Also, C1 is the centroid vertex of node 1, and C2 is the centroid vertex of node 2.
[0065] Next, the encoding device calculates a weighted average of the coordinates of a group of points near the line of intersection where the prediction plane and the shared surface intersect, and generates a temporary vertex at the position of the calculated weighted average. Note that the group of points near the line of intersection refers to, for example, multiple points that are located at a distance from the line of intersection that is equal to or less than a predetermined distance. Figure 13 is a diagram showing an example of generating temporary vertices. In the example shown in Figure 13, temporary vertices A1 and A2 are generated at node 1 and node 2, respectively.
[0066] Specifically, the tentative vertices are generated on the shared surface of the leaf node. Furthermore, the same processing as for the target node is performed on six adjacent nodes in six directions of the target node, and tentative vertices are generated on the shared surface between the adjacent nodes and the target node at the adjacent nodes. For example, in the example shown in FIG. 13 , node 1 is the target node, node 2 is the adjacent node, and tentative vertex A1 is generated on the shared surface of node 1, and tentative vertex A2 is generated on the shared surface of node 2.
[0067] Next, the encoding device determines whether to use the tentative vertex as a new vertex. For example, the following two methods are used for this determination:
[0068] As a first determination, the encoding device determines whether the distance between the tentative vertices A1 and A2 on the shared surface is short, i.e., whether the distance between the tentative vertices A1 and A2 is smaller than a predetermined first threshold.
[0069] As a second determination, the encoding device determines whether a temporary vector v1, which is a vector from the centroid vertex C1 of node 1 to the temporary vertex A1, and a temporary vector v2, which is a vector from the centroid vertex C2 of node 2 to the temporary vertex A2, are substantially parallel. In other words, the encoding device determines whether the angle formed between the temporary vector v1 and the temporary vector v2 is smaller than a predetermined second threshold.
[0070] If both the first and second determinations are true, that is, if the distance between temporary vertices A1 and A2 is short and temporary vectors v1 and v2 are substantially parallel, the encoding device determines that there is a ridge line, and adds a new vertex, a face vertex, to the intersection of the line segment connecting centroid vertex C1 of node 1 and centroid vertex C2 of node 2 with the shared surface.
[0071] The encoding device may perform only one of the first determination and the second determination, or may determine that a ridge line exists if at least one of the first determination and the second determination is true.
[0072] 14 is a diagram illustrating an example of generating face vertices. For example, if both the first determination and the second determination are true, the encoding device generates face vertex B at the intersection of the line segment connecting centroid vertex C1 and centroid vertex C2 and the shared surface.
[0073] [Syntax] The encoding device performs the above-mentioned determination of face vertex generation between the target node and each of its six adjacent nodes. Furthermore, this determination is limited to node faces (candidate faces) that intersect with the prediction plane among the six faces of the target node. Note that the decoding device can also narrow down the candidate faces of the node.
[0074] 15 is a diagram showing an example of the syntax of GDU (geometry_data_unit_data) included in the bitstream. The GDU includes octree information and geometry_trisoup_data. The octree information indicates the configuration of a pruned octree and the positions of multiple leaf nodes included in the pruned octree.
[0075] The geometry_trisoup_data includes the number of edges, edge vertex information, the number of edge vertices, edge vertex position information, the number of candidate faces, and face vertex information.
[0076] The number of edges indicates the number of unique edges. A unique edge is a unique edge that excludes edges with overlapping coordinates. Edge vertex information is provided for each edge, and edge vertex information [i] indicates whether an edge vertex exists on the i-th edge. For example, a value of 0 indicates that no edge vertex exists, and a value of 1 indicates that an edge vertex exists.
[0077] The number of edge vertices indicates the number of vertices on an edge, i.e., the number of edge vertices. Edge vertex position information is provided for each edge vertex, and edge vertex position information [i] indicates the position of the i-th edge vertex.
[0078] The number of candidate faces indicates the number of candidate faces. Face vertex information is provided for each candidate face, and face vertex information [i] indicates whether a face vertex exists on the i-th candidate face (whether a face vertex is to be generated). For example, a value of 0 indicates that no face vertex exists, and a value of 1 indicates that a face vertex exists.
[0079] The decoding device restores information on multiple candidate faces based on the node positions, edge vertex positions, and centroid vertex positions. The decoding device sorts the multiple candidate faces in coordinate order and extracts unique candidate faces by eliminating candidate faces with duplicate coordinates. This uniquely determines the arrangement order of the multiple candidate faces. The decoding device generates face vertices using the face vertex information.
[0080] Alternatively, information indicating the total number of unique faces may be stored in the bitstream instead of the number of candidate faces. Alternatively, the encoding device may not store information indicating the total number of unique faces in the bitstream, and the decoding device may calculate the total number of unique faces from the number of leaf nodes and the arrangement of multiple nodes. Alternatively, the decoding device may calculate a prediction plane and use the prediction plane to calculate the number of candidate faces.
[0081] [Processing Flow] The processing flow in the encoding device and the decoding device will be described below. Fig. 16 is a flowchart of the encoding process by the encoding device.
[0082] First, the encoding device generates a pruned octree and generates a plurality of leaf nodes (a group of leaf nodes) (S101). Next, the encoding device performs the following steps S102 to S104 (loop process 1) for each of the plurality of leaf nodes in the pruned octree.
[0083] First, the encoding device determines edge vertices, approximation planes, and centroid vertices based on the point cloud distribution within the node (S102). Next, the encoding device determines a prediction plane based on the approximation plane and centroid vertices (S103). Next, the encoding device determines tentative vertices based on the point cloud distribution near the intersection line between the prediction plane and the node surface (S104).
[0084] This completes loop process 1 for the target node. Next, the encoding device performs the following steps S105 to S111 (loop process 2) for each of the leaf nodes of the pruned octree.
[0085] First, the encoding device determines whether or not a tentative vertex exists on a shared surface between the target node to be processed and an adjacent node adjacent to the target node (S105).
[0086] If a tentative vertex exists on the shared surface (Yes in S105), it is determined whether the distance between the tentative vertex of the target node on the shared surface and the tentative vertex of the adjacent node is less than or equal to a first threshold, and whether the angular difference between the tentative vector of the target node and the tentative vector of the adjacent node is less than or equal to a second threshold (S106).
[0087] If the distance is less than or equal to the first threshold and the angle difference is less than or equal to the second threshold (Yes in S106), the encoding device sets the face vertex information of this candidate surface to a value of 1 (true) (S107).
[0088] On the other hand, if the distance is greater than the first threshold value, or if the angle difference is greater than the second threshold value (No in S106), the encoding device sets the face vertex information of this candidate surface to a value of 0 (false) (S108).
[0089] Next, the encoding device adds the face vertex information of this candidate face to the face vertex information list (S109).
[0090] The processes in steps S105 to S109 are performed for each of the six adjacent nodes adjacent to the target node in six directions.
[0091] Next, the encoding device entropy encodes the face vertex information list and stores the encoded face vertex information list in the GDU (S110).
[0092] After step S110, or if no tentative vertex exists on the shared surface (No in S105), the encoding device entropy encodes the vertex information, which is the position information of all vertices (edge vertices and centroid vertices) of the target node, and stores the encoded vertex information in the GDU (S111). This completes loop process 2 for the target node.
[0093] 17 is a flowchart of the decoding process performed by the decoding device. First, the decoding device obtains a GDU header and a GDU from the bitstream (S121). Next, the decoding device entropy-decodes octet information from the GDU and generates multiple leaf nodes (a group of leaf nodes) of a pruned octet tree using the octet information (S122).
[0094] Next, the decoding device performs the following steps S123 to S124 (loop process 1) for each of the leaf nodes of the pruned octree. First, the decoding device obtains vertex information indicating the positions of edge vertices and centroid vertices from the GDU (S123). Specifically, the decoding device obtains the vertex information by entropy decoding the encoded vertex information included in the GDU.
[0095] Next, the decoding device calculates a prediction plane from the approximation plane and the centroid vertices, and determines a candidate plane (S124). This completes the loop process 1 for the target node.
[0096] Next, the decoding device acquires a plurality of pieces of face vertex information (face vertex information list) from the GDU and generates a candidate surface list that is a list of all candidate surfaces (S125). Specifically, the decoding device acquires a plurality of pieces of face vertex information by decoding the encoded face vertex information (face vertex information list) included in the GDU. The decoding device generates the candidate surface list using the plurality of pieces of face vertex information.
[0097] Next, the decoding device performs the following steps S126 to S128 (loop process 2) for each of the leaf nodes of the pruned octree. First, the decoding device determines whether the target node has a candidate face and whether the face vertex information of the candidate face is set to a value of 1 (true) (S126).
[0098] If the target node has a candidate face and the face vertex information of the candidate face is value 1 (Yes in S126), the decoding device generates a face vertex at the intersection of the shared face and a line segment connecting the centroid vertex of the target node and the centroid vertex of an adjacent node that shares the candidate face with the target node (S127). On the other hand, if the target node does not have a candidate face or if the face vertex information of the candidate face is value 0 (false) (No in S126), the decoding device does not generate a face vertex. Note that if the target node has multiple candidate faces, the processes of steps S126 and S127 are performed for each candidate face.
[0099] Next, the decoding device generates a group of triangles using all vertices (edge vertices, centroid vertices, and face vertices) in the target node, and generates points on the surface of each triangle (S128). This completes loop process 2 for the target node.
[0100] [Variation 1 of Face Vertex Generation Method] Methods other than those described above may be used as the face vertex generation method. The following method may be used as Variation 1 of the face vertex generation method. Figures 18 and 19 are diagrams for explaining Variation 1 of the face vertex generation method.
[0101] The encoding device calculates a centroid vector, which is a vector connecting the centroid vertex from the approximation plane in the normal direction of the approximation plane, projects the centroid vector onto four parallel faces of the node, and if there is a group of neighboring points on the projection vector, generates a face vertex at the position of the weighted average of the neighboring points.
[0102] For example, as shown in FIG. 18 , the encoding device calculates a centroid vector CV1 connecting the center of gravity G1 of the edge vertex on the approximation plane to the centroid vertex C1 for node 1. Next, the encoding device generates projection vectors PV11, PV12, PV13, and PV14 by projecting the centroid vector CV1 onto the negative y-direction surface, the negative z-direction surface, the positive y-direction surface, and the positive z-direction surface of node 1, respectively. The encoding device determines whether a point group (neighboring point group) exists near the positions indicated by each of the projection vectors PV11, PV12, PV13, and PV14. Note that a neighboring point group refers to, for example, multiple points that are located a predetermined distance or less from the target position. If a neighboring point group exists, the encoding device generates a face vertex at the weighted average position of the neighboring point group. In the example of FIG. 18 , a neighboring point group exists for the projection vector PV11, and as shown in FIG. 19 , a face vertex B1 is generated at the weighted average position of the neighboring point group.
[0103] Similarly, the encoding device calculates a centroid vector CV2 for node 2, connecting the center of gravity G2 of the edge vertex on the approximation plane to the centroid vertex C2. Next, the encoding device generates projection vectors PV21, PV22, PV23, and PV24 by projecting the centroid vector CV2 onto the negative y-direction surface, the negative z-direction surface, the positive y-direction surface, and the positive z-direction surface of node 2, respectively. The encoding device determines whether point groups (neighboring point groups) exist near the positions indicated by each of the projection vectors PV21, PV22, PV23, and PV24, and if neighboring point groups exist, generates a face vertex at the position of the weighted average of the positions of the neighboring point groups. In the example of FIG. 18, a neighboring point group exists for the projection vector PV23, and a face vertex B2 is generated at the position of the weighted average of the positions of the neighboring point groups, as shown in FIG. 19.
[0104] The method of this modification 1 can reduce the amount of processing by eliminating the need to calculate predicted and candidate planes, compare tentative vertices, and compare tentative vectors. Furthermore, since information about adjacent nodes is not used to generate face vertices of the target node, simple processing can be achieved using only information about the target node.
[0105] [Modification 2 of the Face Vertex Generation Method] Next, a description will be given of Modification 2 of the face vertex generation method. Fig. 20 is a diagram for explaining Modification 2 of the face vertex generation method.
[0106] The encoding device generates a face vertex at the position of the weighted average of the neighboring points of the intersection between the prediction plane and the face of the node. Specifically, the encoding device calculates the approximation plane and the prediction plane using the method described with reference to FIG. 12. Next, as shown in FIG. 20, the encoding device determines whether or not a point group (neighboring point group) exists near the intersection between the prediction plane and multiple faces of node 1 (four faces on the positive x-direction side, negative x-direction side, positive y-direction side, and negative y-direction side in the example shown in FIG. 20). If a neighboring point group exists, the encoding device generates a face vertex at the position of the weighted average of the neighboring point group positions.
[0107] The method of this modification 2 can reduce the amount of processing by eliminating the need for comparison of temporary vertices and temporary vectors. Furthermore, since information about adjacent nodes is not used to generate face vertices of the target node, simple processing can be achieved using only information about the target node.
[0108] [Modification 3 of the Face Vertex Generation Method] Next, a description will be given of Modification 3 of the face vertex generation method. Figures 21 and 22 are diagrams for explaining Modification 3 of the face vertex generation method.
[0109] The encoding device calculates an approximate point cloud surface from a neighboring point cloud of the prediction plane, calculates an intersection curve between the approximate point cloud surface and the prediction plane, and generates face vertices at the curve vertices of the intersection curve.
[0110] 21 and 22, an approximate point cloud surface and an intersection curve are obtained for each of node 1 and node 2. Furthermore, a face vertex B1 is generated for node 1, and a face vertex B2 is generated for node 2. Note that although only the processing for the shared surface between node 1 and node 2 is described in FIGS. 21 and 22, similar processing is performed for other surfaces.
[0111] In addition, a face vertex may be generated when the distance from the edge of the vertex of the intersection curve is greater than or equal to a predetermined distance, and a face vertex may not be generated when the distance from the edge of the vertex is less than the predetermined distance.
[0112] The method of this modification 3 can reduce the amount of processing by eliminating the need for preliminary vector comparison. Furthermore, since information on adjacent nodes is not used to generate face vertices of the target node, simple processing can be achieved using only information on the target node.
[0113] [Fourth Modification of the Face Vertex Generation Method] Next, a fourth modification of the face vertex generation method will be described. Fig. 23 is a diagram for explaining the fourth modification of the face vertex generation method.
[0114] If there are a certain number or more points in the vicinity of a line segment connecting two centroid vertices of adjacent nodes, the encoding device generates a face vertex at the intersection of the line segment and the shared surface.
[0115] For example, in the example shown in FIG. 23 , the encoding device generates a line segment L0 by temporarily connecting the centroid vertex C1 of node 1 and the centroid vertex C2 of node 2. The encoding device determines whether the number of points existing near the line segment L0 is equal to or greater than a threshold (first determination). That is, the encoding device determines whether the number of points included in a point cloud (neighborhood point cloud) existing in an area within a predetermined distance from the line segment L0 is equal to or greater than a threshold. The encoding device also determines whether the distribution of the neighborhood point cloud along the line segment L0 is uniform (second determination). For example, the encoding device may divide the area near the line segment L0 into multiple areas and determine that the distribution of the neighborhood point cloud along the line segment L0 is uniform if the difference in the number of points included in each area is equal to or less than a threshold. Alternatively, the encoding device may determine that the distribution of the neighborhood point cloud along the line segment L0 is uniform if the number of points included in each of all areas is equal to or greater than a threshold.
[0116] Furthermore, when both the first determination and the second determination are "true," the encoding device determines that the centroid vertices C1 and C2 are connected, and generates a face vertex at the intersection of the line segment and the shared surface. Note that the encoding device may generate a face vertex when at least one of the first determination and the second determination is "true." Alternatively, the encoding device may perform only one of the first determination and the second determination, and generate a face vertex when that determination is "true."
[0117] [Other Modifications] The following methods may be used as a method for generating face vertices. For example, one or more nodes other than the target node may be selected to determine a set of nodes for creating vertices on the face of a node. Alternatively, one or more adjacent nodes that share points on the face, edge, or node corner of the node with the target node may be specified.
[0118] The following method may be used to determine the face of the node that should form a vertex in the set of nodes defined in this way. One or more faces of the node that exists between the two nodes may be selected. Here, "the face of the node that exists between the two nodes" refers to one or more of the node faces that exist on the line that connects the coordinates of the corners of each of the two nodes. Figure 24 is a diagram showing an example of the face of the node that exists between two nodes. In the example shown in Figure 24, the two nodes are node 1 and node 4, and "the face of the node that exists between two distant nodes" refers to face 1, face 2, and face 3. Furthermore, when two nodes are adjacent, "the face of the node that exists between the two nodes" refers to the shared face of the two nodes.
[0119] The following method may also be used to generate face vertices for the face of the node determined in this manner. For a pair of nodes 1 and 2, the encoding device calculates a line segment connecting a feature point of node 1 with a feature point of node 2, and generates a face vertex at the intersection of the line segment and the face of the node. The feature point here is any of the vertices (edge vertices or centroid vertices) of the node. Alternatively, the feature point may be any one of multiple points of the node. Alternatively, the feature point may be a point indicating a feature value of a point cloud within the node. This feature value may be, for example, a point obtained by interpolating the positions of the point cloud within the node, or may be the weighted average position of the point cloud as shown in FIG. 25. FIG. 25 is a diagram illustrating an example of the weighted average position of the point cloud.
[0120] Alternatively, the following method may be used to set line segments intersecting with the faces of the nodes. The encoding device may define a pair of node 1 and node 2, with a plurality of feature points owned by node 1 and a plurality of feature points owned by node 2, and generate a face vertex at the position of a weighted average of the intersections between the face of the node and one or more line segments connecting the feature points of the pair. FIG. 26 is a diagram showing an example of the position of the weighted average of the intersections. In this example, a face vertex is generated at the position of a weighted average of three intersections between the shared face of node 1 and node 2 and line segments L1, L2, and L3. Note that the weights used for the weighted average may be calculated, for example, based on the distances between the feature points and the face of the node.
[0121] The following method may also be used as a method for transmitting information related to the above processing: The above information for generating face vertices may be stored in the bitstream when a first flag indicating whether or not to generate face vertices is on (e.g., value 1), and may not be stored in the bitstream when the first flag is off.
[0122] The first flag is stored in, for example, an SPS, a GPS, or a GDU. An SPS (Sequence Parameter Set) is metadata (parameter set) common to multiple frames. A GPS (Geometry Parameter Set) is metadata (parameter set) related to encoding of position information. For example, a GPS is metadata common to multiple frames.
[0123] Alternatively, the first flag may not be provided and the above information may be stored in the bitstream at all times.
[0124] In addition to the first flag, one or more flags may be provided to switch between various methods for generating face vertices. For example, the flags may be used to switch between the multiple methods described above.
[0125] Furthermore, the following information may be used as the information for generating face vertices. The information may indicate, for each face sandwiched between two nodes in the group of nodes to be decoded, whether or not to generate face vertices for that face. Here, the face sandwiched between two nodes refers to a shared face between two adjacent nodes with a face in between.
[0126] Alternatively, the information may include identification information (ID) indicating the two nodes and information indicating whether to generate face vertices for all or part of the faces of the nodes on the line segment connecting the two nodes, where the two nodes are, for example, node 1 and node 4 shown in FIG.
[0127] Alternatively, rather than indicating whether or not to generate face vertices for each of all faces, the information may be limited to a number of faces of the node that intersect with the prediction plane shown in Figures 12 and 13, and may indicate whether or not to generate face vertices for each of the multiple faces.
[0128] Alternatively, the information may indicate, for each of all six faces of each node, whether or not to generate face vertices for that face.
[0129] Alternatively, the information may indicate, for each of a plurality of unique faces with overlapping positions omitted, whether or not face vertices are to be generated for the face.
[0130] Alternatively, the information may indicate whether or not to generate face vertices for all or some of the faces that have no adjacent nodes among the multiple faces that each node has.
[0131] The encoding device may also generate attribute information (e.g., color or reflectance) in addition to position information of the generated face vertices or points generated on the surface of a triangle generated using the face vertices. The encoding device may also encode the generated attribute information and store the encoded attribute information in a bitstream.
[0132] Furthermore, in the above-described first to third variations of the face vertex generation method, the face vertices of the target node can be generated without using information about other nodes. In the processes shown in Figures 12 to 14, the temporary vertex A1 of node 1 is used as the face vertex as is, so that the face vertices can be generated without using information about other nodes. The encoding device may perform this process of generating face vertices without using information about other nodes on nodes located at the edges. A node located at the edge is a node that has at least one face with no adjacent nodes.
[0133] The encoding device may also switch the method of generating face vertices depending on the position of the node. Specifically, the encoding device may use a method of generating face vertices using information about other nodes, such as the processes shown in Figures 12 to 14 or Variation 4, for nodes that are not located at the edge, and may use a method that does not use information about other nodes for nodes that are located at the edge.
[0134] Also, in the process shown in Figure 12, the approximation plane is shifted so that the prediction plane includes the centroid vertex, but it is not necessary for the prediction plane to include the centroid vertex, and the approximation plane may be shifted toward the centroid vertex.
[0135] Although the above description has been given using the Trithorpe method, the method of this embodiment is also effective for other methods that express the original point cloud using edge vertices of an approximate surface, other than the Trithorpe method. For example, the approximate surface may be a polygonal surface other than a triangle. The approximate surface may be a flat surface or a curved surface.
[0136] In the above explanation, an example was shown in which a new vertex (face vertex) was generated on the face of a node, but the new vertex does not have to be generated strictly on the node face. Generating a new vertex inside or outside a node may further improve the reproducibility of the original point cloud.
[0137] Furthermore, although the above explanation has solved the problem related to ridge lines, the problem is not limited to ridge lines. Similarly, the solution is not limited to ridge lines. The method of this embodiment is effective against conventional techniques in which the vertices that define the approximate surface are limited to edges.
[0138] [Another Example of Representing Ridge Lines of Point Cloud Surfaces] As mentioned above, in the Trithorpe method, there are cases where the shape of a ridge line (edge line) that spans adjacent nodes cannot be restored. In response to this, the encoding device generates face vertices on the faces that abut adjacent nodes, and restores point clouds on the triangular faces generated from the centroid vertices, face vertices, and edge vertices.
[0139] For example, if there are bends in the point cloud distribution (point cloud surface) inside a leaf node, the corners and edges of the point cloud surface do not intersect, and no vertices are created at the corner positions, so the surface model created by connecting the vertices may not be able to reproduce the shape of the original point cloud.
[0140] 27 is a diagram showing an example of a point cloud in the case where a ridge line is formed by a point cloud distributed across node 1 and node 2. As shown in FIG. 27, an edge vertex 112 is generated based on the point cloud distribution near the edge.
[0141] 28 is a diagram showing an example of generating a centroid vertex in this case. As shown in Fig. 28, a centroid vertex 151 is generated in the normal direction of the approximation plane of the edge vertex group.
[0142] Figure 29 shows an example of the generation of a triangle 131 (Trithorpe surface) in this case. As shown in Figure 29, the triangle 131 is generated by connecting multiple vertices (multiple edge vertices and centroid vertices). In this case, as shown in Figure 29, the point cloud near the node boundary cannot be reproduced.
[0143] This is because the centroid vertices can sample the original point cloud surface, but the current method cannot create a vertex between two centroid vertices of two adjacent nodes. For example, if a ridge line continues to be distributed within a node along one of the x, y, or z axes, a vertex corresponding to the ridge line will not be generated because the ridge line does not cross an edge. This causes this problem.
[0144] In this embodiment, the encoding device predicts the ridge line of the point cloud surface, and if it determines that two adjacent nodes have the same ridge line, it transmits information for connecting the two centroid vertices of the two adjacent nodes with a line segment to the decoding device. This information is, for example, one bit of information assigned to each face between the nodes.
[0145] The decoder uses this information to connect the centroid vertices and generate new vertices (face vertices) at the intersections of the resulting line segments and the shared faces between the nodes. The decoder can use these new vertices to reconstruct the ridge lines when generating triangles 131.
[0146] Furthermore, since the coordinate positions of the face vertices are not quantized, there is no problem of positional deviation due to quantization.
[0147] 30 is a diagram showing an example of generating face vertices. As shown in Fig. 30, the decoding device generates face vertices 161 and generates triangles 131 using the face vertices 161, thereby reproducing the ridge line.
[0148] The above method can reproduce the point cloud surface near the node boundary, resulting in a decoded point cloud that is closer to the original point cloud. Note that in the above explanation, the point cloud surface is used only to explain the issues related to ridge lines, and the ridge lines do not need to be actually obtained.
[0149] [Overview of Generation, Transmission, and Restoration of Face Vertices] First, we will explain the evaluation and restoration of connectivity between centroid vertices. For each node, the encoding device generates a line segment connecting the centroid vertex of the target node to the centroid vertex of the adjacent node, and determines the connectivity between the centroid vertices from the weight of the point cloud near the intersection of the line segment and the shared face between the nodes.
[0150] The encoding device sets a Boolean value (for example, the face vertex information described above) indicating whether or not to connect two centroid vertices for the corresponding face to generate a face vertex.
[0151] The Boolean value of each face is transmitted from the encoding device to the decoding device, and for faces with a Boolean value = true, the decoding device generates a face vertex at the position where the face intersects with the line segment connecting the centroid vertices of the nodes on both sides of the face.
[0152] 31 is a diagram showing an example of faces, among the six faces of a node, for which connectivity evaluation and restoration of centroid vertices are performed. For example, as shown in FIG. 31 , the encoding device and decoding device perform connectivity evaluation and restoration of centroid vertices on both sides of the faces for the three faces with larger coordinates on each axis among the six faces of the node.
[0153] Next, we will explain the outline of reducing the data amount of the bitstream and limiting the face information. To suppress the amount of data to be transmitted, the encoding device applies conditions to the face vertex information (a set of Boolean values) using information already known to the decoding device, thereby reducing the data amount of the face vertex information to be transmitted. Details of this process will be described later.
[0154] Next, we will explain the ordering of vertices within a node. To generate a Trithorpe surface, it is necessary to appropriately select two edge vertices, or an edge vertex and a face vertex. For example, if an edge vertex close to a face vertex is not selected, and an edge vertex far from the face vertex is selected, the surface will not approximate the original point group. Furthermore, an approximate face will not be generated. Therefore, in order to create a surface without gaps or holes within the node, for example, it is necessary to order the edge vertices and face vertices using a rotation order based on the centroid vertex. This will be described in detail later.
[0155] [Processing Flow] Fig. 32 is a flowchart of the encoding process by the encoding device. First, the encoding device generates a pruned octree by dividing a point cloud into octrees, and generates multiple leaf nodes (leaf node group) of the pruned octree. The encoding device then arithmetically encodes (entropy encodes) octree information indicating the pruned octree, and stores the encoded octree information in a bitstream (S201).
[0156] Next, for each of the multiple nodes (leaf nodes), the encoding device generates edge vertices and centroid vertices from the distribution of point clouds within the node, arithmetically encodes (entropy encodes) the vertex information indicating each node's position information, and stores the encoded vertex information in a bitstream (S202).
[0157] Next, the encoding device generates a face vertex at the position where the line segment connecting the centroid vertex of the target node and the centroid vertex of the adjacent node intersects with the face, only for the faces among the multiple faces of each node that meet the geometric conditions (S203).
[0158] Next, the encoding device encodes the face vertex information of the faces that meet the geometry conditions and stores the encoded face vertex information in the bitstream (S204). Here, the face vertex information is information that indicates whether or not to generate a face vertex by connecting the centroids of the nodes on both sides of the face.
[0159] Next, the encoding device performs the following processing (loop processing) of steps S205 to S208 for each of the leaf nodes of the pruned octree. First, the encoding device orders the edge vertices and face vertices in the node counterclockwise (S205). Next, the encoding device connects the vertices in the node (the edge vertices, centroid vertices, and face vertices) to generate a triangle (a trithorpe surface) (S206).
[0160] Next, the encoding device generates multiple points on the surface of the triangle (S207). Next, the encoding device makes the restored point group within the node unique using coordinate values and adds them to the decoded point group (S208). Here, making them unique means excluding points with duplicate coordinate values. This completes the loop processing for the target node.
[0161] 33 is a flowchart of the decoding process performed by the decoding device. First, the decoding device arithmetically decodes (entropy decodes) octet information from the bitstream, generates a pruned octet using the octet information, and generates multiple leaf nodes (leaf node group) (S211).
[0162] Next, the decoding device arithmetically decodes vertex information indicating the positions of edge vertices and centroid vertices from the bitstream (S212).
[0163] Next, the decoding device arithmetically decodes face vertex information only for faces that meet the geometry conditions among the multiple faces of each leaf node (S213). Next, the decoding device generates face vertices based on the face vertex information (S214).
[0164] Next, the decoding device performs the following processing (loop processing) of steps S215 to S218 for each of the leaf nodes of the pruned octree. First, the decoding device orders the edge vertices and face vertices in the node counterclockwise (S215). Next, the decoding device generates a triangle (a trithorpe surface) by connecting the vertices in the node (the edge vertices, the centroid vertex, and the face vertices) (S216).
[0165] Next, the decoding device generates multiple points on the surface of the triangle (S217). Next, the decoding device makes the restored point group within the node unique using coordinate values and adds them to the decoded point group (S218). Here, making them unique means excluding points with duplicate coordinate values. This completes the loop processing for the target node.
[0166] FIG. 34 is a flowchart showing the face vertex information transmission process (details of steps S203 and S204 shown in FIG. 32).
[0167] The encoding device performs the following processing (loop processing) of steps S221 to S226 for each of the multiple faces of each node. First, the encoding device determines whether the target face to be processed satisfies a first condition for generating face vertices (S221). Note that the first condition is a restrictive condition based on geometry information for reducing the data volume of the bitstream described below, and will be described in detail later. Furthermore, by setting this condition, it is possible to exclude faces for which face vertices cannot be generated based on the positional relationships between the node, edge vertices, and centroid vertices, thereby reducing the amount of information transmitted.
[0168] If the first condition is satisfied (Yes in S221), the encoding device determines whether the target surface satisfies a second condition for generating face vertices (S222). The second condition refers to an evaluation of the weight of points near the positions of face vertex candidates in the connectivity evaluation of centroid vertices, which will be described later. This will be described in detail later. By setting this condition, face vertices can be generated based on the distribution of the ridge line shape of the points on the surface.
[0169] If the second condition is met (Yes in S222), the encoding device sets the face vertex information of the target face to "true (vertices present)" and stores the face vertex information for transmission (S223). On the other hand, if the second condition is not met (No in S222), the encoding device sets the face vertex information of the target face to "false (vertices absent)" and stores the face vertex information for transmission (S224).
[0170] If the first condition is not met (No in S221), the encoding device does not generate face vertex information for the target surface, treats it as "false", and does not store the face vertex information for transmission (S225).
[0171] Next, the encoding device generates face vertices on the target surface based on the face vertex information (true / false) (S226). That is, if the face vertex information is "true," the encoding device generates face vertices on the target surface, and if the face vertex information is "false" (or is considered to be "false"), the encoding device does not generate face vertices on the target surface. This completes the loop processing for the target surface.
[0172] Next, the encoding device encodes the accumulated pieces of face vertex information and stores the encoded face vertex information in a bitstream (S227).
[0173] FIG. 35 is a flowchart showing the process of decoding face vertex information (details of steps S213 and S214 shown in FIG. 33).
[0174] The decoding device performs the following processing (loop processing) of steps S231 to S234 for each of the multiple faces of each node. First, the decoding device determines whether the target face satisfies a first condition for generating face vertices (S231). Note that the first condition is the same as the first condition in step S221 shown in FIG. 34.
[0175] If the first condition is satisfied (Yes in S231), the decoding device decodes face vertex information indicating whether or not to generate face vertices on the target surface from the bitstream (S232), thereby determining whether or not to generate face vertices on the target surface ("true" or "false").
[0176] Furthermore, if the first condition is not satisfied (No in S231), the decoding device does not decode the face vertex information of the target surface from the bitstream, and sets the face vertex information of the target surface to "false" (S233).
[0177] Next, the decoding device generates face vertices on the target surface based on the face vertex information (true / false) (S234). That is, if the face vertex information is "true", the decoding device generates face vertices on the target surface, and if the face vertex information is "false", the decoding device does not generate face vertices on the target surface. This completes the loop processing for the target surface.
[0178] In this way, by setting the first condition (a condition that is predetermined by the encoding device and the decoding device and cannot be changed) before the second condition of whether or not to generate a face vertex, the transmission of flag information can be suppressed.
[0179] Furthermore, by combining a predetermined first condition (a condition that cannot be changed) with a second condition (a condition that can be changed) that can be flexibly set in the encoding device and notified by a flag, it is possible to achieve both a reduction in the amount of data and flexibility in the settings.
[0180] [Evaluating and Restoring Connectivity of Centroid Vertices] The encoding device evaluates the weight of points on the line segments connecting the centroid vertices, and if the weight of points near a face vertex candidate is equal to or greater than a threshold, the candidate is set as a face vertex. Here, a candidate is the intersection of the line segment connecting the centroid vertices and the face. The weight of points near a face vertex candidate is the number or density of points included in an area within a predetermined distance from the face vertex candidate.
[0181] 36 is a diagram showing an example of face vertex candidates. In this example, the intersection of a line segment L1 connecting the centroid vertex C1 of node 1 and the centroid vertex C2 of node 2 with the shared face F1 is determined as the face vertex candidate.
[0182] The encoding device adds one bit of information (centroid vertex information) to the bit stream as transmission information to the decoding device, which indicates whether or not a vertex is to be generated on the surface by connecting the centroid vertices of the nodes on both sides of the surface. The encoding device also generates this one bit of information for each of the entire surfaces and stores the generated multiple pieces of one bit information in the bit stream.
[0183] The decoder obtains face vertex information in addition to node position information, edge vertex and centroid vertex position information, and generates face vertices for each face based on the corresponding face vertex information.
[0184] Here, the encoding device sets the intersection of the centroid vertex's line segment and the surface as the position of the face vertex. However, the face vertex may be generated at a position shifted from the intersection, for example, based on the distribution of the point cloud. In this case, in addition to the one-bit information, the encoding device stores an offset amount on the surface between the intersection and the position of the face vertex in the bitstream. This offset amount is expressed, for example, as a two-dimensional value. In other words, information indicating the position of the face vertex may be stored in the bitstream. Note that the information indicating the position of the face vertex is not limited to the offset amount, and may be coordinate information, or a coordinate difference or vector from another vertex (edge vertex or centroid vertex).
[0185] This method allows the positions of the face vertices to better reflect the shape of the ridge lines of the point cloud while maintaining the connectivity of the point cloud surface between nodes, resulting in a high-quality restored point cloud.
[0186] [Reducing the amount of data in the bitstream] The encoding device sets conditions for all faces using information known to the decoding device, thereby reducing the amount of face vertex information to be transmitted. Specifically, since the decoding device knows the position information of edge vertices and centroid vertices, it uses this information to exclude pairs of centroid vertices that cannot be connected based on their geometric relationships.
[0187] For example, the encoding device limits the generation of face vertex information by ANDing the following five conditions (a) to (e): Note that the encoding device may use some of these conditions, or may further combine other conditions.
[0188] (a) The target node has a centroid vertex (C0). (b) There is a node next to the target node (there is an adjacent node). Note that it is also possible for there to be an adjacent node in either the x, y, or z axis direction. (c) The adjacent node has a centroid vertex (C1). This condition is set because a centroid vertex may not be generated if there are few edge vertices. (d) The number of edge vertices on the shared surface shared between nodes is 2 or 3. This condition assumes that a point cloud exists in the form of a ridge line between the nodes.
[0189] (e) The surface resulting from the generation of face vertices is larger than the original surface. Here, the surface is a surface composed of multiple triangles (triangle surface). Specifically, if the following three vectors are not inverted (the inner product of vector Cvec0 and vector NF and the inner product of vector Cvec1 and vector NF are both positive): (1) vector Cvec0 from the center of gravity G0 of the edge vertices of the target node to centroid vertex C0; (2) vector Cvec1 from the center of gravity G1 of the edge vertices of the adjacent node to centroid vertex C1; and (3) vector NF when the foot of the perpendicular line drawn from face vertex candidate F to the line segment formed by the two edge vertices on the shared surface is N), the face vertex information of the target surface is set as a transmission target.
[0190] Fig. 37 is a diagram showing an example of the arrangement of vertices from which face vertices are generated. Fig. 38 is a diagram showing a vector NF. In the example shown in Fig. 37, there are two edge vertices on a shared surface, and the surface expands as face vertices are generated, and face vertex information is transmitted.
[0191] 38 is a unit vector of vector E2E3. x=E2F·u, NF=E2F−xu. If Cvev0·NF>0 and Cvev1·NF>0, the surface expands.
[0192] 39 shows an example of the arrangement of vertices where two edge vertices exist on a shared surface and the surface does not bulge. In this case, face vertex information is not transmitted.
[0193] 40 shows an example of the arrangement of vertices where two edge vertices exist on a shared surface and the surface does not bulge. In this case, face vertex information is not transmitted.
[0194] 41 shows an example of the arrangement of a group of vertices where three edge vertices exist on a shared surface and the surface bulges. In this case, face vertex information is transmitted. In the example shown in FIG. 41, the pair of edge vertices (E2, E3) that is closest to the midpoint of the line segment formed by the two adjacent edge vertices E2, E3, and E4 and is the face vertex candidate is selected, and point N is determined.
[0195] It should be noted that the above determination does not necessarily have to be made. For example, in the case shown in Fig. 40, it may be determined that the point cloud surface bulges.
[0196] As another condition, the encoding device may determine that face vertices are to be generated in the target node if a certain number or more of face vertices have been generated consecutively in a preceding restored node connected to an adjacent node during the reconstruction process.In addition, the encoding device may determine that the point cloud surface is expanding by actually calculating the volume of the point cloud surface, rather than by using the sign of the dot product of the above-mentioned vectors, and determining that the surface is expanding if the volume increases.
[0197] The above determination corresponds to the first condition in step S221 shown in Fig. 34 and step S231 shown in Fig. 35. Also, although an example in which the encoding device makes the determination has been described here, a similar determination is also made in the decoding device.
[0198] [Ordering vertices within a node] Multiple triangles (trisorp surfaces) are generated within a node for point cloud reconstruction. In this process, the vertices must be selected in order from the edge to avoid missing any triangles. Specifically, the decoding device orders the edge vertices and face vertices in a rotational order, centered on the centroid vertex. The decoding device selects two points at a time based on the set order, and generates a triangle using the selected two points and the centroid vertex. This allows triangles to be generated without gaps within the node.
[0199] However, existing methods use the premise that the ordering targets are only edge vertices on the node frame, project the vertices from the major axis (either x, y, or z axis), and achieve ordering through simple sorting. In this embodiment, face vertices are generated on the face of the node, so the ordering targets are no longer limited to the node frame (edge), and simple sorting no longer works.
[0200] 42 to 46 are diagrams for explaining this problem. Fig. 42 is a diagram showing examples of edge vertices and centroid vertices. Fig. 43 is a diagram showing an example of a correctly generated triangle. Fig. 44 is a diagram showing an example of an erroneously generated triangle.
[0201] For example, in the example shown in Figure 42, edge vertices are ordered. For example, the decoding device projects the edge vertices in the x-axis direction and performs the ordering. Next, the decoding device generates a triangle using a centroid vertex and a pair of two adjacent edge vertices, as shown in Figure 43. On the other hand, as shown in Figure 44, if the edge vertices are ordered incorrectly, a hole will appear in the node.
[0202] Fig. 45 shows examples of edge vertices, centroid vertices, and face vertices, while Fig. 46 shows an example of triangle generation when simple sorting is applied in this case.
[0203] For example, when simple sorting is applied to a group of vertices including face vertices, the ordering shown in Figure 45 is performed. Note that the numbers in parentheses in Figure 45 indicate the order set by the ordering. In this case, a hole occurs as shown in Figure 46. In this example, the order of face vertex F2 should be set to (5), but it has been set to (7), causing a hole.
[0204] Instead of simple sorting, the arctangent of each vertex is calculated after the viewpoint is aligned with the circular distribution of edge vertices and face vertices. In addition, to orient the viewpoint with the circular distribution, a rotation matrix is multiplied on the vertices.
[0205] 47 to 50 are diagrams for explaining this processing. In Fig. 47, (A) is the normal unit vector of the edge vertices within the node. (B) is the unit vector in the z-axis direction. (C) is the rotation axis calculated by the cross product of (A) and (B).
[0206] By aligning the origin of the vertex coordinates with the centroid vertex and then multiplying the edge vertices and face vertices by a matrix that rotates (A) to (B), the circular arrangement of the vertices faces the z-axis.
[0207] The amount of rotation (cosθ, sinθ) is found from the dot product of (A) and (B). The axis of rotation (C) is found from the cross product of (A) and (B).
[0208] FIG. 48 is a diagram showing an example of each vertex facing the z-axis after the above processing. The decoding device calculates the arctangent from the x and y coordinates of each vertex facing the z-axis, and sorts and orders the vertices. This results in the ordering shown in FIG. 49. The decoding device selects two points from a group of vertices including edge vertices and face vertices in order, centered on the centroid vertex, and generates a triangle using the selected two points and the centroid vertex. This generates multiple triangles as shown in FIG. 50.
[0209] In the above example, the circular distribution is made to face the viewpoint in the z-axis direction, but the viewpoint may be set in the x-axis direction, the y-axis direction, or any other direction.
[0210] [Syntax] In relation to reducing the amount of data in the bitstream, the information transmitted from the encoding device to the decoding device for restoring face vertices is one bit of information (face vertex information) indicating whether or not to generate a face vertex on each face that is restricted based on geometry.
[0211] 51 is a diagram showing an example of the syntax of GDU (geometry_data_unit_data) included in a bitstream. The GDU includes octree information and geometry_trisoup_data. The geometry_trisoup_data includes the number of edges, edge vertex information, the number of edge vertices, edge vertex position information, the number of transmission faces, and face vertex information. Note that the octree information, the number of edges, edge vertex information, the number of edge vertices, and the edge vertex position information are the same as the information described in FIG. 15.
[0212] The number of transmitted faces indicates the total number of face information (face vertex information) to be transmitted. Face vertex information is provided for each face, and face vertex information [i] is 1-bit information indicating whether a face vertex is generated (whether a face vertex exists) on the i-th face. For example, a value of 0 indicates that a face vertex is not generated, and a value of 1 indicates that a face vertex is generated.
[0213] In addition, the number of transmission faces and face vertex information are included in the bit stream when the face vertex function is enabled, and are not included in the bit stream when the face vertex function is not enabled. The face vertex function is a process for generating the face vertices described above.
[0214] For example, a flag indicating whether the face vertex function is enabled or disabled is provided, and whether the face vertex function is enabled or disabled is determined based on this flag. This flag may be stored in, for example, the GPS or GDU header.
[0215] Furthermore, whether or not this face vertex function is enabled may be set for each node. In this case, multiple flags corresponding to multiple nodes may be stored in the GDU header.
[0216] Fig. 52 is a diagram showing a modified example of the syntax of the GDU. The syntax shown in Fig. 52 differs from the syntax shown in Fig. 51 in that the GDU includes face vertex group information instead of the number of transmission faces and face vertex information.
[0217] The face vertex group information indicates, for each of a plurality of faces, whether or not a face vertex is to be generated on that face. In other words, the face vertex group information is information that compiles the plurality of pieces of face vertex information shown in Fig. 51. Furthermore, the face vertex group information is included in the bitstream when the face vertex function is enabled, and is not included in the bitstream when the face vertex function is not enabled.
[0218] In the syntax shown in Figure 52, the GDU does not include the number of transmission planes. Here, the node information, edge vertex position information, and centroid vertex position information for applying geometry-based restrictions to each plane are known to the decoding device. Therefore, the decoding device can calculate the number of plane information actually transmitted (the number of transmission planes) by applying geometry-based restrictions to all planes determined from the node adjacency relationships.
[0219] Fig. 53 is a diagram showing a modified example of the syntax of the GDU. The syntax shown in Fig. 53 is an example of syntax when, as described above, a face vertex is generated at a position shifted from the intersection of the face and the line segment connecting two centroid vertices. In the syntax shown in Fig. 53, the GDU further includes an offset amount (x) and an offset amount (y) compared to the syntax shown in Fig. 51.
[0220] An offset amount (x) and an offset amount (y) are set for each face vertex. An offset amount (x) [i] indicates the offset amount in the x-axis direction between the intersection point and the i-th face vertex. An offset amount (y) [i] indicates the offset amount in the y-axis direction between the intersection point and the i-th face vertex. In other words, an offset amount (x) and an offset amount (y) indicate the two-dimensional offset amount from the intersection point to the face vertex.
[0221] For example, the encoding device may quantize the two-dimensional offset amount and store it in the bitstream. In this case, the bitstream includes a quantization parameter used for quantization. The decoding device uses the quantization parameter to dequantize the quantized offset amount included in the bitstream and restore the original offset amount.
[0222] 35, the decoding device generates face vertices when the received face vertex information is true, but face vertices may be generated without relying on the face vertex information. For example, the encoding device may not store the face vertex information in the bitstream, and the decoding device may generate face vertices when a first condition is satisfied and may not generate face vertices when the first condition is not satisfied.
[0223] Also, in the flowchart shown in Figure 34, the encoding device determines the value (true or false) of the face vertex information using the first condition and the second condition, but it may also determine the value of the face vertex information using only one of the first condition and the second condition.
[0224] In the ordering of vertices within a node, the rotation axis passing through the centroid vertex is calculated from the cross product of the normal vector of the plane consisting of the edge vertices and an arbitrary coordinate axis, but the method of calculating the rotation axis for ordering vertices (multiple edge vertices and face vertices) is not limited to this. For example, the vertices may be projected in the direction of an arbitrary axis passing through the centroid vertex, and the axis direction may be determined so that the minimum value of the distance between the projection point and the axis is greater than a predetermined value. Alternatively, the axis direction may be determined so that the sum of the squares of the multiple distances is greater than a predetermined value.
[0225] Furthermore, although a Boolean value (1-bit face vertex information) is transmitted to reduce the data amount of the bitstream, other types of information may be transmitted. For example, the face vertex information may indicate three or more values. For example, a value of 0 may mean "do not generate face vertices," a value of 1 may mean "generate face vertices," and a value of 2 may mean "generate face vertices according to the capabilities of the decoding device." Alternatively, when a Boolean value is used and the Boolean value is true, the decoding device may determine whether to generate face vertices according to the capabilities of the decoding device.
[0226] [Summary] As described above, the decoding device (three-dimensional data decoding device) according to the embodiment performs the processing shown in Fig. 54. The decoding device decodes a plurality of three-dimensional points. The decoding device generates a first vertex (e.g., a face vertex) on a first face of a first node at a position other than an edge of the first node (S301), generates a plurality of second vertices (e.g., edge vertices) on a plurality of edges of the first node (S302), generates a third vertex (e.g., a centroid vertex) within the first node based on the plurality of second vertices (S303), generates a triangle (e.g., a Trithorpe surface) within the first node defined by the first vertex, a second vertex of the plurality of second vertices, and the third vertex (S304), and generates a plurality of three-dimensional points on the surface of the triangle (S305).
[0227] According to this method, the decoding device generates a first vertex at a position other than an edge on the face of the node and generates a triangle using the first vertex. This allows for the generation of a triangle that does not pass through an edge of the node, thereby improving the reproducibility of the shape of the original point cloud in the decoded point cloud. Specifically, this method improves the reproducibility of the shape of the original point cloud compared to the Trisorp method, which only provides edge vertices other than the centroid vertex.
[0228] For example, the first surface is a surface common to the first node and a second node adjacent to the first node, and the first vertex represents a plurality of 3D points located around the first surface in the first node and the second node. In this way, by generating a triangle using the first vertex, it is possible to improve the reproducibility of the plurality of 3D points located around the first surface in the decoded point cloud.
[0229] For example, the first vertex is located on the first surface of the first node based on the position of the third vertex of the first node and the position of the third vertex of the second node adjacent to the first node, which can improve the reproducibility of the shape of the original point cloud distributed between the two second vertices.
[0230] For example, the first surface is a surface common to the first node and a second node adjacent to the first node, and the decoding device further generates a plurality of fourth vertices (e.g., edge vertices) on a plurality of edges of the second node, and generates a fifth vertex (e.g., a centroid vertex) within the second node based on the plurality of fourth vertices, and the first vertex is on a line passing through the third vertex and the fifth vertex. This can improve the reproducibility of the shape of a ridge line in the decoded point cloud, for example, when a ridge line is formed across a node boundary.
[0231] For example, the first surface is a surface common to the first node and a second node adjacent to the first node, and the decoding device further generates a plurality of fourth vertices (e.g., edge vertices) on a plurality of edges of the second node, and generates a fifth vertex (e.g., a centroid vertex) within the second node based on the plurality of fourth vertices, and the first vertex is generated based on information indicating that the third vertex is connected to the fifth vertex. This makes it possible to improve the reproducibility of the shape of a ridge line in the decoded point cloud, for example, when a ridge line is formed across a node boundary.
[0232] For example, the information is provided for each of three mutually orthogonal faces of the first node, which makes it possible to control whether or not to generate a first vertex for each face.
[0233] For example, the first vertex is generated based on information provided for each of the three mutually orthogonal faces of the first node, which information indicates whether or not a vertex exists other than on the edge on the corresponding face. This makes it possible to control whether or not to generate a first vertex on each of the three faces of the node.
[0234] For example, the first vertex is determined based on information indicating that the first vertex exists on at least one face of the first node, which allows control over whether or not to generate the first vertex on each face of the node.
[0235] For example, the decoding method further includes receiving a bitstream containing the information. This allows the decoding device to determine whether to generate a first vertex using the information contained in the bitstream. Therefore, the decoding device does not need to make the determination based on other conditions, etc., thereby reducing the amount of processing required by the decoding device.
[0236] For example, the information includes position information of the first vertex. This allows the decoding device to generate the first vertex at any position based on the information. This improves the degree of freedom in determining the position at which the first vertex is generated. For example, the information includes vector information indicating a vector from the third vertex to the first vertex. For example, the information indicates a difference (offset amount) between the intersection of the line passing through the third vertex and the fifth vertex with the first surface and the position of the first vertex.
[0237] For example, a first vertex represents a plurality of 3D points in a first node and a plurality of other 3D points in another node. This allows the first vertex to reproduce not only the 3D points in the first node but also the 3D points of the other node. This improves the reproducibility of the shape of the original point cloud in the decoded point cloud.
[0238] For example, the first vertex is positioned at a predetermined distance or more from the line connecting the two second vertices on the first surface. This allows the first vertex to be used to reproduce a shape that cannot be reproduced using the second vertices. This improves the reproducibility of the shape of the original point cloud in the decoded point cloud.
[0239] For example, the decoding device further receives a bitstream including, for each face of the first node that satisfies a predetermined condition, information indicating whether to generate a first vertex on the face, which can reduce the amount of data stored in the bitstream.
[0240] For example, the first surface is a surface common to a first node and a second node adjacent to the first node, and the decoding device further generates a plurality of fourth vertices (e.g., edge vertices) on a plurality of edges of the second node, and generates a fifth vertex (e.g., face vertex) within the second node based on the plurality of fourth vertices, and the predetermined conditions include at least one of a first condition that the surface includes two or three second vertices, and a second condition that the first vector (e.g., vector Cvec0), the second vector (e.g., vector Cvec1), and the third vector (e.g., vector NF) face in the same direction, and the first vector is a vector from the first center of gravity of the plurality of second vertices to the third vertex, the second vector is a vector from the second center of gravity of the plurality of fourth vertices to the fifth vertex, and the third vector is a vector from a first line connecting two second vertices of the first surface to a tentative first vertex, and the tentative first vertex is located at a position where a second line connecting the third vertex and the fifth vertex intersects with the first surface.
[0241] This makes it possible to omit transmission of information about the first vertex in cases where there is little possibility that generating the first vertex will improve reproducibility.
[0242] For example, in generating a triangle, the decoding device calculates the arctangent of the first vertex and the second vertices while facing the viewpoint directly toward a circular distribution including the first vertex and the second vertices, thereby ordering the first vertex and the second vertices, selects two vertices from the first vertex and the second vertices based on the ordering result, and generates a triangle including the two selected vertices and a third vertex. In this way, the decoding device can appropriately generate a triangle when the first vertex is used.
[0243] 55 is a block diagram of the decoding device 10. For example, the decoding device 10 includes a processor 11 and a memory 12, and the processor 11 uses the memory 12 to perform the above-described processing.
[0244] Furthermore, an encoding device (three-dimensional data encoding device) according to an embodiment performs the processing shown in FIG. 56 . The encoding device encodes a plurality of three-dimensional points. The encoding device generates a first vertex (e.g., a face vertex) on a first face of a first node at a position other than an edge of the first node (S311), generates a plurality of second vertices (e.g., edge vertices) on a plurality of edges of the first node (S312), generates a third vertex (e.g., a centroid vertex) within the first node based on the plurality of second vertices (S313), stores information regarding the first vertex, the plurality of second vertices, and the third vertex in a bitstream (S314), and generates the first vertex, the plurality of second vertices, and the third vertex so as to generate a triangle within the first node defined by the first vertex, a second vertex of the plurality of second vertices, and the third vertex, and the plurality of three-dimensional points are approximated to the triangle.
[0245] According to this, for example, the decoding device can use information included in the bitstream to generate a first vertex at a position other than an edge on the face of a node, and generate a triangle using the first vertex. This allows for the generation of a triangle that passes through a point other than the edge of the node, thereby improving the reproducibility of the shape of the original point cloud in the decoded point cloud.
[0246] For example, the first surface is a surface common to a first node and a second node adjacent to the first node, and the first vertex is generated based on a plurality of three-dimensional points in the first node and a plurality of three-dimensional points in the second node. In this way, since the first vertex is generated based on the three-dimensional points in the first node and the second node, it is possible to improve the reproducibility of the shape of the original point cloud in the decoded point cloud.
[0247] For example, the first vertex is generated based on a plane or a curved surface in the first node and a plane or a curved surface in the second node. This allows the first vertex to be generated based on the distribution of three-dimensional points in the first node and the second node, thereby improving the reproducibility of the shape of the original point cloud in the decoded point cloud.
[0248] For example, the encoding device further generates a plurality of fourth vertices (e.g., edge vertices) on a plurality of edges of the second node, generates a fifth vertex (e.g., a centroid vertex) in the second node based on the plurality of fourth vertices, and in generating the first vertex, shifts a plane or a surface in the first node toward the third vertex and shifts a plane or a surface in the second node toward the fifth vertex, and generates the first vertex based on the shifted plane or the shifted surface in the first node and the shifted plane or the shifted surface in the second node.
[0249] This allows the first vertex to be generated based on the distribution of three-dimensional points within the first node and the second node, thereby improving the reproducibility of the shape of the original point cloud in the decoded point cloud.
[0250] For example, the shifted plane or the shifted surface in the first node passes through the third vertex, and the shifted plane or the shifted surface in the second node passes through the fifth vertex. In this way, the first vertex is generated based on the distribution of three-dimensional points in the first node and the second node, thereby improving the reproducibility of the shape of the original point cloud in the decoded point cloud.
[0251] In the above embodiment, the feature of generating vertices on surfaces other than edges was used in the Trithorpe method for generating triangles, but it may also be used in other methods for generating polygons other than triangles using edge vertices and decoding point groups on the surfaces of the polygons.
[0252] 57 is a block diagram of the encoding device 20. For example, the encoding device 20 includes a processor 21 and a memory 22, and the processor 21 uses the memory 22 to perform the above-described processing.
[0253] The above describes the encoding device (three-dimensional data encoding device) and decoding device (three-dimensional data decoding device) according to the embodiments and modifications of the present disclosure, but the present disclosure is not limited to these embodiments.
[0254] Furthermore, each processing unit included in the encoding device, decoding device, etc. according to the above-described embodiments is typically realized as an LSI, which is an integrated circuit. These may be individually implemented as single chips, or some or all of them may be integrated into a single chip.
[0255] Furthermore, the integrated circuit is not limited to an LSI, but may be realized by a dedicated circuit or a general-purpose processor. An FPGA (Field Programmable Gate Array) that can be programmed after the LSI is manufactured, or a reconfigurable processor that can reconfigure the connections and settings of circuit cells within the LSI may also be used.
[0256] In each of the above embodiments, each component may be configured with dedicated hardware, or may be realized by executing a software program suitable for that component. Each component may be realized by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.
[0257] Furthermore, the present disclosure may be realized as an encoding method (three-dimensional data encoding method) or a decoding method (three-dimensional data decoding method) executed by an encoding device (three-dimensional data encoding device) and a decoding device (three-dimensional data decoding device), etc.
[0258] The present disclosure may also be realized as a program that causes a computer, processor, or device to execute the encoding method or decoding method. The present disclosure may also be realized as a bitstream generated by the encoding method. The present disclosure may also be realized as a recording medium on which the program or the bitstream is recorded. For example, the present disclosure may also be realized as a non-transitory computer-readable recording medium on which the program or the bitstream is recorded.
[0259] The division of functional blocks in the block diagram is an example, and multiple functional blocks may be realized as a single functional block, one functional block may be divided into multiple blocks, or some functions may be moved to another functional block.Furthermore, the functions of multiple functional blocks having similar functions may be processed in parallel or in time-sharing by a single piece of hardware or software.
[0260] The order in which the steps in the flowchart are executed is merely an example for specifically explaining the present disclosure, and other orders may be used. Also, some of the steps may be executed simultaneously (in parallel) with other steps.
[0261] While the encoding device and decoding device according to one or more aspects have been described based on the embodiments, the present disclosure is not limited to these embodiments. As long as they do not deviate from the spirit of the present disclosure, various modifications conceivable by those skilled in the art to the present embodiments and configurations constructed by combining components of different embodiments may also be included within the scope of one or more aspects.
[0262] The present disclosure is applicable to encoding devices and decoding devices.
[0263] 10 Decoding device 11, 21 Processor 12, 22 Memory 20 Encoding device 101 Object space 102, 133 Point cloud 103, 111, 132 Point 104 Leaf node 112 Edge vertex 113, 122 Range 121 Approximation plane 123 Vertex information 131 Triangle 151 Centroid vertex 152 Area 161 Face vertex
Claims
1. A decoding method for decoding a plurality of three-dimensional points, comprising: generating a first vertex on a first face of a first node at a position other than an edge of the first node; generating a plurality of second vertices on a plurality of edges of the first node; generating a third vertex within the first node based on the plurality of second vertices; generating a triangle within the first node defined by the first vertex, a second vertex of the plurality of second vertices, and the third vertex; and generating the plurality of three-dimensional points on the face of the triangle.
2. The decoding method according to claim 1, wherein the first surface is a surface common to the first node and a second node adjacent to the first node, and the first vertex represents a plurality of three-dimensional points located around the first surface in the first node and the second node.
3. The decoding method according to claim 1, wherein the first vertex is placed on the first face of the first node based on the position of the third vertex of the first node and the position of the third vertex of a second node adjacent to the first node.
4. The decoding method according to claim 1, wherein the first face is a face common to the first node and a second node adjacent to the first node, and the decoding method further comprises generating a plurality of fourth vertices on a plurality of edges of the second node, and generating a fifth vertex within the second node based on the plurality of fourth vertices, and the first vertex is generated based on information indicating that the third vertex is connected to the fifth vertex.
5. The decoding method according to claim 4, wherein the information is provided for each of three mutually orthogonal faces of the first node.
6. A decoding method according to claim 1, wherein the first vertex is generated based on information provided for each of three mutually orthogonal faces of the first node, the information indicating whether or not a vertex exists other than the edge on the corresponding face.
7. The decoding method according to claim 4 or 6, further comprising the step of receiving a bitstream containing the information.
8. The decoding method according to claim 4 or 6, wherein the information includes position information of the first vertex.
9. The decoding method of claim 1, wherein the first vertex represents the plurality of 3D points in the first node and a plurality of other 3D points in another node.
10. The decoding method according to claim 1, wherein the first vertex is positioned at a predetermined distance or more from a line connecting two of the second vertices on the first surface.
11. The decoding method according to claim 1, further comprising receiving a bitstream containing information indicating whether or not to generate the first vertex on each of the faces of the first node that satisfies a predetermined condition.
12. The decoding method according to claim 11, wherein the first surface is a surface common to the first node and a second node adjacent to the first node, the decoding method further comprises: generating a plurality of fourth vertices on a plurality of edges of the second node; and generating a fifth vertex within the second node based on the plurality of fourth vertices, the predetermined conditions including at least one of: a first condition that the surface includes two or three of the second vertices; and a second condition that the first vector, the second vector, and the third vector face in the same direction, the first vector being a vector from a first center of gravity of the plurality of second vertices to the third vertex, the second vector being a vector from a second center of gravity of the plurality of fourth vertices to the fifth vertex, and the third vector being a vector from a first line connecting two of the second vertices of the first surface to a tentative first vertex, the tentative first vertex being located at a position where a second line connecting the third vertex and the fifth vertex intersects with the first surface.
13. The decoding method according to claim 1, wherein the generation of the triangle includes: ordering the first vertex and the plurality of second vertices by calculating the arctangent of the first vertex and the plurality of second vertices while facing the viewpoint directly onto a circular distribution including the first vertex and the plurality of second vertices; selecting two vertices from the first vertex and the plurality of second vertices based on the result of the ordering; and generating the triangle including the two selected vertices and the third vertex.
14. A method for encoding a plurality of three-dimensional points, comprising: generating a first vertex on a first face of a first node at a position other than an edge of the first node; generating a plurality of second vertices on a plurality of edges of the first node; generating a third vertex within the first node based on the plurality of second vertices; storing information regarding the first vertex, the plurality of second vertices, and the third vertex in a bitstream; and generating the first vertex, the plurality of second vertices, and the third vertex so as to generate a triangle within the first node defined by the first vertex, a second vertex of the plurality of second vertices, and the third vertex, and the plurality of three-dimensional points are approximated by the triangle.
15. The encoding method according to claim 14, wherein the first surface is a surface common to the first node and a second node adjacent to the first node, and the first vertex is generated based on the plurality of three-dimensional points in the first node and the plurality of three-dimensional points in the second node.
16. The encoding method according to claim 15, wherein the first vertex is generated based on a plane or a curved surface within the first node and a plane or a curved surface within the second node.
17. The encoding method according to claim 16, further comprising: generating a plurality of fourth vertices on a plurality of edges of the second node; generating a fifth vertex in the second node based on the plurality of fourth vertices; and generating the first vertex by shifting the plane or the curved surface in the first node toward the third vertex; and shifting the plane or the curved surface in the second node toward the fifth vertex; and generating the first vertex based on the shifted plane or the shifted curved surface in the first node and the shifted plane or the shifted curved surface in the second node.
18. The encoding method according to claim 17, wherein the shifted plane or the shifted surface in the first node passes through the third vertex, and the shifted plane or the shifted surface in the second node passes through the fifth vertex.
19. A decoding device for decoding a plurality of three-dimensional points, comprising: a processor; and a memory, wherein the processor uses the memory to: generate a first vertex on a first face of a first node at a position other than an edge of the first node; generate a plurality of second vertices on a plurality of edges of the first node; generate a third vertex within the first node based on the plurality of second vertices; generate a triangle within the first node defined by the first vertex, a second vertex of the plurality of second vertices, and the third vertex; and generate the plurality of three-dimensional points on the face of the triangle.
20. A coding device for encoding a plurality of three-dimensional points, comprising: a processor; and a memory, wherein the processor uses the memory to: generate a first vertex on a first face of a first node at a position other than an edge of the first node; generate a plurality of second vertices on a plurality of edges of the first node; generate a third vertex within the first node based on the plurality of second vertices; store information regarding the first vertex, the plurality of second vertices, and the third vertex in a bitstream; the first vertex, the plurality of second vertices, and the third vertex are generated so as to generate a triangle within the first node defined by the first vertex, a second vertex of the plurality of second vertices, and the third vertex, and the plurality of three-dimensional points are approximated by the triangle.
