Mesh decoding device, mesh encoding device, mesh decoding method and program

Abstract

To improve coding efficiency of connectivity.SOLUTION: A mesh decoding device 200 according to an invention comprises: a point group decoding part 201 configured to decode a point group including texture information and static data of a mesh from a coded bit stream; and a mesh conversion part 202 configured to convert the decoded point group to a mesh, with the decoded statistic data.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to a mesh decoding device, a mesh encoding device, a mesh decoding method, and a program. [Background technology]

[0002] Non-Patent Document 1 discloses a technique for encoding a dynamic mesh defined in Non-Patent Document 3 by extending the technique of Non-Patent Document 2 for encoding a video-based point cloud. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Graziosi, Danillo Bracco. "Video-Based Dynamic Mesh Coding." 2021 IEEE International Conference on Image Processing (ICIP). IEEE, 2021. [Non-patent document 2] ISO / IEC FDIS 23090-5:2020, Information technology -Coded representation of immersive media - Part 5: Visual Volumetric Video-based Coding (V3C) and Video-based Point Cloud Compression (V-PCC). [Non-patent document 3] CfP for Dynamic Mesh Coding,ISO / IEC JTC1 / SC29 / WG7 N00231, MPEG136 - OnLine. [Non-patent document 4] ISO / IEC 23008-2:2020, Information technology - High efficiency coding and media delivery in heterogeneous environments - Part 2: High efficiency video coding. [Non-patent document 5] Preliminary metrics for Mesh Coding, ISO / IEC JTC1 / SC29 / WG7 N00113, MPEG133 - OnLine. [Non-patent document 6] Sun, Yiyong, et al. "Triangle mesh-based edge detection and its application to surface segmentation and adaptive surface smoothing." Proceedings. International Conference on Image Processing. Vol. 3. IEEE, 2002. [Non-Patent Document 7] [V-PCC] Generation of mesh connectivity for point clouds, ISO / IEC JTC1 / SC29 / WG11 MPEG2020 / m53533, MPEG130 - Alpbach. Summary of the Invention [Problem to be solved by the invention]

[0004] However, the video-based mesh coding technology disclosed in Non-Patent Document 1 has a problem in that the number of polygons in a mesh is single, making it difficult to optimize for a wide range of bit rates, from low to high. Therefore, the present invention has been made in consideration of the above-mentioned problems, and aims to provide a mesh decoding device, a mesh coding device, a mesh decoding method, and a program that can improve the encoding efficiency of meshes. [Means for solving the problem]

[0005] A first feature of the present invention is that it is a mesh decoding device comprising: a point cloud decoding unit configured to decode statistical data of a point cloud and meshes including texture information from an encoded bitstream; and a mesh conversion unit configured to convert the decoded point cloud into the mesh using the decoded statistical data.

[0006] A second feature of the present invention is that it is a mesh encoding device configured to encode the number of polygons in the order of the patches in which the patches of a point cloud are encoded, and to set the number of polygons to 0 in patches in which the number of polygons is not sent before encoding.

[0007] A third feature of the present invention is a mesh decoding method comprising the steps of: decoding statistical data of a point cloud and meshes including texture information from an encoded bitstream; and converting the decoded point cloud into the mesh using the decoded statistical data.

[0008] A fourth feature of the present invention is a program for causing a computer to function as a mesh decoding device, the mesh decoding device comprising: a point cloud decoding unit configured to decode a point cloud including texture information and statistical data of the mesh from an encoded bitstream; and a mesh conversion unit configured to convert the decoded point cloud into the mesh using the decoded statistical data. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide a mesh decoding device, a mesh encoding device, a mesh decoding method and a program that can improve mesh encoding efficiency. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an example of functional blocks of a mesh coding device 100 according to an embodiment. [Figure 2] Fig. 2A is a diagram showing an example of how a polygon A is divided by the point cloud conversion unit 101 of the mesh coding device 100 according to an embodiment. Fig. 2B is a diagram showing an example of how a polygon A is divided by the point cloud conversion unit 101 of the mesh coding device 100 according to an embodiment. [Figure 3] FIG. 3 is a diagram showing an example of how points of a point cloud are generated by the point cloud conversion unit 101 of the mesh coding device 100 according to an embodiment, taking texture features into consideration. [Figure 4] FIG. 4 is a diagram showing an example of how the color C(p) of a point p is calculated by the point cloud conversion unit 101 of the mesh coding device 100 according to an embodiment. [Figure 5] FIG. 5 is a diagram illustrating an example of functional blocks of a mesh decoding device 200 according to an embodiment. [Figure 6] FIG. 6 is a diagram showing an example of how polygons are generated by the mesh conversion unit 202 of the mesh decoding device 200 according to an embodiment using the Screened Poisson algorithm. [Figure 7] FIG. 7 is a diagram showing an example of how polygons are generated from directive functions by the mesh conversion unit 202 of the mesh decoding device 200 according to an embodiment. [Figure 8] FIG. 8 is a diagram showing an example of how a mesh is simplified and a texture image and UV coordinates are modified by the mesh conversion unit 202 of the mesh decoding device 200 according to an embodiment. [Figure 9] FIG. 9 is a diagram showing an example of how edges are reduced using edge collapse by the mesh conversion unit 202 of the mesh decoding device 200 according to an embodiment. [Figure 10] FIG. 10 is a diagram showing an example of how the problem of insufficient edges between adjacent patches is solved by the mesh conversion unit 202 of the mesh decoding device 200 according to an embodiment. [Figure 11]FIG. 11 is a diagram showing an example of how the mesh conversion unit 202 of the mesh decoding device 200 according to an embodiment fills gaps at the boundary L1 using a zippering algorithm. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the components in the following embodiments can be appropriately replaced with existing components, etc., and various variations, including combinations with other existing components, are possible. Therefore, the description of the following embodiments does not limit the content of the invention described in the claims.

[0012] First Embodiment The mesh encoding device 100 and mesh decoding device 200 according to this embodiment will be described below with reference to Fig. 1 to Fig. 11. Fig. 1 is a diagram showing an example of functional blocks of the mesh encoding device 100 according to this embodiment, and Fig. 2 is a diagram showing an example of functional blocks of the mesh decoding device 200 according to this embodiment.

[0013] As shown in FIG. 1, the mesh coding device 100 includes a point cloud conversion unit 101 and a point cloud coding unit 102.

[0014] The point cloud conversion unit 101 is configured to convert the mesh into a point cloud including color information, taking into consideration texture features or geometric features of the mesh.

[0015] Here, the texture features of the mesh correspond to "Texture map PNG" defined in Non-Patent Document 3, and the geometric features of the mesh correspond to "Mesh OBJ" defined in Non-Patent Document 3.

[0016] Note that "Texture map PNG" includes "Attribute maps" defined in Non-Patent Document 3, and "Mesh OBJ" includes "Connectivity Information," "Geometry Information," and "Mapping Information" defined in Non-Patent Document 3.

[0017] Specifically, in step 1, the point cloud conversion unit 101 is configured to use the vertices of polygons themselves as points of the point cloud based on the geometric characteristics of the mesh ("Geometry Information"), but deletes vertices that are common to adjacent polygons so that they do not overlap in the point cloud.

[0018] In step 2, the point cloud conversion unit 101 is configured to generate points of a point cloud inside the polygon based on the geometric characteristics ("Geometry Information") of the mesh.

[0019] For example, the point cloud conversion unit 101 is configured to divide polygon A using subdivision as shown in FIG. 2A, and then divide polygon A again using subdivision as shown in FIG. 2B.

[0020] Here, the point cloud conversion unit 101 may be configured to divide the polygon again using subdivision if the length of the shortest edge in each polygon divided using subdivision is longer than a threshold value.

[0021] The point cloud conversion unit 101 may be configured to repeat the process in step 2 until there are no polygons whose shortest edge length is longer than the threshold.

[0022] The point cloud conversion unit 101 is configured to use all vertices of the polygons divided in this way as points of the point cloud. However, as described above, vertices that are common to the edges of adjacent polygons are deleted so that they do not overlap in the point cloud.

[0023] Note that the points of the point cloud inside the polygon A thus generated (for example, point X in FIG. 2B) exist in the same plane as the polygon A.

[0024] Here, the point cloud transformation unit 101 may be configured to finely adjust the position coordinates of the points of the point cloud using Non-Patent Document 6 so that the change can be made more smoothly.

[0025] It should be noted that the threshold value is set in advance and may be set for each region rather than for the entire mesh. The magnitude of the threshold value affects the number of points in the point cloud, which in turn affects the coding efficiency.

[0026] The point cloud conversion unit 101 may be configured to generate points of a point cloud inside a polygon using grid sampling, face sampling, or the like described in Non-Patent Document 5.

[0027] Furthermore, the point cloud conversion unit 101 may be configured to adjust the above-mentioned threshold value in consideration of the above-mentioned geometric features.

[0028] For example, the point cloud conversion unit 101 may be configured to use Non-Patent Document 6 to detect geometric edges from a mesh, and to lower (e.g., halve) the above-mentioned threshold for polygons connected to such edges, thereby generating more points in the point cloud inside the polygons.

[0029] As a step 3, the point cloud transformation unit 101 is configured to generate the points of the point cloud taking into account the texture features.

[0030] For example, as shown in Figure 3, when the point cloud conversion unit 101 maps one polygon A to a texture image T, if there is an edge of the texture image T within the polygon A, it generates points B at a fixed distance along the edge.

[0031] The point cloud conversion unit 101 is configured to reverse-map the point B generated in this way onto the original polygon A, find its three-dimensional coordinates, and use them as points in the point cloud. However, points that are common to the points generated in the above steps 1 and 2 are deleted so as not to overlap in the point cloud.

[0032] Furthermore, since each point in the point cloud generated as described above does not have color information, the point cloud conversion unit 101 is configured to assign color information to each point in the point cloud.

[0033] Specifically, the point cloud conversion unit 101 is configured to convert the colors of the vertices of the polygon into the RGB values ​​of the pixels of the texture image taken at the coordinates of the UV mapping.

[0034] For example, the point cloud conversion unit 101 colors the points inside the polygon in the following procedure.

[0035] As shown in FIG. 4, the point mapped onto the texture image is point p, and the pixels around point p are a, b, c, and d.

[0036] In this case, the point cloud conversion unit 101 is configured to calculate the color C(p) of the point p using the following equations (1) to (5).

[0037]

number

[0038]

number

[0039]

number

[0040]

number

[0041]

number

[0042] The point cloud conversion unit 101 is configured to assign the color information calculated in this manner to each point of the point cloud using texture information.

[0043] The point cloud encoding unit 102 is configured to encode the point cloud converted by the point cloud conversion unit 101 and also encode the statistical data of the mesh, similar to the technology defined in Non-Patent Document 2, for example.

[0044] Here, the difference from V-PCC defined in Non-Patent Document 2 is that statistical data of meshes is coded.

[0045] The statistical data may include, for example, the number of polygons, the number of vertices, or the number of edges. When the number of polygons F does not exist, the point cloud encoding unit 102 may be configured to calculate the number of polygons F based on the number of vertices V and the number of edges E using the following formula:

[0046] F=2V F=2E / 3 Non-Patent Document 2 specifies that a point cloud is segmented into multiple patches, each of which is projected onto an appropriate plane, and the color and position coordinates of the point cloud are converted into a set of two-dimensional texture patches and depth patches.

[0047] The point cloud encoding unit 102 may be configured to integrate these patches into one large texture image and depth image, and encode the image using a normal video encoding method such as that shown in Non-Patent Document 4.

[0048] The difference from V-PCC defined in Non-Patent Document 2 is that the point cloud encoding unit 102 encodes the statistical data of the mesh for each patch or for the entire mesh.

[0049] The point cloud encoding unit 102 is configured to include the encoded statistical data as RAW patches in a bitstream.

[0050] When encoding the number of polygons for each patch, the point cloud encoding unit 102 is configured to encode the number of polygons in the order in which the patches are encoded, and to set the number of polygons to 0 for patches for which no polygon number is sent before encoding.

[0051] Here, the point cloud encoding unit 102 may be configured to use predictive coding in encoding the number of polygons. Use predictive coding.

[0052] As shown in FIG. 5, the mesh decoding device 200 includes a point cloud decoding unit 201 and a mesh conversion unit 202.

[0053] The point cloud decoding unit 201 is configured to use Non-Patent Document 2 to decode a point cloud including texture information from the bitstream received from the mesh encoding device 100, and also to decode mesh statistical data.

[0054] Specifically, the point cloud decoding unit 201 may be configured to decode the two-dimensional texture patches and depth patches of each patch from the bitstream, reconstruct a three-dimensional point cloud, perform post-processing, and output the final point cloud, similar to the technique defined in Non-Patent Document 2.

[0055] The point cloud decoding unit 201 may also be configured to decode the number of polygons of a mesh for the entire mesh or for each patch from the RAW patches in the bitstream.

[0056] Here, the point cloud decoding unit 201 may be configured to decode the entire mesh and the number of polygons for each patch coded using predictive coding in the order of the patches of the point cloud, and record the index of the point corresponding to each patch.

[0057] The mesh conversion unit 202 is configured to convert the point cloud decoded by the point cloud decoding unit 201 into a mesh using the statistical data decoded by the point cloud decoding unit 201.

[0058] Specifically, the mesh conversion unit 202 may be configured to convert the point cloud decoded by the point cloud decoding unit 201 into a mesh (including texture information) using statistical data decoded for each patch or for the entire mesh, using Non-Patent Document 7.

[0059] For example, the mesh conversion unit 202 may be configured to convert a point cloud into a mesh in the following procedure: Here, the following procedure remains the same whether the number of polygons included in the statistical data is specified for each patch or for the entire mesh.

[0060] As step 1, the mesh transformation unit 202 is configured to calculate the normals of each point in the point cloud.

[0061] Here, the mesh conversion unit 202 converts a point p i m neighboring points (point p i and calculates the center of gravity c by the following equation (6).

[0062]

number

[0063]

number

[0064] Specifically, when the mesh conversion unit 202 draws the generated polygons on the monitor, it embeds the two-dimensional images of the polygons in a texture image and combines the texture images of the polygons to generate one large texture image.

[0065] Here, this large texture image is the above-mentioned “texture image.” Furthermore, the position of the texture image of each polygon within this large texture image is the UV coordinates.

[0066] 6(a) to 6(d) respectively show a vector field, an indicator gradient, an indicator function, and a surface calculated from the above-mentioned point group and normals. Here, the surface is already a polygon.

[0067] Specifically, the mesh conversion unit 202 may be configured to generate a polygon (surface) from the directive function, as shown in FIG.

[0068] For example, as shown in Fig. 7, the mesh conversion unit 202 can divide the three-dimensional space discretized by the directive function into an internal GI and an external GO of a polygon G. Here, as shown in Fig. 7, the mesh conversion unit 202 may be configured to create a polygon G at a boundary B between the internal GI and external GO of the polygon G.

[0069] That is, the mesh conversion unit 202 may be configured to generate a mesh (M in FIG. 7) from voxels (V in FIG. 7) discretized from a surface.

[0070] In step 3, the mesh conversion unit 202 is configured to simplify the mesh to a specified number of polygons and modify the texture image and UV coordinates.

[0071] However, if the number of polygons specified here is greater than the number of polygons generated in step 2, the mesh conversion unit 202 is configured to skip step 3. The specific process of step 3 is as follows.

[0072] First, the mesh transformation unit 202 is configured to select edges to be removed from the mesh.

[0073] As shown in FIG. 8, the mesh conversion unit 202 is configured to calculate the cost of an edge uv (one side of a triangle) and select the edge with the smallest cost from among the target edges.

[0074] Here, the cost is defined as the magnitude of the visual change when an edge is deleted. In other words, the cost of polygon edges in flat areas or short, fine edges is low.

[0075] Specifically, the mesh conversion unit 202 is configured to calculate the cost of an edge using equation (8).

[0076]

number

[0077] Secondly, the mesh conversion unit 202 is configured to simplify the mesh (polygon) by eliminating selected edges using an operation called edge collapse, as shown in FIG.

[0078] Here, the edge collapse is performed on the vertex V in Figure 9(a). t and vertex V s The edge between vertex V and vertex V is removed as shown in Figure 9(b). t and vertex V s and (vertex V t In this way, performing an edge collapse once reduces the number of polygons by two.

[0079] The operation of refining the mesh (polygon) by changing the state of FIG. 9(b) to the state of FIG. 9(a) is called vetrex split.

[0080] The mesh conversion unit 202 may be configured to repeat the operations shown in FIGS. 8 and 9 until the designated number of polygons is reached.

[0081] If the above-mentioned operations are performed on the entire mesh, no post-processing is required, but if the above-mentioned operations are performed on a patch-by-patch basis, the mesh conversion unit 202 may be configured to perform post-processing on the meshes of adjacent patches.

[0082] For example, the mesh transformation unit 202 may be configured to perform the following steps:

[0083] First, since polygons on the boundaries of patches have insufficient edges, the mesh conversion unit 202 is configured to perform the following processing for each edge on the boundaries.

[0084] It is assumed that L1, which is the boundary of patch P1 shown in Fig. 10, is the edge to be processed. Here, it is assumed that patch P1 and patches P2 / P3 are actually in contact with each other.

[0085] Here, the mesh conversion unit 202 is configured to find the vertices D1 to D3 that are closest to the vertices V1 to V3 of the polygon of the adjacent patch P2 / P3 from the boundary L1.

[0086] Next, the mesh conversion unit 202 is configured to add dotted lines L11 to L13 as new edges if the distance between the vertices D1 to D3 and the closest vertices V1 to V3 is within a threshold value.

[0087] After completing this operation for all boundary edges, the mesh conversion unit 202 can solve the problem of insufficient edges between adjacent patches.

[0088] Here, if a gap occurs at the boundary L1 between patch P1 and patches P2 / P3 due to lossy compression, as shown in Figure 10, the mesh conversion unit 202 may be configured to fill such gap using a zippering algorithm, similar to the technique defined in Non-Patent Document 1.

[0089] In such a case, the mesh conversion unit 202 may be configured to find pairs of vertices where gaps occur (in the example of FIG. 11, the pair of vertices V1 and D1, the pair of vertices V2 and D2, and the pair of vertices V3 and D3), as shown in FIG. 11, and to average and merge the vertices of such pairs.

[0090] According to the present invention, it becomes possible to reuse the video coding chip and the point group coding chip, thereby improving the mesh coding efficiency.

[0091] The mesh encoding device 100 and the mesh decoding device 200 described above may be realized as a program that causes a computer to execute each function (each step). [Industrial Applicability]

[0092] According to this embodiment, for example, it is possible to improve the overall service quality in video communication, which will contribute to the achievement of Goal 9 of the United Nations-led Sustainable Development Goals (SDGs), which is to "Develop resilient infrastructure, promote sustainable industrialization and foster innovation." [Explanation of symbols]

[0093] 100...Mesh coding device 101...Point cloud conversion unit 102...Point cloud encoding unit 200...Mesh decoding device 201...Point cloud decoding unit 202...Mesh conversion unit A...Polygon T...Texture image

Claims

1. A mesh decoding device, comprising: a point cloud decoder configured to decode point cloud and mesh statistics including texture information from the encoded bitstream; a mesh conversion unit configured to convert the decoded point cloud into the mesh using the decoded statistical data, The mesh decoding device is characterized in that the mesh conversion unit is configured to calculate normals from the point cloud, generate polygons using the normals, simplify the mesh to the number of polygons of the entire mesh or a portion of the mesh, and generate a texture image and UV coordinates.

2. A mesh decoding device, a point cloud decoder configured to decode point cloud and mesh statistics including texture information from the encoded bitstream; a mesh conversion unit configured to convert the decoded point cloud into the mesh using the decoded statistical data, The mesh decoding device is characterized in that the mesh conversion unit is configured to calculate the cost of the edges of the polygons in the mesh and repeat the operation of simplifying the polygons starting from the edge with the smallest cost.

3. 3. The mesh decoding device according to claim 1, wherein the statistical data includes at least one of the number of polygons and the number of vertices of the polygons.

4. The mesh decoding device according to claim 1 or 2, wherein the point cloud decoding unit is configured to decode the statistical data from the RAW patches in the bitstream for the entire mesh or for each patch.

5. 3. The mesh decoding device according to claim 2, wherein the mesh conversion unit is configured to simplify the polygons by repeating edge collapse until a target number of polygons is reached.

6. 6. The mesh decoding device according to claim 5, wherein said mesh conversion unit is configured to create new edges for said polygons lacking edges at the boundary of a patch.

7. The mesh decoding device of claim 6, characterized in that the mesh conversion unit is configured to use a zipping algorithm to find pairs of vertices that create gaps at the boundaries of the patches, and to fill the gaps by averaging and merging the vertices of the pairs.

8. 1. A mesh decoding method, comprising: a step A of decoding point cloud and mesh statistics including texture information from the coded bitstream; and a step B of converting the decoded point cloud into the mesh using the decoded statistical data, A mesh decoding method characterized in that in step B, normals are calculated from the point cloud, polygons are generated using the normals, the mesh is simplified to the number of polygons of the entire mesh or a portion of the mesh, and a texture image and UV coordinates are generated.

9. A mesh decoding method, comprising: a step A of decoding point cloud and mesh statistics including texture information from the coded bitstream; and a step B of converting the decoded point cloud into the mesh using the decoded statistical data, In the step B, the mesh decoding method is characterized in that the cost of the edges of the polygons in the mesh is calculated, and an operation of simplifying the polygons starting from the edge with the smallest cost is repeated.

10. A program that causes a computer to function as a mesh decoding device, The mesh decoding device comprises: a point cloud decoder configured to decode point cloud and mesh statistics including texture information from the encoded bitstream; a mesh conversion unit configured to convert the decoded point cloud into the mesh using the decoded statistical data, The mesh conversion unit is configured to calculate normals from the point cloud, generate polygons using the normals, simplify the mesh to the number of polygons of the entire mesh or a portion of the mesh, and generate a texture image and UV coordinates.

11. A program that causes a computer to function as a mesh decoding device, comprising: The mesh decoding device comprises: a point cloud decoder configured to decode point cloud and mesh statistics including texture information from the encoded bitstream; a mesh conversion unit configured to convert the decoded point cloud into the mesh using the decoded statistical data, The program is characterized in that the mesh conversion unit is configured to calculate the cost of the edges of the polygons in the mesh and repeat an operation of simplifying the polygons starting from the edge with the smallest cost.

Images