Mesh Decoding Device, Mesh Encoding Device, Mesh Decoding Method, and Program

Abstract

To improve encoding efficiency of meshes.SOLUTION: A mesh decoding device 200 is equipped with a mesh decoding portion 201 that is configured to decode a mesh in each patch from an encoded bit stream and reconstruct an original polygon number in the mesh on the basis of a variable indicating the polygon number in each patch, and a mesh post-processing portion 202 that is configured to execute post-processing to the mesh of the adjacent patch.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to a mesh decoding device, a mesh encoding device, a mesh decoding method, and a program.

Background Art

[0002] Non-Patent Document 1 discloses a technique for extending the technique of Non-Patent Document 2 for encoding a video-based point cloud and encoding a Dynamic mesh defined in Non-Patent Document 3.

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Non-Patent Document 2

Non-Patent Document 3

Non-Patent Document 4

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the video-based mesh encoding technology disclosed in Non-Patent Document 1, since the number of polygons in the mesh is single, there is a problem that it is difficult to optimize in a wide range of bitrates from low bitrate to high bitrate. Therefore, the present invention has been made in view of the above problems, and an object thereof is to provide a mesh decoding device, a mesh encoding device, a mesh decoding method, and a program capable of improving the encoding efficiency of a mesh.

Means for Solving the Problems

[0005] A first feature of the present invention is a mesh decoding device, which includes a mesh decoding unit configured to decode a mesh in each patch from an encoded bitstream and reconstruct the original number of polygons in the mesh based on a variable indicating the number of polygons in each patch, and a mesh post-processing unit configured to perform post-processing on the mesh of the adjacent patches.

[0006] A second feature of the present invention is a mesh encoding device, which includes a mesh generation unit configured to generate a plurality of meshes having different numbers of polygons for each patch.

[0007] A third feature of the present invention is a mesh decoding method, which includes a step of decoding a mesh in each patch from an encoded bit stream and reconstructing the original number of polygons in the mesh based on a variable indicating the number of polygons in each patch, and a step of performing post-processing on the meshes of the adjacent patches.

[0008] A fourth feature of the present invention is a program for causing a computer to function as a mesh decoding device, the mesh decoding device including a mesh decoding unit configured to decode a mesh in each patch from an encoded bit stream and reconstruct the original number of polygons in the mesh based on a variable indicating the number of polygons in each patch, and a mesh post-processing unit configured to perform post-processing on the meshes of the adjacent patches.

Advantages of the Invention

[0009] According to the present invention, it is possible to provide a mesh decoding device, a mesh encoding device, a mesh decoding method, and a program that can improve the encoding efficiency of a mesh.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

[0011] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the components in the following embodiments can be replaced with existing components as appropriate, and various variations including combinations with other existing components are possible. Therefore, the description of the following embodiments does not limit the content of the invention described in the claims.

[0012] <First Embodiment> Hereinafter, with reference to FIGS. 1 to 6, the mesh encoding apparatus 100 and the mesh decoding apparatus 200 according to the present embodiment will be described. FIG. 1 is a diagram showing an example of the functional blocks of the mesh encoding apparatus 100 according to the present embodiment, and FIG. 2 is a diagram showing an example of the functional blocks of the mesh decoding apparatus 200 according to the present embodiment.

[0013] The mesh encoding apparatus 100 is configured to encode a Dynamic mesh defined in Non-Patent Document 3.

[0014] The feature of the Dynamic mesh is that the number of vertices and the connectivity of triangles are different in the mesh of each frame.

[0015] In addition, in order to cover the range from a low bit rate (e.g., about 2 Mbps) to a high bit rate (e.g., about 32 Mbps), the anchors in Non-Patent Document 3 create meshes of 5K, 10K, 15K, 20K, and 25K from the original mesh, prepare six levels of meshes in combination with the original mesh, and select the optimal mesh according to the bit rate. Generally, in the case of a low bit rate, a mesh with a small number of polygons is encoded, and conversely, in the case of a high bit rate, a mesh with a large number of polygons is encoded.

[0016] As shown in FIG. 1, the mesh encoding device 100 includes a segmentation unit 101, a mesh generation unit 102, and a mesh encoding unit 103.

[0017] The segmentation unit 101 is configured to segment a mesh and generate patches.

[0018] Specifically, as shown in FIG. 1, the segmentation unit 101 is configured to generate patches based on "Texture map PNG" and "Mesh OBJ".

[0019] Note that "Texture map PNG" includes "Attribute maps" defined in Non-Patent Document 3, and "Mesh OBJ" includes "Connectivity Information", "Geometry Information", and "Mapping Information" defined in Non-Patent Document 3.

[0020] Here, the segmentation unit 101 may be configured to use the segmentation method defined in Non-Patent Document 1. In such a case, the segmentation unit 101 divides the original mesh M all into N patches according to the following formula (1).

[0021]

Equation

[0022] For example, the mesh generation unit 102 is configured to generate meshes with polygon numbers of 1 / 2, 1 / 4, and 1 / 8 from the original mesh, and set the variable f indicating the polygon number to 0, 1, 2, and 3. Here, when the variable f is 0, it indicates the original polygon number; when it is 1, it indicates half of the polygon number of the original mesh; when it is 2, it indicates a quarter of the polygon number of the original mesh; and when it is 3, it indicates an eighth of the polygon number of the original mesh.

[0023] Also, the mesh generation unit 102 may use a correspondence table that associates the variable f with the number of polygons. For example, when the variable f is 0, it indicates the original polygon number; when it is 1, it indicates 90% of the polygon number of the original mesh; when it is 2, it indicates 80% of the polygon number of the original mesh; and when it is 3, it indicates 70% of the polygon number of the original mesh.

[0024] Specifically, the mesh generation unit 102 may be configured to generate such a plurality of meshes according to the following procedure.

[0025] The mesh generation unit 102 selects one patch M i and is configured to perform the following procedure when setting the polygon number F i orig of the original to 1, 1 / 2, 1 / 4, 1 / 8 (or 100%, 90%, 80%, 70%).

[0026] As step 1, the mesh generation unit 102 is configured to select the edges to be deleted.

[0027] As shown in FIG. 2, the mesh generation unit 102 is configured to calculate the cost of a certain edge uv (one side of a triangle) and select the edge with the lowest cost from the target edges.

[0028] Here, the cost is defined as the magnitude of the visual change when the edge is deleted. That is, the cost of the edges of polygons in flat regions or short and fine edges is low.

[0029] Specifically, the mesh generation unit 102 is configured to calculate the edge cost cost(u, v) using Equation (2).

[0030]

Equation

[0031] As step 2, as shown in FIG. 3, the mesh generation unit 102 is configured to simplify the mesh (polygon) by reducing the selected edge using an operation called edge collapse.

[0032] Here, edge collapse targets the edge between vertex V t and vertex V s in FIG. 3(a), and as shown in FIG. 3(b), combines vertex V t and vertex V s (deletes vertex V t ). In this way, when edge collapse is performed once, the number of polygons decreases by two.

[0033] Note that the operation of refining the mesh (polygon) by changing the state of FIG. 3(b) to the state of FIG. 3(a) is called vertex split.

[0034] The mesh generation unit 102 may be configured to repeatedly perform the operations shown in FIGS. 2 and 3 until the number of polygons in the original mesh becomes 1 / 2 of the original number of polygons.

[0035] Furthermore, the mesh generation unit 102 may repeatedly perform such operations until the number of polygons in the original mesh becomes 1 / 4, and further 1 / 8 of the original number of polygons, and save the mesh as patches M 2 i and M 3 i and M

[0036] Finally, the mesh generation unit 102 is configured such that for 1, 1 / 2, 1 / 4, 1 / 8 (or 100%, 90%, 80%, 70%) of the original number of polygons F, the variable fi is set to 0, 1, 2, 3, respectively. i orig Note that the number of polygons F of M

[0037] and M i 0 and M i 1 and M i 2 and M i 3 is calculated by the following formula (3). i k

Equation

[0038]

Equation

[0039] The patch encoding unit 103 has a plurality of meshes M with different numbers of polygons in each patch i 0 and M i 1 and M i 2 and Mi 3 Since [optimal meshes] are prepared, it is configured to select the optimal mesh from among them.

[0040] For example, the patch encoding unit 103 calculates the cost (Cost(M i f )) according to the following formula (4), and the mesh with the smallest cost (Cost(M i f )) is selected as the optimal mesh, and the corresponding variable f i opt is configured to be recorded.

[0041] Cost(M i f ) = Rate(M i f ) + λ · Distortion(M i f ) (4)

[0042]

Equation

[0043] The patch encoding unit 103 is configured to put the selected optimal mesh and the corresponding variable f i opt into the bit stream.

[0044] In this way, encoding the variable f i can improve the encoding efficiency rather than directly encoding the number of polygons. Here, the variable f i is 0, 1, 2, or 3.

[0045] Also, the patch encoding unit 103 can calculate from the variable f i for the number of polygons of the selected mesh according to the following formula (6).

[0046]

Number

[0047] The patch encoding unit 103 may be configured to encode the coordinates of the vertices, Connectivity, and texture information of the optimal mesh selected in each patch by the method of Non-Patent Document 1.

[0048] Specifically, the method of Non-Patent Document 1 takes a mesh as input and segments it into a plurality of patches.

[0049] And, in the method of Non-Patent Document 1, since vertices are connected by edges in each patch, it projects onto a plane in units of triangles.

[0050] Furthermore, in the method of Non-Patent Document 1, since the projection of triangles is sparser compared to a point cloud, it uses rasterization to convert a polygon into pixel data and uses it as a Geometry image defined in Non-Patent Document 2.

[0051] Here, the Geometry image is encoded by a video encoding technique as defined in Non-Patent Document 2. Also, the Connectivity as information without a point cloud is encoded using Edgebreaker of Non-Patent Document 4.

[0052] For simplicity, the patch encoding unit 103 calculates Rate and Distortion only using the Geometry image. Also, since Connectivity is Lossless, Distortion is zero.

[0053] Note that when the patch encoding unit 103 counts the Rate(M f i )(Rate with only the Geometry image), it may be configured to project each patch onto a plane, rasterize the resulting Geometry image, and encode it using video encoding technology.

[0054] In such video encoding technology, it is possible to set a Q value for controlling the amount of encoding for each patch. In such video encoding technology, using a larger Q value results in a smaller amount of encoding.

[0055] The patch encoding unit 103 can set the Q value for each such patch in the following steps.

[0056] As step 1, the patch encoding unit 103 allocates the total amount of encoding proportional to the size of each patch to the corresponding patch, and sets an initial Q value Q1 of the Q value based on the allocated amount of encoding. f i to be set.

[0057] As step 2, the patch encoding unit 103 adjusts the initial Q value Q1 of the Q value to Q2 f i according to the number of polygons of the corresponding mesh M of the corresponding patch. f i Basically, when the number of polygons is large, the patch encoding unit 103 adjusts it to a smaller Q value. f i As step 3, since there are patches that are displayed and patches that are not displayed according to the viewer's viewpoint and line of sight, the patch encoding unit 103 adjusts Q2

[0058] to Q3 f i which is the final Q value, f i based on the degree to which the corresponding patch is visible. Basically, for patches that are not displayed, the patch encoding unit 103 adjusts it to a larger Q value.

[0059] As shown in FIG. 4, the mesh decoding device 200 includes a mesh decoding unit 201 and a mesh post-processing unit 202.

[0060] The mesh decoding unit 201 decodes the mesh (specifically, the vertex coordinates, Connectivity, and texture information) in each patch in the method of Non-Patent Document 1, and reconstructs the original number of polygons in the decoded mesh based on the variable f selected in each patch. i It is configured to reconstruct the original number of polygons.

[0061] The mesh decoding unit 201 is configured to calculate the above-mentioned original number of polygons (target number of polygons) F by the following formula (7). i orig It is configured to calculate.

[0062]

Equation

[0063] As Step 1, the mesh decoding unit 201 is configured to select vertices to be divided in patch M. i In patch M, it is configured to select vertices to be divided.

[0064] Here, the mesh decoding unit 201 is configured to use the cost of the edge defined by the mesh generation unit 102 as described above.

[0065] As shown in FIG. 2, the mesh decoding unit 201 calculates the cost of an edge uv (one side of a triangle) and is configured to select the edge with the largest cost from patch M. i It is configured to select the edge with the largest cost from patch M.

[0066] Here, the mesh decoding unit 201 is configured to add vertices to the edges of polygons in non-flat regions or polygons with large areas.

[0067] The mesh decoding unit 201 is configured to select, as an operation target vertex, any one of the vertices that are not boundaries from the two vertices of the selected edge. If both vertices are boundary vertices, the mesh decoding unit 201 sets the cost of the edge to zero and repeats step 1.

[0068] As step 2, the mesh decoding unit 201 is configured to perform polygon refinement to divide the vertex selected in step 1 using an operation called vertex sprit shown in FIG. 3.

[0069] According to vertex sprit, the vertex V in FIG. 3(b) s is targeted for division and divided into vertex V t and vertex V s . New edges are created between vertex V t and vertex V s , between vertex V l and vertex V t , and between vertex V r and vertex V t . The original edges e1 and e2 connected to vertex V s are reconnected to the new vertex V t .

[0070] When dividing a vertex, the mesh decoding unit 201 is configured to set the difference vector of the coordinates of the two newly generated vertices to half the length of the edge uv shown in FIG. 2. Here, when vertex sprit is performed once, the number of polygons increases by two.

[0071] The mesh decoding unit 201 is configured to repeatedly perform the above-described step 1 and step 2 until the original number of polygons is reached.

[0072] The mesh post-processing unit 202 is configured to perform post-processing on the meshes of adjacent patches.

[0073] Specifically, the mesh post-processing unit 202 is configured to perform the following steps.

[0074] As Step 1, since the polygon at the boundary of the patch has insufficient edges, the mesh post-processing unit 202 is configured to perform the following processing for each edge at the boundary.

[0075] Let L1, which is the boundary of patch P1 shown in FIG. 5, be the edge to be processed. Here, it is assumed that patch P1 is actually in contact with patch P2 / P3.

[0076] Here, the mesh post-processing unit 202 is configured to find the vertices D1 to D3 that are closest to the vertices V1 to V3 of the polygon of the adjacent patch P2 / P3 from the boundary L1.

[0077] Next, if the distances between the vertices D1 to D3 and the closest vertices V1 to V3 are within the threshold, the mesh post-processing unit 202 is configured to add the dotted lines L11 to L13 as new edges.

[0078] Note that after this operation for all the boundary edges is completed, the mesh post-processing unit 202 can solve the problem of insufficient edges between adjacent patches.

[0079] As Step 2, due to Lossy compression, if a gap occurs at the boundary L1 between patch P1 and patch P2 / P3 as shown in FIG. 5, the mesh post-processing unit 202 may be configured to fill such a gap using the zippering algorithm in the same manner as the technique defined in Non-Patent Document 1.

[0080] In such a case, as shown in FIG. 6, the mesh post-processing unit 202 may be configured to find pairs of vertices where gaps occur (in the example of FIG. 6, the pair of vertex V1 and vertex D1, the pair of vertex V2 and vertex D2, and the pair of vertex V3 and vertex D3), and average and merge the vertices of such pairs.

[0081] The above-mentioned mesh encoding device 100 and mesh decoding device 200 may be realized by a program that causes a computer to execute each function (each process).

Industrial Applicability

[0082] According to this embodiment, for example, in video communication, since an improvement in overall service quality can be realized, it is possible to contribute to Goal 9 of the Sustainable Development Goals (SDGs) led by the United Nations, "Build resilient infrastructure, promote sustainable industrialization, and foster innovation."

Explanation of Signs

[0083] 100... Mesh encoding device 101... Segmentation unit 102... Mesh generation unit 103... Mesh encoding unit 200... Mesh decoding device 201... Mesh decoding unit 202... Mesh post-processing unit

Claims

1. A mesh decoding device comprising: a mesh decoding unit configured to decode a mesh in each patch from an encoded bit stream and reconstruct an original number of polygons in the mesh based on a variable indicating the number of polygons in each patch; a mesh post-processing unit configured to perform post-processing on the meshes of the adjacent patches; wherein the mesh decoding unit is configured to calculate the original number of polygons based on a variable indicating the number of polygons for each patch, and repeatedly perform Vertex split until the calculated original number of polygons is reached.

2. The mesh decoding device according to claim 1, wherein the mesh decoding unit is configured to calculate a cost of an edge in the patch and repeatedly perform an operation of increasing polygons from the edge having the largest cost.

3. A mesh decoding device comprising: a mesh decoding unit configured to decode a mesh in each patch from an encoded bit stream and reconstruct an original number of polygons in the mesh based on a variable indicating the number of polygons in each patch; a mesh post-processing unit configured to perform post-processing on the meshes of the adjacent patches; wherein the mesh post-processing unit is configured to create a new edge for a polygon lacking an edge at a boundary of the patch.

4. The mesh decoding device according to claim 3, wherein the mesh post-processing unit is configured to find a pair of vertices where a gap occurs using a zippering algorithm at a boundary of the patch and average and merge the vertices of the pair.

5. A mesh decoding method comprising: step A of decoding a mesh in each patch from an encoded bit stream and reconstructing an original number of polygons in the mesh based on a variable indicating the number of polygons in each patch; step B of performing post-processing on the meshes of the adjacent patches; wherein in step A, the original number of polygons is calculated based on a variable indicating the number of polygons for each patch, and Vertex split is repeatedly performed until the calculated original number of polygons is reached. **Claim 6** A mesh decoding method, comprising: Step A of decoding a mesh in each patch from the encoded bit stream and reconstructing the original number of polygons in the mesh based on a variable indicating the number of polygons in each patch; and Step B of performing post-processing on the meshes of the adjacent patches, wherein in Step B, a new edge is created for a polygon with insufficient edges at the boundary of the patch. The mesh decoding method is characterized by this. **Claim 7** A program for causing a computer to function as a mesh decoding device, wherein the mesh decoding device comprises a mesh decoding unit configured to decode a mesh in each patch from the encoded bit stream and reconstruct the original number of polygons in the mesh based on a variable indicating the number of polygons in each patch; and a mesh post-processing unit configured to perform post-processing on the meshes of the adjacent patches, wherein the mesh decoding unit is configured to calculate the original number of polygons based on a variable indicating the number of polygons for each patch and repeatedly perform Vertex split until the calculated original number of polygons is reached. The program is characterized by this. **Claim 8** A program for causing a computer to function as a mesh decoding device, wherein the mesh decoding device comprises a mesh decoding unit configured to decode a mesh in each patch from the encoded bit stream and reconstruct the original number of polygons in the mesh based on a variable indicating the number of polygons in each patch; and a mesh post-processing unit configured to perform post-processing on the meshes of the adjacent patches, wherein the mesh post-processing unit is configured to create a new edge for a polygon with insufficient edges at the boundary of the patch. The program is characterized by this.

Images