Mesh decoding device, mesh decoding method, and program

Abstract

A mesh decoding device 200 according to the present invention is provided with: an atlas data decoding unit 207 that decodes an atlas bitstream to generate and output control information; a basic mesh decoding unit 202 that generates and outputs a basic mesh using the control information and a basic mesh bitstream as inputs; a subdivision unit 203 that outputs a subdivided mesh and subdivision vertex normals using the control information and the basic mesh as inputs; a mesh decoding unit 204 that generates a decoded mesh using the control information, displacement amounts, the subdivided mesh, and the subdivision vertex normals as inputs; and a boundary correction unit 208 that receives the control information and the decoded mesh and outputs a boundary-corrected decoded mesh.

Description

Mesh decoding device, mesh decoding method and program

[0001] The present invention relates to a mesh decoding device, a mesh decoding method, and a program.

[0002] In Non-Patent Document 1, meshes are decoded by dividing them into basic meshes that represent rough information and displacements that represent detailed information, and the displacements are decoded using a video codec.

[0003] Khaled Mammou, Jungsun Kim, Alexis Tourapis, Dimitri Podborski, Krasimir Kolarov, “[V-CG] Apple's Dynamic Mesh Coding CfP Response,” ISO / IEC JTC 1 / SC 29 / WG 7 m59281, April 2022. Google Draco, accessed May 26, 2022 [Online], https: / / google.github.io / draco

[0004] However, in Non-Patent Document 1, if mesh data of an interframe is coded in units of sub-meshes, holes may appear at the boundaries of the sub-meshes, which may result in a decrease in subjective performance.

[0005] In order to prevent holes from appearing in the decoded mesh, it is necessary to correct the submesh boundaries during decoding. To perform such correction, it is necessary to embed submesh boundary information in the bitstream, but transmitting this information as is results in a problem of large data volume. Therefore, the present invention has been made in consideration of the above-mentioned problems, and aims to provide a mesh decoding device, a mesh decoding method, and a program that can efficiently encode boundary information by using the boundary information of a reference frame as the boundary information of an interframe.

[0006] A first feature of the present invention is summarized as a mesh decoding device comprising an atlas data decoding unit configured to decode an atlas bitstream and generate and output control information; a base mesh decoding unit configured to receive the control information and a base mesh bitstream as input and generate and output a base mesh; a subdivision unit configured to receive the control information and the base mesh as input and output a subdivision mesh and subdivision vertex normals; a mesh decoding unit configured to receive the control information, a displacement amount, the subdivision mesh, and the subdivision vertex normals as input and generate a decoded mesh; and a boundary correction unit configured to receive the control information and the decoded mesh as input and output a boundary-corrected decoded mesh.

[0007] A second feature of the present invention is a mesh decoding method comprising the steps of: decoding an atlas bitstream to generate and output control information; generating and outputting a base mesh using the control information and a base mesh bitstream as input; outputting a subdivision mesh and subdivision vertex normals using the control information and the base mesh as input; generating a decoded mesh using the control information, displacement, the subdivision mesh, and the subdivision vertex normals as input; and inputting the control information and the decoded mesh to output a boundary-corrected decoded mesh.

[0008] A third feature of the present invention is summarized as a program that causes a computer to function as a mesh decoding device, the mesh decoding device comprising: an atlas data decoding unit configured to decode an atlas bitstream and generate and output control information; a basic mesh decoding unit configured to receive the control information and a basic mesh bitstream as input and generate and output a basic mesh; a subdivision unit configured to receive the control information and the basic mesh as input and output a subdivision mesh and subdivision vertex normals; a mesh decoding unit configured to receive the control information, a displacement amount, the subdivision mesh, and the subdivision vertex normals as input and generate a decoded mesh; and a boundary correction unit configured to receive the control information and the decoded mesh as input and output a boundary-corrected decoded mesh.

[0009] According to the present invention, it is possible to provide a mesh decoding device, a mesh decoding method and a program that can efficiently encode boundary information by using the boundary information of a reference frame as is as boundary information of an interframe.

[0010] FIG. 1 is a diagram showing an example of the configuration of a mesh processing system 1 according to an embodiment. FIG. 2 is a diagram showing an example of functional blocks of a mesh decoding device 200 according to an embodiment. FIG. 3 is a diagram showing an example of the syntax configuration of AVE in the first embodiment. FIG. 4 is a diagram showing an example of a submesh decoded by the basic mesh decoding unit 202. FIG. 5 is a diagram for explaining an example of a boundary pair index in the first embodiment. FIG. 6 is a diagram showing an example of the syntax configuration of AVE in Modification Example 2. FIG. 7 is a diagram for explaining an example of a boundary pair index in Modification Example 2. FIG. 8 is a diagram showing an example of the syntax configuration of AVE in Modification Example 3. FIG. 9 is a diagram showing an example of a basic mesh. FIG. 10 is a diagram showing an example of a subdivision mesh. FIG. 11 is a diagram schematically showing a boundary correction method.

[0011] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the components in the following embodiments can be appropriately replaced with existing components, etc., and various variations, including combinations with other existing components, are possible. Therefore, the description of the following embodiments does not limit the content of the invention described in the claims.

[0012] First Embodiment A mesh processing system 1 according to this embodiment will be described below with reference to FIGS.

[0013] 1 is a diagram showing an example of the configuration of a mesh processing system 1 according to this embodiment. As shown in FIG. 1, the mesh processing system 1 includes a mesh encoding device 100 and a mesh decoding device 200.

[0014] FIG. 2 is a diagram showing an example of functional blocks of a mesh decoding device 200 according to this embodiment.

[0015] As shown in Figure 2, the mesh decoding device 200 has a demultiplexing unit 201, a basic mesh decoding unit 202, a subdivision unit 203, a mesh decoding unit 204, a displacement amount decoding unit 205, a video decoding unit 206, an atlas data decoding unit 207, and a boundary correction unit 208.

[0016] The demultiplexing unit 201 is configured to separate the multiplexed bit stream into an atlas bit stream, a base mesh bit stream, a displacement amount bit stream, and a texture bit stream.

[0017] The basic mesh decoding unit 202 is configured to decode the basic mesh bit stream based on the control information, generate and output the control information and basic meshes.

[0018] The subdivision unit 203 is configured to generate and output subdivision vertices and their connection information from the basic mesh decoded by the basic mesh decoding unit 202 using the subdivision method indicated by the control information.

[0019] Here, the basic mesh, the added subdivision vertices, and the connection information therebetween are collectively referred to as a "subdivision mesh."

[0020] The mesh decoding unit 204 is configured to generate and output a decoded mesh using the control information, the subdivision mesh, the subdivision vertex normals, and the displacement amounts.

[0021] The displacement amount decoding unit 205 is configured to decode the displacement amount bit stream based on the control information to generate and output the displacement amount.

[0022] The video decoding unit 206 is configured to decode the texture using a video codec and output the decoded texture.

[0023] The atlas data decoding unit 207 is configured to decode the atlas bitstream and output the control information.

[0024] The boundary correction unit 208 is configured to correct the submesh boundaries of the input decoded mesh and output the boundary-corrected decoded mesh.

[0025] (Atlas Data Decoding Unit 207) Hereinafter, the control information decoded by the atlas data decoding unit 207 will be described with reference to FIGS.

[0026] First, the atlas bitstream may include an afps vmc extension (AVE), which is a collection of control information for the atlas. The AVE may be defined for each frame. The control information may also be defined in supplemental enhancement information (SEI).

[0027] (First embodiment) Hereinafter, the syntax structure of AVE and its decoding method will be described with reference to Figs.

[0028] 3 is a diagram showing an example of the syntax configuration of AVE. Here, if the syntax functions are similar, syntax names different from those shown in FIG. 3 may be used.

[0029] In the AVE syntax structure shown in Figure 3, the Description column indicates how each syntax is coded. Also, ue(v) indicates an unsigned zeroth-order exponential-Golomb code, and u(n) indicates an n-bit flag.

[0030] The Description is merely an example, and an encoding method other than the Description of the diagram, such as u(n), ue(v), or ae(v), may be selected.

[0031] AVE includes a control signal afps_submesh_correction_flag that indicates whether submesh boundary correction is performed for each frame.

[0032] The AVE includes a control signal afps_border_coding_flag that indicates whether to send border information for each frame.

[0033] AVE contains a control signal afps_border_vertex_count_minus1 that represents the number of boundary pairs per frame.

[0034] Here, a boundary pair is a combination of vertices with the same coordinates. Details of boundary pairs will be described later.

[0035] The AVE includes a control signal afps_border_vertex_pair_count_minus1[i] that indicates the number of vertices that make up the i-th border pair per frame.

[0036] The AVE includes a control signal afps_border_submesh_id[i][j] that represents the index of the submesh of the jth vertex that makes up the ith boundary pair for each frame.

[0037] The AVE includes a control signal afps_border_vertex_id[i][j] that represents the index of the jth vertex that constitutes the ith border pair for each frame.

[0038] The method for decoding boundary information will be described below with reference to FIGS.

[0039] FIG. 4 shows an example of a sub-mesh decoded by the basic mesh decoding unit 202. As shown in FIG.

[0040] 4, the displacement amounts are defined for the vertices on the sub-mesh and their subdivision vertices, but even if the vertex coordinates are the same, the displacement amounts may differ if the sub-mesh indexes are different. As a result, when the displacement amounts are added by the mesh decoding unit 204, holes may appear at the sub-mesh boundaries.

[0041] To solve this problem, the boundary correction unit 208 corrects the vertices of the decoded mesh output from the mesh decoding unit 204 based on the boundary information.

[0042] Here, the boundary information is information about vertices that have the same coordinates even though they are on different sub-meshes at the time of sub-mesh decoding.

[0043] Note that the boundary information can be obtained by decoding the atlas bitstream described above, and the boundary vertices can be decoded as a set of boundary pairs.

[0044] In the example of FIG. 5, when the boundary pair index i=2, afps_border_submesh_id[2]=[0,1,2] and afps_border_vertex_id[2]=[5,2,0].

[0045] This indicates that submesh vertex v(0,5), submesh vertex v(1,2), and submesh vertex v(2,0) are the same vertex, where v(i,j) indicates vertex j of submesh i.

[0046] These pairs of identical vertices are called boundary pairs. In the example above, the boundary pair is 2, but so are all other boundary pairs.

[0047] Any boundary information can be decoded using the same procedure as in the above example.

[0048] Second Embodiment The syntax structure of AVE and its decoding method will be described below with reference to Figures 4, 6 and 7. Since this is basically the same as the first embodiment, only the differences from the first embodiment will be described.

[0049] 6 is a diagram showing an example of the syntax configuration of AVE. Here, if the syntax functions are similar, syntax names different from those shown in FIG. 4 may be used.

[0050] AVE contains a control signal afps_border_vertex_count_minus1 that represents the number of border vertices per frame.

[0051] The AVE contains a control signal afps_border_submesh_id[i] that represents the index of the submesh of the i-th border vertex.

[0052] The AVE includes a control signal afps_border_pair_id[i] that represents the index of the border pair of the i-th border vertex.

[0053] The AVE includes a control signal afps_border_vertex_id[i] that represents the i-th border vertex index.

[0054] The method for decoding boundary information will be described below with reference to FIGS.

[0055] For example, let i be the index of all border vertices that satisfy the border pair index afps_border_pair_id[i]=2.

[0056] In this case, afps_border_submesh_id[i] and afps_border_vertex_id[i] are [0, 1, 2] and [5, 2, 0], respectively.

[0057] This indicates that the submesh vertex v(0,5), the submesh vertex v(1,2) and the submesh vertex v(2,0) are the same vertex.

[0058] In the example above, the boundary pair is 2, but so are all other boundary pairs.

[0059] Any boundary information can be decoded using the same procedure as in the above example.

[0060] Third Embodiment The syntax configuration of AVE will be described below with reference to Fig. 8. Since this is basically the same as the second embodiment, only the differences from the second embodiment will be described.

[0061] AVE includes a control signal afps_border_submesh_delta_id[i] that represents the difference between the index of the submesh of the (i-1)th border vertex and the index of the submesh of the i-th border vertex.

[0062] The submesh index of the i-th boundary vertex can be decoded by the following procedure.

[0063] afps_border_submesh_id[i] = afps_border_submesh_delta_id[i] when i=0 afps_border_submesh_id[i] = afps_border_submesh_delta_id[i] + afps_border_submesh_id[i-1] when i>0 The AVE contains a control signal afps_border_delta_pair_id[i] representing the difference between the index of the (i-1)th border vertex and the index of the border pair of the i-th border vertex.

[0064] The index of the boundary pair of the i-th boundary vertex can be decoded as follows:

[0065] When i=0, afps_border_pair_id[i] = afps_border_delta_pair_id[i] When i>0, afps_border_pair_id[i] = afps_border_delta_pair_id[i] + afps_border_pair_id[i-1] The AVE contains a control signal afps_border_vertex_delta_id[i] representing the difference between the index of the (i-1)th border vertex and the index of the i-th border vertex.

[0066] The index of the boundary vertex of the i-th boundary vertex can be decoded by the following procedure.

[0067] When i = 0, afps_border_vertex_id[i] = afps_border_vertex_delta_id[i] When i > 0, afps_border_vertex_id[i] = afps_border_vertex_delta_id[i] + afps_border_vertex_id[i-1]

[0068] (Basic Mesh Decoding Unit 202) The basic mesh decoding unit 202 is configured to decode the basic mesh bit stream based on the control information, and generate and output a basic mesh.

[0069] Here, a base mesh is made up of one or more sub-meshes, each of which is made up of a plurality of vertices in three-dimensional space and edges connecting these vertices.

[0070] As shown in FIG. 9, the basic mesh is formed by combining basic faces each represented by three vertices.

[0071] The base mesh decoder 202 may be configured to decode the base mesh bitstream using, for example, Draco as described in Non-Patent Document 2.

[0072] The base mesh decoder 202 may be configured to use Draco to independently decode multiple sub-meshes.

[0073] (Subdivision Unit 203) A method for decoding a subdivision mesh in the subdivision unit 203 will be described with reference to FIG.

[0074] The subdivision unit 203 is configured to output a subdivision mesh, subdivision vertex normals, and control information based on the control information and the base mesh.

[0075] 10 shows an example of a subdivision mesh that has been subdivided once using the mid-edge division method. The mid-edge division method generates a subdivision mesh by dividing the midpoints of the edges that make up the mesh.

[0076] By repeating the above process, a finer refined mesh can be generated.

[0077] The subdivision unit 203 is configured to store information about which vertices the subdivision vertex is located between when the subdivision vertex is generated, and to output this information to the boundary correction unit 208 as control information.

[0078] The subdivision mesh may be configured such that each sub-mesh can be decoded independently.

[0079] (Displacement Amount Decoding Unit 205) The displacement amount decoding unit 205 is configured to decode the displacement amount bitstream to generate and output a displacement amount.

[0080] The displacement may be defined so that it can be decoded independently for each sub-mesh.

[0081] (Mesh Decoding Unit 204) The mesh decoding unit 204 is configured to output a decoded mesh based on the input control information, the subdivision mesh, and the amount of displacement.

[0082] The mesh decoding unit 204 adds the displacement amounts to the vertices of the input subdivision mesh, and decodes the decoded mesh.

[0083] At this time, the mesh decoding unit 204 may output a plurality of decoded meshes for each submesh based on the displacement amount corresponding to the submesh and the subdivision mesh.

[0084] In addition, the mesh decoding unit 204 may be configured to set a coordinate system for the displacement amount based on a control signal representing a coordinate system for the displacement amount defined for each submesh, and calculate and output the decoded mesh by adding the displacement amount to the subdivision vertex on the set coordinate system.

[0085] (Boundary Correction Unit 208) A method of decoding a boundary-corrected decoded mesh in the boundary correction unit 208 will be described with reference to FIG.

[0086] The boundary correction unit 208 is configured to correct the submesh boundaries of the input decoded mesh and output the boundary-corrected decoded mesh.

[0087] When afps_border_signal_coding_flag is "0", the boundary correction unit 208 uses the sub-mesh boundary information of the reference frame as it is as the sub-mesh boundary information.

[0088] The boundary correction unit 208 may be configured to skip boundary correction when the frame to be corrected is not an interframe.

[0089] The boundary correction unit 208 may be configured to skip boundary correction and output the decoded mesh as is when the sub-mesh index or vertex index contained in the boundary information does not exist.

[0090] Furthermore, the boundary correction unit 208 may be configured to skip correction of only the boundary pairs of sub-mesh indexes or vertex indexes when the sub-mesh indexes or vertex indexes contained in the boundary information do not exist.

[0091] 11 is a diagram showing a schematic diagram of the boundary correction method, and the boundary correction procedure will be described below with reference to FIG.

[0092] Since the atlas data decoding unit 207 only finds boundary pairs of submesh vertices, it is necessary to find boundary pairs of subdivision vertices in order to perform corrections on the subdivision vertices as well.

[0093] Here we describe the procedure for decoding boundary pairs of subdivision vertices.

[0094] First, the boundary correction unit 208 finds the midpoints of the submesh boundary vertices.

[0095] In the example of FIG. 11, the midpoint between the bounding vertex v(0,3) and the bounding vertex v(0,4) is v(0,5).

[0096] Next, the boundary correction unit 208 determines the boundary vertex pairs of the boundary vertices v(0,3) and v(0,4). The boundary vertex pairs of the boundary vertices v(0,3) and v(0,4) are the boundary vertices v(1,0) and v(1,1).

[0097] Next, the boundary correction unit 208 finds the midpoint between the boundary vertices v(1,0) and v(1,1). This midpoint becomes v(1,4).

[0098] Finally, the boundary correction unit 208 sets the boundary vertex v(0,5) and the boundary vertex v(1,4) as a boundary vertex pair.

[0099] The boundary correction unit 208 can obtain boundary vertex pairs by repeating the above procedure even when the number of subdivisions is two or more.

[0100] The boundary correction unit 208 calculates the average value of the vertex coordinates of the vertices that make up each boundary vertex pair, and corrects the vertices that make up the boundary vertex pair using the average value.

[0101] According to this embodiment, boundary information of a reference frame is used as it is as boundary information of an inter-frame, so that boundary information can be coded efficiently.

[0102] Furthermore, according to this embodiment, the boundary information can be efficiently encoded by utilizing the redundancy contained in the index of the boundary information.

[0103] The mesh encoding device 100 and mesh decoding device 200 described above may be realized as a program that causes a computer to execute each function (each step).

[0104] According to this embodiment, for example, it is possible to improve the overall service quality in video communication, which makes it possible to contribute to Goal 9 of the Sustainable Development Goals (SDGs) led by the United Nations, which is to "Develop resilient infrastructure, promote sustainable industrialization and foster innovation."

[0105] REFERENCE SIGNS LIST 1... Mesh processing system 100... Mesh encoding device 200... Mesh decoding device 201... Demultiplexing unit 202... Basic mesh decoding unit 203... Subdivision unit 204... Mesh decoding unit 205... Displacement amount decoding unit 206... Video decoding unit 207... Atlas data decoding unit 208... Boundary correction unit

Claims

1. A mesh decoding device comprising: an atlas data decoding unit configured to decode an atlas bitstream and generate and output control information; a base mesh decoding unit configured to generate and output a base mesh using the control information and a base mesh bitstream as input; a subdivision unit configured to output a subdivision mesh and subdivision vertex normals using the control information and the base mesh as input; a mesh decoding unit configured to generate a decoded mesh using the control information, a displacement amount, the subdivision mesh, and the subdivision vertex normals as input; and a boundary correction unit configured to input the control information and the decoded mesh and output a boundary-corrected decoded mesh.

2. The mesh decoding device according to claim 1, wherein said atlas data decoding unit decodes a control signal indicating whether or not to send boundary information.

3. The mesh decoding device according to claim 2, characterized in that, when the boundary information is not sent, the atlas data decoding unit decodes the boundary information of the reference frame as is as the boundary information.

4. A mesh decoding method comprising the steps of: decoding an atlas bitstream to generate and output control information; generating a base mesh using the control information and a basic mesh bitstream as input and outputting the control information; using the control information and the base mesh as input and outputting a subdivision mesh and subdivision vertex normals; generating a decoded mesh using the control information, the displacement amount, the subdivision mesh and the subdivision vertex normals as inputs; and inputting the control information and the decoded mesh and outputting a boundary-corrected decoded mesh.

5. A program that causes a computer to function as a mesh decoding device, the mesh decoding device comprising: an atlas data decoding unit configured to decode an atlas bitstream and generate and output control information; a base mesh decoding unit configured to generate and output a base mesh using the control information and a base mesh bitstream as input; a subdivision unit configured to output a subdivision mesh and subdivision vertex normals using the control information and the base mesh as input; a mesh decoding unit configured to generate a decoded mesh using the control information, the displacement amount, the subdivision mesh, and the subdivision vertex normals as input; and a boundary correction unit configured to input the control information and the decoded mesh and output a boundary-corrected decoded mesh.

Images