Mesh decoding device, mesh decoding method, and program

Abstract

The present invention performs correction even when the number of subdivisions is different for each submesh. In a mesh decoding device 200 according to the present invention, a boundary correction unit 208 skips calculation of boundary information of a subdivision vertex when a subdivision vertex calculation control signal is "1", and performs a boundary correction of a submesh on the basis of only boundary information that is stored in an atlas bitstream.

Description

Mesh decoding device, mesh decoding method and program

[0001] The present invention relates to a mesh decoding device, a mesh decoding method, and a program.

[0002] In Non-Patent Document 1, meshes are decoded by dividing them into basic meshes that represent rough information and displacements that represent detailed information, and the displacements are decoded using a video codec.

[0003] Khaled Mammou, Jungsun Kim, Alexis Tourapis, Dimitri Podborski, Krasimir Kolarov, “[V-CG] Apple's Dynamic Mesh Coding CfP Response,” ISO / IEC JTC 1 / SC 29 / WG 7 m59281, April 2022. Google Draco, accessed May 26, 2022 [Online], https: / / google.github.io / draco

[0004] However, the existing methods have a problem in that they cannot correct the subdivision boundaries when the number of subdivisions differs for each submesh. Therefore, the present invention has been made in consideration of the above-mentioned problem, and an object of the present invention is to provide a mesh decoding device, a mesh decoding method, and a program that can perform correction even when the number of subdivisions differs for each submesh.

[0005] a mesh decoding unit configured to receive the first control information, the second control information, and the base mesh as input, and output a subdivision mesh and subdivision vertex normals; a mesh decoding unit configured to receive the first control information, the second control information, and the base mesh as input, and output a subdivision mesh and subdivision vertex normals; a mesh decoding unit configured to receive the first control information, a displacement amount, the subdivision mesh, and the subdivision vertex normals as input, and generate a decoded mesh; and a boundary correction unit configured to receive the first control information and the decoded mesh as input, and output a boundary-corrected decoded mesh;

[0006] A second feature of the present invention is a mesh decoding method comprising: step A of decoding an atlas bitstream to generate and output first control information; step B of using a base mesh bitstream as input to generate and output a base mesh and second control information; step C of using the first control information, the second control information, and the base mesh as input to output a subdivision mesh and subdivision vertex normals; step D of generating a decoded mesh using the first control information, a displacement amount, the subdivision mesh, and the subdivision vertex normals as input; and step E of inputting the first control information and the decoded mesh to output a boundary-corrected decoded mesh, wherein in step E, when a subdivision vertex calculation control signal is "1", calculation of boundary information for the subdivision vertices is skipped, and boundary correction of the submeshes is performed based only on the boundary information stored in the atlas bitstream.

[0007] A third aspect of the present invention is a program that causes a computer to function as a mesh decoding device, the mesh decoding device comprising: an atlas data decoding unit configured to decode an atlas bitstream and generate and output first control information; a base mesh decoding unit configured to receive a base mesh bitstream as input and generate and output a base mesh and second control information; a subdivision unit configured to receive the first control information, the second control information, and the base mesh as input and output a subdivision mesh and subdivision vertex normals; a mesh decoding unit configured to receive the first control information, a displacement amount, the subdivision mesh, and the subdivision vertex normals as input and generate a decoded mesh; and a boundary correction unit configured to receive the first control information and the decoded mesh as input and output a boundary-corrected decoded mesh, wherein when a subdivision vertex calculation control signal is "1", the boundary correction unit skips calculation of boundary information for subdivision vertices and performs boundary correction for submeshes based only on the boundary information stored in the atlas bitstream.

[0008] According to the present invention, it is possible to provide a mesh decoding device, a mesh decoding method, and a program that can perform correction even when the number of subdivisions differs for each submesh.

[0009] FIG. 1 is a diagram showing an example of the configuration of a mesh processing system 1 according to an embodiment. FIG. 2 is a diagram showing an example of functional blocks of a mesh decoding device 200 according to an embodiment. FIG. 3 is a diagram showing an example of the configuration of an AVE syntax in the first embodiment. FIG. 4 is a diagram showing an example of the configuration of an AFMI syntax in the first embodiment. FIG. 5 is a diagram showing an example of the configuration of a Zippering syntax. FIG. 6 is a diagram showing an example of a submesh decoded by the basic mesh decoding unit 202. FIG. 7 is an example of zp_border_submesh_idx and zp_border_vertex_idx. FIG. 8 is a diagram showing an example of a basic mesh. FIG. 9 is a diagram showing an example of a subdivision mesh. FIG. 10 is a diagram schematically showing a method of boundary correction.

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the components in the following embodiments can be appropriately replaced with existing components, etc., and various variations, including combinations with other existing components, are possible. Therefore, the description of the following embodiments does not limit the content of the invention described in the claims.

[0011] First Embodiment A mesh processing system 1 according to this embodiment will be described below with reference to FIGS.

[0012] 1 is a diagram showing an example of the configuration of a mesh processing system 1 according to this embodiment. As shown in FIG. 1, the mesh processing system 1 includes a mesh encoding device 100 and a mesh decoding device 200.

[0013] FIG. 2 is a diagram showing an example of functional blocks of a mesh decoding device 200 according to this embodiment.

[0014] As shown in Figure 2, the mesh decoding device 200 has a demultiplexing unit 201, a basic mesh decoding unit 202, a subdivision unit 203, a mesh decoding unit 204, a displacement amount decoding unit 205, a video decoding unit 206, an atlas data decoding unit 207, and a boundary correction unit 208.

[0015] The demultiplexing unit 201 is configured to separate the multiplexed bit stream into an atlas bit stream, a base mesh bit stream, a displacement amount bit stream, and a texture bit stream.

[0016] The basic mesh decoding unit 202 is configured to decode the basic mesh bit stream based on the control information (first control information) to generate and output control information (second control information) and a basic mesh.

[0017] The subdivision unit 203 is configured to generate and output subdivision vertices and their connection information from the basic mesh decoded by the basic mesh decoding unit 202, using the subdivision method indicated by the control information (first control information and second control information). The basic mesh is composed of one submesh or multiple submeshes.

[0018] Here, the basic mesh, the added subdivision vertices, and the information on their connections are collectively referred to as a "subdivision mesh." Also, the submesh, the added subdivision vertices, and the information on their connections are collectively referred to as a "subdivision submesh."

[0019] The mesh decoding unit 204 is configured to generate and output a decoded mesh using the control information, the subdivision mesh, the subdivision vertex normals, and the displacement amounts.

[0020] The displacement amount decoding unit 205 is configured to decode the displacement amount bit stream based on the control information to generate and output the displacement amount.

[0021] The video decoding unit 206 is configured to decode the texture using a video codec and output the decoded texture.

[0022] The atlas data decoding unit 207 is configured to decode the atlas bitstream and output control information (first control information).

[0023] The boundary correction unit 208 is configured to correct the submesh boundaries of the input decoded mesh and output the boundary-corrected decoded mesh.

[0024] (Atlas Data Decoding Unit 207) Hereinafter, the control information decoded by the atlas data decoding unit 207 will be described with reference to FIGS.

[0025] First, the atlas bitstream may include an AVE (asps vdmc extension), which is a collection of control information for the atlas.

[0026] Second, the atlas bitstream may include AFMI (afps frame mesh information), which is a collection of frame control information for the atlas.

[0027] Third, the atlas bitstream may include Zippering, which is a collection of boundary correction related control information.

[0028] AVE, AFMI, and Zippering may be defined for each mesh data, or may be defined at a finer granularity such as frame, patch, or tile.

[0029] The structure of the syntax decoded by the atlas data decoding unit 207 and the decoding method thereof will be described below with reference to FIGS.

[0030] 3 is a diagram showing an example of the structure of the AVE syntax. Here, if the syntax functions are similar, syntax names different from those shown in FIG. 3 may be used.

[0031] In the AVE syntax structure shown in Figure 3, the Description column indicates how each syntax is coded. Also, ue(v) indicates an unsigned zeroth-order exponential-Golomb code, and u(n) indicates an n-bit flag.

[0032] The Description is merely an example, and an encoding method other than the Description of the diagram, such as u(n), ue(v), or ae(v), may be selected.

[0033] The same applies to the syntax structure of AFMI and zippering.

[0034] AVE is configured not to define asve_subdivision_method, asve_subdivision_iteration_count, asve_displacement_coordinate_system, vdmc_lifting_transform_parameters(0, AspsSubdivisionCount), asve_1d_displacement_flag, and asve_displacement_frame_qp_minus_N when the geometry information flag vps_geometry_video_present_flag is "1".

[0035] Here, asve_subdivision_method is a control signal representing the subdivision method, asve_subdivision_iteration_count is a control signal representing the number of subdivisions, asve_displacement_coordinate_system is a control signal representing the coordinate system that defines the displacement amount, vdmc_lifting_transform_parameters(0, AspsSubdivisionCount) is a control signal related to the displacement amount, asve_1d_displacement_flag is a control signal representing one-dimensional displacement amount encoding, and asve_displacement_frame_qp_minus_N is a control signal representing the quantized value of the displacement amount.

[0036] 4 is a diagram showing an example of the structure of AFMI syntax. Here, if the syntax functions are similar, syntax names different from those shown in FIG. 4 may be used.

[0037] AFMI includes a single mesh control signal afmi_use_single_mesh_flag.

[0038] When afmi_use_single_mesh_flag is "0", AFMI includes afmi_num_submeshes_minus2, and NumSubMeshes can be decoded by adding 2 to afmi_num_submeshes_minus2.

[0039] When afmi_use_single_mesh_flag is “1”, NumSubMeshes is decoded as 1.

[0040] A method for decoding boundary information will be described below with reference to FIGS.

[0041] FIG. 5 is a diagram showing an example of the structure of Zippering syntax.

[0042] Zippering includes a control signal zp_persistence_flag that indicates whether to send boundary information for each frame.

[0043] Zippering includes a control signal zp_border_vertex_pair_count_minus1 that represents the number of border pairs per frame.

[0044] Here, a boundary pair is a combination of vertices with the same coordinates. Details of boundary pairs will be described later.

[0045] Zippering includes a control signal zp_border_vertex_pair_length_minus1[i] that indicates the number of vertices that make up the i-th border pair per frame.

[0046] Zippering includes a control signal zp_border_submesh_idx[i][j] that represents the submesh index of the jth vertex that makes up the ith boundary pair for each frame.

[0047] Zippering includes a control signal zp_border_vertex_idx[i][j] that represents the vertex index of the jth vertex that makes up the ith border pair for each frame.

[0048] Zippering includes the submesh subdivision vertex calculation control signal zp_subdivision_vertex_border_calculation_flag.

[0049] FIG. 6 shows an example of a sub-mesh decoded by the basic mesh decoding unit 202. As shown in FIG.

[0050] 6, the displacement amounts are defined for the vertices on the sub-mesh and their subdivision vertices, but even if the vertex coordinates are the same, the displacement amounts may differ if the sub-mesh indexes are different. As a result, when the displacement amounts are added by the mesh decoding unit 204, holes may appear at the sub-mesh boundaries.

[0051] To solve this problem, the boundary correction unit 208 corrects the vertices of the decoded mesh output from the mesh decoding unit 204 based on the boundary information.

[0052] Here, the boundary information is information about vertices that have the same coordinates even though they are on different sub-meshes at the time of sub-mesh decoding.

[0053] Note that the boundary information can be obtained by decoding the atlas bitstream described above, and the boundary vertices can be decoded as a set of boundary pairs.

[0054] FIG. 7 is an example of zp_border_submesh_idx and zp_border_vertex_idx.

[0055] In the example of FIG. 7, when the index of the boundary pair is "2", zp_border_submesh_idx[2] = [0, 1, 2] and zp_border_vertex_idx[2] = [5, 2, 0].

[0056] This indicates that submesh vertex v(0,5), submesh vertex v(1,2), and submesh vertex v(2,0) are the same vertex, where v(i,j) indicates vertex j of submesh i.

[0057] These pairs of identical vertices are called boundary pairs. In the example above, the boundary pair is 2, but so are all other boundary pairs.

[0058] When the border pair index is i, the vertex pairs are obtained as v(zp_border_submesh_idx[i][0], zp_border_vertex_idx[i][0]), ..., v(zp_border_submesh_idx[i][zp_border_vertex_pair_length_minus1[i]], zp_border_vertex_idx[i][zp_border_vertex_pair_length_minus1[i]])).

[0059] By repeating the above procedure for all boundary pair indexes from 0 to zp_border_pair_count_minus1, it is possible to find boundary pairs included in all boundary information.

[0060] Any boundary information can be decoded using the same procedure as in the above example.

[0061] (Basic Mesh Decoding Unit 202) The basic mesh decoding unit 202 is configured to decode the basic mesh bit stream based on the control information, and generate and output a basic mesh.

[0062] Here, a base mesh is made up of one or more sub-meshes, each of which is made up of a plurality of vertices in three-dimensional space and edges connecting these vertices.

[0063] As shown in FIG. 8, the basic mesh is formed by combining basic faces each represented by three vertices.

[0064] The base mesh decoder 202 may be configured to decode the base mesh bitstream using, for example, Draco as described in Non-Patent Document 2 or other known mesh decoding techniques.

[0065] The base mesh decoder 202 may be configured to independently decode multiple sub-meshes using Draco or other known mesh decoding techniques.

[0066] (Subdivision Unit 203) A method for decoding a subdivision mesh in the subdivision unit 203 will be described with reference to FIG.

[0067] The subdivision unit 203 is configured to output a subdivision mesh, subdivision vertex normals, and control information based on the control information and the base mesh.

[0068] 10 shows an example of a subdivision mesh that has been subdivided once using the mid-edge division method. The mid-edge division method generates a subdivision mesh by dividing the midpoints of the edges that make up the mesh.

[0069] By repeating the above process, a finer subdivision mesh can be generated.

[0070] The subdivision unit 203 is configured to store information about which vertices the subdivision vertex is located between when the subdivision vertex is generated, and to output this information to the boundary correction unit 208 as control information.

[0071] The subdivision unit 203 may be configured not to record the above information when zp_subdivision_vertex_border_calculation_flag is "1".

[0072] The subdivision mesh may be configured such that each sub-mesh can be decoded independently.

[0073] (Displacement Amount Decoding Unit 205) The displacement amount decoding unit 205 is configured to decode the displacement amount bitstream to generate and output a displacement amount.

[0074] The displacement may be defined so that it can be decoded independently for each sub-mesh.

[0075] (Mesh Decoding Unit 204) The mesh decoding unit 204 is configured to output a decoded mesh based on the input control information, the subdivision mesh, and the amount of displacement.

[0076] The mesh decoding unit 204 adds the displacement amounts to the vertices of the input subdivision mesh, and decodes the decoded mesh.

[0077] At this time, the mesh decoding unit 204 may output a plurality of decoded meshes for each submesh based on the displacement amount corresponding to the submesh and the subdivision mesh.

[0078] In addition, the mesh decoding unit 204 may be configured to set a coordinate system for the displacement amount based on a control signal representing a coordinate system for the displacement amount defined for each submesh, and calculate and output the decoded mesh by adding the displacement amount to the subdivision vertex on the set coordinate system.

[0079] (Boundary Correction Unit 208) A method of decoding a boundary-corrected decoded mesh in the boundary correction unit 208 will be described with reference to FIG.

[0080] The boundary correction unit 208 is configured to correct the submesh boundaries of the input decoded mesh and output the boundary-corrected decoded mesh.

[0081] When zp_persistence_flag is "1", the boundary correction unit 208 uses the sub-mesh boundary information of the reference frame as it is as the sub-mesh boundary information.

[0082] The boundary correction unit 208 may be configured to skip boundary correction when the frame to be corrected is not an interframe.

[0083] The boundary correction unit 208 may be configured to skip boundary correction and output the decoded mesh as is when the sub-mesh index or vertex index contained in the boundary information does not exist.

[0084] Furthermore, the boundary correction unit 208 may be configured to skip correction of only the boundary pairs of sub-mesh indexes or vertex indexes when the sub-mesh indexes or vertex indexes contained in the boundary information do not exist.

[0085] 10 is a diagram showing a schematic diagram of the boundary correction method, and the boundary correction procedure will be described below with reference to FIG.

[0086] When zp_subdivision_vertex_border_calculation_flag is "0", the atlas data decoding unit 207 only finds boundary pairs of submesh vertices, so it is necessary to find boundary pairs of subdivision vertices in order to perform corrections on the subdivision vertices as well.

[0087] However, when zp_subdivision_vertex_border_calculation_flag is “1”, the boundary correction unit 208 may perform correction processing only on the boundary pairs determined by the atlas data decoding unit 207 .

[0088] Here we describe the procedure for decoding boundary pairs of subdivision vertices.

[0089] First, the boundary correction unit 208 finds the midpoints of the submesh boundary vertices.

[0090] In the example of FIG. 10, the midpoint between the bounding vertex v(0,3) and the bounding vertex v(0,4) is v(0,5).

[0091] Next, the boundary correction unit 208 determines the boundary vertex pairs of the boundary vertices v(0,3) and v(0,4). The boundary vertex pairs of the boundary vertices v(0,3) and v(0,4) are the boundary vertices v(1,0) and v(1,1).

[0092] Next, the boundary correction unit 208 finds the midpoint between the boundary vertices v(1,0) and v(1,1). This midpoint becomes v(1,4).

[0093] Finally, the boundary correction unit 208 sets the boundary vertex v(0,5) and the boundary vertex v(1,4) as a boundary vertex pair.

[0094] The boundary correction unit 208 can obtain boundary vertex pairs by repeating the above procedure even when the number of subdivisions is two or more.

[0095] The boundary correction unit 208 calculates the average value of the vertex coordinates of the vertices that make up each boundary vertex pair, and corrects the vertices that make up the boundary vertex pair using the average value.

[0096] According to this embodiment, when the geometry information flag vps_geometry_video_present_flag is "1", syntax relating to the displacement amount and subdivision is not coded, thereby making it possible to reduce the amount of bits.

[0097] Furthermore, according to this embodiment, Zippering is introduced, and when Zippering (submesh subdivision vertex calculation control signal) is "1", calculation of boundary information for subdivision vertices is skipped and boundary correction of submeshes is performed based only on the boundary information stored in the bitstream. This allows correction even when the number of subdivisions differs for each submesh.

[0098] Furthermore, according to this embodiment, when the single mesh control signal afmi_use_single_mesh_flag is "1", the signal representing the number of sub-meshes is added by 1, and in other cases, the control signal representing the number of sub-meshes is added by 2, thereby decoding the number of sub-meshes. In this way, by using afmi_use_single_mesh_flag in combination, the range of the number of sub-meshes that can be represented can be expanded.

[0099] The mesh encoding device 100 and mesh decoding device 200 described above may be realized as a program that causes a computer to execute each function (each step).

[0100] According to this embodiment, for example, it is possible to improve the overall service quality in video communication, which makes it possible to contribute to Goal 9 of the Sustainable Development Goals (SDGs) led by the United Nations, which is to "Develop resilient infrastructure, promote sustainable industrialization and foster innovation."

[0101] REFERENCE SIGNS LIST 1... Mesh processing system 100... Mesh encoding device 200... Mesh decoding device 201... Demultiplexing unit 202... Basic mesh decoding unit 203... Subdivision unit 204... Mesh decoding unit 205... Displacement amount decoding unit 206... Video decoding unit 207... Atlas data decoding unit 208... Boundary correction unit

Claims

1. A mesh decoding device, comprising: an atlas data decoding unit configured to decode an atlas bit stream to generate and output first control information; a basic mesh decoding unit configured to input a basic mesh bit stream and generate and output a basic mesh and second control information; a subdivision unit configured to input the first control information, the second control information, and the basic mesh and output a subdivided mesh and a subdivided vertex normal; a mesh decoding unit configured to input the first control information, a displacement amount, the subdivided mesh, and the subdivided vertex normal and generate a decoded mesh; and a boundary correction unit configured to input the first control information and the decoded mesh and output a boundary-corrected decoded mesh, wherein the boundary correction unit skips calculation of boundary information of subdivided vertices when a subdivided vertex calculation control signal is "1" and performs boundary correction of a sub-mesh based only on boundary information stored in the atlas bit stream.

2. A mesh decoding method, comprising: step A of decoding an atlas bit stream to generate and output first control information; step B of inputting a basic mesh bit stream and generating and outputting a basic mesh and second control information; step C of inputting the first control information, the second control information, and the basic mesh and outputting a subdivided mesh and a subdivided vertex normal; step D of inputting the first control information, a displacement amount, the subdivided mesh, and the subdivided vertex normal and generating a decoded mesh; and step E of inputting the first control information and the decoded mesh and outputting a boundary-corrected decoded mesh, wherein in step E, when a subdivided vertex calculation control signal is "1", calculation of boundary information of subdivided vertices is skipped and boundary correction of a sub-mesh is performed based only on boundary information stored in the atlas bit stream.

3. A program for causing a computer to function as a mesh decoding device, wherein the mesh decoding device includes: - An atlas data decoding unit configured to decode an atlas bit stream and generate and output first control information; - A basic mesh decoding unit configured to input a basic mesh bit stream and generate and output a basic mesh and second control information; - A subdivision unit configured to input the first control information, the second control information, and the basic mesh and output a subdivided mesh and subdivided vertex normals; - A mesh decoding unit configured to input the first control information, a displacement amount, the subdivided mesh, and the subdivided vertex normals and generate a decoded mesh; and - A boundary correction unit configured to input the first control information and the decoded mesh and output a boundary-corrected decoded mesh, wherein the boundary correction unit skips the calculation of the boundary information of the subdivided vertices when a subdivided vertex calculation control signal is "1" and performs boundary correction of the sub-meshes based only on the boundary information stored in the atlas bit stream.

Images