Mesh decoding device, mesh decoding method, and program

Abstract

To facilitate design.SOLUTION: A mesh decoding device 200 comprises an atlas data decoding section 207 configured to decode a syntax that defines a maximal value of the number of repetitions of subdivision. The value of the syntax decoded by the atlas data decoding section 207 is 15 or less.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a mesh decoding device, a mesh decoding method, and a program. [Background technology]

[0002] Non-Patent Document 1 discloses a technique for dividing a mesh into a rough basic mesh and a detailed displacement amount, and encoding and decoding are performed by the division.

[0003] Specifically, in this technology, when decoding, the bit stream is demultiplexed into a basic mesh bit stream and a displacement bit stream. The basic mesh bit stream is decoded by a basic mesh decoding unit, and then repeatedly subdivided by a subdivision unit to decode fine meshes. Then, the displacement is added or subtracted to output the decoded mesh.

[0004] In Non-Patent Document 2, a design is made to realize the technology disclosed in Non-Patent Document 1. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Khaled Mammou, Jungsun Kim, Alexis Tourapis, Dimitri Podborski, Krasimir Kolarov, “[V-CG] Apple's Dynamic Mesh Coding CfP Response,” ISO / IEC JTC 1 / SC 29 / WG 7 m59281, April 2022. [Non-Patent Document 2] “WD 3.0 of V-DMC,” ISO / IEC JTC 1 / SC 29 / WG7 w22775, April 2023 Summary of the Invention [Problem to be solved by the invention]

[0006] However, in Non-Patent Document 2, an irrational number of repetitions of subdivision is set, which makes design difficult. Therefore, the present invention has been made in consideration of the above-mentioned problems, and aims to provide a mesh decoding device, a mesh decoding method, and a program that can be realized with a simple design. [Means for solving the problem]

[0007] A first feature of the present invention is a mesh decoding device comprising an atlas data decoding unit configured to decode a syntax that specifies a maximum number of subdivision iterations, and the value of the syntax decoded by the atlas data decoding unit is 15 or less.

[0008] A second feature of the present invention is a mesh decoding device comprising an atlas data decoding unit configured to decode a syntax that specifies a maximum number of subdivision iterations, and the value of the syntax decoded by the atlas data decoding unit is 7 or less.

[0009] A third feature of the present invention is a mesh decoding device comprising an atlas data decoding unit configured to decode a syntax that specifies a maximum number of subdivision iterations, and when high quality is set, the value of the syntax decoded by the atlas data decoding unit is 7 or less, and when general quality is set, the value of the syntax decoded by the atlas data decoding unit is 15 or less.

[0010] A fourth feature of the present invention is a mesh decoding method comprising a step of decoding a syntax that specifies a maximum number of times subdivision is repeated, and the value of the syntax decoded in the step is 15 or less.

[0011] A fifth feature of the present invention is a program for causing a computer to function as a mesh decoding device, the mesh decoding device comprising an atlas data decoding unit configured to decode a syntax that specifies a maximum number of subdivision iterations, and the value of the syntax decoded by the atlas data decoding unit is 15 or less. Effect of the Invention

[0012] According to the present invention, it is possible to provide a mesh decoding device, a mesh decoding method, and a program that can be realized with a simple design. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a mesh processing system 1 according to an embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of functional blocks of a mesh decoding device 200 according to an embodiment. [Diagram 3] FIG. 3 is a flowchart showing an example of the operation of the subdivision section 203. [Figure 4] FIG. 4 is an example of parameter sets and data units to which subdivision pertains. [Diagram 5] FIG. 5 is a diagram illustrating an example of an ASPS syntax configuration. [Figure 6] FIG. 6 is a diagram illustrating an example of an AFPS syntax configuration. [Figure 7] FIG. 7 is a diagram illustrating an example of a PDU syntax configuration. [Figure 8A] FIG. 8A is a diagram showing a basic mesh input to the subdivision unit 203. As shown in FIG. [Figure 8B] FIG. 8B is a diagram showing the subdivision mesh after one degree of subdivision. [Figure 9A] FIG. 9A is a diagram showing an example of a tetrahedron to be subdivision. [Figure 9B]FIG. 9B is a table summarizing the number of times subdivision is repeated, the number of bits required, the number of faces, and the number of vertices when the tetrahedron shown in FIG. 9A is subdivided. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, the embodiments of the present invention will be described with reference to the drawings. Note that the components in the following embodiments can be appropriately replaced with existing components, and various variations including combinations with other existing components are possible. Therefore, the description of the following embodiments does not limit the content of the invention described in the claims.

[0015] First Embodiment Hereinafter, the mesh processing system according to this embodiment will be described with reference to FIGS.

[0016] 1 is a diagram showing an example of the configuration of a mesh processing system 1 according to this embodiment. As shown in FIG.

[0017] FIG. 2 is a diagram showing an example of functional blocks of the mesh decoding device 200 according to this embodiment.

[0018] As shown in FIG. 2, the mesh decoding device 200 includes a demultiplexing unit 201, a basic mesh decoding unit 202, a subdivision unit 203, a mesh decoding unit 204, a displacement amount decoding unit 205, a video decoding unit 206, and an atlas data decoding unit 207.

[0019] The demultiplexing unit 201 is configured to separate the multiplexed bit stream into a base mesh bit stream, a displacement amount bit stream, a texture bit stream, and an atlas bit stream.

[0020] The base mesh decoding unit 202 is configured to decode the base mesh bitstream and generate and output a base mesh.

[0021] The subdivision unit 203 is configured to generate and output added subdivision vertices and their connection information from the basic mesh decoded by the basic mesh decoding unit 202 using a subdivision method indicated by the control information.

[0022] Here, the basic mesh, the added subdivision vertices, and their connection information are collectively referred to as a "subdivision mesh."

[0023] The mesh decoding unit 204 is configured to generate and output a decoded mesh using the subdivision mesh generated by the subdivision unit 203 and the amount of displacement decoded by the displacement amount decoding unit 205 .

[0024] The displacement amount decoding unit 205 is configured to decode the displacement amount bitstream to generate and output the displacement amount.

[0025] The video decoding unit 206 is configured to decode the texture by video encoding and output the texture. For example, the video decoding unit 206 may use HEVC in Non-Patent Document 1.

[0026] The atlas data decoding unit 207 is configured to decode the atlas bitstream and output the control information.

[0027] (Subdivision section 203) FIG. 3 is a flowchart showing an example of the operation of the subdivision section 203.

[0028] As shown in steps S2031 to S2033 in FIG. 3, the subdivision section 205 repeats the subdivision a determined number of times.

[0029] (Structure of Displacement Bit Stream) An example of a bitstream configuration related to subdivision will be described below with reference to Figs.

[0030] Figure 4 shows an example of a parameter set and data unit related to the atlas, including information controlling subdivision. In Non-Patent Document 2, this is control information obtained by the atlas data decoder 207, but if the meaning is the same, it may be moved to another bitstream and decoded there.

[0031] As shown in FIG. 4, first, the atlas bitstream may include an Atlas Sequence Parameter Set (ASPS), which is a set of control information related to decoding of the atlas information of a sequence.

[0032] Secondly, the atlas bitstream may include an atlas frame parameter set (AFPS), which is a set of control information corresponding to a frame, after the ASPS.

[0033] Thirdly, the atlas bitstream may include a patch data unit (PDU) after the AFPS, which is a data unit for each patch.

[0034] Figures 5 to 7 are diagrams showing examples of syntax. In Figures 5 to 7, the Descriptor column indicates how each syntax is coded. In Figures 5 to 7, u(n) indicates an n-bit code.

[0035] FIG. 5 is a diagram illustrating an example of an ASPS syntax configuration.

[0036] In FIG. 5, the ASPS includes a subdivision method (afps_vdmc_ext_subdivision_method).

[0037] For example, when afps_vdmc_ext_subdivision_method is other than 0, it is defined that subdivision is performed, and ASPS includes the number of subdivision iterations (afps_vdmc_ext_subdivision_iteration_count), which is given by u(8) in Non-Patent Document 2.

[0038] FIG. 6 is a diagram illustrating an example of an AFPS syntax configuration.

[0039] AFPS includes a flag (afps_vdmc_ext_overriden_flag) indicating whether to overwrite.

[0040] For example, when afps_vdmc_ext_overriden_flag is 1, AFPS includes a flag (afps_vdmc_ext_subdivision_enable_flag) indicating whether or not to enable subdivision.

[0041] For example, when afps_vdmc_ext_subdivision_enable_flag is 1, subdivision is enabled.

[0042] When subdivision is enabled, AFPS includes a method (afps_vdmc_ext_subdivision_method) that indicates the subdivision.

[0043] For example, when afps_vdmc_ext_subdivision_method is non-zero and some subdivision method is specified, AFPS includes the number of subdivision iterations (afps_vdmc_ext_subdivision_iteration_count), which is given by u(8) in Non-Patent Document 2.

[0044] 7 is a diagram showing an example of a PDU syntax configuration. PDF is data for each patch index.

[0045] The PDU includes a flag (pdu_parameters_enable_flag) indicating whether or not to enable PDU parameters.

[0046] For example, when pdu_parameters_enable_flag is 1, the PDU includes a flag (pdu_subdivision_enable_flag) indicating whether or not to enable subdivision.

[0047] For example, when pdu_subdivision_enable_flag is 1, subdivision is enabled.

[0048] When subdivision is enabled, the PDU contains a method (pdu_subdivision_method) that indicates the subdivision.

[0049] For example, when pdu_subdivision_method is non-zero and some subdivision method is specified, the PDU includes the number of subdivision iterations (pdu_subdivision_iteration_count), which is given by u(8) in Non-Patent Document 2.

[0050] (subdivision part 2032) 8A and 8B show how the number of vertices and the number of faces change in one subdivision.

[0051] FIG. 8A shows a basic mesh input to the subdivision unit 203, and FIG. 8B shows a subdivision mesh after one degree of subdivision.

[0052] In general, when a subdivision method such as Midpoint is used on a triangular mesh, the number of subdivision meshes increases by about four times after one subdivision.

[0053] It can also be seen in Figure 8B that the number of faces is four times the basic number of faces compared to Figure 8A. In other words, each time subdivision is performed, the number of faces becomes the number of faces before division x 4.

[0054] In addition, each time a subdivision is performed, the number of edges becomes the number of edges before the division x 2 + the number of faces x 3. This is because the edge before the division is split into two, and three new edges are created inside one face.

[0055] In addition, each time a subdivision is performed, the number of vertices becomes the number of edges before the subdivision plus the number of vertices before the subdivision. This is because a new vertex is created at the midpoint of the edge before the subdivision, and the number of vertices before the subdivision plus the number of vertices before the subdivision becomes the new number of vertices. The number of faces is about twice the number of vertices.

[0056] Now consider an extremely small base mesh (eg, a tetrahedron with only four faces) as shown in FIG. 9A.

[0057] FIG. 9B shows a table summarizing the number of times subdivision is repeated when the tetrahedron shown in FIG. 9A is subdivided, the number of bits required, the number of faces, the number of vertices, and the memory required to represent the vertices.

[0058] As shown in FIG. 9B, in the case of a typical subdivision up to three times, the number of subdivision iterations can be represented by two bits, resulting in 256 faces and 130 vertices.

[0059] If the number of repeated subdivisions is set to seven, the upper limit that can be represented by three bits, the number of faces will be 65,536 and the number of vertices will be 32,770.

[0060] If the number of subdivision repetitions is set to 15, the maximum number that can be represented by 4 bits, the number of faces will be 4,294,967,296 (approximately 4.3 GB) and the number of vertices will be 2,147,483,650 (approximately 2.2 GB) (rounded to one decimal place).

[0061] If the subdivision is repeated 16 times, one more than 15, which requires 5 bits to represent, the number of faces will be 17,179,869,184 (approximately 17.2 Giga), and the number of vertices will be 8,589,934,594 (approximately 8.6 Giga) (rounded to one decimal place).

[0062] Here, the memory required to represent one frame of meshes when the mesh data compressed under the conditions in FIG. 9B is decoded will be roughly calculated for a specific number of repetitions of subdivision.

[0063] In reality, memory is also required for the data loaded into memory and for the decoding process, but since the memory required to decode and hold one frame is assumed to be large, this is estimated here.

[0064] <16 times> When the number of repetitions of subdivision is 16, which is the minimum number of repetitions when expressing the number of repetitions of subdivision with 5 bits, the number of vertices is approximately 8.6 giga.

[0065] The x, y and z coordinates of a vertex are often represented as 32-bit decimals, meaning that one vertex requires 12 bytes. The representation of the vertices alone requires 103.1 GB per frame.

[0066] In addition, in the mesh representation, connectivity information that connects the vertices is also necessary, and in order to represent the connectivity information, it is necessary to output the vertex index at least once for each vertex.

[0067] If the number of vertices is 8.6 gigabytes, 33 bits are required to represent the index of each vertex, which is about 4 bytes.

[0068] To represent connectivity information, 8.6 x 4 = 34.4 GB is required per frame. Therefore, to represent meshes, not including textures, a minimum of 103.1 + 34.4 = 137.5 GB is required per frame.

[0069] Even in today's high-end desktop computers, a maximum memory of 128 GB is typical, making 16 subdivision iterations impractical.

[0070] As described above, mesh representation requires handling various pieces of information for faces and vertices, which results in a huge amount of memory being required, making implementation extremely difficult from a memory standpoint.

[0071] <15 times> When the number of subdivision repetitions is expressed by 4 bits, which is the maximum number of repetitions, 15, the number of vertices is approximately 2.2 giga.

[0072] The x, y and z coordinates of a vertex are often represented as 32-bit decimals, meaning that one vertex requires 12 bytes. The representation of the vertices alone requires 25.8 GB per frame.

[0073] In addition, in the mesh representation, connectivity information that connects the vertices is also necessary, and in order to represent the connectivity information, it is necessary to output the vertex index at least once for each vertex.

[0074] If the number of vertices is 2.2 gigabytes, 31 bits are required to represent the index of each vertex, which is about 4 bytes.

[0075] To represent connectivity information, 2.2 x 4 = 8.8 GB is required for one frame. Therefore, to represent meshes, not including textures, a minimum of 25.8 + 8.8 = 34.6 GB is required per frame.

[0076] However, this representation is possible with a memory of 64 GB to 128 GB. It is important to set the number of repetitions of subdivision within a reasonable range, and it is preferable to set it to 15 or less.

[0077] However, 64GB or more of memory is required for high-end devices, and there are problems in popularizing it. Also, when textures, buffers, free shapes, etc. are considered, 15 subdivisions may be difficult.

[0078] <8 times> When the number of subdivision repetitions is 8, which is the minimum number of times that the number of subdivision repetitions can be expressed using 4 bits, the number of vertices is 131,074.

[0079] The x, y and z coordinates of a vertex are often expressed as 32-bit decimals, meaning that one vertex requires 12 bytes. The representation of the vertices alone requires 1,572,888B per frame.

[0080] In addition, in the mesh representation, connectivity information that connects the vertices is also necessary, and in order to represent the connectivity information, it is necessary to output the vertex index at least once for each vertex.

[0081] If the number of vertices is 131,074, 18 bits are required to represent the index of each vertex, which is approximately 2 bytes.

[0082] To represent connectivity information, one frame requires 131,074 x 2 = 262,148 B. Therefore, to represent meshes not including textures, a minimum of 1,572,888 + 262,148 = 1,835,036 ≒ 1.8 MB is required per frame.

[0083] As described above, if the shape is not complex and does not include texture information, it can be expressed with approximately 1 GB of memory, which is generally the minimum.

[0084] However, the basic mesh shape without subdivision is somewhat complex, and the number of vertices is set to 514 instead of 4.

[0085] This is also seen in the number of vertices per frame before subdivision used in the MPEG test sequence, which results in approximately 65,0000 faces after three subdivision iterations.

[0086] If the number of subdivision iterations is 8, the number of faces will be 67,108,864, and the number of vertices will be approximately 33,554,434. The vertex representation alone will be 33,554,434 x 12 = 402,653,208B per frame.

[0087] To represent the connectivity information, 26 bits are required to represent the index of each vertex, which is about 3 bytes. That is, 33,554,434 x 3 = 100,663,302 B. 503,316,510 B (about 503 MB) is required per frame to represent the mesh.

[0088] Considering cases where data of multiple past frames is referred to in the decoding process, even in the absence of texture, it may be necessary to limit the number of times subdivision is repeated to only some patches.

[0089] Also, when textures are taken into account, the xyz coordinates of the vertices need to be mapped to uv coordinates. If u and v are each 32-bit decimals, the total becomes 33,554,434 x 8 = 268,435,472B. In total, about 750MB of memory is required per frame.

[0090] When memory is around 1GB, the smallest unit of memory for modern PCs and sufficient for edge devices, there is almost no memory to represent textures, and complex decoding processes such as storing past frames in a buffer are difficult to implement.

[0091] <7th time> When the number of repetitions of subdivision is expressed as 7, which is the maximum number of repetitions when the number of repetitions is expressed with 3 bits, the number of vertices is about 32,770. The x, y, and z coordinates of a vertex are often expressed as 32-bit decimal numbers. In other words, one vertex requires 12 bytes. One frame requires 393,240B (about 394kB) just to represent the vertices.

[0092] In addition, in the mesh representation, connectivity information that connects the vertices is also necessary, and in order to represent the connectivity information, it is necessary to output the vertex index at least once for each vertex.

[0093] If the number of vertices is 32,770, 16 bits are required to represent the index of each vertex, which is about 2 bytes. To represent the connectivity information, one frame requires 32,770 x 2 = 65,540 B (about 66 kB).

[0094] Therefore, a minimum of 393,240 + 65,540 = 458,780B (approximately 458kB) is required per frame to represent the mesh, excluding textures, etc.

[0095] There is greater freedom in retaining loaded data, the decoding process, memory required for textures, and the number of vertices before subdivision.

[0096] Here, as in the case where the number of repeated subdivisions is eight, it is assumed that the number of basic mesh vertices before subdivision is 514 instead of four.

[0097] This is about 1 / 4 of the amount of data required 8 times, so even including the texture coordinates per frame, it comes to about 190MB.

[0098] Even with just 1GB of memory, it will be possible to represent complex decoding processes that retain past frames and color texture information.

[0099] <3 times> When the number of repetitions of the subdivision is expressed by 2 bits, the maximum number of repetitions is 3. This is used as the default value of the current reference software, and it can be decoded even when the texture is complex, and it is the baseline. Because it is the baseline, it is easy to design, including the memory.

[0100] On the other hand, depending on the use case, unevenness may be subjectively noticeable, and it may be desirable to increase the number of subdivision repetitions to four or more depending on the use case.

[0101] In summary, the number of repeated subdivisions (16 times) that require 5 bits for expression is impossible on almost any terminal from a memory standpoint, and a number of times less than 4 bits (15 times or less) is preferable.

[0102] However, in the case of 4 bits, there are constraints such as high-spec memory requirements (15 times) and the difficulty of combining the expression of complex shapes, complex decoding processes, and complex textures (8 times), so a subdivision repetition count of 3 bits or less (7 times or less) is more desirable.

[0103] Also, the upper limit of the number of repetitions of subdivision may be changed depending on the desired level. For example, assuming three levels, the number of repetitions of subdivision may be set to a range that can be expressed in 2 bits for a low level of normal quality, to a range that can be expressed in 3 bits for a medium level of high quality, and to a range that can be expressed in 4 bits for a high level of very high quality.

[0104] According to this embodiment, it is possible to facilitate design by defining reasonable constraints on the number of times subdivision is repeated.

[0105] The above-mentioned mesh encoding device 100 and mesh decoding device 200 may be realized as a program that causes a computer to execute each function (each step). [Industrial Applicability]

[0106] According to this embodiment, for example, it is possible to improve the overall service quality in video communication, which makes it possible to contribute to Goal 9 of the Sustainable Development Goals (SDGs) led by the United Nations, which is to "build resilient infrastructure, promote sustainable industrialization and foster innovation." [Explanation of symbols]

[0107] 1…Mesh processing system 100...Mesh coding device 200…Mesh decoding unit 201…Demultiplexer 202...Basic mesh decoding unit 203…Subdivision part 204...Mesh decoding unit 205...Displacement amount decoding unit 206...Video decoding section 207...Atlas data decoding unit

Claims

1. A mesh decoding device, comprising: An atlas data decoding unit configured to decode a syntax that specifies a maximum number of subdivision iterations, A mesh decoding device, characterized in that the syntax value decoded by the atlas data decoding unit is 15 or less.

2. A mesh decoding device, comprising: An atlas data decoding unit configured to decode a syntax that specifies a maximum number of subdivision iterations, A mesh decoding device, characterized in that the syntax value decoded by the atlas data decoding unit is 7 or less.

3. A mesh decoding device, comprising: An atlas data decoding unit configured to decode a syntax that specifies a maximum number of subdivision iterations, When set to high quality, the syntax value decoded by the atlas data decoding unit is 7 or less; A mesh decoding device characterized in that, when set to general quality, the syntax value decoded by the atlas data decoding unit is 15 or less.

4. A mesh decoding method, comprising: Decoding a syntax that specifies a maximum number of subdivision iterations, A mesh decoding method, characterized in that the syntax value decoded in the step is 15 or less.

5. A program for causing a computer to function as a mesh decoding device, comprising: The mesh decoding device comprises an atlas data decoding unit configured to decode a syntax that specifies a maximum value of the number of subdivision iterations, The program, wherein the syntax value decoded by the atlas data decoding unit is 15 or less.

Images