Mesh decoding device, mesh decoding method, and program
The mesh decoding device and method address the issue of incorrect mesh reconstruction by using an atlas and level value images to determine the appropriate displacement decoding method, ensuring accurate and efficient mesh reconstruction.
Patent Information
- Application Number
- JP2024109215
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2024-07-05
- Publication Date
- 2026-01-19
AI Technical Summary
Existing mesh decoding methods do not support multiple methods of decoding displacement amounts, leading to incorrect reconstruction of meshes.
A mesh decoding device and method that utilizes an atlas, a basic mesh, and a level value image or a first coefficient sequence to reconstruct meshes, incorporating a submesh reconstruction unit and a submesh combining unit, with a displacement amount decoding method determination unit to select the appropriate decoding method based on control information.
Enables correct reconstruction of meshes by determining the appropriate displacement amount decoding method, reducing calculation complexity and ensuring accurate mesh reconstruction.
Smart Images
Figure 2026008478000001_ABST
Abstract
Description
[Technical Field]
[0001] The present invention relates to a mesh decoding device, a mesh decoding method, and a program. [Background technology]
[0002] Non-Patent Document 1 discloses a technique for decoding a point group using video information and additional information called an atlas.
[0003] Non-Patent Document 2 discloses a technique that extends Non-Patent Document 1 to mesh decoding applications.
[0004] Specifically, Non-Patent Document 2 first decodes a rough base mesh and a detailed displacement, and secondly, reconstructs a mesh from both. There are two methods for decoding the displacement: a method using a video codec and a method using arithmetic decoding. The decoding of the displacement may also be omitted. In the method using arithmetic decoding, the level value is decoded from the syntax, and then the displacement is decoded. [Prior art documents] [Non-patent literature]
[0005] [Non-Patent Document 1] “Information technology - Coded Representation of Immersive Media - Part 5: Visual Volumetric Video-based Coding (V3C) and Video-based Point Cloud Compression (V-PCC),” ISO / IEC JTC 1 / SC 29 / WG 7, ISO / IEC 23090-5:2021(2E). [Non-patent document 2] “WD 6.0 of V-DMC,” ISO / IEC JTC 1 / SC 29 / WG 7 N00822. Summary of the Invention [Problem to be solved by the invention]
[0006] However, the method of Non-Patent Document 2 does not support multiple methods of decoding displacement amounts when reconstructing a mesh, and therefore has a problem in that the mesh cannot be reconstructed correctly.
[0007] Therefore, the present invention has been made in consideration of the above-mentioned problems, and an object of the present invention is to provide a mesh decoding device, a mesh decoding method, and a program that can correctly reconstruct meshes. [Means for solving the problem]
[0008] A first feature of the present invention is summarized as a mesh decoding device comprising: a mesh reconstruction unit configured to reconstruct and output a mesh using an atlas, a basic mesh, and a level value image or a first coefficient sequence obtained from a bitstream; the mesh reconstruction unit comprising: a submesh reconstruction unit configured to generate and output a submesh sequence using the atlas, the basic mesh, and the level value image or the first coefficient sequence; and a submesh combining unit configured to generate and output a single mesh by combining the submesh sequences; and the submesh reconstruction unit comprises a displacement amount decoding method determination unit configured to output the level value image or the first coefficient sequence by determining a displacement amount decoding method from information indicated by control information.
[0009] A second feature of the present invention is a mesh decoding method comprising steps of reconstructing and outputting a mesh using an atlas obtained from a bitstream, a basic mesh, and a level value image or a first coefficient sequence, the steps comprising: step A of generating and outputting a sub-mesh sequence using the atlas, the basic mesh, and the level value image or the first coefficient sequence; and step B of generating and outputting a single mesh by combining the sub-mesh sequences, wherein in step A, the level value image or the first coefficient sequence is output by determining a decoding method for a displacement amount from information indicated by control information.
[0010] A third feature of the present invention is summarized as a program causing a computer to function as a mesh decoding device, the mesh decoding device comprising: a mesh reconstruction unit configured to reconstruct and output a mesh using an atlas, a basic mesh, and a level value image or a first coefficient sequence obtained from a bitstream; the mesh reconstruction unit comprising: a submesh reconstruction unit configured to generate and output a submesh sequence using the atlas, the basic mesh, and the level value image or the first coefficient sequence; and a submesh combining unit configured to generate and output a single mesh by combining the submesh sequences; and the submesh reconstruction unit comprising: a displacement amount decoding method determination unit configured to output the level value image or the first coefficient sequence by determining a displacement amount decoding method from information indicated by control information. [Effects of the Invention]
[0011] According to the present invention, it is possible to provide a mesh decoding device, a mesh decoding method and a program that can correctly reconstruct a mesh. [Brief explanation of the drawings]
[0012] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a mesh processing system 1 according to an embodiment. [Figure 2]FIG. 2 is a diagram illustrating an example of functional blocks of a mesh decoding device 200 according to an embodiment. [Figure 3] FIG. 3 is a diagram illustrating an example of the configuration of a V3C unit. [Figure 4] FIG. 4 is a diagram showing an example of the syntax configuration of a V3C unit header. [Figure 5] FIG. 5 is a diagram showing an example of the structure of a bitstream. [Figure 6] FIG. 6 is a diagram showing an example of the configuration of a displacement amount bit stream. [Figure 7] FIG. 7 is a diagram illustrating an example of the syntax configuration of displ_nal_unit_header. [Figure 8] FIG. 8 is a diagram showing an example of the correspondence between displ_nal_unit_type and the contents of the payload. [Figure 9] FIG. 9 is a diagram illustrating an example of the syntax configuration of displ_layer_rbsp. [Figure 10] FIG. 10 is a diagram illustrating an example of the syntax configuration of displ_layer_rbsp. [Figure 11] FIG. 11 is a diagram illustrating an example of functional blocks of the displacement amount decoding unit 204 of the mesh decoding device 200 according to an embodiment. [Figure 12] FIG. 12 is a diagram showing an example of functional blocks of the mesh reconstruction unit 206 of the mesh decoding device 200 according to an embodiment. [Figure 13] FIG. 13 is a diagram showing an example of the configuration of the submesh reconstructing unit 206A of the mesh reconstructing unit 206 of the mesh decoding device 200 according to an embodiment. [Figure 14] FIG. 14 is a flowchart showing an example of processing by the arithmetic decoding unit 204B of the displacement amount decoding unit 204 of the mesh decoding device 200 according to an embodiment. [Figure 15]FIG. 15 is a flowchart showing an example of processing by the submesh reconstruction unit 206A of the mesh reconstruction unit 206 of the mesh decoding device 200 according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION
[0013] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the components in the following embodiments can be appropriately replaced with existing components, etc., and various variations, including combinations with other existing components, are possible. Therefore, the description of the following embodiments does not limit the content of the invention described in the claims.
[0014] First Embodiment This embodiment is based on V3C described in Non-Patent Document 1. An atlas in V3C is a collection of rectangular regions and related information arranged on a frame, and corresponds to a three-dimensional space.
[0015] Hereinafter, a mesh processing system 1 according to this embodiment will be described with reference to FIGS.
[0016] 1 is a diagram showing an example of the configuration of a mesh processing system 1 according to this embodiment. As shown in FIG. 1, the mesh processing system 1 includes a mesh encoding device 100 and a mesh decoding device 200.
[0017] FIG. 2 is a diagram showing an example of functional blocks of the mesh decoding device 200 according to this embodiment.
[0018] As shown in FIG. 2, the mesh decoding device 200 includes a V3C unit separation unit 201, an atlas decoding unit 202, a basic mesh decoding unit 203, a displacement amount decoding unit 204, a texture decoding unit 205, and a mesh reconstruction unit 206.
[0019] The V3C unit separation unit 201 is configured to separate the bitstream into an atlas bitstream, a base mesh bitstream, a displacement bitstream, and a texture bitstream.
[0020] The atlas decoding unit 202 is configured to decode and output the atlas from the atlas bitstream.
[0021] The base mesh decoding unit 203 is configured to decode and output the base mesh from the base mesh bitstream.
[0022] The displacement decoding unit 204 is configured to decode and output the level value image or the first coefficient sequence from the displacement bitstream.
[0023] The texture decoding unit 205 is configured to decode and output texture from the texture bitstream. The texture decoding unit 205 may decode the texture using HEVC.
[0024] The mesh reconstruction unit 206 is configured to reconstruct and output a mesh using the atlas decoded by the atlas decoding unit 201, the basic mesh decoded by the basic mesh decoding unit 203, and the level value image or first coefficient sequence decoded by the displacement amount decoding unit 204.
[0025] (Bitstream configuration) A bitstream may be composed of a combination of multiple V3C units.
[0026] 3 is a diagram showing an example of the configuration of a V3C unit. A V3C unit may be composed of a V3C unit header and a V3C unit payload.
[0027] 4 is a diagram showing an example of the syntax configuration of a V3C unit header. The V3C unit header may include vuh_unit_type, which indicates the type of the V3C unit.
[0028] 5 is a diagram showing an example of the configuration of a bitstream. The bitstream may be composed of a V3C parameter set, an atlas bitstream, a base mesh bitstream, a displacement bitstream, and a texture bitstream.
[0029] vuh_unit_type may store any of the following: V3C_VPS indicating a V3C parameter set, V3C_AD indicating an atlas bitstream, V3C_BMD indicating a basic mesh bitstream, V3C_GVD indicating a displacement bitstream, or V3C_AVD indicating a texture bitstream.
[0030] (Structure of displacement bitstream) 6 is a diagram showing an example of the configuration of a displacement amount bitstream. The displacement amount bitstream may have a V3C unit payload configured with displ_nal_unit indicating an NAL unit.
[0031] The displ_nal_unit may be composed of a displ_nal_unit_header indicating the header of the NAL unit in the displacement bitstream and the payload of the NAL unit in the displacement bitstream.
[0032] FIG. 7 is a diagram illustrating an example of the syntax configuration of displ_nal_unit_header.
[0033] The displ_nal_unit_header may be composed of a displ_nal_forbidden_zero_bit indicating the leading write-protected section and a displ_nal_unit_type indicating the type of data in the payload.
[0034] FIG. 8 is a diagram showing an example of the correspondence between displ_nal_unit_type and the contents of the payload.
[0035] The contents of the payload may be defined as displ_sequence_parameter_set_rbsp indicating the sequence parameter set of the displacement amount when displ_nal_unit_type is 0, displ_frame_parameter_set_rbsp indicating the frame parameter set of the displacement amount when displ_nal_unit_type is 1, and displ_layer_rbsp indicating the displacement amount data when displ_nal_unit_type is 2.
[0036] 9 is a diagram showing an example of the syntax configuration of displ_layer_rbsp. displ_layer_rbsp may include displ_header indicating a displacement amount header and displ_data_unit indicating displacement amount data.
[0037] FIG. 10 is a diagram illustrating an example of the syntax configuration of displ_data_unit.
[0038] The displ_data_unit may include a ddu_lod_count that indicates the number of subdivisions.
[0039] The displ_data_unit may include ddu_vertex_count_lod, which indicates the number of displacements.
[0040] The displ_data_unit may include ddu_num_subblock_lod, which indicates the number of subblocks.
[0041] The displ_data_unit may include a ddu_nz_subBlock that indicates whether the subblock has at least one non-zero level value.
[0042] The displ_data_unit may include ddu_coeff_abs_level_gt0, which indicates whether the absolute value of the level value is 1 or greater.
[0043] The displ_data_unit may include ddu_coeff_abs_level_gt1, which indicates whether the absolute value of the level value is 2 or greater.
[0044] The displ_data_unit may include ddu_coeff_abs_level_gtn, which indicates whether the absolute value of the level value is greater than or equal to n.
[0045] The displ_data_unit may include a ddu_coeff_sign that indicates the sign of the level value.
[0046] The displ_data_unit may include ddu_coeff_abs_level_rem, which indicates the absolute value of the level value.
[0047] (Displacement amount decoding unit 204) Fig. 11 is a diagram showing an example of the configuration of the displacement decoding unit 204. As shown in Fig. 11, the displacement decoding unit 204 includes a video decoding unit 204A, an arithmetic decoding unit 204B, a first inverse quantization unit 204C, an adder 204D, an inter prediction unit 204E, and a frame buffer 204F.
[0048] The video decoding unit 204A is configured to decode the level value image from the displacement amount bitstream using a video codec and output the level value image. The arithmetic decoding unit 204B is configured to decode the displacement amount bitstream into a level value sequence by arithmetic decoding and output the level value sequence. Here, the arithmetic decoding unit 204B first initializes all level values to 0.
[0049] When ddu_nz_subBlock=0, the arithmetic decoding unit 204B omits determining the level value by arithmetic decoding. Details will be described later in "Processing of the arithmetic decoding unit." This makes it possible to reduce the amount of calculation required for decoding the level value.
[0050] The first inverse quantization unit 204C is configured to generate and output a first coefficient sequence by inverse quantizing the level value sequence.
[0051] The adder 204D is configured to obtain a prediction coefficient sequence from the inter prediction unit 204E and a first coefficient sequence (actually, a prediction residual sequence) from the first inverse quantization unit 204C, and add them together to generate a coefficient sequence, which is output to the frame buffer 204F.
[0052] The inter prediction unit 204E is configured to read a reference frame indicated by the reference list from the frame buffer 204F, and perform inter prediction using the first coefficient sequence of the reference frame, thereby generating and outputting a prediction coefficient sequence.
[0053] The inter prediction unit 204E may determine prediction coefficients for each frequency in the current frame by directly referencing the coefficients for the corresponding frequency in the reference frame.
[0054] The frame buffer 204F is configured to acquire and store the prediction residual sequence or the prediction coefficient sequence from the first inverse quantization unit 204C or the adder 204D, respectively.
[0055] The frame buffer 204F is configured to output the first coefficient sequence of the reference frame in accordance with control information (not shown).
[0056] (Mesh reconstruction unit 206) Fig. 12 is a diagram showing an example of the configuration of the mesh reconstructing unit 206. As shown in Fig. 12, the mesh reconstructing unit 206 includes a submesh reconstructing unit 206A and a submesh combining unit 206B.
[0057] The submesh reconstructing unit 206A is configured to generate and output a submesh sequence using the basic mesh and the level value image or the first coefficient sequence.
[0058] The submesh combining unit 206B is configured to combine the submesh rows to generate and output a single mesh.
[0059] Fig. 13 is a diagram showing an example of the configuration of the submesh reconstructing unit 206 A. As shown in Fig. 13, the submesh reconstructing unit 206 A includes a subdivision unit 206 A1, a displacement decoding method determining unit 206 A2, a displacement sequence generating unit 206 A3, a level value image processing unit 206 A4, a coefficient sequence processing unit 206 A5, an image expanding unit 206 A6, a second inverse quantization unit 206 A7, an inverse transform unit 206 A8, and a displacement adding unit 206 A9.
[0060] The subdivision unit 206A1 is configured to generate the added subdivision vertices and their connection information from the matching base mesh according to the subdivision method indicated by the control information.
[0061] The subdivision unit 206A1 outputs a subdivision mesh that combines the basic mesh, the added subdivision vertices, and their connection information.
[0062] The displacement amount decoding method determining unit 206A2 is configured to determine a method for decoding the displacement amount from the information indicated by the control information, and output a level value image or a first coefficient sequence.
[0063] The displacement amount sequence generation unit 206A3 is configured to generate an N-dimensional zero vector using the number N of displacement amounts indicated by the control information, thereby generating and outputting a displacement amount sequence.
[0064] The level value image processing unit 206A4 is configured to process the level value image to generate and output a processed level value image.
[0065] The coefficient sequence processing unit 206A5 is configured to process the first coefficient sequence to generate and output a processed coefficient sequence.
[0066] The image development unit 206A6 is configured to develop the processing level value image to generate and output a level value sequence.
[0067] The second inverse quantization unit 206A7 is configured to inversely quantize the level value sequence to generate and output a second coefficient sequence.
[0068] The inverse transform unit 206A8 is configured to generate and output a displacement sequence by inversely transforming the second coefficient sequence or the processed coefficient sequence. Here, the inverse transform unit 206A8 may inversely transform the processed coefficient sequence using an inverse DCT or an inverse wavelet transform.
[0069] The displacement amount adding unit 206A9 is configured to generate and output sub-meshes by adding the displacement amount sequence to the finely divided mesh.
[0070] The above-described configuration of the submesh reconstructing unit 206A determines the method for decoding the amount of displacement, and performs appropriate branching and processing accordingly, thereby enabling correct reconstruction of the mesh.
[0071] (Processing of the arithmetic decoding unit 204B) FIG. 14 is a flowchart showing an example of the process of the arithmetic decoding unit 204B.
[0072] As shown in Fig. 14, in step S101, the arithmetic decoding unit 204B initializes all level values to 0. This makes it possible to reduce the amount of calculation required to decode the level values in the subsequent steps.
[0073] In step S102, the arithmetic decoding unit 204B decodes the ddu_nz_subBlock.
[0074] In step S103, the arithmetic decoding unit 204B determines whether ddu_nz_subBlock is 0 or 1.
[0075] If ddu_nz_subBlock=0, the operation proceeds to step S111, and if ddu_nz_subBlock=1, the operation proceeds to step S107.
[0076] In step S104, the arithmetic decoding unit 204B determines the level value.
[0077] In step S105, the arithmetic decoding unit 204B determines whether or not the level value currently being processed is the last in the sub-block.
[0078] If yes, the operation proceeds to step S107, and if no, the operation proceeds to step S106.
[0079] In step S106, the arithmetic decoding unit 204B proceeds to process the next level value.
[0080] In step S107, the arithmetic decoding unit 204B determines whether the sub-block currently being processed is the final sub-block.
[0081] If yes, the operation proceeds to step S106, and if no, the operation proceeds to step S113.
[0082] In step S108, the arithmetic decoding unit 204B proceeds to process the next sub-block.
[0083] In step S109, the arithmetic decoding unit 204B determines whether the block currently being processed is the final block.
[0084] If yes, the operation proceeds to step S111, and if no, the operation proceeds to step S110.
[0085] In step S110, the arithmetic decoding unit 204B proceeds to processing the next block.
[0086] In step S111, the arithmetic decoding unit 204B determines whether the dimension currently being processed is the final dimension.
[0087] If Yes, this operation ends, and if No, this operation proceeds to step S107.
[0088] In step S112, the arithmetic decoding unit 204B proceeds to processing of the next dimension.
[0089] (Processing of the submesh reconstruction unit 206A) FIG. 15 is a flowchart showing an example of the processing of the submesh reconstruction unit 206A.
[0090] As shown in FIG. 15, in step S201, the submesh reconstructing unit 206A determines the displacement amount decoding method.
[0091] If the displacement amount decoding method is no displacement amount, this operation proceeds to step S202, if the displacement amount decoding method is video codec, this operation proceeds to step S203, and if the displacement amount decoding method is arithmetic decoding, this operation proceeds to step S204.
[0092] In step S202, the submesh reconstructing unit 206A generates a coefficient sequence.
[0093] In step S203, the submesh reconstructing unit 206A processes the level value image.
[0094] In step S204, the submesh reconstructing unit 206A processes the coefficient sequence.
[0095] In step S205, the submesh reconstructing unit 206A performs image expansion.
[0096] In step S206, the submesh reconstructing unit 206A performs inverse quantization.
[0097] In step S207, the submesh reconstructing unit 206A adds the displacement amounts, and then this operation ends.
[0098] According to this embodiment, when decoding the amount of displacement using arithmetic decoding, all level values are first initialized to 0, thereby making it possible to reduce the amount of calculation required to decode the level values.
[0099] According to this embodiment, the method for decoding the displacement amount is determined when reconstructing the mesh, and appropriate branching and processing is performed accordingly, thereby enabling the mesh to be reconstructed correctly.
[0100] The mesh encoding device 100 and the mesh decoding device 200 described above may be realized as a program that causes a computer to execute each function (each step). [Industrial Applicability]
[0101] According to this embodiment, for example, it is possible to improve the overall service quality in video communication, which will contribute to the achievement of Goal 9 of the United Nations-led Sustainable Development Goals (SDGs), which is to "Develop resilient infrastructure, promote sustainable industrialization and foster innovation." [Explanation of symbols]
[0102] 1...Mesh processing system 100...Mesh coding device 200...Mesh decoding device 201...V3C unit separation section 202...Atlas decoding unit 203...Basic mesh decoding unit 204...Displacement amount decoding unit 204A...Video decoding unit 204B...Arithmetic decoding unit 204C...first inverse quantization section 204D...adder 204E…Inter prediction section 204F...frame buffer 205...Texture decoding unit 206...Mesh reconstruction unit 206A...Submesh reconstruction section 206A...Submesh joint 206A1…Subdivision section 206A2...Displacement amount decoding method determination unit 206A3...Displacement sequence generation unit 206A4...Level value image processing section 206A5…Coefficient row processing section 206A6...Image development section 206A7…Second inverse quantization unit 206A8…Inverse conversion unit 206A9...Displacement amount adding unit
Claims
1. A mesh decoding device, comprising: a mesh reconstruction unit configured to reconstruct and output a mesh using the atlas, basic mesh, and level value image or the first coefficient sequence obtained from the bitstream; The mesh reconstruction unit a submesh reconstruction unit configured to generate and output a submesh sequence using the atlas, the basic mesh, and the level value image or the first coefficient sequence; a submesh combining unit configured to combine the submesh rows to generate and output a single mesh, The mesh decoding device is characterized in that the submesh reconstruction unit includes a displacement amount decoding method determination unit configured to output the level value image or the first coefficient sequence by determining a displacement amount decoding method from information indicated by control information.
2. The submesh reconstruction unit a level value image processing unit configured to process the level value image to generate and output a processed level value image; 2. The mesh decoding device according to claim 1, further comprising: a coefficient sequence processing unit configured to process the first coefficient sequence to generate and output a processed coefficient sequence.
3. 1. A mesh decoding method, comprising: a step configured to reconstruct and output a mesh using the atlas, basic mesh, and level value image or the first coefficient sequence obtained from the bitstream; The process comprises: a step A of generating and outputting a submesh sequence using the atlas, the basic mesh, and the level value image or the first coefficient sequence; and step B of generating and outputting a single mesh by combining the sub-mesh rows, A mesh decoding method, characterized in that in the step A, the level value image or the first coefficient sequence is output by determining a decoding method for the amount of displacement from information indicated by control information.
4. A program that causes a computer to function as a mesh decoding device, The mesh decoding device comprises: a mesh reconstruction unit configured to reconstruct and output a mesh using the atlas, basic mesh, and level value image or the first coefficient sequence obtained from the bitstream; The mesh reconstruction unit a submesh reconstruction unit configured to generate and output a submesh sequence using the atlas, the basic mesh, and the level value image or the first coefficient sequence; a submesh combining unit configured to combine the submesh rows to generate and output a single mesh, the submesh reconstruction unit includes a displacement amount decoding method determination unit configured to output the level value image or the first coefficient sequence by determining a displacement amount decoding method from information indicated by control information.