Mesh decoding device, mesh decoding method, and program

Abstract

The purpose of the present invention is to correctly reconstruct a mesh. A mesh reconstruction unit 206 of a mesh decoding device 200 according to the present invention comprises: a sub-mesh reconstruction unit 206A constituted to generate and output a sub-mesh column using an atlas, a basic mesh, and a level value image or a first coefficient sequence; and a sub-mesh connection unit 206B constituted to generate and output a single mesh by combining the sub-mesh columns. The sub-mesh reconstruction unit 206A includes a displacement amount decoding method determination unit 206A2 constituted to output a level value image or a first coefficient sequence by determining a displacement amount decoding method from information indicated by control information.

Description

Mesh decoding device, mesh decoding method and program

[0001] The present invention relates to a mesh decoding device, a mesh decoding method, and a program.

[0002] Non-Patent Document 1 discloses a technique for decoding a point group using video information and additional information called an atlas.

[0003] Non-Patent Document 2 discloses a technique that extends Non-Patent Document 1 to mesh decoding applications.

[0004] Specifically, Non-Patent Document 2 first decodes a rough base mesh and a detailed displacement, and secondly, reconstructs a mesh from both. There are two types of displacement decoding methods: a method using a video codec and a method using arithmetic decoding. Furthermore, the decoding of the displacement may be omitted. In the method using arithmetic decoding, the level value is decoded from the syntax, and then the displacement is decoded.

[0005] “Information technology - Coded Representation of Immersive Media - Part 5: Visual Volumetric Video-based Coding (V3C) and Video-based Point Cloud Compression (V-PCC),” ISO / IEC JTC 1 / SC 29 / WG 7, ISO / IEC 23090-5:2021(2E). “WD 6.0 of V-DMC,” ISO / IEC JTC 1 / SC 29 / WG 7 N00822.

[0006] However, the method of Non-Patent Document 2 does not support multiple methods of decoding displacement amounts when reconstructing a mesh, and therefore has a problem in that the mesh cannot be reconstructed correctly.

[0007] Therefore, the present invention has been made in consideration of the above-mentioned problems, and an object of the present invention is to provide a mesh decoding device, a mesh decoding method, and a program that can correctly reconstruct meshes.

[0008] A first feature of the present invention is summarized as a mesh decoding device comprising: a mesh reconstruction unit configured to reconstruct and output a mesh using an atlas, a basic mesh, and a level value image or a first coefficient sequence obtained from a bitstream; the mesh reconstruction unit comprising: a submesh reconstruction unit configured to generate and output a submesh sequence using the atlas, the basic mesh, and the level value image or the first coefficient sequence; and a submesh combining unit configured to generate and output a single mesh by combining the submesh sequences; and the submesh reconstruction unit comprises a displacement amount decoding method determination unit configured to output the level value image or the first coefficient sequence by determining a displacement amount decoding method from information indicated by control information.

[0009] A second feature of the present invention is a mesh decoding method comprising steps of reconstructing and outputting a mesh using an atlas obtained from a bitstream, a basic mesh, and a level value image or a first coefficient sequence, the steps comprising: step A of generating and outputting a sub-mesh sequence using the atlas, the basic mesh, and the level value image or the first coefficient sequence; and step B of generating and outputting a single mesh by combining the sub-mesh sequences, wherein in step A, the level value image or the first coefficient sequence is output by determining a decoding method for a displacement amount from information indicated by control information.

[0010] A third feature of the present invention is summarized as a program causing a computer to function as a mesh decoding device, the mesh decoding device comprising: a mesh reconstruction unit configured to reconstruct and output a mesh using an atlas, a basic mesh, and a level value image or a first coefficient sequence obtained from a bitstream; the mesh reconstruction unit comprising: a submesh reconstruction unit configured to generate and output a submesh sequence using the atlas, the basic mesh, and the level value image or the first coefficient sequence; and a submesh combining unit configured to generate and output a single mesh by combining the submesh sequences; and the submesh reconstruction unit comprising a displacement amount decoding method determination unit configured to output the level value image or the first coefficient sequence by determining a displacement amount decoding method from information indicated by control information.

[0011] According to the present invention, it is possible to provide a mesh decoding device, a mesh decoding method and a program that can correctly reconstruct a mesh.

[0012] FIG. 1 is a diagram showing an example of the configuration of a mesh processing system 1 according to an embodiment. FIG. 2 is a diagram showing an example of functional blocks of a mesh decoding device 200 according to an embodiment. FIG. 3 is a diagram showing an example of the configuration of a V3C unit. FIG. 4 is a diagram showing an example of the syntax configuration of a V3C unit header. FIG. 5 is a diagram showing an example of the configuration of a bit stream. FIG. 6 is a diagram showing an example of the configuration of a displacement bit stream. FIG. 7 is a diagram showing an example of the syntax configuration of displ_nal_unit_header. FIG. 8 is a diagram showing an example of the correspondence between displ_nal_unit_type and the contents of the payload. FIG. 9 is a diagram showing an example of the syntax configuration of displ_layer_rbsp. FIG. 10 is a diagram showing an example of the syntax configuration of displ_layer_rbsp. FIG. 11 is a diagram showing an example of functional blocks of a displacement decoding unit 204 of a mesh decoding device 200 according to an embodiment. Fig. 12 is a diagram showing an example of functional blocks of the mesh reconstruction unit 206 of the mesh decoding device 200 according to an embodiment. Fig. 13 is a diagram showing an example of the configuration of the submesh reconstruction unit 206A of the mesh reconstruction unit 206 of the mesh decoding device 200 according to an embodiment. Fig. 14 is a flowchart showing an example of the processing of the arithmetic decoding unit 204B of the displacement amount decoding unit 204 of the mesh decoding device 200 according to an embodiment. Fig. 15 is a flowchart showing an example of the processing of the submesh reconstruction unit 206A of the mesh reconstruction unit 206 of the mesh decoding device 200 according to an embodiment.

[0013] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the components in the following embodiments can be appropriately replaced with existing components, etc., and various variations, including combinations with other existing components, are possible. Therefore, the description of the following embodiments does not limit the content of the invention described in the claims.

[0014] First Embodiment This embodiment is based on V3C described in Non-Patent Document 1. An atlas in V3C is a collection of rectangular regions and related information arranged on a frame, and corresponds to a three-dimensional space.

[0015] Hereinafter, a mesh processing system 1 according to this embodiment will be described with reference to FIGS.

[0016] 1 is a diagram showing an example of the configuration of a mesh processing system 1 according to this embodiment. As shown in FIG. 1, the mesh processing system 1 includes a mesh encoding device 100 and a mesh decoding device 200.

[0017] FIG. 2 is a diagram showing an example of functional blocks of a mesh decoding device 200 according to this embodiment.

[0018] As shown in FIG. 2, the mesh decoding device 200 includes a V3C unit separation unit 201, an atlas decoding unit 202, a basic mesh decoding unit 203, a displacement amount decoding unit 204, a texture decoding unit 205, and a mesh reconstruction unit 206.

[0019] The V3C unit separation unit 201 is configured to separate the bitstream into an atlas bitstream, a basic mesh bitstream, a displacement bitstream, and a texture bitstream.

[0020] The atlas decoding unit 202 is configured to decode and output the atlas from the atlas bitstream.

[0021] The base mesh decoding unit 203 is configured to decode and output base meshes from the base mesh bitstream.

[0022] The displacement amount decoding unit 204 is configured to decode and output the level value image or the first coefficient sequence from the displacement amount bit stream.

[0023] The texture decoding unit 205 is configured to decode and output the texture from the texture bitstream. The texture decoding unit 205 may decode the texture using HEVC.

[0024] The mesh reconstruction unit 206 is configured to reconstruct and output a mesh using the atlas decoded by the atlas decoding unit 201, the basic mesh decoded by the basic mesh decoding unit 203, and the level value image or first coefficient sequence decoded by the displacement amount decoding unit 204.

[0025] (Bitstream Structure) A bitstream may be formed by combining a plurality of V3C units.

[0026] 3 is a diagram showing an example of the configuration of a V3C unit. A V3C unit may be composed of a V3C unit header and a V3C unit payload.

[0027] 4 is a diagram showing an example of the syntax configuration of a V3C unit header. The V3C unit header may include vuh_unit_type, which indicates the type of the V3C unit.

[0028] 5 is a diagram showing an example of the configuration of a bitstream. The bitstream may be composed of a V3C parameter set, an atlas bitstream, a base mesh bitstream, a displacement bitstream, and a texture bitstream.

[0029] vuh_unit_type may store any of the following: V3C_VPS indicating a V3C parameter set, V3C_AD indicating an atlas bitstream, V3C_BMD indicating a basic mesh bitstream, V3C_GVD indicating a displacement bitstream, and V3C_AVD indicating a texture bitstream.

[0030] (Structure of Displacement Amount Bitstream) Fig. 6 is a diagram showing an example of the structure of a displacement amount bitstream. The displacement amount bitstream may have a V3C unit payload configured with displ_nal_unit indicating a NAL unit.

[0031] The displ_nal_unit may be composed of a displ_nal_unit_header indicating the header of the NAL unit in the displacement amount bitstream and the payload of the NAL unit in the displacement amount bitstream.

[0032] FIG. 7 is a diagram illustrating an example of the syntax configuration of the display_nal_unit_header.

[0033] The display_nal_unit_header may be composed of a display_nal_forbidden_zero_bit indicating the leading write-protected section and a display_nal_unit_type indicating the type of payload data.

[0034] FIG. 8 is a diagram showing an example of the correspondence between the display_nal_unit_type and the contents of the payload.

[0035] The contents of the payload may be defined as displ_sequence_parameter_set_rbsp indicating the sequence parameter set of the displacement amount when displ_nal_unit_type is 0, as displ_frame_parameter_set_rbsp indicating the frame parameter set of the displacement amount when displ_nal_unit_type is 1, and as displ_layer_rbsp indicating the displacement amount data when displ_nal_unit_type is 2.

[0036] 9 is a diagram showing an example of the syntax configuration of display_layer_rbsp. display_layer_rbsp may include display_header indicating a displacement amount header and display_data_unit indicating displacement amount data.

[0037] FIG. 10 is a diagram illustrating an example of the syntax configuration of the display_data_unit.

[0038] The display_data_unit may include a ddu_lod_count that indicates the number of subdivisions.

[0039] The display_data_unit may include ddu_vertex_count_lod, which indicates the number of displacements.

[0040] The display_data_unit may include ddu_num_subblock_lod, which indicates the number of subblocks.

[0041] The display_data_unit may include a ddu_nz_subBlock that indicates whether the subblock has at least one non-zero level value.

[0042] The display_data_unit may include ddu_coeff_abs_level_gt0, which indicates whether the absolute value of the level value is 1 or greater.

[0043] The display_data_unit may include ddu_coeff_abs_level_gt1, which indicates whether the absolute value of the level value is 2 or greater.

[0044] The display_data_unit may include ddu_coeff_abs_level_gtn, which indicates whether the absolute value of the level value is greater than or equal to n.

[0045] The display_data_unit may include a ddu_coeff_sign that indicates the sign of the level value.

[0046] The display_data_unit may include ddu_coeff_abs_level_rem, which indicates the absolute value of the level value.

[0047] (Displacement Amount Decoding Unit 204) Fig. 11 is a diagram showing an example of the configuration of the displacement amount decoding unit 204. As shown in Fig. 11, the displacement amount decoding unit 204 includes a video decoding unit 204A, an arithmetic decoding unit 204B, a first inverse quantization unit 204C, an adder 204D, an inter prediction unit 204E, and a frame buffer 204F.

[0048] The video decoding unit 204A is configured to decode and output a level value image from the displacement amount bit stream using a video codec. The arithmetic decoding unit 204B is configured to decode and output a level value sequence from the displacement amount bit stream using arithmetic decoding. Here, the arithmetic decoding unit 204B first initializes all level values ​​to 0.

[0049] When ddu_nz_subBlock=0, the arithmetic decoding unit 204B omits determining the level value by arithmetic decoding. Details will be described later in "Processing of the arithmetic decoding unit." This makes it possible to reduce the amount of calculation required for decoding the level value.

[0050] The first inverse quantization unit 204C is configured to inversely quantize the level value sequence to generate and output a first coefficient sequence.

[0051] The adder 204D is configured to receive the prediction coefficient sequence from the inter prediction unit 204E and the first coefficient sequence (actually, the prediction residual sequence) from the first inverse quantization unit 204C, and add them together to generate a coefficient sequence, which is output to the frame buffer 204F.

[0052] The inter prediction unit 204E is configured to read a reference frame indicated by the reference list from the frame buffer 204F, and perform inter prediction using a first coefficient sequence of the reference frame, thereby generating and outputting a prediction coefficient sequence.

[0053] The inter prediction unit 204E may determine prediction coefficients for each frequency in the current frame by directly referencing the coefficients for the corresponding frequency in the reference frame.

[0054] The frame buffer 204F is configured to acquire and store the prediction residual sequence or the prediction coefficient sequence from the first inverse quantization unit 204C or the adder 204D, respectively.

[0055] The frame buffer 204F is configured to output a first coefficient sequence of a reference frame in accordance with control information (not shown).

[0056] (Mesh Reconstruction Unit 206) Fig. 12 is a diagram showing an example of the configuration of the mesh reconstruction unit 206. As shown in Fig. 12, the mesh reconstruction unit 206 includes a submesh reconstruction unit 206A and a submesh connection unit 206B.

[0057] The submesh reconstructing unit 206A is configured to generate and output a submesh sequence using the basic mesh and the level value image or the first coefficient sequence.

[0058] The submesh combining unit 206B is configured to combine the submesh rows to generate and output a single mesh.

[0059] Fig. 13 is a diagram showing an example of the configuration of the submesh reconstructing unit 206 A. As shown in Fig. 13, the submesh reconstructing unit 206 A includes a subdivision unit 206 A1, a displacement decoding method determining unit 206 A2, a displacement sequence generating unit 206 A3, a level value image processing unit 206 A4, a coefficient sequence processing unit 206 A5, an image expanding unit 206 A6, a second inverse quantization unit 206 A7, an inverse transform unit 206 A8, and a displacement adding unit 206 A9.

[0060] The subdivision unit 206A1 is configured to generate the added subdivision vertices and their connection information from the matching base mesh according to the subdivision method indicated by the control information.

[0061] The subdivision unit 206A1 outputs a subdivision mesh that combines the basic mesh, the added subdivision vertices, and their connection information.

[0062] The displacement amount decoding method determining unit 206A2 is configured to determine the method of decoding the displacement amount from the information indicated by the control information, and output the level value image or the first coefficient sequence.

[0063] The displacement amount sequence generator 206A3 is configured to generate an N-dimensional zero vector using the number N of displacement amounts indicated by the control information, thereby generating and outputting a displacement amount sequence.

[0064] The level value image processing unit 206A4 is configured to process the level value image to generate and output a processed level value image.

[0065] The coefficient sequence processing unit 206A5 is configured to process the first coefficient sequence to generate and output a processed coefficient sequence.

[0066] The image development unit 206A6 is configured to develop the processing level value image to generate and output a level value sequence.

[0067] The second inverse quantization unit 206A7 is configured to inversely quantize the level value sequence to generate and output a second coefficient sequence.

[0068] The inverse transform unit 206A8 is configured to generate and output a displacement sequence by inversely transforming the second coefficient sequence or the processed coefficient sequence. Here, the inverse transform unit 206A8 may inversely transform the processed coefficient sequence using an inverse DCT or an inverse wavelet transform.

[0069] The displacement amount adding unit 206A9 is configured to generate and output sub-meshes by adding a sequence of displacement amounts to the finely divided meshes.

[0070] The above-described configuration of the submesh reconstructing unit 206A determines the method for decoding the displacement amount, and performs appropriate branching and processing accordingly, thereby enabling correct mesh reconstruction.

[0071] (Processing of Arithmetic Decoding Unit 204B) FIG. 14 is a flowchart showing an example of processing of the arithmetic decoding unit 204B.

[0072] 14, in step S101, the arithmetic decoding unit 204B initializes all level values ​​to 0. This makes it possible to reduce the amount of calculation required to decode the level values ​​in the subsequent steps.

[0073] In step S102, the arithmetic decoding unit 204B decodes the ddu_nz_subBlock.

[0074] In step S103, the arithmetic decoding unit 204B determines whether ddu_nz_subBlock is 0 or 1.

[0075] If ddu_nz_subBlock=0, the operation proceeds to step S111, and if ddu_nz_subBlock=1, the operation proceeds to step S107.

[0076] In step S104, the arithmetic decoding unit 204B determines a level value.

[0077] In step S105, the arithmetic decoding unit 204B determines whether the level value currently being processed is the last in the sub-block.

[0078] If yes, the operation proceeds to step S107, and if no, the operation proceeds to step S106.

[0079] In step S106, the arithmetic decoding unit 204B proceeds to processing the next level value.

[0080] In step S107, the arithmetic decoding unit 204B determines whether the sub-block currently being processed is the final sub-block.

[0081] If the answer is Yes, the operation proceeds to step S106, and if the answer is No, the operation proceeds to step S113.

[0082] In step S108, the arithmetic decoding unit 204B proceeds to process the next sub-block.

[0083] In step S109, the arithmetic decoding unit 204B determines whether the block currently being processed is the final block.

[0084] If yes, the operation proceeds to step S111, and if no, the operation proceeds to step S110.

[0085] In step S110, the arithmetic decoding unit 204B proceeds to process the next block.

[0086] In step S111, the arithmetic decoding unit 204B determines whether the dimension currently being processed is the final dimension.

[0087] If yes, this operation ends, and if no, this operation proceeds to step S107.

[0088] In step S112, the arithmetic decoding unit 204B proceeds to processing of the next dimension.

[0089] (Processing of the Submesh Reconstruction Unit 206A) FIG. 15 is a flowchart showing an example of processing of the submesh reconstruction unit 206A.

[0090] As shown in FIG. 15, in step S201, the submesh reconstructing unit 206A determines the displacement decoding method.

[0091] If the displacement amount decoding method is no displacement amount, this operation proceeds to step S202, if the displacement amount decoding method is video codec, this operation proceeds to step S203, and if the displacement amount decoding method is arithmetic decoding, this operation proceeds to step S204.

[0092] In step S202, the submesh reconstruction unit 206A generates a coefficient sequence.

[0093] In step S203, the submesh reconstructing unit 206A processes the level value image.

[0094] In step S204, the submesh reconstructing unit 206A processes the coefficient sequence.

[0095] In step S205, the submesh reconstruction unit 206A performs image expansion.

[0096] In step S206, the submesh reconstructing unit 206A performs inverse quantization.

[0097] In step S207, the submesh reconstructing unit 206A adds the displacement amounts, and then this operation ends.

[0098] According to this embodiment, when decoding the amount of displacement using arithmetic decoding, all level values ​​are first initialized to 0, thereby making it possible to reduce the amount of calculation required to decode the level values.

[0099] According to this embodiment, the method for decoding the displacement amount is determined when reconstructing the mesh, and appropriate branching and processing is performed accordingly, thereby enabling the mesh to be reconstructed correctly.

[0100] The mesh encoding device 100 and mesh decoding device 200 described above may be realized as a program that causes a computer to execute each function (each step).

[0101] According to this embodiment, for example, it is possible to improve the overall service quality in video communication, which makes it possible to contribute to Goal 9 of the Sustainable Development Goals (SDGs) led by the United Nations, which is to "Develop resilient infrastructure, promote sustainable industrialization and foster innovation."

[0102] 1...Mesh processing system 100...Mesh encoding device 200...Mesh decoding device 201...V3C unit separation unit 202...Atlas decoding unit 203...Basic mesh decoding unit 204...Displacement decoding unit 204A...Video decoding unit 204B...Arithmetic decoding unit 204C...First inverse quantization unit 204D...Adder 204E...Inter prediction unit 204F...Frame buffer 205...Texture decoding unit 206...Mesh reconstruction unit 206A...Submesh reconstruction unit 206A...Submesh combining unit 206A1...Subdivision unit 206A2...Displacement decoding method determination unit 206A3...Displacement sequence generation unit 206A4...Level value image processing unit 206A5...Coefficient sequence processing unit 206A6...Image expansion unit 206A7...Second inverse quantization unit 206A8...Inverse transformation unit 206A9...Displacement amount adding unit

Claims

1. A mesh decoding device comprising: a mesh reconstruction unit configured to reconstruct and output a mesh using an atlas, a basic mesh, and a level value image or a first coefficient sequence obtained from a bitstream, wherein the mesh reconstruction unit comprises: a submesh reconstruction unit configured to generate and output a submesh sequence using the atlas, the basic mesh, and the level value image or the first coefficient sequence; and a submesh combining unit configured to generate and output a single mesh by combining the submesh sequences, wherein the submesh reconstruction unit comprises a displacement amount decoding method determination unit configured to output the level value image or the first coefficient sequence by determining a displacement amount decoding method from information indicated by control information.

2. The mesh decoding device described in claim 1, characterized in that the submesh reconstruction unit comprises: a level value image processing unit configured to process the level value image to generate and output a processed level value image; and a coefficient sequence processing unit configured to process the first coefficient sequence to generate and output a processed coefficient sequence.

3. A mesh decoding method comprising steps configured to reconstruct and output a mesh using an atlas obtained from a bitstream, a basic mesh, and a level value image or a first coefficient sequence, said steps comprising: step A of generating and outputting a sub-mesh sequence using said atlas, said basic mesh, and said level value image or said first coefficient sequence; and step B of generating and outputting a single mesh by combining said sub-mesh sequences, wherein in step A, said level value image or said first coefficient sequence is output by determining a decoding method for the amount of displacement from information indicated by control information.

4. A program that causes a computer to function as a mesh decoding device, wherein the mesh decoding device comprises: a mesh reconstruction unit configured to reconstruct and output a mesh using an atlas, a basic mesh, and a level value image or a first coefficient sequence obtained from a bit stream; the mesh reconstruction unit comprises: a submesh reconstruction unit configured to generate and output a submesh sequence using the atlas, the basic mesh, and the level value image or the first coefficient sequence; and a submesh combining unit configured to generate and output a single mesh by combining the submesh sequences; and the submesh reconstruction unit comprises a displacement amount decoding method determination unit configured to output the level value image or the first coefficient sequence by determining a displacement amount decoding method from information indicated by control information.

Images