Mesh decoding device, mesh decoding method, and program

Abstract

The present invention improves coding efficiency, making it possible to reduce the amount of coding if arithmetic decoding is used. A mesh patch decoding unit 207D of an atlas decoding unit 207 of this mesh decoding device 200 is provided with: a displacement codec determination unit 207Dm1 / 207Di1 for determining whether an amount of displacement is decoding by video codec or is arithmetic decoding; and a geometric video position information decoding unit 207Dm2 / 207Di2 for decoding information for identifying the extent of a patch in a frame of video of the decoded amount of displacement if the amount of displacement was determined to be decoding by video codec. The geometric video position information decoding unit 207Dm2 / 207Di2 does not decode information for identifying the extent of a patch in a frame of video of the decoded amount of displacement if the amount of displacement was determined not to be decoding by video codec.

Description

Mesh decoding device, mesh decoding method and program

[0001] The present invention relates to a mesh decoding device, a mesh decoding method, and a program.

[0002] Non-Patent Document 1 discloses a technique for decoding a point group using video information and additional information called an atlas.

[0003] Non-Patent Document 2 discloses a technique in which a mesh is decomposed into a rough basic mesh and a detailed displacement amount, and the displacement amount is converted into an image and then encoded by a video codec.

[0004] Non-Patent Document 3 discloses specifications for a technique for decoding a displacement amount using a video codec and a technique for arithmetic decoding. Either video codec decoding or arithmetic decoding can be used for decoding the displacement amount. Non-Patent Document 3 also discloses specifications for image packing of the displacement amount.

[0005] "Information technology - Coded Representation of Immersive Media - Part 5: Visual Volumetric Video-based Coding (V3C) and Video-based Point Cloud Compression (V-PCC)," ISO / IEC JTC 1 / SC 29 / WG 7, ISO / IEC 23090-5:2021(2E).Khaled Mammou, Jungsun Kim, Alexis Tourapis, Dimitri Podborski, Krasimir Kolarov, "[V-CG] Apple's Dynamic Mesh Coding CfP Response," ISO / IEC JTC 1 / SC 29 / WG 7 m59281, April 2022."WD 6.0 of V-DMC," ISO / IEC JTC 1 / SC 29 / WG7 w23617, March 2024.Jungsun Kim et.al., “[V-DMC][new] Usage of parameters in the patches in the Atlas Data sub-bitstream, ISO / IEC JTC 1 / SC 20 / WG7 m65852, January 2024.Patrice Rndao Alface et. al., “[V-DMC][EE4.7-related][New] LOD-based displacement frame packing into rectangular regions, ISO / IEC JTC 1 / SC 29 / WG7 m66206, January 2024.

[0006] Here, in the techniques disclosed in Non-Patent Documents 2 and 3, the displacement, which is one of the geometric information, can be coded using video coding by a video codec or arithmetic coding.

[0007] However, when decoding the displacement amount by arithmetic decoding, there is a problem that the amount of code increases. Therefore, the present invention has been made in consideration of the above-mentioned problem, and an object of the present invention is to provide a mesh decoding device, a mesh decoding method, and a program that can reduce the amount of code when using arithmetic decoding, which can improve coding efficiency and can reduce the amount of code when not using a video codec when decoding the displacement amount.

[0008] A first feature of the present invention is a mesh decoding device comprising an atlas decoding unit configured to decode and output an atlas bitstream, the atlas decoding unit comprising a mesh patch decoding unit that decodes patches, the mesh patch decoding unit comprising a displacement amount codec determination unit that determines whether the displacement amount has been decoded using a video codec, and a geometric video position information decoding unit that, when it is determined that the displacement amount has been decoded using the video codec, decodes information for specifying the range of existence of the patch in a frame of video of the decoded displacement amount, and when it is determined that the displacement amount has not been decoded using the video codec, the geometric video position information decoding unit does not decode the information for specifying the range of existence of the patch in a frame of video of the decoded displacement amount.

[0009] A second feature of the present invention is a mesh decoding device comprising an atlas decoding unit configured to decode and output an atlas bitstream, the atlas decoding unit comprising a mesh patch decoding unit that decodes patches, the mesh patch decoding unit comprising a displacement amount codec determination unit that determines whether the displacement amount is decoded using a video codec, and a geometric video position information decoding unit that decodes the number of vertices in a video frame of the decoded displacement amount when it is determined that the displacement amount is decoded using the video codec, and the geometric video position information decoding unit does not decode the number of vertices in the video frame of the decoded displacement amount when it is determined that the displacement amount is not decoded using the video codec.

[0010] A third feature of the present invention is summarized as a mesh decoding device including an atlas sequence parameter set decoding unit that decodes an atlas sequence parameter set, wherein the atlas sequence parameter set decoding unit includes a determination unit that determines whether a displacement amount is to be decoded by a video codec, and a geometric video packing method decoding unit that decodes a packing method of the decoded displacement amount when it is determined that the displacement amount is to be decoded by the video codec, and the geometric video packing method decoding unit does not decode the packing method of the decoded displacement amount when it is determined that the displacement amount is not to be decoded by the video codec.

[0011] A fourth feature of the present invention is summarized as a mesh decoding method comprising a step of decoding and outputting an atlas bitstream, the step including a step A of decoding a patch, the step A including a step A1 of determining whether or not a displacement amount has been decoded using a video codec, a step A2 of decoding information for identifying the range of existence of the patch in a frame of video of the decoded displacement amount when it is determined that the displacement amount has been decoded using the video codec, and a step A3 of not decoding information for identifying the range of existence of the patch in a frame of video of the decoded displacement amount when it is determined that the displacement amount has not been decoded using the video codec.

[0012] A fifth feature of the present invention is summarized as a program that causes a computer to function as a mesh decoding device, the mesh decoding device comprising an atlas decoding unit configured to decode and output an atlas bitstream, the atlas decoding unit comprising a mesh patch decoding unit that decodes patches, the mesh patch decoding unit comprising a displacement amount codec determination unit that determines whether the displacement amount has been decoded using a video codec, and a geometric video position information decoding unit that, when it is determined that the displacement amount has been decoded using the video codec, decodes information for specifying the range of existence of the patch in a frame of video of the decoded displacement amount, and when it is determined that the displacement amount has not been decoded using the video codec, does not decode the information for specifying the range of existence of the patch in a frame of video of the decoded displacement amount.

[0013] According to the present invention, it is possible to provide a mesh decoding device, a mesh decoding method and a program that can improve coding efficiency when a video codec is not used when decoding a displacement amount.

[0014] FIG. 1 is a diagram showing an example of the configuration of a mesh processing system 1 according to an embodiment. FIG. 2 is a diagram showing an example of functional blocks of a mesh decoding device 200 according to an embodiment. FIG. 3 is a diagram showing an example of functional blocks of a displacement decoding unit 205 of the mesh decoding device 200 according to an embodiment. FIG. 4 is a diagram showing an example of functional blocks of an atlas decoding unit 207 of the mesh decoding device 200 according to an embodiment. FIG. 5 is a diagram showing an example of the configuration of a bitstream. FIG. 6 is a diagram showing an example of the syntax configuration of profile_tier_level. FIG. 7 is a diagram showing an example of the configuration of a displacement bitstream. FIG. 8 is a diagram showing an example of the syntax configuration of DPS. FIG. 9 is a diagram showing an example of the syntax configuration of DFH. FIG. 10 is a diagram showing an example of an atlas bitstream. FIG. 11 is a diagram showing an example of the syntax configuration of ASVE. FIG. 12 is a diagram showing an example of the syntax configuration of MDU. FIG. 13 is a diagram showing an example of the syntax configuration of ASVE. Fig. 14 is a diagram showing an example of the syntax configuration of an MDU. Fig. 15 is a diagram showing an example of the syntax configuration of an IMDU. Fig. 16 is a flowchart showing an example of the operation of the atlas data decoding unit 207 to decode a mesh patch data unit. Fig. 17 is a flowchart showing an example of the operation of the atlas data decoding unit 207 to decode a mesh patch data unit. Fig. 18 is a flowchart showing an example of the operation of the atlas data decoding unit 207 to decode an ASVE.

[0015] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the components in the following embodiments can be appropriately replaced with existing components, etc., and various variations, including combinations with other existing components, are possible. Therefore, the description of the following embodiments does not limit the content of the invention described in the claims.

[0016] First Embodiment A mesh processing system according to this embodiment will be described below with reference to FIGS.

[0017] 1 is a diagram showing an example of the configuration of a mesh processing system 1 according to this embodiment. As shown in FIG. 1, the mesh processing system 1 includes a mesh encoding device 100 and a mesh decoding device 200.

[0018] FIG. 2 is a diagram showing an example of functional blocks of a mesh decoding device 200 according to this embodiment.

[0019] As shown in Figure 2, the mesh decoding device 200 has a demultiplexing unit 201, a basic mesh decoding unit 202, a subdivision unit 203, a mesh decoding unit 204, a displacement amount decoding unit 205, a video decoding unit 206, and an atlas data decoding unit 207.

[0020] The demultiplexing unit 201 is configured to separate the multiplexed bit stream into a base mesh bit stream, a displacement bit stream, a texture bit stream, and an atlas bit stream.

[0021] The base mesh decoder 202 is configured to decode the base mesh bitstream, generate and output a base mesh.

[0022] The subdivision unit 203 is configured to generate and output added subdivision vertices and their connection information from the basic mesh decoded by the basic mesh decoding unit 202 using the subdivision method indicated by the control information.

[0023] Here, the basic mesh, the added subdivision vertices, and the connection information therebetween are collectively referred to as a "subdivision mesh."

[0024] The mesh decoding unit 204 is configured to generate and output a decoded mesh using the subdivision mesh generated by the subdivision unit 203 and the displacement amount decoded by the displacement amount decoding unit 205.

[0025] The displacement amount decoding unit 205 is configured to decode the displacement amount bitstream to generate and output the displacement amount.

[0026] The video decoding unit 206 is configured to decode the texture bitstream, which is attribute information, using a video codec to generate texture. For example, the video decoding unit 206 may use HEVC, as described in Non-Patent Document 1. The atlas data decoding unit 207 is configured to decode the atlas bitstream and output various parameters related to the atlas to the base mesh decoding unit 202, the displacement amount decoding unit 205, and the video decoding unit 206.

[0027] (Displacement Amount Decoding Unit 205) Fig. 3 is a diagram showing an example of the configuration of the displacement amount decoding unit 205. As shown in Fig. 3, the displacement amount decoding unit 205 includes an arithmetic decoding unit 205A, a video codec decoding unit 205B, an image expansion unit 205C, an inverse quantization unit 205D, and an inverse transform unit 205H.

[0028] The arithmetic decoding unit 205A is configured to generate coefficient level values ​​by performing arithmetic decoding on the received displacement bitstream.

[0029] The video codec decoding unit 205B is configured to generate an image by decoding the received displacement bitstream using a video codec.

[0030] The image development unit 205C is configured to develop an image to generate coefficient level values.

[0031] The inverse quantization unit 205D is configured to generate transform coefficients by performing inverse quantization on the received coefficient level values.

[0032] The inverse quantization unit 205D is a common functional block whether the video codec decoding unit 205B or the arithmetic decoding unit 205A is used.

[0033] The inverse transform unit 205H is configured to generate a decoded displacement by performing an inverse transform on the received transform coefficients. The inverse transform may be, for example, an inverse DCT or an inverse wavelet transform.

[0034] (Atlas Data Decoding Unit 207) Fig. 4(a) is a diagram showing an example of the configuration of the atlas data decoding unit 207. As shown in Fig. 4(a), the atlas data decoding unit 207 includes an atlas sequence parameter set decoding unit 207A, an atlas frame parameter set decoding unit 207B, an atlas tile decoding unit 207C, and a mesh patch decoding unit 207D.

[0035] The atlas sequence parameter set decoding unit 207A is configured to decode the atlas sequence parameter set (ASPS) and its extension ASVE (Atlas Sequence Parameter Set V-DMC Extension) for 3D mesh compression.

[0036] The atlas frame parameter set decoding unit 207B is configured to decode the AFPS (Atlas Sequence Parameter Set) and its extension AFVE (Atlas Frame Parameter Set V-DMC Extension) for 3D mesh compression.

[0037] The atlas tile decoder 207C is configured to decode various information about the tiles.

[0038] The mesh patch decoder 207D is configured to decode various information related to patches within a tile, including MDUs (Mesh Patch Data Units) and IMDUs (Inter Mesh Patch Data Units).

[0039] (ASVE Decoding Unit 207A1) FIG. 4A is a diagram showing an example of the configuration of the ASVE decoding unit 207A1.

[0040] The ASVE decoding unit 207A1 decodes ASVE parameters using a displacement codec determination unit 207A1a, a geometric video packing method decoding unit 207A1b, and an ASVE other syntax decoding unit 207A1c.

[0041] The geometric video packing method decoding unit 207A1b decodes parameters that are required only when the displacement video codec is used.

[0042] For example, the geometric video packing method decoding unit 207A1b may decode asve_packing_method, which is an image packing method.

[0043] The other syntax decoder 207A1c in ASVE decodes parameters that are commonly used when the displacement amount video codec is used or parameters that are not required when the displacement amount video codec is used.

[0044] For example, other syntax decoding units 207A1c within ASVE may decode asve_subdivision_method indicating the subdivision method, asve_subdivision_iteration_count indicating the number of subdivision iterations, asve_1d_displacement_flag indicating the 1D displacement amount flag, asve_displacement_reference_qp indicating the displacement amount QP, and asve_num_attribute_video indicating the number of videos of the attribute.

[0045] (Mesh Patch Decoding Unit 207D) FIG. 4B is a diagram showing an example of the configuration of the mesh patch decoding unit 207D.

[0046] The mesh patch decoding unit 207D obtains decoded patch data by a patch mode determination unit 207D1, an MDU decoding unit 207Dm, and an IMDU decoding unit 207Di.

[0047] The MDU decoder 207Dm includes a displacement amount codec determination unit 207Dm1, a geometric video position information decoder 207Dm2, a displacement amount decoding parameter decoder 207Dm3, and an attribute information parameter decoder 207Dm4.

[0048] The IMDU decoder 207Di includes a displacement amount codec determination unit 207Di1, a geometric video position information decoder 207Di2, and an attribute information parameter decoder 207Di3.

[0049] The displacement decoding parameter decoder 207Dm3 decodes parameters required for decoding the displacement regardless of the displacement codec, that is, in common with both the video codec and arithmetic decoding.

[0050] For example, such parameters include an override flag mdu_parameters_override_flag, a subdivision override flag mdu_subdivision_override_flag, a quantization override flag mdu_quantization_override_flag, a transformation method override flag mdu_transform_method_override_flag, a transformation parameter override flag mdu_transform_method_override_flag, a subdivision override flag mdu_subdivision_override_flag, a quantization override flag mdu_quantization_override_flag, a transformation parameter override flag mdu_transform_method ... The parameters may include an override flag mdu_transform_parameters_override_flag for the data, a subdivision method mdu_subdivision_method, the number of subdivision iterations mdu_subdivision_iteration_count, a coordinate system for the displacement mdu_displacement_coordinate_system, and a transformation method mdu_transform_method.

[0051] (Bitstream Structure) Fig. 5 is a diagram showing an example of the structure of a bitstream. The bitstream may be composed of a V3C parameter set, an atlas bitstream, a base mesh bitstream, a displacement bitstream, and a texture bitstream.

[0052] The V3C parameter set may include a profile_tier_level that indicates the profile tier level.

[0053] 6 is a diagram showing an example of the syntax configuration of profile_tier_level. profile_tier_level may include ptl_profile_codec_group_idc indicating a codec ID of the amount of displacement. When the amount of displacement is decoded using a video codec, ptl_profile_codec_group_idc may be an integer ranging from 0 to 4, or ptl_profile_codec_group_idc may be 127. When the amount of displacement is arithmetically decoded, ptl_profile_codec_group_idc may be 5.

[0054] (Configuration of Displacement Amount Bitstream - Arithmetic Decoding) Hereinafter, an example of the configuration of a displacement amount bitstream will be described with reference to FIGS.

[0055] FIG. 7 is a diagram showing an example of the configuration of a displacement amount bit stream.

[0056] In FIGS. 8 to 10, u(n) means an n-bit code, and ue(v) means an unsigned variable-length zeroth-order exponential-Golomb code.

[0057] First, as shown in FIG. 7, the displacement bitstream may include a DPS (Displacement Parameter Set), which is a set of control information related to the decoding of the displacement.

[0058] FIG. 8 is a diagram showing an example of the syntax configuration of a DPS.

[0059] As shown in FIG. 8, the DPS may include an ID (displacement_codec_id) indicating whether arithmetic decoding or video codec decoding is to be performed.

[0060] For example, it may be defined that when displacement_codec_id is 1, arithmetic coding is performed, and when displacement_codec_id is 0, decoding using a video codec is performed.

[0061] When arithmetic coding is performed (i.e., when displacement_codec_id is 1), the displacement bitstream may include a Displacement Frame Header (DFH), which is a set of control information corresponding to a frame, as shown in FIG.

[0062] FIG. 9 is a diagram illustrating an example of a syntax configuration of DFH.

[0063] As shown in FIG. 9, the DFH may include a frame type (displacement_frame_type).

[0064] For example, when displacement_frame_type is 1, it may be defined that inter prediction is performed in the corresponding frame, and when displacement_frame_type is 0, it may be defined that inter prediction is not performed in the corresponding frame.

[0065] 7, the displacement bitstream may include, after the DFH, a DDU (Displacement Data Unit) that is an encoded displacement corresponding to a frame. The DDU may include the number of vertices ddu_vertex_count_lod.

[0066] (Configuration of Displacement Amount Bitstream - Decoding by Video Codec) When decoding by a video codec is performed, the mesh patch decoding unit 207D may use the decoded displacement amount video to decode the atlas bitstream or the displacement amount bitstream shown in Fig. 10. The displacement amount codec determination unit 207Dm1 / 207Di1 is configured to determine whether a video codec is used for encoding and decoding the displacement amount. The displacement amount codec determination unit 207Dm1 / 207Di1 may determine that a video codec is used when ptl_profile_codec_group_idc is an integer in the range of 0 to 4 or when ptl_profile_codec_group_idc is 127.

[0067] The displacement codec determining unit 207Dm1 / 207Di1 may determine that a video codec has been used to encode and decode the displacement when displacement_codec_id is 0.

[0068] In the following description, the case where a video codec is used in the displacement amount codec determination unit 207Dm1 / 207Di1 will be described as displacement_codec_id being 0, but this is not limiting.

[0069] The atlas bitstream may include an atlas sequence parameter set (ASPS), which is a set of control information related to the decoding of displacements, and its extension, the atlas sequence parameter set V-DMC extension (ASVE), for 3D mesh compression.

[0070] As shown in FIG. 11, the ASVE may include, for example, asve_num_attribute_video, which indicates the number of attribute videos.

[0071] Furthermore, ASVE may include asve_packing_method, which indicates a packing method for the displacements. asve_packing_method may define the ascending or descending order in which the displacements are packed into the image.

[0072] Furthermore, ASVE may include asve_attribute_subtexture_enabled_flag, which is a flag indicating whether each attribute video is enabled or disabled.

[0073] As shown in FIG. 10, the atlas bitstream may include, following the ASPS, an AFPS (Atlas Frame Parameter Set) corresponding to the frame and its extension AFVE (Atlas Frame Parameter Set V-DMC Extension) for 3D mesh compression.

[0074] AFVE may include, for example, a flag afve_projection_textcoord_present_flag[i] indicating whether or not texture coordinates corresponding to submesh index i exist.

[0075] The AFVE may also include information regarding tile division of geometric and attribute information, and inter and intra mesh information.

[0076] As shown in Figure 10, the atlas bitstream may include, next to the AFPS, ATDUs (Atlas Tile Data Units), which are coded displacements corresponding to tiles, and their extensions for 3D mesh compression.

[0077] As shown in FIG. 10, the atlas bitstream may include mesh patch data units (MDUs) in an ATDU that correspond to patches within a tile.

[0078] As shown in FIG. 12, the MDU may include information for identifying the range of the patch in the frame of the decoded displacement image.

[0079] For example, the MDU may include the following syntax that indicates the coordinates that are the reference for the patch in the tile (usually the x and y coordinates of the top left corner of the tile) and the size of the patch, such as the width x and height y.

[0080] This indicates the position of the patch in the displacement video tile corresponding to the tile ID (tileID) and patch index (patchIdx). The x and y coordinates of the displacement video patch may be mdu_2d_pos_x[tileID][patchIdx] and mdu_2d_pas_y[tileID][patchIdx], respectively. Also, the width (quantized)-1 and height (quantized)-1 of the displacement video patch may be mdu_2d_size_x_minus1[tileID][patchIdx] and mdu_2d_size_y_minus1[tileID][patchIdx], respectively.

[0081] The image development unit 205C extracts the corresponding coefficient level value from the patch position. The inverse quantization unit 205D corrects the coefficient level value. The inverse transformation unit 205H performs an inverse transformation such as wavelet transformation and outputs the decoded displacement amount.

[0082] As shown in FIG. 12, the MDU may include parameters for decoding attribute information.

[0083] For example, the MDU may include information for identifying the extent to which patches exist in a frame of video of a decoded attribute (texture).

[0084] For example, the MDU may include the following syntax related to the coordinates of the reference patch in the attribute video tile (usually the x and y coordinates of the top left corner of the tile) and the size of the patch, such as the width x and height y:

[0085] This indicates the patch position in the attribute video tile corresponding to the tile ID (tileID) and patch index (patchIdx). The x and y coordinates of the patch may be mdu_attribute_2d_pos_x[tileID][patchIdx][j] and mdu_attribute_2d_pos_y[tileID][patchIdx][j], respectively. In addition, the width (quantized) - 1 and height (quantized) - 1 of the patch may be mdu_attribute_2d_sice_x_minus1[tileID][patchIdx][j] and mdu_attribute_2d_sice_y_minus1[tileID][patchIdx][j], respectively. j is the index of the attribute video for which the sub-texture is valid.

[0086] (Deletion of unnecessary or redundant syntax) However, when arithmetic decoding is used for the displacement amount in an atlas bitstream, unnecessary information is encoded or the same information is encoded in duplicate, increasing the amount of code.

[0087] Therefore, it is necessary to control the arithmetic decoding so that such information is not coded or decoded.

[0088] The atlas sequence parameter set decoding unit 207A in the atlas data decoding unit 207 is configured to decode ASVE. The asve_packing_method, which indicates the packing method of the displacement, is currently used only in decoding by a video codec and is therefore not required during arithmetic decoding.

[0089] The atlas sequence parameter set decoding unit 207A may be provided with a displacement amount codec determination unit 207A1a, which may encode asve_packing_method only when it is determined that a video codec is being used, i.e., when displacement_codec_id is 0, and may not encode asve_packing_method in other cases.

[0090] During decoding, the displacement amount codec determination unit 207A1a may decode asve_packing_method only when it is determined that a video codec is being used, i.e., when displacement_codec_id is 0, as shown in Figure 13, and may not decode asve_packing_method in other cases.

[0091] In the mesh patch decoding unit 207D of the atlas data decoding unit 207, as shown in Figure 12, even when the displacement amount is not decoded using a video codec in the MDU, the syntax mdu_2d_pos_x, mdu_2d_pos_y, mdu_2d_size_x_minus1, and mdu_2d_size_y_minus1 related to the patch existence range in the video frame of the displacement amount, which is geometric information, are present, but these are unnecessary.

[0092] In addition, mdu_vertex_count_minus1, which is the number of vertices minus 1, may also hold the number of vertices in the base mesh or arithmetic decoding, so the information is duplicated (hereinafter, a numerical value that can derive the number of vertices, such as the number of vertices minus 1, may be referred to as the number of vertices for simplicity).

[0093] On the other hand, the MDU contains attribute information mdu_attribute_2d_pos_x, mdu_attribute_2d_pos_y, mdu_attribute_2d_size_x_minus1, and mdu_attribute_2d_size_y_minus1, which relate to the patch existence range in the texture video frame. These attribute information parameters (attribute information parameters) are often required even during arithmetic decoding.

[0094] When asve_num_attribute_video is 0 and asve_attribute_subtexture_enabled_flag is 1, unlike the patch existence range of the video of the geometric information, the specifications allow the decoding of these attribute information parameters to be omitted, as shown in FIG.

[0095] For displacement video, as with attribute information parameters, unnecessary syntax or redundant syntax should not be coded or decoded. Similarly, for IMDU, unnecessary or redundant syntax exists. These should also not be coded or decoded, as with attribute information parameters.

[0096] Therefore, in the proposed syntax, in the decoding of the MDU by the MDU decoder 207Dm of the mesh patch decoder 207D, a displacement codec determination unit 207Dm1 is provided, and the displacement codec determination unit 207Dm1 determines the existence range of the patch in the frame of the decoded displacement video only when it is determined that a video codec is used, that is, only when displacement_codec_id is 0. If displacement_codec_id is other than 0, mdu_2d_pos_x, mdu_2d_pos_y, mdu_2d_size_x_minus1, and mdu_2d_size_y_minus1 are not decoded.

[0097] Similarly, during encoding, only when it is determined that a video codec is used, is the range in which the patch exists in the frame of the displacement video coded; otherwise, no coding is performed.

[0098] An example of the syntax configuration of an MDU is shown in Fig. 14. Only when displacement_codec_id is 0, mdu_2d_pos_x, mdu_2d_pos_y, mdu_2d_size_x_minus1, and mdu_2d_size_y_minus1 are decoded, and when displacement_codec_id is anything else, mdu_2d_pos_x, mdu_2d_pos_y, mdu_2d_size_x_minus1, and mdu_2d_size_y_minus1 are not decoded.

[0099] Note that the syntax elements are not limited to mdu_2d_pos_x, mdu_2d_pos_y, mdu_2d_size_x_minus1, and mdu_2d_size_y_minus1, as long as they can indicate the patch existence range. This makes it possible to reduce the amount of code related to syntax elements that are unnecessary when the displacement amount is not decoded by a video codec.

[0100] Furthermore, mdu_vertex_count_minus1, which is the number of vertices minus 1, is coded only when displacement_codec_id is 0, and is not coded when displacement_codec_id is other than 0.

[0101] During decoding, mdu_vertex_count_minus1 is decoded only when displacement_codec_id is 0, and mdu_vertex_count_minus1 is not decoded when displacement_codec_id is other than 0. This makes it possible to delete the code amount related to overlapping syntax when arithmetically decoding the displacement amount.

[0102] The same applies to the range of existence of the patch in the frame of the image of the decoded displacement amount in the IMDU, which is an Inter meshpatch data unit syntax other than the MDU.

[0103] In the decoding of the IMDU by the mesh patch decoder 207D, a displacement codec determination unit 207Di1 is provided, and only when the displacement codec determination unit 207Di1 determines that a video codec is used, that is, only when displacement_codec_id is 0, imdu_2d_delta_pos_x, imdu_2d_delta_pos_y, imdu_2d_delta_size_x and imdu_2d_delta_size_y are decoded, and if displacement_codec_id is anything else, imdu_2d_delta_pos_x, imdu_2d_delta_pos_y, imdu_2d_delta_size_x and imdu_2d_delta_size_y are not decoded.

[0104] Similarly, during encoding, only if it is determined that a video codec is used, the range of existence of the patch in the frame of the video of the displacement is encoded, and otherwise, no encoding is performed. Figure 15 shows an example of the syntax configuration of an IMDU. As with the MDU, a displacement amount codec determination unit is provided, and only when the determination unit determines that a video codec has been used, i.e., only when displacement_codec_id is 0, imdu_2d_delta_pos_x, imdu_2d_delta_pos_y, imdu_2d_delta_size_x, and imdu_2d_delta_size_y are decoded, and if displacement_codec_id is anything other than this, imdu_2d_delta_pos_x, imdu_2d_delta_pos_y, imdu_2d_delta_size_x, and imdu_2d_delta_size_y are not decoded.

[0105] Note that the syntax elements are not limited to imdu_2d_delta_pos_x, imdu_2d_delta_pos_y, imdu_2d_delta_size_x, and imdu_2d_delta_size_y, as long as they can indicate the patch existence range. This makes it possible to reduce the amount of code related to syntax elements that are unnecessary when arithmetic decoding the amount of displacement.

[0106] 15, imdu_delta_vertex_count_minus1, which is the number of vertices minus 1, is coded only when displacement_codec_id is 0, and is not coded when displacement_codec_id is other than 0.

[0107] During decoding, imdu_delta_vertex_count_minus1 is decoded only when displacement_codec_id is 0, and imdu_delta_vertex_count_minus1 is not decoded when displacement_codec_id is other than 0. This makes it possible to delete the code amount related to the syntax that is duplicated during arithmetic decoding of the displacement amount.

[0108] In addition, in the contributions (m66206 in the above-mentioned non-patent document 4 and m65852 in the non-patent document 5), a syntax for displacement amount video within an MDU has been proposed, and there is a possibility that it will be adopted into a standard in the future.

[0109] These also include syntax that is only used for decoding by a video codec, so by signaling encoding / decoding only when displacement_codec_id is 0, the amount of code for unnecessary syntax or redundant syntax can be reduced when arithmetically decoding the displacement.

[0110] In m65852 of Non-Patent Document 4, pdu_vertex_block_count_minus1 and pdu_vertex_count_inlastblock correspond to the range in which the patch exists in the frame of the image of the displacement amount, and the number of vertices is also calculated.

[0111] pdu_vertex_block_count_minus1 may indicate the number of blocks per LoD indicating the level of subdivision minus 1. The block size is PatchPackingBlockSize x PatchPackingBlockSize.

[0112] pdu_vertex_count_inlastblock may indicate the number of coefficients in the last block.

[0113] In m66206 of Non-Patent Document 5, the range of existence of patches in a frame of a displacement image corresponds to mdu_2d_pos_x, mdu_2d_pos_y, mdu_2d_size_x_minus1, and mdu_2d_size_y_minus1, and the number of vertices - 1 corresponds to mdu_vertex_count_minus1.

[0114] Non-Patent Documents 4 and 5 propose to retain information on the range of existence of the patch in the frame of the image of the displacement and the number of vertices for each LoD (Level of Detail) in the patch. If such a proposal is adopted, the amount of coding will further increase. Therefore, if the amount of displacement is not decoded by the image, the effect of reducing the amount of coding will be further enhanced by not encoding and decoding these syntaxes.

[0115] (Operation of Mesh Patch Decoding Unit 207D in Atlas Data Decoding Unit 207) Hereinafter, an example of the operation of the atlas data decoding unit 207 will be described with reference to FIGS. 16 and 17. FIG.

[0116] First, an example of the operation of the atlas data decoding unit 207 to decode a mesh patch will be described with reference to Figures 16 and 17. Here, Figure 16 is a flowchart showing the processing when the mesh patch mode is intra, and Figure 17 is a flowchart showing the processing when the mesh patch mode is inter.

[0117] First, the process when the mesh patch mode is intra will be described with reference to FIG.

[0118] As shown in FIG. 16(a), in step S101, in the patch mode determination, the mesh patch decoding unit 207D in the atlas data decoding unit 207 determines whether the mesh patch mode (meshpatchMode) is intra (P_INTRA or I_INTRA).

[0119] If the mesh patch mode is intra, in step S102, the mesh patch decoding unit 207D decodes the patch identification ID. The identification ID may include mdu_submesh_id indicating the identification ID of a sub-mesh obtained by dividing the entire mesh.

[0120] In step S103, in the displacement codec determination, the mesh patch decoding unit 207D determines whether or not a geometric video exists, that is, whether or not the displacement is decoded by a video codec (whether or not displacement_codec_id is 0).

[0121] If geometric video exists, i.e., if the displacement is decoded by a video codec (displacement_codec_id is 0), proceed to step S104 and step S105. Otherwise, proceed to step S106.

[0122] In steps S104 and S105, the mesh patch decoding unit 207D decodes the number of vertices (mdu_vertex_count_minus1) and the patch existence range information (mdu_2d_pos_x, mdu_2d_pos_y, mdu_2d_size_x_minus1, and mdu_2d_size_y_minus1), respectively.

[0123] On the other hand, in step S103, if there is no geometric video, that is, if the displacement amount is not decoded by a video codec (displacement_codec_id is other than 0), the mesh patch decoding unit 207D does not decode the number of vertices (mdu_vertex_count_minus1) and the patch existence range information (mdu_2d_pos_x, mdu_2d_pos_y, mdu_2d_size_x_minus1 and mdu_2d_size_y_minus1).

[0124] The number of vertices can be calculated from the number of vertices of the base mesh, and may also be stored in the displacement bitstream, so it does not need to be included in the syntax of the mesh patch data unit. In this case, the processing of step S104 is not performed. Furthermore, when using these, the syntax mdu_vertex_count_minus1 is deleted from the MDU and does not need to be decoded.

[0125] In step S106, the mesh patch decoder 207D decodes the remaining syntax in the MDU.

[0126] In step S106, the mesh patch decoder 207D decodes the parameters in the remaining MDUs.

[0127] For example, in step S106, the mesh patch decoding unit 207D may decode the syntax used in the displacement decoding in step S107, which is displacement recovery parameter decoding, regardless of the video codec or arithmetic decoding.

[0128] This includes the override flag mdu_parameters_override_flag, the subdivision override flag mdu_subdivision_override_flag, the quantization override flag mdu_quantization_override_flag, the transformation method override flag mdu_transform_method_override_flag, and the transformation parameter override flag mdu_transform_method_override_flag. It may include an override flag mdu_transform_parameters_override_flag, a subdivision method mdu_subdivision_method, a subdivision iteration count mdu_subdivision_iteration_count, a coordinate system for the displacement mdu_displacement_coordinate_system, a transformation method mdu_transform_method, etc.

[0129] Also, in step S106, the mesh patch decoding unit 207D may decode the position information of the patch of the attribute information (information for identifying the range in which the patch exists in the frame of the image of the decoded attribute) etc. in the attribute information parameter decoding unit in step 108.

[0130] This may include mdu_attribute_2d_pos_x[tileID][patchIdx][j] and mdu_attribute_2d_pos_y[tileID][patchIdx][j], and may also include mdu_attribute_2d_sice_x_minus1[tileID][patchIdx][j] and mdu_attribute_2d_sice_y_minus1[tileID][patchIdx][j], which are the quantized values ​​of the patch width -1 and height -1.

[0131] The operation of the geometric video position information decoding step S105 will be described with reference to Figure 16(b). mdu_2d_pos_x and mdu_2d_pos_y, which respectively relate to the vertex coordinates x and y of the patch's existence range, are decoded from the atlas bitstream. mdu_2d_pos_x is multiplied by PatchPackingBlockSize, which is the block size when packing into an image, and TileMeshpatch2dPosX, which is the x coordinate of the patch vertex, is output.

[0132] Similarly, mdu_2d_pos_y is multiplied by PatchPackingBlockSize, which is the block size when packing into an image, and TileMeshpatch2dPosY, which is the y coordinate of the patch vertex, is output.

[0133] Furthermore, mdu_2d_size_x_minus1 and mdu_2d_size_y_minus1, which respectively relate to the width and height of the range in which the patch exists, are decoded from the atlas bitstream. Regarding the width, 1 is added to mdu_2d_size_y_minus1.

[0134] PatchSizeXQuantizer, which is the X-direction quantized value of the patch size, is multiplied by mdu_2d_size_x_minus1+1 to output TileMeshpatch2dSizeX, which is the patch width. Similarly, for the height, 1 is added to mdu_2d_size_y_minus1.

[0135] PatchSizeYQuantizer, which is the Y-direction quantized value of the patch size, is multiplied by mdu_2d_size_y_minus1+1, and TileMeshpatch2dSizeY, which is the height of the patch, is output.

[0136] Next, the processing when the mesh patch mode is inter will be described with reference to FIG.

[0137] As shown in FIG. 17(a), in the patch mode determination in step S101, the mesh patch decoding unit 207D in the atlas data decoding unit 207 determines whether the mesh patch mode (meshpatchMode) is inter (P_INTER) or not.

[0138] If the mesh patch mode is inter, then in step S102, the mesh patch decoder 207D decodes parameters related to the patch ID, which may include imdu_patch_index.

[0139] In the displacement codec determination in step S103, the mesh patch decoding unit 207D determines whether or not a geometric video exists, that is, whether or not the displacement is decoded by a video codec (whether or not displacement_codec_id is 0).

[0140] If geometric video exists, i.e., if the displacement is decoded by a video codec (displacement_codec_id is 0), proceed to step S104 and step S105. Otherwise, proceed to step S106.

[0141] In steps S104 and S105, the mesh patch decoding unit 207D decodes the difference in the number of vertices (imdu_delta_vertex_count_minus1) and the difference in the patch existence range (imdu_2d_delta_pos_x, imdu_2d_delta_pos_y, imdu_2d_delta_size, and imdu_2d_delta_size_y), respectively.

[0142] On the other hand, in step S103, if there is no geometric video, that is, if the displacement amount is not decoded by a video codec (displacement_codec_id is other than 0), the mesh patch decoding unit 207D does not decode the difference in the number of vertices (imdu_delta_vertex_count_minus1) and the difference in the patch existence range (imdu_2d_delta_pos_x, imdu_2d_delta_pos_y, imdu_2d_delta_size_x and imdu_2d_delta_size_y).

[0143] The number of vertices can be calculated from the number of vertices of the base mesh and may also be stored in the displacement bitstream, so it does not need to be included in the syntax of the mesh patch data unit. In this case, the processing of step S104 is not performed. Furthermore, when using these, the syntax imdu_delta_vertex_count_minus1 is deleted from the IMDU and does not need to be decoded.

[0144] In step S106, the mesh patch decoder 207D decodes the syntax in the remaining IMDU.

[0145] For example, in step S106, the mesh patch decoding unit 207D may decode the difference in patch position information of the attribute information (information for identifying the range of existence of the patch in the frame of the image of the decoded attribute) by attribute information parameter decoding in step S107.

[0146] This may include imdu_attribute_2d_delta_pos_x[tileID][patchIdx][j] and imdu_attribute_2d_delta_pos_y[tileID][patchIdx][j], and may also include imdu_attribute_2d_delta_size_x[tileID][patchIdx][j] and imdu_attribute_2d_delta_size_y[tileID][patchIdx][j], which relate to the difference in width and height of the patch.

[0147] An example of the operation of decoding the patch identification ID in step S102 will be described with reference to FIG.

[0148] In step S102, the prediction index PredictoIdx is added to the decoded inter patch index imdu_patch_index to obtain the reference patch index RefPatchIdx.

[0149] The operation of step S105 for decoding geometric video position information will be described with reference to FIG. 17(c).

[0150] TileMeshpatch2dPosX and TileMeshpatch2dPosY having the reference patch index RefPatchIdx are copied to refMeshpatch2dPosX and refMeshpatch2dPosY, respectively.

[0151] Information imdu_2d_delta_pos_x and imdu_2d_delta_pos_y relating to the difference in vertex positions within the range in which the patch exists are decoded from the atlas bitstream.

[0152] imdu_2d_delta_pos_x is multiplied by PatchPackingBlockSize, which is the block size when packing into an image, and refMeshpatch2dPosX, which is the x coordinate of the vertex of the reference patch, is added to this, and TileMeshpatch2dPosX, which is the x coordinate of the patch vertex, is output.

[0153] Similarly, PatchPackingBlockSize is multiplied by imdu_2d_delta_pos_y, and refMeshpatch2dPosY, which is the y coordinate of the vertex of the reference patch, is added to it, and TileMeshpatch2dPosY, which is the y coordinate of the patch vertex, is output.

[0154] Furthermore, TileMeshpatch2dSize and TileMeshpatch2dSizeY having the reference patch index RefPatchIdx are copied to refMeshpatch2dSizeX and refMeshpatch2dSizeY, respectively. Here, in addition to TileMeshpatch2dPosX and TileMeshpatch2dPosY, TileMeshpatch2dSize and TileMeshpatch2dSizeY are set as reference tile mesh patch position information.

[0155] Information relating to the width x and height y of the patch's existence range, imdu_2d_delta_size_x and imdu_2d_delta_size_y, are decoded from the atlas bitstream.

[0156] PatchSizeXQuantizer, which is the X-direction quantized value of the patch size, is multiplied by mdu_2d_size_x, and refMeshpatch2dSizeX, which is the width of the reference patch, is added to it, and TileMeshpatch2dSizeX, which is the width of the patch, is output.

[0157] Similarly, mdu_2d_size_y is multiplied by PatchSizeYQuantizer, which is the Y-direction quantized value of the patch size, and refMeshpatch2dSizeY, which is the height of the reference patch, is added to it, and TileMeshpatch2dSizeY, which is the height of the patch, is output.

[0158] Secondly, with reference to FIG. 18, an example of the operation of the atlas sequence parameter set decoding unit 207A in the atlas data decoding unit 207 to decode ASVE will be described.

[0159] As shown in FIG. 18, in step S101, the displacement codec determination unit 207A1a of the ASVE decoding unit 207A1 in the atlas sequence parameter set decoding unit 207A determines whether or not geometric video exists, i.e., whether or not the displacement is decoded by a video codec (whether or not displacement_codec_id is 0).

[0160] If geometric video is present, i.e., if the displacement is decoded by a video codec (displacement_codec_id is 0), proceed to step S102. Otherwise, proceed to step S103. In the geometric video packing method decoding in step S102, the ASVE decoding unit 207A1 in the atlas sequence parameter set decoding unit 207A decodes the image packing method asve_packing_method.

[0161] In step S103, the atlas sequence parameter set decoding unit 207A decodes other syntax in the ASVE, and some of the other syntax may be decoded before the displacement codec determination in S101.

[0162] According to the mesh decoding device 200 of this embodiment, even when arithmetic decoding is used, unnecessary information is not signaled, so that the amount of code can be reduced.

[0163] Although the image packing method has been described as the syntax of ASVE, it is the same even when it is moved to another parameter set. For example, even when the image packing method is moved to the displacement amount parameter set of DPS, the same processing may be performed by replacing ASVE with DPS.

[0164] The mesh encoding device 100 and mesh decoding device 200 described above may be realized as a program that causes a computer to execute each function (each step).

[0165] According to this embodiment, for example, it is possible to improve the overall service quality in video communication, which makes it possible to contribute to Goal 9 of the Sustainable Development Goals (SDGs) led by the United Nations, which is to "Develop resilient infrastructure, promote sustainable industrialization and foster innovation."

[0166] 1...Mesh processing system 100...Mesh encoding device 200...Mesh decoding unit 201...Demultiplexing unit 202...Basic mesh decoding unit 203...Subdivision unit 204...Mesh decoding unit 205...Displacement decoding unit 205A...Arithmetic decoding unit 205B...Video codec decoding unit 205C...Image expansion unit 205D...Inverse quantization unit 205H...Inverse transform unit 206...Video decoding unit 207...Atlas data decoding unit 207A...Atlas sequence parameter set decoding unit 207B...Atlas frame parameter set decoding unit 207C...Atlas style decoding unit 207D...Mesh patch decoding unit 207A1...ASVE decoding unit 207A1a...Displacement codec determination unit 207A1b...Geometric video packing method decoding unit 207A1c...Other syntax decoding unit in ASVE 207D1...patch mode determination unit 207Dm...MDU decoding unit 207Dm1...displacement amount codec determination unit 207Dm2...geometric video position information decoding unit 207Dm3...displacement amount decoding parameter decoding unit 207Dm4...attribute information parameter decoding unit 207Di...MDU decoding unit 207Di1...displacement amount codec determination unit 207Di2...geometric video position information decoding unit 207Di3...attribute information parameter decoding unit

Claims

1. A mesh decoding device comprising an atlas decoding unit configured to decode and output an atlas bitstream, the atlas decoding unit comprising a mesh patch decoding unit that decodes patches, the mesh patch decoding unit comprising: a displacement amount codec determination unit that determines whether the displacement amount has been decoded using a video codec; and a geometric video position information decoding unit that decodes information for specifying the range of existence of the patch in the frame of the decoded displacement amount video when it is determined that the displacement amount has been decoded using the video codec, wherein the geometric video position information decoding unit does not decode information for specifying the range of existence of the patch in the frame of the decoded displacement amount video when it is determined that the displacement amount has not been decoded using the video codec.

2. The mesh decoding device described in claim 1, characterized in that the information for identifying the range of existence of the patch in the frame of the image of the decoded displacement amount includes vertex coordinates indicating any one point of the rectangle or quantized versions of the vertex coordinates, and the width and height of the rectangle or quantized versions of the width and height of the rectangle.

3. The mesh decoding device described in claim 1, characterized in that the information for identifying the range of existence of the patch in the frame of the image of the decoded displacement includes the starting position of the block and the number of vertices of the last block.

4. A mesh decoding device comprising an atlas decoding unit configured to decode and output an atlas bitstream, wherein the atlas decoding unit comprises a mesh patch decoding unit that decodes patches, wherein the mesh patch decoding unit comprises: a displacement amount codec determination unit that determines whether the displacement amount has been decoded using a video codec; and a geometric video position information decoding unit that decodes the number of vertices in a frame of video of the decoded displacement amount when it is determined that the displacement amount has been decoded using the video codec, wherein the geometric video position information decoding unit does not decode the number of vertices in a frame of video of the decoded displacement amount when it is determined that the displacement amount has not been decoded using the video codec.

5. A mesh decoding device comprising: an atlas decoding unit configured to decode and output an atlas bitstream, wherein the atlas decoding unit comprises an atlas sequence parameter set decoding unit that decodes an atlas sequence parameter set, wherein the atlas sequence parameter set decoding unit comprises: a displacement amount codec determination unit that determines whether or not a displacement amount is decoded using a video codec; and a geometric video packing method decoding unit that decodes a packing method of the decoded displacement amount when it is determined that the displacement amount is decoded using the video codec, wherein the geometric video packing method decoding unit does not decode the packing method of the decoded displacement amount when it is determined that the displacement amount is not decoded using the video codec.

6. A mesh decoding method comprising a step of decoding and outputting an atlas bitstream, the step comprising step A of decoding a patch, the step A comprising step A1 of determining whether or not the displacement amount has been decoded using a video codec, step A2 of decoding information for specifying the range of existence of the patch in a frame of video of the decoded displacement amount when it is determined that the displacement amount has been decoded using the video codec, and step A3 of not decoding the information for specifying the range of existence of the patch in a frame of video of the decoded displacement amount when it is determined that the displacement amount has not been decoded using the video codec.

7. A program that causes a computer to function as a mesh decoding device, wherein the mesh decoding device comprises an atlas decoding unit configured to decode and output an atlas bitstream, the atlas decoding unit comprising a mesh patch decoding unit that decodes patches, the mesh patch decoding unit comprising: a displacement amount codec determination unit that determines whether the displacement amount has been decoded using a video codec; and a geometric video position information decoding unit that decodes information for specifying the range of existence of the patch in the frame of the decoded displacement amount video when it is determined that the displacement amount has been decoded using the video codec, and the geometric video position information decoding unit does not decode information for specifying the range of existence of the patch in the frame of the decoded displacement amount video when it is determined that the displacement amount has not been decoded using the video codec.

Images