Mesh decoding device, mesh decoding method, and program
The mesh decoding device and method optimize coding efficiency by selectively using video or arithmetic decoding to reduce redundant code, addressing inefficiencies in existing mesh decoding techniques.
Patent Information
- Application Number
- JP2024064880
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2024-04-12
- Publication Date
- 2025-10-24
AI Technical Summary
Existing mesh decoding techniques using arithmetic decoding result in an increased amount of code, leading to inefficiencies in coding efficiency.
A mesh decoding device and method that selectively decodes displacement amounts using either a video codec or arithmetic decoding, depending on the displacement codec determination, thereby reducing unnecessary code by omitting redundant information during arithmetic decoding.
Improves coding efficiency by minimizing the amount of code when arithmetic decoding is used, without relying on a video codec.
Smart Images
Figure 2025161576000001_ABST
Abstract
Description
[Technical Field]
[0001] The present invention relates to a mesh decoding device, a mesh decoding method, and a program. [Background technology]
[0002] Non-Patent Document 1 discloses a technique for decoding a point group using video information and additional information called an atlas.
[0003] Non-Patent Document 2 discloses a technique in which a mesh is decomposed into a rough basic mesh and a detailed displacement amount, and the displacement amount is converted into an image and then encoded by a video codec.
[0004] Non-Patent Document 3 discloses the specifications of a technique for decoding displacement amounts using a video codec and a technique for arithmetic decoding. Either video codec decoding or arithmetic decoding can be used for decoding displacement amounts. Non-Patent Document 3 also discloses the specifications of image packing of displacement amounts. [Prior art documents] [Non-patent literature]
[0005] [Non-Patent Document 1] "Information technology - Coded Representation of Immersive Media - Part 5: Visual Volumetric Video-based Coding (V3C) and Video-based Point Cloud Compression (V-PCC)," ISO / IEC JTC 1 / SC 29 / WG 7, ISO / IEC 23090-5:2021(2E). [Non-patent document 2] Khaled Mammou, Jungsun Kim, Alexis Tourapis, Dimitri Podborski, Krasimir Kolarov, "[V-CG] Apple's Dynamic Mesh Coding CfP Response," ISO / IEC JTC 1 / SC 29 / WG 7 m59281, April 2022. [Non-patent document 3] "WD 6.0 of V-DMC," ISO / IEC JTC 1 / SC 29 / WG7 w23617, March 2024. [Non-patent document 4] Jungsun Kim et.al., “[V-DMC][new] Usage of parameters in the patches in the Atlas Data sub-bitstream, ISO / IEC JTC 1 / SC 20 / WG7 m65852, January 2024. [Non-Patent Document 5] Patrice Rndao Alface et. al., “[V-DMC][EE4.7-related][New] LOD-based displacement frame packing into rectangular regions, ISO / IEC JTC 1 / SC 29 / WG7 m66206, January 2024. Summary of the Invention [Problem to be solved by the invention]
[0006] Here, in the techniques disclosed in Non-Patent Documents 2 and 3, the displacement, which is one piece of geometric information, can be coded using video coding by a video codec or arithmetic coding.
[0007] However, when decoding the displacement amount by arithmetic decoding, there is a problem that the amount of code increases. Therefore, the present invention has been made in consideration of the above-mentioned problem, and an object of the present invention is to provide a mesh decoding device, a mesh decoding method, and a program that can reduce the amount of code when using arithmetic decoding, which can improve coding efficiency and can reduce the amount of code when not using a video codec when decoding the displacement amount. [Means for solving the problem]
[0008] A first feature of the present invention is a mesh decoding device comprising an atlas decoding unit configured to decode and output an atlas bitstream, the atlas decoding unit comprising a mesh patch decoding unit that decodes patches, the mesh patch decoding unit comprising a displacement amount codec determination unit that determines whether the displacement amount is decoded using a video codec, and a geometric video position information decoding unit that decodes information for identifying the range of existence of the patch in a frame of video of the decoded displacement amount when it is determined that the displacement amount is decoded using the video codec, and the geometric video position information decoding unit does not decode information for identifying the range of existence of the patch in a frame of video of the decoded displacement amount when it is determined that the displacement amount is not decoded using the video codec.
[0009] A second feature of the present invention is a mesh decoding device comprising an atlas decoding unit configured to decode and output an atlas bitstream, the atlas decoding unit comprising a mesh patch decoding unit that decodes patches, the mesh patch decoding unit comprising a displacement amount codec determination unit that determines whether the displacement amount is decoded using a video codec, and a geometric video position information decoding unit that decodes the number of vertices in a video frame of the decoded displacement amount when it is determined that the displacement amount is decoded using the video codec, and the geometric video position information decoding unit does not decode the number of vertices in the video frame of the decoded displacement amount when it is determined that the displacement amount is not decoded using the video codec.
[0010] A third feature of the present invention is summarized as a mesh decoding device including an atlas sequence parameter set decoding unit that decodes an atlas sequence parameter set, wherein the atlas sequence parameter set decoding unit includes a determination unit that determines whether a displacement amount is decoded by a video codec, and a geometric video packing method decoding unit that decodes a packing method of the decoded displacement amount when it is determined that the displacement amount is decoded by the video codec, and the geometric video packing method decoding unit does not decode the packing method of the decoded displacement amount when it is determined that the displacement amount is not decoded by the video codec.
[0011] A fourth feature of the present invention is summarized as a mesh decoding method comprising a step of decoding and outputting an atlas bitstream, the step including a step A of decoding a patch, the step A including a step A1 of determining whether or not a displacement amount has been decoded using a video codec, a step A2 of decoding information for identifying the range of existence of the patch in a frame of video of the decoded displacement amount when it is determined that the displacement amount has been decoded using the video codec, and a step A3 of not decoding information for identifying the range of existence of the patch in a frame of video of the decoded displacement amount when it is determined that the displacement amount has not been decoded using the video codec.
[0012] A fifth feature of the present invention is summarized as a program that causes a computer to function as a mesh decoding device, the mesh decoding device comprising an atlas decoding unit configured to decode and output an atlas bitstream, the atlas decoding unit comprising a mesh patch decoding unit that decodes patches, the mesh patch decoding unit comprising a displacement amount codec determination unit that determines whether the displacement amount has been decoded using a video codec, and a geometric video position information decoding unit that, when it is determined that the displacement amount has been decoded using the video codec, decodes information for identifying the range of existence of the patch in a frame of video of the decoded displacement amount, and when it is determined that the displacement amount has not been decoded using the video codec, does not decode the information for identifying the range of existence of the patch in a frame of video of the decoded displacement amount. [Effects of the Invention]
[0013] According to the present invention, it is possible to provide a mesh decoding device, a mesh decoding method and a program that can improve coding efficiency when a video codec is not used when decoding a displacement amount. [Brief explanation of the drawings]
[0014] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a mesh processing system 1 according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of functional blocks of a mesh decoding device 200 according to an embodiment. [Figure 3] FIG. 3 is a diagram showing an example of functional blocks of the displacement amount decoding unit 205 of the mesh decoding device 200 according to an embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of functional blocks of the atlas decoding unit 207 of the mesh decoding device 200 according to an embodiment. [Figure 5] FIG. 5 is a diagram showing an example of the structure of a bitstream. [Figure 6]FIG. 6 is a diagram illustrating an example of the syntax configuration of profile_tier_level. [Figure 7] FIG. 7 is a diagram showing an example of the configuration of a displacement amount bit stream. [Figure 8] FIG. 8 is a diagram showing an example of the syntax configuration of a DPS. [Figure 9] FIG. 9 is a diagram showing an example of the syntax configuration of the DFH. [Figure 10] FIG. 10 is a diagram illustrating an example of an atlas bitstream. [Figure 11] FIG. 11 is a diagram showing an example of a syntax configuration of ASVE. [Figure 12] FIG. 12 is a diagram illustrating an example of the syntax configuration of an MDU. [Figure 13] FIG. 13 is a diagram showing an example of a syntax configuration of ASVE. [Figure 14] FIG. 14 is a diagram illustrating an example of the syntax configuration of an MDU. [Figure 15] FIG. 15 is a diagram illustrating an example of the syntax configuration of an IMDU. [Figure 16] FIG. 16 is a flowchart showing an example of the operation of the atlas data decoding unit 207 to decode a mesh patch data unit. [Figure 17] FIG. 17 is a flowchart showing an example of the operation of the atlas data decoding unit 207 to decode a mesh patch data unit. [Figure 18] FIG. 18 is a flowchart showing an example of the operation of the atlas data decoding unit 207 to decode ASVE. DETAILED DESCRIPTION OF THE INVENTION
[0015] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the components in the following embodiments can be appropriately replaced with existing components, etc., and various variations, including combinations with other existing components, are possible. Therefore, the description of the following embodiments does not limit the content of the invention described in the claims.
[0016] First Embodiment The mesh processing system according to this embodiment will be described below with reference to FIGS.
[0017] 1 is a diagram showing an example of the configuration of a mesh processing system 1 according to this embodiment. As shown in FIG. 1, the mesh processing system 1 includes a mesh encoding device 100 and a mesh decoding device 200.
[0018] FIG. 2 is a diagram showing an example of functional blocks of the mesh decoding device 200 according to this embodiment.
[0019] As shown in Figure 2, the mesh decoding device 200 has a demultiplexing unit 201, a basic mesh decoding unit 202, a subdivision unit 203, a mesh decoding unit 204, a displacement amount decoding unit 205, a video decoding unit 206, and an atlas data decoding unit 207.
[0020] The demultiplexing unit 201 is configured to separate the multiplexed bit stream into a basic mesh bit stream, a displacement amount bit stream, a texture bit stream, and an atlas bit stream.
[0021] The base mesh decoding unit 202 is configured to decode the base mesh bitstream, generate and output a base mesh.
[0022] The subdivision unit 203 is configured to generate and output the added subdivision vertices and their connection information from the basic mesh decoded by the basic mesh decoding unit 202 using the subdivision method indicated by the control information.
[0023] Here, the basic mesh, the added subdivision vertices, and the connection information therebetween are collectively referred to as a "subdivision mesh."
[0024] The mesh decoding unit 204 is configured to generate and output a decoded mesh using the subdivision mesh generated by the subdivision unit 203 and the displacement amount decoded by the displacement amount decoding unit 205.
[0025] The displacement decoding unit 205 is configured to decode the displacement bitstream to generate and output the displacement.
[0026] The video decoding unit 206 is configured to decode the texture bitstream, which is attribute information, using a video codec to generate texture. For example, the video decoding unit 206 may use HEVC in Non-Patent Document 1. The atlas data decoding unit 207 is configured to decode the atlas bitstream and output various parameters related to the atlas to the basic mesh decoding unit 202, the displacement amount decoding unit 205, and the video decoding unit 206.
[0027] (Displacement amount decoding unit 205) Fig. 3 is a diagram showing an example of the configuration of the displacement amount decoding unit 205. As shown in Fig. 3, the displacement amount decoding unit 205 includes an arithmetic decoding unit 205A, a video codec decoding unit 205B, an image development unit 205C, an inverse quantization unit 205D, and an inverse transform unit 205H.
[0028] The arithmetic decoding unit 205A is configured to generate coefficient level values by performing arithmetic decoding on the received displacement amount bitstream.
[0029] The video codec decoding unit 205B is configured to generate an image by decoding the received displacement bitstream using a video codec.
[0030] The image development unit 205C is configured to develop an image to generate coefficient level values.
[0031] The inverse quantization unit 205D is configured to generate transform coefficients by performing inverse quantization on the received coefficient level values.
[0032] The inverse quantization unit 205D is a common functional block whether the video codec decoding unit 205B or the arithmetic decoding unit 205A is used.
[0033] The inverse transform unit 205H is configured to generate a decoded displacement by performing an inverse transform on the received transform coefficients. The inverse transform may be, for example, an inverse DCT or an inverse wavelet transform.
[0034] (Atlas data decoding unit 207) Fig. 4(a) is a diagram showing an example of the configuration of the atlas data decoding unit 207. As shown in Fig. 4(a), the atlas data decoding unit 207 includes an atlas sequence parameter set decoding unit 207A, an atlas frame parameter set decoding unit 207B, an atlas tile decoding unit 207C, and a mesh patch decoding unit 207D.
[0035] The atlas sequence parameter set decoding unit 207A is configured to decode the atlas sequence parameter set (ASPS) and its extension, the atlas sequence parameter set V-DMC extension (ASVE) for 3D mesh compression.
[0036] The atlas frame parameter set decoder 207B is configured to decode the Atlas Sequence Parameter Set (AFPS) and its extension Atlas Frame Parameter Set V-DMC Extension (AFVE) for 3D mesh compression.
[0037] The atlas tile decoder 207C is configured to decode various information about the tiles.
[0038] The mesh patch decoder 207D is configured to decode various information related to patches within a tile, including MDUs (Mesh Patch Data Units) and IMDUs (Inter Mesh Patch Data Units).
[0039] (ASVE decoding section 207A1) FIG. 4(a) is a diagram showing an example of the configuration of the decoding unit 207A1 of the ASVE.
[0040] The ASVE decoding unit 207A1 decodes ASVE parameters using a displacement codec determination unit 207A1a, a geometric video packing method decoding unit 207A1b, and another syntax decoding unit 207A1c in ASVE.
[0041] The geometric video packing method decoding unit 207A1b decodes parameters that are required only when the displacement video codec is used.
[0042] For example, the geometric video packing method decoding unit 207A1b may decode the image packing method asve_packing_method.
[0043] Another syntax decoder 207A1c in the ASVE decodes parameters that are commonly used when the displacement amount video codec is used or parameters that are not required when the displacement amount video codec is used.
[0044] For example, another syntax decoding unit 207A1c within ASVE may decode asve_subdivision_method indicating the subdivision method, asve_subdivision_iteration_count indicating the number of subdivision iterations, asve_1d_displacement_flag indicating the 1D displacement amount flag, asve_displacement_reference_qp indicating the displacement amount QP, and asve_num_attribute_video indicating the number of videos of the attribute.
[0045] (Mesh patch decoding unit 207D) FIG. 4(b) is a diagram showing an example of the configuration of the mesh patch decoding unit 207D.
[0046] The mesh patch decoding unit 207D obtains decoded patch data by the patch mode determination unit 207D1, the MDU decoding unit 207Dm, and the IMDU decoding unit 207Di.
[0047] The MDU decoder 207Dm includes a displacement amount codec determiner 207Dm1, a geometric video position information decoder 207Dm2, a displacement amount decoding parameter decoder 207Dm3, and an attribute information parameter decoder 207Dm4.
[0048] The IMDU decoder 207Di includes a displacement amount codec determiner 207Di1, a geometric video position information decoder 207Di2, and an attribute information parameter decoder 207Di3.
[0049] The displacement decoding parameter decoder 207Dm3 decodes parameters required for decoding the displacement regardless of the displacement codec, that is, parameters required for decoding the displacement in common for both video codec and arithmetic decoding.
[0050] For example, such parameters may include an override flag mdu_parameters_override_flag, a subdivision override flag mdu_subdivision_override_flag, a quantization override flag mdu_quantization_override_flag, a transform method override flag mdu_transform_method_override_flag, a transform parameter override flag mdu_transform_parameters_override_flag, a subdivision method mdu_subdivision_method, a subdivision iteration count mdu_subdivision_iteration_count, a displacement coordinate system mdu_displacement_coordinate_system, a transform method mdu_transform_method, etc. (Bitstream configuration) 5 is a diagram showing an example of the configuration of a bitstream. The bitstream may be composed of a V3C parameter set, an atlas bitstream, a base mesh bitstream, a displacement bitstream, and a texture bitstream.
[0051] A V3C parameter set may contain a profile_tier_level that indicates the profile tier level.
[0052] 6 is a diagram showing an example of the syntax configuration of profile_tier_level. profile_tier_level may include ptl_profile_codec_group_idc indicating the codec ID of the amount of displacement. When the amount of displacement is decoded using a video codec, ptl_profile_codec_group_idc may be an integer ranging from 0 to 4, or ptl_profile_codec_group_idc may be 127. When the amount of displacement is arithmetically decoded, ptl_profile_codec_group_idc may be 5.
[0053] (Displacement Bitstream Construction - Arithmetic Decoding) An example of the configuration of a displacement amount bit stream will be described below with reference to FIGS.
[0054] FIG. 7 is a diagram showing an example of the configuration of a displacement amount bit stream.
[0055] 8 to 10, u(n) means an n-bit code, and ue(v) means an unsigned variable-length zeroth-order exponential-Golomb code.
[0056] First, as shown in FIG. 7, the displacement bitstream may include a DPS (Displacement Parameter Set), which is a set of control information related to the decoding of the displacement.
[0057] FIG. 8 is a diagram showing an example of the syntax configuration of a DPS.
[0058] As shown in FIG. 8, the DPS may include an ID (displacement_codec_id) indicating whether arithmetic decoding or video codec decoding is to be performed.
[0059] For example, it may be defined that when displacement_codec_id is 1, arithmetic coding is performed, and when displacement_codec_id is 0, decoding using a video codec is performed.
[0060] When arithmetic coding is performed (i.e., when displacement_codec_id is 1), the displacement bitstream may include a DFH (Displacement Frame Header), which is a collection of control information corresponding to a frame, as shown in Figure 7.
[0061] FIG. 9 is a diagram showing an example of the syntax configuration of the DFH.
[0062] As shown in FIG. 9, the DFH may include a frame type (displacement_frame_type).
[0063] For example, when displacement_frame_type is 1, it may be defined that inter prediction is performed in the corresponding frame, and when displacement_frame_type is 0, it may be defined that inter prediction is not performed in the corresponding frame.
[0064] 7, the displacement bitstream may include, after the DFH, a DDU (Displacement Data Unit) that is an encoded displacement corresponding to a frame. The DDU may include the number of vertices ddu_vertex_count_lod.
[0065] (Structure of displacement bitstream - Decoding by video codec) When decoding is performed using a video codec, the mesh patch decoding unit 207D may use the decoded displacement video to decode the atlas bitstream and displacement bitstream shown in FIG.
[0066] The displacement codec determination unit 207Dm1 / 207Di1 is configured to determine whether a video codec is used for encoding and decoding the displacement. The displacement amount codec determination unit 207Dm1 / 207Di1 may determine that a video codec has been used when ptl_profile_codec_group_idc is an integer in the range of 0 to 4 or when ptl_profile_codec_group_idc is 127.
[0067] The displacement codec determination unit 207Dm1 / 207Di1 may determine that a video codec has been used for encoding and decoding the displacement when displacement_codec_id is 0.
[0068] In the following description, the case where a video codec is used in the displacement amount codec determination unit 207Dm1 / 207Di1 will be described as displacement_codec_id being 0, but this is not limitative.
[0069] The atlas bitstream may include an Atlas Sequence Parameter Set (ASPS), which is a set of control information related to the decoding of displacements, and its extension, the Atlas Sequence Parameter Set V-DMC Extension (ASVE), for 3D mesh compression.
[0070] As shown in FIG. 11, the ASVE may include, for example, asve_num_attribute_video, which indicates the number of attribute videos.
[0071] The ASVE may also include asve_packing_method, which indicates a packing method for the displacements, and may define the ascending or descending order when packing the displacements into the image.
[0072] The ASVE may also include asve_attribute_subtexture_enabled_flag, which is a flag indicating whether each attribute video is enabled or disabled.
[0073] As shown in Figure 10, the atlas bitstream may include, following the ASPS, an AFPS (Atlas Frame Parameter Set) corresponding to the frame and its extension AFVE (Atlas Frame Parameter Set V-DMC Extension) for 3D mesh compression.
[0074] The AFVE may include, for example, a flag afve_projection_textcoord_present_flag[i] indicating whether or not a texture coordinate corresponding to the submesh index i exists.
[0075] The AFVE may also include information regarding tile division of geometric and attribute information, and inter- and intra-mesh information.
[0076] As shown in Figure 10, the atlas bitstream may include, next to the AFPS, an ATDU (Atlas Tile Data Unit), which is a coded displacement corresponding to a tile, or its extension for 3D mesh compression.
[0077] As shown in FIG. 10, the atlas bitstream may include, in an ATDU, MDUs (Mesh Patch Data Units) corresponding to patches within a tile.
[0078] As shown in FIG. 12, the MDU may include information for identifying the range of the patch in the frame of the decoded displacement image.
[0079] For example, the MDU may include the following syntax that indicates the coordinates of the patch in the tile (usually the x and y coordinates of the top left corner of the tile) and the size of the patch, such as the width x and height y:
[0080] This indicates the position of the patch in the displacement video tile corresponding to the tile ID (tileID) and patch index (patchIdx). The x and y coordinates of the displacement video patch may be mdu_2d_pos_x[tileID][patchIdx] and mdu_2d_pas_y[tileID][patchIdx], respectively. Also, the width(quantized)-1 and height(quantized)-1 of the displacement video patch may be mdu_2d_size_x_minus1[tileID][patchIdx] and mdu_2d_size_y_minus1[tileID][patchIdx], respectively.
[0081] The image development unit 205C extracts the corresponding coefficient level value from the patch position. The inverse quantization unit 205D corrects the coefficient level value. The inverse transformation unit 205H performs an inverse transformation such as wavelet transformation and outputs the decoded displacement amount.
[0082] As shown in FIG. 12, the MDU may include a parameter for decoding attribute information.
[0083] For example, the MDU may include information for identifying the extent to which patches of a decoded attribute (texture) exist in a frame of video.
[0084] For example, the MDU may include the following syntax related to the coordinates of the patch in the attribute video tile (usually the x and y coordinates of the top left corner of the tile) and the size of the patch, such as the width x and height y:
[0085] This indicates the patch position in the attribute video tile corresponding to the tile ID (tileID) and patch index (patchIdx). The x and y coordinates of the patch may be mdu_attribute_2d_pos_x[tileID][patchIdx][j] and mdu_attribute_2d_pos_y[tileID][patchIdx][j], respectively. The width (quantized)-1 and height (quantized)-1 of the patch may be mdu_attribute_2d_sice_x_minus1[tileID][patchIdx][j] and mdu_attribute_2d_sice_y_minus1[tileID][patchIdx][j], respectively. j is the index of the attribute video for which subtexture is enabled.
[0086] (Deleting unnecessary and duplicated syntax) However, when arithmetic decoding is used for the displacement amount in the atlas bitstream, unnecessary information is coded or the same information is coded in duplicate, resulting in an increase in the amount of code.
[0087] Therefore, it is necessary to control the arithmetic decoding so that such information is not coded or decoded.
[0088] The atlas sequence parameter set decoding unit 207A in the atlas data decoding unit 207 is configured to decode ASVE. The asve_packing_method, which indicates the packing method for the displacement, is currently used only in decoding by a video codec and is therefore not required during arithmetic decoding.
[0089] The atlas sequence parameter set decoding unit 207A may be provided with a displacement amount codec determination unit 207A1a, which may encode asve_packing_method only when it is determined that a video codec is being used, i.e., when displacement_codec_id is 0, and may not encode asve_packing_method in other cases.
[0090] During decoding, the displacement amount codec determination unit 207A1a may decode asve_packing_method only when it is determined that a video codec is being used, i.e., when displacement_codec_id is 0, as shown in FIG. 13, and may not decode asve_packing_method in other cases.
[0091] In the mesh patch decoding unit 207D of the atlas data decoding unit 207, as shown in Figure 12, even when the displacement amount is not decoded using a video codec in the MDU, the syntax mdu_2d_pos_x, mdu_2d_pos_y, mdu_2d_size_x_minus1, and mdu_2d_size_y_minus1 related to the patch existence range in the video frame of the displacement amount, which is geometric information, are present, but these are unnecessary.
[0092] In addition, mdu_vertex_count_minus1, which is the number of vertices minus 1, may also contain the number of vertices in the base mesh or arithmetic decoding, so the information is duplicated (hereafter, for simplicity, numbers that can be used to derive the number of vertices, such as the number of vertices minus 1, may be referred to as the number of vertices).
[0093] On the other hand, the MDU contains attribute information mdu_attribute_2d_pos_x, mdu_attribute_2d_pos_y, mdu_attribute_2d_size_x_minus1, and mdu_attribute_2d_size_y_minus1, which relate to the patch existence range in the texture video frame. These attribute information parameters (attribute information parameters) are often required even during arithmetic decoding.
[0094] When asve_num_attribute_video is 0 and asve_attribute_subtexture_enabled_flag is 1, unlike the patch existence range of the video of the geometric information, the specifications allow the decoding of these attribute information parameters to be omitted, as shown in FIG.
[0095] For displacement video, as with attribute information parameters, unnecessary syntax or redundant syntax should not be coded or decoded. Similarly, for IMDU, unnecessary or redundant syntax exists. As with attribute information parameters, these should also not be coded or decoded.
[0096] Therefore, in the proposed syntax, when decoding an MDU by the MDU decoding unit 207Dm of the mesh patch decoding unit 207D, a displacement amount codec determination unit 207Dm1 is provided, and the displacement amount codec determination unit 207Dm1 decodes mdu_2d_pos_x, mdu_2d_pos_y, mdu_2d_size_x_minus1, and mdu_2d_size_y_minus1, which specify the range of existence of the patch in the frame of the decoded displacement amount video, only when it is determined that a video codec has been used, i.e., only when displacement_codec_id is 0, and does not decode mdu_2d_pos_x, mdu_2d_pos_y, mdu_2d_size_x_minus1, and mdu_2d_size_y_minus1 if displacement_codec_id is other than 0.
[0097] Similarly, during encoding, only when it is determined that a video codec is used, is the range in which the patch exists in the frame of the displacement video coded; otherwise, no coding is performed.
[0098] An example of the syntax configuration of an MDU is shown in Fig. 14. Only when displacement_codec_id is 0, mdu_2d_pos_x, mdu_2d_pos_y, mdu_2d_size_x_minus1, and mdu_2d_size_y_minus1 are decoded, and when displacement_codec_id is anything else, mdu_2d_pos_x, mdu_2d_pos_y, mdu_2d_size_x_minus1, and mdu_2d_size_y_minus1 are not decoded.
[0099] Note that the syntax elements are not limited to mdu_2d_pos_x, mdu_2d_pos_y, mdu_2d_size_x_minus1, and mdu_2d_size_y_minus1, as long as they can indicate the patch existence range. This makes it possible to reduce the amount of code related to syntax elements that are unnecessary when the displacement amount is not decoded by a video codec.
[0100] Also, mdu_vertex_count_minus1, which is the number of vertices minus 1, is coded only when displacement_codec_id is 0, and is not coded when displacement_codec_id is other than 0.
[0101] During decoding, mdu_vertex_count_minus1 is decoded only when displacement_codec_id is 0, and mdu_vertex_count_minus1 is not decoded when displacement_codec_id is other than 0. This makes it possible to delete the amount of code related to overlapping syntax when arithmetically decoding the displacement amount.
[0102] The same applies to the range of existence of a patch in a video frame of a decoded displacement amount in IMDU, which is an Inter meshpatch data unit syntax other than MDU.
[0103] In decoding IMDUs by the mesh patch decoding unit 207D, a displacement codec determination unit 207Di1 is provided, and only when the displacement codec determination unit 207Di1 determines that a video codec is used, i.e., only when displacement_codec_id is 0, does it decode imdu_2d_delta_pos_x, imdu_2d_delta_pos_y, imdu_2d_delta_size_x, and imdu_2d_delta_size_y, and if displacement_codec_id is anything other than this, it does not decode imdu_2d_delta_pos_x, imdu_2d_delta_pos_y, imdu_2d_delta_size_x, and imdu_2d_delta_size_y.
[0104] Similarly, during encoding, only when it is determined that a video codec is used, is the range in which the patch exists in the frame of the displacement video coded; otherwise, no coding is performed. An example of the syntax configuration of an IMDU is shown in Figure 15. As with the MDU, a displacement codec determination unit is provided, and imdu_2d_delta_pos_x, imdu_2d_delta_pos_y, imdu_2d_delta_size_x, and imdu_2d_delta_size_y are decoded only when the determination unit determines that a video codec is used, i.e., only when displacement_codec_id is 0, and imdu_2d_delta_pos_x, imdu_2d_delta_pos_y, imdu_2d_delta_size_x, and imdu_2d_delta_size_y are not decoded when displacement_codec_id is anything else.
[0105] Note that the syntax elements are not limited to imdu_2d_delta_pos_x, imdu_2d_delta_pos_y, imdu_2d_delta_size_x, and imdu_2d_delta_size_y, as long as they can indicate the patch existence range. This makes it possible to reduce the amount of code related to syntax elements that are unnecessary when arithmetically decoding the amount of displacement.
[0106] 15, imdu_delta_vertex_count_minus1, which is the number of vertices minus 1, is coded only when displacement_codec_id is 0, and is not coded when displacement_codec_id is other than 0.
[0107] During decoding, imdu_delta_vertex_count_minus1 is decoded only when displacement_codec_id is 0, and imdu_delta_vertex_count_minus1 is not decoded when displacement_codec_id is other than 0. This makes it possible to delete the amount of code related to redundant syntax when arithmetically decoding the displacement amount.
[0108] In addition, in the contributions (m66206 in the above-mentioned Non-Patent Document 4 and m65852 in the above-mentioned Non-Patent Document 5), a syntax for displacement amount video within MDU has been proposed, which may be adopted in the standard in the future.
[0109] These also include syntax that is only used for decoding by the video codec, so by signaling encoding / decoding only when displacement_codec_id is 0, it is possible to reduce the amount of code for unnecessary syntax or redundant syntax when arithmetically decoding the displacement.
[0110] In m65852 of Non-Patent Document 4, pdu_vertex_block_count_minus1 and pdu_vertex_count_inlastblock correspond to the range in which the patch exists in the frame of the image of the displacement amount, and the number of vertices is also calculated.
[0111] pdu_vertex_block_count_minus1 may indicate the number of blocks per LoD indicating the level of subdivision minus 1. The block size is PatchPackingBlockSize x PatchPackingBlockSize.
[0112] pdu_vertex_count_inlastblock may indicate the number of coefficients in the last block.
[0113] In m66206 of Non-Patent Document 5, the range in which the patch exists in the frame of the image of the displacement amount corresponds to mdu_2d_pos_x, mdu_2d_pos_y, mdu_2d_size_x_minus1, and mdu_2d_size_y_minus1, and the number of vertices minus 1 corresponds to mdu_vertex_count_minus1.
[0114] Non-patent documents 4 and 5 propose storing information on the range of existence of the patch in the displacement image frame and the number of vertices for each LoD (Level of Detail) within the patch. However, if such a proposal is adopted, the amount of coding will increase further. Therefore, if the amount of displacement is not decoded by the image, not encoding and decoding these syntaxes will further increase the effect of reducing the amount of coding.
[0115] (Operation of the mesh patch decoding unit 207D in the atlas data decoding unit 207) An example of the operation of the atlas data decoding unit 207 will be described below with reference to FIGS.
[0116] First, an example of the operation of the atlas data decoding unit 207 to decode a mesh patch will be described with reference to Fig. 16 and Fig. 17. Here, Fig. 16 is a flowchart showing the processing when the mesh patch mode is intra, and Fig. 17 is a flowchart showing the processing when the mesh patch mode is inter.
[0117] First, with reference to FIG. 16, the process when the mesh patch mode is intra will be described.
[0118] As shown in FIG. 16(a), in step S101, in the patch mode determination, the mesh patch decoding unit 207D in the atlas data decoding unit 207 determines whether the mesh patch mode (meshpatchMode) is intra (P_INTRA or I_INTRA).
[0119] If the mesh patch mode is intra, in step S102, the mesh patch decoding unit 207D decodes the patch identification ID. The identification ID may include mdu_submesh_id, which indicates the identification ID of a sub-mesh obtained by dividing the entire mesh.
[0120] In step S103, in determining the displacement codec, the mesh patch decoding unit 207D determines whether or not a geometric video exists, that is, whether or not the displacement is decoded by a video codec (whether or not displacement_codec_id is 0).
[0121] If a geometric video exists, that is, if the displacement is decoded by a video codec (displacement_codec_id is 0), proceed to step S104 and step S105. Otherwise, proceed to step S106.
[0122] In steps S104 and S105, the mesh patch decoding unit 207D decodes the number of vertices (mdu_vertex_count_minus1) and the patch existence range information (mdu_2d_pos_x, mdu_2d_pos_y, mdu_2d_size_x_minus1, and mdu_2d_size_y_minus1), respectively.
[0123] On the other hand, in step S103, if there is no geometric video, that is, if the displacement amount is not decoded by a video codec (displacement_codec_id is other than 0), the mesh patch decoding unit 207D does not decode the number of vertices (mdu_vertex_count_minus1) and the patch existence range information (mdu_2d_pos_x, mdu_2d_pos_y, mdu_2d_size_x_minus1 and mdu_2d_size_y_minus1).
[0124] The number of vertices can be calculated from the number of vertices of the base mesh, and may be stored in the displacement bitstream, so it does not need to be included in the syntax of the mesh patch data unit. In this case, step S104 is not performed. When using these, the mdu_vertex_count_minus1 syntax is deleted from the MDU and does not need to be decoded.
[0125] In step S106, the mesh patch decoder 207D decodes the remaining syntax in the MDU.
[0126] In step S106, the mesh patch decoder 207D decodes the parameters in the remaining MDU.
[0127] For example, in step S106, the mesh patch decoding unit 207D may decode the syntax used in decoding the displacement amount in step S107, which is displacement amount recovery parameter decoding, regardless of the video codec or arithmetic decoding.
[0128] This may include an override flag mdu_parameters_override_flag, a subdivision override flag mdu_subdivision_override_flag, a quantization override flag mdu_quantization_override_flag, a transform method override flag mdu_transform_method_override_flag, a transform parameter override flag mdu_transform_parameters_override_flag, a subdivision method mdu_subdivision_method, a subdivision iteration count mdu_subdivision_iteration_count, a displacement coordinate system mdu_displacement_coordinate_system, a transform method mdu_transform_method, etc.
[0129] Also, in step S106, the mesh patch decoding unit 207D may decode the position information of the patch of the attribute information (information for identifying the range in which the patch exists in the frame of the image of the decoded attribute) etc. in the attribute information parameter decoding unit in step 108.
[0130] This may include mdu_attribute_2d_pos_x[tileID][patchIdx][j] and mdu_attribute_2d_pos_y[tileID][patchIdx][j], and may also include mdu_attribute_2d_sice_x_minus1[tileID][patchIdx][j] and mdu_attribute_2d_sice_y_minus1[tileID][patchIdx][j], which are the quantized values of the patch width -1 and height -1.
[0131] The operation of the geometric video position information decoding step S105 will be described with reference to Figure 16(b). mdu_2d_pos_x and mdu_2d_pos_y, which respectively relate to the vertex coordinates x and y of the patch's existence range, are decoded from the atlas bitstream. mdu_2d_pos_x is multiplied by PatchPackingBlockSize, which is the block size used for packing into an image, and TileMeshpatch2dPosX, the x coordinate of the patch vertex, is output.
[0132] Similarly, mdu_2d_pos_y is multiplied by PatchPackingBlockSize, which is the block size when packing into an image, and TileMeshpatch2dPosY, which is the y coordinate of the patch vertex, is output.
[0133] Additionally, mdu_2d_size_x_minus1 and mdu_2d_size_y_minus1, which respectively relate to the width and height of the patch's existence range, are decoded from the atlas bitstream. For the width, 1 is added to mdu_2d_size_y_minus1.
[0134] PatchSizeXQuantizer, the X-direction quantization value of the patch size, is multiplied by mdu_2d_size_x_minus1+1 to output TileMeshpatch2dSizeX, the width of the patch. Similarly, for the height, 1 is added to mdu_2d_size_y_minus1.
[0135] PatchSizeYQuantizer, the Y-direction quantized value of the patch size, is multiplied by mdu_2d_size_y_minus1+1, and TileMeshpatch2dSizeY, the patch height, is output.
[0136] Next, with reference to FIG. 17, a process when the mesh patch mode is inter will be described.
[0137] As shown in FIG. 17(a), in the patch mode determination in step S101, the mesh patch decoding unit 207D in the atlas data decoding unit 207 determines whether the mesh patch mode (meshpatchMode) is inter (P_INTER).
[0138] If the mesh patch mode is inter, in step S102, the mesh patch decoder 207D decodes parameters related to the patch ID, which may include imdu_patch_index.
[0139] In determining the displacement codec in step S103, the mesh patch decoding unit 207D determines whether or not a geometric video exists, that is, whether or not the displacement is decoded by a video codec (whether or not displacement_codec_id is 0).
[0140] If a geometric video exists, that is, if the displacement is decoded by a video codec (displacement_codec_id is 0), proceed to step S104 and step S105. Otherwise, proceed to step S106.
[0141] In steps S104 and S105, the mesh patch decoding unit 207D decodes the difference in the number of vertices (imdu_delta_vertex_count_minus1) and the difference in the patch existence range (imdu_2d_delta_pos_x, imdu_2d_delta_pos_y, imdu_2d_delta_size, and imdu_2d_delta_size_y), respectively.
[0142] On the other hand, in step S103, if there is no geometric video, that is, if the displacement amount is not decoded by a video codec (displacement_codec_id is other than 0), the mesh patch decoding unit 207D does not decode the difference in the number of vertices (imdu_delta_vertex_count_minus1) and the difference in the patch existence range (imdu_2d_delta_pos_x, imdu_2d_delta_pos_y, imdu_2d_delta_size_x and imdu_2d_delta_size_y).
[0143] The number of vertices can be calculated from the number of vertices of the base mesh, and may also be stored in the displacement bitstream, so it does not need to be included in the syntax of the mesh patch data unit. In this case, the processing of step S104 is not performed. Furthermore, when using these, the imdu_delta_vertex_count_minus1 syntax is deleted from the IMDU and does not need to be decoded.
[0144] In step S106, the mesh patch decoder 207D decodes the syntax in the remaining IMDU.
[0145] For example, in step S106, the mesh patch decoding unit 207D may decode the difference in patch position information of the attribute information (information for identifying the range of existence of the patch in the frame of the image of the decoded attribute) by the attribute information parameter decoding in step S107.
[0146] This may include imdu_attribute_2d_delta_pos_x[tileID][patchIdx][j] and imdu_attribute_2d_delta_pos_y[tileID][patchIdx][j], and may also include imdu_attribute_2d_delta_size_x[tileID][patchIdx][j] and imdu_attribute_2d_delta_size_y[tileID][patchIdx][j], which relate to the patch width and height delta.
[0147] An example of the operation of decoding the patch identification ID in step S102 will be described with reference to FIG. 17(b).
[0148] In step S102, the prediction index PredictoIdx is added to the decoded inter patch index imdu_patch_index to obtain the reference patch index RefPatchIdx.
[0149] The operation of step S105 for decoding geometric video position information will be described with reference to FIG. 17(c).
[0150] Copy TileMeshpatch2dPosX and TileMeshpatch2dPosY with reference patch index RefPatchIdx to refMeshpatch2dPosX and refMeshpatch2dPosY, respectively.
[0151] Information imdu_2d_delta_pos_x and imdu_2d_delta_pos_y relating to the difference in vertex positions within the range of patch existence are decoded from the atlas bitstream.
[0152] The block size used for packing into an image, PatchPackingBlockSize, is multiplied by imdu_2d_delta_pos_x, and refMeshpatch2dPosX, the x-coordinate of the vertex of the reference patch, is added to it, and TileMeshpatch2dPosX, the x-coordinate of the patch vertex, is output.
[0153] Similarly, PatchPackingBlockSize is multiplied by imdu_2d_delta_pos_y, and refMeshpatch2dPosY, which is the y coordinate of the vertex of the reference patch, is added to it, and TileMeshpatch2dPosY, which is the y coordinate of the patch vertex, is output.
[0154] Furthermore, TileMeshpatch2dSize and TileMeshpatch2dSizeY having the reference patch index RefPatchIdx are copied to refMeshpatch2dSizeX and refMeshpatch2dSizeY, respectively. Here, in addition to TileMeshpatch2dPosX and TileMeshpatch2dPosY, TileMeshpatch2dSize and TileMeshpatch2dSizeY are set as reference tile mesh patch position information.
[0155] imdu_2d_delta_size_x and imdu_2d_delta_size_y, which are information relating to the width x and height y of the patch's existence range, are decoded from the atlas bitstream.
[0156] The PatchSizeXQuantizer, which is the X-direction quantized value of the patch size, is multiplied by mdu_2d_size_x, and refMeshpatch2dSizeX, which is the width of the reference patch, is added to it, and TileMeshpatch2dSizeX, which is the width of the patch, is output.
[0157] Similarly, mdu_2d_size_y is multiplied by PatchSizeYQuantizer, which is the Y-direction quantized value of the patch size, and refMeshpatch2dSizeY, which is the height of the reference patch, is added to it, and TileMeshpatch2dSizeY, which is the height of the patch, is output.
[0158] Secondly, with reference to FIG. 18, an example of the operation of the atlas sequence parameter set decoding unit 207A in the atlas data decoding unit 207 to decode ASVE will be described.
[0159] As shown in FIG. 18, in step S101, the displacement codec determination unit 207A1a of the ASVE decoding unit 207A1 in the atlas sequence parameter set decoding unit 207A determines whether geometric video exists, i.e., whether the displacement is decoded by a video codec (whether displacement_codec_id is 0 or not).
[0160] If a geometric video exists, that is, if the displacement is decoded by a video codec (displacement_codec_id is 0), proceed to step S102. Otherwise, proceed to step S103. In the geometric video packing method decoding in step S102, the ASVE decoding unit 207A1 in the atlas sequence parameter set decoding unit 207A decodes asve_packing_method, which is the image packing method.
[0161] In step S103, the atlas sequence parameter set decoding unit 207A decodes other syntax within the ASVE, and some of the other syntax may be decoded before the displacement codec determination in S101.
[0162] According to the mesh decoding device 200 of this embodiment, even when arithmetic decoding is used, unnecessary information is not signaled, so that the amount of code can be reduced.
[0163] The image packing method is described as the syntax of ASVE, but it is the same even if it is moved to another parameter set. For example, even if the image packing method is moved to the displacement parameter set of DPS, the same processing can be performed by replacing ASVE with DPS.
[0164] The mesh encoding device 100 and the mesh decoding device 200 described above may be realized as a program that causes a computer to execute each function (each step). [Industrial Applicability]
[0165] According to this embodiment, for example, it is possible to improve the overall service quality in video communication, which makes it possible to contribute to Goal 9 of the Sustainable Development Goals (SDGs) led by the United Nations, which is to "Develop resilient infrastructure, promote sustainable industrialization and foster innovation." [Explanation of symbols]
[0166] 1. Mesh processing system 100...Mesh coding device 200...Mesh decoding unit 201…Demultiplexer 202...Basic mesh decoding unit 203…Subdivision part 204...Mesh decoding unit 205...Displacement amount decoding unit 205A...Arithmetic decoding unit 205B...Video codec decoding unit 205C...Image development section 205D...Dequantization section 205H…Reverse conversion unit 206...Video decoding unit 207...Atlas data decoding unit 207A... Atlas sequence parameter set decoding unit 207B... Atlas frame parameter set decoding unit 207C...Atlas Style Decoding Section 207D…Mesh patch decoding unit 207A1…ASVE decoding section 207A1a...Displacement amount codec determination unit 207A1b...Geometric video packing method decoding unit 207A1c...Other syntax decoding units in ASVE 207D1...Patch mode determination section 207Dm...MDU decoding section 207Dm1...Displacement codec determination section 207Dm2...Geometric video position information decoding unit 207Dm3...Displacement amount decoding parameter decoding unit 207Dm4...Attribute information parameter decoding section 207Di…MDU decoding unit 207Di1...Displacement codec determination unit 207Di2...Geometric video position information decoding unit 207Di3...Attribute information parameter decoding unit
Claims
1. A mesh decoding device, comprising: an atlas decoding unit configured to decode and output an atlas bitstream; the atlas decoder comprises a mesh patch decoder configured to decode patches; The mesh patch decoding unit a displacement codec determination unit that determines whether the displacement is decoded by a video codec; a geometric video position information decoding unit that, when it is determined that the amount of displacement is decoded by the video codec, decodes information for specifying an existing range of a patch in a video frame of the decoded amount of displacement; The mesh decoding device is characterized in that the geometric video position information decoding unit does not decode information for identifying the range of existence of a patch in a frame of video of the decoded displacement amount when it is determined that the displacement amount is not decoded by the video codec.
2. The mesh decoding device of claim 1, characterized in that the information for identifying the range of existence of the patch in the frame of the image of the decoded displacement amount includes vertex coordinates indicating any one point of a rectangle or quantized versions of the vertex coordinates, and the width and height of the rectangle or quantized versions of the width and height of the rectangle.
3. The mesh decoding device according to claim 1, characterized in that the information for identifying the range of existence of the patch in the frame of the image of the decoded displacement includes the starting position of the block and the number of vertices of the last block.
4. A mesh decoding device, comprising: an atlas decoding unit configured to decode and output an atlas bitstream; the atlas decoder comprises a mesh patch decoder configured to decode patches; The mesh patch decoding unit a displacement codec determination unit that determines whether the displacement is decoded by a video codec; a geometric video position information decoding unit that decodes the number of vertices in a video frame of the decoded displacement amount when it is determined that the displacement amount is decoded by the video codec; The mesh decoding device is characterized in that the geometric video position information decoding unit does not decode the number of vertices in the frame of the video of the decoded displacement amount when it is determined that the displacement amount is not decoded by the video codec.
5. A mesh decoding device, comprising: an atlas decoding unit configured to decode and output an atlas bitstream; the atlas decoding unit comprises an atlas sequence parameter set decoding unit that decodes an atlas sequence parameter set; The atlas sequence parameter set decoding unit a displacement codec determination unit that determines whether the displacement is decoded by a video codec; a geometric video packing method decoding unit that decodes a packing method of the decoded displacement amount when it is determined that the displacement amount is decoded by the video codec, The mesh decoding device, characterized in that the geometric video packing method decoding unit does not decode a packing method of the decoded displacement amount when it is determined that the displacement amount is not decoded by the video codec.
6. 1. A mesh decoding method, comprising: decoding and outputting the atlas bitstream; The steps include a step A of decoding the patch; The step A comprises: A step A1 of determining whether the displacement is decoded by a video codec; a step A2 of decoding information for specifying a patch existence range in a video frame of the decoded displacement amount when it is determined that the displacement amount is decoded by the video codec; A mesh decoding method characterized by having a step A3 of not decoding information for identifying the range of existence of a patch in a frame of video of the decoded displacement amount when it is determined that the displacement amount is not decoded by the video codec.
7. A program that causes a computer to function as a mesh decoding device, the mesh decoding device comprises an atlas decoding unit configured to decode and output an atlas bitstream; the atlas decoder comprises a mesh patch decoder configured to decode patches; The mesh patch decoding unit a displacement codec determination unit that determines whether the displacement is decoded by a video codec; a geometric video position information decoding unit that, when it is determined that the amount of displacement is decoded by the video codec, decodes information for specifying an existing range of a patch in a video frame of the decoded amount of displacement; The program is characterized in that the geometric video position information decoding unit does not decode information for identifying the range of existence of a patch in a frame of video of the decoded displacement amount when it is determined that the displacement amount is not decoded by the video codec.