Mesh decoding device, mesh decoding method, and program

Abstract

The present invention improves encoding efficiency. A basic mesh decoding unit 202 of a mesh decoding device 200 according to the present invention comprises: a control signal acquisition unit 202F that acquires a control signal; and a determination unit 202G that determines whether to decode inter-prediction related information from the basic mesh bit stream by a control signal.

Description

Mesh decoding device, mesh decoding method and program

[0001] The present invention relates to a mesh decoding device, a mesh decoding method, and a program.

[0002] Non-Patent Document 4 discloses a technology that utilizes the framework of Non-Patent Document 5 to decode a mesh by dividing it into a rough base mesh and a detailed displacement amount, decodes information such as vertex coordinates, connectivity, and UV coordinates (a type of attribute) for the base mesh using a base mesh decoding unit, and reconstructs a mesh from the base mesh and displacement amount by adding the decoded atlas using an atlas data decoding unit.

[0003] In addition, the decoding of the above-mentioned basic mesh uses any one of intra-frame decoding, inter-frame decoding, and skip-frame decoding.

[0004] Non-Patent Document 4 discloses that in addition to meshes, video data called textures are decoded by a video decoder as a type of Attribute.

[0005] Khaled Mammou, Jungsun Kim, Alexis M Tourapis, Dimitri Podborski, and Krasimir Kolarov, “[V-CG] Apple's Dynamic Mesh Coding CfP Response,” April 2022, ISO / IEC JTC 1 / SC 29 / WG 7 m59281.Google Draco, accessed May 26, 2022 [Online], https: / / google.github.io / dracoJean-Eudes Marvie, Olivier Mocquard, “[V-DMC][EE4.4-related] An efficient EdgeBreaker implementation,” April 2023, ISO / IEC JTC 1 / SC 29 / WG 7 m63344. “Text of ISO / IEC CD 23090-29 Video-based mesh coding,” ISO / IEC JTC 1 / SC 29 / WG7 N00885, April 2024. “Information technology - Coded Representation of Immersive Media - Part 5: Visual Volumetric Video-based Coding (V3C) and Video-based Point Cloud Compression (V-PCC),” ISO / IEC JTC 1 / SC 29 / WG 7, ISO / IEC 23090-5:2021(2E).

[0006] However, non-patent document 4 has the problem that when there is only one frame in the basic mesh (when bmptc_one_mesh_frame_only_flag is 1) or when all frames are intra-frames (when bmptc_intra_frames_only_flag is 1), unnecessary information such as inter-prediction and reference frame-related syntax is coded and decoded, resulting in poor coding efficiency.

[0007] Furthermore, there is a problem in that a flag for only one frame in the basic mesh (bmptc_one_mesh_frame_only_flag) and a flag for only one frame in the atlas (ptc_one_v3c_frame_only_flag) may conflict.

[0008] Therefore, the present invention has been made in consideration of the above-mentioned problems, and has an object to provide a mesh decoding device, a mesh decoding method, and a program that can improve coding efficiency.

[0009] Another object of the present invention is to provide a mesh decoding device, a mesh decoding method and a program that can avoid conflicts between the basic mesh control signal bmptc_one_mesh_frame_only_flag and the atlas control signal ptc_one_v3c_frame_only_flag.

[0010] A first feature of the present invention is a mesh decoding device comprising an atlas data decoding unit configured to decode and output an atlas bitstream, and a base mesh decoding unit configured to decode and output a base mesh bitstream, wherein the base mesh decoding unit comprises a control signal acquisition unit that acquires a control signal, and a judgment unit that judges, based on the control signal, whether to decode inter-prediction related information from the base mesh bitstream.

[0011] A second feature of the present invention is a mesh decoding method comprising a step A of decoding and outputting an atlas bitstream, and a step B of decoding and outputting a base mesh bitstream, wherein the step B comprises a step of acquiring a control signal, and a step of determining, based on the control signal, whether to decode inter-prediction related information from the base mesh bitstream.

[0012] A third feature of the present invention is a program that causes a computer to function as a mesh decoding device, the mesh decoding device comprising an atlas data decoding unit configured to decode and output an atlas bitstream, and a basic mesh decoding unit configured to decode and output a basic mesh bitstream, and the basic mesh decoding unit comprising a control signal acquisition unit that acquires a control signal, and a judgment unit that judges whether to decode inter-prediction related information from the basic mesh bitstream based on the control signal.

[0013] According to the present invention, it is possible to provide a mesh decoding device, a mesh decoding method and a program that can improve coding efficiency.

[0014] Furthermore, according to the present invention, it is possible to provide a mesh decoding device, a mesh decoding method and a program that can avoid conflicts between the basic mesh control signal bmptc_one_mesh_frame_only_flag and the atlas control signal ptc_one_v3c_frame_only_flag.

[0015] FIG. 1 is a diagram showing an example of the configuration of a mesh processing system 1 according to an embodiment. FIG. 2 is a diagram showing an example of functional blocks of a mesh decoding device 200 according to an embodiment. FIG. 3 is a diagram showing an example of a base mesh and a subdivision mesh. FIG. 4 is a diagram showing an example of functional blocks of a base mesh decoding unit 202 of the mesh decoding device 200 according to an embodiment. FIG. 5 is a diagram showing an example of functional blocks of an intra decoding unit 202B of the base mesh decoding unit 202 of the mesh decoding device 200 according to an embodiment. FIG. 6 is a diagram showing an example of a correspondence relationship between vertices of a base mesh of a P frame and vertices of a base mesh of an I frame. FIG. 7 is a diagram showing an example of the configuration of a base mesh bitstream. FIG. 8 is a diagram showing an example of a control signal for imposing restrictions on a profile toolset. FIG. 9 is a diagram showing an example of a BMSPS. FIG. 10 is a diagram showing an example of a BMFPS. FIG. 11 is a diagram showing an example of a BMSH. FIG. 12 is a diagram showing an example of a BMSPS. FIG. 13 is a diagram showing an example of a BMFPS. FIG. 14 is a diagram showing an example of a BMSH. FIG. 15 is a diagram showing an example of a control signal related to the Attributes of a base mesh. Fig. 16 is a diagram showing an example of functional blocks of the inter decoding unit 202E. Fig. 17 is a diagram for explaining an example of a method for calculating the MVP of a vertex to be decoded by the motion vector prediction unit 202E3 of the inter decoding unit 202 of the basic mesh decoding unit 200 according to an embodiment. Fig. 18 is a diagram for explaining the mesh buffer unit 202C of the basic mesh decoding unit 202 of the mesh decoding device 200 according to an embodiment. Fig. 19 is a diagram showing an example of a NAL header. Fig. 20 is a diagram showing an example of a BMSPS.

[0016] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the components in the following embodiments can be appropriately replaced with existing components, etc., and various variations, including combinations with other existing components, are possible. Therefore, the description of the following embodiments does not limit the content of the invention described in the claims.

[0017] First Embodiment A mesh processing system according to this embodiment will be described below with reference to FIGS.

[0018] 1 is a diagram showing an example of the configuration of a mesh processing system 1 according to this embodiment. As shown in FIG. 1, the mesh processing system 1 includes a mesh encoding device 100 and a mesh decoding device 200.

[0019] FIG. 2 is a diagram showing an example of functional blocks of a mesh decoding device 200 according to this embodiment.

[0020] As shown in Figure 2, the mesh decoding device 200 has a demultiplexing unit 201, a basic mesh decoding unit 202, a subdivision unit 203, a mesh decoding unit 204, a patch integration unit 205, a displacement amount decoding unit 206, a video decoding unit 207, and an atlas data decoding unit 208.

[0021] Here, the basic mesh decoding unit 202, the subdivision unit 203, the mesh decoding unit 204 and the displacement amount decoding unit 206 are configured to perform processing in units of patches into which the mesh is divided, and the results of these processes may then be integrated by the patch integration unit 205.

[0022] In the example of FIG. 3, the mesh is divided into a patch 1 made up of basic faces 1 and 2 and a patch 2 made up of basic faces 3 and 4.

[0023] The demultiplexing unit 201 is configured to separate the multiplexed bit stream into a base mesh bit stream, a displacement bit stream, a texture bit stream, and an atlas bit stream.

[0024] The subdivision unit 203 is configured to generate and output subdivision vertices and their connection information from the basic mesh decoded by the basic mesh decoding unit 202, using the subdivision method indicated by the control information (first control information and second control information). The basic mesh is composed of one submesh or multiple submeshes.

[0025] Here, the basic mesh, the added subdivision vertices, and the connection information therebetween are collectively referred to as the "subdivision mesh." Also, the submesh, the added subdivision vertices, and the connection information therebetween are collectively referred to as the "subdivision submesh."

[0026] The mesh decoding unit 204 is configured to generate and output a decoded mesh using the control information, the subdivision mesh, the subdivision vertex normals, and the displacement amounts.

[0027] The displacement amount decoding unit 206 is configured to decode the displacement amount bit stream based on the control information to generate and output the displacement amount.

[0028] The video decoding unit 207 is configured to decode the texture using a video codec and output the decoded texture.

[0029] The atlas data decoder 208 is configured to decode the atlas bitstream and output control information, which may be used as metadata by the base mesh decoder 202, the subdivision unit 203, the mesh decoder 204, the displacement decoder 206, and the video decoder 207.

[0030] <Basic Mesh Decoding Unit 202> The basic mesh decoding unit 202 is configured to decode the basic mesh bitstream, generate a basic mesh, and output it.

[0031] Here, the basic mesh is made up of a plurality of vertices in a three-dimensional space and edges connecting these vertices.

[0032] The base mesh decoder 202 may be configured to decode the base mesh bitstream using, for example, the techniques described in Draco in Non-Patent Document 2 or in Non-Patent Document 3.

[0033] As shown in FIG. 4, the basic mesh decoding unit 202 includes a separating unit 202A, an intra decoding unit 202B, a mesh buffer unit 202C, a connection information decoding unit 202D, and an inter decoding unit 202E.

[0034] (Separator 202A) The separator 202A is configured to classify the basic mesh bitstream into a bitstream of I frames and a bitstream of P frames.

[0035] (Intra decoding unit 202B) The intra decoding unit 202B is configured to decode the coordinates and connection information of the vertices of the I frame from the bit stream of the I frame, for example, by using Draco shown in Non-Patent Document 2 or the technology described in Non-Patent Document 3.

[0036] FIG. 5 is a diagram showing an example of functional blocks of the intra decoder 202B.

[0037] As shown in FIG. 5, the intra decoding unit 202B includes an arbitrary intra decoding unit 202B1 and an alignment unit 202B2.

[0038] The optional intra decoder 202B1 is configured to decode the coordinates and connectivity information of the unordered vertices of the I-frame from the bitstream of the I-frame using any method including Draco shown in Non-Patent Document 2 or the technology described in Non-Patent Document 3.

[0039] The sorting unit 202B2 is configured to output the vertices by sorting the unordered vertices into a predetermined order.

[0040] The predetermined order may be, for example, a Morton code order or a raster scan order.

[0041] Furthermore, the alignment unit 202B2 may combine overlapping vertices, which are multiple vertices with the same coordinates in the decoded basic mesh, into a single vertex, and then rearrange them in a predetermined order.

[0042] (Mesh Buffer Unit 202C) The mesh buffer unit 202C is configured to store the coordinates and connection information of the vertices of the I frame decoded by the intra decoding unit 202B. Here, a specific buffer may be provided to store pairs of vertex indices A(k) and B(k) of overlapping vertices in a predetermined order.

[0043] (Connection Information Decoding Unit 202D) The connection information decoding unit 202D is configured to convert the connection information of the I frame or reference frame extracted from the mesh buffer unit 202C into connection information of a P frame.

[0044] (Inter decoding unit 202E) The inter decoding unit 202E is configured to decode the coordinates of the vertices of the P frame by adding the coordinates of the vertices of the reference frame retrieved from the mesh buffer unit 202C and the motion vectors decoded from the bit stream of the P frame.

[0045] Furthermore, the inter-decoding unit 202E can adjust the index of the vertex of the P frame using the pair of vertex indices A(k) and B(k) that exist as overlapping vertices stored in the specific buffer.

[0046] In this embodiment, there is a correspondence between the vertices of the base mesh of the P frame and the vertices of the base mesh of the reference frame (I frame or P frame), as shown in Figure 6. Here, the motion vector decoded by the inter decoding unit 202E is a difference vector between the coordinates of the vertices of the base mesh of the P frame and the coordinates of the vertices of the base mesh of the I frame.

[0047] (Configuration of Basic Mesh Bit Stream) Hereinafter, an example of the configuration of a basic mesh bit stream will be described with reference to FIGS.

[0048] FIG. 7 is a diagram showing an example of the configuration of a basic mesh bit stream.

[0049] 8 to 13, u(n) means an n-bit code, and ue(v) means an unsigned variable-length zeroth-order exponential-Golomb code.

[0050] As shown in Figure 7, the base mesh bitstream may include BMSPS (Basemesh Sequence Parameter Set), BMFPS (Basemesh Frame Parameter Set), BMSH (Basemesh Submesh Header), and BMSDU (Basemesh Submesh Data Unit), which are sets of control information related to the decoding of the base mesh.

[0051] As shown in FIG. 7, the BMSPS may include Profile-related syntax.

[0052] Furthermore, as shown in FIG. 8, the Profile-related syntax may include control signals that impose restrictions on the profile toolset, such as bmptc_one_mesh_frame_only_flag and bmptc_intra_frames_only_flag.

[0053] bmptc_one_mesh_frame_only_flag is a flag (first control signal) that indicates that the number of frames in the basic mesh is 1.

[0054] bmptc_intra_frames_only_flag is a flag (second control signal) indicating that the sub-mesh of the basic mesh is an intra-frame (I-frame) only.

[0055] In addition to the Profile-related syntax, the BMSPS may also include control signals used in the inter-decoding unit 202E and reference frame-related control signals, as shown in Figure 9, such as bmsps_inter_mesh_max_num_neighbors_minus1, which indicates the maximum number of adjacent vertices for predicting motion vectors, and bmesh_ref_list_struct(), the reference frame list Structure.

[0056] As shown in FIG. 10, the BMFPS may include a reference frame-related control signal, for example, bmfps_num_ref_idx_default_active_minus1 indicating an active reference frame list Structure.

[0057] As shown in FIG. 11, the BMSH may include control signals related to reference frames in sub-meshes, such as bmsh_ref_mesh_frame_list_msps_flag and bmsh_ref_mesh_frame_list_idx.

[0058] (Redundancy Removal) However, the amount of code increases in the basic mesh bit stream due to unnecessary information being coded by the control signal of the profile.

[0059] For example, when bmtpc_one_mesh_frame_only_flag or bmtpc_intra_frames_only_flag is 1, there are no P frames, and therefore it is not necessary to encode information related to motion vectors or reference frame related information in BMSPS, BMFPS, BMSH, etc. In this embodiment, such information is called inter prediction related information.

[0060] Therefore, when bmtpc_one_mesh_frame_only_flag or bmtpc_intra_frames_only_flag is 1, it is necessary to control so that inter prediction related information is not signaled.

[0061] In this embodiment, in order to control so that inter-prediction related information is not signaled, as shown in Figure 4, the basic mesh decoding unit 202 further includes a control signal acquisition unit 202F that acquires a control signal, and a judgment unit 202G that judges whether to decode the inter-prediction related information based on the control signal acquired by the control signal acquisition unit 202F.

[0062] The control signal acquisition unit 202F may acquire bmtpc_one_mesh_frame_only_flag or bmtpc_intra_frames_only_flag.

[0063] For example, if bmtpc_one_mesh_frame_only_flag or bmtpc_intra_frames_only_flag is 1, inter prediction related information may not be signaled.

[0064] Therefore, when bmtpc_one_mesh_frame_only_flag or bmtpc_intra_frames_only_flag is 1, the determination unit 202G determines not to decode the inter prediction related information.

[0065] However, when the inter prediction related information is not signaled, the determining unit 202G may set the value of the inter prediction related information to a predetermined value.

[0066] Alternatively, the control signal acquisition unit 202F may acquire bmsps_enable_prediction_flag (third control signal) indicating that inter prediction information is to be coded.

[0067] For example, if bmsps_enable_prediction_flag is 0, no inter-prediction related information may be signaled.

[0068] Therefore, when bmsps_enable_prediction_flag is 0, the determining unit 202G determines not to decode the inter prediction related information.

[0069] However, when the inter prediction related information is not decoded, the determining unit 202G may set the value of the inter prediction related information to a predetermined value.

[0070] Note that the control signal acquisition unit 202F may calculate bmsps_enable_prediction_flag using Equation 1. In this case, bmsps_enable_prediction_flag is not signaled.

[0071] bmsps_enable_prediction_flag=Not(bmtpc_one_mesh_frame_only_flag Or bmtpc_intra_frames_only_flag) (Equation 1) In other words, when bmtpc_one_mesh_frame_only_flag is 1, the control signal acquisition unit 202F sets bmsps_enable_prediction_flag to 0.

[0072] Furthermore, when bmtpc_intra_frames_only_flag is 1, the control signal acquisition unit 202F sets bmsps_enable_prediction_flag to 0.

[0073] Alternatively, when the bmsps_enable_prediction_flag is signaled, the bmsps_enable_prediction_flag may be included in the syntax of the BMSPS, and the control signal acquirer 202F may decode the bmsps_enable_prediction_flag from the BMSPS, as shown in Fig. 12. In this case, the bmsps_enable_prediction_flag is signaled.

[0074] If bmsps_enable_prediction_flag does not exist in the bitstream, the control signal acquisition unit 202F sets bmsps_enable_prediction_flag to 1.

[0075] However, if bmsps_enable_prediction_flag is 1, the control signal acquisition unit 202F sets bmsps_enable_prediction_flag to 0.

[0076] Furthermore, when bmtpc_intra_frames_only_flag is 1, the control signal acquisition unit 202F sets bmsps_enable_prediction_flag to 0.

[0077] For example, the inter prediction related information is a control signal (for example, BMSPS, BMFPS, or BMSH) as shown in FIGS.

[0078] However, the determining unit 202G may make a determination using the frame type in the BMSH shown in FIG.

[0079] According to this configuration, the inter prediction related information is coded only when necessary, which is expected to improve coding efficiency.

[0080] In addition, a conformance requirement is provided that the value of the flag indicating that there is only one frame in the basic mesh bitstream (control signal bmptc_one_mesh_frame_only_flag in non-patent document 4) and the value of the flag indicating that there is only one frame in the atlas (control signal ptc_one_v3c_frame_only_flag in non-patent document 5) are the same.

[0081] With this configuration, it is expected that the conflict between the bmptc_one_mesh_frame_only_flag and the ptc_one_v3c_frame_only_flag can be avoided.

[0082] (Attribute count / index related information) Hereinafter, an example of a control signal related to the Attribute of a basic mesh in an atlas bit stream and a basic mesh bit stream will be described with reference to FIG.

[0083] The atlas bitstream may include a first control signal vps_ext_bmesh_data_attribute_count indicating the number of Attributes in the base mesh, as shown in FIG.

[0084] The atlas bitstream may also include a second control signal vps_ext_bmesh_attribute_type that indicates the type of each Attribute of the base mesh.

[0085] The BMSPS of the basic mesh bitstream may include a third control signal bmsps_mesh_attribute_count indicating the number of Attributes of the basic mesh, as shown in FIG.

[0086] In addition, the BMSPS of the basic mesh bitstream may include a fourth control signal bmsps_mesh_attribute_type_id indicating the type of each Attribute of the basic mesh.

[0087] In addition, the BMSPS of the basic mesh bitstream may include a fifth control signal bmsps_mesh_attribute_index indicating the index of each Attribute of the I frame.

[0088] In Non-Patent Document 4, the control signal acquisition unit 202F may decode vps_ext_bmesh_data_attribute_count using u(8). vps_ext_bmesh_data_attribute_count has a range of 0 to 255. That is, the maximum value of vps_ext_bmesh_data_attribute_count is 255.

[0089] Therefore, if the Attribute in Non-Patent Document 4 is a 4096×4096 video, 48 MB is required for just one frame of Attribute.

[0090] Here, the product memory required is proportional to the number of Attributes. If the maximum number of Attributes is 255, even if each Attribute only stores one frame, a maximum of 12,240 MB of memory may be required. Therefore, the product must have at least 12 GB of memory.

[0091] On the other hand, if the maximum number of Attributes is 127, then when saving one frame for each Attribute, a maximum of 6,096 MB of memory will be sufficient, which is expected to reduce the required memory capacity by half.

[0092] Note that a certain number is required for vps_ext_bmesh_data_attribute_count defined in Non-Patent Document 5. The minimum value is equal to or greater than the control signal ai_attribute_count, which indicates the number of textures decoded by the video decoding unit 207.

[0093] On the other hand, the control signal acquisition unit 202F may decode the ai_attribute_count using u(7). The ai_attribute_count has a range of 0 to 127. That is, the maximum value of the ai_attribute_count is 127.

[0094] Therefore, in this embodiment, the minimum value of vps_ext_bmesh_data_attribute_count is set to the maximum value of ai_attribute_count, making the range as small as possible.

[0095] For example, the control signal acquisition unit 202F may decode the vps_ext_bmesh_data_attribute_count with u(7). In this case, the vps_ext_bmesh_data_attribute_count has a range of 0 to 127.

[0096] If the maximum value of ai_attribute_count is redefined elsewhere, such as in a profile or level, the minimum value of vps_ext_bmesh_data_attribute_count may be set to the redefined maximum value, making the range as small as possible.

[0097] For example, if the maximum value of ai_attribute_count is set to MAX_ATTRIBUTE_COUNT (a predetermined value, for example, 15) in a certain profile or level, the maximum value of vps_ext_bmesh_data_attribute_count may be set to MAX_ATTRIBUTE_COUNT (for example, 15) or a value slightly larger than MAX_ATTRIBUTE_COUNT (for example, 31).

[0098] In addition, the fifth control signal bmsps_mesh_attribute_index, which indicates the index of each Attribute of the I frame of the basic mesh, has a range of 0 to 127.

[0099] Therefore, the basic mesh decoding unit 202 may decode the fifth control signal bmsps_mesh_attribute_index, which indicates the index of each Attribute of the I frame of the basic mesh, using u(7).

[0100] In addition, the possible values ​​of the second control signal indicating the type of each Attribute of the basic mesh of the atlas bitstream and the fourth control signal indicating the type of each Attribute of the basic mesh of the basic mesh bitstream may be the same.

[0101] For example, an example of the atlas bitstream and the basic mesh bitstream is shown in Table 1 or Table 2.

[0102]

[0103] According to this configuration, by rationalizing the maximum value of vps_ext_bmesh_data_attribute_count, it is possible to expect the effect of making it possible to manufacture products while significantly reducing product costs.

[0104] (Inter Decoding Unit 202E) FIG. 16 is a diagram showing an example of functional blocks of the inter decoding unit 202E.

[0105] As shown in FIG. 16, the inter decoding unit 202E has a motion vector residual decoding unit 202E1, a motion vector buffer unit 202E2, a motion vector prediction unit 202E3, a motion vector calculation unit 202E4, and an adder 202E5.

[0106] The motion vector residual decoding unit 202E1 is configured to generate a motion vector residual (MVR) from a P frame bitstream.

[0107] Here, MVR is a motion vector residual indicating the difference between MV (Motion Vector) and MVP (Motion Vector Prediction). MV is a difference vector (motion vector) between the coordinates of a vertex of the corresponding I frame and the coordinates of a vertex of the corresponding P frame. MVP is a predicted value of the MV of the target vertex using MV (predicted value of the motion vector).

[0108] The motion vector buffer unit 202E2 is configured to sequentially store the motion vectors output by the motion vector calculation unit 202E4.

[0109] The motion vector prediction unit 202E3 is configured to obtain decoded MVs from the motion vector buffer unit 202E2 for vertices connected to the vertex to be decoded, and to output the MVP of the vertex to be decoded using all or part of the obtained decoded MVs, as shown in Figure 17.

[0110] The motion vector calculation unit 202E4 is configured to add the MVR generated by the motion vector residual decoding unit 202E1 and the MVP output from the motion vector prediction unit 202E3, and output the MV of the vertex to be decoded.

[0111] The adder 202E5 is configured to add the coordinates of the vertex corresponding to the vertex to be decoded, obtained from the decoded basic mesh of the corresponding reference frame (I frame or P frame), to the motion vector MV output from the motion vector calculation unit 202E3, and output the coordinates of the vertex to be decoded.

[0112] (Mesh Buffer Unit 202C) The mesh buffer unit 202C is configured to store one or more reference decoded basic meshes in a predetermined order.

[0113] Such a basic mesh has metadata such as a frame number and a submesh number, as well as at least the coordinates of each vertex and the index of that vertex, and is stored in the mesh buffer unit 202C in a predetermined order determined by the reference frame list.

[0114] As shown in FIG. 18, the reference frame list (ref_list0) is a list of information that identifies all reference decoding basic meshes stored in the mesh buffer unit 202C.

[0115] The reference frame list may be determined by control signals decoded from the bitstream, as shown in FIG. 18, or may be calculated naturally from the decoding order of the frames.

[0116] The control signal decoded from the bitstream may be represented by a relative distance from the frame to be decoded, or may be represented by a frame index that is an absolute value.

[0117] Additionally, the control signal may use a short-term or long-term frame of reference.

[0118] For example, when a short-term reference frame is used, the absolute value (abs_delta_mfoc_st) of the difference in display order (Display Order) between the current frame (cur) and the reference frame (ref) and its sign (sign_flag) may be decoded from the bitstream, and the display order (Display Order) of the reference frame may be specified by the following formula: If(sign_flag) { Display Order(ref) = Display Order(cur) + abs_delta_mfoc_st} else { Display Order(ref) = Display Order(cur) - abs_delta_mfoc_st} Furthermore, when a method of calculating naturally from the decoding order of frames is used, for example, when no control signal is present in the reference frame list, frames may be arranged in order of a certain number of frames starting from the most recently decoded frame. That is, the reference frame list may be set to {0, -1, -2, ..., -(N-1)}.

[0119] Basically, the reference frame list does not change for each frame except in special circumstances (for example, when a re-ordering instruction is received).

[0120] The mesh buffer unit 202C may be updated as follows.

[0121] When a basic mesh is decoded, in the case of an I frame or a P frame, the mesh buffer unit 202C deletes one or more existing reference frames in a predetermined order determined by the reference frame list, and inserts one or more basic meshes including the basic mesh of the decoded frame, or creates and inserts one basic mesh from multiple basic meshes, thereby adjusting the order of the reference frames.

[0122] This deletion operation may be performed only when the mesh buffer unit 202C is full. The number of basic meshes that can be stored in the mesh buffer unit 202C is predetermined. In this embodiment, the mesh buffer unit 202C is defined as being full when the number of basic meshes is reached.

[0123] In the above-described creation process, a single basic mesh may be created by weighting the coordinates of the vertices corresponding to the basic meshes of the decoded frame and the existing basic meshes stored in the mesh buffer unit 202C.

[0124] The weights used in such a weighted average may be predetermined, calculated using the frame index, or decoded from the control signal.

[0125] In addition, when the mesh buffer unit 202C receives a control signal indicating a re-ordering instruction from a control signal decoded from the bitstream, it updates the reference frame list as shown in Figure 14 and adjusts the order of the reference frames according to the specified order determined by the updated reference frame list (ref_list0).

[0126] When sub-meshes as defined in the above-mentioned non-patent document 4 exist, all sub-meshes are set to the same control signal (smh_mesh_from_order_cnt_lsb) or the control signal (smh_mesh_from_order_cnt_lsb) is applied to all sub-meshes.

[0127] The value indicated by this control signal (smh_mesh_from_order_cnt_lsb) may be a difference from the display order of the frame to be decoded, or may be an order within a predetermined frame group MaxMeshFrmOrderCntLsb.

[0128] In addition, when the decoding order (Decode Order) and the display order (Display Order) are different, if the decoded basic meshes are arranged in the decoding order (Decode Order), the basic mesh decoding unit 202 may rearrange the decoded basic meshes in the display order (Display Order).

[0129] In order to achieve temporal scalability, control signals indicating whether or not to decode the basic mesh, the displacement, and the texture are defined for each frame, and are decoded from the bitstream.

[0130] Furthermore, the Temporal_ID of the atlas and the base mesh may be matched within the same frame. Furthermore, the Temporal_ID of the atlas and the texture may be matched within the same frame. Furthermore, the Temporal_ID of the atlas and the displacement amount may be matched within the same frame.

[0131] According to this configuration, it is expected that it will be possible to avoid frames that cannot be decoded and unnecessary data.

[0132] It is desirable that the interval between adjacent frames having the same Temporal_ID is constant.

[0133] Adjacent frames with the same Temporal_ID have the closest POC.

[0134] By keeping the frame intervals constant as described above, it is expected that a constant frame rate can be maintained when displaying decoded frames.

[0135] Furthermore, the decoding orders of atlases and base meshes having the same display order may be matched, the decoding orders of atlases and displacements having the same display order may be matched, and the decoding orders of atlases and textures having the same display order may be matched.

[0136] Alternatively, the random access points of an atlas and a base mesh that have the same display order may be matched. Alternatively, the random access points of an atlas and a displacement that have the same display order may be matched. Alternatively, the random access points of an atlas and a texture that have the same display order may be matched. Note that random access points are defined in Non-Patent Document 4 or Non-Patent Document 5.

[0137] According to this configuration, when decoding the basic mesh, the displacement amount, and the texture, it is possible to expect the effect that the mesh can be reproduced without waiting for each other to be decoded.

[0138] Furthermore, a frame having a Temporal_ID higher than the control signal Temporal_ID of the current frame to be decoded is not used as a reference frame for the current frame to be decoded.

[0139] This is expected to have the effect of eliminating the possibility of reference frames being discarded.

[0140] An example of realizing temporal scalability using the above-mentioned Temporal_ID will be described below.

[0141] The atlas, base mesh, displacement, and texture bitstreams are encapsulated by Network Abstraction Layer (NAL) units, which may have a NAL header as shown in Figure 22.

[0142] The TID, defined as the last 3 bits in the NAL header, is the Temporal_ID plus 1. The range of TID is 1 to 7, with zero being prohibited.

[0143] LayerID / R6, defined as the 6 bits immediately preceding the TID in the NAL header, specifies the identifier of the layer to which the NAL unit belongs.

[0144] The value of LayerID / R6 must be in the range of 0 to 62. The value 63 may be specified by ISO / IEC in the future.

[0145] For purposes other than determining the amount of data in a decoded unit of the bitstream, the mesh decoding device 200 ignores all data following the value 63 in a NAL unit, and a mesh decoding device 200 conforming to the specified profile will ignore (i.e., remove from the bitstream and discard) all NAL units whose LayerID-R6 value is not 0.

[0146] The value 63 of LayerID / R6 can be used to indicate an enhancement layer identifier in future enhancements.

[0147] When sub-meshes defined in Non-Patent Document 4 exist, all sub-meshes are assigned the same TID or the TID is applied to all sub-meshes.

[0148] Non-Patent Document 5 can be used for the atlas, and HEVC or VVC, a video coding method, can be used for the displacement and texture, so the following will explain the basic mesh.

[0149] As shown in FIG. 20, the BMSPS of the basic mesh bitstream may include a control signal bmsps_max_sub_layers_minus1 indicating the maximum number of Temporal sub-layers at u(3).

[0150] It may also include a control signal bmsps_max_dec_mesh_frame_buffering_minus1 indicating the buffer size of the maximum basic mesh in each Temporal sublayer and a control signal bmsps_max_num_reorder_frames indicating the difference from the display order of the maximum frame to be decoded.

[0151] The LayerID / R6 values ​​of all BMCL NAL units of a coded basic mesh frame must be the same. The LayerID / R6 value of a coded basic mesh frame is the LayerID / R6 value of the BMCL NAL unit of the coded basic mesh frame.

[0152] If NALType is equal to NAL_EOB, the value of LayerID / R6 must be equal to 0.

[0153] If the NALType is in the range from NAL_BLA_W_LP to NAL_RSV_BMCL_29 defined in Non-Patent Document 4, i.e., if it belongs to an IRAP coded basic mesh frame, Temporal_ID must be 0.

[0154] If NALType is equal to NAL_TSA_R or NAL_TSA_N, Temporal_ID must not be equal to 0.

[0155] If NALType is equal to 0 and NALType is equal to NAL_STSA_R or NAL_STSA_N, Temporal_ID must not be equal to 0.

[0156] The value of Temporal_ID must be the same for all BMCL NAL units within an access unit.

[0157] The value of Temporal_ID of a coded basic mesh frame or access unit is the value of Temporal_ID of the BMCL NAL unit of the coded basic mesh frame or access unit.

[0158] The value of Temporal_ID of a sub-layer representation is the maximum value of Temporal_ID of all BMCL NAL units within the sub-layer representation.

[0159] The value of Temporal_ID for non-BMCL NAL units is restricted as follows: - if NALType is equal to NAL_BMSPS, Temporal_ID shall be 0 and the Temporal_ID of the access unit containing the NAL unit shall be 0. - otherwise, if NALType is equal to NAL_EOS or NAL_EOB, Temporal_ID shall be 0. - otherwise, if NALType is equal to NAL_AUD or NALLFDD, Temporal_ID shall be equal to the Temporal_ID of the access unit containing the NAL unit. - otherwise, Temporal_ID shall be greater than or equal to the Temporal_ID of the access unit containing the NAL unit.

[0160] Note that if the NAL unit is not BMCL, the value of Temporal_ID is equal to the minimum value of the Temporal_ID values ​​of all access units to which the non-BMCL NAL unit applies.

[0161] When NALType is equal to NAL_BMFPS, Temporal_ID can be greater than or equal to the Temporal_ID of the included access unit, since all basic mesh frame parameter sets (BMFPS) are included at the beginning of the bitstream, where the Temporal_ID of the first coded basic mesh frame is 0.

[0162] The skip decoding unit 202F refers to the specified tIDTarget and discards, without decoding, any NAL unit whose Temporal_ID is higher than tIDTarget.

[0163] Here, tIDTarget may be specified by a predetermined value, or may be specified based on the network status or the terminal capability of the mesh decoding device 200 .

[0164] For example, a lower tIDTarget is specified in the wireless case than in the wired case. Also, a lower tIDTarget is specified when the network condition is poor. Also, a lower tIDTarget is specified when decoding is performed by a mesh decoding device 200 with low specifications.

[0165] However, the requirement for bitstream conformance is that there must be at least one NAL unit in the bitstream whose Temporal_ID is not higher than tIDTarget.

[0166] Below, a modified example will be described in which temporal scalability is achieved using the above-mentioned Temporal_ID.

[0167] The atlas, base mesh, displacement and texture bitstreams are encapsulated by Network Abstraction Layer (NAL) units, which may have a NAL header as shown in Figure 19.

[0168] The TID, defined as the last 3 bits in the NAL header, is the Temporal_ID plus 1. The range of TID is 1 to 7, with zero being prohibited.

[0169] LayerID / R6, defined as the 6 bits immediately preceding the TID in the NAL header, specifies the identifier of the layer to which the NAL unit belongs.

[0170] The value of LayerID / R6 must be in the range of 0 to 62. The value 63 may be specified by ISO / IEC in the future.

[0171] For purposes other than determining the amount of data in a decoded unit of the bitstream, the mesh decoding device 200 ignores all data following the value 63 in a NAL unit, and a mesh decoding device 200 conforming to the specified profile will ignore (i.e., remove from the bitstream and discard) all NAL units whose LayerID-R6 value is not 0.

[0172] The value 63 of LayerID / R6 can be used to indicate an enhancement layer identifier in future enhancements.

[0173] When sub-meshes of the basic mesh defined in Non-Patent Document 4 exist, all sub-meshes are assigned the same TID or the TID is applied to all sub-meshes.

[0174] The bitstreams of the atlas, base mesh, displacement, and texture may each have their own TID. For example, for the atlas, the TID is fixed to zero according to Annex A of Non-Patent Document 5. Therefore, the base mesh, displacement, and texture each have their own TID.

[0175] That is, depending on the content, at least one of the basic mesh, displacement, and texture may have Temporal_ID fixed to zero. In that case, the LD setting can also be used. Table 3 shows an example.

[0176]

[0177] Even if they are set independently in each bitstream, the displacement and texture are determined using the video coding method HEVC or VVC, and the basic mesh is determined using the above-described embodiment.

[0178] When set independently in each bitstream, the tIDTarget specified in each bitstream is referenced, and NAL units with TIDs higher than tIDTarget are discarded without being decoded.

[0179] Here, tIDTarget may be specified by a predetermined value, or may be specified based on the network status or the terminal capability of the mesh decoding device 200 .

[0180] For example, a lower tIDTarget is specified in the wireless case than in the wired case. Also, a lower tIDTarget is specified when the network condition is poor. Also, a lower tIDTarget is specified when decoding is performed by a mesh decoding device 200 with low specifications.

[0181] However, the requirement for bitstream conformance is that there must be at least one NAL unit in the bitstream whose TID is not higher than tIDTarget.

[0182] On the other hand, if each bitstream is set independently, if any of the basic mesh, displacement, and texture is discarded in a specific frame, the others will also be discarded.

[0183] Alternatively, if each bitstream is set independently, if the base mesh is discarded in a specific frame, the displacement and texture are also discarded, and the reconstruction process is not performed. However, if the displacement is discarded, all displacement values ​​are set to zero, and the reconstruction process is performed. Also, if the texture is discarded, all texture values ​​are set to zero, and the reconstruction process is performed.

[0184] The number of sub-meshes may differ for each frame (intraframe, interframe, and skip frame).

[0185] In this case, the intra decoder 202B, the inter decoder 202E, and the skip decoder 202F assign a unique submesh ID to each submesh in each frame.

[0186] Furthermore, the intra decoder 202B and the inter decoder 202E may assign different Submesh IDs to corresponding submeshes between frames.

[0187] However, the inter-decoding unit 202E can only refer to submeshes that have the same SubmeshID in the reference frame.

[0188] Alternatively, the inter-decoding unit 202E can refer to only sub-meshes that have the same number of vertices in the reference frame.

[0189] Alternatively, the intra decoder 202B and the inter decoder 202E can refer to a submesh specified in a reference frame.

[0190] In this case, if there are multiple submeshes in the reference frame, the inter-decoding unit 202E may decode a control signal specifying the Submesh ID of a referenceable submesh from the bit stream of the current submesh.

[0191] On the other hand, when there is only one submesh in the reference frame, the inter decoding unit 202E may treat this submesh as a referenceable submesh.

[0192] However, if the above-mentioned control signal does not exist, the inter decoding unit 202E or the skip decoding unit 202F sets the submesh ID of the referable submesh to the same submesh ID as the submesh in the current frame.

[0193] The inter decoding unit 202E may also decode, from the bitstream, a control signal indicating whether or not the above-mentioned control signal is present.

[0194] The inter-decoding unit 202E may decode a control signal for selecting the method for determining the above-mentioned referenceable sub-meshes.

[0195] The subdivision unit 203 and the displacement amount decoding unit 206 may be configured in accordance with Non-Patent Document 4.

[0196] According to this embodiment, unnecessary information is not coded and decoded, thereby improving coding efficiency.

[0197] Specifically, according to this embodiment, when there is only one frame in the basic mesh (when bmptc_one_mesh_frame_only_flag in non-patent document 4 is 1) or when all frames are intra-frames (when bmptc_intra_frames_only_flag in non-patent document 4 is 1), the amount of code can be reduced by not encoding and decoding unnecessary information such as inter-prediction and reference frame-related syntax.

[0198] Furthermore, according to this embodiment, it is possible to avoid a conflict between the basic mesh control signal bmptc_one_mesh_frame_only_flag and the atlas control signal ptc_one_v3c_frame_only_flag.

[0199] Specifically, according to this embodiment, a conformance requirement that harmonizes the bmptc_one_mesh_frame_only_flag and the ptc_one_v3c_frame_only_flag is provided, thereby making it possible to avoid conflicts between the two control signals.

[0200] Furthermore, according to this embodiment, by rationalizing the maximum value of the attribute count (reducing the maximum value), it is possible to manufacture products while significantly reducing product costs.

[0201] Specifically, according to this embodiment, by rationalizing the maximum value of the attribute count (that is, by reducing the maximum value), it becomes possible to manufacture a product while significantly reducing the product cost.

[0202] The mesh encoding device 100 and mesh decoding device 200 described above may be realized as a program that causes a computer to execute each function (each step).

[0203] According to this embodiment, for example, it is possible to improve the overall service quality in video communication, which makes it possible to contribute to Goal 9 of the Sustainable Development Goals (SDGs) led by the United Nations, which is to "Develop resilient infrastructure, promote sustainable industrialization and foster innovation."

[0204] 1...Mesh processing system 100...Mesh encoding device 200...Mesh decoding device 201...Demultiplexing unit 202...Basic mesh decoding unit 202A...Separation unit 202B...Intra decoding unit 202B1...Arbitrary intra decoding unit 202B2...Alignment unit 202C...Mesh buffer unit 202D...Connection information decoding unit 202E...Inter decoding unit 202E1...Motion vector residual decoding unit 202E2...Motion vector buffer unit 202E3...Motion vector prediction unit 202E4...Motion vector calculation unit 202E5...Adder 203...Subdivision unit 204...Mesh decoding unit 205...Patch integration unit 206...Displacement amount decoding unit 207...Video decoding unit 208...Atlas data decoding unit

Claims

1. A mesh decoding device comprising: an atlas data decoding unit configured to decode and output an atlas bitstream; and a base mesh decoding unit configured to decode and output a base mesh bitstream, wherein the base mesh decoding unit comprises: a control signal acquisition unit that acquires a control signal; and a judgment unit that determines whether to decode inter-prediction related information from the base mesh bitstream based on the control signal.

2. The mesh decoding device of claim 1, characterized in that the control signal acquisition unit acquires a first control signal indicating that the number of frames in the basic mesh is 1 or a second control signal indicating that the sub-mesh of the basic mesh is only intra-frames, and the judgment unit decides not to decode the inter-prediction related information if the first control signal or the second control signal is 1, and if the inter-prediction related information is not decoded, sets the value of the inter-prediction related information to a predetermined value.

3. The mesh decoding device of claim 1, characterized in that the control signal acquisition unit acquires a third control signal indicating that the inter-prediction information is to be encoded, and the judgment unit decides not to decode the inter-prediction related information if the third control signal is 0, and if the inter-prediction related information is not to be decoded, sets the value of the inter-prediction related information to a predetermined value.

4. The mesh decoding device described in claim 3, characterized in that when the control signal acquisition unit acquires the third control signal, it sets the third control signal to 0 when the first control signal indicating that the number of frames in the basic mesh is 1 is 1, and sets the third control signal to 0 when the second control signal indicating that the submesh of the basic mesh is only an intraframe is 1.

5. The mesh decoding device described in claim 3, characterized in that when the control signal acquisition unit acquires a third control signal indicating that inter-prediction information is to be coded, it includes the third control signal in the syntax of the BMSPS and decodes the third control signal from the BMSPS, and when the third control signal does not exist in the bitstream, it sets the third control signal to 1.

6. The mesh decoding device according to claim 1, wherein the value of a flag indicating that there is only one frame in the basic mesh bitstream is the same as the value of a flag indicating that there is only one frame in the atlas.

7. A mesh decoding method comprising: a step A of decoding and outputting an atlas bitstream; and a step B of decoding and outputting a base mesh bitstream, wherein step B comprises: a step of acquiring a control signal; and a step of determining, based on the control signal, whether to decode inter-prediction related information from the base mesh bitstream.

8. A program that causes a computer to function as a mesh decoding device, the mesh decoding device comprising: an atlas data decoding unit configured to decode and output an atlas bitstream; and a basic mesh decoding unit configured to decode and output a basic mesh bitstream, the basic mesh decoding unit comprising: a control signal acquisition unit that acquires a control signal; and a judgment unit that judges whether to decode inter-prediction related information from the basic mesh bitstream based on the control signal.

Images