Mesh decoding device, mesh decoding method, and program

Abstract

When using AC displacement values, maintain independence between the atlas decoding unit and the displacement value decoding unit to correctly decode the mesh. [Solution] The mesh decoding device 200 according to the present invention includes a mesh decoding unit 204 that generates and outputs a decoded mesh using control information, a subdivided mesh, a subdivided vertex normal, and a displacement amount. The mesh decoding unit 204 calculates mapping information between the displacement amount ID and the displacement amount index based on a unique control signal placed in the atlas bitstream.

Description

Technical Field

[0001] The present invention relates to a mesh decoding device, a mesh decoding method, and a program.

Background Art

[0002] Non-Patent Document 4 discloses a technique that utilizes the framework of Non-Patent Document 5 to decode a mesh by dividing it into a rough basic mesh and a detailed displacement amount. For the basic mesh, information such as vertex coordinates, connectivity, and UV coordinates (a type of attribute) is decoded by a basic mesh decoding unit, and an atlas decoded by an atlas data decoding unit is added to reconstruct the mesh from the basic mesh and the displacement amount.

[0003] In addition, for the decoding of the above-mentioned basic mesh, any one of decoding by an intra-frame (I-frame), decoding by an inter-frame (P-frame), and decoding by a skip frame is used.

[0004] In addition, there are two types of methods for decoding the above-mentioned displacement amount: a method using a video codec and a method using arithmetic decoding. In this specification, they are referred to as video displacement amount and AC displacement amount, respectively. Decoding of the displacement amount may be omitted.

[0005] Note that Non-Patent Document 4 discloses that, in addition to a mesh, video data called texture is decoded as a type of attribute in a video decoding unit.

Prior Art Documents

Non-Patent Documents

[0006]

Non-Patent Document 1

[0007] Non-patent document 4 states that when using AC displacement, the mapping information DisplIDToIndex and DisplIndexToID between the displacement ID and the displacement index are shared between the displacement decoding unit and the atlas data decoding unit, which leads to a violation of independence.

[0008] Therefore, the present invention has been made in view of the above-mentioned problems, and aims to provide a mesh decoding device, mesh decoding method, and program that can correctly decode a mesh while maintaining the independence of the atlas decoding unit and the displacement decoding unit when using AC displacement. [Means for solving the problem]

[0009] The first feature of the present invention is a mesh decoding device comprising a mesh reconstruction unit that generates and outputs a decoded mesh using control information, a subdivided mesh, a subdivided vertex normal, and a displacement amount, wherein the mesh reconstruction unit calculates mapping information between the displacement amount ID and the displacement amount index using a unique control signal input into the atlas bitstream.

[0010] A second feature of the present invention is a mesh decoding method comprising the steps of generating and outputting a decoded mesh using control information, a subdivided mesh, a subdivided vertex normal, and a displacement amount, wherein in the above step, mapping information between the displacement amount ID and the displacement amount index is calculated using a unique control signal placed in the atlas bitstream.

[0011] A third feature of the present invention is a program that causes a computer to function as a mesh decoding device, wherein the mesh decoding device includes a mesh reconstruction unit that generates and outputs a decoded mesh using control information, a subdivided mesh, a subdivided vertex normal, and a displacement amount, and the mesh reconstruction unit calculates mapping information between the displacement amount ID and the displacement amount index using a unique control signal placed in the atlas bitstream. [Effects of the Invention]

[0012] According to the present invention, when using an AC displacement amount, it is possible to provide a mesh decoding device, a mesh decoding method, and a program that can maintain the independence between an atlas decoding unit and a displacement amount decoding unit and correctly decode a mesh.

Brief Description of the Drawings

[0013] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a mesh processing system 1 according to an embodiment. [Figure 2] FIG. 2 is a diagram showing an example of the functional blocks of a mesh decoding device 200 according to an embodiment. [Figure 3] FIG. 3 is a diagram showing an example of a basic mesh and a subdivided mesh. [Figure 4] FIG. 4 is a diagram showing an example of the configuration of the syntax of an atlas bit stream. [Figure 5] FIG. 5 is a diagram showing an example of the functional blocks of a basic mesh decoding unit 202 of a mesh decoding device 200 according to an embodiment. [Figure 6] FIG. 6 is a diagram showing an example of the functional blocks of an intra decoding unit 202B of a basic mesh decoding unit 202 of a mesh decoding device 200 according to an embodiment. [Figure 7] FIG. 7 is a diagram showing an example of the correspondence between the vertices of the basic mesh of a P frame and the vertices of the basic mesh of an I frame. [Figure 8] FIG. 8 is a diagram showing an example of an ASPS. [Figure 9] FIG. 9 is a diagram showing an example of an AFPS. [Figure 10] FIG. 10 is a diagram showing an example of an ATH. [Figure 11] FIG. 11 is a diagram showing an example of the configuration of a basic mesh bit stream. [Figure 12] FIG. 12 is a diagram showing an example of a BMSPS. [Figure 13] FIG. 13 is a diagram showing an example of a BMFPS. [Figure 14]FIG. 14 is a diagram showing an example of BMSH. [Figure 15] FIG. 15 is a diagram showing an example of a functional block of a displacement decoder unit 206 of a mesh decoder device 200 according to an embodiment. [Figure 16] FIG. 16 is a diagram for explaining problems of the prior art. [Figure 17] FIG. 17 is a flowchart for explaining corresponding methods 1-1 and 1-2 by an adder 206E. [Figure 18] FIG. 18 is a flowchart for explaining corresponding methods 2-1 and 2-2 by an adder 206E. [Figure 19] FIG. 19 is a diagram showing an example of DSPS. [Figure 20] FIG. 20 is a diagram showing an example of DFPS. [Figure 21] FIG. 21 is a diagram showing an example of DH. [Figure 22] FIG. 22 is a diagram showing an example of a control signal related to Attribute of a basic mesh. [Figure 23] FIG. 23 is a diagram showing an example of a functional block of an inter-decoder unit 202E. [Figure 24] FIG. 24 is a diagram for explaining a mesh buffer unit 202C of a basic mesh decoder unit 202 of a mesh decoder device 200 according to an embodiment. [Figure 25] FIG. 25 is a flowchart showing an example of processing of the mesh buffer unit 202C. [Figure 26] FIG. 26 is a diagram showing an example of a NAL header. [Figure 27] FIG. 27 is a diagram showing an example of BMSPS.

MODE FOR CARRYING OUT THE INVENTION

[0014] Embodiments of the present invention will be described below with reference to the drawings. Note that the components in the following embodiments can be replaced with existing components as appropriate, and various variations are possible, including combinations with other existing components. Therefore, the description of the following embodiments does not limit the content of the invention as described in the claims.

[0015] <First Embodiment> The mesh processing system according to this embodiment will be described below with reference to Figures 1 to 27.

[0016] Figure 1 shows an example of the configuration of the mesh processing system 1 according to this embodiment. As shown in Figure 1, the mesh processing system 1 includes a mesh encoding device 100 and a mesh decoding device 200.

[0017] Figure 2 shows an example of the functional block of the mesh decoding device 200 according to this embodiment.

[0018] As shown in Figure 2, the mesh decoding device 200 includes a multiplexing unit 201, a basic mesh decoding unit 202, a subdivision unit 203, a mesh reconstruction unit 204, a patch integration unit 205, a displacement decoding unit 206, a video decoding unit 207, and an atlas data decoding unit 208.

[0019] Here, the basic mesh decoding unit 202, the subdivision unit 203, the mesh reconstruction unit 204, and the displacement decoding unit 206 are configured to process the mesh in patch units, and the results of these processes may then be integrated in the patch integration unit 205.

[0020] In the example shown in Figure 3, the mesh is divided into patch 1, which consists of base surfaces 1 and 2, and patch 2, which consists of base surfaces 3 and 4.

[0021] The multiplexing unit 201 is configured to separate the multiplexed bitstream into a basic mesh bitstream, a displacement bitstream, a texture bitstream, and an atlas bitstream.

[0022] The subdivision unit 203 is configured to generate and output subdivided vertices and their connection information from the basic mesh decoded by the basic mesh decoding unit 202 using a subdivision method indicated by the control information (first control information and second control information). The basic mesh consists of one or more submeshes.

[0023] Here, the base mesh, the added subdivided vertices, and their connection information are collectively referred to as the "subdivided mesh." Similarly, the submesh, the added subdivided vertices, and their connection information are collectively referred to as the "subdivided submesh."

[0024] The mesh reconstruction unit 204 is configured to generate and output a decoded mesh using control information, a subdivided mesh, subdivided vertex normals, and displacement amounts.

[0025] When using AC displacement values, the mesh reconstruction unit 204 must use Disp1IDToIndex, which is information for mapping the displacement value ID to the displacement value index.

[0026] However, Non-Patent Document 4 states that when using AC displacement amounts, the mapping information Disp1IDToIndex and Disp1IndecToID between the displacement amount ID and the displacement amount index are shared by the displacement amount decoding unit 206 and the atlas data decoding unit 208, which violates the principle of independence.

[0027] Here, Disp1IndecToID is information for mapping the displacement index to the displacement ID.

[0028] Therefore, the mesh decoding device 200 according to this embodiment incorporates its own control signal into the Atlas bitstream, and the mesh reconstruction unit 204 independently calculates mapping information Disp1IDToIndex between the displacement ID and the displacement index using this unique control signal.

[0029] Specifically, as shown in Figure 4, the Atlas bitstream may include a control signal afmi_signalled_displ_id_delta_length indicating the number of bits in afmi_displ_id[i] when using AC displacement.

[0030] Furthermore, each submesh may include a control signal afmi_displ_id[i] indicating the corresponding submesh ID.

[0031] Here, afmi_displ_id[i] and afmi_signalled_displ_id_delta_length correspond to the custom control signals mentioned above.

[0032] Specifically, in the current Atlas style, if tile ID is equal to tileID, the value obtained by adding Ceil(Log2(NumSubmeshes)) to afmi_signalled_displ_id_delta_length specifies the number of bits used to represent the syntax afmi_displ_id[i] and syntax mdu_displ_id[tileID][patchIdx] in the mesh patch data unit index patchIdx.

[0033] The value of afmi_signalled_displ_id_delta_length must be in the range of 0 to 15 (including both ends). If di_signalled_displ_id_delta_length does not exist, its value is assumed to be 0.

[0034] Additionally, afmi_displ_id[i] specifies the ID of the i-th displacement.

[0035] The length of afmi_displ_id[i] is afmi_signalled_displ_id_delta_length + Ceil(Log2(NumSubmeshes)) bits. If afmi_displ_id[i] does not exist, its value is estimated to be i for each i in the range from 0 to (NumSubmeshes-1).

[0036] As a requirement for bitstream conformance, afmi_displ_id[i] must not be equal to afmi_displ_id[j] for all i (≠j).

[0037] The variable FirstDisplID is calculated as follows:

[0038] FirstDisplID = afmi_displ_id

[0000] for ( i = 1; i < NumSubmeshes; i++ ) FirstDispID = Min(FirstDisplID, afmi_displ_id[ i ]) Then, as shown in Figure 4, the mapping relationship between displacement ID() and displacement index (DisplIDToIndex and DisplIndexToID) is calculated as follows.

[0039] for( i = 0; i < NumSubMeshes; i++ ) { DisplIDToIndex[ afmi_displ_id[ i ] ] = i DisplIndexToID[ i ] = afmi_displ_id[ i ] } According to the above configuration, when using AC displacement, it is expected that the independence of the atlas data decoding unit 208 and the displacement decoding unit 206 will be maintained, and the mesh will be correctly decoded.

[0040] The displacement amount decoding unit 206 is configured to decode the displacement amount bitstream based on control information, generate a displacement amount, and output it.

[0041] The displacement decoding unit 206 may decode and output the displacement amount from the displacement bitstream using a video codec, or it may decode and output the displacement amount from the displacement bitstream using arithmetic decoding.

[0042] The video decoding unit 207 is configured to decode and output textures using a video codec.

[0043] The Atlas data decoding unit 208 is configured to decode the Atlas bitstream and output control information. This control signal may be used as metadata by the basic mesh decoding unit 202, the subdivision unit 203, the mesh reconstruction unit 204, the displacement decoding unit 206, and the video decoding unit 207.

[0044] <Basic Mesh Decoding Unit 202> The basic mesh decoding unit 202 is configured to decode the basic mesh bitstream, generate a basic mesh, and output it.

[0045] Here, the basic mesh consists of multiple vertices in three-dimensional space and edges that connect these multiple vertices.

[0046] The basic mesh decoding unit 202 may be configured to decode the basic mesh bitstream using, for example, the Draco described in Non-Patent Document 2 or the technology described in Non-Patent Document 3.

[0047] As shown in Figure 4, the basic mesh decoding unit 202 comprises a separation unit 202A, an intra decoding unit 202B, a mesh buffer unit 202C, a connection information decoding unit 202D, and an inter-decoding unit 202E.

[0048] (Separation part 202A) The separation unit 202A is configured to classify the basic mesh bitstream into I-frame bitstreams and P-frame bitstreams, and to extract data for sub-mesh data units from each bitstream.

[0049] Specifically, data for a submesh data unit having a submesh ID "submeshID" is extracted from the basic mesh submesh data unit bmesh_submesh_unit(submeshID) which has the submesh ID "submeshID".

[0050] The submesh ID "submeshID" can be freely set in the mesh encoding device 100.

[0051] For example, in a basic mesh with two submeshes, the "submeshID" of the first submesh can be set to 10000, and the "submeshID" of the second submesh can be set to 20000. Note that "submeshID" cannot be duplicated.

[0052] In this way, being able to freely set the submesh ID "submeshID" makes parallel processing much easier.

[0053] Furthermore, the submesh ID can also be set hierarchically.

[0054] For example, if there are three human characters and five small objects such as balls, when encoding each object as a submesh, the submesh IDs for the human characters can be set to 100, 200, and 300, and the submesh IDs for the small objects such as balls can be set to 10000, 10001, 10002, 10003, and 10004.

[0055] Setting submesh IDs in this way has the advantage of making it easy to add or remove character models and small objects.

[0056] (Intra-decoding unit 202B) The intra-decoding unit 202B is configured to decode the coordinates of the I-frame vertices and connection information from the I-frame bitstream, for example, using Draco as shown in Non-Patent Document 2 or the technology described in Non-Patent Document 3.

[0057] Figure 6 shows an example of the functional block of the intra-decoding unit 202B.

[0058] As shown in Figure 6, the intra-decoding unit 202B includes an arbitrary intra-decoding unit 202B1 and an alignment unit 202B2.

[0059] The arbitrary intra decoding unit 202B1 is configured to decode the coordinates and connection information of the unordered vertices of the I-frame from the bitstream of the I-frame using any scheme including Draco as shown in Non-Patent Document 2 or the technology described in Non-Patent Document 3.

[0060] The alignment unit 202B2 is configured to output vertices by rearranging unordered vertices into a predetermined order.

[0061] The predetermined order may be, for example, Morton code order or raster scan order.

[0062] Furthermore, the alignment unit 202B2 may group duplicate vertices, which are multiple vertices with the same coordinates in the decoded basic mesh, into a single vertex, and then rearrange them in a predetermined order.

[0063] (Mesh buffer section 202C) The mesh buffer unit 202C is configured to store the coordinates and connection information of the vertices of the I-frame decoded by the intra-decoder unit 202B. A specific buffer may be provided here to store pairs of vertex indices A(k) and B(k) of duplicate vertices in a predetermined order.

[0064] (Connection information decoding unit 202D) The connection information decoding unit 202D is configured to convert the connection information of the I-frame or reference frame extracted from the mesh buffer unit 202C into the connection information of the P-frame.

[0065] (Inter-decoding unit 202E) The inter-decoding unit 202E is configured to decode the vertex coordinates of the P-frame by adding the vertex coordinates of the reference frame extracted from the mesh buffer unit 202C with the motion vector decoded from the bitstream of the P-frame.

[0066] Furthermore, the inter-decoding unit 202E can adjust the vertex indices of the P-frame using pairs of vertex indices A(k) and B(k) of vertices that exist as duplicate vertices stored in the specific buffer.

[0067] In this embodiment, as shown in Figure 7, a correspondence exists between the vertices of the basic mesh of the P frame and the vertices of the basic mesh of the reference frame (I frame or P frame). Here, the motion vector decoded by the inter-decoding unit 202E is the difference vector between the coordinates of the vertices of the basic mesh of the P frame and the coordinates of the vertices of the basic mesh of the I frame.

[0068] (Configuration of Atlas Bitstream) The Atlas bitstream may include ASPS (Atlas Sequence Parameter Set), AFPS (Atlas Frame Parameter Set), and ATS (Atlas Tile Header), which are sets of control information related to Atlas decoding.

[0069] The following describes an example of the configuration of an Atlas bitstream, referring to Figures 8 to 10.

[0070] In Figures 8 to 10, u(n) represents an n-bit code, and ue(v) represents an unsigned variable-length zero-order exponential Golomb code.

[0071] As shown in Figure 8, an ASPS may include a control signal, asps_atlas_sequence_parameter_set_id, that indicates its own ASPS ID.

[0072] The control signal asps_atlas_sequence_parameter_set_id is a type of APSP ID and is decoded with ue(v). However, the range of the control signal asps_atlas_sequence_parameter_set_id is limited to 0 to 15.

[0073] Furthermore, as shown in Figure 9, the AFPS may include a control signal afps_atlas_sequence_parameter_set_id that indicates the ASPS ID it references, and a control signal afps_atlas_frame_parameter_set_id that indicates its own AFPS ID.

[0074] Here, the control signal afps_atlas_sequence_parameter_set_id is a type of ASPS ID and is decoded with ue(v). However, the range of the control signal afps_atlas_sequence_parameter_set_id is limited to 0 to 15.

[0075] Furthermore, the control signal afps_atlas_frame_parameter_set_id is a type of AFPS ID and is decoded with ue(v). However, the range of the control signal afps_atlas_frame_parameter_set_id is limited to 0 to 63.

[0076] Furthermore, as shown in Figure 10, ATH may include a control signal ath_atlas_frame_parameter_set_id that indicates the AFPS ID it is referencing.

[0077] Here, the control signal ath_atlas_frame_parameter_set_id is a type of AFPS ID and is decoded by ue(v). However, the range of the control signal ath_atlas_frame_parameter_set_id is limited to 0 to 63.

[0078] (Basic Mesh Bitstream Configuration) The following describes an example of the configuration of a basic mesh bitstream, referring to Figures 11 to 14.

[0079] Figure 11 shows an example of the configuration of a basic mesh bitstream.

[0080] As shown in Figure 11, the base mesh bitstream may include BMSPS (Basemesh Sequence Parameter Set), BMFPS (Basemesh Frame Parameter Set), BMSH (Basemesh Submesh Header), and BMSDU (Basemesh Submesh Data Unit), which are sets of control information related to the decoding of the base mesh.

[0081] Furthermore, as shown in Figure 12, the BMSPS may include a control signal bmsps_sequence_parameter_set_id (fourth control signal) that indicates its own BMSPS ID.

[0082] Here, the control signal bmsps_sequence_parameter_set_id is a type of BMSPS ID.

[0083] As shown in Non-Patent Document 4, when the control signal bmsps_sequence_parameter_set_id is decoded by u(4), it does not match the corresponding control signal asps_atlas_sequence_parameter_set_id of Atlas.

[0084] In this embodiment, the control signal bmsps_sequence_parameter_set_id is decoded by ue(v). Furthermore, the range of the control signal bmsps_sequence_parameter_set_id is limited to 0 to 15.

[0085] Furthermore, as shown in Figure 13, the BMFPS may include a control signal bmfps_sequence_parameter_set_id that indicates the BMFPS ID it references, and a control signal bmfps_frame_parameter_set_id that indicates its own BMFPS ID.

[0086] Here, the control signal bmfps_sequence_parameter_set_id is a type of BMSPS ID.

[0087] As shown in Non-Patent Document 4, when the control signal bmfps_sequence_parameter_set_id is decoded by u(4), it does not match the corresponding control signal afps_atlas_sequence_parameter_set_id of Atlas.

[0088] In this embodiment, the control signal bmfps_sequence_parameter_set_id is decoded by ue(v). Furthermore, the range of the control signal bmfps_sequence_parameter_set_id is limited to 0 to 15.

[0089] Note that the control signal bmfps_frame_parameter_set_id is a type of BMFPS ID.

[0090] As shown in Non-Patent Document 4, when the control signal bmfps_frame_parameter_set_id is decoded by u(4) and the range limit of the control signal bmfps_frame_parameter_set_id is 0 to 15, the control signal bmfps_frame_parameter_set_id does not match the corresponding control signal afps_atlas_frame_parameter_set_id of Atlas.

[0091] In this embodiment, the control signal bmfps_frame_parameter_set_id is decoded using ue(v). Furthermore, the range of the control signal bmfps_frame_parameter_set_id is limited to 0 to 63.

[0092] However, as an example of a change, the control signal bmfps_frame_parameter_set_id may be modified to match the range limit of the atlas, and the decoding method may be changed from ue(v) to u(6).

[0093] Furthermore, as shown in Figure 14, BMSH may include a control signal bmsh_basemesh_frame_parameter_set_id that indicates the BMFPS ID it is referencing.

[0094] Here, the control signal bmsh_basemesh_frame_parameter_set_id is a type of BMFPS ID.

[0095] As shown in Non-Patent Document 4, when the control signal bmsh_basemesh_frame_parameter_set_id is decoded by u(4) and the range limit of the control signal bmsh_basemesh_frame_parameter_set_id is 0 to 15, the control signal bmsh_basemesh_frame_parameter_set_id does not match the corresponding control signal ath_atlas_frame_parameter_set_id of Atlas.

[0096] In this embodiment, the control signal bmsh_basemesh_frame_parameter_set_id is decoded by ue(v). Furthermore, the range of the control signal bmsh_basemesh_frame_parameter_set_id is limited to 0 to 63.

[0097] According to this embodiment, by matching the decoding method and range of the control signals of the basic mesh with those of the atlas, consistency with the number of FPS IDs in the atlas is maintained, and it is expected that the degree of freedom and performance will be maximized without limiting the capabilities of the atlas.

[0098] (Displacement decoding unit 206) Figure 15 shows an example of the configuration of the displacement decoding unit 206. As shown in Figure 15, the displacement decoding unit 206 comprises an arithmetic decoding unit 206A, an inverse quantization unit 206B, a frame buffer 206C, an interpretation unit 206D, and an adder 206E.

[0099] The arithmetic decoding unit 206A is configured to decode a sequence of level values ​​for each vertex from the displacement bitstream by arithmetic decoding and output it.

[0100] The inverse quantization unit 206B is configured to generate and output a coefficient sequence by inverse quantizing a sequence of level values.

[0101] The frame buffer 206C is configured to acquire and store a sequence of coefficients from the inverse quantization unit 206B or the adder 206E. The frame buffer 206C is configured to output a sequence of coefficients in the reference frame of the displacement amount according to control information (not shown).

[0102] The interpretation unit 206D is configured to read the reference frame of the displacement amount indicated by the reference list of displacement amounts from the frame buffer 206C, perform interpretation using the coefficient sequence, generate a prediction coefficient sequence, and output it.

[0103] The interpretation unit 206D may determine the prediction coefficients for each frequency in the decoding target frame by directly referencing the coefficients for the corresponding frequencies in the displacement reference frame.

[0104] Furthermore, if there are multiple sub-displacement values, the information for each sub-displacement value must be saved in the item that contains the AC displacement value reference frame list.

[0105] In this embodiment, the inter-prediction unit 206D stores information on each sub-displacement amount in an item in the AC displacement amount reference frame list in index order. The inter-prediction unit 206D then uses the index instead of the sub-displacement amount ID to retrieve reference sub-displacement amount information from the AC displacement amount reference frame list. Therefore, the inter-prediction unit 206D must calculate the index from the sub-displacement amount ID.

[0106] In such cases, the mapping relationship (DisplIDToIndex and DisplIndexToID) is calculated as follows:

[0107] for( i = 0; < NumDispls; i++ ) DisplIDToIndex[ di_displ_id[ i ] ] = i DisplIndexToID[ i ] = di_displ_id[ i ] } However, di_displ_id[i] is a control signal parsed from the AC displacement sub-bitstream and indicates the ID of the i-th sub-displacement. NumDispls indicates the number of sub-displacement values ​​in the AC displacement.

[0108] The adder 206E is configured to obtain a sequence of predicted coefficients from the interpretation unit 206D, obtain a sequence of coefficients (actually a sequence of predicted residuals) from the inverse quantization unit 206B, and add them together to generate a sequence of coefficients (displacement).

[0109] The coefficient sequence generated by adder 206E is output to frame buffer 206C. However, as shown in Figure 16, if the number of subdivisions differs between the reference frame for the AC displacement and the frame to be decoded for the AC displacement, the number of vertices will also differ.

[0110] Therefore, in this embodiment, the following countermeasures can be adopted.

[0111] <Solution 1-1> As shown in the flowchart in Figure 17, the adder 206E may perform the above-described addition process up to the number of vertices in the reference frame, refTotalVertCount.

[0112] The adder 206E may retrieve the number of vertices in the reference frame, refTotalVertCount, from the frame buffer 206C, or from the memory of a global variable.

[0113] Therefore, the adder 206E outputs the number of vertices along with the coefficient sequence of the generated decoded frame to the frame buffer 206C or to the memory of a global variable.

[0114] The adder 206E may set the displacement coefficients of vertices greater than or equal to the number of vertices in the reference frame (refTotalVertCount) to zero, or it may leave them as the prediction coefficient sequence obtained from the interpretation unit 206D.

[0115] Alternatively, adder 206E does not need to calculate displacement coefficients for vertices greater than or equal to the number of vertices in the reference frame (refTotalVertCount). In this case, adder 206E uses the values ​​from the last addition for the number of vertices and subdivisions of the frame to be decoded.

[0116] <Solution 1-2> As shown in the flowchart in Figure 17, the adder 206E may perform the above-described addition process up to the number of vertices in the reference frame, refTotalVertCount.

[0117] The adder 206E may also calculate the number of vertices in the reference frame, refTotalVertCount, from the size of the coefficient sequence of the reference frame read from the frame buffer 206C.

[0118] The adder 206E may set the displacement coefficients of vertices greater than or equal to the number of vertices in the reference frame (refTotalVertCount) to zero, or it may leave them as the prediction coefficient sequence obtained from the interpretation unit 206D.

[0119] Alternatively, adder 206E does not need to calculate displacement coefficients for vertices greater than or equal to the number of vertices in the reference frame (refTotalVertCount). In this case, adder 206E uses the values ​​from the last addition for the number of vertices and subdivisions of the frame to be decoded.

[0120] <Solution 2-1> As shown in the flowchart of Figure 18, the adder 206E may perform the above-described addition process up to the number of subdivisions of the reference frame, refSubdivisionCount.

[0121] The adder 206E may retrieve the number of subdivisions of the reference frame, refSubdivisionCount, from the frame buffer 206C, or from the memory of a global variable.

[0122] Therefore, the adder 206E outputs the number of re-partitions along with the coefficient sequence of the generated decoded frame to the frame buffer 206C or to the memory of a global variable.

[0123] The adder 206E may set the displacement coefficients of vertices that are equal to or greater than the number of subdivisions of the reference frame (refSubdivisionCount), or it may leave them as the prediction coefficient sequence obtained from the interpretation unit 206D.

[0124] Alternatively, adder 206E does not need to calculate the vertex displacement coefficient for the subdivision count refSubdivisionCount of the reference frame. In this case, adder 206E uses the values ​​from the last addition for the number of vertices and subdivision count of the frame to be decoded.

[0125] <Solution 2-2> As shown in the flowchart of Figure 18, the adder 206E may perform the above-described addition process up to the number of subdivisions of the reference frame, refSubdivisionCount.

[0126] The adder 206E may calculate the number of subdivisions of the reference frame, refSubdivisionCount, from the size of the coefficient sequence of the reference frame read from the frame buffer 206C.

[0127] Specifically, first, the adder 206E calculates the number of vertices in the reference frame, refTotalVertCount, from the size of the coefficient sequence of the reference frame read from the frame buffer 206C.

[0128] Then, the adder 206E calculates the number of subdivisions of the reference frame, refSubdivisionCount, based on the property that the number of vertices in LOD(n+1) is four times the number of vertices in LODn, or on the number of subdivisions and the number of vertices of the frame to be decoded.

[0129] For example, adder 206E calculates the number of subdivisions of a reference frame, refSubdivisionCount, as follows:

[0130] refSubdivisionCount = GetRefSubdivisionCount() { for( i = curSubdivisionCount; i < 1; i-- ) if (refTotalVertCount == levelOfDetailVertexCount[i]) { return i } return 0 } Here, curVertCount[i] is the number of vertices from LOD0 to LODi in the frame to be decoded.

[0131] The adder 206E may set the displacement coefficients of vertices that are equal to or greater than the number of subdivisions of the reference frame (refSubdivisionCount), or it may leave them as the prediction coefficient sequence obtained from the interpretation unit 206D.

[0132] Alternatively, adder 206E does not need to calculate displacement coefficients for vertices that are equal to or greater than the number of subdivisions of the reference frame (refSubdivisionCount). In this case, adder 206E uses the values ​​from the last addition for the number of vertices and subdivisions of the frame to be decoded.

[0133] According to the above configuration, when decoding the AC displacement, it is expected that the displacement coefficient can be calculated even if the number of vertices in the reference frame and the number of vertices in the frame to be decoded are different.

[0134] (Construction of the bitstream for arithmetic coding) Non-patent document 4 states that the displacement decoding unit 206 has two methods for decoding the displacement: one using video coding and the other using arithmetic coding.

[0135] The following describes an example of the configuration of the displacement bitstream for arithmetic coding, referring to Figures 19 to 21.

[0136] The arithmetic coding displacement bitstream may include a set of control information related to the decoding of the displacement, such as a DSPS (Displacement Sequence Parameter Set), a DFPS (Displacement Frame Parameter Set), or a DH (Displacement Header).

[0137] Furthermore, as shown in Figure 19, the DSPS may include a control signal dsps_sequence_parameter_set_id that indicates its own DSPS ID.

[0138] Here, the control signal dsps_sequence_parameter_set_id is a type of DSPS ID.

[0139] As shown in Non-Patent Document 4, when the control signal dsps_sequence_parameter_set_id is decoded by u(4), it does not match the corresponding control signal asps_atlas_sequence_parameter_set_id of Atlas.

[0140] In this embodiment, the control signal dsps_sequence_parameter_set_id is decoded by ue(v). Furthermore, the range of the control signal dsps_sequence_parameter_set_id is limited to 0 to 15.

[0141] Furthermore, as shown in Figure 20, the DFPS may include a control signal dfps_displ_sequence_parameter_set_id that indicates the DSPS ID it references, and a control signal dfps_displ_frame_parameter_set_id that indicates its own DFPS ID.

[0142] Here, the control signal dfps_displ_sequence_parameter_set_id is a type of DSPS ID.

[0143] As shown in Non-Patent Document 4, when the control signal dfps_displ_sequence_parameter_set_id is decoded by u(4), it does not match the corresponding control signal afps_atlas_sequence_parameter_set_id of Atlas.

[0144] In this embodiment, the control signal dfps_displ_sequence_parameter_set_id is decoded by ue(v). Furthermore, the range of the control signal dfps_displ_sequence_parameter_set_id is limited to 0 to 15.

[0145] Note that the control signal dfps_displ_frame_parameter_set_id is a type of DFPS ID.

[0146] As shown in Non-Patent Document 4, when the control signal dfps_displ_frame_parameter_set_id is decoded by u(4) and the range limit of the control signal dfps_displ_frame_parameter_set_id is 0 to 15, the control signal dfps_displ_frame_parameter_set_id does not match the corresponding control signal afps_atlas_frame_parameter_set_id of Atlas.

[0147] In this embodiment, the control signal dfps_displ_frame_parameter_set_id is decoded by ue(v). Furthermore, the range of the control signal dfps_displ_frame_parameter_set_id is limited to 0 to 63.

[0148] However, as an example of a change, the control signal dfps_displ_frame_parameter_set_id may be modified to match the range limit of the atlas, and the decoding method may be changed from ue(v) to u(6).

[0149] Furthermore, as shown in Figure 21, DH may include a control signal dh_frame_parameter_set_id that indicates the DFPS ID it is referencing.

[0150] Here, the control signal dh_frame_parameter_set_id is a type of DFPS ID.

[0151] As shown in Non-Patent Document 4, when the control signal dh_frame_parameter_set_id is decoded by u(4) and the range limit of the control signal dh_frame_parameter_set_id is 0 to 15, the control signal dh_frame_parameter_set_id does not match the corresponding control signal ath_atlas_frame_parameter_set_id of the atlas.

[0152] In this embodiment, the control signal dh_frame_parameter_set_id is decoded by ue(v). Furthermore, the range of the control signal dh_frame_parameter_set_id is limited to 0 to 63.

[0153] However, as an example of a change, the control signal dh_frame_parameter_set_id may be modified to match the range limit of the atlas, and the decoding method may be changed from ue(v) to u(6).

[0154] According to this embodiment, by matching the decoding method and range of the displacement control signal using arithmetic coding with that of Atlas, it is expected that consistency with the number of FPS IDs in Atlas will be maintained, and the degree of freedom and performance will be maximized without limiting the capabilities of Atlas.

[0155] (Attribute count / index related information) The following describes an example of attribute-related control signals for the basic mesh in the atlas bitstream and basic mesh bitstream, with reference to Figure 22.

[0156] The atlas bitstream may include a first control signal vps_ext_bmesh_data_attribute_count indicating the number of attributes of the basic mesh, as shown in Figure 22.

[0157] Furthermore, the BMSPS of the basic mesh bitstream may include a second control signal bmsps_mesh_attribute_count indicating the number of attributes of the basic mesh, as shown in Figure 22.

[0158] Furthermore, the BMSPS of the basic mesh bitstream may include bmsps_mesh_attribute_count third control signals, bmsps_mesh_attribute_index, which indicate the index of the corresponding attribute in the I-frame.

[0159] Here, the third control signal bmsps_mesh_attribute_index may indicate the index of the corresponding attribute in any submesh of any I-frame, or it may indicate the index of the corresponding attribute in the first submesh of the first I-frame. In the former case, the value of the corresponding attribute index is the same in all submeshes of all I-frames.

[0160] Additionally, each I-frame in the basic mesh bitstream may include a fourth control signal, mesh_attribute_count, which indicates the number of its own attributes.

[0161] Non-patent document 4 imposes a constraint that the value of the first control signal vps_ext_bmesh_data_attribute_count must be less than or equal to the number of attributes in the basic mesh bitstream (i.e., the value of the second control signal bmsps_mesh_attribute_count).

[0162] However, the above constraints do not apply to the value of the second control signal bmsps_mesh_attribute_count.

[0163] In this embodiment, the value of the second control signal bmsps_mesh_attribute_count is constrained to be less than or equal to the number of attributes in any submesh of any I-frame in the basic mesh bitstream (i.e., any value of mesh_attribute_count).

[0164] According to this embodiment, it is ensured that the control signal bmsps_mesh_attribute_count, which indicates the number of attributes of the basic mesh, is less than or equal to the number of attributes in any submesh of any I-frame in the basic mesh bitstream (i.e., any value of mesh_attribute_count), thereby preventing inconsistencies regarding the number of attributes of the basic mesh and enabling correct decoding of the basic mesh.

[0165] (Inter-decoding unit 202E) Figure 23 shows an example of the functional block of the inter-decoding unit 202E.

[0166] As shown in Figure 23, the inter-decoding unit 202E includes a motion vector residual decoding unit 202E1, a motion vector buffer unit 202E2, a motion vector prediction unit 202E3, a motion vector calculation unit 202E4, and an adder 202E5.

[0167] The motion vector residual decoding unit 202E1 is configured to generate an MVR (Motion Vector Residual) from the bitstream of the P frame.

[0168] Here, MVR is the motion vector residual that shows the difference between MV (Motion Vector) and MVP (Motion Vector Prediction). MV is the difference vector (motion vector) between the coordinates of the vertex in the corresponding I-frame and the vertex in the P-frame. MVP is the predicted value of the MV of the target vertex (predicted value of the motion vector) using MV.

[0169] The motion vector buffer unit 202E2 is configured to sequentially save the MV output by the motion vector calculation unit 202E4.

[0170] The motion vector prediction unit 202E3 is configured to obtain decoded MVs from the motion vector buffer unit 202E2 for vertices connected to the vertex to be decoded, and to output the MVP of the vertex to be decoded using all or part of the obtained decoded MVs, as shown in Figure 21.

[0171] The motion vector calculation unit 202E4 is configured to add the MVR generated by the motion vector residual decoding unit 202E1 and the MVP output from the motion vector prediction unit 202E3, and output the MV of the vertex to be decoded.

[0172] The adder 202E5 is configured to add the coordinates of the vertices to be decoded, obtained from the decoded base mesh of the corresponding reference frame (I-frame or P-frame), to the motion vector MV output from the motion vector calculation unit 202E3, and output the coordinates of the vertices to be decoded.

[0173] (Mesh buffer section 202C) The mesh buffer unit 202C is configured to store one or more reference decoding base meshes in a predetermined order.

[0174] Furthermore, such a basic mesh contains metadata such as frame numbers and sub-mesh numbers, as well as at least the coordinates of each vertex and the index of that vertex, and is stored in the mesh buffer unit 202C in a predetermined order determined by the reference frame list.

[0175] Here, as shown in Figure 24, the reference frame list (ref_list0) is a list of information that identifies all the reference decoded base meshes stored in the mesh buffer unit 202C.

[0176] The reference framelist may be determined by the control signals decoded from the bitstream, as shown in Figure 24, or it may be naturally calculated from the decoding order of the frames.

[0177] The control signal decoded from the bitstream may be expressed as a relative distance to the frame being decoded, or as an absolute value of the frame index.

[0178] Furthermore, control signals may be used to utilize short-term or long-term reference frames.

[0179] For example, using a short-term reference frame, the absolute value (abs_delta_mfoc_st) and its sign (sign_flag) of the difference in display order between the frame to be decoded (cur) and the reference frame (ref) can be decoded from the bitstream, and the display order of the reference frame can be specified by the following formula. If(sign_flag){ Display Order(ref)=Display Order(cur)+abs_delta_mfoc_st else{ Display Order(ref)=Display Order(cur)-abs_delta_mfoc_st } Furthermore, if a method is used that naturally calculates the order from the decoding order of the frames, for example, in the reference frame list, when no control signals are present, the frames may be arranged sequentially in fixed numbers starting from the most recently decoded frame. In other words, the reference frame list may be {0, -1, -2, ..., -(N-1)}.

[0180] Basically, the reference frame list does not change from frame to frame except in special circumstances (for example, when a re-ordering instruction is received).

[0181] Furthermore, when there are multiple submeshes, the information for each submesh must be stored in an item in the reference frame list (for example, ref_list0[0]).

[0182] Non-patent document 4 states that when saving information about each submesh to an item in the reference frame list, the information is saved in a buffer using the submesh ID.

[0183] However, because the submesh ID can be freely set, it can sometimes result in a huge waste of buffer space in the reference frame.

[0184] For example, consider a basic mesh with two submeshes, where the ID of the first submesh is 10000 and the ID of the second submesh is 20000.

[0185] If we save the submesh using the submesh ID, the buffer size becomes 20,000 times the size of the submesh.

[0186] In this embodiment, the mesh buffer unit 202C stores the information of each submesh in an item in the reference frame list in index order to avoid waste.

[0187] In other words, the mesh buffer unit 202C stores only two sub-mesh information items in the reference frame list, in the order of the first and second sub-meshes.

[0188] Then, the mesh buffer unit 202C uses an index instead of a submesh ID to retrieve reference submesh information from the reference frame list.

[0189] Therefore, the mesh buffer unit 202C must calculate an index from the submesh ID. As shown in Figure 25 and below, this mapping relationship BasemeshSubmeshIDToIndex is calculated.

[0190] for( i = 0; i < NumBmeshSubMeshes; i++ ) { BasemeshSubmeshIDToIndex[ bmsi_submesh_id[ i ] ] = i BasemeshSubmeshIndexToID[ i ] = bmsi_submesh_id[ i ] } Here, bmsi_submesh_id[i] is a control signal parsed from the sub-bitstream of the base mesh, and indicates the ID of the i-th submesh. NumbmeshSubMeshes indicates the number of submeshes.

[0191] According to this embodiment, a reference frame list is constructed using an index instead of a submesh ID, which is expected to significantly reduce memory usage.

[0192] The mesh buffer section 202C may be updated as follows.

[0193] When the basic mesh is decoded, the mesh buffer unit 202C, in the case of I-frames and P-frames, deletes one or more existing reference frames in a predetermined order determined by the reference frame list, inserts one or more basic meshes including the basic mesh of the decoded frame, or creates and inserts one basic mesh from multiple basic meshes, thereby adjusting the order of the reference frames.

[0194] Such deletion operations may be performed only when the mesh buffer unit 202C is full. The number of basic meshes that can be stored in the mesh buffer unit 202C is predetermined. In this embodiment, the mesh buffer unit 202C is defined as being full when the number of such basic meshes is reached.

[0195] In the creation process described above, the coordinates of the vertices corresponding to the decoded frame's base mesh and the existing base mesh stored in the mesh buffer section 202C may be weighted and averaged to create a single base mesh.

[0196] The weights used in such a weighted average may be predetermined, calculated using frame indices, or decoded from control signals.

[0197] Furthermore, when the mesh buffer unit 202C receives a control signal indicating a re-ordering instruction via a control signal decoded from the bitstream, it updates the reference frame list and adjusts the order of the reference frames according to a predetermined order determined by the updated reference frame list (ref_list0).

[0198] Furthermore, when a submesh exists as defined in Non-Patent Document 4 above, all submeshes will either be given the same control signal (smh_mesh_frm_order_cnt_lsb) or the control signal (smh_mesh_frm_order_cnt_lsb) will be applied to all submeshes.

[0199] The value indicated by such a control signal (smh_mesh_frm_order_cnt_lsb) may be the difference from the display order of the frames to be decoded, or it may be the order within a predetermined frame set MaxMeshFrmOrderCntLsb.

[0200] Furthermore, if the Decode Order and Display Order are different, and the decoded basic meshes are arranged in the Decode Order, the basic mesh decoding unit 202 may rearrange the decoded basic meshes to the Display Order.

[0201] To achieve temporal scalability, control signals are defined for each frame to indicate whether to decode the base mesh, displacement, and texture, and these are decoded from the bitstream accordingly.

[0202] Furthermore, the Temporal_IDs of the atlas and the base mesh may be matched within the same frame. Similarly, the Temporal_IDs of the atlas and the texture may be matched within the same frame. Finally, the Temporal_IDs of the atlas and the displacement may be matched within the same frame.

[0203] This configuration is expected to have the effect of avoiding frame decoding failures and unnecessary data.

[0204] Furthermore, it is desirable that the interval between adjacent frames with the same Temporal_ID remains constant.

[0205] Adjacent frames with the same Temporal_ID are closest to the POC.

[0206] As mentioned above, by keeping the frame interval constant, it is possible to maintain a constant frame rate when displaying the decoded frames.

[0207] Furthermore, the decoding order of atlases and base meshes with the same display order may be matched. The decoding order of atlases and displacement values ​​with the same display order may also be matched. Additionally, the decoding order of atlases and textures with the same display order may be matched.

[0208] Alternatively, the random access points of atlases and base meshes with the same display order may be matched. Furthermore, the random access points of atlases and displacement amounts with the same display order may be matched. Also, the random access points of atlases and textures with the same display order may be matched. Note that random access points are defined in Non-Patent Document 4 or Non-Patent Document 5.

[0209] With this configuration, it is expected that the mesh can be reconstructed without waiting for the decoding of the basic mesh, displacement, and texture to be completed.

[0210] Furthermore, frames with a Temporal_ID higher than the control signal Temporal_ID of the frame to be decoded will not be used as reference frames for that frame.

[0211] This is expected to have the effect of eliminating the possibility of reference frames being discarded.

[0212] The following describes an example of achieving temporal scalability using the aforementioned Temporal_ID.

[0213] The atlas, base mesh, displacement, and texture bitstreams are encapsulated by a Network Abstraction Layer (NAL) unit. The NAL unit may have a NAL header as shown in Figure 26.

[0214] The TID, defined as the last 3 bits in the NAL header, is Temporal_ID plus 1. The TID ranges from 1 to 7, and zero is prohibited.

[0215] The LayerID / R6, defined as the six bits immediately preceding the TID in the NAL header, specifies the identifier of the layer to which the NAL unit belongs.

[0216] The LayerID / R6 value must be within the range of 0 to 62. The value 63 may be specified by ISO / IEC in the future.

[0217] Aside from determining the amount of data in the bitstream's decode unit, the mesh decoder 200 ignores all data following the value 63 in the NAL unit, and a mesh decoder 200 conforming to a specified profile ignores (i.e., removes and discards) all NAL units where the LayerID-R6 value is not 0.

[0218] The LayerID / R6 value of 63 can be used in future extensions to indicate an extended layer identifier.

[0219] Furthermore, if a submesh exists as defined in Non-Patent Document 4, all submeshes will be assigned the same TID, or the TID will be applied to all submeshes.

[0220] Regarding the atlas, Non-Patent Document 5 can be used, and for displacement and texture, HEVC and VVC video encoding schemes can be used, so the basic mesh will be described below.

[0221] As shown in Figure 27, the BMSPS of the basic mesh bitstream may include a control signal bmsps_max_sub_layers_minus1 in u(3) that indicates the maximum number of temporal sublayers.

[0222] Furthermore, each Temporal sublayer may include a control signal bmsps_max_dec_mesh_frame_buffering_minus1 indicating the buffer size of the largest basic mesh, and a control signal bmsps_max_num_reorder_frames indicating the difference from the display order of the largest decoded frame.

[0223] The LayerID / R6 values ​​of all BMCL NAL units in the encoded base mesh frame must be the same. The LayerID / R6 value of the encoded base mesh frame is the LayerID / R6 value of the BMCL NAL unit in the encoded base mesh frame.

[0224] If NALType is equal to NAL_EOB, the value of LayerID / R6 must be equal to 0.

[0225] If the NALType falls within the range of NAL_BLA_W_LP to NAL_RSV_BMCL_29 as defined in Non-Patent Document 4, that is, if it belongs to an IRAP-encoded basic mesh frame, then the Temporal_ID must be 0.

[0226] If NALType is equal to NAL_TSA_R or NAL_TSA_N, Temporal_ID must not be equal to 0.

[0227] If NALType is equal to 0, and NALType is equal to NAL_STSA_R or NAL_STSA_N, then Temporal_ID must not be equal to 0.

[0228] The value of Temporal_ID must be the same for all BMCL NAL units within the access unit.

[0229] The Temporal_ID value of the coded base mesh frame or access unit is the Temporal_ID value of the BMCL NAL unit of the coded base mesh frame or access unit.

[0230] The Temporal_ID value of a sublayer representation is the maximum value of the Temporal_IDs of all BMCL NAL units within that sublayer representation.

[0231] The Temporal_ID value for non-BMCL NAL units is restricted as follows: - If NALType is equal to NAL_BMSPS, then Temporal_ID must be 0, and the Temporal_ID of the access unit containing the NAL unit must be 0. - Otherwise, if NALType is equal to NAL_EOS or NAL_EOB, Temporal_ID must be 0. - Otherwise, if NALType is equal to NAL_AUD or NAL_FD, Temporal_ID must be equal to the Temporal_ID of the access unit containing the NALL unit. - Otherwise, the Temporal_ID must be greater than or equal to the Temporal_ID of the access unit containing the NAL unit.

[0232] If the NAL unit is not a BMCL, the Temporal_ID value will be equal to the minimum Temporal_ID value of all access units to which the non-BMCL NAL unit applies.

[0233] If NALType is equal to NAL_BMFPS, then Temporal_ID can be greater than or equal to the Temporal_ID of the included access unit, since the entire set of basic mesh frame parameters (BMFPS) is included at the beginning of the bitstream where the Temporal_ID of the first encoded basic mesh frame is 0.

[0234] Furthermore, the skip decoding unit 202F will refer to the specified tIDTarget and discard any NAL units whose Temporal_ID is higher than tIDTarget without decoding them.

[0235] Here, tIDTarget may be specified by a predetermined value, or it may be specified by the network conditions or the terminal capabilities of the mesh decoding device 200.

[0236] For example, a lower tIDTarget is specified for wireless connections than for wired connections. Also, a lower tIDTarget is specified when the network conditions are poor. Furthermore, a lower tIDTarget is specified when decoding is performed by a low-spec mesh decoder 200.

[0237] However, a requirement for bitstream conformance is that the bitstream must contain at least one NAL unit whose Temporal_ID is not higher than tIDTarget.

[0238] The following describes an example of a change that achieves temporal scalability using the aforementioned Temporal_ID.

[0239] The atlas, base mesh, displacement, and texture bitstreams are encapsulated by a Network Abstraction Layer (NAL) unit. The NAL unit may have a NAL header as shown in Figure 26.

[0240] The TID, defined as the last 3 bits in the NAL header, is Temporal_ID plus 1. The TID ranges from 1 to 7, and zero is prohibited.

[0241] The LayerID / R6, defined as the six bits immediately preceding the TID in the NAL header, specifies the identifier of the layer to which the NAL unit belongs.

[0242] The LayerID / R6 value must be within the range of 0 to 62. The value 63 may be specified by ISO / IEC in the future.

[0243] Aside from determining the amount of data in the bitstream's decode unit, the mesh decoder 200 ignores all data following the value 63 in the NAL unit, and a mesh decoder 200 conforming to a specified profile ignores (i.e., removes and discards) all NAL units where the LayerID-R6 value is not 0.

[0244] The LayerID / R6 value of 63 can be used in future extensions to indicate an extended layer identifier.

[0245] Furthermore, if there are submeshes of the basic mesh as defined in Non-Patent Document 4, all submeshes shall be given the same TID, or the TID shall be applied to all submeshes.

[0246] The bitstreams for the atlas, base mesh, displacement, and texture may each have their own independently set TID. For example, the TID for the atlas is fixed to zero according to Annex A of Non-Patent Document 5. Thus, the base mesh, displacement, and texture each have their own independently set TID.

[0247] That is, depending on the content, at least one of the basic mesh, displacement amount, and texture may have its Temporal_ID fixed to zero. In that case, the LD setting can also be used. Table 1 shows an example.

[0248]

Table 1

[0249] Even if set independently for each bitstream, for the displacement amount and texture, HEVC or VVC of the video encoding method is used, and for the basic mesh, the above-described embodiment is used.

[0250] ... When set independently for each bitstream, referring to the tIDTarget specified in each bitstream, NAL units with a TID higher than tIDTarget are discarded without being decoded. ...

[0251] ... ... Here, tIDTarget may be specified by a pre-determined value, or may be specified according to the network situation or the terminal capabilities of the mesh decoder 200. ...

[0252] ... ... For example, a lower tIDTarget is specified for a wireless case than for a wired case. Also, a lower tIDTarget is specified when the network situation is poor. Also, a lower tIDTarget is specified when decoding is performed by a low-spec mesh decoder 200. ...

[0253] ... <0OO0896> ... However, as a requirement for the compatibility of the bitstream, it is stipulated that there must be at least one NAL unit in the bitstream whose TID is not higher than tIDTarget. ...

[0254] ... ... On the other hand, when each bitstream is set independently, if one of the basic mesh, displacement amount, and texture is discarded in a specific frame, the others are also discarded.

[0255] Alternatively, if each bitstream is set independently, when a basic mesh is discarded in a specific frame, the displacement amount and texture will also be discarded, and the Reconstruction process will not be performed. However, if the displacement amount is discarded, all the values of the displacement amount will be set to zero, and the Reconstruction process will be performed. Also, if the texture is discarded, all the values of the texture will be set to zero, and the Reconstruction process will be performed.

[0256] Note that the number of sub-meshes may be different for each frame (intra-frame, inter-frame, and skip frame).

[0257] In such a case, the intra decoder 202B, the inter decoder 202E, and the skip decoder 202F assign non-overlapping sub-mesh IDs to each of the sub-meshes in each frame.

[0258] Also, the intra decoder 202B and the inter decoder 202E may assign different SubmeshIDs (sub-mesh IDs) to corresponding sub-meshes between frames.

[0259] However, it is assumed that the inter decoder 202E can only refer to sub-meshes having the same SubmeshID in the reference frame.

[0260] Alternatively, it is assumed that the inter decoder 202E can only refer to sub-meshes having the same number of vertices in the reference frame.

[0261] Alternatively, it is assumed that the intra decoder 202B and the inter decoder 202E can refer to the specified sub-meshes in the reference frame.

[0262] In such cases, if there are multiple submeshes in the reference frame, the inter-decoding unit 202E may decode a control signal from the bitstream of the current submesh that specifies the SubmeshID of the referenceable submesh.

[0263] On the other hand, if there is only one submesh in the reference frame, the inter-decoding unit 202E may treat that submesh as a referenceable submesh.

[0264] However, if the above-mentioned control signals are not present, the inter-decoding unit 202E or the skip decoding unit 202F will set the SubmeshID of the accessible submesh to the same SubmeshID as the submesh in the current frame.

[0265] Furthermore, the inter-decoding unit 202E may decode a control signal from the bitstream indicating whether the above-mentioned control signal exists.

[0266] The inter-decoding unit 202E may also decode a control signal that selects the method for determining the above-mentioned referable submesh.

[0267] The subdivision section 203 and the displacement decoding section 206 may conform to Non-Patent Document 4.

[0268] Non-patent document 4 defines various profiles and levels. For example, as shown in Table 2, non-patent document 4 defines the levels of atlases.

[0269] [Table 2]

[0270] Furthermore, Non-Patent Document 4 defines the basic mesh levels, as shown in Table 3.

[0271] [Table 3]

[0272] However, in Non-Patent Document 4, the levels shown in Table 2 or Table 3 cannot be achieved due to the syntax of the atlas and the basic mesh.

[0273] This embodiment modifies the related syntax of the atlas and the basic mesh, and minimizes the memory so as to achieve such levels.

[0274] Specifically, in this embodiment, a control signal afmi_num_submeshes_minus2 indicating (the number of sub-meshes - 2) in the atlas is set to U(14) instead of u(8), and its range is from 0 to 16,383. [[ID=ll]]

[0275] In addition, in this embodiment, the range of a control signal bmsi_num_submeshes_minus2 indicating (the number of sub-meshes - 2) in the basic mesh is set to from 0 to 511.

[0276] Note that in this embodiment, the range of a control signal di_num_displs_minus2 indicating (the number of sub-displacements - 2) in the AC displacement amount is set to from 0 to 511.

[0277] The above-mentioned mesh encoding device 100 and mesh decoding device 200 may be realized by a program that causes a computer to execute each function (each process).

Industrial Applicability

Explanation of Signs

[0279] 1... Mesh processing system 100... Mesh encoding device 200... Mesh Decoder 201…Demultiplexer 202...Basic Mesh Decoding Unit 202A…Separation part 202B...Intra Decoding Unit 202B1... Optional intra-decoding unit 202B2…Alignment section 202C...Mesh buffer section 202D...Connection Information Decoding Unit 202E...Inter-decoding section 202E1... Motion vector residue decoding unit 202E2... Motion vector buffer section 202E3... Motion vector prediction unit 202E4... Motion vector calculation unit 202E5…Adder 202F...Control information acquisition unit 202G...judgment section 203…Subdivision part 204...Mesh decoding section 205... Patch Integration Department 206...Displacement Decoding Unit 206A... Arithmetic Decoding Unit 206B...Inverse quantization section 206C... Frame buffer 206D...Interface prediction unit 206E... Adder 207...Video Decoding Unit 208... Atlas Data Decoding Unit

Claims

1. A mesh decoding device, It includes a mesh reconstruction unit that generates and outputs a decoded mesh using control information, a subdivided mesh, subdivided vertex normals, and displacement amounts. The mesh decoding device is characterized in that the mesh reconstruction unit calculates mapping information between displacement IDs and displacement indexes using a proprietary control signal fed into the Atlas bitstream.

2. The aforementioned mapping information is DisplIDToIndex, which shows information for mapping the ID of the displacement amount to the displacement amount index, Includes DisplIndexToID, which indicates information for mapping the displacement index to the displacement ID, The aforementioned DispliIDToIndex and DispliIndexToID are, for( i = 0; i < NumSubMeshes; i++ ) { DisplIDToIndex[ afmi_displ_id[ i ] ] = i DisplIndexToID[ i ] = afmi_displ_id[ i ] } Calculated by, The afmi_displ_id[i] is the unique control signal, and indicates the ID of the i-th displacement amount. The mesh decoding device according to claim 1, wherein NumSubMeshes indicates the number of submeshes.

3. A mesh decoding method, The process includes generating and outputting a decoded mesh using control information, a subdivided mesh, subdivided vertex normals, and displacement amounts. A mesh decoding method characterized in that, in the aforementioned step, mapping information between the displacement ID and the displacement index is calculated using a proprietary control signal input into the Atlas bitstream.

4. A program that makes a computer function as a mesh decoding device, The mesh decoding device is It includes a mesh reconstruction unit that generates and outputs a decoded mesh using control information, a subdivided mesh, subdivided vertex normals, and displacement amounts. The mesh reconstruction unit is a program characterized by calculating mapping information between displacement IDs and displacement indexes using a proprietary control signal fed into the Atlas bitstream.

Images