Mesh decoding device, mesh decoding method, and program

Abstract

To provide a mesh decoding device, a mesh decoding method, and a program which accurately decode a mesh even when the number of submeshes is different between a decoded atlas and a basic mesh.SOLUTION: In a mesh decoding device 200, a variable indicating the number of submeshes in a decoding object frame is calculated from a control signal defined by an atlas sub-bit stream, and a basic mesh is decoded at a basic mesh decoding section 202, and a displacement and a texture are added to submeshes of a basic mesh so that a number of submeshes equal to the calculated submesh number are reproduced.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a mesh decoding device, a mesh decoding method, and a program. [Background technology]

[0002] Non-Patent Document 1 or Non-Patent Document 4 discloses a technique for encoding a mesh using Non-Patent Document 2 or 3 in accordance with the framework of Non-Patent Document 5. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Khaled Mammou, Jungsun Kim, Alexis M Tourapis, Dimitri Podborski, and Krasimir Kolarov, “[V-CG] Apple's Dynamic Mesh Coding CfP Response,” April 2022, ISO / IEC JTC 1 / SC 29 / WG 7 m59281. [Non-patent document 2] Google Draco, accessed May 26, 2022 [Online], https: / / google.github.io / draco [Non-patent document 3] Jean-Eudes Marvie, Olivier Mocquard, “[V-DMC][EE4.4-related] An efficient EdgeBreaker implementation,” April 2023, ISO / IEC JTC 1 / SC 29 / WG 7 m63344. [Non-patent document 4] “WD 6.0 of V-DMC,” Jan. 2024, ISO / IEC JTC 1 / SC 29 / WG 7 N00822. [Non-Patent Document 5] “Information technology - Coded Representation of Immersive Media - Part 5: Visual Volumetric Video-based Coding (V3C) and Video-based Point Cloud Compression (V-PCC),” ISO / IEC JTC 1 / SC 29 / WG 7, ISO / IEC 23090-5:2021(2E). Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the conventional technology, if the number of sub-meshes differs between the decoded atlas and the base mesh, the mesh cannot be decoded correctly. Therefore, the present invention has been made in consideration of the above-mentioned problem, and aims to provide a mesh decoding device, a mesh decoding method, and a program that can correctly decode a mesh even if the number of sub-meshes differs between the decoded atlas and the base mesh. [Means for solving the problem]

[0005] A first feature of the present invention is a mesh decoding device comprising an atlas data decoding unit that decodes an atlas bitstream and outputs control information, and a basic mesh decoding unit that decodes a basic mesh bitstream and generates and outputs a basic mesh, wherein the basic mesh decoding unit calculates and outputs the number of sub-meshes of the basic mesh of the frame to be decoded from a control signal defined in the basic mesh sub-bitstream, a variable indicating the number of sub-meshes in the frame to be decoded is calculated from the control signal defined in the atlas sub-bitstream, and the sub-meshes of the basic mesh decoded by the basic mesh decoding unit are reproduced by adding a displacement amount and a texture to the sub-meshes of the basic mesh.

[0006] A second feature of the present invention is a mesh decoding method comprising step A of decoding an atlas bitstream and outputting control information, and step B of decoding a basic mesh bitstream and generating and outputting a basic mesh, wherein in step B, the number of sub-meshes of the basic mesh of the frame to be decoded is calculated and output from a control signal defined in the basic mesh sub-bitstream, a variable indicating the number of sub-meshes in the frame to be decoded is calculated from a control signal defined in the atlas sub-bitstream, and a displacement amount and texture are added to the sub-meshes of the basic mesh decoded in step B to reproduce the sub-meshes of the number of sub-meshes.

[0007] A third feature of the present invention is a program that causes a computer to function as a mesh decoding device, the mesh decoding device comprising an atlas data decoding unit that decodes an atlas bitstream and outputs control information, and a basic mesh decoding unit that decodes a basic mesh bitstream and generates and outputs a basic mesh, wherein the basic mesh decoding unit calculates and outputs the number of sub-meshes of the basic mesh of the frame to be decoded from a control signal defined in the basic mesh sub-bitstream, a variable indicating the number of sub-meshes in the frame to be decoded is calculated from the control signal defined in the atlas sub-bitstream, and the sub-meshes of the basic mesh decoded by the basic mesh decoding unit are reproduced by adding a displacement amount and texture to the sub-meshes of the basic mesh. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide a mesh decoding device, a mesh decoding method, and a program that can correctly decode a mesh even if the number of sub-meshes differs between the decoded atlas and the base mesh. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a mesh processing system 1 according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of functional blocks of a mesh decoding device 200 according to an embodiment. [Figure 3A] FIG. 3A is a diagram showing an example of a basic mesh and a subdivision mesh. [Figure 3B] FIG. 3B is a diagram showing an example of a basic mesh and a subdivision mesh. [Figure 4] FIG. 4 is a diagram showing an example of functional blocks of the basic mesh decoding unit 202 of the mesh decoding device 200 according to an embodiment. [Figure 5] FIG. 5 is a diagram showing an example of functional blocks of the intra-decoding unit 202B of the basic mesh decoding unit 202 of the mesh decoding device 200 according to an embodiment. [Figure 6] FIG. 6 is a diagram showing an example of the correspondence between the vertices of the basic mesh of a P frame and the vertices of the basic mesh of an I frame. [Figure 7] FIG. 7 is a diagram for explaining Modification Example 1 and Modification Example 2-1. [Figure 8] FIG. 8 is a diagram for explaining modified example 2-2. [Figure 9] FIG. 9 is a diagram showing an example of measurement results of coding efficiency of a control signal. [Figure 10] FIG. 10 is a diagram showing an example of functional blocks of the inter decoding unit 202E of the basic mesh decoding unit 202 of the mesh decoding device 200 according to an embodiment. [Figure 11] FIG. 11 is a diagram illustrating an example of a method for calculating the MVP of a vertex to be decoded by the motion vector prediction unit 202E3 of the inter decoding unit 202E of the basic mesh decoding unit 202 of the mesh decoding device 200 according to one embodiment. [Figure 12] FIG. 12 shows a flowchart illustrating an example of the operation of the motion vector prediction unit 202E3 of the inter decoding unit 202E of the basic mesh decoding unit 202 of the mesh decoding device 200 according to an embodiment. [Figure 13]FIG. 13 is a diagram illustrating the mesh buffer unit 202C of the basic mesh decoding unit 202 of the mesh decoding device 200 according to an embodiment. [Figure 14] FIG. 14 is a diagram illustrating the mesh buffer unit 202C of the basic mesh decoding unit 202 of the mesh decoding device 200 according to an embodiment. [Figure 15] FIG. 15 is a diagram illustrating a modification of the basic mesh decoding unit 202 of the mesh decoding device 200 according to the second modification. [Figure 16] FIG. 21 is a diagram illustrating a modification of the basic mesh decoding unit 202 of the mesh decoding device 200 according to the second modification. [Figure 17] FIG. 17 is a diagram illustrating an example of an NAL header. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the components in the following embodiments can be appropriately replaced with existing components, etc., and various variations, including combinations with other existing components, are possible. Therefore, the description of the following embodiments does not limit the content of the invention described in the claims.

[0011] First Embodiment The mesh processing system according to this embodiment will be described below with reference to FIGS.

[0012] 1 is a diagram showing an example of the configuration of a mesh processing system 1 according to this embodiment. As shown in FIG. 1, the mesh processing system 1 includes a mesh encoding device 100 and a mesh decoding device 200.

[0013] FIG. 2 is a diagram showing an example of functional blocks of the mesh decoding device 200 according to this embodiment.

[0014] As shown in Figure 2, the mesh decoding device 200 includes a demultiplexing unit 201, a basic mesh decoding unit 202, a subdivision unit 203, a mesh decoding unit 204, a patch integration unit 205, a displacement amount decoding unit 206, a video decoding unit 207, and an atlas data decoding unit 208.

[0015] Here, the basic mesh decoding unit 202, the subdivision unit 203, the mesh decoding unit 204, and the displacement amount decoding unit 206 are configured to perform processing in units of patches into which the mesh is divided, and the results of these processes may then be integrated by the patch integration unit 205.

[0016] In the example of FIG. 3A, the mesh is divided into patch 1 consisting of basic faces 1 and 2, and patch 2 consisting of basic faces 3 and 4.

[0017] The demultiplexing unit 201 is configured to separate the multiplexed bit stream into a basic mesh bit stream, a displacement bit stream, a texture bit stream, and an atlas bit stream.

[0018] The subdivision unit 203 is configured to generate and output subdivision vertices and their connection information from the basic mesh decoded by the basic mesh decoding unit 202, using a subdivision method indicated by the control information (first control information and second control information). The basic mesh is composed of one submesh or multiple submeshes.

[0019] Here, the basic mesh, the added subdivision vertices, and the information on their connections are collectively referred to as a "subdivision mesh." Also, the submesh, the added subdivision vertices, and the information on their connections are collectively referred to as a "subdivision submesh."

[0020] The mesh decoding unit 204 is configured to generate and output a decoded mesh using the control information, the subdivision mesh, the subdivision vertex normals, and the displacement amounts.

[0021] The displacement decoding unit 206 is configured to decode the displacement bitstream based on the control information to generate and output the displacement.

[0022] The video decoding unit 207 is configured to decode the texture using a video codec and output the decoded texture.

[0023] The atlas data decoder 208 is configured to decode the atlas bitstream and output control information, which may be used as metadata by the basic mesh decoder 202, the subdivision unit 203, the mesh decoder 204, the displacement decoder 206, and the video decoder 207.

[0024] <Basic mesh decoding unit 202> The base mesh decoding unit 202 is configured to decode the base mesh bitstream, generate and output a base mesh.

[0025] Here, the basic mesh is made up of a plurality of vertices in a three-dimensional space and edges connecting these vertices.

[0026] As shown in Figure 3A, a basic mesh or a decoded mesh is composed of a combination of basic faces or faces represented by three vertices. Therefore, the requirement for bitstream compatibility is that the decoded mesh has three or more vertices, or the control signal indicating the number of vertices is three or more.

[0027] Note that the basic mesh or the decoded mesh may be constructed by combining lines represented by two vertices, and the requirement for bitstream conformance is that the decoded mesh has two or more vertices, or the control signal indicating the number of vertices is two or more.

[0028] The base mesh decoder 202 may be configured to decode the base mesh bitstream using, for example, the techniques described in Draco in Non-Patent Document 2 or in Non-Patent Document 3.

[0029] As shown in FIG. 4, the basic mesh decoding unit 202 includes a separating unit 202A, an intra decoding unit 202B, a mesh buffer unit 202C, a connection information decoding unit 202D, and an inter decoding unit 202E.

[0030] The separating unit 202A is configured to classify the basic mesh bitstream into a bitstream of I frames and a bitstream of P frames.

[0031] (Intra decoding unit 202B) The intra decoder 202B is configured to decode the coordinates and connection information of the vertices of the I-frame from the bitstream of the I-frame, for example, using the technique Draco shown in Non-Patent Document 2 or the technique described in Non-Patent Document 3.

[0032] FIG. 5 is a diagram showing an example of functional blocks of the intra decoding unit 202B.

[0033] As shown in FIG. 5, the intra decoding unit 202B includes an arbitrary intra decoding unit 202B1 and an alignment unit 202B2.

[0034] The optional intra decoder 202B1 is configured to decode the coordinates and connectivity information of the unordered vertices of the I-frame from the bitstream of the I-frame using any method including Draco described in Non-Patent Document 2 or the technique described in Non-Patent Document 3.

[0035] The sorting unit 202B2 is configured to output the vertices by sorting the unordered vertices into a predetermined order.

[0036] The predetermined order may be, for example, a Morton code order or a raster scan order.

[0037] Furthermore, the alignment unit 202B2 may combine overlapping vertices, which are multiple vertices with the same coordinates in the decoded basic mesh, into a single vertex, and then rearrange them in a predetermined order.

[0038] The mesh buffer unit 202C is configured to store the coordinates and connection information of the vertices of the I frame decoded by the intra decoding unit 202B. Here, a specific buffer may be provided to store pairs of vertex indices A(k) and B(k) of overlapping vertices in a predetermined order.

[0039] The connection information decoding unit 202D is configured to convert the connection information of the I frame or the reference frame extracted from the mesh buffer unit 202C into connection information of the P frame.

[0040] The inter-decoding unit 202E is configured to decode the coordinates of the vertices of the P frame by adding the coordinates of the vertices of the reference frame retrieved from the mesh buffer unit 202C to the motion vectors decoded from the bitstream of the P frame.

[0041] Furthermore, the inter decoding unit 202E can adjust the index of the vertex of the P frame using the pair of vertex indices A(k) and B(k) that exist as overlapping vertices stored in the specific buffer.

[0042] Here, all or part of the indexes are decoded from the bitstream. The decoding method may be arithmetic coding. As a result, it is expected that there is no limit to the maximum value of the index to be decoded using arithmetic coding.

[0043] For example, we can use arithmetic coding ue(v), which denotes leftmost bit-first exponential-Golomb coding of unsigned integers of order 0 (Exp-Golomb).

[0044] Specifically, the parsing process for a syntax element of ue(v) begins at the current position in the bitstream, reading the bits containing the first non-zero bit, and counting the number of leading bits equal to 0. This process is specified as follows: leadingZeroBits=-1 for(b=0;!b;leadingZeroBits++ b = read_bits(l) Next, the variable codeNum is assigned as follows:

[0045] codeNum=2 leadingZeroBits -1+read_bits(leadingZeroBits) However, the value returned by read_bits(leadingZeroBits) is interpreted as a binary representation of an unsigned integer with the most significant bit written first, and the value of ue(v) is equal to the value of codeNum.

[0046] Table 1 shows the structure of an Exp-Golomb code, separating the bit string into "prefix" and "suffix" bits.

[0047] [Table 1] Here, the "prefix" bits are the bits that are parsed as specified in the leadingZeroBits calculation and appear as 0 or 1 in the Bit String column of Table 1.

[0048] The "suffix" bits are the bits parsed in the codeNum calculation, and are denoted by x in Table 1. i where i ranges from 0 to leadingZeroBits-1. Each x i is equal to either 0 or 1.

[0049] Table 2 shows how to explicitly assign bit strings to values ​​of codeNum, where the value of ue(v) is equal to the value of codeNum.

[0050] [Table 2] In this embodiment, there is a correspondence between the vertices of the base mesh of the P frame and the vertices of the base mesh of the reference frame (I frame or P frame), as shown in Fig. 6. Here, the motion vector decoded by the inter decoding unit 202E is a difference vector between the coordinates of the vertices of the base mesh of the P frame and the coordinates of the vertices of the base mesh of the I frame.

[0051] The inter decoding unit 202E may decode, from the bitstream, a control signal indicating the number of vertices or motion vectors of the current frame or current submesh.

[0052] <Modification 1: When the control signal indicates the number of vertices in the current frame or current submesh> Here, when the inter-decoding unit 202E decodes the number of vertices of the current frame or current submesh from the above-mentioned bitstream, the compatibility requirement of such a bitstream is that the number of vertices of the decoded current frame or current submesh must be equal to the number of vertices of the reference frame or reference submesh.

[0053] In addition, when decoding the number of vertices of the current frame or current submesh from the above-mentioned bitstream, if the number of vertices of the decoded current frame or current submesh differs from the number of vertices of the reference frame or reference submesh, the inter-decoding unit 202E is configured to preferentially use the number of vertices of the current frame or current submesh decoded from the bitstream.

[0054] Furthermore, the inter decoding unit 202E may use the number of vertices of the reference frame or reference submesh as the number of vertices of the current frame or current submesh.

[0055] [Table 3] In such a case, the inter-decoding unit 202E may add a dummy vertex to the reference frame or reference submesh if the number of vertices in the current frame or current submesh is greater than the number of vertices in the reference frame or reference submesh (CurrSubmesh case 2 in Table 3).

[0056] Here, the inter decoding unit 202E may set the coordinates of such dummy vertices to fixed values ​​(for example, (0,0,0)), or may copy them from a predetermined vertex (for example, the last vertex of the reference frame).

[0057] However, since such dummy vertices do not have connection information, the inter-decoding unit 202E can discard such dummy vertices during processing or as post-processing, and when discarding such dummy vertices, it changes the number of vertices of the current sub-mesh accordingly.

[0058] In addition, when the number of vertices in the current frame or current submesh is decoded from the bitstream and the number of vertices in the current frame or current submesh is less than the number of vertices in the reference frame or reference submesh (CurrSubmesh case 2 in Table 3), the inter-decoding unit 202E calculates the vertex coordinates up to the number of vertices in the reference frame or reference submesh, and sets the number of vertices in the current frame or current submesh to the number of vertices in the reference frame or reference submesh.

[0059] Specifically, first, the inter decoding unit 202E calculates the vertex coordinates up to the number of vertices in the current frame or current submesh. However, there are vertices whose vertex coordinates have not been calculated, such as v3, v4, and v5 in Figure 7(c).

[0060] Here, since the connectivity information is copied from the reference submesh, there are faces that have connectivity information but no vertex coordinates, such as faces (v0, v2, v3), (v2, v4, v3), (v2, v5, v4), and (v1, v5, v2) shown by dotted lines in Figure 7(c).

[0061] Hereinafter, a vertex for which vertex coordinates have not been calculated will be referred to as an abnormal vertex, and a face for which there is connectivity information but no vertex coordinates will be referred to as an abnormal face.

[0062] When an abnormal surface is subdivisioned, there is a problem that the coordinates of the midpoints cannot be calculated as one of the outputs of the subdivision.

[0063] To address this issue, the following solutions 1 / 2 are available.

[0064] Solution 1: When the number of vertices in the current frame or current submesh is less than that of the reference submesh, the inter-decoding unit 202E may set the coordinates of the abnormal vertex to a fixed value (e.g., (0,0,0)) or copy the coordinates from a predetermined vertex (e.g., select one from the non-abnormal vertices). Also, the inter-decoding unit 202E changes the number of vertices in the current frame or current submesh to the number of vertices in the reference submesh.

[0065] Solution 2: When the number of vertices in the current frame or current submesh is less than that of the reference submesh, the inter-decoding unit 202E may copy the coordinates of the abnormal vertices from the vertices of the reference submesh with the same vertex index. Also, the inter-decoding unit 202E changes the number of vertices in the current frame or current submesh to the number of vertices in the reference submesh.

[0066] According to this configuration, it is expected that the decoding operation of the current frame or the current submesh can be guaranteed.

[0067] <Modification 2: When the control signal indicates the number of motion vectors of the current frame or current submesh> In this modified example 2, a case where the current submesh is used will be described. Note that this modified example 2 can also be applied to a case where the current frame is used.

[0068] (Change example 2-1) Here, when the inter-decoding unit 202E decodes the number of motion vectors of the current submesh from the above-mentioned bitstream, if the decoded number of motion vectors of the current submesh differs from the number of vertices of the reference submesh, it adjusts the number of vertices of the current submesh and the coordinates of some of the vertices.

[0069] First, as an initialization operation, the inter-decoding unit 202E copies the submesh ID, number of faces, vertex coordinates and connection information of the reference submesh to the current submesh, and calculates the number of vertices of the current submesh using equation (A).

[0070] DecSummesh.verCoordCount=max(refSubmesh.verCoordCount,bm_vertex_count[submeshID]) … Formula (A) where DecSummesh.verCoordCount is a control signal indicating the number of vertices of the current submesh, refSubmesh.verCoordCount is a control signal indicating the number of vertices of the reference submesh, and bm_vertex_count[submeshID] is a control signal indicating the number of motion vectors of the current submesh.

[0071] That is, when the number of motion vectors of the current submesh is greater than the number of vertices of the reference submesh, the inter decoding unit 202E sets the number of vertices of the current submesh to the number of motion vectors of the current submesh.

[0072] As a result of the above operations, the number of vertices of the current submesh may become the same as the number of vertices of the reference submesh, or the number of vertices of the current submesh may become greater than the number of vertices of the reference submesh.

[0073] However, at this point, the number of vertices included in the vertex coordinates of the current submesh is the same as that of the reference submesh.

[0074] Furthermore, as shown in Table 4 and Figure 7, there are cases where the number of motion vectors in the current submesh is the same as the number of vertices in the reference submesh (currSubmesh SW), where the number of motion vectors in the current submesh is less than the number of vertices in the reference submesh (currSubmesh case 1), or where the number of motion vectors in the current submesh is more than the number of vertices in the reference submesh (currSubmesh case 2).

[0075] [Table 4] In such a case, when the number of vertices of the current submesh is greater than the number of vertices of the reference submesh (currSubmesh case 2 in Table 4), and the vertex index of the current submesh is greater than or equal to the number of vertices of the reference submesh (there is no corresponding vertex of the reference submesh), the inter-decoding unit 202E calculates the vertex coordinates using the following formula (B).

[0076] - Otherwise(v is greater than refSubmesh.verCoordCount-1),DecSummesh.verCoords[v][k]=currentSubmeshMotionVectors[v][k],where k=0..2 … Formula (B) where refSubmesh.verCoordCount is the number of vertices in the reference submesh, currentSubmeshMotionVectors[v][k] is the k-th dimension motion vector of the v-th vertex of the current submesh, and DecSummesh.verCoords[v][k] is the calculated k-th dimension vertex coordinate of the v-th vertex of the current submesh.

[0077] Furthermore, since the above-mentioned dummy vertices do not have connection information, the inter-decoding unit 202E can discard such dummy vertices during processing or as post-processing, and if such dummy vertices are discarded, the number of vertices of the current sub-mesh is changed accordingly.

[0078] By the above-mentioned operation, when the number of motion vectors of the current submesh is greater than the number of vertices of the reference submesh, the inter-decoding unit 202E calculates vertex coordinates up to the number of vertices of the reference submesh using motion vectors, and leaves the other vertex coordinates as motion vectors.

[0079] In such a case, when the number of motion vectors of the current submesh is less than or equal to the number of vertices of the reference submesh, the inter-decoding unit 202E calculates vertex coordinates using motion vectors up to the number of motion vectors of the current submesh, and leaves the other vertex coordinates as they are in the reference submesh.

[0080] According to this configuration, it is expected that the decoding operation of the current frame or the current submesh can be guaranteed.

[0081] (Change example 2-2) Here, when the inter-decoding unit 202E decodes the number of motion vectors of the current submesh from the above-mentioned bitstream, if the decoded number of motion vectors of the current submesh differs from the number of vertices of the reference submesh, it adjusts the number of vertices of the current submesh and some of the vertex coordinates.

[0082] First, as an initialization operation, the inter-decoding unit 202E copies the submesh ID, the number of vertices, the number of faces, the vertex coordinates, and the connection information of the reference submesh to the current submesh.

[0083] The above operation ensures that the number of vertices in the current submesh is the same as the number of vertices in the reference submesh.

[0084] However, as shown in Table 5 and Figure 8, the number of motion vectors may be the same as the number of vertices of the reference submesh (currSubmesh SW), may be less than the number of vertices of the reference submesh (currSubmesh case 1), or may be more than the number of vertices of the reference submesh (currSubmesh case 2).

[0085] [Table 5] In such a case, if the number of motion vectors of the current submesh is less than or equal to the number of vertices of the reference submesh, the inter-decoding unit 202E calculates vertex coordinates up to the number of motion vectors of the current submesh using the decoded motion vectors, and leaves the other vertex coordinates as they are in the reference submesh.

[0086] In such a case, if the number of motion vectors of the current submesh is greater than the number of vertices of the reference submesh, the inter-decoding unit 202E sets the number of vertices of the current submesh to the number of vertices of the reference frame, and calculates vertex coordinates up to the number of vertices of the reference submesh using the decoded motion vectors.

[0087] To summarize the above operations, the inter decoding unit 202E calculates the vertex coordinates when the vertex index of the current submesh is between 0 and min(refSubmesh.vertex_count,bm_vertex_count[submeshID])-1.

[0088] where refSubmesh.vertex_count is the number of vertices in the reference submesh, and bm_vertex_count[submeshID] is the number of motion vectors in the current submesh.

[0089] In other words, when the number of motion vectors of the current submesh is greater than the number of vertices of the reference submesh, the inter-decoding unit 202E sets the number of vertices of the current frame or current submesh to the number of vertices of the reference frame, and calculates vertex coordinates up to the number of vertices of the reference submesh using the decoded motion vectors.

[0090] According to this configuration, it is expected that the decoding operation of the current frame or the current submesh can be guaranteed.

[0091] The inter-decoding unit 202E may use the bit analysis method shown in Table 6 to decode the control signal indicating the number of vertices of the basic mesh or after subdivision for each frame or each submesh.

[0092] [Table 6] For example, such control signals include mesh_vertex_count, pdu_vertex_count_minus_1[titleID][patchIdx], sismu_inter_vertex_count[subMeshID], and the like.

[0093] Furthermore, the inter decoding unit 202E may decode the above control signal using u(v), ue(v), ae(v), vu(v), or the like.

[0094] For example, the inter-decoding unit 202E may decode the above-mentioned control signal using arithmetic coding called ue(v), where ue(v) represents leftmost bit-first unsigned integer zeroth-order exponential-Golomb coding (Exp-Golomb).

[0095] Furthermore, the inter decoding unit 202E may decode the above control signal using arithmetic coding called ae(v) using CABAC.

[0096] As a result, it is expected that there will be no limit to the maximum number of vertices to be decoded using arithmetic coding.

[0097] However, since the value of the control signal indicating the number of vertices is large, there is a problem in that the efficiency of ue(v) and ae(v) is low.

[0098] Therefore, when there is a maximum number of vertices maxVC for all frames or submeshes, the inter decoding unit 202E may decode the above control signal using u(v) instead of ue(v).

[0099] In this case, the inter decoding unit 202E calculates v from the maximum number of vertices maxVC of all frames or submeshes. Specifically, the inter decoding unit 202E calculates v using the following formula (C).

[0100]

number

[0101] Furthermore, using vu(v) instead of u(v) has the effect of improving coding efficiency compared to using ue(v), as shown in Figure 9, and vu(v) can be analyzed using the following procedure without calculating v.

[0102] That is, the first bit of each 8-bit string is a flag indicating whether the next 8 bits should be parsed, and the other 7 bits indicate the value.

[0103] vu(v) { value = 0 do { continue = read_bits( 1 ) partial_value = read_bits( 7 ) value = ( value << 7 ) | partial_value } while( continue ) return value In addition, for the control signal indicating the number of vertices of the basic mesh of the current frame or current submesh, using the same bit analysis method as other syntaxes indicating the number of vertices has the effect of simplifying the hardware design.

[0104] Furthermore, in the bit analysis method of syntax indicating the number of faces such as face_count, vu(v) may be changed to ue(v).

[0105] The syntax indicating the number of vertices and faces mentioned above is a large value, so unifying the syntax bit analysis method has the effect of reducing the complexity of the decoder and the effect of reducing the complexity of the hardware design and enabling chip sharing.

[0106] In addition, a requirement for bitstream compatibility may be that the above-mentioned pdu_vertex_count_minus_1[titleID][patchIdx] and sismu_inter_vertex_count[subMeshID] do not contradict each other, or that pdu_vertex_count_minus_1[titleID][patchIdx] and mesh_vertex_count do not contradict each other.

[0107] In addition, if there is a conflict between pdu_vertex_count_minus_1[titleID][patchIdx] and sismu_inter_vertex_count[subMeshID], or if there is a conflict between pdu_vertex_count_minus_1[titleID][patchIdx] and mesh_vertex_count, priority may be given to sismu_inter_vertex_count[subMeshID] or mesh_vertex_count.

[0108] The atlas data decoding unit 208 calculates a variable called AtlasTotalNumMeshpatches from a control signal called afmi_num_submeshes_minus2 defined in the atlas sub-bitstream.

[0109] Here, AtlasTotalNumMeshpatches indicates the displacement amount and the number of texture sub-meshes in the corresponding frame.

[0110] The basic mesh decoding unit 202 calculates the number of sub-meshes of the basic mesh in the frame to be decoded using bmsi_num_submeshes_minus2 defined in the basic mesh sub-bitstream.

[0111] Because afmi_num_submeshes_minus2 and bmsi_num_submeshes_minus2 are independent, there is a problem in that the displacement amount indicated by the calculated AtlasTotalNumMeshpatches or the number of texture submeshes may differ from the number of submeshes of the base mesh.

[0112] In the above-mentioned non-patent document 4, displacement and texture are added to the sub-meshes of the basic mesh, and the number of sub-meshes indicated by AtlasTotalNumMeshpatches calculated using the control signal output from the atlas data decoding unit 208 is reproduced as the final sub-meshes.

[0113] Therefore, when the number of sub-meshes of the base mesh is less than the number of sub-meshes indicated by AtlasTotalNumMeshpatches, the base mesh does not have enough sub-meshes to reproduce the missing final sub-meshes.

[0114] Conversely, when the number of submeshes of a base mesh is greater than the number indicated by AtlasTotalNumMeshpatches, there will be excess submeshes of the base mesh, and the displacement and texture of the excess final submeshes may be treated as non-existent.

[0115] Furthermore, the remaining final sub-meshes may be reconstructed using the existing displacements and textures of the remaining basic meshes. This reconstruction is performed in the reconstruction process (see FIG. 2). This reconstruction process is performed by the subdivision unit 203, mesh decoding unit 204, and patch integration unit 205.

[0116] In this embodiment, the requirement for bitstream compatibility may be that AtlasTotalNumMeshpatches is less than the number of submeshes of the base mesh, or that AtlasTotalNumMeshpatches is the same as the number of submeshes of the base mesh.

[0117] Also, when AtlasTotalNumMeshpatches is greater than the number of submeshes of the base mesh, AtlasTotalNumMeshpatches may be changed to the number of submeshes of the base mesh.

[0118] Alternatively, when AtlasTotalNumMeshpatches is greater than the number of submeshes of the base mesh, the submeshes of the base mesh are insufficient, so the missing parts may be made into submeshes of a specific base mesh.

[0119] For example, the first submesh among the submeshes with the same number of vertices (or the last submesh of the basic mesh) may be selected as a submesh of the specific basic mesh.

[0120] This configuration is expected to solve the problem that the atlas data decoding unit 208 cannot reproduce some sub-meshes.

[0121] (Inter-decoding unit 202E) FIG. 10 is a diagram illustrating an example of functional blocks of the inter decoding unit 202E.

[0122] As shown in FIG. 10, the inter decoding unit 202E includes a motion vector residual decoding unit 202E1, a motion vector buffer unit 202E2, a motion vector prediction unit 202E3, a motion vector calculation unit 202E4, and an adder 202E5.

[0123] The motion vector residual decoding unit 202E1 is configured to generate a motion vector residual (MVR) from a P frame bitstream.

[0124] Here, MVR is a motion vector residual indicating the difference between MV (Motion Vector) and MVP (Motion Vector Prediction). MV is a difference vector (motion vector) between the coordinates of the corresponding vertex in the I frame and the coordinates of the vertex in the P frame. MVP is a predicted value (predicted value of the motion vector) of the MV of the target vertex using MV.

[0125] The motion vector buffer unit 202E2 is configured to sequentially store the MVs output by the motion vector calculation unit 202E4.

[0126] The motion vector prediction unit 202E3 is configured to obtain the decoded MV for the vertex connected to the vertex to be decoded from the motion vector buffer unit 202E2, and output the MVP of the vertex to be decoded using all or part of the obtained decoded MV, as shown in Figure 11.

[0127] The motion vector calculation unit 202E4 is configured to add the MVR generated by the motion vector residual decoding unit 202E1 and the MVP output from the motion vector prediction unit 202E3, and output the MV of the vertex to be decoded.

[0128] The adder 202E5 is configured to add the coordinates of the vertex corresponding to the vertex to be decoded, obtained from the decoded basic mesh of the corresponding reference frame (I frame or P frame), to the motion vector MV output from the motion vector calculation unit 202E3, and output the coordinates of the vertex to be decoded.

[0129] The individual components of inter decoding unit 202E will be described in detail below.

[0130] 12 is a flowchart showing an example of the operation of the motion vector predictor 202E3. Hereinafter, the operation of the motion vector predictor 202E3 will be referred to as the "average prediction method."

[0131] As shown in FIG. 12, in step S1001, the motion vector prediction unit 202E3 sets MVP and N to 0.

[0132] In step S1002, the motion vector prediction unit 202E3 obtains a set of MVs of vertices around the vertex to be decoded from the motion vector buffer unit 202E2, identifies vertices for which subsequent processing has not been completed, and transitions to No. If subsequent processing has been completed for all vertices, it transitions to Yes.

[0133] In step S1003, the motion vector prediction unit 202E3 transitions to No if the MV of the vertex to be processed has not been decoded, and transitions to Yes if the MV of the vertex to be processed has been decoded.

[0134] In step S1004, the motion vector prediction unit 202E3 adds MV to MVP and adds 1 to N.

[0135] In step S1005, if N is greater than 0, the motion vector prediction unit 202E3 outputs the result of dividing MVP by N, and if N is 0, outputs 0, and ends the process.

[0136] That is, the motion vector prediction unit 202E3 is configured to output the MVP to be decoded by averaging the decoded motion vectors of the vertices around the vertex to be decoded.

[0137] The motion vector prediction unit 202E3 may be configured to set MVP to 0 when the set of decoded motion vectors is an empty set.

[0138] The motion vector calculation unit 202E4 may be configured to calculate the MV of the vertex to be decoded from the MVP output by the motion vector prediction unit 202E3 and the MVR generated by the motion vector residual decoding unit 202E1 using equation (1).

[0139] MV(k)=MVP(k)+MVR(k) … (1) where k is the index of the vertex. MV, MVR and MVP are vectors with x, y and z components.

[0140] According to this configuration, since only the MVR is coded instead of the MV using the MVP, it is expected that the coding efficiency will be improved.

[0141] The adder 202E5 is configured to calculate the coordinates of a vertex by adding the MV of the vertex calculated by the motion vector calculation unit 202E4 to the coordinates of the vertex in the reference frame corresponding to the vertex, and to leave the connectivity information (Connectivity) as it is in the reference frame.

[0142] Specifically, the adder 202E5 calculates the coordinate v' of the kth vertex using equation (2). i (k) may be calculated.

[0143] v' i (k)=v' j (k)+MV(k) … (2) where v' i (k) is the coordinate of the kth vertex to be decoded in the frame to be decoded, and v' j (k) is the coordinate of the decoded k-th vertex of the reference frame, and MV(k) is the k-th MV of the frame to be decoded, where k=1, 2, . . . , K.

[0144] Furthermore, the connection information of the frame to be decoded is made the same as the connection information of the reference frame.

[0145] Note that the motion vector prediction unit 202E3 calculates the MVP using the decoded MV, and therefore the order of decoding affects the MVP.

[0146] The decoding order is the order in which the vertices of the base mesh in the reference frame are decoded. Generally, if a decoding method is used that uses a fixed repetition pattern to increase the number of base faces one by one from the starting edge, the order of the vertices of the decoded base mesh is determined during the decoding process.

[0147] For example, the motion vector prediction unit 202E3 may use an Edgebreaker to determine the order of decoding vertices in the base mesh of the reference frame.

[0148] According to this configuration, since MVs from a reference frame are coded instead of vertex coordinates, it is expected that coding efficiency will be improved.

[0149] (Mesh buffer section 202C) The mesh buffer unit 202C is configured to store one or more decoded reference basic meshes in a predetermined order.

[0150] The basic mesh has metadata such as a frame number and a submesh number, and at least the coordinates of each vertex and the index of that vertex, and is stored in the mesh buffer unit 202C in a predetermined order determined by the reference frame list.

[0151] As shown in FIG. 13, the reference frame list (ref_list0) is a list of information that identifies all reference decoding basic meshes stored in the mesh buffer unit 202C.

[0152] The reference frame list may be determined by control signals decoded from the bitstream, as shown in FIG. 13, or may be calculated naturally from the decoding order of the frames.

[0153] The control signal decoded from the bitstream may be represented by a relative distance from the frame to be decoded, or may be represented by a frame index that is an absolute value.

[0154] Additionally, the control signal may use a short-term or long-term frame of reference.

[0155] For example, when a short-term reference frame is used, the absolute value (abs_delta_mfoc_st) of the difference in display order between the current frame (cur) and the reference frame (ref) and its sign (sign_flag) can be decoded from the bitstream, and the display order of the reference frame can be specified using the following formula: If(sign_flag){ Display Order(ref)=Display Order(cur)+abs_delta_mfoc_st }else{ Display Order(ref)=Display Order(cur)-abs_delta_mfoc_st } In addition, when a method of calculating the frame decoding order is used, for example, when there is no control signal in the reference frame list, the frames may be arranged in order of a certain number of frames from the most recently decoded frame. That is, the reference frame list may be {0, -1, -2, ..., -(N-1)}.

[0156] Basically, the reference frame list does not change for each frame except in special circumstances (for example, when a re-ordering instruction is received).

[0157] The mesh buffer unit 202C may be updated as follows.

[0158] When a basic mesh is decoded, in the case of an I frame or a P frame, the mesh buffer unit 202C deletes one or more existing reference frames in a predetermined order determined by the reference frame list, and inserts one or more basic meshes including the basic mesh of the decoded frame, or creates and inserts one basic mesh from multiple basic meshes, thereby adjusting the order of the reference frames.

[0159] This deletion operation may be performed only when the mesh buffer unit 202C is full. The number of basic meshes that can be stored in the mesh buffer unit 202C is determined in advance. In this embodiment, the mesh buffer unit 202C is defined as being full when the number of basic meshes is reached. In the above-mentioned creation work, a single basic mesh may be created by taking a weighted average of the coordinates of the vertices corresponding to the basic meshes of the decoded frame and the existing basic meshes stored in the mesh buffer unit 202C.

[0160] The weights used in such a weighted average may be predetermined, calculated using the frame index, or decoded from the control signal.

[0161] In addition, when the mesh buffer unit 202C receives a control signal indicating a re-ordering instruction from a control signal decoded from the bitstream, it updates the reference frame list as shown in Figure 14 and adjusts the order of the reference frames according to the specified order determined by the updated reference frame list (ref_list0).

[0162] (Inter-decoding unit 202E) The inter-decoding unit 202E is configured to decode the coordinates of the vertices of the P frame by adding the coordinates of the vertices of the reference frame retrieved from the mesh buffer unit 202C to the motion vectors decoded from the bitstream of the P frame.

[0163] Furthermore, the inter-decoding unit 202E can adjust the indexes of the vertices of the P frame using the pair of indexes A(k) and B(k) of the vertices existing as overlapping vertices stored in the specific buffer. All or part of these indexes are decoded from the bitstream. This decoding method may be arithmetic coding. This configuration is expected to have the effect of eliminating the limit on the maximum value of the index to be decoded using arithmetic coding. For example, ue(v) may be used as the arithmetic coding.

[0164] Furthermore, the above-mentioned method of signaling and parsing information about overlapping vertices may be as follows.

[0165] [Table 6A] Here, bmidu_derived_mv_present_flag[submeshID] is a flag that indicates whether bmidu_mv_signalled_flag is present in the bitstream.

[0166] bmidu_mv_signalled_flag_count[submeshID] is the number of bmidu_mv_signalled_flag.

[0167] bmidu_mv_signalled_flag is a flag that indicates whether motion vectors for overlapping vertices are present in the bitstream.

[0168] In parsing, the number of motion vectors that are not signaled in the bitstream is counted, and the index of the corresponding motion vector is not specified in the submesh.

[0169] In addition, during the decoding process, a conversion process is performed to identify the index of the corresponding motion vector within the sub-mesh.

[0170] According to this configuration, it is expected that parsing can be performed only up to the end of the stream without relying on complex processing such as decoding processing.

[0171] (Modification 2 of the basic mesh decoding unit 202) Hereinafter, a second modification of the basic mesh decoding unit 202 will be described with reference to FIG.

[0172] As shown in FIG. 15, in the inter decoding unit 202E, the decoding order and the display order are different in order to allow reference to subsequent frames.

[0173] Here, the display order is the same as the input order when encoding, and is the same as the output order when decoding.

[0174] On the other hand, the decoding order is the same as the output order when encoding, and is the same as the input order when decoding.

[0175] The reference frame may be calculated by taking a weighted average of the subsequent frame and one or more other frames.

[0176] However, when referring to multiple frames including subsequent frames, MR_SUBMESH (MR frame or B frame) is defined as a new frame type (smh_type), and MR_SUBMESH is decoded from the bitstream.

[0177] Furthermore, the other frame may be a decoded frame immediately before the target frame, as shown in FIG.

[0178] Such weights may be calculated using the frame interval between the target frame and the subsequent frame and the frame interval between the target frame and another frame, or may be determined in advance.

[0179] The basic mesh decoding unit 202 decodes the control signal (smh_mesh_frm_order_cnt_lsb) from the bitstream and decodes the order of the outputs.

[0180] When sub-meshes as defined in the above-mentioned non-patent document 4 exist, all sub-meshes are set to the same control signal (smh_mesh_frm_order_cnt_lsb) or the control signal (smh_mesh_frm_order_cnt_lsb) is applied to all sub-meshes.

[0181] The value indicated by this control signal (smh_mesh_frm_order_cnt_lsb) may be a difference from the display order of the frame to be decoded, or may be an order within a predetermined frame group MaxMeshFrmOrderCntLsb.

[0182] In addition, when the decoding order and the display order are different, if the decoded basic meshes are arranged in the decoding order, the basic mesh decoding unit 202 may rearrange the decoded basic meshes in the display order.

[0183] To achieve temporal scalability, control signals indicating whether to decode the base mesh, displacement, and texture are defined for each frame, and each is decoded from the bitstream.

[0184] Also, the Temporal_IDs of the atlas and the base mesh may be matched within the same frame. Also, the Temporal_IDs of the atlas and the texture may be matched within the same frame. Also, the Temporal_IDs of the atlas and the displacement may be matched within the same frame.

[0185] According to this configuration, it is expected that it will be possible to avoid frames that cannot be decoded and unnecessary data.

[0186] It is desirable that the interval between adjacent frames having the same Temporal_ID is constant.

[0187] Adjacent frames with the same Temporal_ID have the closest POC.

[0188] By keeping the frame intervals constant as described above, it is expected that a constant frame rate can be maintained when displaying decoded frames.

[0189] Furthermore, the decoding orders of atlases and base meshes having the same display order may be matched, the decoding orders of atlases and displacements having the same display order may be matched, and the decoding orders of atlases and textures having the same display order may be matched.

[0190] Alternatively, the random access points of an atlas and a base mesh that have the same display order may be matched. Alternatively, the random access points of an atlas and a displacement that have the same display order may be matched. Alternatively, the random access points of an atlas and a texture that have the same display order may be matched. Note that random access points are defined in Non-Patent Document 4 or Non-Patent Document 5.

[0191] According to this configuration, when decoding the basic mesh, the displacement amount, and the texture, it is possible to expect the effect that the mesh can be reproduced without waiting for each other to be decoded.

[0192] Furthermore, a frame having a Temporal_ID higher than the control signal Temporal_ID of the frame to be decoded is not used as a reference frame for the frame to be decoded.

[0193] This is expected to have the effect of eliminating the possibility of reference frames being discarded.

[0194] An example of realizing temporal scalability using the above-mentioned Temporal_ID will be described below.

[0195] The atlas, base mesh, displacement, and texture bitstreams are encapsulated by Network Abstraction Layer (NAL) units, which may have a NAL header as shown in Figure 22.

[0196] The TID, defined as the last 3 bits in the NAL header, is the Temporal_ID plus 1. The range of TID is 1 to 7, with zero being prohibited.

[0197] LayerID / R6, defined as the 6 bits immediately preceding the TID in the NAL header, specifies the identifier of the layer to which the NAL unit belongs.

[0198] The value of LayerID / R6 must be in the range of 0 to 62. The value 63 may be specified by ISO / IEC in the future.

[0199] For purposes other than determining the amount of data in a decoded unit of the bitstream, the mesh decoding device 200 ignores all data following the value 63 in an NAL unit, and a mesh decoding device 200 conforming to the specified profile will ignore (i.e., remove from the bitstream and discard) all NAL units with a LayerID-R6 value other than 0.

[0200] The value 63 of LayerID / R6 can be used to indicate an extension layer identifier in future extensions.

[0201] When sub-meshes defined in Non-Patent Document 4 exist, all sub-meshes are assigned the same TID or the TID is applied to all sub-meshes.

[0202] Non-Patent Document 5 can be used for the atlas, and HEVC or VVC video coding methods can be used for the displacement and texture, so the following will explain the basic mesh.

[0203] The LayerID / R6 values ​​of all BMCL NAL units of a coded basic mesh frame MUST be the same. The LayerID / R6 value of a coded basic mesh frame is the LayerID / R6 value of the BMCL NAL units of the coded basic mesh frame.

[0204] If NALType is equal to NAL_EOB, the value of LayerID / R6 must be equal to 0.

[0205] If the NALType is in the range from NAL_BLA_W_LP to NAL_RSV_BMCL_29 defined in Non-Patent Document 4, i.e., belongs to an IRAP coded basic mesh frame, Temporal_ID must be 0.

[0206] If NALType is equal to NAL_TSA_R or NAL_TSA_N, Temporal_ID must not be equal to 0.

[0207] If NALType is equal to 0 and NALType is equal to NAL_STSA_R or NAL_STSA_N, Temporal_ID must not be equal to 0.

[0208] The value of Temporal_ID must be the same for all BMCL NAL units within an access unit.

[0209] The value of Temporal_ID of a coded basic mesh frame or access unit is the value of Temporal_ID of the BMCL NAL unit of the coded basic mesh frame or access unit.

[0210] The value of Temporal_ID of a sub-layer representation is the maximum value of the Temporal_ID of all BMCL NAL units within the sub-layer representation.

[0211] The values ​​of Temporal_ID for non-BMCL NAL units are restricted as follows: - If NALType is equal to NAL_BMSPS, Temporal_ID MUST be 0 and the Temporal_ID of the access unit containing the NAL unit MUST be 0. Otherwise, if NALType is equal to NAL_EOS or NAL_EOB, Temporal_ID must be 0. Otherwise, if NALType is equal to NAL_AUD or NALLFDD, Temporal_ID shall be equal to the Temporal_ID of the access unit that contains the NALL unit. - Otherwise, the Temporal_ID MUST be greater than or equal to the Temporal_ID of the access unit that contains the NAL unit.

[0212] If the NAL unit is not BMCL, the value of Temporal_ID is equal to the minimum value of the Temporal_ID values ​​of all access units to which the non-BMCL NAL unit is applied.

[0213] If NALType is equal to NAL_BMFPS, Temporal_ID can be greater than or equal to the Temporal_ID of the included access unit, since all basic mesh frame parameter sets (BMFPS) are included at the beginning of the bitstream, where the Temporal_ID of the first coded basic mesh frame is 0.

[0214] The skip decoding unit 202F refers to the specified tIDTarget and discards NAL units whose Temporal_ID is higher than tIDTarget without decoding them.

[0215] Here, tIDTarget may be specified by a predetermined value, or may be specified based on the network status or the terminal capability of the mesh decoding device 200.

[0216] For example, a lower tIDTarget is specified in the wireless case than in the wired case. Also, a lower tIDTarget is specified when the network condition is poor. Also, a lower tIDTarget is specified when decoding is performed by a mesh decoding device 200 with low specifications.

[0217] However, the requirement for bitstream conformance is that there must be at least one NAL unit in the bitstream whose Temporal_ID is not higher than tIDTarget.

[0218] Below, we will describe a modification example that uses the above-mentioned Temporal_ID to achieve temporal scalability.

[0219] The atlas, base mesh, displacement, and texture bitstreams are encapsulated by Network Abstraction Layer (NAL) units, which may have a NAL header as shown in Figure 17.

[0220] The TID, defined as the last 3 bits in the NAL header, is the Temporal_ID plus 1. The range of TID is 1 to 7, with zero being prohibited.

[0221] LayerID / R6, defined as the 6 bits immediately preceding the TID in the NAL header, specifies the identifier of the layer to which the NAL unit belongs.

[0222] The value of LayerID / R6 must be in the range of 0 to 62. The value 63 may be specified by ISO / IEC in the future.

[0223] For purposes other than determining the amount of data in a decoded unit of the bitstream, the mesh decoding device 200 ignores all data following the value 63 in an NAL unit, and a mesh decoding device 200 conforming to the specified profile will ignore (i.e., remove from the bitstream and discard) all NAL units with a LayerID-R6 value other than 0.

[0224] The value 63 of LayerID / R6 can be used to indicate an extension layer identifier in future extensions.

[0225] When sub-meshes of the basic mesh defined in Non-Patent Document 4 exist, all sub-meshes are assigned the same TID or the TID is applied to all sub-meshes.

[0226] The bitstreams of the atlas, base mesh, displacement, and texture may each have their own TID. For example, the TID of the atlas is fixed to zero according to Annex A of Non-Patent Document 5. Therefore, the base mesh, displacement, and texture each have their own TID.

[0227] That is, depending on the content, at least one of the basic mesh, displacement, and texture may have Temporal_ID fixed to zero. In that case, the LD setting can also be used. Table 7 shows an example.

[0228] [Table 7] Even if they are set independently in each bitstream, the displacement and texture are determined using the video coding method HEVC or VVC, and the basic mesh is determined using the above-described embodiment.

[0229] When set independently for each bitstream, the tIDTarget specified in each bitstream is referenced, and NAL units with TIDs higher than the tIDTarget are discarded without being decoded.

[0230] Here, tIDTarget may be specified by a predetermined value, or may be specified based on the network status or the terminal capability of the mesh decoding device 200.

[0231] For example, a lower tIDTarget is specified in the wireless case than in the wired case. Also, a lower tIDTarget is specified when the network condition is poor. Also, a lower tIDTarget is specified when decoding is performed by a mesh decoding device 200 with low specifications.

[0232] However, the requirement for bitstream conformance is that there must be at least one NAL unit in the bitstream whose TID is not higher than tIDTarget.

[0233] On the other hand, if each bitstream is set independently, if any of the basic mesh, displacement, and texture is discarded in a specific frame, the others will also be discarded.

[0234] Alternatively, if each bitstream is set independently, if the base mesh is discarded in a specific frame, the displacement and texture are also discarded and the reconstruction process is not performed. However, if the displacement is discarded, all displacement values ​​are set to zero and the reconstruction process is performed. Also, if the texture is discarded, all texture values ​​are set to zero and the reconstruction process is performed.

[0235] The number of sub-meshes may differ for each frame (intraframe, interframe, and skip frame).

[0236] In this case, the intra decoder 202B, the inter decoder 202E, and the skip decoder 202F assign a unique submesh ID to each submesh in each frame.

[0237] Furthermore, the intra decoder 202B and the inter decoder 202E may assign different Submesh IDs to corresponding submeshes between frames.

[0238] However, the inter-decoding unit 202E can only refer to submeshes that have the same SubmeshID in the reference frame.

[0239] Alternatively, the inter-decoding unit 202E can refer to only sub-meshes that have the same number of vertices in the reference frame.

[0240] Alternatively, the intra decoder 202B and the inter decoder 202E can refer to a submesh specified in a reference frame.

[0241] In this case, if there are multiple submeshes in the reference frame, the inter decoding unit 202E may decode a control signal specifying the SubmeshID of a referenceable submesh from the bit stream of the current submesh.

[0242] On the other hand, when there is only one submesh in the reference frame, the inter decoding unit 202E may treat this submesh as a referenceable submesh.

[0243] However, if the above-mentioned control signal does not exist, the inter decoding unit 202E or the skip decoding unit 202F sets the Submesh ID of the referable submesh to the same Submesh ID as the submesh in the current frame.

[0244] Furthermore, the inter decoding unit 202E may decode, from the bitstream, a control signal indicating whether or not the above-mentioned control signal is present.

[0245] The inter-decoding unit 202E may decode a control signal for selecting the method for determining the above-mentioned referenceable sub-meshes.

[0246] The subdivision unit 203 and the displacement decoding unit 206 may comply with Non-Patent Document 4.

[0247] According to the present invention, the amount of calculation can be reduced by reusing the reference frame itself without searching for adjacent vertices that have already been decoded.

[0248] Furthermore, according to this embodiment, in inter-prediction coding, the base mesh of the current frame can be decoded even if the number of vertices of the base mesh of the current frame differs from the number of vertices of the reference frame or reference sub-mesh.

[0249] Furthermore, according to this embodiment, in inter-prediction coding, it is possible to avoid a situation in which the number of vertices of the base mesh of the current frame differs from the number of vertices of the reference frame or reference sub-mesh.

[0250] Furthermore, according to this embodiment, in inter-prediction coding, it is possible to identify which sub-mesh to refer to by introducing a control signal into the reference frame of the current frame indicating which sub-mesh to refer to.

[0251] Furthermore, according to this embodiment, in inter-prediction coding, even if there is no information in the reference frame of the current frame as to which sub-mesh to refer to, it is possible to specify which sub-mesh to refer to.

[0252] Furthermore, according to this embodiment, it is possible to ensure that the basic mesh has at least one face.

[0253] Furthermore, according to this embodiment, the Temporal_scalability function can be realized.

[0254] Furthermore, according to this embodiment, the encoding efficiency of the mesh can be improved.

[0255] The mesh encoding device 100 and the mesh decoding device 200 described above may be realized as a program that causes a computer to execute each function (each step). [Industrial Applicability]

[0256] According to this embodiment, for example, it is possible to improve the overall service quality in video communication, which makes it possible to contribute to Goal 9 of the Sustainable Development Goals (SDGs) led by the United Nations, which is to "Develop resilient infrastructure, promote sustainable industrialization and foster innovation." [Explanation of symbols]

[0257] 1...Mesh processing system 100...Mesh coding device 200...Mesh decoding device 201…Demultiplexer 202...Basic mesh decoding unit 202A…Separation part 202B... Intra decoding unit 202B1...Optional intra-decoding unit 202B2...Alignment section 202C...Mesh buffer section 202D...Connection information decoding unit 202E...Inter-decoding unit 202E1...Motion vector residual decoding unit 202E2...Motion vector buffer unit 202E3...Motion vector prediction unit 202E4...Motion vector calculation unit 202E5...adder 203…Subdivision part 204...Mesh decoding unit 205…Patch Integration Department 206...Displacement amount decoding unit 207...Video decoding unit 208...Atlas data decoding unit

Claims

1. A mesh decoding device, comprising: an atlas data decoder that decodes the atlas bitstream and outputs control information; a basic mesh decoding unit that decodes the basic mesh bitstream, generates a basic mesh, and outputs the generated basic mesh; The basic mesh decoding unit calculates and outputs the number of sub-meshes of the basic mesh of the frame to be decoded from the control signal defined in the basic mesh sub-bitstream; A mesh decoding device characterized in that a variable indicating the number of submeshes in the frame to be decoded is calculated from a control signal defined in an atlas sub-bitstream, and the submeshes of the basic mesh are decoded in the basic mesh decoding unit, and displacement and texture are added to reproduce the submeshes of the number of submeshes.

2. The mesh decoding device described in claim 1, characterized in that the requirement for bitstream compatibility is that the number of submeshes indicated by the variable is less than the number of submeshes of the basic mesh output by the basic mesh decoding unit, or is the same as the number of submeshes of the basic mesh.

3. The mesh decoding device described in claim 1, characterized in that if the number of submeshes indicated by the variable is greater than the number of submeshes of the basic mesh output by the basic mesh decoding unit, the number of submeshes indicated by the variable is changed to the number of submeshes of the basic mesh.

4. The mesh decoding device described in claim 1, characterized in that if the number of submeshes indicated by the variable is greater than the number of submeshes of the base mesh, the submeshes of the base mesh that are insufficient to the number of submeshes indicated by the variable are made submeshes of a specific base mesh.

5. 5. The mesh decoding device according to claim 4, wherein the submesh of the specific basic mesh is made the first submesh among the submeshes with the same number of vertices.

6. 5. The mesh decoding device according to claim 4, wherein the submesh of the specific basic mesh is the last submesh of the basic mesh.

7. 1. A mesh decoding method, comprising: a step A of decoding the atlas bitstream and outputting control information; and a step B of decoding the basic mesh bitstream to generate and output a basic mesh; In the step B, the number of sub-meshes of the basic mesh of the frame to be decoded is calculated and output from the control signal defined in the basic mesh sub-bitstream; A mesh decoding method characterized in that a variable indicating the number of submeshes in the frame to be decoded is calculated from a control signal defined in the atlas sub-bitstream, and in step B, a displacement amount and texture are added to the submeshes of the basic mesh decoded to reproduce the submeshes of the number of submeshes.

8. A program that causes a computer to function as a mesh decoding device, The mesh decoding device comprises: an atlas data decoder that decodes the atlas bitstream and outputs control information; a basic mesh decoding unit that decodes the basic mesh bitstream, generates a basic mesh, and outputs the generated basic mesh; The basic mesh decoding unit calculates and outputs the number of sub-meshes of the basic mesh of the frame to be decoded from the control signal defined in the basic mesh sub-bitstream; A program characterized in that a variable indicating the number of submeshes in a frame to be decoded is calculated from a control signal defined in an atlas sub-bitstream, and the submeshes of the basic mesh are decoded by the basic mesh decoding unit, and a displacement amount and texture are added to the submeshes of the basic mesh to reproduce the submeshes of the number of submeshes.

Images