Mesh decoding device, mesh decoding method, and program

Abstract

The present invention decodes a basic mesh or a displacement amount when the number of vertices of a current frame or a current sub-mesh is different from the number of vertices of a reference frame or a reference sub-mesh. In a mesh decoding device 200 according to the present invention, an inter-decoding unit 202E decodes the number of motion vectors of a current frame or a current sub-mesh from a bit stream, and adjusts the number of vertices and some vertex coordinates of the current frame or the current sub-mesh when the number of decoded motion vectors is different from the number of vertices of a reference frame or a reference sub-mesh.

Description

Mesh decoding device, mesh decoding method and program

[0001] The present invention relates to a mesh decoding device, a mesh decoding method, and a program.

[0002] Non-Patent Document 1 or Non-Patent Document 4 discloses a technique for encoding a mesh using Non-Patent Document 2 or 3 according to the framework of Non-Patent Document 5.

[0003] Khaled Mammou, Jungsun Kim, Alexis M Tourapis, Dimitri Podborski, and Krasimir Kolarov, “[V-CG] Apple's Dynamic Mesh Coding CfP Response,” April 2022, ISO / IEC JTC 1 / SC 29 / WG 7 m59281.Google Draco, accessed May 26, 2022 [Online], https: / / google.github.io / dracoJean-Eudes Marvie, Olivier Mocquard, “[V-DMC][EE4.4-related] An efficient EdgeBreaker implementation,” April 2023, ISO / IEC JTC 1 / SC 29 / WG 7 m63344. “WD 6.0 of V-DMC,” Jan. 2024, ISO / IEC JTC 1 / SC 29 / WG 7 N00822. “Information technology - Coded Representation of Immersive Media - Part 5: Visual Volumetric Video-based Coding (V3C) and Video-based Point Cloud Compression (V-PCC),” ISO / IEC JTC 1 / SC 29 / WG 7, ISO / IEC 23090-5:2021(2E).

[0004] However, in the prior art, there was a problem that in inter-prediction coding, if the number of vertices in the current frame or current submesh differs from the number of vertices in the reference frame or reference submesh, the mesh could not be decoded correctly. Therefore, the present invention has been made in consideration of the above-mentioned problem, and aims to provide a mesh decoding device, mesh decoding method, and program that can decode a basic mesh or a displacement amount even if the number of vertices in the current frame or current submesh differs from the number of vertices in the reference frame or reference submesh.

[0005] A first feature of the present invention is a mesh decoding device comprising an inter-decoding unit that decodes a base mesh by adding a motion vector decoded from an inter-frame bitstream to a base mesh of a reference frame, and a subdivision unit that subdivides the faces of the base mesh using the base mesh, wherein the inter-decoding unit decodes the number of motion vectors of a current frame or a current sub-mesh from the bitstream, and when the decoded number of motion vectors differs from the number of vertices of a reference frame or a reference sub-mesh, adjusts the number of vertices of the current frame or the current sub-mesh and some of the vertex coordinates.

[0006] A second feature of the present invention is a mesh decoding method comprising a step A of decoding a basic mesh by adding a motion vector decoded from an interframe bitstream to a basic mesh of a reference frame, and a step B of subdividing the faces of the basic mesh using the basic mesh, wherein in step A, the number of motion vectors of a current frame or a current submesh is decoded from the bitstream, and when the decoded number of motion vectors differs from the number of vertices of a reference frame or a reference submesh, the number of vertices of the current frame or the current submesh and the coordinates of some of the vertices are adjusted.

[0007] A third feature of the present invention is a program that causes a computer to function as a mesh decoding device, the mesh decoding device comprising an inter-decoding unit that decodes a base mesh by adding a motion vector decoded from an inter-frame bitstream to a base mesh of a reference frame, and a subdivision unit that subdivides the faces of the base mesh using the base mesh, wherein the inter-decoding unit decodes the number of motion vectors of a current frame or current sub-mesh from the bitstream, and when the decoded number of motion vectors differs from the number of vertices of a reference frame or reference sub-mesh, adjusts the number of vertices of the current frame or current sub-mesh and some of the vertex coordinates.

[0008] According to the present invention, it is possible to provide a mesh decoding device, a mesh decoding method and a program that can decode a basic mesh or a displacement amount even when the number of vertices of the current frame or current submesh is different from the number of vertices of the reference frame or reference submesh.

[0009] FIG. 1 is a diagram illustrating an example of the configuration of a mesh processing system 1 according to an embodiment. FIG. 2 is a diagram illustrating an example of functional blocks of a mesh decoding device 200 according to an embodiment. FIG. 3A is a diagram illustrating an example of a base mesh and a subdivision mesh. FIG. 3B is a diagram illustrating an example of a base mesh and a subdivision mesh. FIG. 4 is a diagram illustrating an example of functional blocks of a basic mesh decoding unit 202 of a mesh decoding device 200 according to an embodiment. FIG. 5 is a diagram illustrating an example of functional blocks of an intra-decoding unit 202B of the basic mesh decoding unit 202 of the mesh decoding device 200 according to an embodiment. FIG. 6 is a diagram illustrating an example of a correspondence relationship between vertices of a basic mesh of a P frame and vertices of a basic mesh of an I frame. FIG. 7 is a diagram illustrating Modifications 1 and 2-1. FIG. 8 is a diagram illustrating Modification 2-2. FIG. 9 is a diagram illustrating an example of measurement results of coding efficiency of a control signal. FIG. 10 is a diagram illustrating an example of functional blocks of an inter-decoding unit 202E of the basic mesh decoding unit 202 of a mesh decoding device 200 according to an embodiment. Fig. 11 is a diagram for explaining an example of a method for calculating the MVP of a vertex to be decoded by the motion vector prediction unit 202E3 of the inter decoding unit 202 of the basic mesh decoding unit 200 of an embodiment. Fig. 12 is a flowchart showing an example of the operation of the motion vector prediction unit 202E3 of the inter decoding unit 202 of the basic mesh decoding unit 202 of the mesh decoding device 200 of an embodiment. Fig. 13 is a diagram for explaining the mesh buffer unit 202C of the basic mesh decoding unit 202 of the mesh decoding device 200 of an embodiment. Fig. 14 is a diagram for explaining the mesh buffer unit 202C of the basic mesh decoding unit 202 of the mesh decoding device 200 of an embodiment. Fig. 15 is a diagram for explaining a modified example of the basic mesh decoding unit 202 of the mesh decoding device 200 according to Modification 2. Fig. 21 is a diagram for explaining a modified example of the basic mesh decoding unit 202 of the mesh decoding device 200 according to Modification 2. Fig. 17 is a diagram showing an example of a NAL header.

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the components in the following embodiments can be appropriately replaced with existing components, etc., and various variations, including combinations with other existing components, are possible. Therefore, the description of the following embodiments does not limit the content of the invention described in the claims.

[0011] First Embodiment A mesh processing system according to this embodiment will be described below with reference to FIGS.

[0012] 1 is a diagram showing an example of the configuration of a mesh processing system 1 according to this embodiment. As shown in FIG. 1, the mesh processing system 1 includes a mesh encoding device 100 and a mesh decoding device 200.

[0013] FIG. 2 is a diagram showing an example of functional blocks of a mesh decoding device 200 according to this embodiment.

[0014] As shown in Figure 2, the mesh decoding device 200 has a demultiplexing unit 201, a basic mesh decoding unit 202, a subdivision unit 203, a mesh decoding unit 204, a patch integration unit 205, a displacement amount decoding unit 206, a video decoding unit 207, and an atlas data decoding unit 208.

[0015] Here, the basic mesh decoding unit 202, the subdivision unit 203, the mesh decoding unit 204 and the displacement amount decoding unit 206 are configured to perform processing in units of patches into which the mesh is divided, and the results of these processes may then be integrated by the patch integration unit 205.

[0016] In the example of FIG. 3A, the mesh is divided into patch 1 consisting of basic faces 1 and 2 and patch 2 consisting of basic faces 3 and 4.

[0017] The demultiplexing unit 201 is configured to separate the multiplexed bit stream into a base mesh bit stream, a displacement bit stream, a texture bit stream, and an atlas bit stream.

[0018] The subdivision unit 203 is configured to generate and output subdivision vertices and their connection information from the basic mesh decoded by the basic mesh decoding unit 202, using the subdivision method indicated by the control information (first control information and second control information). The basic mesh is composed of one submesh or multiple submeshes.

[0019] Here, the basic mesh, the added subdivision vertices, and the information on their connections are collectively referred to as a "subdivision mesh." Also, the submesh, the added subdivision vertices, and the information on their connections are collectively referred to as a "subdivision submesh."

[0020] The mesh decoding unit 204 is configured to generate and output a decoded mesh using the control information, the subdivision mesh, the subdivision vertex normals, and the displacement amounts.

[0021] The displacement amount decoding unit 206 is configured to decode the displacement amount bit stream based on the control information to generate and output the displacement amount.

[0022] The video decoding unit 207 is configured to decode the texture using a video codec and output the decoded texture.

[0023] The atlas data decoder 208 is configured to decode the atlas bitstream and output control information, which may be used as metadata by the base mesh decoder 202, the subdivision unit 203, the mesh decoder 204, the displacement decoder 206, and the video decoder 207.

[0024] <Basic Mesh Decoding Unit 202> The basic mesh decoding unit 202 is configured to decode the basic mesh bitstream, generate a basic mesh, and output it.

[0025] Here, the basic mesh is made up of a plurality of vertices in a three-dimensional space and edges connecting these vertices.

[0026] As shown in Figure 3A, a basic mesh or a decoded mesh is composed of a combination of basic faces or faces represented by three vertices. Therefore, the requirement for bitstream compatibility is that the decoded mesh has three or more vertices, or the control signal indicating the number of vertices is three or more.

[0027] Note that the basic mesh or the decoded mesh may be constructed by combining lines represented by two vertices, and therefore the requirement for bitstream conformance is that the decoded mesh has two or more vertices, or the control signal indicating the number of vertices is two or more.

[0028] The base mesh decoder 202 may be configured to decode the base mesh bitstream using, for example, the techniques described in Draco in Non-Patent Document 2 or in Non-Patent Document 3.

[0029] As shown in FIG. 4, the basic mesh decoding unit 202 includes a separating unit 202A, an intra decoding unit 202B, a mesh buffer unit 202C, a connection information decoding unit 202D, and an inter decoding unit 202E.

[0030] The separator 202A is configured to classify the basic mesh bitstream into a bitstream of I frames and a bitstream of P frames.

[0031] (Intra decoding unit 202B) The intra decoding unit 202B is configured to decode the coordinates and connection information of the vertices of the I frame from the bit stream of the I frame, for example, by using Draco shown in Non-Patent Document 2 or the technology described in Non-Patent Document 3.

[0032] FIG. 5 is a diagram showing an example of functional blocks of the intra decoder 202B.

[0033] As shown in FIG. 5, the intra decoding unit 202B includes an arbitrary intra decoding unit 202B1 and an alignment unit 202B2.

[0034] The optional intra decoder 202B1 is configured to decode the coordinates and connectivity information of the unordered vertices of the I-frame from the bitstream of the I-frame using any method including Draco shown in Non-Patent Document 2 or the technology described in Non-Patent Document 3.

[0035] The sorting unit 202B2 is configured to output the vertices by sorting the unordered vertices into a predetermined order.

[0036] The predetermined order may be, for example, a Morton code order or a raster scan order.

[0037] Furthermore, the alignment unit 202B2 may combine overlapping vertices, which are multiple vertices with the same coordinates in the decoded basic mesh, into a single vertex, and then rearrange them in a predetermined order.

[0038] The mesh buffer unit 202C is configured to store the coordinates and connection information of the vertices of the I frame decoded by the intra decoding unit 202B. Here, a specific buffer may be provided to store pairs of vertex indices A(k) and B(k) of overlapping vertices in a predetermined order.

[0039] The connection information decoding unit 202D is configured to convert the connection information of the I frame or reference frame extracted from the mesh buffer unit 202C into connection information of the P frame.

[0040] The inter-decoding unit 202E is configured to decode the coordinates of the vertices of the P frame by adding the coordinates of the vertices of the reference frame retrieved from the mesh buffer unit 202C and the motion vectors decoded from the bitstream of the P frame.

[0041] Furthermore, the inter-decoding unit 202E can adjust the index of the vertex of the P frame using the pair of vertex indices A(k) and B(k) that exist as overlapping vertices stored in the specific buffer.

[0042] Here, all or part of the indexes are decoded from the bitstream. The decoding method may be arithmetic coding. As a result, it is expected that there is no limit to the maximum value of the index to be decoded using arithmetic coding.

[0043] For example, arithmetic coding called ue(v) may be used, where ue(v) represents leftmost bit-first unsigned integer zeroth-order exponential-Golomb coding (Exp-Golomb).

[0044] Specifically, the parsing process for the syntax element of ue(v) begins from the current position in the bitstream, reading the bits containing the first non-zero bit, and counting the number of leading bits equal to 0. This process is specified as follows: leadingZeroBits=-1 for(b=0;!b;leadingZeroBits++ b=read_bits(l)) Next, the variable codeNum is assigned as follows:

[0045] codeNum = 2 leadingZeroBits - 1 + read_bits (leadingZeroBits) where the value returned by read_bits (leadingZeroBits) is interpreted as a binary representation of an unsigned integer with the most significant bit written first. Also, the value of ue(v) is equal to the value of codeNum.

[0046] Table 1 shows the structure of an Exp-Golomb code, separating the bit string into "prefix" bits and "suffix" bits.

[0047]

[0048] Here, the "prefix" bits are the bits that are parsed as specified in the calculation of leadingZeroBits and are represented as 0 or 1 in the Bit String column of Table 1.

[0049] The "suffix" bits are the bits that are parsed in the calculation of codeNum, and are denoted by x in Table 1. ｉ where i ranges from 0 to leadingZeroBits-1. Each x ｉis equal to either 0 or 1.

[0050] Table 2 shows how to explicitly assign bit strings to values ​​of codeNum, where the value of ue(v) is equal to the value of codeNum.

[0051]

[0052] In this embodiment, there is a correspondence between the vertices of the base mesh of the P frame and the vertices of the base mesh of the reference frame (I frame or P frame), as shown in Figure 6. Here, the motion vector decoded by the inter decoding unit 202E is a difference vector between the coordinates of the vertices of the base mesh of the P frame and the coordinates of the vertices of the base mesh of the I frame.

[0053] In addition, the inter decoding unit 202E may decode a control signal indicating the number of vertices or the number of motion vectors of the current frame or current submesh from the bitstream. <Modification 1: When the control signal indicates the number of vertices of the current frame or current submesh> Here, when the inter decoding unit 202E decodes the number of vertices of the current frame or current submesh from the above-mentioned bitstream, the requirement for the compatibility of such a bitstream is that the number of vertices of the decoded current frame or current submesh must be equal to the number of vertices of the reference frame or reference submesh.

[0054] In addition, when decoding the number of vertices of the current frame or current submesh from the above-mentioned bitstream, if the number of vertices of the decoded current frame or current submesh differs from the number of vertices of the reference frame or reference submesh, the inter-decoding unit 202E is configured to preferentially use the number of vertices of the current frame or current submesh decoded from the bitstream.

[0055] Furthermore, the inter decoding unit 202E may use the number of vertices of the reference frame or reference submesh as the number of vertices of the current frame or current submesh.

[0056]

[0057] In such a case, if the number of vertices in the current frame or current submesh is greater than the number of vertices in the reference frame or reference submesh (Current Submesh case 2 in Table 3), the inter-decoding unit 202E may add dummy vertices to the reference frame or reference submesh.

[0058] Here, the inter decoding unit 202E may set the coordinates of such dummy vertices to fixed values ​​(for example, (0,0,0)), or may copy them from a predetermined vertex (for example, the last vertex of the reference frame).

[0059] However, since such dummy vertices do not have connection information, the inter-decoding unit 202E can discard such dummy vertices during processing or as post-processing, and when discarding such dummy vertices, it changes the number of vertices of the current submesh accordingly.

[0060] In addition, when the number of vertices in the current frame or current submesh is decoded from the bitstream and the number of vertices in the current frame or current submesh is less than the number of vertices in the reference frame or reference submesh (Current Submesh case 2 in Table 3), the inter decoding unit 202E calculates the vertex coordinates up to the number of vertices in the reference frame or reference submesh, and sets the number of vertices in the current frame or current submesh to the number of vertices in the reference frame or reference submesh.

[0061] Specifically, the inter-decoding unit 202E first calculates the vertex coordinates up to the number of vertices in the current frame or current submesh. However, there are vertices whose vertex coordinates have not been calculated, such as v3, v4, and v5 in Figure 7(c).

[0062] Here, since the connection information is copied from the reference submesh, there are faces that have connection information but do not have vertex coordinates, such as faces (v0, v2, v3), (v2, v4, v3), (v2, v5, v4), and (v1, v5, v2) shown by dotted lines in Figure 7(c).

[0063] Hereinafter, a vertex for which vertex coordinates have not been calculated will be referred to as an abnormal vertex, and a face for which there is connectivity information but no vertex coordinates will be referred to as an abnormal face.

[0064] When an abnormal surface is subjected to subdivision, there is a problem in that the coordinates of the midpoint cannot be calculated as one of the outputs of the subdivision.

[0065] To address this issue, the following solutions 1 / 2 are available.

[0066] Solution 1: When the number of vertices in the current frame or current submesh is smaller than that of the reference submesh, the inter-decoding unit 202E may set the coordinates of the abnormal vertex to a fixed value (e.g., (0,0,0)) or copy the coordinates from a predetermined vertex (e.g., select one from the non-abnormal vertices). Also, the inter-decoding unit 202E changes the number of vertices in the current frame or current submesh to the number of vertices in the reference submesh.

[0067] Solution 2: When the number of vertices in the current frame or current submesh is smaller than that in the reference submesh, the inter-decoding unit 202E may copy the coordinates of the abnormal vertices from the vertices of the reference submesh with the same vertex index, and change the number of vertices in the current frame or current submesh to the number of vertices in the reference submesh.

[0068] With this configuration, it is expected that the decoding operation of the current frame or current submesh will be guaranteed. <Modification 2: When the control signal indicates the number of motion vectors of the current frame or current submesh> Modification 2 will explain the case where the current submesh is used. Note that Modification 2 can also be applied to the case where the current frame is used. (Modification 2-1) Here, when the inter decoding unit 202E decodes the number of motion vectors of the current submesh from the above-mentioned bitstream, if the number of motion vectors of the decoded current submesh differs from the number of vertices of the reference submesh, it adjusts the number of vertices of the current submesh and the coordinates of some of the vertices.

[0069] First, as an initialization operation, the inter-decoding unit 202E copies the submesh ID, number of faces, vertex coordinates and connection information of the reference submesh to the current submesh, and calculates the number of vertices of the current submesh using equation (A).

[0070] DecSummesh.verCoordCount = max(refSubmesh.verCoordCount, bm_vertex_count[submeshID]) ... Equation (A) where DecSummesh.verCoordCount is a control signal indicating the number of vertices of the current submesh, refSubmesh.verCoordCount is a control signal indicating the number of vertices of the reference submesh, and bm_vertex_count[submeshID] is a control signal indicating the number of motion vectors of the current submesh.

[0071] That is, when the number of motion vectors of the current submesh is greater than the number of vertices of the reference submesh, the inter decoding unit 202E sets the number of vertices of the current submesh to the number of motion vectors of the current submesh.

[0072] As a result of the above operations, the number of vertices of the current submesh may become the same as the number of vertices of the reference submesh, or the number of vertices of the current submesh may become greater than the number of vertices of the reference submesh.

[0073] However, at this point, the number of vertices included in the vertex coordinates of the current submesh is the same as that of the reference submesh.

[0074] Furthermore, as shown in Table 4 and FIG. 7, there are cases where the number of motion vectors of the current submesh is the same as the number of vertices of the reference submesh (currSubmesh SW), where the number of motion vectors of the current submesh is fewer than the number of vertices of the reference submesh (currSubmesh case 1), or where the number of motion vectors of the current submesh is more than the number of vertices of the reference submesh (currSubmesh case 2).

[0075]

[0076] In such a case, when the number of vertices of the current submesh is greater than the number of vertices of the reference submesh (currentSubmesh case 2 in Table 4), and when the vertex index of the current submesh is greater than or equal to the number of vertices of the reference submesh (there is no corresponding vertex of the reference submesh), the inter-decoding unit 202E calculates the vertex coordinates using the following formula (B):

[0077] - Otherwise (v is greater than refSubmesh.verCoordCount-1), DecSummesh.verCoords[v][k] = currentSubmeshMotionVectors[v][k], where k = 0..2 ... formula (B) where refSubmesh.verCoordCount is the number of vertices of the reference submesh, currentSubmeshMotionVectors[v][k] is the k-th dimension motion vector of the v-th vertex of the current submesh, and DecSummesh.verCoords[v][k] is the calculated k-th dimension vertex coordinate of the v-th vertex of the current submesh.

[0078] Furthermore, since the above-mentioned dummy vertices do not have connection information, the inter-decoding unit 202E can discard such dummy vertices during processing or as post-processing, and if such dummy vertices are discarded, the number of vertices of the current submesh is changed accordingly.

[0079] By the above-mentioned operation, when the number of motion vectors of the current submesh is greater than the number of vertices of the reference submesh, the inter-decoding unit 202E calculates vertex coordinates up to the number of vertices of the reference submesh using motion vectors, and leaves the other vertex coordinates as motion vectors.

[0080] In such a case, when the number of motion vectors of the current submesh is less than or equal to the number of vertices of the reference submesh, the inter-decoding unit 202E calculates vertex coordinates using motion vectors up to the number of motion vectors of the current submesh, and leaves the other vertex coordinates as they are in the reference submesh.

[0081] According to this configuration, it is expected that the decoding operation of the current frame or the current submesh can be guaranteed. (Modification 2-2) Here, when the inter decoding unit 202E decodes the number of motion vectors of the current submesh from the above-mentioned bitstream, if the decoded number of motion vectors of the current submesh differs from the number of vertices of the reference submesh, the inter decoding unit 202E adjusts the number of vertices of the current submesh and the coordinates of some of the vertices.

[0082] First, as an initialization operation, the inter-decoding unit 202E copies the submesh ID, the number of vertices, the number of faces, the vertex coordinates, and the connection information of the reference submesh to the current submesh.

[0083] The above operation ensures that the number of vertices in the current submesh is the same as the number of vertices in the reference submesh.

[0084] However, as shown in Table 5 and FIG. 8, the number of motion vectors may be the same as the number of vertices of the reference submesh (currSubmesh SW), may be less than the number of vertices of the reference submesh (currSubmesh case 1), or may be more than the number of vertices of the reference submesh (currSubmesh case 2).

[0085]

[0086] In such a case, if the number of motion vectors of the current submesh is less than or equal to the number of vertices of the reference submesh, the inter-decoding unit 202E calculates vertex coordinates up to the number of motion vectors of the current submesh using the decoded motion vectors, and leaves the other vertex coordinates as they are in the reference submesh.

[0087] In such a case, if the number of motion vectors of the current submesh is greater than the number of vertices of the reference submesh, the inter-decoding unit 202E sets the number of vertices of the current submesh to the number of vertices of the reference frame, and calculates vertex coordinates up to the number of vertices of the reference submesh using the decoded motion vectors.

[0088] To summarize the above operations, the inter-decoding unit 202E calculates the vertex coordinates when the vertex index of the current submesh is between 0 and min(refSubmesh.vertex_count, bm_vertex_count[submeshID])-1.

[0089] where refSubmesh.vertex_count is the number of vertices of the reference submesh, and bm_vertex_count[submeshID] is the number of motion vectors of the current submesh.

[0090] In other words, when the number of motion vectors of the current submesh is greater than the number of vertices of the reference submesh, the inter-decoding unit 202E sets the number of vertices of the current frame or current submesh to the number of vertices of the reference frame, and calculates vertex coordinates up to the number of vertices of the reference submesh using the decoded motion vectors.

[0091] According to this configuration, it is expected that the decoding operation of the current frame or the current submesh can be guaranteed.

[0092] The inter-decoding unit 202E may decode the control signal indicating the number of vertices of the basic mesh or after subdivision for each frame or each submesh using the bit analysis method described in Table 6.

[0093]

[0094] For example, such control signals include mesh_vertex_count, pdu_vertex_count_minus_1[titleID][patchIdx], sismu_inter_vertex_count[subMeshID], and the like.

[0095] Furthermore, the inter decoding unit 202E may decode the above control signal using u(v), ue(v), ae(v), vu(v), or the like.

[0096] For example, the inter decoder 202E may decode the above-described control signal using arithmetic coding called ue(v), where ue(v) represents leftmost bit-first unsigned integer zeroth-order exponential-Golomb coding (Exp-Golomb).

[0097] Furthermore, the inter decoding unit 202E may decode the above control signal using arithmetic coding called ae(v) using CABAC.

[0098] As a result, it is expected that there will be no limit to the maximum number of vertices to be decoded using arithmetic coding.

[0099] However, since the value of the control signal indicating the number of vertices is large, there is a problem in that the efficiency of ue(v) and ae(v) is poor.

[0100] Therefore, if there is a maximum number of vertices maxVC in all frames or submeshes, the inter decoding unit 202E may decode the above control signal using u(v) instead of ue(v).

[0101] In this case, the inter decoding unit 202E calculates v from the maximum number of vertices maxVC of all frames or sub-meshes. Specifically, the inter decoding unit 202E calculates v using the following equation (C).

[0102]

[0103] This configuration is expected to have the effect of reducing the amount of code required for the control signal indicating the number of vertices. Note that the effect of setting the maximum number of vertices maxVC for all frames or submeshes is useful when defining profiles and levels.

[0104] Furthermore, using vu(v) instead of u(v) has the effect of improving coding efficiency compared to using ue(v), as shown in Figure 9, and vu(v) can be analyzed using the following procedure without calculating v.

[0105] That is, the first bit of every eight bits is a flag indicating whether the next eight bits should be analyzed, and the other seven bits indicate the value.

[0106] vu(v) { value = 0 do { continue = read_bits( 1 ) partial_value = read_bits( 7 ) value = ( value << 7 ) | partial_value} while( continue ) return value Note that using the same bit analysis method for the control signal indicating the number of vertices in the basic mesh of the current frame or current submesh as for other syntaxes indicating the number of vertices has the effect of simplifying hardware design.

[0107] Furthermore, in the bit analysis method of syntax indicating the number of faces such as face_count, vu(v) may be changed to ue(v).

[0108] Since the syntax indicating the number of vertices and faces described above is a large value, unifying the syntax bit analysis method has the effect of reducing the complexity of the decoder and the effect of reducing the complexity of the hardware design and enabling sharing of chips.

[0109] In addition, a requirement for bitstream compatibility may be that the above-mentioned pdu_vertex_count_minus_1[titleID][patchIdx] and sismu_inter_vertex_count[subMeshID] do not contradict each other, or that pdu_vertex_count_minus_1[titleID][patchIdx] and mesh_vertex_count do not contradict each other.

[0110] Furthermore, if there is a conflict between pdu_vertex_count_minus_1[titleID][patchIdx] and sismu_inter_vertex_count[subMeshID], or if there is a conflict between pdu_vertex_count_minus_1[titleID][patchIdx] and mesh_vertex_count, priority may be given to sismu_inter_vertex_count[subMeshID] or mesh_vertex_count.

[0111] The atlas data decoding unit 208 calculates a variable called AtlasTotalNumMeshpatches from a control signal called afmi_num_submeshes_minus2 defined in the atlas sub-bitstream.

[0112] Here, AtlasTotalNumMeshpatches indicates the displacement amount and the number of texture sub-meshes in the corresponding frame.

[0113] The basic mesh decoding unit 202 calculates the number of sub-meshes of the basic mesh in the frame to be decoded using bmsi_num_submeshes_minus2 defined in the basic mesh sub-bitstream.

[0114] Since afmi_num_submeshes_minus2 and bmsi_num_submeshes_minus2 are independent, there is a problem in that the displacement amount indicated by the calculated AtlasTotalNumMeshpatches or the number of submeshes of the texture may differ from the number of submeshes of the base mesh.

[0115] In the above-mentioned non-patent document 4, displacement and texture are added to sub-meshes of a basic mesh, and the number of sub-meshes indicated by AtlasTotalNumMeshpatches calculated using a control signal output from the atlas data decoding unit 208 is reproduced as the final sub-meshes.

[0116] Therefore, when the number of sub-meshes of the basic mesh is less than the number of sub-meshes indicated by AtlasTotalNumMeshpatches, the final sub-meshes corresponding to the missing number cannot be reproduced because there are not enough sub-meshes of the basic mesh.

[0117] Conversely, when the number of submeshes of the base mesh is greater than the number indicated by AtlasTotalNumMeshpatches, there will be excess submeshes of the base mesh, and the displacement and texture of the excess final submeshes may be treated as non-existent.

[0118] Alternatively, the remaining final sub-meshes may be reconstructed using the existing displacements and textures of the remaining basic meshes. This reconstruction is performed in the reconstruction process (see FIG. 2). This reconstruction process is performed by the subdivision unit 203, mesh decoding unit 204, and patch integration unit 205.

[0119] In this embodiment, the requirement for bitstream compatibility may be that AtlasTotalNumMeshpatches is less than the number of submeshes of the base mesh, or that AtlasTotalNumMeshpatches is the same as the number of submeshes of the base mesh.

[0120] Furthermore, when AtlasTotalNumMeshpatches is greater than the number of sub-meshes of the basic mesh, AtlasTotalNumMeshpatches may be changed to the number of sub-meshes of the basic mesh.

[0121] Alternatively, when AtlasTotalNumMeshpatches is greater than the number of submeshes of the basic mesh, the submeshes of the basic mesh are insufficient, and the insufficient submeshes may be made into submeshes of a specific basic mesh.

[0122] For example, the first submesh among the submeshes with the same number of vertices (or the last submesh of the basic mesh) may be selected as a submesh of the specific basic mesh.

[0123] This configuration is expected to solve the problem of some sub-meshes not being able to be reproduced by the atlas data decoding unit 208. (Inter Decoding Unit 202E) Fig. 10 is a diagram showing an example of functional blocks of the inter decoding unit 202E.

[0124] As shown in FIG. 10, the inter decoding unit 202E includes a motion vector residual decoding unit 202E1, a motion vector buffer unit 202E2, a motion vector prediction unit 202E3, a motion vector calculation unit 202E4, and an adder 202E5.

[0125] The motion vector residual decoding unit 202E1 is configured to generate a motion vector residual (MVR) from a P frame bitstream.

[0126] Here, MVR is a motion vector residual indicating the difference between MV (Motion Vector) and MVP (Motion Vector Prediction). MV is a difference vector (motion vector) between the coordinates of a vertex of the corresponding I frame and the coordinates of a vertex of the corresponding P frame. MVP is a value (motion vector prediction value) predicted by using MV of the MV of the target vertex.

[0127] The motion vector buffer unit 202E2 is configured to sequentially store the motion vectors output by the motion vector calculation unit 202E4.

[0128] The motion vector prediction unit 202E3 is configured to obtain decoded MVs from the motion vector buffer unit 202E2 for vertices connected to the vertex to be decoded, and to output the MVP of the vertex to be decoded using all or part of the obtained decoded MVs, as shown in Figure 11.

[0129] The motion vector calculation unit 202E4 is configured to add the MVR generated by the motion vector residual decoding unit 202E1 and the MVP output from the motion vector prediction unit 202E3, and output the MV of the vertex to be decoded.

[0130] The adder 202E5 is configured to add the coordinates of the vertex corresponding to the vertex to be decoded, obtained from the decoded basic mesh of the corresponding reference frame (I frame or P frame), to the motion vector MV output from the motion vector calculation unit 202E3, and output the coordinates of the vertex to be decoded.

[0131] Each component of the inter decoding unit 202E will be described in detail below.

[0132] 12 is a flowchart showing an example of the operation of the motion vector prediction unit 202E3. Hereinafter, the operation of the motion vector prediction unit 202E3 will be referred to as the "average prediction method."

[0133] As shown in FIG. 12, in step S1001, the motion vector prediction unit 202E3 sets MVP and N to 0.

[0134] In step S1002, the motion vector prediction unit 202E3 obtains a set of MVs of vertices surrounding the vertex to be decoded from the motion vector buffer unit 202E2, identifies vertices for which subsequent processing has not been completed, and transitions to No. If subsequent processing has been completed for all vertices, it transitions to Yes.

[0135] In step S1003, if the MV of the vertex to be processed has not been decoded, the motion vector prediction unit 202E3 transitions to No, and if the MV of the vertex to be processed has been decoded, the motion vector prediction unit 202E3 transitions to Yes.

[0136] In step S1004, the motion vector prediction unit 202E3 adds MV to MVP and adds 1 to N.

[0137] In step S1005, if N is greater than 0, the motion vector prediction unit 202E3 outputs the result of dividing MVP by N, and if N is 0, it outputs 0, and ends the process.

[0138] That is, the motion vector prediction unit 202E3 is configured to output the MVP to be decoded by averaging the decoded motion vectors of the vertices around the vertex to be decoded.

[0139] The motion vector prediction unit 202E3 may be configured to set the MVP to 0 if the set of decoded motion vectors is an empty set.

[0140] The motion vector calculation unit 202E4 may be configured to calculate the MV of the vertex to be decoded from the MVP output by the motion vector prediction unit 202E3 and the MVR generated by the motion vector residual decoding unit 202E1 using equation (1).

[0141] MV(k)=MVP(k)+MVR(k) (1) where k is the index of the vertex, and MV, MVR, and MVP are vectors having x, y, and z components.

[0142] According to this configuration, since only the MVR is coded instead of the MV using the MVP, it is expected that the coding efficiency will be improved.

[0143] The adder 202E5 is configured to calculate the coordinates of a vertex by adding the MV of the vertex calculated by the motion vector calculation unit 202E4 to the coordinates of the vertex in the reference frame corresponding to the vertex, and to leave the connectivity information (Connectivity) as it is in the reference frame.

[0144] Specifically, the adder 202E5 calculates the coordinate v' of the kth vertex using equation (2). ｉ (k) may be calculated.

[0145] v' ｉ (k)=v′ ｊ (k) + MV(k) ... (2) where v' ｉ (k) is the coordinate of the kth vertex to be decoded in the frame to be decoded, and v' ｊ (k) is the coordinate of the decoded k-th vertex of the reference frame, and MV(k) is the k-th MV of the frame to be decoded, where k=1, 2, . . . , K.

[0146] Furthermore, the connection information of the frame to be decoded is made the same as the connection information of the reference frame.

[0147] It should be noted that the motion vector prediction unit 202E3 calculates the MVP using the decoded MV, and therefore the order of decoding affects the MVP.

[0148] The decoding order is the order in which the vertices of the base mesh in the reference frame are decoded. Generally, if a decoding method is used that uses a fixed repetition pattern to increase the number of base faces by one from the starting edge, the order of the vertices of the decoded base mesh is determined during the decoding process.

[0149] For example, the motion vector prediction unit 202E3 may use an Edgebreaker to determine the order in which vertices are decoded in the base mesh of the reference frame.

[0150] According to this configuration, since MVs from the reference frame are encoded instead of vertex coordinates, it is expected that the encoding efficiency will be improved. (Mesh Buffer Unit 202C) The mesh buffer unit 202C is configured to store one or more reference decoded base meshes in a predetermined order.

[0151] Such a basic mesh has metadata such as a frame number and a submesh number, as well as at least the coordinates of each vertex and the index of that vertex, and is stored in the mesh buffer unit 202C in a predetermined order determined by the reference frame list.

[0152] As shown in FIG. 13, the reference frame list (ref_list0) is a list of information that identifies all reference decoding basic meshes stored in the mesh buffer unit 202C.

[0153] The reference frame list may be determined by control signals decoded from the bitstream, as shown in FIG. 13, or may be calculated naturally from the decoding order of the frames.

[0154] The control signal decoded from the bitstream may be represented by a relative distance from the frame to be decoded, or may be represented by a frame index that is an absolute value.

[0155] Additionally, the control signal may use a short-term or long-term frame of reference.

[0156] For example, when a short-term reference frame is used, the absolute value (abs_delta_mfoc_st) of the difference in display order (Display Order) between the current frame (cur) and the reference frame (ref) and its sign (sign_flag) may be decoded from the bitstream, and the display order (Display Order) of the reference frame may be specified by the following formula: If(sign_flag) { Display Order(ref) = Display Order(cur) + abs_delta_mfoc_st} else { Display Order(ref) = Display Order(cur) - abs_delta_mfoc_st} Furthermore, when a method of calculating naturally from the decoding order of frames is used, for example, when no control signal is present in the reference frame list, frames may be arranged in order of a certain number of frames starting from the most recently decoded frame. That is, the reference frame list may be set to {0, -1, -2, ..., -(N-1)}.

[0157] Basically, the reference frame list does not change for each frame except in special circumstances (for example, when a re-ordering instruction is received).

[0158] The mesh buffer unit 202C may be updated as follows.

[0159] When a basic mesh is decoded, in the case of an I frame or a P frame, the mesh buffer unit 202C deletes one or more existing reference frames in a predetermined order determined by the reference frame list, and inserts one or more basic meshes including the basic mesh of the decoded frame, or creates and inserts one basic mesh from multiple basic meshes, thereby adjusting the order of the reference frames.

[0160] This deletion operation may be performed only when the mesh buffer unit 202C is full. The number of basic meshes that can be stored in the mesh buffer unit 202C is predetermined. In this embodiment, the mesh buffer unit 202C is defined as being full when the number of basic meshes is reached.

[0161] In the above-described creation process, a single basic mesh may be created by weighting the coordinates of the vertices corresponding to the basic meshes of the decoded frame and the existing basic meshes stored in the mesh buffer unit 202C.

[0162] The weights used in such a weighted average may be predetermined, calculated using the frame index, or decoded from the control signal.

[0163] When the mesh buffer unit 202C receives a control signal indicating a re-ordering instruction from the control signal decoded from the bitstream, it updates the reference frame list as shown in Figure 14 and adjusts the order of the reference frames according to the predetermined order determined by the updated reference frame list (ref_list0). (Inter decoding unit 202E) The inter decoding unit 202E is configured to decode the coordinates of the vertices of the P frame by adding the coordinates of the vertices of the reference frame extracted from the mesh buffer unit 202C to the motion vectors decoded from the bitstream of the P frame.

[0164] Furthermore, the inter-decoding unit 202E can adjust the indexes of the vertices of the P frame using the pair of indexes A(k) and B(k) of the vertices existing as overlapping vertices stored in the specific buffer. All or part of these indexes are decoded from the bitstream. This decoding method may be arithmetic coding. This configuration is expected to have the effect of eliminating the limit on the maximum value of the index to be decoded using arithmetic coding. For example, arithmetic coding called ue(v) may be used.

[0165] Furthermore, the above-mentioned method of signaling and parsing information about overlapping vertices may be as follows.

[0166]

[0167] Here, bmidu_derived_mv_present_flag[submeshID] is a flag indicating whether bmidu_mv_signaled_flag is present in the bitstream.

[0168] bmidu_mv_signaled_flag_count[submeshID] is the number of bmidu_mv_signaled_flag.

[0169] bmidu_mv_signaled_flag is a flag that indicates whether motion vectors for overlapped vertices are present in the bitstream.

[0170] In parsing, the number of motion vectors that are not signaled in the bitstream is counted, and the index of the corresponding motion vector is not specified in the submesh.

[0171] In addition, during the decoding process, a conversion process is performed to identify the index of the corresponding motion vector within the sub-mesh.

[0172] With this configuration, it is expected that only parsing can be performed to the end of the stream without relying on complex processing such as decoding, etc. (Modification 2 of basic mesh decoding unit 202) Hereinafter, modification 2 of the basic mesh decoding unit 202 will be described with reference to Fig. 15 .

[0173] As shown in FIG. 15, in the inter decoding unit 202E, the decoding order and the display order are different in order to allow reference to subsequent frames.

[0174] Here, the display order is the same as the input order when encoding, and is the same as the output order when decoding.

[0175] On the other hand, the decoding order is the same as the output order when encoding, and is the same as the input order when decoding.

[0176] The reference frame may be calculated by taking a weighted average of the subsequent frame and one or more other frames.

[0177] However, when referring to multiple frames including subsequent frames, MR_SUBMESH (MR frame or B frame) is defined as a new frame type (smh_type), and MR_SUBMESH is decoded from the bitstream.

[0178] Furthermore, the other frame may be a decoded frame immediately before the target frame, as shown in FIG.

[0179] Such weights may be calculated using the frame interval between the target frame and the subsequent frame and the frame interval between the target frame and another frame, or may be determined in advance.

[0180] The basic mesh decoding unit 202 decodes the control signal (smh_mesh_from_order_cnt_lsb) from the bitstream and decodes the order of the outputs.

[0181] When sub-meshes as defined in the above-mentioned non-patent document 4 exist, all sub-meshes are set to the same control signal (smh_mesh_from_order_cnt_lsb) or the control signal (smh_mesh_from_order_cnt_lsb) is applied to all sub-meshes.

[0182] The value indicated by this control signal (smh_mesh_from_order_cnt_lsb) may be a difference from the display order of the frame to be decoded, or may be an order within a predetermined frame group MaxMeshFrmOrderCntLsb.

[0183] In addition, when the decoding order (Decode Order) and the display order (Display Order) are different, if the decoded basic meshes are arranged in the decoding order (Decode Order), the basic mesh decoding unit 202 may rearrange the decoded basic meshes in the display order (Display Order).

[0184] In order to achieve temporal scalability, control signals indicating whether or not to decode the basic mesh, the displacement, and the texture are defined for each frame, and are decoded from the bitstream.

[0185] Furthermore, the Temporal_ID of the atlas and the base mesh may be matched within the same frame. Furthermore, the Temporal_ID of the atlas and the texture may be matched within the same frame. Furthermore, the Temporal_ID of the atlas and the displacement amount may be matched within the same frame.

[0186] According to this configuration, it is expected that it will be possible to avoid frames that cannot be decoded and unnecessary data.

[0187] It is desirable that the interval between adjacent frames having the same Temporal_ID is constant.

[0188] Adjacent frames with the same Temporal_ID have the closest POC.

[0189] By keeping the frame intervals constant as described above, it is expected that a constant frame rate can be maintained when displaying decoded frames.

[0190] Furthermore, the decoding orders of atlases and base meshes having the same display order may be matched, the decoding orders of atlases and displacements having the same display order may be matched, and the decoding orders of atlases and textures having the same display order may be matched.

[0191] Alternatively, the random access points of an atlas and a base mesh that have the same display order may be matched. Alternatively, the random access points of an atlas and a displacement that have the same display order may be matched. Alternatively, the random access points of an atlas and a texture that have the same display order may be matched. Note that random access points are defined in Non-Patent Document 4 or Non-Patent Document 5.

[0192] According to this configuration, when decoding the basic mesh, the displacement amount, and the texture, it is possible to expect the effect that the mesh can be reproduced without waiting for each other to be decoded.

[0193] Furthermore, a frame having a Temporal_ID higher than the control signal Temporal_ID of the current frame to be decoded is not used as a reference frame for the current frame to be decoded.

[0194] This is expected to have the effect of eliminating the possibility of reference frames being discarded.

[0195] An example of realizing temporal scalability using the above-mentioned Temporal_ID will be described below.

[0196] The atlas, base mesh, displacement, and texture bitstreams are encapsulated by Network Abstraction Layer (NAL) units, which may have a NAL header as shown in Figure 22.

[0197] The TID, defined as the last 3 bits in the NAL header, is the Temporal_ID plus 1. The range of TID is 1 to 7, with zero being prohibited.

[0198] LayerID / R6, defined as the 6 bits immediately preceding the TID in the NAL header, specifies the identifier of the layer to which the NAL unit belongs.

[0199] The value of LayerID / R6 must be in the range of 0 to 62. The value 63 may be specified by ISO / IEC in the future.

[0200] For purposes other than determining the amount of data in a decoded unit of the bitstream, the mesh decoding device 200 ignores all data following the value 63 in a NAL unit, and a mesh decoding device 200 conforming to the specified profile will ignore (i.e., remove from the bitstream and discard) all NAL units whose LayerID-R6 value is not 0.

[0201] The value 63 of LayerID / R6 can be used to indicate an enhancement layer identifier in future enhancements.

[0202] When sub-meshes defined in Non-Patent Document 4 exist, all sub-meshes are assigned the same TID or the TID is applied to all sub-meshes.

[0203] Non-Patent Document 5 can be used for the atlas, and HEVC or VVC, a video coding method, can be used for the displacement and texture, so the following will explain the basic mesh.

[0204] The LayerID / R6 values ​​of all BMCL NAL units of a coded basic mesh frame must be the same. The LayerID / R6 value of a coded basic mesh frame is the LayerID / R6 value of the BMCL NAL unit of the coded basic mesh frame.

[0205] If NALType is equal to NAL_EOB, the value of LayerID / R6 must be equal to 0.

[0206] If the NALType is in the range from NAL_BLA_W_LP to NAL_RSV_BMCL_29 defined in Non-Patent Document 4, i.e., if it belongs to an IRAP coded basic mesh frame, Temporal_ID must be 0.

[0207] If NALType is equal to NAL_TSA_R or NAL_TSA_N, Temporal_ID must not be equal to 0.

[0208] If NALType is equal to 0 and NALType is equal to NAL_STSA_R or NAL_STSA_N, Temporal_ID must not be equal to 0.

[0209] The value of Temporal_ID must be the same for all BMCL NAL units within an access unit.

[0210] The value of Temporal_ID of a coded basic mesh frame or access unit is the value of Temporal_ID of the BMCL NAL unit of the coded basic mesh frame or access unit.

[0211] The value of Temporal_ID of a sub-layer representation is the maximum value of Temporal_ID of all BMCL NAL units within the sub-layer representation.

[0212] The value of Temporal_ID for non-BMCL NAL units is restricted as follows: - if NALType is equal to NAL_BMSPS, Temporal_ID shall be 0 and the Temporal_ID of the access unit containing the NAL unit shall be 0. - otherwise, if NALType is equal to NAL_EOS or NAL_EOB, Temporal_ID shall be 0. - otherwise, if NALType is equal to NAL_AUD or NALLFDD, Temporal_ID shall be equal to the Temporal_ID of the access unit containing the NAL unit. - otherwise, Temporal_ID shall be greater than or equal to the Temporal_ID of the access unit containing the NAL unit.

[0213] Note that if the NAL unit is not BMCL, the value of Temporal_ID is equal to the minimum value of the Temporal_ID values ​​of all access units to which the non-BMCL NAL unit applies.

[0214] When NALType is equal to NAL_BMFPS, Temporal_ID can be greater than or equal to the Temporal_ID of the included access unit, since all basic mesh frame parameter sets (BMFPS) are included at the beginning of the bitstream, where the Temporal_ID of the first coded basic mesh frame is 0.

[0215] The skip decoding unit 202F refers to the specified tIDTarget and discards, without decoding, any NAL unit whose Temporal_ID is higher than tIDTarget.

[0216] Here, tIDTarget may be specified by a predetermined value, or may be specified based on the network status or the terminal capability of the mesh decoding device 200 .

[0217] For example, a lower tIDTarget is specified in the wireless case than in the wired case. Also, a lower tIDTarget is specified when the network condition is poor. Also, a lower tIDTarget is specified when decoding is performed by a mesh decoding device 200 with low specifications.

[0218] However, the requirement for bitstream conformance is that there must be at least one NAL unit in the bitstream whose Temporal_ID is not higher than tIDTarget.

[0219] Below, a modified example will be described in which temporal scalability is achieved using the above-mentioned Temporal_ID.

[0220] The atlas, base mesh, displacement and texture bitstreams are encapsulated by Network Abstraction Layer (NAL) units, which may have a NAL header as shown in Figure 17.

[0221] The TID, defined as the last 3 bits in the NAL header, is the Temporal_ID plus 1. The range of TID is 1 to 7, with zero being prohibited.

[0222] LayerID / R6, defined as the 6 bits immediately preceding the TID in the NAL header, specifies the identifier of the layer to which the NAL unit belongs.

[0223] The value of LayerID / R6 must be in the range of 0 to 62. The value 63 may be specified by ISO / IEC in the future.

[0224] For purposes other than determining the amount of data in a decoded unit of the bitstream, the mesh decoding device 200 ignores all data following the value 63 in a NAL unit, and a mesh decoding device 200 conforming to the specified profile will ignore (i.e., remove from the bitstream and discard) all NAL units whose LayerID-R6 value is not 0.

[0225] The value 63 of LayerID / R6 can be used to indicate an enhancement layer identifier in future enhancements.

[0226] When sub-meshes of the basic mesh defined in Non-Patent Document 4 exist, all sub-meshes are assigned the same TID or the TID is applied to all sub-meshes.

[0227] The bitstreams of the atlas, base mesh, displacement, and texture may each have their own TID. For example, for the atlas, the TID is fixed to zero according to Annex A of Non-Patent Document 5. Therefore, the base mesh, displacement, and texture each have their own TID.

[0228] That is, depending on the content, at least one of the basic mesh, displacement, and texture may have Temporal_ID fixed to zero. In that case, the LD setting can also be used. Table 7 shows an example.

[0229]

[0230] Even if they are set independently in each bitstream, the displacement and texture are determined using the video coding method HEVC or VVC, and the basic mesh is determined using the above-described embodiment.

[0231] When set independently in each bitstream, the tIDTarget specified in each bitstream is referenced, and NAL units with TIDs higher than tIDTarget are discarded without being decoded.

[0232] Here, tIDTarget may be specified by a predetermined value, or may be specified based on the network status or the terminal capability of the mesh decoding device 200 .

[0233] For example, a lower tIDTarget is specified in the wireless case than in the wired case. Also, a lower tIDTarget is specified when the network condition is poor. Also, a lower tIDTarget is specified when decoding is performed by a mesh decoding device 200 with low specifications.

[0234] However, the requirement for bitstream conformance is that there must be at least one NAL unit in the bitstream whose TID is not higher than tIDTarget.

[0235] On the other hand, if each bitstream is set independently, if any of the basic mesh, displacement, and texture is discarded in a specific frame, the others will also be discarded.

[0236] Alternatively, if each bitstream is set independently, if the base mesh is discarded in a specific frame, the displacement and texture are also discarded, and the reconstruction process is not performed. However, if the displacement is discarded, all displacement values ​​are set to zero, and the reconstruction process is performed. Also, if the texture is discarded, all texture values ​​are set to zero, and the reconstruction process is performed.

[0237] The number of sub-meshes may differ for each frame (intraframe, interframe, and skip frame).

[0238] In this case, the intra decoder 202B, the inter decoder 202E, and the skip decoder 202F assign a unique submesh ID to each submesh in each frame.

[0239] Furthermore, the intra decoder 202B and the inter decoder 202E may assign different Submesh IDs to corresponding submeshes between frames.

[0240] However, the inter-decoding unit 202E can only refer to submeshes that have the same SubmeshID in the reference frame.

[0241] Alternatively, the inter-decoding unit 202E can refer to only sub-meshes that have the same number of vertices in the reference frame.

[0242] Alternatively, the intra decoder 202B and the inter decoder 202E can refer to a submesh specified in a reference frame.

[0243] In this case, if there are multiple submeshes in the reference frame, the inter-decoding unit 202E may decode a control signal specifying the Submesh ID of a referenceable submesh from the bit stream of the current submesh.

[0244] On the other hand, when there is only one submesh in the reference frame, the inter decoding unit 202E may treat this submesh as a referenceable submesh.

[0245] However, if the above-mentioned control signal does not exist, the inter decoding unit 202E or the skip decoding unit 202F sets the submesh ID of the referable submesh to the same submesh ID as the submesh in the current frame.

[0246] The inter decoding unit 202E may also decode, from the bitstream, a control signal indicating whether or not the above-mentioned control signal is present.

[0247] The inter-decoding unit 202E may decode a control signal for selecting the method for determining the above-mentioned referenceable sub-meshes.

[0248] The subdivision unit 203 and the displacement amount decoding unit 206 may be configured in accordance with Non-Patent Document 4.

[0249] According to the present invention, the amount of calculation can be reduced by reusing the reference frame itself without searching for adjacent vertices that have already been decoded.

[0250] Furthermore, according to this embodiment, in inter-prediction coding, the base mesh of the current frame can be decoded even if the number of vertices of the base mesh of the current frame differs from the number of vertices of the reference frame or reference sub-mesh.

[0251] Furthermore, according to this embodiment, in inter-prediction coding, it is possible to avoid a situation in which the number of vertices of the base mesh of the current frame differs from the number of vertices of the reference frame or reference sub-mesh.

[0252] Furthermore, according to this embodiment, in inter-prediction coding, it is possible to identify which sub-mesh to refer to by introducing a control signal indicating which sub-mesh to refer to in the reference frame of the current frame.

[0253] Furthermore, according to this embodiment, in inter-prediction coding, even if there is no information in the reference frame of the current frame as to which sub-mesh to refer to, it is possible to identify which sub-mesh to refer to.

[0254] Furthermore, according to this embodiment, it is possible to ensure that the base mesh has at least one face.

[0255] Furthermore, according to this embodiment, the Temporal_scalability function can be realized.

[0256] Furthermore, according to this embodiment, the encoding efficiency of the mesh can be improved.

[0257] The mesh encoding device 100 and mesh decoding device 200 described above may be realized as a program that causes a computer to execute each function (each step).

[0258] According to this embodiment, for example, it is possible to improve the overall service quality in video communication, which makes it possible to contribute to Goal 9 of the Sustainable Development Goals (SDGs) led by the United Nations, which is to "Develop resilient infrastructure, promote sustainable industrialization and foster innovation."

[0259] 1...Mesh processing system 100...Mesh encoding device 200...Mesh decoding device 201...Demultiplexing unit 202...Basic mesh decoding unit 202A...Separation unit 202B...Intra decoding unit 202B1...Arbitrary intra decoding unit 202B2...Alignment unit 202C...Mesh buffer unit 202D...Connection information decoding unit 202E...Inter decoding unit 202E1...Motion vector residual decoding unit 202E2...Motion vector buffer unit 202E3...Motion vector prediction unit 202E4...Motion vector calculation unit 202E5...Adder 203...Subdivision unit 204...Mesh decoding unit 205...Patch integration unit 206...Displacement amount decoding unit 207...Video decoding unit 208...Atlas data decoding unit

Claims

1. A mesh decoding device comprising: an inter-decoding unit that decodes a base mesh by adding a motion vector decoded from an inter-frame bitstream to a base mesh of a reference frame; and a subdivision unit that subdivides the faces of the base mesh using the base mesh, wherein the inter-decoding unit decodes the number of motion vectors of a current frame or current sub-mesh from the bitstream, and when the decoded number of motion vectors differs from the number of vertices of the reference frame or reference sub-mesh, adjusts the number of vertices and some of the vertex coordinates of the current frame or current sub-mesh.

2. The mesh decoding device described in claim 1, characterized in that when the number of motion vectors is less than or equal to the number of vertices of the reference submesh, the inter-decoding unit calculates vertex coordinates up to the number of motion vectors using the motion vectors, and leaves the other vertex coordinates in the reference submesh.

3. The mesh decoding device described in claim 1, characterized in that when the number of motion vectors is greater than the number of vertices of the reference submesh, the inter-decoding unit sets the number of vertices of the current frame or the current submesh as the number of motion vectors, calculates vertex coordinates up to the number of vertices of the reference submesh using the motion vectors, and leaves the other vertex coordinates as the motion vectors.

4. The mesh decoding device described in claim 1, characterized in that when the number of motion vectors is greater than the number of vertices of the reference submesh, the inter-decoding unit sets the number of vertices of the current frame or the current submesh as the number of vertices of the reference frame, and calculates vertex coordinates using the motion vectors up to the number of vertices of the reference submesh.

5. A mesh decoding method comprising: step A of decoding a base mesh by adding a motion vector decoded from an interframe bitstream and a base mesh of a reference frame; and step B of subdividing the faces of the base mesh using the base mesh, wherein in step A, the number of motion vectors of the current frame or current submesh is decoded from the bitstream, and when the decoded number of motion vectors differs from the number of vertices of the reference frame or reference submesh, the number of vertices and some of the vertex coordinates of the current frame or current submesh are adjusted.

6. A program that causes a computer to function as a mesh decoding device, the mesh decoding device comprising: an inter-decoding unit that decodes a base mesh by adding a motion vector decoded from an inter-frame bitstream to a base mesh of a reference frame; and a subdivision unit that subdivides the faces of the base mesh using the base mesh, wherein the inter-decoding unit decodes the number of motion vectors of a current frame or current sub-mesh from the bitstream, and when the decoded number of motion vectors differs from the number of vertices of the reference frame or reference sub-mesh, adjusts the number of vertices and some of the vertex coordinates of the current frame or current sub-mesh.

Images