Mesh decoding device, mesh decoding method, and program

Abstract

To improve the encoding efficiency of meshes.SOLUTION: In a mesh decoding device 200 according to the present invention, in a mode 1, a motion vector calculation unit 202E4 adds a motion vector residual and a motion vector prediction value and outputs a motion vector of a vertex of a decoding target, whereas in a mode 0, the motion vector calculation unit 202E4 outputs the motion vector residual as the motion vector of the vertex of the decoding target. The motion vector calculation unit 202E4 sets the mode to be the same for the motion vectors of N (N≥1) sequential vertices in the order of decoding. The N vertices constitute one group. The motion vector calculation unit 202E4 decodes, from a bit stream, a control signal with which a group size indicating the size of the group can be calculated.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to a mesh decoding device, a mesh decoding method, and a program. [Background technology]

[0002] Non-Patent Document 1 discloses a technique for encoding a mesh using Non-Patent Document 2 or 3. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Khaled Mammou, Jungsun Kim, Alexis M Tourapis, Dimitri Podborski, and Krasimir Kolarov, “[V-CG] Apple's Dynamic Mesh Coding CfP Response,” April 2022, ISO / IEC JTC 1 / SC 29 / WG 7. [Non-Patent Document 2] Google Draco, accessed May 26, 2022 [Online], https: / / google.github.io / draco [Non-Patent Document 3] Jean-Eudes Marvie, Olivier Mocquard, “[V-DMC][EE4.4-related] An efficient EdgeBreaker implementation,” April 2023, ISO / IEC JTC 1 / SC 29 / WG 7 m63344. [Non-Patent Document 4] “WD 3.0 of V-DMC,” April 2023, ISO / IEC JTC 1 / SC 29 / WG 7 N00611. Summary of the Invention [Problem to be solved by the invention]

[0004] However, the conventional technology has a problem that the coding efficiency of the motion vector is low. Therefore, the present invention has been made in consideration of the above-mentioned problem, and an object of the present invention is to provide a mesh decoding device, a mesh decoding method, and a program that can improve the coding efficiency of the mesh. [Means for solving the problem]

[0005] A first feature of the present invention is a mesh decoding device comprising: a motion vector residual decoding unit that generates a motion vector residual from an interframe bit stream; a motion vector prediction unit that acquires decoded motion vectors from a vertex to be decoded and surrounding vertices connected to the vertex to be decoded, and outputs a motion vector prediction value of the vertex to be decoded using all or a part of the decoded motion vector; and a motion vector calculation unit that outputs the motion vector of the vertex to be decoded, wherein in mode 1, the motion vector calculation unit adds the motion vector residual and the motion vector prediction value to output the motion vector of the vertex to be decoded, and in mode 0, the motion vector calculation unit outputs the motion vector residual as the motion vector of the vertex to be decoded, the motion vector calculation unit sets the modes of motion vectors of N (N≧1) consecutive vertices in decoding order to be the same, the N vertices constitute one group, and the motion vector calculation unit decodes a control signal from the bit stream that can calculate a group size indicating the size of the group.

[0006] A second feature of the present invention is a mesh decoding method comprising: a step A of generating a motion vector residual from an interframe bit stream; a step B of acquiring decoded motion vectors from a vertex to be decoded and surrounding vertices connected to the vertex to be decoded, and outputting a motion vector prediction value of the vertex to be decoded using all or a part of the decoded motion vector; and a step C of outputting the motion vector of the vertex to be decoded, wherein in mode 1, in step C, the motion vector residual and the motion vector prediction value are added to output the motion vector of the vertex to be decoded, and in mode 0, the motion vector residual is output as the motion vector of the vertex to be decoded, and the N vertices form one group, and a control signal capable of calculating a group size indicating the size of the group is decoded from the bit stream.

[0007] A third feature of the present invention is a program for causing a computer to function as a mesh decoding device, the mesh decoding device comprising: a motion vector residual decoding unit that generates a motion vector residual from an interframe bit stream; a motion vector prediction unit that acquires decoded motion vectors from a vertex to be decoded and surrounding vertices connected to the vertex to be decoded, and outputs a motion vector prediction value of the vertex to be decoded using all or a part of the decoded motion vector; and a motion vector calculation unit that outputs the motion vector of the vertex to be decoded, wherein in mode 1, the motion vector calculation unit adds the motion vector residual and the motion vector prediction value to output the motion vector of the vertex to be decoded, and in mode 0, the motion vector calculation unit outputs the motion vector residual as the motion vector of the vertex to be decoded, the motion vector calculation unit sets the modes of motion vectors of N (N≧1) consecutive vertices in decoding order to be the same, the N vertices constitute one group, and the motion vector calculation unit decodes a control signal from the bit stream that can calculate a group size indicating the size of the group. Effect of the Invention

[0008] According to the present invention, it is possible to provide a mesh decoding device, a mesh decoding method, and a program capable of improving the encoding efficiency of a mesh. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a mesh processing system 1 according to an embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of functional blocks of a mesh decoding device 200 according to an embodiment. [Figure 3A] FIG. 3A is a diagram showing an example of a base mesh and a subdivision mesh. [Figure 3B] FIG. 3B is a diagram showing an example of a base mesh and a subdivision mesh. [Figure 4] FIG. 4 is a diagram showing an example of functional blocks of the basic mesh decoding unit 202 of the mesh decoding device 200 according to an embodiment. [Diagram 5] FIG. 5 is a diagram showing an example of functional blocks of the intra-decoding unit 202B of the basic mesh decoding unit 202 of the mesh decoding device 200 according to an embodiment. [Figure 6] FIG. 6 is a diagram showing an example of the correspondence between the vertices of the basic mesh of a P frame and the vertices of the basic mesh of an I frame. [Figure 7] FIG. 7 is a diagram showing an example of functional blocks of an inter decoding unit 202E of the basic mesh decoding unit 202 of the mesh decoding device 200 according to an embodiment. [Figure 8] FIG. 8 is a diagram illustrating an example of a method for calculating the MVP of a vertex to be decoded by the motion vector prediction unit 202E3 of the inter decoding unit 202E of the basic mesh decoding unit 202 of the mesh decoding device 200 according to one embodiment. [Figure 9] FIG. 9 shows a flowchart showing an example of the operation of the motion vector prediction unit 202E3 of the inter decoding unit 202E of the basic mesh decoding unit 202 of the mesh decoding device 200 according to one embodiment. [Figure 10A]FIG. 10A is a diagram showing an example of a decoding order for meshes. [Figure 10B] FIG. 10B is an example of a list of vertices around the vertex to be decoded. [Figure 11] FIG. 11 is a diagram showing an example of statistical data indicating the relationship between the number of decoded motion vectors and the number of vertices around the vertex to be decoded. [Figure 12] FIG. 12 is a diagram for explaining an example of the worst case. [Figure 13] FIG. 13 is a diagram illustrating a second modification of the inter decoding unit 202E of the basic mesh decoding unit 202 of the mesh decoding device 200 according to an embodiment. [Figure 14] FIG. 14 is a diagram illustrating a second modification of the inter decoding unit 202E of the basic mesh decoding unit 202 of the mesh decoding device 200 according to an embodiment. [Figure 15] FIG. 15 is a diagram showing a modification of the functional blocks of Modification 1 of the basic mesh decoding unit 202 of the mesh decoding device 200 according to an embodiment. [Figure 16] FIG. 16 is a diagram illustrating a first modification of the basic mesh decoding unit 202 of the mesh decoding device 200 according to an embodiment. [Figure 17] FIG. 17 is a diagram illustrating the mesh buffer unit 202C of the basic mesh decoding unit 202 of the mesh decoding device 200 according to an embodiment. [Figure 18] FIG. 18 is a diagram illustrating the mesh buffer unit 202C of the basic mesh decoding unit 202 of the mesh decoding device 200 according to an embodiment. [Figure 19] FIG. 19 is a diagram illustrating a modification of the basic mesh decoding unit 202 of the mesh decoding device 200 according to the second modification. [Figure 20] FIG. 20 is a diagram illustrating a modification of the basic mesh decoding unit 202 of the mesh decoding device 200 according to the second modification. [Figure 21]FIG. 21 is a diagram showing an example of functional blocks of the subdivision unit 203 of the mesh decoding device 200 according to an embodiment. [Figure 22] FIG. 22 is a diagram showing an example of functional blocks of the basic mesh subdivision unit 203A of the subdivision unit 203 in the mesh decoding device 200 according to an embodiment. [Diagram 23] FIG. 23 is a diagram illustrating an example of a method for dividing a base surface by the basic surface dividing unit 203A5 of the basic mesh subdivision unit 203A of the subdivision unit 203 in the mesh decoding device 200 according to an embodiment. [Figure 24] FIG. 24 is a flowchart showing an example of the operation of the basic mesh subdivision unit 203A of the subdivision unit 203 in the mesh decoding device 200 according to an embodiment. [Diagram 25] FIG. 25 is a diagram showing an example of functional blocks of the subdivision mesh adjustment unit 203B of the subdivision unit 203 of the mesh decoding device 200 according to an embodiment. [Figure 26] FIG. 26 is a diagram showing an example of a case in which an edge division point on a basic surface ABC is moved by the edge division point moving unit 701 of the subdivision mesh adjustment unit 203B of the subdivision unit 203 of the mesh decoding device 200 according to an embodiment. [Figure 27] FIG. 27 is a diagram showing an example of a case in which subdivision surface X in a base surface is re-subdivided by the subdivision surface division unit 702 of the subdivision mesh adjustment unit 203B of the subdivision unit 203 of the mesh decoding device 200 according to one embodiment. [Figure 28] FIG. 28 is a diagram showing an example of a case in which all subdivision surfaces are re-subdivided by the subdivision surface division unit 702 of the subdivision mesh adjustment unit 203B of the subdivision unit 203 of the mesh decoding device 200 according to one embodiment. [Figure 29] FIG. 29 is a diagram showing an example of functional blocks of the displacement amount decoding unit 206 of the mesh decoding device 200 according to one embodiment (when inter prediction is performed in the spatial domain). [Diagram 30] FIG. 30 is a diagram showing an example of the configuration of a displacement amount bit stream. [Diagram 31]FIG. 31 is a diagram showing an example of a syntax configuration of a DPS. [Diagram 32] FIG. 32 is a diagram illustrating an example of a syntax configuration of the DPH. [Diagram 33] FIG. 33 is a diagram illustrating an example of a correspondence relationship in frequency between a reference frame and a current frame to be decoded when inter prediction is performed in the spatial domain. [Diagram 34] FIG. 34 is a diagram showing an example of functional blocks of the displacement amount decoding unit 206 of the mesh decoding device 200 according to one embodiment (when inter prediction is performed in the frequency domain). [Diagram 35] FIG. 35 is a diagram illustrating an example of a correspondence relationship in frequency between a reference frame and a current frame to be decoded when inter prediction is performed in the frequency domain. [Diagram 36] FIG. 36 is a flowchart showing an example of the operation of the displacement amount decoding unit 206 of the mesh decoding device 200 according to an embodiment. [Figure 37] FIG. 37 is a diagram illustrating an example of functional blocks of the displacement amount decoding unit 206 according to the first modification. [Figure 38] FIG. 38 is a diagram illustrating an example of functional blocks of the displacement amount decoding unit 206 according to the second modification. [Figure 39] FIG. 39 is a diagram showing an example of a NAL header. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments of the present invention will be described with reference to the drawings. Note that the components in the following embodiments can be appropriately replaced with existing components, and various variations including combinations with other existing components are possible. Therefore, the description of the following embodiments does not limit the contents of the invention described in the claims.

[0011] First Embodiment The mesh processing system according to this embodiment will be described below with reference to FIGS.

[0012] 1 is a diagram showing an example of the configuration of a mesh processing system 1 according to this embodiment. As shown in FIG.

[0013] FIG. 2 is a diagram showing an example of functional blocks of the mesh decoding device 200 according to this embodiment.

[0014] As shown in FIG. 2, the mesh decoding device 200 includes a demultiplexing unit 201, a basic mesh decoding unit 202, a subdivision unit 203, a mesh decoding unit 204, a patch integration unit 205, a displacement amount decoding unit 206, and a video decoding unit 207.

[0015] Here, the basic mesh decoding unit 202, the subdivision unit 203, the mesh decoding unit 204 and the displacement amount decoding unit 206 are configured to perform processing on a patch basis into which the mesh is divided, and the results of these processes may then be configured to be integrated by the patch integration unit 205.

[0016] In the example of FIG. 3A, the mesh is divided into patch 1 consisting of basic faces 1 and 2, and patch 2 consisting of basic faces 3 and 4.

[0017] The demultiplexing unit 201 is configured to separate the multiplexed bit stream into a base mesh bit stream, a displacement amount bit stream, and a texture bit stream.

[0018] <Basic mesh decoding unit 202> The base mesh decoding unit 202 is configured to decode the base mesh bitstream and generate and output a base mesh.

[0019] Here, the base mesh is composed of a plurality of vertices in a three-dimensional space and edges connecting these vertices.

[0020] As shown in FIG. 3A, the basic mesh is formed by combining basic faces each represented by three vertices.

[0021] The base mesh decoder 202 may be configured to decode the base mesh bitstream, for example using the techniques described in Draco in Non-Patent Document 2 or in Non-Patent Document 3.

[0022] Furthermore, the basic mesh decoding unit 202 may be configured to generate "subdivision_method_id" (described later) as control information for controlling the type of subdivision method.

[0023] As shown in FIG. 4, the basic mesh decoding unit 202 includes a separation unit 202A, an intra decoding unit 202B, a mesh buffer unit 202C, a connection information decoding unit 202D, and an inter decoding unit 202E.

[0024] The separator 202A is configured to classify the base mesh bitstream into a bitstream of I frames and a bitstream of P frames. (Intra decoding unit 202B) The intra decoder 202B is configured to decode the coordinates and connection information of the vertices of the I-frame from the bit stream of the I-frame, for example, using the technique Draco shown in Non-Patent Document 2 or the technique described in Non-Patent Document 3.

[0025] FIG. 5 is a diagram showing an example of functional blocks of the intra decoding unit 202B.

[0026] As shown in FIG. 5, the intra decoding unit 202B includes an arbitrary intra decoding unit 202B1 and an alignment unit 202B2.

[0027] The arbitrary intra decoder 202B1 is configured to decode the coordinates and connectivity information of the unordered vertices of the I-frame from the bitstream of the I-frame using any method including Draco shown in Non-Patent Document 2 or the technique described in Non-Patent Document 3.

[0028] The sorting unit 202B2 is configured to output the vertices by sorting the unordered vertices into a predetermined order.

[0029] As the predetermined order, for example, the Morton code order or the raster scan order may be used.

[0030] Furthermore, the alignment unit 202B2 may combine overlapping vertices, which are multiple vertices with the same coordinates in the decoded basic mesh, into a single vertex, and then rearrange the vertices in a predetermined order.

[0031] The mesh buffer unit 202C is configured to accumulate the coordinates and connection information of the vertices of the I frame decoded by the intra decoder 202B. Here, a specific buffer may be provided to store pairs of vertex indexes A(k) and B(k) of overlapping vertices in a predetermined order.

[0032] The connection information decoding unit 202D is configured to convert the connection information of the I frame or the reference frame extracted from the mesh buffer unit 202C into connection information of a P frame.

[0033] The inter decoding unit 202E is configured to decode the coordinates of the vertices of the P frame by adding the coordinates of the vertices of the reference frame retrieved from the mesh buffer unit 202C to the motion vectors decoded from the bit stream of the P frame.

[0034] Furthermore, the inter decoding unit 202E can adjust the index of the vertex of the P frame by the pair of vertex indexes A(k) and B(k) existing as overlapping vertices stored in the specific buffer.

[0035] Here, all or part of the indexes are decoded from the bitstream. The decoding method may be arithmetic coding. As a result, it is expected that there is no limit to the maximum value of the index to be decoded using arithmetic coding.

[0036] For example, we may use the arithmetic encoding ue(v), which denotes the leftmost bit first exponential-Golomb encoding of an unsigned integer of order 0.

[0037] Specifically, the parsing process for a syntax element of ue(v) begins at the current position in the bitstream, reading the bits that contain the first non-zero bit, and counting the number of leading bits that are equal to 0. This process is specified as follows: leadingZeroBits=-1 for(b=0;!b;leadingZeroBits++ b = read_bits(l) Next, the variable codeNum is assigned as follows:

[0038] codeNum=2 leadingZeroBits -1+read_bits(leadingZeroBits) where the value returned by read_bits(leadingZeroBits) is interpreted as the binary representation of an unsigned integer written most significant bit first, and the value of ue(v) is equal to the value of codeNum.

[0039] Table 1 shows the structure of an Exp-Golomb code, separating the bit string into "prefix" bits and "suffix" bits.

[0040] [Table 1] Here, the "prefix" bits are those bits that are parsed as specified in the calculation of leadingZeroBits, and are represented as 0 or 1 in the bit string column of Table 1.

[0041] The "suffix" bits are the bits parsed in the calculation of codeNum, and are denoted by x in Table 1. i where i ranges from 0 to leadingZeroBits-1. Each x i is equal to either 0 or 1.

[0042] Table 2 shows how to explicitly assign bit strings to values ​​of codeNum, where the value of ue(v) is equal to the value of codeNum.

[0043] [Table 2] In this embodiment, as shown in Fig. 6, there is a correspondence between the vertices of the basic mesh of the P frame and the vertices of the basic mesh of the reference frame (I frame or P frame). Here, the motion vector decoded by the inter decoding unit 202E is a difference vector between the coordinates of the vertices of the basic mesh of the P frame and the coordinates of the vertices of the basic mesh of the I frame.

[0044] (Inter-decoding unit 202E) FIG. 7 is a diagram illustrating an example of functional blocks of the inter decoding unit 202E.

[0045] As shown in FIG. 7, the inter decoding unit 202E includes a motion vector residual decoding unit 202E1, a motion vector buffer unit 202E2, a motion vector prediction unit 202E3, a motion vector calculation unit 202E4, and an adder 202E5.

[0046] The motion vector residual decoding unit 202E1 is configured to generate a motion vector residual (MVR) from a bit stream of a P frame.

[0047] Here, MVR is a motion vector residual indicating the difference between MV (Motion Vector) and MVP (Motion Vector Prediction). MV is a difference vector (motion vector) between the coordinates of the vertex of the corresponding I frame and the coordinates of the vertex of the corresponding P frame. MVP is a predicted value of the MV of the target vertex using MV (predicted value of the motion vector).

[0048] The motion vector buffer unit 202E2 is configured to sequentially store the MVs output by the motion vector calculation unit 202E4.

[0049] The motion vector prediction unit 202E3 is configured to obtain a decoded MV from the motion vector buffer unit 202E2 for a vertex connected to the vertex to be decoded, and output an MVP of the vertex to be decoded using all or part of the obtained decoded MV, as shown in Figure 8.

[0050] The motion vector calculation unit 202E4 is configured to add the MVR generated by the motion vector residual decoding unit 202E1 and the MVP output from the motion vector prediction unit 202E3, and output the MV of the vertex to be decoded.

[0051] The adder 202E5 is configured to add the coordinates of a vertex corresponding to the vertex to be decoded obtained from the decoded basic mesh of a corresponding reference frame (I frame or P frame) to the motion vector MV output from the motion vector calculation unit 202E3, and output the coordinates of the vertex to be decoded.

[0052] Hereinafter, each component of inter decoding unit 202E will be described in detail.

[0053] 9 shows a flowchart illustrating an example of the operation of the motion vector prediction unit 202E3. Hereinafter, the operation of the motion vector prediction unit 202E3 will be referred to as an "average prediction method."

[0054] As shown in FIG. 9, in step S1001, the motion vector prediction unit 202E3 sets MVP and N to 0.

[0055] In step S1002, the motion vector prediction unit 202E3 obtains a set of MVs of vertices around the vertex to be decoded from the motion vector buffer unit 202E2, identifies vertices for which subsequent processing has not been completed, and transitions to No, or transitions to Yes if subsequent processing has been completed for all vertices.

[0056] In step S1003, if the MV of the vertex to be processed has not been decoded, the motion vector prediction unit 202E3 transitions to No, and if the MV of the vertex to be processed has been decoded, the motion vector prediction unit 202E3 transitions to Yes.

[0057] In step S1004, the motion vector prediction unit 202E3 adds MV to MVP and adds 1 to N.

[0058] In step S1005, if N is greater than 0, the motion vector prediction unit 202E3 outputs the result of dividing MVP by N, and if N is 0, outputs 0, and the process ends.

[0059] That is, the motion vector prediction unit 202E3 is configured to output the MVP to be decoded by averaging the decoded motion vectors of vertices around the vertex to be decoded.

[0060] The motion vector prediction unit 202E3 may be configured to set the MVP to 0 when the set of decoded motion vectors is an empty set.

[0061] The motion vector calculation unit 202E4 may be configured to calculate the MV of the vertex to be decoded from the MVP output by the motion vector prediction unit 202E3 and the MVR generated by the motion vector residual decoding unit 202E1, using equation (1).

[0062] MV(k)=MVP(k)+MVR(k) … (1) where k is the index of the vertex. MV, MVR and MVP are vectors with x, y and z components.

[0063] According to this configuration, since only the MVR is encoded instead of the MV using the MVP, it is expected to have the effect of improving the encoding efficiency.

[0064] The adder 202E5 is configured to calculate the coordinates of a vertex by adding the MV of the vertex calculated by the motion vector calculation unit 202E4 to the coordinates of the vertex in the reference frame corresponding to the vertex, and to leave the connectivity information (Connectivity) as it is in the reference frame.

[0065] Specifically, the adder 202E5 calculates the coordinate v' of the kth vertex using the formula (2). i (k) may be calculated.

[0066] v' i (k)=v' j (k) + MV(k) … (2) where v' i (k) is the coordinate of the kth vertex to be decoded in the frame to be decoded, and v' j (k) is the coordinate of the k-th decoded vertex of the reference frame, and MV(k) is the k-th MV of the frame to be decoded, where k=1, 2, . . . , K.

[0067] Moreover, the connection information of the frame to be decoded is made the same as the connection information of the reference frame.

[0068] Note that since the motion vector prediction unit 202E3 calculates the MVP using a decoded MV, the order of decoding affects the MVP.

[0069] The order of the decoding is the order of the vertices of the base mesh in the reference frame. In general, if a decoding method is used that uses a certain repetitive pattern to increase the number of base faces one by one from the starting edge, the order of the vertices of the decoded base mesh is determined during the decoding process.

[0070] For example, the motion vector prediction unit 202E3 may use an Edgebreaker to determine the order of decoding vertices in the base mesh of the reference frame.

[0071] According to this configuration, since MVs from a reference frame are encoded instead of vertex coordinates, it is expected that the encoding efficiency can be improved.

[0072] (Modification 1 of the inter-decoding unit 202E) The following describes a first modification of the inter decoding unit 202E.

[0073] The motion vector prediction unit 202E3 of the inter-decoding unit 202E uses an "average prediction method" that averages the decoded motion vectors of vertices surrounding the vertex to be decoded, and calculates an MVP using all or only a portion of the decoded motion vectors of vertices surrounding the vertex to be decoded so as not to exceed a predetermined maximum number of uses.

[0074] Note that the predetermined maximum usage number is decoded from the bitstream as a control signal.

[0075] Furthermore, when the number of decoded motion vectors of vertices around the vertex to be decoded exceeds the maximum number of uses, the motion vector prediction unit 202E3 picks up up to the maximum number of uses according to a certain rule.

[0076] For example, the motion vector prediction unit 202E3 may select the first or last vertex in decoding order as a rule.

[0077] The decoding order for a mesh such as that shown in FIG. 10A is as indicated by the arrows for vertex v D →vC →v A →v B It is.

[0078] FIG. 10B shows the maximum number of decoded adjacent vertices set to 3 for each vertex v A ~v D This is a list of vertices around the vertex to be decoded that is used when calculating the MVP of

[0079] According to this configuration, by determining the maximum number of adjacent vertices, it is expected that the amount of calculation and the amount of memory can be reduced while maintaining or slightly decreasing the coding efficiency.

[0080] However, in order to achieve the above-mentioned effect, it is necessary to set an appropriate maximum number of adjacent vertices in the mesh encoding device 100 and write it into the bitstream as an associated control signal.

[0081] Therefore, since the range that can be set for the above-mentioned maximum number of adjacent vertices determines the amount of memory to be prepared in the mesh decoding device 200, the maximum number of adjacent vertices is encoded / decoded so that it is less than or equal to a predetermined maximum value as a reasonable constraint on the maximum number of adjacent vertices.

[0082] In this way, by defining reasonable constraints on the maximum number of adjacent vertices, it is expected that the design of the mesh decoding device 200 will be facilitated.

[0083] Generally, the average number of adjacent vertices in a closed 2-manifold triangle mesh is about 6, but statistically, the maximum number of adjacent vertices is often 7 to 8. As shown in Fig. 11, the number of decoded motion vectors (vertical axis) dynamically changes according to the number of vertices around the vertex to be decoded (horizontal axis).

[0084] Therefore, it is desirable to narrow the range within which the above-mentioned maximum number of adjacent vertices can be set.

[0085] For example, as shown in FIG. 11, by including "3", which is the number of vertices around the vertex to be decoded that has the largest number of decoded motion vectors statistically, within the range that can be set as the maximum number of adjacent vertices in the above-mentioned control signal, or by setting a certain percentage (e.g., 50% or 120%) of the average statistical number of adjacent vertices or a value not greater than a natural number that can cover up to N bits (e.g., 3 bits), as the upper limit (maximum value) of the range that can be set as the maximum number of adjacent vertices in the above-mentioned control signal, it is possible to achieve the effect of reducing the amount of calculation and memory.

[0086] On the other hand, if the range that can be set as the maximum number of adjacent vertices is set to a large value, for example, the worst case of 256 or 8 bits, there is a possibility that the effect of reducing not only the memory amount but also the calculation amount cannot be achieved.

[0087] Fig. 12 shows an example of the worst case, where when n ≥ 256, the number of decoded adjacent vertices exceeds 256. In Fig. 12, the number of decoded adjacent vertices for vertex n+1 is n.

[0088] If the upper limit of the maximum number of adjacent vertices is set to 256, the mesh decoding device 200 will require not only a huge amount of memory but also a huge amount of calculation, as shown in Fig. 10B. Therefore, the upper limit (maximum value) of the range that can be set as the above-mentioned maximum number of adjacent vertices may be set to 8.

[0089] Furthermore, the range that can be set as the maximum number of adjacent vertices in the above-mentioned control signal may be a clear value, or may be calculated from other control signals or data.

[0090] For example, Level 1 may define a range that can be set as the maximum number of adjacent vertices in a control signal.

[0091] Alternatively, the upper limit of the range that can be set as the maximum number of adjacent vertices in the control signal may be calculated from the number of vertices in the basic mesh using the following formula (3).

[0092] The upper limit of the range that can be set as the maximum number of adjacent vertices in the control signal = log2 (number of vertices in the base mesh) ... Equation (3) According to this configuration, it is possible to determine an appropriate range for the maximum number of adjacent vertices, and it is expected to reliably reduce both the amount of calculation and the amount of memory required, even in the worst case.

[0093] (Modification 2 of the inter-decoding unit 202E) Hereinafter, a second modification of the inter decoding unit 202E will be described with reference to FIG.

[0094] The motion vector calculation unit 202E4 of the inter decoding unit 202E has a mode 1 and a mode 0.

[0095] In mode 1, the motion vector calculation unit 202E4 adds the MVR generated by the motion vector residual decoding unit 202E1 and the MVP output from the motion vector prediction unit 202E3, and outputs the MV of the vertex to be decoded (see A in FIG. 13).

[0096] On the other hand, in mode 0, the motion vector calculation unit 202E4 outputs the MVR generated by the motion vector residual decoding unit 202E1 as the MV of the vertex to be decoded (see B in FIG. 13).

[0097] The operation of the motion vector calculation unit 202E4 in mode 0 corresponds to the operation of setting the MVP output from the motion vector prediction unit 202E3 to zero.

[0098] Furthermore, the motion vector calculation unit 202E4 may set the MV modes of N (N≧1) consecutive vertices in decoding order to the same mode.

[0099] The motion vector calculation unit 202E4 groups the above-mentioned N vertices into one group. The size of the group (group size) N is equal to or greater than 1. The motion vector calculation unit 202E4 decodes a control signal (group size shown in FIG. 13) from the bit stream that allows the calculation of the group size.

[0100] However, if the number of vertices remaining in the last group is smaller than the group size, the motion vector calculation section 202E4 puts all of the remaining vertices into the group.

[0101] In this way, by setting N consecutive vertices to the same mode, the amount of code for the mode can be reduced, and this is expected to have the effect of improving the coding efficiency.

[0102] Here, the more consecutive vertices with the same mode there are, the greater the effect of reducing the amount of code for the mode. Therefore, it is necessary to set an appropriate group size in the mesh encoding device 100 and decode it from the bitstream as a control signal in the mesh decoding device 200.

[0103] Therefore, it is desirable that the settable range of such a control signal is no smaller than the number of consecutive vertices that actually have the same mode.

[0104] For example, if a mode is selected in which all vertices are almost the same, the group size may be set to the total number of vertices.

[0105] Table 1 shows examples where the number of vertices that select mode 0 is 80% or more, and where the number of vertices that select mode 1 is 90% or more.

[0106] Therefore, the settable range of the above control signal must cover from 1 to a predetermined maximum value, which must be equal to or greater than the total number of vertices in the base mesh.

[0107] [Table 3] When the above-mentioned control signal (group size) is a natural number, if it is set to be equal to or greater than the total number of vertices, the absolute value is large, so the amount of code increases.

[0108] Therefore, the above control signal may be logarithmic. Specifically, the control signal may be log2_group_size, and the group size may be calculated by the following formula (4).

[0109] group size=2log2_group_size … Equation (4) Here, if there is only one group in the frame, make it the last group. In other words, if the group size is greater than the number of vertices, put all vertices into a group.

[0110] Furthermore, the settable range of the above-mentioned control signal may be a definite value, or may be calculated from other control signals or data.

[0111] For example, the range that can be set in such a control signal may be defined by Level 1.

[0112] Alternatively, the settable range of such a control signal may be calculated from the number of vertices of the base mesh.

[0113] For example, the range that can be set in such a control signal may be the smallest natural number that is a power of two that covers the number of vertices of the base mesh.

[0114] Furthermore, the settable range in the above-mentioned control signal may be narrowed, and a predetermined flag (Mode flag) of another control signal may be introduced as shown in Fig. 14. In this case, as shown in Fig. 14, if the predetermined flag is TRUE (Mode flag = 1), the motion vector calculation unit 202E4 puts all vertices into one group (i.e., the number of all vertices is the group size), and if it is FALSE, it leaves the group size calculated from the above-mentioned control signal unchanged.

[0115] The above control signal may be set for each sequence or for each frame. When the above control signal is set for each sequence, the group size of all frames is the same.

[0116] According to this configuration, by determining the range in which the group size can be set appropriately, it is possible to deal with any situation, reliably reduce the amount of code in the mode, and improve the coding efficiency.

[0117] (Modification 1 of the basic mesh decoding unit 202) Hereinafter, a first modification of the basic mesh decoding unit 202 will be described with reference to Figs.

[0118] As shown in FIG. 15, the basic mesh decoding unit 202 according to the present modified example 1 includes a separating unit 202A, an intra decoding unit 202B, a mesh buffer unit 202C, an inter decoding unit 202E, and a skip decoding unit 202F.

[0119] The skip decoding unit 202F is configured to decode the base mesh of the frame to be decoded by directly using the decoding base mesh of the specified reference frame.

[0120] In this embodiment, the frame may be either a mesh or a submesh.

[0121] For example, as shown in FIG. 16, "P_SUBMESH" in smh_type may correspond to a P frame, "I_SUBMESH" in smh_type may correspond to an I frame, and "SKIP_SUBMESH" in smh_type may correspond to an S frame.

[0122] (Skip decoding unit 202F) The skip decoding unit 202F is configured to extract a decoded base mesh (a reference decoding base mesh) of a specified reference frame from the mesh buffer unit 202C, and to decode the vertex coordinates and vertex indexes of the base mesh of the frame to be decoded using the vertex coordinates and vertex indexes of the extracted reference decoding base mesh as they are.

[0123] Here, the mesh buffer unit 202C has at least one reference frame and is configured to store at least one decoded base mesh for each reference frame.

[0124] The skip decoding unit 202F may identify the designated reference decoding base mesh using a control signal decoded from the bitstream or a predetermined rule.

[0125] For example, the predetermined rule may be to extract the first reference frame in the reference frame list from the mesh buffer unit 202C, or to extract the reference frame whose frame index is closest to the frame to be decoded.

[0126] In this embodiment, a frame in which the coordinates of the vertices of a base mesh are decoded using the coordinates of the vertices of the decoded base mesh for reference and the indexes of the vertices as they are is called an "S frame."

[0127] According to this configuration, since the skip decoding unit 202F can eliminate the need for motion vectors, a significant reduction in the amount of code and the amount of calculation can be expected.

[0128] (Mesh buffer section 202C) The mesh buffer unit 202C is configured to store one or more reference decoded basic meshes in a predetermined order.

[0129] The basic mesh has metadata such as a frame number and a submesh number, and at least the coordinates of each vertex and an index of the vertex, and is stored in the mesh buffer unit 202C in a predetermined order determined by the reference frame list.

[0130] Here, as shown in FIG. 17, the reference frame list (ref_list0) is a list of information that identifies all of the reference decoding basic meshes stored in the mesh buffer unit 202C.

[0131] The reference frame list may be determined by control signals decoded from the bitstream, as shown in FIG. 17, or may be calculated naturally from the decoding order of the frames.

[0132] Note that the control signal decoded from the bit stream may be indicated by a relative distance from the frame to be decoded, or may be an absolute frame index.

[0133] Additionally, the control signal may use a short-term or a long-term frame of reference.

[0134] For example, when a short-term reference frame is used, the absolute value (abs_delta_mfoc_st) of the difference in display order between the current frame (cur) and the reference frame (ref) and its sign (sign_flag) may be decoded from the bitstream, and the display order of the reference frame may be specified using the following formula: if(sign_flag) { Display Order(ref)=Display Order(cur)+abs_delta_mfoc_st }else{ Display Order(ref)=Display Order(cur)-abs_delta_mfoc_st } In addition, when a method of calculating from the natural frame decoding order is used, for example, when there is no control signal in the reference frame list, the frames may be arranged in sequence starting from the last decoded frame by a certain number of frames. That is, the reference frame list may be {0, -1, -2, ..., -(N-1)}.

[0135] Basically, the reference frame list does not change for each frame except in special circumstances (for example, when a re-ordering instruction is received).

[0136] The mesh buffer unit 202C may be updated as follows.

[0137] When a basic mesh is decoded, in the case of an I frame or a P frame, the mesh buffer unit 202C adjusts the order of the reference frames by deleting one or more existing reference frames in a predetermined order determined by the reference frame list, and inserting one or more basic meshes including the basic mesh of the decoded frame, or by creating and inserting one basic mesh from the multiple basic meshes.

[0138] Such a deletion operation may be performed only when the mesh buffer unit 202C is full. The number of basic meshes that can be stored in the mesh buffer unit 202C is determined in advance. In this embodiment, the mesh buffer unit 202C is defined as being full when the number of basic meshes is reached. In the above-mentioned creation work, a single basic mesh may be generated by taking a weighted average of the coordinates of vertices corresponding to the basic meshes of the decoded frame and the existing basic meshes stored in the mesh buffer unit 202C.

[0139] The weights used in such a weighted average may be predetermined, calculated using the frame index, or decoded from the control signal.

[0140] However, the mesh buffer unit 202C may or may not perform such an update when it is an S frame.

[0141] In addition, when the mesh buffer unit 202C receives a control signal indicating a re-ordering instruction from a control signal decoded from the bit stream, it updates the reference frame list as shown in Figure 18, and adjusts the order of the reference frames according to the specified order determined by the updated reference frame list (ref_list0).

[0142] (Inter-decoding unit 202E) The inter decoding unit 202E is configured to decode the coordinates of the vertices of the P frame by adding the coordinates of the vertices of the reference frame retrieved from the mesh buffer unit 202C to the motion vectors decoded from the bit stream of the P frame.

[0143] Furthermore, the inter-decoding unit 202E can adjust the indexes of the vertices of the P frame by a pair of vertex indexes A(k) and B(k) existing as overlapping vertices stored in the specific buffer. All or part of the indexes are decoded from the bitstream. The decoding method may be arithmetic coding. With this configuration, it is expected that there is no limit to the maximum value of the index to be decoded using arithmetic coding. For example, arithmetic coding called ue(v) may be used.

[0144] (Modification 2 of the basic mesh decoding unit 202) Hereinafter, the second modification of the basic mesh decoding unit 202 will be described with reference to FIG.

[0145] Although the skip decoding unit 202F will be described below, it may also be applied to the inter decoding unit 202E.

[0146] As shown in FIG. 19, in the skip decoding unit 202F, since it is possible to refer to subsequent frames, the decoding order and the display order are different.

[0147] Here, the display order is the same as the input order during encoding, and is the same as the output order during decoding.

[0148] On the other hand, the decoding order is the same as the output order during encoding, and is the same as the input order during decoding.

[0149] The reference frame may be calculated by taking a weighted average of the subsequent frame and one or more other frames.

[0150] However, when referring to a plurality of frames including the subsequent frame, MR_SUBMESH (MR frame or B frame) is defined as a new frame type (smh_type) in FIG. 14, and MR_SUBMESH is decoded from the bitstream.

[0151] Furthermore, the other frame may be a decoded frame immediately before the target frame, as shown in FIG.

[0152] Such weights may be calculated using the frame interval between the target frame and the subsequent frame and the frame interval between the target frame and another frame, or may be determined in advance.

[0153] The basic mesh decoding unit 202 decodes the control signal (smh_mesh_frm_order_cnt_lsb) from the bit stream and decodes the order of the output.

[0154] In addition, when sub-meshes as defined in the above-mentioned non-patent document 4 exist, all sub-meshes are set to the same control signal (smh_mesh_frm_order_cnt_lsb) or the control signal (smh_mesh_frm_order_cnt_lsb) is applied to all sub-meshes.

[0155] The value indicated by this control signal (smh_mesh_frm_order_cnt_lsb) may be a difference from the display order of the frame to be decoded, or may be an order within a predetermined frame group MaxMeshFrmOrderCntLsb.

[0156] In addition, when the decoding order and the display order are different, if the decoded basic meshes are arranged in the decoding order, the basic mesh decoding unit 202 may rearrange the decoded basic meshes into the display order.

[0157] In addition, in an S frame that can reference a subsequent frame, two mesh buffers 202C may be provided, or when only one mesh buffer unit 202 is provided, there is at least one reference frame, including a reference frame whose display order is later than the frame to be decoded.

[0158] The skip decoding unit 202F specifies a reference frame by receiving a control signal decoded from the bitstream, a predetermined rule, or a Re-ordering instruction.

[0159] Specifically, the skip decoding unit 202F specifies a reference frame in the reference frame list by the control signal.

[0160] Alternatively, the skip decoding unit 202F designates the first reference frame in the reference frame list.

[0161] Alternatively, upon receiving the Re-ordering instruction, the skip decoding unit 202F updates the reference frame order in the reference frame list and the mesh buffer unit 202C, and specifies the first reference frame in the reference frame list.

[0162] In this embodiment, even if an S frame is not decoded, it does not affect the decoding of other frames. Therefore, if some or all of the S frames are not decoded, temporal scalability can be realized.

[0163] Furthermore, the base mesh decoding unit 202 may decode a base mesh of an S frame by integrating a plurality of reference frames in response to a control signal.

[0164] For example, the base mesh decoding unit 202 may be configured to average the coordinates of corresponding vertices in the base meshes of two previous and next reference frames, and use the average coordinates and the vertex index as is to decode the coordinates of the vertices and the index of the vertex of the base mesh of the frame to be decoded.

[0165] According to this configuration, the skip decoding unit 202F or the inter decoding unit 202E can generate a high-quality basic mesh without using a motion vector, so that the effect of improving the quality of the decoded mesh can be expected. Furthermore, the effect of realizing temporal scalability can be expected.

[0166] However, in order to achieve temporal scalability, a control signal Temporal_ID is defined for each frame, which indicates whether or not the corresponding base mesh, displacement, and texture are to be decoded, and each is decoded from the bitstream.

[0167] In addition, if it is the same frame, the control signal Tempora_ID of the base mesh, the displacement amount, and the texture is made to match. If the Tempora_ID of the base mesh, the displacement amount, and the texture to be decoded match, it is expected that the frame cannot be decoded and unnecessary data can be avoided.

[0168] It is desirable that the interval between adjacent frames having the same Tempora_ID be constant.

[0169] Adjacent frames with the same Tempora_ID have the closest POC.

[0170] By making the frame intervals constant as described above, it is expected that a constant frame rate can be maintained when displaying decoded frames.

[0171] Furthermore, the decoding order of the base meshes, displacements, and textures having the same display order is made consistent.

[0172] By matching the decoding order in this way, it is expected that the mesh can be reproduced without waiting for the decoding of each of the basic mesh, the displacement amount, and the texture.

[0173] If the decoding order of the base mesh, the displacement amount, and the texture is different, it is necessary to wait for the component that was decoded last, which causes problems such as an increase in buffer usage and a decoding delay.

[0174] Furthermore, a frame having a Temporal_ID higher than the control signal Temporal_ID of the frame to be decoded is not used as a reference frame for the frame to be decoded.

[0175] This is expected to have the effect of eliminating the possibility of the reference frame being discarded.

[0176] An example of achieving temporal scalability using the above-mentioned Temporal_ID will be described below.

[0177] The bitstreams of the base mesh, displacements and textures are encapsulated by Network Abstraction Layer (NAL) units, which may have a NAL header as shown in Figure 39.

[0178] The TID, defined as the last 3 bits in the NAL header, is the Temporal_ID plus 1. The range of TID is 1 to 7, with zero being prohibited.

[0179] When sub-meshes defined in Non-Patent Document 4 exist, all sub-meshes are to have the same TID or the TID is to be applied to all sub-meshes.

[0180] Since the video coding method HEVC or VVC is used for the displacement amount and texture, only the basic mesh will be described below.

[0181] If the NALType is in the range from NAL_BLA_W_LP to NAL_RSV_BMCL_29 defined in Non-Patent Document 4, i.e., belongs to an IRAP coded basic mesh frame, then Temporal_ID must be 0.

[0182] If NALType is equal to NAL_TSA_R or NAL_TSA_N, Temporal_ID must not be equal to 0.

[0183] If NALType is equal to 0, and NALType is equal to NAL_STSA_R or NAL_STSA_N, Temporal_ID shall not be equal to 0.

[0184] The value of Temporal_ID MUST be the same for all BMCL NAL units within an access unit.

[0185] The value of Temporal_ID of a coded basic mesh frame or access unit is the value of Temporal_ID of the BMCL NAL unit of the coded basic mesh frame or access unit.

[0186] The value of Temporal_ID of a sublayer representation is the maximum value of the Temporal_IDs of all BMCL NAL units in the sublayer representation.

[0187] The values ​​of Temporal_ID for non-BMCL NAL units are restricted as follows: - if NALType is equal to NAL_BMSPS, then Temporal_ID MUST be 0 and the Temporal_ID of the access unit that contains the NAL unit MUST be 0. Otherwise, if NALType is equal to NAL_EOS or NAL_EOB, Temporal_ID must be 0. Otherwise, if NALType is equal to NAL_AUD or NALLFDD, Temporal_ID shall be equal to the Temporal_ID of the access unit that contains the NALL unit. - Otherwise, the Temporal_ID MUST be greater than or equal to the Temporal_ID of the access unit that contains the NAL unit.

[0188] Note that if the NAL unit is not BMCL, the value of Temporal_ID is equal to the minimum value of the Temporal_ID values ​​of all access units to which the non-BMCL NAL unit applies.

[0189] When NALType is equal to NAL_BMFPS, Temporal_ID can be greater than or equal to the Temporal_ID of the included access unit because all basic mesh frame parameter sets (BMFPS) are included at the beginning of the bitstream where the Temporal_ID of the first encoded basic mesh frame is 0.

[0190] <Subdivision section 203> The subdivision unit 203 is configured to generate and output added subdivision vertices and their connection information from the basic mesh decoded by the basic mesh decoding unit 202 using a subdivision method indicated by the control information.

[0191] Here, the basic mesh, the added subdivision vertices, and their connection information are collectively referred to as a "subdivision mesh."

[0192] The subdivision unit 202 is configured to identify the type of subdivision method from subdivision_method_id, which is control information generated by decoding the basic mesh bit stream.

[0193] The subdivision unit 202 will now be described with reference to FIGS. 3A and 3B.

[0194] 3A and 3B are diagrams for explaining an example of an operation for generating subdivision vertices from a basic mesh.

[0195] FIG. 3A is a diagram showing an example of a base mesh composed of five vertices.

[0196] Here, the subdivision may be performed, for example, by using a mid-edge division method in which the midpoints of the sides of each basic face are connected to each other, resulting in a basic face being divided into four faces.

[0197] Figure 3B shows an example of a subdivision mesh obtained by dividing a base mesh consisting of five vertices, in which eight subdivision vertices (white circles) are generated in addition to the original five vertices (black circles).

[0198] By decoding the displacement amount for each subdivision vertex thus generated by the displacement amount decoder 206, it is possible to expect improvement in coding performance.

[0199] Also, a different subdivision method may be applied to each patch. This allows the displacement decoded by the displacement decoder 206 to be adaptively changed for each patch, which is expected to improve coding performance. Information on the divided patches is received as patch_id, which is control information.

[0200] The subdivision section 203 will be described below with reference to Fig. 21. Fig. 21 is a diagram showing an example of functional blocks of the subdivision section 203.

[0201] As shown in FIG. 21, the subdivision section 203 has a basic mesh subdivision section 203A and a subdivision mesh adjustment section 203B.

[0202] (Basic mesh refinement section 203A) The basic mesh subdivision unit 203A is configured to calculate the number of divisions (number of subdivisions) for each basic surface and basic patch based on the input basic mesh and division information of the basic mesh, subdivide the basic mesh based on the number of divisions, and output the subdivision surface.

[0203] In other words, the basic mesh subdivision unit 203A may be configured to be able to change the above-mentioned number of divisions for each basic surface and basic patch.

[0204] Here, a base face is a face that constitutes a base mesh, and a base patch is a collection of several base faces.

[0205] In addition, the basic mesh subdivision unit 203A may be configured to predict the number of subdivisions of the base surface and calculate the number of subdivisions of the base surface by adding a predicted subdivision number residual to the predicted number of subdivisions of the base surface.

[0206] The basic mesh subdivision unit 203A may be configured to calculate the number of subdivisions of a basic surface based on the number of subdivisions of an adjacent basic surface of the basic surface.

[0207] The basic mesh subdivision unit 203A may be configured to calculate the number of subdivisions of a base surface based on the number of subdivisions of the base surface that has been stored immediately before.

[0208] The basic mesh subdivision unit 203A may be configured to generate vertices that divide the three sides that constitute the basic surface, and to connect the generated vertices to subdivide the basic surface.

[0209] As shown in FIG. 21, a basic mesh subdivision section 203A is followed by a subdivision mesh adjustment section 203B, which will be described later.

[0210] An example of the processing of the basic mesh subdivision section 203A will be described below with reference to FIGS.

[0211] FIG. 22 is a diagram showing an example of functional blocks of the basic mesh subdivision unit 203A, and FIG. 24 is a flowchart showing an example of the operation of the basic mesh subdivision unit 203A.

[0212] As shown in FIG. 22, the basic mesh subdivision unit 203A has a basic surface division number buffer unit 203A1, a basic surface division number reference unit 203A2, a basic surface division number prediction unit 203A3, an addition unit 203A4, and a basic surface division unit 203A5.

[0213] The basic surface division number buffer unit 203A1 stores division information of basic surfaces including the division number of the basic surface, and is configured to output the division information of the basic surface to the basic surface division number reference unit 203A2.

[0214] Here, the size of the basic surface division number buffer unit 203A1 may be set to 1, and the unit may be configured to output the division number of the basic surface that was stored immediately before to the basic surface division number reference unit 203A2.

[0215] That is, by setting the size of the basic surface division number buffer unit 203A1 to 1, it may be configured to refer to only the last decoded fine division number (the subdivision number decoded immediately before).

[0216] The basic surface division number reference unit 203A2 is configured to output a reference not possible to the basic surface division number prediction unit 203A3 when there is no adjacent basic surface to the basic surface to be decoded, or when there is an adjacent basic surface to the basic surface to be decoded but the number of divisions has not been determined.

[0217] On the other hand, when a base face adjacent to the base face to be decoded exists and the number of divisions is determined, the basic face division number reference unit 203A2 is configured to output the number of divisions to the basic face division number prediction unit 203A3.

[0218] The basic surface division number prediction unit 203A3 is configured to predict the division number (subdivision number) of a basic surface based on one or more input division numbers, and output the predicted division number (predicted division number) to the addition unit 203A4.

[0219] Here, the basic surface division number prediction unit 203A3 is configured to output 0 to the addition unit 203A4 when only reference impossible is input from the basic surface division number reference unit 203A2.

[0220] In addition, when one or more division numbers are input, the basic surface division number prediction unit 203A3 may be configured to generate a predicted division number using any of the statistical values ​​such as the average value, maximum value, minimum value, or mode of the input division numbers.

[0221] The basic face division number prediction unit 203A3 may be configured to generate the division number of the most adjacent face as the predicted division number when one or more division numbers are input.

[0222] The addition unit 203A4 is configured to output the division number obtained by adding the prediction division number residual decoded from the prediction residual bitstream and the prediction division number obtained from the basic surface division number prediction unit 203A3 to the basic surface division unit 203A5.

[0223] The basic surface division unit 203A5 is configured to finely divide the basic surface based on the number of divisions input from the addition unit 203A4.

[0224] FIG. 23 is an example of a case where the basic surface is divided into nine parts. With reference to FIG. 23, the method of dividing the basic surface by the basic surface division unit 203A5 will be described.

[0225] The basic surface division unit 203A5 generates points A_1,…, A_(N - 1) that divide the side AB constituting the basic surface into N equal parts (N = 3).

[0226] Similarly, the basic surface division unit 203A5 also divides the sides BC and CA into N equal parts, and generates points B_1,…, B_(N - 1) and C_1,…, C_(N - 1) respectively.

[0227] Hereinafter, the points on the sides AB, BC, and CA are referred to as "side division points".

[0228] For all i (i = 1, 2,…, N - 1), the basic surface division unit 203A5 generates the sides A_i B_(N - i), B_i C_(N - i), C_i A_(N - i), and generates N 2 fine division surfaces.

[0229] Next, with reference to FIG. 24, the processing procedure of the basic mesh division unit 203A will be described.

[0230] In step S2201, the basic mesh division unit 203A determines whether the division process for the last basic surface has been completed. If the process is completed, it ends; otherwise, it proceeds to step S2202.

[0231] In step S2202, the basic mesh division unit 203A determines whether Depth < mdu_max_depth.

[0232] Here, Depth is a variable representing the current depth, with an initial value of 0, and mdu_max_depth represents the maximum depth determined for each basic surface.

[0233] If the condition in step S2202 is met, the process proceeds to step S2203; if the condition is not met, the process returns to step S2201.

[0234] In step S2203, the basic mesh subdivision unit 203A determines whether or not mdu_subdivision_flag is 1 at the current depth.

[0235] If the answer is Yes, the process returns to step S2201; if the answer is No, the process proceeds to step S2204.

[0236] In step S2204, the basic mesh subdivision unit 203A further subdivides all the subdivision surfaces within the basic surface.

[0237] Here, if the subdivision process has never been performed on the basic surface, the basic mesh subdivision unit 203A subdivides the basic surface.

[0238] The subdivision method is the same as that described in step S2204.

[0239] Specifically, if a basic surface has never been subdivided, the basic surface is subdivided as shown in FIG. 23. If the basic surface has been subdivided at least once, the subdivision surface is 2 In the example of Figure 23, the face consisting of vertices A_2, B, and B_1 is further divided into N 2 Generate faces.

[0240] When the subdivision process is completed, the process proceeds to step S2205.

[0241] In step S2205, the basic mesh subdivision unit 203A adds 1 to Depth, and the processing procedure returns to step S2202.

[0242] (Fine division mesh adjustment unit 203B) Next, a specific example of the process performed by the finely divided mesh adjustment unit 203B will be described. An example of the process performed by the finely divided mesh adjustment unit 203B will be described below with reference to Figs.

[0243] FIG. 25 is a diagram showing an example of functional blocks of the finely divided mesh adjustment unit 203B.

[0244] As shown in FIG. 25, the subdivision mesh adjustment unit 203B has an edge division point moving unit 701 and a subdivision surface division unit 702.

[0245] (Edge division point moving unit 701) The edge division point moving unit 701 is configured to move an edge division point of a basic face to any of the edge division points of an adjacent basic face for an input initial subdivision face, and output a subdivision face.

[0246] Fig. 26 shows an example in which an edge division point on a basic face ABC is moved. For example, as shown in Fig. 26, the edge division point moving unit 701 may be configured to move an edge division point of the basic face ABC to an edge division point of the nearest adjacent basic face.

[0247] (Subdivision surface division part 702) The subdivision surface division unit 702 is configured to re-subdivide the input subdivision surface and output a decomposed subdivision surface.

[0248] FIG. 27 is a diagram showing an example of a case where subdivision is performed again on a subdivision surface X in a base surface.

[0249] As shown in FIG. 27, the subdivision surface division unit 702 may be configured to generate a new subdivision surface within a base surface by connecting vertices constituting the subdivision surface to edge division points of an adjacent base surface.

[0250] FIG. 28 is a diagram showing an example of a case where the above-mentioned subdivision processing is performed on all subdivision surfaces.

[0251] The mesh decoding unit 204 is configured to generate and output a decoded mesh using the subdivision mesh generated by the subdivision unit 203 and the displacement amount decoded by the displacement amount decoding unit 206 .

[0252] Specifically, the mesh decoding unit 204 is configured to generate a decoded mesh by adding to each subdivision vertex a corresponding displacement amount, where information about which subdivision vertex each displacement amount corresponds to is indicated by control information.

[0253] The patch integration unit 205 is configured to integrate the decoded mesh generated by the mesh decoding unit 206 by the amount of a plurality of patches and output the integrated mesh.

[0254] Here, the patch division method is defined by the mesh encoding device 100. For example, the patch division method may be configured to calculate a normal vector for each basic face, select a basic face having the most similar normal vector among adjacent basic faces, combine both basic faces into the same patch, and repeat such a procedure for the next basic face in sequence.

[0255] The video decoding unit 207 is configured to decode and output the texture by video encoding. For example, the video decoding unit 207 may use HEVC.

[0256] <Displacement Amount Decoding Unit 206> The displacement amount decoding unit 206 is configured to decode the displacement amount bitstream to generate and output the displacement amount.

[0257] 3B is a diagram showing an example of a displacement amount for a certain subdivision vertex. In the example of FIG. 3B, since there are eight subdivision vertices, the displacement amount decoding unit 206 is configured to define eight displacement amounts expressed as scalars or vectors for each subdivision vertex.

[0258] The displacement amount decoding unit 206 will be described below with reference to Fig. 29. Fig. 29 is a diagram showing an example of functional blocks of the displacement amount decoding unit 206.

[0259] As shown in FIG. 29, the displacement amount decoding unit 206 has a decoding unit 206A, an inverse quantization unit 206B, an inverse wavelet transform unit 206C, an adder 206D, an inter prediction unit 206E, and a frame buffer 206F.

[0260] The decoding unit 206A is configured to decode and output the level value and the control information by performing variable length decoding on the received displacement bit stream. Here, the level value obtained by the variable length decoding is output to the inverse quantization unit 206B, and the control information is output to the inter prediction unit 206E.

[0261] An example of the configuration of a displacement amount bit stream will be described below with reference to Fig. 30. Fig. 30 is a diagram showing an example of the configuration of a displacement amount bit stream.

[0262] As shown in FIG. 30, first, the displacement bit stream may include a Displacement Parameter Set (DPS) which is a set of control information related to decoding of the displacement.

[0263] Secondly, the displacement bitstream may include a Displacement Patch Header (DPH), which is a collection of control information corresponding to a patch.

[0264] Third, the displacement bitstream may contain, next to the DPH, the encoded displacements that make up the patch.

[0265] As described above, the displacement bit stream has a configuration in which each coded displacement corresponds to one DPH and one DPS.

[0266] Note that the configuration in Fig. 30 is merely an example. As long as the DPH and DPS correspond to each coded displacement amount, elements other than those described above may be added as components of the displacement amount bit stream.

[0267] For example, as shown in FIG. 30, the displacement bit stream may include a sequence parameter set (SPS).

[0268] FIG. 31 is a diagram showing an example of a syntax configuration of a DPS.

[0269] In FIG. 31, the Descriptor column indicates how each syntax element is coded.

[0270] In addition, in FIG. 31, ue(v) means an unsigned zeroth-order exponential Golomb code, and u(n) means an n-bit flag.

[0271] When multiple DPSs exist, the DPS includes at least DPS id information (dps_displacement_parameter_set_id) for identifying each DPS.

[0272] Furthermore, the DPS may include a flag (interprediction_enabled_flag) that controls whether or not inter prediction is performed.

[0273] For example, it may be defined that inter prediction is not performed when interprediction_enabled_flag is 0, and that inter prediction is performed when interprediction_enabled_flag is 1. It may be defined that inter prediction is not performed when interprediction_enabled_flag is not included.

[0274] The DPS may include a flag (dct_enabled_flag) that controls whether or not to perform an inverse DCT.

[0275] For example, it may be defined that the inverse DCT is not performed when dct_enabled_flag is 0, and that the inverse DCT is performed when dct_enabled_flag is 1. It may be defined that the inverse DCT is not performed when dct_enabled_flag is not included.

[0276] FIG. 32 is a diagram illustrating an example of a syntax configuration of the DPH.

[0277] As shown in FIG. 32, a DPH includes at least DPS id information for specifying a DPS corresponding to each DPH.

[0278] The inverse quantization unit 206B is configured to inversely quantize the level values ​​decoded by the decoding unit 206A to generate and output transform coefficients.

[0279] The inverse wavelet transform unit 206C is configured to perform an inverse wavelet transform on the transform coefficients generated by the inverse quantization unit 206B to generate and output prediction residuals.

[0280] (Inter prediction unit 206E) The inter prediction unit 206E is configured to perform inter prediction using the decoded displacement amount of the reference frame read from the frame buffer 206F, thereby generating and outputting a predicted displacement amount.

[0281] The inter prediction unit 206E is configured to perform such inter prediction only when interprediction_enabled_flag is set to one.

[0282] The inter prediction unit 206E may perform inter prediction in the spatial domain or may perform inter prediction in the frequency domain. The inter prediction may be bidirectional prediction using a past reference frame and a future reference frame in terms of time.

[0283] FIG. 33 is a diagram illustrating an example of a correspondence relationship between subdivision vertices between a reference frame and a current frame for decoding when inter prediction is performed in the spatial domain.

[0284] FIG. 34 illustrates an example of functional blocks of the inter prediction unit 206E when performing inter prediction in the frequency domain.

[0285] When performing inter prediction in the frequency domain, the inter prediction unit 206E may determine predicted wavelet transform coefficients of a frequency in a frame to be decoded by directly referring to decoded wavelet transform coefficients of a corresponding frequency in a reference frame.

[0286] The inter prediction unit 206E may perform stochastic inter prediction according to a normal distribution with estimated mean and variance, using the decoded displacement amounts or decoded wavelet transform coefficients of the subdivision vertices in a plurality of reference frames.

[0287] The inter prediction unit 206E may perform inter prediction based on a regression curve estimated using the decoded displacement amounts or decoded wavelet transform coefficients of the subdivision vertices in a plurality of reference frames, with time as an explanatory variable and the displacement amount as a response variable.

[0288] The inter prediction unit 206E may be configured to perform bidirectional inter prediction using a temporally past reference frame and a future reference frame.

[0289] In the mesh coding device 100, the order of the decoded wavelet transform coefficients may be rearranged in order to improve the coding efficiency for each frame.

[0290] The frequency correspondence between the reference frame and the frame to be decoded is indicated by control information.

[0291] FIG. 35 is a diagram illustrating an example of a correspondence relationship in frequency between a reference frame and a current frame to be decoded when inter prediction is performed in the frequency domain.

[0292] In addition, when the subdivision unit 203 divides the base mesh into a plurality of patches, the inter prediction unit 206E is also configured to perform inter prediction for each divided patch. This increases the time correlation between frames, and is expected to improve coding performance.

[0293] The adder 206D receives the prediction residual from the inverse wavelet transform unit 206C and the prediction displacement amount from the inter prediction unit 206E.

[0294] The adder 206D is configured to add the prediction residual and the prediction displacement amount to calculate and output the decoded displacement amount.

[0295] The decoded displacement calculated by the adder 206D is also output to the frame buffer 206F.

[0296] The frame buffer 206F is configured to receive and store the decoded displacement amount from the adder 206D.

[0297] Here, the frame buffer 206F outputs the decoded displacement amount at the corresponding vertex in the reference frame in response to control information (not shown).

[0298] FIG. 36 is a flowchart showing an example of the operation of the displacement amount decoding unit 206.

[0299] As shown in FIG. 36, in step S3501, the displacement decoding unit 206 determines whether or not this process has been completed for all patches.

[0300] If Yes, the operation ends, and if No, the operation proceeds to step S3502.

[0301] In step S3502, the displacement decoding unit 206 performs inverse DCT, and then performs inverse quantization and inverse wavelet transform on the patch to be decoded.

[0302] In step S3503, the displacement decoding unit 206 determines whether or not the interpretation_enabled flag is 1.

[0303] If yes, the operation proceeds to step S3504; if no, the operation returns to step S3501.

[0304] In step S3504, the displacement decoding unit 206 performs the above-mentioned inter prediction and addition.

[0305] <Variation 1> Hereinafter, with reference to FIG. 37, a first modification of the first embodiment will be described, focusing on the differences from the first embodiment.

[0306] FIG. 37 is a diagram showing an example of functional blocks of the displacement amount decoding unit 206 according to the present first modification.

[0307] As shown in FIG. 37, the displacement amount decoding unit 206 according to the present first modification includes an inverse DCT unit 206G subsequent to the decoding unit 206A, that is, between the decoding unit 206A and the inverse quantization unit 206B.

[0308] That is, in the present first modification, the inverse quantization unit 206B is configured to generate a prediction residual by performing an inverse wavelet transform on the level values ​​output from the inverse DCT unit 202G.

[0309] <Variation 2> Hereinafter, the second modification of the first embodiment will be described with reference to FIG. 38, focusing on the differences from the first embodiment.

[0310] As shown in FIG. 38, the displacement amount decoding unit 206 according to the present second modification includes a video decoding unit 2061, an image developing unit 2062, an inverse quantization unit 2063, and an inverse wavelet transformation unit 2064.

[0311] The video decoding unit 2061 is configured to decode the received displacement amount bit stream by video coding, and output a video.

[0312] For example, the video decoding unit 2061 may use HEVC in Non-Patent Document 1.

[0313] Furthermore, the video decoding unit 2061 may use a video encoding method in which a motion vector is always set to 0. For example, the video decoding unit 2061 may always set a motion vector of HEVC to 0 and always use inter prediction at the same position.

[0314] Furthermore, the video decoding unit 2061 may use a video encoding method in which conversion is always skipped. For example, the video decoding unit 2061 may always perform HEVC conversion in a conversion skip mode and use a video encoding method without conversion.

[0315] The image development unit 2062 is configured to develop the video decoded by the video decoding unit 2061 as a level value for each image (frame) and output it.

[0316] In such a development method, the image development unit 2062 can specify it by calculating backwards from the arrangement of the level values ​​in the image indicated by the control information.

[0317] The image development unit 2062 may arrange the level values ​​in the image, for example, in the order of raster operation, from high frequency components to low frequency components.

[0318] The inverse quantization unit 2063 is configured to inversely quantize the level values ​​generated by the image development unit 2062 to generate and output transformation coefficients.

[0319] The inverse wavelet transform unit 2064 is configured to perform an inverse wavelet transform on the transform coefficients generated by the inverse quantization unit 2063, thereby generating and outputting a decoded displacement amount.

[0320] The above-mentioned mesh encoding device 100 and mesh decoding device 200 may be realized as a program that causes a computer to execute each function (each step). [Industrial Applicability]

[0321] According to this embodiment, for example, it is possible to improve the overall service quality in video communication, which makes it possible to contribute to Goal 9 of the Sustainable Development Goals (SDGs) led by the United Nations, which is to "build resilient infrastructure, promote sustainable industrialization and foster innovation." [Explanation of symbols]

[0322] 1. Mesh processing system 100...Mesh coding device 200…Mesh decoding unit 201…Demultiplexer 202...Basic mesh decoding unit 202A…Separation part 202B... Intra decoding unit 202B1...Optional intra-decoding unit 202B2…Alignment section 202C…Mesh buffer section 202D…Connection information decoding unit 202E…Inter-decoding section 202E1...Motion vector decoding unit 202E2...Motion vector buffer unit 202E3...Motion vector prediction unit 202E4...Motion vector calculation unit 202E5…Adder 202F…Skip decoding section 203…Subdivision part 203A…Basic mesh refinement section 203A1...Basic face division number buffer section 203A2…Basic surface division number reference section 203A3...Basic surface division number prediction unit 203A4…Addition section 203A5…Basic surface division part 203B…Subdivision mesh adjustment section 701...Edge division point moving part 702…Subdivision surface division part 204...Mesh decoding unit 205…Patch Integration Department 206...Displacement amount decoding unit 206A…Decoding section 206B, 2063...Dequantization section 206C, 2064...Inverse wavelet transform section 206D…Adder 206E…Inter prediction section 206F…Frame buffer 206G…Reverse DCT section 2062…Image development section 207, 2061...Video decoding section

Claims

1. A mesh decoding device, comprising: a motion vector residual decoding unit that generates motion vector residuals from an inter-frame bitstream; a motion vector prediction unit that acquires decoded motion vectors from a vertex to be decoded and surrounding vertices connected to the vertex to be decoded, and outputs a motion vector prediction value of the vertex to be decoded using all or part of the decoded motion vectors; a motion vector calculation unit that outputs a motion vector of the vertex to be decoded, In mode 1, the motion vector calculation unit adds the motion vector residual and the motion vector prediction value to output the motion vector of the vertex to be decoded; In mode 0, the motion vector calculation unit outputs the motion vector residual as a motion vector of the vertex to be decoded; the motion vector calculation unit sets the modes of motion vectors of N (N≧1) consecutive vertices in decoding order to be the same; The N vertices form one group, The mesh decoding device is characterized in that the motion vector calculation unit decodes a control signal from a bitstream, which can calculate a group size indicating the size of the group.

2. 2. The mesh decoding device according to claim 1, wherein the motion vector calculation unit is configured so that the range that can be set as the group size covers from 1 to a predetermined maximum value.

3. 3. The mesh decoding device according to claim 2, wherein the maximum value is equal to or greater than the number of all vertices of the base mesh.

4. The mesh decoding device according to claim 1 , wherein the motion vector calculation unit groups all vertices in the group into one group when the group size is larger than the number of vertices in the group.

5. 2. The mesh decoding device according to claim 1, wherein the control signal is a natural number or a logarithm.

6. The mesh decoding device according to claim 1, wherein the control signal is log2_group_size, and the group size is calculated by the following equation (4): group size=2 log2_group_size....

7. 5. The mesh decoding device according to claim 4, wherein the motion vector calculation unit calculates the group size using an exponential function when the control signal is a logarithm.

8. 2. The mesh decoding device according to claim 1, wherein the range that can be set as the group size is a fixed value or is calculated from other control signals or data.

9. The mesh decoding device described in claim 8, characterized in that the range that can be set as the group size, when calculated from the other control signal or data, is the smallest natural number that is a power of 2 that can cover the number of vertices of the basic mesh.

10. The mesh decoding device described in claim 1, characterized in that when a predetermined flag is TRUE, the motion vector calculation unit sets the number of all vertices to the group size, and when the predetermined flag is FALSE, the group size calculated from the control signal is not changed.

11. 2. The mesh decoding device according to claim 1, wherein the control signal is set for each sequence or each frame.

12. 1. A mesh decoding method, comprising: A step A of generating motion vector residuals from an inter-frame bitstream; a step B of acquiring decoded motion vectors from a vertex to be decoded and surrounding vertices connected to the vertex to be decoded, and outputting a motion vector prediction value of the vertex to be decoded using all or part of the decoded motion vectors; and a step C of outputting a motion vector of the vertex to be decoded, In the step C, In mode 1, the motion vector residual and the motion vector prediction value are added together to output the motion vector of the vertex to be decoded; In mode 0, the motion vector residual is output as a motion vector of the vertex to be decoded; The modes of the motion vectors of N (N≧1) consecutive vertices in decoding order are made the same; The N vertices form one group, A mesh decoding method, characterized by decoding a control signal from a bitstream, which can calculate a group size indicating the size of the group.

13. A program that causes a computer to function as a mesh decoding device, The mesh decoding device comprises: a motion vector residual decoding unit that generates motion vector residuals from an inter-frame bitstream; a motion vector prediction unit that acquires decoded motion vectors from a vertex to be decoded and surrounding vertices connected to the vertex to be decoded, and outputs a motion vector prediction value of the vertex to be decoded using all or part of the decoded motion vectors; a motion vector calculation unit that outputs a motion vector of the vertex to be decoded, In mode 1, the motion vector calculation unit adds the motion vector residual and the motion vector prediction value to output the motion vector of the vertex to be decoded; In mode 0, the motion vector calculation unit outputs the motion vector residual as a motion vector of the vertex to be decoded; the motion vector calculation unit sets the modes of motion vectors of N (N≧1) consecutive vertices in decoding order to be the same; The N vertices form one group, The program, wherein the motion vector calculation unit decodes a control signal from a bitstream, which allows calculation of a group size indicating the size of the group.

Images