Mesh decoding device, mesh decoding method, and program

Abstract

To improve mesh coding efficiency.SOLUTION: In a mesh coding device 200 according to an invention, a motion vector calculation unit 202E4 adds a motion vector residual and a motion vector prediction value and outputs a motion vector of the apex of a decoding target in a mode 1, and the motion vector calculation unit 202E4 outputs the motion vector residual as the motion vector of the apex of the decoding target in a mode 0.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to a mesh decoding device, a mesh decoding method, and a program. [Background technology]

[0002] Non-patent document 1 discloses a technique for encoding a mesh using non-patent document 2 or 3. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Khaled Mammou, Jungsun Kim, Alexis M Tourapis, Dimitri Podborski, and Krasimir Kolarov, “[V-CG] Apple's Dynamic Mesh Coding CfP Response,” April 2022, ISO / IEC JTC 1 / SC 29 / WG 7. [Non-Patent Document 2] Google Draco, accessed May 26, 2022 [Online], https: / / google.github.io / draco [Non-Patent Document 3] Jean-Eudes Marvie, Olivier Mocquard, “[V-DMC][EE4.4-related] An efficient EdgeBreaker implementation,” April 2023, ISO / IEC JTC 1 / SC 29 / WG 7 m63344. [Overview of the project] [Problems that the invention aims to solve]

[0004] However, conventional technology had the problem of low encoding efficiency for motion vectors. Therefore, the present invention has been made in view of the above-mentioned problems, and aims to provide a mesh decoding device, a mesh decoding method, and a program that can improve the encoding efficiency of meshes. [Means for solving the problem]

[0005] The first feature of the present invention is a mesh decoding device comprising an inter-decoding unit that decodes the coordinates of a vertex in a P-frame by adding the coordinates of a vertex in a reference frame and a motion vector decoded from the bitstream of the P-frame, wherein the inter-decoding unit comprises a motion vector residual decoding unit that generates a motion vector residual from the bitstream of the interframe, a motion vector prediction unit that predicts the decoded motion vector from the vertex to be decoded and the surrounding vertices connected to the vertex to be decoded, calculates a motion vector prediction value using all or part of the decoded motion vectors of the vertices around the vertex to be decoded so as not to exceed the maximum number of uses, decodes a control signal from the bitstream that can calculate the maximum number of adjacent vertices, and a motion vector calculation unit that outputs the motion vector of the vertex to be decoded, wherein in mode 1, the motion vector calculation unit outputs the motion vector of the vertex to be decoded by adding the motion vector residual and the motion vector prediction value, and in mode 0, the motion vector calculation unit outputs the motion vector residual as the motion vector of the vertex to be decoded.

[0006] A second feature of the present invention is a mesh decoding method comprising the steps of decoding the coordinates of a vertex in a P-frame by adding the coordinates of a vertex in a reference frame to a motion vector decoded from the bitstream of the P-frame, wherein the steps include: step A, which generates a motion vector residual from the bitstream of the interframe; step B, which predicts the decoded motion vector from the vertex to be decoded and the surrounding vertices connected to the vertex to be decoded, calculates a motion vector prediction value using all or part of the decoded motion vectors of the vertices surrounding the vertex to be decoded so as not to exceed the maximum number of uses, and decodes a control signal from the bitstream that can calculate the maximum number of adjacent vertices; and step C, which outputs the motion vector of the vertex to be decoded, wherein in step C, in mode 1, the motion vector residual and the motion vector prediction value are added to output the motion vector of the vertex to be decoded, and in mode 0, the motion vector residual is output as the motion vector of the vertex to be decoded.

[0007] A third feature of the present invention is a program that causes a computer to function as a mesh decoder, wherein the mesh decoder includes an inter-decoding unit that decodes the coordinates of the vertices of a P-frame by adding the coordinates of the vertices of a reference frame and the motion vectors decoded from the bitstream of the P-frame, the inter-decoding unit includes a motion vector residual decoding unit that generates motion vector residuals from the bitstream of the interframe, a motion vector prediction unit that predicts the decoded motion vectors from the vertex to be decoded and the surrounding vertices connected to the vertex to be decoded, calculates a motion vector prediction value using all or part of the decoded motion vectors of the vertices around the vertex to be decoded so as not to exceed the maximum number of uses, decodes a control signal from the bitstream that can calculate the maximum number of adjacent vertices, and a motion vector calculation unit that outputs the motion vector of the vertex to be decoded, the gist of which is that in mode 1, the motion vector calculation unit outputs the motion vector of the vertex to be decoded by adding the motion vector residual and the motion vector prediction value, and in mode 0, the motion vector calculation unit outputs the motion vector residual as the motion vector of the vertex to be decoded. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide a mesh decoding device, a mesh decoding method, and a program that can improve the coding efficiency of the mesh. [Brief explanation of the drawing]

[0009] [Figure 1] Figure 1 shows an example of the configuration of a mesh processing system 1 according to one embodiment. [Figure 2] Figure 2 shows an example of a functional block of a mesh decoding device 200 according to one embodiment. [Figure 3A] Figure 3A shows an example of a basic mesh and a subdivided mesh. [Figure 3B] Figure 3B shows an example of a basic mesh and a subdivided mesh. [Figure 4]FIG. 4 is a diagram showing an example of a functional block of a basic mesh decoding unit 202 of a mesh decoding apparatus 200 according to an embodiment. [Figure 5] FIG. 5 is a diagram showing an example of a functional block of an intra decoding unit 202B of a basic mesh decoding unit 202 of a mesh decoding apparatus 200 according to an embodiment. [Figure 6] FIG. 6 is a diagram showing an example of a correspondence relationship between vertices of a basic mesh of a P frame and vertices of a basic mesh of an I frame. [Figure 7] FIG. 7 is a diagram showing an example of a functional block of an inter decoding unit 202E of a basic mesh decoding unit 202 of a mesh decoding apparatus 200 according to an embodiment. [Figure 8] FIG. 8 is a diagram for explaining an example of a method for calculating an MVP of a vertex to be decoded by a motion vector prediction unit 202E3 of an inter decoding unit 202E of a basic mesh decoding unit 202 of a mesh decoding apparatus 200 according to an embodiment. [Figure 9] FIG. 9 shows a flowchart showing an example of an operation of a motion vector prediction unit 202E3 of an inter decoding unit 202E of a basic mesh decoding unit 202 of a mesh decoding apparatus 200 according to an embodiment. [Figure 10A] FIG. 10A is a diagram showing an example of a decoding order for a mesh. [Figure 10B] FIG. 10B is an example of a list of vertices around a vertex to be decoded. [Figure 11] FIG. 11 is a diagram showing an example of statistical data showing a relationship between the number of decoded motion vectors and the number of vertices around a vertex to be decoded. [Figure 12] FIG. 12 is a diagram for explaining an example of a worst case. [Figure 13] FIG. 13 is a diagram for explaining a modification example 2 of an inter decoding unit 202E of a basic mesh decoding unit 202 of a mesh decoding apparatus 200 according to an embodiment. [Figure 14] FIG. 14 is a diagram for explaining a modification example 2 of an inter decoding unit 202E of a basic mesh decoding unit 202 of a mesh decoding apparatus 200 according to an embodiment. [Figure 15] Figure 15 shows an example of a modification of the functional block in the basic mesh decoding unit 202 of a mesh decoding device 200 according to one embodiment. [Figure 16] Figure 16 is a diagram illustrating a modification example 1 of the basic mesh decoding unit 202 of a mesh decoding device 200 according to one embodiment. [Figure 17] Figure 17 is a diagram illustrating the mesh buffer section 202C of the basic mesh decoding section 202 of a mesh decoding device 200 according to one embodiment. [Figure 18] Figure 18 is a diagram illustrating the mesh buffer section 202C of the basic mesh decoding section 202 of a mesh decoding device 200 according to one embodiment. [Figure 19] Figure 19 is a diagram illustrating an example of a modification to the basic mesh decoding unit 202 of the mesh decoding device 200 according to modification example 2. [Figure 20] Figure 20 is a diagram illustrating an example of a modification to the basic mesh decoding unit 202 of the mesh decoding device 200 according to modification example 2. [Figure 21] Figure 21 is a diagram showing an example of a functional block of the subdivision section 203 of a mesh decoding device 200 according to one embodiment. [Figure 22] Figure 22 shows an example of a functional block of the basic mesh subdivision section 203A of the subdivision section 203 of the mesh decoding device 200 according to one embodiment. [Figure 23] Figure 23 is a diagram illustrating an example of a method for dividing a basic surface by a basic surface division section 203A5 of the basic mesh subdivision section 203A of the mesh decoding device 200 according to one embodiment. [Figure 24] Figure 24 is a flowchart showing an example of the operation of the basic mesh subdivision section 203A of the subdivision section 203 of the mesh decoding device 200 according to one embodiment. [Figure 25] Figure 25 shows an example of the functional block of the subdivided mesh adjustment unit 203B of the subdivided section 203 of the mesh decoding device 200 according to one embodiment. [Figure 26] Figure 26 shows an example of a case in which the edge division points on the basic plane ABC are moved by the edge division point movement unit 701 of the subdivided mesh adjustment unit 203B of the subdivided division unit 203 of the mesh decoding device 200 according to one embodiment. [Figure 27] Figure 27 shows an example of a case in which the subdivided surface X within the basic plane is further subdivided by the subdivided mesh adjustment unit 203B of the subdivided division unit 203 of the mesh decoding device 200 according to one embodiment. [Figure 28] Figure 28 shows an example of a case in which all subdivided surfaces are subdivided again by the subdivided surface division section 702 of the subdivided mesh adjustment section 203B of the subdivided section 203 of the mesh decoding device 200 according to one embodiment. [Figure 29] Figure 29 shows an example of the functional block of the displacement decoding unit 206 of the mesh decoding device 200 according to one embodiment (when interpretation is performed in the spatial domain). [Figure 30] Figure 30 shows an example of the configuration of a displacement bitstream. [Figure 31] Figure 31 shows an example of the syntax configuration of DPS. [Figure 32] Figure 32 shows an example of the syntax configuration of DPH. [Figure 33] Figure 33 illustrates an example of the frequency correspondence between a reference frame and a decoded frame when interpretation is performed in the spatial domain. [Figure 34] Figure 34 is a diagram showing an example of the functional block of the displacement decoding unit 206 of the mesh decoding device 200 according to one embodiment (when interpretation is performed in the frequency domain). [Figure 35] Figure 35 illustrates an example of the frequency correspondence between a reference frame and a decoded frame when interpretation is performed in the frequency domain. [Figure 36]Figure 36 is a flowchart showing an example of the operation of the displacement decoding unit 206 of a mesh decoding device 200 according to one embodiment. [Figure 37] Figure 37 shows an example of the functional block of the displacement decoding unit 206 according to Modification 1. [Figure 38] Figure 38 shows an example of the functional block of the displacement decoding unit 206 according to the modified example 2. [Figure 39] Figure 39 shows an example of a NAL header. [Modes for carrying out the invention]

[0010] Embodiments of the present invention will be described below with reference to the drawings. Note that the components in the following embodiments can be replaced with existing components as appropriate, and various variations are possible, including combinations with other existing components. Therefore, the description of the following embodiments does not limit the content of the invention as described in the claims.

[0011] <First Embodiment> The mesh processing system according to this embodiment will be described below with reference to Figures 1 to 38.

[0012] Figure 1 shows an example of the configuration of the mesh processing system 1 according to this embodiment. As shown in Figure 1, the mesh processing system 1 includes a mesh encoding device 100 and a mesh decoding device 200.

[0013] Figure 2 shows an example of the functional block of the mesh decoding device 200 according to this embodiment.

[0014] As shown in Figure 2, the mesh decoding device 200 includes a multiplexing unit 201, a basic mesh decoding unit 202, a subdivision unit 203, a mesh decoding unit 204, a patch integration unit 205, a displacement decoding unit 206, and a video decoding unit 207.

[0015] Here, the basic mesh decoding unit 202, the subdivision unit 203, the mesh decoding unit 204, and the displacement decoding unit 206 are configured to process the mesh in patch units, and the results of these processes may then be integrated in the patch integration unit 205.

[0016] In the example shown in Figure 3A, the mesh is divided into patch 1, which consists of basic surfaces 1 and 2, and patch 2, which consists of basic surfaces 3 and 4.

[0017] The multiplexing unit 201 is configured to separate the multiplexed bitstream into a basic mesh bitstream, a displacement bitstream, and a texture bitstream.

[0018] <Basic Mesh Decoding Unit 202> The basic mesh decoding unit 202 is configured to decode the basic mesh bitstream, generate a basic mesh, and output it.

[0019] Here, the basic mesh consists of multiple vertices in three-dimensional space and edges that connect these multiple vertices.

[0020] As shown in Figure 3A, the basic mesh is constructed by combining basic faces, each represented by three vertices.

[0021] The basic mesh decoding unit 202 may be configured to decode the basic mesh bitstream using, for example, the Draco described in Non-Patent Document 2 or the technology described in Non-Patent Document 3.

[0022] Furthermore, the basic mesh decoding unit 202 may be configured to generate "subdivision_method_id," described later, as control information for controlling the type of subdivision method.

[0023] As shown in Figure 4, the basic mesh decoding unit 202 comprises a separation unit 202A, an intra decoding unit 202B, a mesh buffer unit 202C, a connection information decoding unit 202D, and an inter-decoding unit 202E.

[0024] The separation unit 202A is configured to classify the basic mesh bitstream into I-frame bitstreams and P-frame bitstreams.

[0025] (Intra-decoding unit 202B) The intra-decoding unit 202B is configured to decode the coordinates of the I-frame vertices and connection information from the I-frame bitstream using, for example, Draco as shown in Non-Patent Document 2 or the technology described in Non-Patent Document 3.

[0026] Figure 5 shows an example of the functional block of the intra-decoding unit 202B.

[0027] As shown in Figure 5, the intra-decoding unit 202B includes an arbitrary intra-decoding unit 202B1 and an alignment unit 202B2.

[0028] The arbitrary intra decoding unit 202B1 is configured to decode the coordinates and connection information of the unordered vertices of the I-frame from the bitstream of the I-frame using any scheme including Draco as shown in Non-Patent Document 2 or the technology described in Non-Patent Document 3.

[0029] The alignment unit 202B2 is configured to output vertices by rearranging unordered vertices into a predetermined order.

[0030] The predetermined order may be, for example, Morton code order or raster scan order.

[0031] Furthermore, the alignment unit 202B2 may group duplicate vertices, which are multiple vertices with the same coordinates in the decoded basic mesh, into a single vertex, and then rearrange them in a predetermined order.

[0032] The mesh buffer unit 202C is configured to store the coordinates and connection information of the vertices of the I-frame decoded by the intra-decoder unit 202B. A specific buffer may be provided here to store pairs of vertex indices A(k) and B(k) of duplicate vertices in a predetermined order.

[0033] The connection information decoding unit 202D is configured to convert the connection information of the I-frame or reference frame extracted from the mesh buffer unit 202C into the connection information of the P-frame.

[0034] The inter-decoding unit 202E is configured to decode the vertex coordinates of the P-frame by adding the vertex coordinates of the reference frame extracted from the mesh buffer unit 202C with the motion vector decoded from the bitstream of the P-frame.

[0035] Furthermore, the inter-decoding unit 202E can adjust the vertex indices of the P-frame using pairs of vertex indices A(k) and B(k) of vertices that exist as duplicate vertices stored in the specific buffer.

[0036] Here, all or part of the aforementioned index is decoded from the bitstream. This decoded method may be arithmetic coding. As a result, it can be expected that there will be no limit on the maximum value of the index to be decoded using arithmetic coding.

[0037] For example, we can use the arithmetic coding ue(v). ue(v) represents left-bit first unsigned integer zero-order exponential Golomb coding (Exp-Golomb).

[0038] Specifically, the parsing process for the syntactic elements of ue(v) begins at the current position in the bitstream, reading the bits containing the first non-zero bit and counting the number of leading bits equal to 0. This process is specified as follows: leadingZeroBits=-1 for(b=0;!b;leadingZeroBits++ b = read_bits(l) Next, the variable codeNum is assigned as follows:

[0039] coldNum=2 leadingZeroBits -1 + read_bits(leadingZeroBits) However, the value returned by read_bits(leadingZeroBits) is interpreted as a binary representation of an unsigned integer where the most significant bit is written first. Also, the value of ue(v) is equal to the value of codeNum.

[0040] Table 1 shows the structure of the Exp-Golomb code by separating the bit sequence into "prefix" bits and "suffix" bits.

[0041] [Table 1] Here, the “prefix” bits are the bits that are parsed as specified in the calculation of leadingZeroBits, and are represented as 0 or 1 in the bit sequence column of Table 1.

[0042] The "suffix" bit is the bit that is parsed in the calculation of codeNum, and in Table 1, x i It is displayed as follows. i is in the range from 0 to leadingZeroBits-1. Each x i It is equal to either 0 or 1.

[0043] Table 2 shows how to explicitly assign a bit string to the value of codeNum, where the value of ue(v) is equal to the value of codeNum.

[0044] [Table 2] In this embodiment, as shown in Figure 6, a correspondence exists between the vertices of the basic mesh of the P frame and the vertices of the basic mesh of the reference frame (I frame or P frame). Here, the motion vector decoded by the inter-decoding unit 202E is the difference vector between the coordinates of the vertices of the basic mesh of the P frame and the coordinates of the vertices of the basic mesh of the I frame.

[0045] (Inter-decoding unit 202E) Figure 7 shows an example of the functional block of the inter-decoding unit 202E.

[0046] As shown in Figure 7, the inter-decoding unit 202E includes a motion vector residual decoding unit 202E1, a motion vector buffer unit 202E2, a motion vector prediction unit 202E3, a motion vector calculation unit 202E4, and an adder 202E5.

[0047] The motion vector residual decoding unit 202E1 is configured to generate an MVR (Motion Vector Residual) from the bitstream of the P frame.

[0048] Here, MVR is the motion vector residual that shows the difference between MV (Motion Vector) and MVP (Motion Vector Prediction). MV is the difference vector (motion vector) between the coordinates of the vertex in the corresponding I-frame and the vertex in the P-frame. MVP is the predicted value of the MV of the target vertex (predicted value of the motion vector) using MV.

[0049] The motion vector buffer unit 202E2 is configured to sequentially save the MV output by the motion vector calculation unit 202E4.

[0050] The motion vector prediction unit 202E3 is configured to obtain decoded MVs from the motion vector buffer unit 202E2 for vertices connected to the vertex to be decoded, and to output the MVP of the vertex to be decoded using all or part of the obtained decoded MVs, as shown in Figure 8.

[0051] The motion vector calculation unit 202E4 is configured to add the MVR generated by the motion vector residual decoding unit 202E1 and the MVP output from the motion vector prediction unit 202E3, and output the MV of the vertex to be decoded.

[0052] The adder 202E5 is configured to add the coordinates of the vertices to be decoded, obtained from the decoded base mesh of the corresponding reference frame (I-frame or P-frame), to the motion vector MV output from the motion vector calculation unit 202E3, and output the coordinates of the vertices to be decoded.

[0053] The following describes the details of each part of the inter-decoding unit 202E.

[0054] Figure 9 shows a flowchart illustrating an example of the operation of the motion vector prediction unit 202E3. Hereafter, the operation of the motion vector prediction unit 202E3 will be referred to as the "average prediction method".

[0055] As shown in Figure 9, in step S1001, the motion vector prediction unit 202E3 sets MVP and N to 0.

[0056] In step S1002, the motion vector prediction unit 202E3 obtains a set of MVs of vertices around the vertex to be decoded from the motion vector buffer unit 202E2, identifies vertices for which subsequent processing has not been completed, and transitions to No. If subsequent processing has been completed for all vertices, it transitions to Yes.

[0057] In step S1003, the motion vector prediction unit 202E3 transitions to No if the MV of the vertex to be processed has not been decoded, and transitions to Yes if the MV of the vertex to be processed has been decoded.

[0058] In step S1004, the motion vector prediction unit 202E3 adds MV to MVP and adds 1 to N.

[0059] In step S1005, the motion vector prediction unit 202E3 outputs the result of dividing MVP by N if N is greater than 0, and outputs 0 if N is 0, and terminates the process.

[0060] In other words, the motion vector prediction unit 202E3 is configured to output the MVP of the vertex to be decoded by averaging the decoded motion vectors of the vertices surrounding the vertex to be decoded.

[0061] The motion vector prediction unit 202E3 may be configured to set the MVP to 0 if the set of decoded motion vectors is an empty set.

[0062] The motion vector calculation unit 202E4 may be configured to calculate the MV of the vertex to be decoded from the MVP output by the motion vector prediction unit 202E3 and the MVR generated by the motion vector residual decoding unit 202E1, according to equation (1).

[0063] MV(k) = MVP(k) + MVR(k) … (1) Here, k is the index of the vertex. MV, MVR, and MVP are vectors having x, y, and z components.

[0064] With this configuration, since MVP is used to encode only MVR instead of MV, it is expected that encoding efficiency will be improved.

[0065] The adder 202E5 calculates the coordinates of a vertex by adding the motion vector (MV) of the vertex calculated by the motion vector calculation unit 202E4 to the coordinates of the vertex in the corresponding reference frame, thereby keeping the connectivity information as it is in the reference frame.

[0066] Specifically, adder 202E5 uses equation (2) to determine the coordinates v' of the k-th vertex. i It may be configured to calculate (k).

[0067] v' i (k=v' j (k) + MV(k) … (2) Here, v' i (k) is the coordinate of the k-th vertex to be decoded in the frame to be decoded, and v' j (k) is the coordinate of the k-th decoded vertex of the reference frame, and MV(k) is the k-th MV of the frame being decoded, where k = 1, 2, ..., K.

[0068] Furthermore, the connection information of the frame to be decrypted will be identical to the connection information of the reference frame.

[0069] Furthermore, since the motion vector prediction unit 202E3 calculates the MVP using the decoded MV, the order of decoding affects the MVP.

[0070] The order of decoding is determined by the decoding order of the vertices of the base mesh in the reference frame. Generally, if a decoding method uses a fixed repeating pattern to add one base face at a time from a starting edge, the order of the decoded base mesh vertices is determined during the decoding process.

[0071] For example, the motion vector prediction unit 202E3 may use Edgebreaker to determine the order of vertex decoding in the basic mesh of the reference frame.

[0072] This configuration is expected to improve encoding efficiency because it encodes the MV from the reference frame instead of the vertex coordinates.

[0073] (Example 1 of modifications to the inter-decoding unit 202E) The following describes example 1 of the modifications to the inter-decoding unit 202E.

[0074] In the "average prediction method" where the motion vector prediction unit 202E3 of the inter-decoding unit 202E averages the decoded motion vectors of the vertices around the vertex to be decoded, in order not to exceed the predetermined maximum number of uses, it calculates the MVP by using all or only some of the decoded motion vectors of the vertices around the vertex to be decoded.

[0075] Note that the predetermined maximum number of uses is decoded from the bitstream as a control signal.

[0076] Also, when the number of decoded motion vectors of the vertices around the vertex to be decoded exceeds the maximum number of uses, the motion vector prediction unit 202E3 picks up to the maximum number of uses according to a certain rule.

[0077] [[ID=]12]For example, as such a rule, the motion vector prediction unit 202E3 decides to select the first or last vertex in the decoding order.

[0078] The decoding order for the mesh as shown in FIG. 10A is, as indicated by the arrow, vertex v D →v C →v A →v B That is.

[0079] FIG. 10B is a list of the vertices around the vertex to be decoded used when calculating the MVP of each vertex v A ~v D when the maximum value of the number of decoded adjacent vertices is set to 3.

[0080] According to such a configuration, by determining the maximum number of adjacent vertices, it is possible to expect the effect of reducing the calculation amount and the memory amount while maintaining or reducing the coding efficiency.

[0081] However, in order to exhibit the above effects, it is necessary to set an appropriate maximum number of adjacent vertices in the mesh encoding device 100 and write it into the bitstream as a related control signal.

[0082] Therefore, the range within which the maximum number of adjacent vertices can be set as described above determines the amount of memory provided by the mesh decoding device 200. As a reasonable constraint regarding the maximum number of adjacent vertices, the maximum number of adjacent vertices is encoded / decoded so that it is less than or equal to a predetermined maximum value.

[0083] Thus, by defining reasonable constraints on the maximum number of adjacent vertices, it is expected that the design of the mesh decoding device 200 will be made easier.

[0084] Generally, the average number of adjacent vertices in a closed 2-manifold triangle mesh is about 6, but statistically, the maximum number of adjacent vertices is often 7-8. As shown in Figure 11, the number of decoded motion vectors (vertical axis) changes dynamically depending on the number of vertices surrounding the vertex to be decoded (horizontal axis).

[0085] Therefore, it is desirable to narrow the range of possible settings for the maximum number of adjacent vertices mentioned above.

[0086] For example, as shown in Figure 11, by including "3," which is the number of vertices surrounding the vertex to be decoded that has the most decoded motion vectors statistically, within the range that can be set as the maximum number of adjacent vertices in the control signal described above, or by setting the upper limit (maximum value) of the range that can be set as the maximum number of adjacent vertices in the control signal described above to a certain ratio of the average number of adjacent vertices statistically (for example, 50% or 120%) or a value not greater than a natural number that can cover up to N bits (for example, 3 bits), the computational and memory usage can be reduced.

[0087] On the other hand, if the maximum number of adjacent vertices is set to a large value, such as the worst-case 256 or 8 bits, it may not be possible to achieve the desired reduction in both memory usage and computational complexity.

[0088] Figure 12 shows an example of the worst case, where the number of decoded neighboring vertices exceeds 256 when n ≥ 256. In Figure 12, the number of decoded neighboring vertices at vertex n+1 is n.

[0089] If the upper limit for the maximum number of adjacent vertices is set to 256, the mesh decoder 200 will require not only a huge amount of memory but also a huge amount of computation, as shown in Figure 10B. Therefore, the upper limit (maximum value) of the range in which the above-mentioned maximum number of adjacent vertices can be set may be set to 8.

[0090] Furthermore, the range that can be set as the maximum number of adjacent vertices in the aforementioned control signal may be a clearly defined value, or it may be calculated from other control signals or data.

[0091] For example, Level 1 may define the range that can be set as the maximum number of adjacent vertices in the control signal.

[0092] Alternatively, the upper limit of the range that can be set as the maximum number of adjacent vertices in the control signal may be calculated from the number of vertices of the basic mesh using the following equation (3).

[0093] The upper limit of the maximum number of adjacent vertices that can be set in the control signal = log2(number of vertices in the basic mesh) ... Equation (3) With this configuration, it is possible to appropriately determine the range within which the maximum number of adjacent vertices can be set, and even in the worst case, it can be expected that both the computational load and memory usage will be reliably reduced.

[0094] (Example 2 of modifications to the inter-decoding unit 202E) Below, with reference to Figure 13, we will explain Example 2 of the modification of the inter-decoding unit 202E.

[0095] The motion vector calculation unit 202E4 of the inter-decoding unit 202E has modes 1 and mode 0.

[0096] In mode 1, the motion vector calculation unit 202E4 adds the MVR generated by the motion vector residual decoding unit 202E1 and the MVP output from the motion vector prediction unit 202E3 to output the MV of the vertex to be decoded (see Figure 13A).

[0097] On the other hand, in mode 0, the motion vector calculation unit 202E4 outputs the MVR generated by the motion vector residual decoding unit 202E1 as the MV of the vertex to be decoded (see Figure 13B).

[0098] In mode 0, the operation of the motion vector calculation unit 202E4 corresponds to setting the MVP output from the motion vector prediction unit 202E3 to zero.

[0099] Furthermore, the motion vector calculation unit 202E4 may make the modes of the MVs of N (N≧1) consecutive vertices in the decoding order the same.

[0100] The motion vector calculation unit 202E4 groups the aforementioned N vertices into one group. The size N of such a group is 1 or greater. The motion vector calculation unit 202E4 decodes a control signal (group size shown in Figure 13) from the bitstream that allows it to calculate this group size.

[0101] However, if the number of vertices remaining in the last group is less than the group size, the motion vector calculation unit 202E4 will include all of the remaining vertices in the group.

[0102] By assigning the same mode to N consecutive vertices in this way, the code size of the mode can be reduced, which is expected to improve coding efficiency.

[0103] Here, the greater the number of consecutive vertices with the same mode, the greater the reduction in the code size of the mode. Therefore, it is necessary to set an appropriate group size in the mesh coding device 100 and decode it from the bitstream as a control signal in the mesh decoding device 200.

[0104] Therefore, it is desirable that the range of settings that can be configured in such a control signal is not smaller than the actual number of consecutive vertices that have the same mode.

[0105] For example, if all vertices are selected in a mode that is almost identical, you can set the group size to the total number of vertices.

[0106] Table 1 shows examples of cases where more than 80% of vertices have selected Mode 0, or where more than 90% of vertices have selected Mode 1.

[0107] Therefore, the control signal described above is designed to cover a range from 1 to a predetermined maximum value. This maximum value is set to be greater than or equal to the total number of vertices in the basic mesh.

[0108] [Table 3] Furthermore, if the aforementioned control signal (group size) is set to a natural number, setting it to a number greater than or equal to the total number of vertices will result in a large absolute value, thus increasing the sign quantity.

[0109] Therefore, it is also possible to make the above control signal logarithmic. Specifically, the control signal can be denoted as log2_group_size, and the group size can be calculated using the following equation (4).

[0110] group size=2log2_group_size … Equation (4) Here, if there is only one group in the frame, that group is made the last group. In other words, if the group size is greater than the number of vertices, all vertices are placed in the group.

[0111] Furthermore, the configurable range in the aforementioned control signal may be a clearly defined value, or it may be calculated from other control signals or data.

[0112] For example, Level 1 may define the range of settings that can be configured in such control signals.

[0113] Alternatively, the range that can be set in such a control signal may be calculated from the number of vertices in the basic mesh.

[0114] For example, the range of configurable values ​​in such a control signal may be the smallest natural number that is a power of 2 and can cover the number of vertices in the basic mesh.

[0115] Furthermore, the range of configurable parameters in the aforementioned control signal may be narrowed, and a predetermined flag (Mode flag) of another control signal may be introduced, as shown in Figure 14. In this case, as shown in Figure 14, if the predetermined flag is TRUE (Mode flag=1), the motion vector calculation unit 202E4 groups all vertices into one group (i.e., the group size is the total number of vertices), and if it is FALSE, it leaves the group size as calculated from the aforementioned control signal.

[0116] The aforementioned control signals may be set per sequence or per frame. If the aforementioned control signals are set per sequence, the group size of all frames will be the same.

[0117] With this configuration, by appropriately determining the range within which the group size can be set, it is possible to handle all situations, reliably reduce the amount of code in each mode, and is expected to improve coding efficiency.

[0118] (Example 1 of modifications to the basic mesh decoding unit 202) The following describes an example of modification 1 of the basic mesh decoding unit 202, with reference to Figures 15 and 16.

[0119] As shown in Figure 15, the basic mesh decoding unit 202 in this modified example 1 comprises a separation unit 202A, an intra decoding unit 202B, a mesh buffer unit 202C, an inter-decoding unit 202E, and a skip decoding unit 202F.

[0120] The skip decoding unit 202F is configured to decode the base mesh of the frame to be decoded using the base mesh of the specified reference frame as is.

[0121] In this embodiment, the frame may be either a mesh or a submesh.

[0122] For example, as shown in Figure 16, "P_SUBMESH" in smh_type may correspond to a P frame, "I_SUBMESH" in smh_type may correspond to an I frame, and "SKIP_SUBMESH" in smh_type may correspond to an S frame.

[0123] (Skip decoding unit 202F) The skip decoding unit 202F is configured to retrieve the decoded base mesh (reference decoded base mesh) of the specified reference frame from the mesh buffer unit 202C, and to decode the vertex coordinates and vertex indices of the base mesh of the frame to be decoded using the coordinates and vertex indices of the vertices of the retrieved reference decoded base mesh as is.

[0124] Here, the mesh buffer unit 202C has at least one reference frame and is configured to store at least one decoded basic mesh for each reference frame.

[0125] The skip decoding unit 202F may identify a designated reference decoding base mesh using a control signal decoded from the bitstream or a predetermined rule.

[0126] For example, such a predetermined rule may be to retrieve the first reference frame from the reference frame list from the mesh buffer unit 202C, or to retrieve the reference frame whose frame index is closest to the frame to be decoded.

[0127] In this embodiment, a frame in which the coordinates of the vertices of the base mesh are decoded using the coordinates of the vertices of the base mesh used for reference and the index of those vertices is referred to as an "S-frame".

[0128] With this configuration, the motion vector can be eliminated in the skip decoding unit 202F, which is expected to result in a significant reduction in both the amount of code and the amount of computation.

[0129] (Mesh buffer section 202C) The mesh buffer unit 202C is configured to store one or more reference decoding base meshes in a predetermined order.

[0130] Furthermore, such a basic mesh contains metadata such as frame numbers and sub-mesh numbers, as well as at least the coordinates of each vertex and the index of that vertex, and is stored in the mesh buffer unit 202C in a predetermined order determined by the reference frame list.

[0131] Here, as shown in Figure 17, the reference frame list (ref_list0) is a list of information that identifies all the reference decoded base meshes stored in the mesh buffer unit 202C.

[0132] The reference frame list may be determined by the control signals decoded from the bitstream, as shown in Figure 17, or it may be naturally calculated from the decoding order of the frames.

[0133] The control signal decoded from the bitstream may be expressed as a relative distance to the frame being decoded, or as an absolute value of the frame index.

[0134] Furthermore, control signals may be used to utilize short-term or long-term reference frames.

[0135] For example, using a short-term reference frame, the absolute value (abs_delta_mfoc_st) and its sign (sign_flag) of the difference in display order between the current frame (cur) and the reference frame (ref) can be decoded from the bitstream, and the display order of the reference frame can be specified by the following formula. If(sign_flag){ Display Order(ref)=Display Order(cur)+abs_delta_mfoc_st else{ Display Order(ref)=Display Order(cur)-abs_delta_mfoc_st } Furthermore, if a method is used that naturally calculates the order of frame decoding, for example, in the reference frame list, when no control signals are present, the frames may be arranged sequentially in fixed numbers starting from the most recently decoded frame. In other words, the reference frame list may be {0, -1, -2, ..., -(N-1)}.

[0136] Basically, the reference frame list does not change from frame to frame except in special circumstances (for example, when a re-ordering instruction is received).

[0137] The mesh buffer section 202C may be updated as follows.

[0138] When the basic mesh is decoded, the mesh buffer unit 202C, in the case of I-frames and P-frames, deletes one or more existing reference frames in a predetermined order determined by the reference frame list, inserts one or more basic meshes including the basic mesh of the decoded frame, or creates and inserts one basic mesh from multiple basic meshes, thereby adjusting the order of the reference frames.

[0139] Such deletion operations may be performed only when the mesh buffer unit 202C is full. The number of basic meshes that can be stored in the mesh buffer unit 202C is predetermined. In this embodiment, the mesh buffer unit 202C is defined as being full when the number of such basic meshes is reached. In the creation process described above, the coordinates of the vertices corresponding to the decoded frame's base mesh and the existing base mesh stored in the mesh buffer section 202C may be weighted and averaged to create a single base mesh.

[0140] The weights used in such a weighted average may be predetermined, calculated using frame indices, or decoded from control signals.

[0141] However, the mesh buffer unit 202C may or may not perform such an update when it is an S-frame.

[0142] Furthermore, when the mesh buffer unit 202C receives a control signal indicating a re-ordering instruction via a control signal decoded from the bitstream, it updates the reference frame list as shown in Figure 18 and adjusts the order of the reference frames according to a predetermined order determined by the updated reference frame list (ref_list0).

[0143] (Inter-decoding unit 202E) The inter-decoding unit 202E is configured to decode the vertex coordinates of the P-frame by adding the vertex coordinates of the reference frame extracted from the mesh buffer unit 202C with the motion vector decoded from the bitstream of the P-frame.

[0144] Furthermore, the inter-decoding unit 202E can adjust the vertex indices of the P-frame using pairs of vertex indices A(k) and B(k) that exist as duplicate vertices stored in the specific buffer. All or part of these indices are decoded from the bitstream. Such decoding method may be arithmetic coding. With this configuration, it can be expected that there will be no limit on the maximum value of the index to be decoded using arithmetic coding. For example, the arithmetic coding ue(v) may be used.

[0145] (Example 2 of modifications to the basic mesh decoding unit 202) Below, with reference to Figure 19, we will explain Example 2 of the modification of the basic mesh decoding unit 202.

[0146] The skip decoding unit 202F will be described below, but it may also be applied to the inter-decoding unit 202E.

[0147] As shown in Figure 19, in the skip decoding unit 202F, the decoding order and display order are different in order to allow subsequent frames to be referenced.

[0148] Here, the display order is the same as the input order during encoding and the same as the output order during decoding.

[0149] On the other hand, the decoding order is the same as the output order during encoding, and the same as the input order during decoding.

[0150] The reference frame may also be calculated by weighting and averaging subsequent frames with one or more other frames.

[0151] However, when referencing multiple frames, including subsequent frames, a new frame type (smh_type) is defined in Figure 14, called MR_SUBMESH (MR frame or B frame), and MR_SUBMESH is decoded from the bitstream.

[0152] Furthermore, such other frames may be the decoded frame immediately preceding the target frame, as shown in Figure 20.

[0153] Such weights may be calculated using the frame interval between the target frame and subsequent frames, and the frame interval between the target frame and other frames, or they may be predetermined.

[0154] The basic mesh decoding unit 202 decodes the control signal (smh_mesh_frm_order_cnt_lsb) from the bitstream and decodes the order of the output.

[0155] Furthermore, when a submesh exists as defined in Non-Patent Document 4 above, all submeshes will either be given the same control signal (smh_mesh_frm_order_cnt_lsb) or the control signal (smh_mesh_frm_order_cnt_lsb) will be applied to all submeshes.

[0156] The value indicated by such a control signal (smh_mesh_frm_order_cnt_lsb) may be the difference from the display order of the frames to be decoded, or it may be the order within a predetermined frame set MaxMeshFrmOrderCntLsb.

[0157] Furthermore, if the Decode Order and Display Order are different, and the decoded basic meshes are arranged in the Decode Order, the basic mesh decoding unit 202 may rearrange the decoded basic meshes to the Display Order.

[0158] In addition, in an S-frame that can reference subsequent frames, two mesh buffers 202C may be provided, or if only one mesh buffer 202 is provided, there may be at least one reference frame, including a reference frame whose display order is later than the frame to be decoded.

[0159] The skip decoding unit 202F specifies a reference frame by receiving a control signal, a predetermined rule, or a re-ordering instruction decoded from the bitstream.

[0160] Specifically, the skip decoding unit 202F uses this control signal to specify a reference frame in the reference frame list.

[0161] Alternatively, the skip decoding unit 202F specifies the first reference frame in the reference frame list.

[0162] Alternatively, the skip decoding unit 202F, upon receiving a Re-ordering instruction, updates the reference frame order of the reference frame list and the mesh buffer unit 202C, and specifies the first reference frame in the reference frame list.

[0163] In this embodiment, decoding an S-frame does not affect the decoding of other frames. Therefore, if some or all of the S-frames are not decoded, temporal scalability can be achieved.

[0164] Furthermore, the basic mesh decoding unit 202 may decode the basic mesh of the S frame by integrating multiple reference frames using a control signal.

[0165] For example, the basic mesh decoding unit 202 may be configured to average the coordinates of corresponding vertices in the basic meshes of the two preceding and succeeding reference frames, and then use the average coordinates and vertex indices as they are to decode the vertex coordinates and vertex indices of the basic mesh of the frame to be decoded.

[0166] With this configuration, it is possible to obtain a high-quality basic mesh without requiring motion vectors in the skip decoding unit 202F or the inter-decoding unit 202E, thus improving the quality of the decoded mesh. Furthermore, it is expected to enable temporal scalability.

[0167] However, in order to achieve temporal scalability, a control signal Temporal_ID is defined for each frame, indicating whether to decode the corresponding base mesh, displacement amount, and texture, and these are decoded from the bitstream.

[0168] Furthermore, for the same frame, the Tempora_ID of the control signal for the base mesh, displacement, and texture should be matched. Matching the Tempora_ID of the base mesh, displacement, and texture to be decoded is expected to have the effect of avoiding frames that cannot be decoded and unnecessary data.

[0169] Furthermore, it is desirable that the interval between adjacent frames with the same Tempora_ID remains constant.

[0170] Adjacent frames with the same Tempora_ID are closest to the POC.

[0171] As mentioned above, by keeping the frame interval constant, it is expected that a constant frame rate can be maintained when displaying the decoded frames.

[0172] Furthermore, the decoding order of the base mesh, displacement, and textures that have the same display order is matched.

[0173] By matching the decoding order in this way, it is expected that the mesh can be reconstructed without having to wait for each other to decode when decoding the base mesh, displacement, and texture.

[0174] If the decoding order of the base mesh, displacement, and texture is different, it becomes necessary to wait for the slowest-decoded component, which can lead to increased buffer usage and decoding delays.

[0175] Furthermore, frames with a Temporal_ID higher than the control signal Temporal_ID of the frame to be decoded will not be used as reference frames for that frame.

[0176] This is expected to have the effect of eliminating the possibility of reference frames being discarded.

[0177] The following describes an example of achieving temporal scalability using the aforementioned Temporal_ID.

[0178] The bitstreams of the base mesh, displacement, and texture are encapsulated by a Network Abstraction Layer (NAL) unit. The NAL unit may have a NAL header as shown in Figure 39.

[0179] The TID, defined as the last 3 bits in the NAL header, is Temporal_ID plus 1. The TID ranges from 1 to 7, and zero is prohibited.

[0180] Furthermore, if a submesh exists as defined in Non-Patent Document 4, all submeshes will be assigned the same TID, or the TID will be applied to all submeshes.

[0181] Since HEVC and VVC video encoding schemes are used for displacement and texture, only the basic mesh will be explained below.

[0182] If the NALType falls within the range of NAL_BLA_W_LP to NAL_RSV_BMCL_29 as defined in Non-Patent Document 4, that is, if it belongs to an IRAP-encoded basic mesh frame, then the Temporal_ID must be 0.

[0183] If NALType is equal to NAL_TSA_R or NAL_TSA_N, Temporal_ID must not be equal to 0.

[0184] If NALType is equal to 0, and NALType is equal to NAL_STSA_R or NAL_STSA_N, then Temporal_ID must not be equal to 0.

[0185] The value of Temporal_ID must be the same for all BMCL NAL units within the access unit.

[0186] The Temporal_ID value of the coded base mesh frame or access unit is the Temporal_ID value of the BMCL NAL unit of the coded base mesh frame or access unit.

[0187] The Temporal_ID value of a sublayer representation is the maximum value of the Temporal_IDs of all BMCL NAL units within that sublayer representation.

[0188] The Temporal_ID value for non-BMCL NAL units is restricted as follows: - If NALType is equal to NAL_BMSPS, then Temporal_ID must be 0, and the Temporal_ID of the access unit containing the NAL unit must be 0. - Otherwise, if NALType is equal to NAL_EOS or NAL_EOB, Temporal_ID must be 0. - Otherwise, if NALType is equal to NAL_AUD or NALLFDD, Temporal_ID must be equal to the Temporal_ID of the access unit containing the NALL unit. - Otherwise, the Temporal_ID must be greater than or equal to the Temporal_ID of the access unit containing the NAL unit.

[0189] If the NAL unit is not a BMCL, the Temporal_ID value will be equal to the minimum Temporal_ID value of all access units to which the non-BMCL NAL unit applies.

[0190] If NALType is equal to NAL_BMFPS, then Temporal_ID can be greater than or equal to the Temporal_ID of the included access unit, since the entire set of basic mesh frame parameters (BMFPS) is included at the beginning of the bitstream where the Temporal_ID of the first encoded basic mesh frame is 0.

[0191] <Subdivision section 203> The subdivision unit 203 is configured to generate and output additional subdivision vertices and their connection information from the basic mesh decoded by the basic mesh decoding unit 202, using the subdivision method indicated by the control information.

[0192] Here, the basic mesh, the added subdivided vertices, and their connection information are collectively referred to as the "subdivided mesh."

[0193] The subdivision unit 202 is configured to identify the type of subdivision method from the subdivision_method_id, which is control information generated by decoding the basic mesh bitstream.

[0194] The subdivided section 202 will be described below with reference to Figures 3A and 3B.

[0195] Figures 3A and 3B illustrate an example of the process of generating subdivided vertices from a basic mesh.

[0196] Figure 3A shows an example of a basic mesh consisting of five vertices.

[0197] Here, for subdivision, for example, a mid-edge subdivision method may be used, which connects the midpoints of each edge on each base face. In this way, a base face will be divided into four faces.

[0198] Figure 3B shows an example of a subdivided mesh obtained by dividing a basic mesh composed of five vertices. In the subdivided mesh shown in Figure 3B, eight subdivided vertices (white circles) are generated in addition to the original five vertices (black circles).

[0199] By decoding the displacement amount for each of the subdivided vertices generated in this way using the displacement amount decoding unit 206, an improvement in encoding performance can be expected.

[0200] Furthermore, different subdivision methods may be applied to each patch. This allows the displacement amount decoded by the displacement decoding unit 206 to be adaptively changed for each patch, which is expected to improve encoding performance. The information of the divided patches is received as control information, namely patch_id.

[0201] The subdivision section 203 will be described below with reference to Figure 21. Figure 21 is a diagram showing an example of the functional blocks of the subdivision section 203.

[0202] As shown in Figure 21, the subdivision section 203 has a basic mesh subdivision section 203A and a subdivision mesh adjustment section 203B.

[0203] (Basic mesh subdivision section 203A) The basic mesh subdivision unit 203A is configured to calculate the number of divisions (subdivision count) for each basic surface and basic patch based on the input basic mesh and basic mesh subdivision information, subdivide the basic mesh based on these division counts, and output the subdivided surfaces.

[0204] In other words, the basic mesh subdivision section 203A may be configured to allow the number of divisions to be changed on a basic surface and basic patch basis.

[0205] Here, a basic surface is a surface that makes up the basic mesh, and a basic patch is a collection of several basic cells.

[0206] Furthermore, the basic mesh subdivision unit 203A may be configured to predict the number of subdivisions of the basic surface and calculate the number of subdivisions of the basic surface by adding the predicted subdivision residual to the predicted number of subdivisions of the basic surface.

[0207] Furthermore, the basic mesh subdivision unit 203A may be configured to calculate the number of subdivisions of a basic surface based on the number of subdivisions of adjacent basic surfaces.

[0208] Furthermore, the basic mesh subdivision unit 203A may be configured to calculate the number of subdivisions of the basic surface based on the number of subdivisions of the basic surface accumulated immediately beforehand.

[0209] Furthermore, the basic mesh subdivision unit 203A may be configured to subdivide the basic face by generating vertices that divide the three edges constituting the basic face and connecting the generated vertices.

[0210] As shown in Figure 21, the basic mesh subdivision unit 203A is followed by a subdivision mesh adjustment unit 203B, which will be described later.

[0211] Below, an example of the processing of the basic mesh subdivision section 203A will be explained using Figures 22 to 24.

[0212] Figure 22 is a diagram showing an example of the functional blocks of the basic mesh subdivision unit 203A, and Figure 24 is a flowchart showing an example of the operation of the basic mesh subdivision unit 203A.

[0213] As shown in Figure 22, the basic mesh subdivision unit 203A includes a basic surface division buffer unit 203A1, a basic surface division reference unit 203A2, a basic surface division prediction unit 203A3, an addition unit 203A4, and a basic surface division unit 203A5.

[0214] The basic surface division buffer unit 203A1 stores basic surface division information, including the number of divisions of the basic surface, and is configured to output basic surface division information to the basic surface division reference unit 203A2.

[0215] Here, the size of the basic surface division buffer unit 203A1 may be set to 1, and it may be configured to output the number of basic surface divisions that was immediately accumulated to the basic surface division reference unit 203A2.

[0216] In other words, by setting the size of the basic face division buffer 203A1 to 1, it may be configured to refer only to the last decoded fine division number (the most recently decoded fine division number).

[0217] The basic face division number reference unit 203A2 is configured to output "reference unavailable" to the basic face division number prediction unit 203A3 if there are no adjacent basic faces to the basic face to be decoded, or if there are adjacent basic faces to the basic face to be decoded but the number of divisions has not been determined.

[0218] On the other hand, the basic face division number reference unit 203A2 is configured to output the division number to the basic face division number prediction unit 203A3 if there is an adjacent basic face to the basic face to be decoded and the division number is determined.

[0219] The basic surface division number prediction unit 203A3 is configured to predict the number of divisions (subdivisions) of a basic surface based on one or more input division numbers, and to output the predicted number of divisions (predicted divisions) to the addition unit 203A4.

[0220] Here, the basic surface division prediction unit 203A3 is configured to output 0 to the adder 203A4 if only "unable to reference" is input from the basic surface division reference unit 203A2.

[0221] Furthermore, the basic surface division number prediction unit 203A3 may be configured to generate a predicted number of divisions using one or more statistical values ​​such as the average, maximum, minimum, or mode of the input number of divisions when one or more division numbers are input.

[0222] The basic face division number prediction unit 203A3 may be configured to generate the number of divisions of the nearest adjacent face as the predicted number of divisions when one or more division numbers are input.

[0223] The addition unit 203A4 is configured to output to the basic surface division unit 203A5 the number of divisions obtained by adding the predicted division number residual decoded from the predicted residual bitstream and the predicted division number obtained from the basic surface division number prediction unit 203A3.

[0224] The basic surface division unit 203A5 is configured to subdivide the basic surface based on the number of divisions input from the addition unit 203A4.

[0225] Figure 23 shows an example of a case where the basic surface is divided into nine parts. Referring to Figure 23, the method of dividing the basic surface by the basic surface division section 203A5 will be explained.

[0226] The basic face division unit 203A5 generates points A_1, ..., A_(N-1) that divide the edge AB constituting the basic face into N equal parts (N=3).

[0227] Similarly, the basic face division section 203A5 also divides edges BC and CA into N equal parts, generating points B_1, ..., B_(N-1) and C_1, ..., C_(N-1), respectively.

[0228] Hereafter, points on sides AB, BC, and CA will be referred to as "edge division points."

[0229] The basic face division section 203A5 generates edges A_i B_(Ni), B_i C_(Ni), C_i A_(Ni) for all i (i=1,2,...,N-1), and N 2 Generates individual subdivision surfaces.

[0230] Next, with reference to Figure 24, the processing procedure for the basic mesh subdivision section 203A will be described.

[0231] In step S2201, the basic mesh subdivision unit 203A determines whether the subdivision process for the last basic surface has been completed. If the process is completed, it ends; otherwise, it proceeds to step S2202.

[0232] In step S2202, the basic mesh subdivision unit 203A determines whether Depth < mdu_max_depth.

[0233] Here, Depth is a variable representing the current depth, with an initial value of 0, and mdu_max_depth represents the maximum depth determined for each basic surface.

[0234] If the condition in step S2202 is satisfied, this processing procedure proceeds to step S2203; if the condition is not satisfied, this processing procedure returns to step S2201.

[0235] In step S2203, the basic mesh subdivision unit 203A determines whether mdu_subdivision_flag at the current depth is 1.

[0236] If Yes, this processing procedure returns to step S2201; if No, this processing procedure proceeds to step S2204.

[0237] In step S2204, the basic mesh subdivision unit 203A further subdivides all the subdivided surfaces within the basic surface.

[0238] Here, if the basic mesh subdivision unit 203A has never performed a subdivision process on the basic surface, it subdivides the basic surface.

[0239] Note that the subdivision method is the same as the method described in step S2204.

[0240] Specifically, if the basic surface has never been subdivided, it is subdivided as shown in FIG. 23 for the basic surface. If it has been subdivided at least once, the subdivided surface is N2 Divide it into individual parts. Taking Fig. 23 as an example, the surface composed of vertex A_2, vertex B, and vertex B_1 is further divided in the same way as when dividing the basic surface to generate N 2 surfaces.

[0241] When the subdivision process is completed, this processing procedure proceeds to step S2205.

[0242] In step S2205, the basic mesh subdivision unit 203A adds 1 to Depth, and this processing procedure returns to step S2202.

[0243] (Subdivision Mesh Adjustment Unit 203B) Next, a specific example of the processing performed by the subdivision mesh adjustment unit 203B will be described. Hereinafter, an example of the processing performed by the subdivision mesh adjustment unit 203B will be described using Figs. 25 to 28.

[0244] Fig. 25 is a diagram showing an example of the functional block of the subdivision mesh adjustment unit 203B.

[0245] As shown in Fig. 25, the subdivision mesh adjustment unit 203B has an edge division point movement unit 701 and a subdivision surface division unit 702.

[0246] (Edge Division Point Movement Unit 701) The edge division point movement unit 701 is configured to move the edge division point of the basic surface to either the edge division point of the adjacent basic surface for the input initial subdivision surface and output the subdivision surface.

[0247] Fig. 26 is an example of moving the edge division point on the basic surface ABC. For example, as shown in Fig. 26, the edge division point movement unit 701 may be configured to move the edge division point of the basic surface ABC to the edge division point of the nearest adjacent basic surface.

[0248] (Subdivision Surface Division Unit 702) The subdivision surface division unit 702 is configured to further divide the input subdivision surface and output the decoded subdivision surface.

[0249] Figure 27 shows an example of a case where the subdivision plane X within the basic plane is further subdivided.

[0250] As shown in Figure 27, the subdivision surface division section 702 may be configured to generate a new subdivision surface within the base surface by connecting the vertices constituting the subdivision surface with the edge division points of the adjacent base surface.

[0251] Figure 28 shows an example of a case where the above subdivision process is performed on all subdivision surfaces.

[0252] The mesh decoding unit 204 is configured to generate and output a decoded mesh using the subdivided mesh generated by the subdivided mesh unit 203 and the displacement amount decoded by the displacement amount decoding unit 206.

[0253] Specifically, the mesh decoding unit 204 is configured to generate a decoded mesh by adding a corresponding displacement amount to each subdivided vertex. Here, information on which subdivided vertex each displacement amount corresponds to is indicated by control information.

[0254] The patch integration unit 205 is configured to integrate and output the decoded mesh generated by the mesh decoding unit 206 for multiple patches.

[0255] Here, the patch division method is defined by the mesh coding device 100. For example, the patch division method may be configured to calculate a normal vector for each base face, select the base face with the most similar normal vector among adjacent base faces, combine both base faces into the same patch, and then repeat this procedure sequentially for the next base face.

[0256] The video decoding unit 207 is configured to decode and output a texture by video encoding. For example, the video decoding unit 207 may use HEVC.

[0257] <Displacement amount decoding unit 206> The displacement amount decoding unit 206 is configured to decode the displacement amount bitstream, generate a displacement amount, and output it.

[0258] Figure 3B shows an example of the displacement amount for a certain subdivision vertex. In the example in Figure 3B, there are eight subdivision vertices, so the displacement amount decoding unit 206 is configured to define eight displacement amounts, each expressed as a scalar or vector, for each subdivision vertex.

[0259] The displacement decoding unit 206 will be described below with reference to Figure 29. Figure 29 is a diagram showing an example of the functional blocks of the displacement decoding unit 206.

[0260] As shown in Figure 29, the displacement decoding unit 206 includes a decoding unit 206A, an inverse quantization unit 206B, an inverse wavelet transform unit 206C, an adder 206D, an interpretation unit 206E, and a frame buffer 206F.

[0261] The decoding unit 206A is configured to decode and output level values ​​and control information by performing variable-length decoding on the received displacement bitstream. Here, the level values ​​obtained by variable-length decoding are output to the inverse quantization unit 206B, and the control information is output to the interpretation unit 206E.

[0262] The following describes an example of the configuration of a displacement bitstream, referring to Figure 30. Figure 30 is a diagram showing an example of the configuration of a displacement bitstream.

[0263] As shown in Figure 30, firstly, the displacement bitstream may include a Displacement Parameter Set (DPS), which is a set of control information related to the decoding of the displacement.

[0264] Second, the displacement bit stream may include a DPH (Displacement Patch Header), which is a set of control information corresponding to a patch.

[0265] Third, the displacement bit stream may include, after the DPH, the encoded displacements that constitute the patch.

[0266] As described above, the displacement bit stream is configured such that one DPH and one DPS respectively correspond to each encoded displacement.

[0267] Note that the configuration in FIG. 30 is merely an example. As long as the configuration is such that one DPH and one DPS respectively correspond to each encoded displacement, elements other than those described above may be added as components of the displacement bit stream.

[0268] For example, as shown in FIG. 30, the displacement bit stream may include an SPS (Sequence Parameter Set).

[0269] FIG. 31 is a diagram showing an example of the syntax configuration of the DPS.

[0270] In FIG. 31, the Descriptor field indicates how each syntax is encoded.

[0271] Also, in FIG. 31, ue(v) means that it is an unsigned zero-order exponential Golomb code, and u(n) means that it is an n-bit flag.

[0272] When there are multiple DPSs, the DPS includes at least DPS id information (dps_displacement_parameter_set_id) for identifying each DPS.

[0273] Additionally, DPS may include a flag (interprediction_enabled_flag) that controls whether or not interprediction is performed.

[0274] For example, you could define that interprediction is disabled when `interprediction_enabled_flag` is 0, and enabled when `interprediction_enabled_flag` is 1. Alternatively, you could define that interprediction is disabled when `interprediction_enabled_flag` is not included.

[0275] DPS may include a flag (dct_enabled_flag) that controls whether or not to perform reverse DCT.

[0276] For example, you could define that when dct_enabled_flag is 0, reverse DCT will not be performed, and when dct_enabled_flag is 1, reverse DCT will be performed. Alternatively, you could define that when dct_enabled_flag is not included, reverse DCT will not be performed.

[0277] Figure 32 shows an example of the syntax configuration of DPH.

[0278] As shown in Figure 32, the DPH includes at least DPS ID information to specify the DPS corresponding to each DPH.

[0279] The inverse quantization unit 206B is configured to generate and output conversion coefficients by inverse quantizing the level values ​​decoded by the decoding unit 206A.

[0280] The inverse wavelet transform unit 206C is configured to generate and output predicted residuals by applying an inverse wavelet transform to the transformation coefficients generated by the inverse quantization unit 206B.

[0281] (Interpretation Unit 206E) The interpretation unit 206E is configured to generate and output a predicted displacement amount by performing interpretation prediction using the decoded displacement amount of the reference frame read from the frame buffer 206F.

[0282] The interprediction unit 206E is configured to perform such interprediction only when the interprediction_enabled_flag is set to 1.

[0283] The interpretation unit 206E may perform interpretation in the spatial domain or in the frequency domain. Interpretation may also be performed bidirectionally using a reference frame from the past and a reference frame from the future.

[0284] Figure 33 illustrates an example of the correspondence between subdivided vertices between a reference frame and a frame to be decoded when interpretation is performed in the spatial domain.

[0285] Figure 34 shows an example of the functional block of the interpretation unit 206E when interpretation is performed in the frequency domain.

[0286] When performing interpretation in the frequency domain, the interpretation prediction unit 206E may determine the predicted wavelet transform coefficients for the frequency in the frame to be decoded by directly referencing the decoded wavelet transform coefficients for the corresponding frequency in the reference frame.

[0287] The interpretation unit 206E may also perform interpretation probabilistically according to a normal distribution whose mean and variance are estimated using the decoded displacement amount or decoded wavelet transform coefficient of the subdivided vertices in multiple reference frames.

[0288] The interpretation unit 206E may also perform interpretation based on a regression curve estimated with time as the explanatory variable and displacement as the dependent variable, using the decoded displacement amount or decoded wavelet transform coefficient of the subdivided vertices in multiple reference frames.

[0289] The interpretation unit 206E may be configured to perform bidirectional interpretation using a reference frame from the past and a reference frame from the future.

[0290] In the mesh coding device 100, the order of the decoded wavelet transform coefficients may be rearranged for each frame in order to improve coding efficiency.

[0291] The frequency correspondence between the reference frame and the frame to be decoded is indicated by the control information.

[0292] Figure 35 illustrates an example of the frequency correspondence between a reference frame and a decoded frame when interpretation is performed in the frequency domain.

[0293] Furthermore, if the subdivision unit 203 divides the basic mesh into multiple patches, the interpretation unit 206E is also configured to perform interpretation for each divided patch. This increases the time correlation between frames, which is expected to improve encoding performance.

[0294] The adder 206D receives the predicted residual from the inverse wavelet transform unit 206C and the predicted displacement from the interpretation unit 206E.

[0295] The adder 206D is configured to calculate and output the decoded displacement by adding the predicted residual and the predicted displacement.

[0296] The decoded displacement calculated by adder 206D is also output to frame buffer 206F.

[0297] The frame buffer 206F is configured to acquire and store the decoded displacement amount from the adder 206D.

[0298] Here, the frame buffer 206F outputs the decoded displacement amount at the corresponding vertex in the reference frame, according to control information (not shown).

[0299] Figure 36 is a flowchart showing an example of the operation of the displacement decoding unit 206.

[0300] As shown in Figure 36, in step S3501, the displacement decoding unit 206 determines whether or not this process has been completed for all patches.

[0301] If the answer is Yes, this operation ends; if the answer is No, this operation proceeds to step S3502.

[0302] In step S3502, the displacement decoding unit 206 performs inverse DCT on the patch to be decoded, followed by inverse quantization and inverse wavelet transform.

[0303] In step S3503, the displacement decoding unit 206 determines whether or not the interpretation_enabled_flag is 1.

[0304] If the answer is Yes, the process proceeds to step S3504; if the answer is No, the process returns to step S3501.

[0305] In step S3504, the displacement decoding unit 206 performs the above-described interpretation and addition.

[0306] <Example 1> Hereinafter, with reference to Figure 37, Modification 1 of the first embodiment described above will be explained, focusing on the differences from the first embodiment described above.

[0307] Figure 37 shows an example of the functional block of the displacement decoding unit 206 according to this modified example 1.

[0308] As shown in Figure 37, the displacement decoding unit 206 according to this modified example 1 includes an inverse DCT unit 206G after the decoding unit 206A, that is, between the decoding unit 206A and the inverse quantization unit 206B.

[0309] In other words, in this modified example 1, the inverse quantization unit 206B is configured to generate a predicted residual by applying an inverse wavelet transform to the level value output from the inverse DCT unit 202G.

[0310] <Modification 2> Hereinafter, with reference to Figure 38, a modified example 2 of the first embodiment described above will be explained, focusing on the differences from the first embodiment described above.

[0311] As shown in Figure 38, the displacement decoding unit 206 according to this modified example 2 includes an image decoding unit 2061, an image unfolding unit 2062, an inverse quantization unit 2063, and an inverse wavelet transform unit 2064.

[0312] The video decoding unit 2061 is configured to output video by decoding the received displacement bitstream using video encoding.

[0313] For example, the video decoding unit 2061 may use HEVC as described in Non-Patent Document 1.

[0314] Furthermore, the video decoding unit 2061 may use a video encoding scheme in which the motion vector is always zero. For example, the video decoding unit 2061 may always set the HEVC motion vector to zero and always use interpretation at the same position.

[0315] Furthermore, the video decoding unit 2061 may use a video encoding scheme that always skips conversion. For example, the video decoding unit 2061 may always use a conversion skip mode for HEVC conversion and use a video encoding scheme without conversion.

[0316] The image unpacking unit 2062 is configured to unpack the video decoded by the video decoding unit 2061 as level values ​​for each image (frame) and output them.

[0317] In this unfolding method, the image unfolding unit 2062 can be identified by working backward from the arrangement of level values ​​in the image indicated by the control information.

[0318] The image unfolding unit 2062 may, for example, arrange the level values ​​in the image in the order of raster operation, from high-frequency components to low-frequency components.

[0319] The inverse quantization unit 2063 is configured to generate and output conversion coefficients by inverse quantizing the level values ​​generated by the image unfolding unit 2062.

[0320] The inverse wavelet transform unit 2064 is configured to generate and output a decoded displacement by applying an inverse wavelet transform to the transformation coefficients generated by the inverse quantization unit 2063.

[0321] The mesh coding device 100 and mesh decoding device 200 described above may be implemented as programs that cause a computer to execute each function (each process). [Industrial applicability]

[0322] Furthermore, according to this embodiment, for example, it is possible to achieve an overall improvement in service quality in video communication, thereby contributing to Goal 9 of the United Nations-led Sustainable Development Goals (SDGs), "Build resilient infrastructure, promote sustainable industrialization and foster innovation." [Explanation of symbols]

[0323] 1…Mesh processing system 100... Mesh coding device 200... Mesh decoding section 201…Demultiplexer 202...Basic Mesh Decoding Unit 202A…Separation part 202B...Intra Decoding Unit 202B1... Optional intra-decoding unit 202B2…Alignment section 202C...Mesh buffer section 202D...Connection Information Decoding Unit 202E...Inter-decoding section 202E1... Motion vector decoding unit 202E2... Motion vector buffer section 202E3... Motion vector prediction unit 202E4... Motion vector calculation unit 202E5…Adder 202F...Skip Decoding Unit 203…Subdivision part 203A...Basic mesh subdivision section 203A1...Basic surface division buffer section 203A2…Basic surface division number reference section 203A3...Basic surface division number prediction unit 203A4... Addition section 203A5…Basic surface division part 203B... Subdivision Mesh Adjustment Section 701... Edge division point movement section 702…Subdivision surface division part 204...Mesh decoding section 205... Patch Integration Department 206...Displacement Decoding Unit 206A...Decoding section 206B, 2063...Dequantization section 206C, 2064... Inverse wavelet transform section 206D... Adder 206E...Interface prediction unit 206F... Frame buffer 206G…Reverse DCT section 2062…Image Expansion Section 207, 2061…Video decoding unit

Claims

1. A mesh decoding device, The system includes an inter-decoding unit that decodes the coordinates of the vertices of the P-frame by adding the coordinates of the vertices of the reference frame and the motion vector decoded from the bitstream of the P-frame. The aforementioned inter-decoding unit is A motion vector residual decoding unit that generates motion vector residuals from the interframe bitstream, A motion vector prediction unit predicts decoded motion vectors from the vertex to be decoded and the surrounding vertices connected to the vertex to be decoded, calculates a motion vector prediction value using all or part of the decoded motion vectors of the vertices surrounding the vertex to be decoded so as not to exceed the maximum number of uses, and decodes a control signal from the bitstream that can calculate the maximum number of adjacent vertices. The system includes a motion vector calculation unit that outputs the motion vector of the vertex to be decoded, In mode 1, the motion vector calculation unit adds the motion vector residual and the motion vector prediction value to output the motion vector of the vertex to be decoded. In mode 0, the mesh decoding device is characterized in that the motion vector calculation unit outputs the motion vector residual as the motion vector of the vertex to be decoded.

2. The mesh decoding device according to claim 1, characterized in that the motion vector prediction unit decodes the value of the maximum number of adjacent vertices so that it is less than or equal to a predetermined maximum value.

3. The mesh decoding device according to claim 2, characterized in that the maximum value is not greater than a constant ratio of the average number of adjacent vertices statistically or a natural number that can cover up to N bits.

4. The mesh decoding device according to claim 2, characterized in that the maximum value is 8 or 3 bits.

5. The mesh decoding device according to claim 1, characterized in that the range that can be set as the maximum number of adjacent vertices is calculated from other control signals or Level 1 or the number of vertices of the basic mesh.

6. A mesh decoding method, The process includes a step of decoding the coordinates of the vertices of the P-frame by adding the coordinates of the vertices of the reference frame and the motion vector decoded from the bitstream of the P-frame, The above process is, Step A involves generating motion vector residuals from the interframe bitstream, Step B involves decoding a control signal from a bitstream that can predict the decoded motion vector from the vertex to be decoded and the surrounding vertices connected to the vertex to be decoded, calculate a motion vector prediction value using all or part of the decoded motion vectors of the vertices surrounding the vertex to be decoded so as not to exceed the maximum number of uses, and calculate the maximum number of adjacent vertices. The process includes step C, which outputs the motion vector of the vertex to be decoded, In step C, In mode 1, the motion vector residual and the motion vector prediction value are added together to output the motion vector of the vertex to be decoded. In mode 0, the mesh decoding method is characterized by outputting the motion vector residual as the motion vector of the vertex to be decoded.

7. A program that makes a computer function as a mesh decoding device, The mesh decoding device includes an inter-decoding unit that decodes the coordinates of the vertices of the P-frame by adding the coordinates of the vertices of the reference frame and the motion vector decoded from the bitstream of the P-frame. The aforementioned inter-decoding unit is A motion vector residual decoding unit that generates motion vector residuals from the interframe bitstream, A motion vector prediction unit predicts decoded motion vectors from the vertex to be decoded and the surrounding vertices connected to the vertex to be decoded, calculates a motion vector prediction value using all or part of the decoded motion vectors of the vertices surrounding the vertex to be decoded so as not to exceed the maximum number of uses, and decodes a control signal from the bitstream that can calculate the maximum number of adjacent vertices. The system includes a motion vector calculation unit that outputs the motion vector of the vertex to be decoded, In mode 1, the motion vector calculation unit adds the motion vector residual and the motion vector prediction value to output the motion vector of the vertex to be decoded. In mode 0, the motion vector calculation unit is characterized by outputting the motion vector residual as the motion vector of the vertex to be decoded.

Images