Mesh Decoder, Mesh Encoder, Mesh Decoding Method, and Program

Abstract

To improve the efficiency of coding a mesh.SOLUTION: A displacement amount decoding unit 206 of a mesh decoder 200 of the present invention includes: a decoding unit 206A for decoding a level value from a displacement amount bit stream; an inverse quantization unit 206B for inverse-quantizing the level value and generating a conversion coefficient; a reverse-wavelet transforming unit 206C for generating a prediction residual by performing a reverse-wavelet transform on the conversion coefficient; and an adding unit 206D for adding the prediction residual and the prediction displacement amount and calculating the displacement amount.SELECTED DRAWING: Figure 28

Description

Technical Field

[0001] The present invention relates to a mesh decoding device, a mesh encoding device, a mesh decoding method, and a program.

Background Art

[0002] Non-Patent Document 1 discloses a technique for encoding a mesh using Non-Patent Document 2.

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Non-Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the prior art, in order to reversibly encode the coordinates and connection information of all vertices constituting a dynamic mesh, there is a problem that the amount of information cannot be reduced even under conditions where loss is allowed, and the encoding efficiency is low. Therefore, the present invention has been made in view of the above problems, and an object thereof is to provide a mesh decoding device, a mesh encoding device, a mesh decoding method, and a program capable of improving the encoding efficiency of a mesh.

Means for Solving the Problems

[0005] The first feature of the present invention is a mesh decoding device, which includes a displacement amount decoding unit configured to decode a displacement amount bit stream to generate and output a displacement amount. The displacement amount decoding unit includes a decoding unit configured to decode a level value from the displacement amount bit stream, an inverse quantization unit configured to generate a conversion coefficient by inverse quantizing the level value, an inverse wavelet transform unit configured to generate a prediction residual by performing an inverse wavelet transform on the conversion coefficient, and an adder configured to calculate the displacement amount by adding a predicted displacement amount and the prediction residual. The gist of the invention is to include these components.

[0006] The second feature of the present invention is a mesh decoding method, which includes a step of decoding a level value from a displacement amount bit stream, a step of generating a conversion coefficient by inverse quantizing the level value, a step of generating a prediction residual by performing an inverse wavelet transform on the conversion coefficient, and a step of calculating the displacement amount by adding a predicted displacement amount and the prediction residual. The gist of the invention is to have these steps.

[0007] The third feature of the present invention is a program for causing a computer to function as a mesh decoding device. The mesh decoding device includes a displacement amount decoding unit configured to decode a displacement amount bit stream to generate and output a displacement amount. The displacement amount decoding unit includes a decoding unit configured to decode a level value from the displacement amount bit stream, an inverse quantization unit configured to generate a conversion coefficient by inverse quantizing the level value, an inverse wavelet transform unit configured to generate a prediction residual by performing an inverse wavelet transform on the conversion coefficient, and an adder configured to calculate the displacement amount by adding a predicted displacement amount and the prediction residual. The gist of the invention is to include these components.

Advantages of the Invention

[0008] According to the present invention, it is possible to provide a mesh decoding device, a mesh encoding device, a mesh decoding method, and a program that can improve the encoding efficiency of a mesh.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3A

Figure 3B

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Figure 31

Figure 32

Figure 33

Figure 34

Figure 35

Figure 36

Figure 37

Figure 38

Figure 39

DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the components in the following embodiments can be replaced with existing components as appropriate, and various variations including combinations with other existing components are possible. Therefore, the description of the following embodiments does not limit the content of the invention described in the claims.

[0011] <First Embodiment> Hereinafter, with reference to FIGS. 1 to 35, the mesh processing system according to this embodiment will be described.

[0012] FIG. 1 is a diagram showing an example of the configuration of a mesh processing system 1 according to this embodiment. As shown in FIG. 1, the mesh processing system 1 includes a mesh encoding device 100 and a mesh decoding device 200.

[0013] FIG. 2 is a diagram showing an example of the functional blocks of the mesh decoding device 200 according to this embodiment.

[0014] As shown in FIG. 2, the mesh decoding device 200 includes a multiplex separation unit 201, a basic mesh decoding unit 202, a sub-division unit 203, a mesh decoding unit 204, a patch integration unit 205, a displacement amount decoding unit 206, and a video decoding unit 207.

[0015] Here, the basic mesh decoding unit 202, the sub-division unit 203, the mesh decoding unit 204, and the displacement amount decoding unit 206 are configured to perform processing in units of patches obtained by dividing a mesh, and thereafter, the processing results thereof may be configured to be integrated by the patch integration unit 205.

[0016] In the example of FIG. 3A, the mesh is divided into patch 1 composed of basic surfaces 1 and 2 and patch 2 composed of basic surfaces 3 and 4.

[0017] The multiplex separation unit 201 is configured to separate the multiplexed bit stream into a basic mesh bit stream, a displacement amount bit stream, and a texture bit stream.

[0018] <Basic mesh decoding unit 202> The basic mesh decoding unit 202 is configured to decode the basic mesh bit stream and generate and output a basic mesh.

[0019] Here, the basic mesh is composed of a plurality of vertices in a three-dimensional space and edges connecting such a plurality of vertices.

[0020] Note that, as shown in FIG. 3A, the basic mesh is configured by combining basic planes represented by three vertices.

[0021] The basic mesh decoding unit 202 may be configured to decode the basic mesh bit stream using, for example, Draco shown in Non-Patent Document 2.

[0022] Also, the basic mesh decoding unit 202 may be configured to generate "subdivision_method_id" described later as control information for controlling the type of the subdivision method.

[0023] Hereinafter, with reference to FIGS. 4 to 5, the control information decoded by the basic mesh decoding unit 202 will be described.

[0024] FIG. 4 is a diagram showing an example of the syntax configuration of the basic mesh bit stream.

[0025] As shown in FIG. 4, first, the basic mesh bit stream may include a BPH (Base Patch header), which is a set of control information corresponding to a basic mesh patch. Second, the basic mesh bit stream may include, after the BPH, basic mesh patch data obtained by encoding the basic mesh patch.

[0026] As described above, the basic mesh bit stream is configured such that one BPH corresponds to each patch data. Note that the configuration in FIG. 4 is merely an example, and other elements than those described above may be added as components of the basic mesh bit stream as long as the configuration is such that one BPH corresponds to each patch data.

[0027] For example, as shown in FIG. 4, the basic mesh bit stream may include an SPS (Sequence Parameter Set), or may include an FH (Frame Header), which is a set of control information corresponding to a frame, or may include an MH (Mesh Header), which is control information corresponding to a mesh.

[0028] FIG. 5 is a diagram showing an example of the syntax configuration of the BPH. Here, as long as the functions of the syntax are the same, different syntax names may be used even if they are different from the syntax shown in FIG. 5.

[0029] In the syntax configuration of the BPH shown in FIG. 5, the Description column indicates how each syntax is encoded. Also, ue(v) means that it is an unsigned 0th order exponential Golomb code, and u(n) means that it is an n-bit flag.

[0030] The BPH includes at least a control signal (mdu_face_count_minus1) that specifies the number of basic faces included in the basic mesh patch.

[0031] Also, the BPH includes at least a control signal (mdu_subdivision_method_id) that specifies the type of basic mesh subdivision method for each basic patch.

[0032] In addition, BPH may include a control signal (mdu_subdivision_num_method_id) that specifies the type of the subdivision number generation method for each basic mesh patch. For example, when mdu_subdivision_num_method_id = 0, it may be defined that the subdivision number of the basic surface is generated based on the predicted division residual, and when mdu_subdivision_num_method_id = 1, it may be defined that the subdivision number of the basic surface is generated recursively.

[0033] When BPH generates the subdivision number of the basic surface based on the predicted division residual, it may include a control signal (mdu_subdivision_resuiduals) that specifies the predicted division residual of the basic surface for each index i (i = 0, …, mdu_face_count_minus1).

[0034] When BPH generates the subdivision number of the basic surface recursively, it may include a control signal (mdu_max_depth) for identifying the upper limit of the number of recursive subdivisions for each basic mesh patch.

[0035] BPH may include a control signal (mdu_subdivision_flag) that specifies whether to recursively subdivide the basic surface for each index i (i = 0, …, mdu_face_count_minus1) and j (j = 0, …, mdu_subdivision_depth_index).

[0036] As shown in FIG. 6, the basic mesh decoding unit 202 includes a separation unit 202A, an intra decoding unit 202B, a mesh buffer unit 202C, a connection information decoding unit 202D, and an inter decoding unit 202E.

[0037] The separation unit 202A is configured to classify the basic mesh bit stream into the bit stream of the I frame (reference frame) and the bit stream of the P frame.

[0038] (Intra decoding unit 202B) The intra decoder 202B is configured to decode the coordinates and connection information of the vertices of the I-frame from the bit stream of the I-frame, for example, using Draco shown in Non-Patent Document 2.

[0039] FIG. 39 is a diagram showing an example of the functional blocks of the intra decoder 202B.

[0040] As shown in FIG. 39, the intra decoder 202B includes a separation unit 202A, an arbitrary intra decoder 202B1, and an alignment unit 202B2.

[0041] The arbitrary intra decoder 202B1 is configured to decode the coordinates and connection information of the unordered vertices of the I-frame from the bit stream of the I-frame using an arbitrary method including Draco shown in Non-Patent Document 2.

[0042] The alignment unit 202B2 is configured to output vertices by rearranging the unordered vertices in a predetermined order.

[0043] As the predetermined order, for example, the Morton code order or the raster scan order may be used.

[0044] Also, a plurality of vertices with matching coordinates, that is, duplicate vertices, may be grouped together as a single vertex and then rearranged in a predetermined order.

[0045] The mesh buffer unit 202C is configured to store the coordinates and connection information of the vertices of the I-frame decoded by the intra decoder 202B.

[0046] The connection information decoder 202D is configured to convert the connection information of the I-frame retrieved from the mesh buffer unit 202C into the connection information of the P-frame.

[0047] The inter-decoding unit 202E is configured to decode the vertex coordinates of the P-frame by adding the vertex coordinates of the I-frame retrieved from the mesh buffer unit 202C and the motion vectors decoded from the bit stream of the P-frame.

[0048] In the present embodiment, as shown in FIG. 7, there is a correspondence relationship between the vertices of the basic mesh of the P-frame and the vertices of the basic mesh of the I-frame (reference frame). Here, the motion vectors decoded by the inter-decoding unit 202E are the difference vectors between the vertex coordinates of the basic mesh of the P-frame and the vertex coordinates of the basic mesh of the I-frame.

[0049] (Inter-decoding unit 202E) FIG. 8 is a diagram showing an example of the functional block of the inter-decoding unit 202E.

[0050] As shown in FIG. 8, the inter-decoding unit 202E includes a motion vector residual decoding unit 202E1, a motion vector buffer unit 202E2, a motion vector prediction unit 202E3, a motion vector calculation unit 202E4, and an adder 202E5.

[0051] The motion vector residual decoding unit 202E1 is configured to generate an MVR (Motion Vector Residual) from the bit stream of the P-frame.

[0052] Here, the MVR is a motion vector residual indicating the difference between the MV (Motion Vector) and the MVP (Motion Vector Prediction). The MV is a difference vector (motion vector) between the vertex coordinates of the corresponding I-frame and the vertex coordinates of the P-frame. The MVP is a predicted value of the MV of the target vertex using the MV (predicted value of the motion vector).

[0053] The motion vector buffer unit 202E2 is configured to sequentially store the MVs output by the motion vector calculation unit 202E4.

[0054] The motion vector prediction unit 202E3 is configured to obtain the decoded MV from the motion vector buffer unit 202E2 for the vertices connected to the vertex to be decoded, and output the MVP of the vertex to be decoded using all or part of the obtained decoded MV as shown in FIG. 9.

[0055] The motion vector calculation unit 202E4 is configured to add the MVR generated by the motion vector residual decoding unit 202E1 and the MVP output from the motion vector prediction unit 202E3, and output the MV of the vertex to be decoded.

[0056] The adder 202E5 is configured to add the coordinates of the vertex corresponding to the vertex to be decoded obtained from the decoded basic mesh of the corresponding I frame (reference frame) and the motion vector MV output from the motion vector calculation unit 202E3, and output the coordinates of the vertex to be decoded.

[0057] Hereinafter, the details of each part of the inter-decoding unit 202E will be described.

[0058] FIG. 10 shows a flowchart illustrating an example of the operation of the motion vector prediction unit 202E3.

[0059] As shown in FIG. 10, in step S1001, the motion vector prediction unit 202E3 sets 0 for MVP and N.

[0060] In step S1002, the motion vector prediction unit 202E3 obtains a set of MVs of the vertices around the vertex to be decoded from the motion vector buffer unit 202E2, identifies the vertices for which the subsequent processing has not ended, transitions to No, and transitions to Yes when the subsequent processing has ended for all vertices.

[0061] In step S1003, the motion vector prediction unit 202E3 transitions to No if the MV of the vertex to be processed is not decoded, and transitions to Yes if the MV of the vertex to be processed is decoded.

[0062] In step S1004, the motion vector prediction unit 202E3 adds the MV to the MVP and adds 1 to N.

[0063] In step S1005, if N is greater than 0, the motion vector prediction unit 202E3 outputs the result of dividing the MVP by N, and if N is 0, it outputs 0 and ends the process.

[0064] That is, the motion vector prediction unit 202E3 is configured to output the MVP to be decoded by averaging the decoded motion vectors of the vertices around the vertex to be decoded.

[0065] Note that the motion vector prediction unit 202E3 may be configured to set the MVP to 0 when the set of such decoded motion vectors is an empty set.

[0066] The motion vector calculation unit 202E4 may be configured to calculate the MV of the vertex to be decoded from the MVP output by the motion vector prediction unit 202E3 and the MVR generated by the motion vector residual decoding unit 202E1 according to formula (1).

[0067] MV(k)=MVP(k)+MVR(k) … (1) Here, k is the index of the vertex. MV, MVR, and MVP are vectors having x, y, and z components.

[0068] According to such a configuration, since only the MVR is encoded instead of the MV using the MVP, an effect of improving the encoding efficiency can be expected.

[0069] The adder 202E5 calculates the coordinates of such a vertex by adding the MV of the vertex calculated by the motion vector calculation unit 202E4 and the coordinates of the vertex of the reference frame corresponding to such a vertex, and is configured to keep the connection information (Connectivity) as the reference frame.

[0070] Specifically, the adder 202E5 may be configured to calculate the coordinates v’ i (k) of the k-th vertex using Equation (2).

[0071] v’ i (k) = v’ j (k) + MV(k) … (2) Here, v’ i (k) is the coordinate of the k-th vertex to be decoded in the frame to be decoded, v’ j (k) is the coordinate of the decoded k-th vertex of the reference frame, MV(k) is the k-th MV of the frame to be decoded, and k = 1, 2, …, K.

[0072] Also, the connection information of the frame to be decoded is made the same as the connection information of the reference frame.

[0073] Note that since the motion vector prediction unit 202E3 calculates the MVP using the decoded MV, the decoding order affects the MVP.

[0074] Such a decoding order is set to the decoding order of the vertices of the basic mesh of the reference frame. Generally, if a decoding method that increases the basic surface one by one from the starting edge using a certain repeating pattern is used, the order of the vertices of the decoded basic mesh is determined during the decoding process.

[0075] For example, the motion vector prediction unit 202E3 may determine the decoding order of the vertices using Edgebreaker in the basic mesh of the reference frame.

[0076] According to such a configuration, since the MV from the reference frame is encoded instead of the coordinates of the vertices, an effect of improving the encoding efficiency can be expected.

[0077] (Modification Example 1 of the Inter-Decoder 202E) The MVP calculated by the flowchart shown in FIG. 10 is calculated by the simple average of the decoded surrounding MVs, but it may be calculated by the weighted average.

[0078] That is, the motion vector prediction unit 202E3 may be configured to output a predicted value of the motion vector to be decoded by weighted-averaging the decoded motion vectors of the vertices around the vertex to be decoded with weights according to the distances between the vertices of the reference frames corresponding to the vertex to be decoded and the vertices around the vertex to be decoded.

[0079] Note that the motion vector prediction unit 202E3 may be configured to output a predicted value of the motion vector to be decoded by weighted-averaging a part of the decoded motion vectors of the vertices around the vertex to be decoded with weights according to the distances between the vertices of the reference frames corresponding to the vertex to be decoded and the vertices around the vertex to be decoded.

[0080] In this first modification example, the motion vector prediction unit 202E3 of the inter-decoding unit 202E is configured to calculate the MVP according to the following procedure.

[0081] First, the motion vector prediction unit 202E3 is configured to calculate weights.

[0082] FIG. 11 shows a flowchart illustrating an example of an operation of calculating the sum Total_D of distances to the decoded surrounding vertices.

[0083] As shown in FIG. 11, in step S1101, the motion vector prediction unit 202E3 sets 0 to Total_D.

[0084] Step S1102 is the same as step S1002.

[0085] Step S1103 is the same as step S1003.

[0086] In step S1104, the motion vector prediction unit 202E3 adds e(k) to Total_D.

[0087] That is, the motion vector prediction unit 202E3 refers to a set of vertices around the vertex to be decoded, and adds the distances of the decoded vertices.

[0088] In the first modification example, the motion vector prediction unit 202E3 is configured to calculate weights using distances in a reference frame where the correspondence between vertices is known.

[0089] That is, e(k) in step S1104 of FIG. 11 is the distance between corresponding vertices in the reference frame.

[0090] And the motion vector prediction unit 202E3 may be configured to calculate the weight w(k) by formulas (3) and (4).

[0091]

Equation

[0092] Note that the motion vector prediction unit 202E3 may be configured to set weights according to a rule determined in advance according to the distance.

[0093] For example, the motion vector prediction unit 202E3 may be configured to set the weight to 1 when e(k) is smaller than the threshold TH1, set the weight to 0.5 when e(k) is smaller than the threshold TH2, and set the weight to 0 (not using the weight) in other cases.

[0094] According to such a configuration, it is possible to expect the effect that the MVP can be calculated with higher accuracy by increasing the weight when the distance to the vertex to be decoded is close.

[0095] Second, the motion vector prediction unit 202E3 is configured to refer to the MVP.

[0096] FIG. 12 shows a flowchart illustrating an example of an operation of calculating the MVP using weighted average.

[0097] As shown in FIG. 12, in step S1201, the motion vector prediction unit 202E3 sets 0 for the MVP and N.

[0098] Step S1202 is the same as step S1002.

[0099] Step S1203 is the same as step S1003.

[0100] In step S1204, the motion vector prediction unit 202E3 adds w(k)×MV(k) to the MVP and adds 1 to N.

[0101] Step S1205 is the same as step S1005.

[0102] Alternatively, the motion vector prediction unit 202E3 may be configured to calculate the MVP by the formula (5).

[0103]

Equation

[0104] According to such a configuration, since a higher-precision MVP can be calculated by weighted average, an effect of improving the coding efficiency can be expected by reducing the value of the MVR and concentrating it near zero.

[0105] (Modification Example 2 of the Inter Decoder 202E) In this second modification example, the motion vector prediction unit 202E3 is configured to select one MV instead of calculating the MVP using a plurality of surrounding MVs.

[0106] That is, the motion vector prediction unit 202E3 may be configured to select the MV of the nearest vertex among the decoded MVs stored in the motion vector buffer unit 202E2 as the MV of the vertex connected to the vertex to be decoded.

[0107] Here, the motion vector prediction unit 202E3 may be configured to construct a candidate list consisting of the MVs of the vertices connected to the vertex to be decoded from among the decoded MVs stored in the motion vector buffer unit 202E2, and select a motion vector from the candidate list based on the index decoded from the bitstream of the P frame (the frame to be decoded).

[0108] FIG. 13 shows a flowchart illustrating an example of an operation of selecting an MV from a set of candidate MVs as the MVP.

[0109] As shown in FIG. 13, in step S1301, the motion vector prediction unit 202E3 decodes the list ID from the bitstream of the P frame.

[0110] In step S1302, the motion vector prediction unit 202E3 selects, as the MVP, the MV to which the list ID is attached from among the candidate MVs.

[0111] Note that in the set of candidate MVs in FIG. 13, the decoded surrounding MVs and the MVs calculated by combining them are arranged in a certain order.

[0112] FIG. 14 shows a flowchart illustrating an example of an operation of creating such a set of candidate MVs.

[0113] As shown in FIG. 14, in step S1401, the motion vector prediction unit 202E3 refers to the set of MVs of the vertices around the vertex to be decoded, and determines whether the processing for all the vertices around the vertex to be decoded has been completed.

[0114] If such processing has been completed, this operation ends. If such processing has not been completed, this operation proceeds to step S1402.

[0115] In step S1402, the motion vector prediction unit 202E3 determines whether the MV of the target vertex has been decoded.

[0116] If such MV has been decoded, this operation proceeds to step S1403. If such MV has not been decoded, this operation returns to step S1401.

[0117] In step S1403, the motion vector prediction unit 202E3 determines whether such MV overlaps with other decoded MVs.

[0118] If there is an overlap, this operation returns to step S1401. If there is no overlap, this operation proceeds to step S1404.

[0119] In step S1404, the motion vector prediction unit 202E3 determines the list ID to be assigned to such MV, and includes it in the set of candidate MVs in step S1405.

[0120] Note that in FIG. 14, when the motion vector prediction unit 202E3 determines the list ID, the list ID may be incremented one by one in order, or the list ID may be determined in the order of the distance (e(k) in Equation (3)) between the vertex to be decoded and the vertex corresponding to vertex k in the reference frame.

[0121] According to such a configuration, since selecting one of the candidate MVs as the MVP may be closer to the MV than the average in some cases, in such cases, an effect of improving the coding efficiency can be expected.

[0122] Furthermore, the motion vector prediction unit 202E3 may be configured to add, as a new candidate MV, an MV obtained by averaging consecutive MV0 and MV1 from among the above-described candidate MVs to the list. As the addition destination of such an MV, as shown in Table 1, it is after MV0 and MV1.

[0123] [Table 1] According to such a configuration, an effect of increasing the possibility that the selected candidate MV is closer to the MV of the vertex to be decoded can be expected.

[0124] Furthermore, the motion vector prediction unit 202E3 may be configured to select the MV of the vertex closest to the candidate MV set without encoding the list ID. According to such a configuration, an effect of further improving the encoding efficiency can be expected.

[0125] (Modification Example 3 of the Inter Decoder 202E) In the above-described embodiments and modification examples 1 to 2, the surrounding vertices were the vertices connected to the vertex to be decoded.

[0126] In contrast, in this modification example 3, the motion vector prediction unit 202E3 is configured to calculate the MVP by parallelogram prediction, that is, by using vertices that are not directly connected to the vertex to be decoded.

[0127] As shown in FIG. 15, in parallelogram prediction, the vertex D on the opposite side of the decoded surface having the shared edge BC with the vertex A to be decoded is also used.

[0128] In addition, the shared edges of the vertex A to be decoded are CE and BG in addition to AB. Therefore, in parallelogram prediction, vertices F and H can also be used in the same manner.

[0129] For example, the motion vector prediction unit 202E3 may be configured to calculate the MVP using the plane BCD shown in FIG. 15 according to Equation (6).

[0130] MVP = MV(B) + MV(C) - MV(D) … (6) Here, MV(X) is the motion vector of vertex X, and MVP is the predicted value of the motion vector of vertex A to be decoded.

[0131] Also, when there are multiple shared edges as described above, the motion vector prediction unit 202E3 may average each MVP, or may select the plane with the closest center of gravity of the plane.

[0132] (Modification Example 4 of Inter-decoder 202E) In this modification example, the MVR generated by the motion vector residual decoder unit 202E1 is not left as it is, but is configured such that the quantization width when expressing the MVR as an integer is controlled.

[0133] In this modification example, the motion vector residual decoder unit 202E1 is configured to decode the adaptive_mesh_flag, the adaptive_bit_flag, and the accuracy control parameter as control information for controlling the quantization width of the MVR.

[0134] That is, the motion vector residual decoder unit 202E1 is configured to decode the adaptive_mesh_flag for the entire basic mesh and the adaptive_bit_flag for each basic patch.

[0135] Here, the adaptive_mesh_flag and the adaptive_bit_flag are flags indicating whether to adjust the quantization width of the above-described MVR, and take either a value of 0 or 1.

[0136] Here, the motion vector residual decoder unit 202E1 decodes the adaptive_bit_flag only when the adaptive_mesh_flag is valid (i.e., 1).

[0137] Also, when the adaptive_mesh_flag is invalid (i.e., 0), the motion vector residual decoding unit 202E1 regards the adaptive_bit_flag as invalid (i.e., 0).

[0138] FIG. 16 shows a flowchart illustrating an example of an operation for controlling the quantization width of the decoded MVR from the adaptive_mesh_flag, adaptive_bit_flag, and precision control parameters, which are control information generated by decoding the basic mesh bit stream.

[0139] As shown in FIG. 16, in step S1601, the motion vector prediction unit 202E3 determines whether the adaptive_mesh_flag is 0.

[0140] If it is determined that the adaptive_mesh_flag for the entire mesh is 0, this operation ends.

[0141] On the other hand, if it is determined that the adaptive_mesh_flag for the entire mesh is 1, this operation proceeds to step S1602.

[0142] In step S1602, the motion vector prediction unit 202E3 determines whether there is an unprocessed patch within the frame.

[0143] In step S1603, the motion vector prediction unit 202E3 determines whether the adaptive_mesh_flag decoded for each patch is 0.

[0144] If it is determined that the adaptive_mesh_flag is 0, this operation returns to step S1601.

[0145] On the other hand, if it is determined that the adaptive_mesh_flag is 1, this operation proceeds to step S1604.

[0146] In step S1604, the motion vector prediction unit 202E3 controls the quantization width of the MVR based on the accuracy control parameter described later.

[0147] Note that the value of the MVR whose quantization width is controlled in this way is called "MVRQ (Motion Vector Residual Quantization)".

[0148] Here, the motion vector prediction unit 202E3 may be configured to use the quantization width of the MVR corresponding to the quantization width control parameter generated by decoding the basic mesh bit stream with reference to a table such as Table 2, for example.

[0149]

Table 2

[0150] (Modification Example 5 of the Inter Decoder Unit 202E) When the MVR generated by the motion vector residual decoder unit 202E1 is not encoded, an error occurs. In this modification example 5, in order to correct such an error, discrete motion vector differences are encoded.

[0151] Specifically, as shown in FIG. 17, the MVR can take values of 1, 2, 4, and 8 in six directions of the x-axis, y-axis, and z-axis. An example of such encoding is shown in Tables 3 and 4.

[0152] Also, the MVR may be encoded in a combination of a plurality of directions. For example, correction may be performed in the order of 2 in the + direction of the x-axis and 1 in the + direction of the y-axis.

[0153]

Table 3

[0154]

Table 4

[0155] Hereinafter, a further modification example of the inter decoder 202E will be described.

[0156] In the further modification example of the above-described inter decoder 202E, before implementing the above-described inter decoder 202E, it is configured to add the following functional blocks.

[0157] Specifically, as shown in FIG. 18, in addition to the configuration shown in FIG. 8, the inter decoder 202E includes a duplicate vertex search unit 202E6, a duplicate vertex determination unit 202E7, a motion vector acquisition unit 202E8, and an All skip mode single and a skip mode single.

[0158] Here, the All skip mode signal is at the beginning of the bit stream of the P frame, has at least two values, and is 1 bit or 1 bit or more.

[0159] One of them (when the All skip mode signal indicates Yes, for example, when it is 1) is a signal for copying the motion vectors of all duplicate vertices of the P frame without decoding the motion vectors from the bit stream.

[0160] Another one (when the All skip mode signal indicates No, for example, when it is 0) is a signal that performs different processing at each vertex of the P frame. Further, another one may have other values. For example, another one is a single that does not perform the processing in the motion vector acquisition unit 202E8 for the motion vectors of all duplicate vertices and performs the same processing as the inter-decoding unit 202E shown in FIG. 8.

[0161] Here, when the All skip mode signal indicates No, the Skip mode signal has two values for each duplicate vertex and is 1 bit.

[0162] When the All skip mode signal indicates Yes (for example, when it is 1), the Skip mode signal is a signal that does not decode the motion vector of the vertex from the bit stream but copies the motion vectors of the duplicated vertices.

[0163] When the All skip mode signal indicates No (for example, when it is 0), the Skip mode signal is a single that does not perform the processing in the motion vector acquisition unit 202E8 for the motion vector of the vertex and performs the same processing as the inter-decoding unit 202E shown in FIG. 8.

[0164] Note that the above-described Skip mode signal may be directly decoded from the bit stream, or data (for example, the index of the duplicate vertex) that identifies duplicate vertices that perform the same processing as the inter-decoding unit 202E shown in FIG. 8 is decoded from the bit stream, and the Skip mode signal may be calculated from such data.

[0165] Furthermore, as shown in FIG. 38, without calculating the Skip mode signal, using data (for example, the index of the duplicate vertex) that identifies duplicate vertices that perform the same processing as the inter-decoding unit 202E shown in FIG. 8 above, the motion vector decoding method of the vertex may be determined in the same manner as in the above case.

[0166] The duplicate vertex search unit 202E6 is configured to search for the indices of vertices with matching coordinates (hereinafter referred to as duplicate vertices) from the geometric information of the basic mesh of the decoded reference frame and store them in a buffer (not shown).

[0167] Specifically, the input to the duplicate vertex search unit 202E6 is the index (in decoding order) and position coordinates of each vertex of the basic mesh of the decoded reference frame.

[0168] Also, the output of the duplicate vertex search unit 202E6 is a list of pairs of the index of the vertex where the duplicate vertex exists (vindex0) and the index of such duplicate vertex (vindex1). Here, such a list of pairs is stored in the buffer repVert in the order of index0.

[0169] Also, since the vertex of vindex1 is decoded before that of vindex0, the relationship vindex0 > vindex1 holds.

[0170] Note that as a method for finding duplicate vertices in the basic mesh of the reference frame, for vertices where duplicate vertices exist, instead of position coordinates, the index of the duplicate vertex is decoded by a special signal. With such a special signal, a pair of the index of the corresponding vertex and the index of the duplicate vertex can be stored in decoding order.

[0171] The duplicate vertex determination unit 202E7 is configured to determine whether there is a duplicate vertex among the decoded vertices of the corresponding vertex.

[0172] Here, if the index of the corresponding vertex is among the indices of the vertices where duplicate vertices exist, the duplicate vertex determination unit 202E7 determines that there is a duplicate vertex among the decoded vertices. Since the corresponding vertex comes in decoding order, the above search is unnecessary.

[0173] Here, when the duplicate vertex determination unit 202E7 determines that there is no duplicate vertex for the corresponding vertex, the same processing as the inter-decoding unit 202E shown in FIG. 8 is performed.

[0174] When there are duplicate vertices for the corresponding vertex, when the All skip mode signal indicates Yes, or when the All skip mode signal indicates No and the Skip mode signal for the corresponding vertex indicates Yes, the motion vector acquisition unit 202E8 acquires the motion vector of the vertex having the same index as the duplicate vertex from the motion vector buffer unit 202E2 that stores the decoded motion vector, and is configured to use it as the motion vector of the corresponding vertex.

[0175] Here, when the All skip mode signal indicates No and the Skip mode signal for the corresponding vertex indicates No, instead of the motion vector acquisition unit 202E8, the same processing as that of the inter-decoding unit 202E shown in FIG. 8 is performed.

[0176] According to such a configuration, it is possible to expect the effect of reducing the decoding calculation of the motion vector and the amount of code for the vertex having duplicate vertices.

[0177] In a further modification example of the above-described inter-decoding unit 202E, the inter-decoding unit 202E acquires the correspondence relationship between the vertices of the reference frame and the vertices of the frame to be decoded from the decoded basic mesh of the reference frame.

[0178] Then, based on such a correspondence relationship, the inter-decoding unit 202E is configured to make the connection information of the vertices of the frame to be decoded the same as the connection information of the decoded vertices of the reference frame without encoding the connection information of the vertices of the frame to be decoded.

[0179] Also, the inter-decoding unit 202E divides the basic mesh of the frame to be decoded into two types of regions based on the signal in the decoding order of the vertices of the reference frame. In the first region, decoding is performed using inter-processing, and in the second region, decoding is performed using intra-processing.

[0180] Note that the above-described regions are defined as regions formed by a plurality of consecutive vertices in the decoding order when decoding the basic mesh of the reference frame.

[0181] In addition, with respect to the means for decoding the coordinates of the vertices of the basic mesh of the frame to be decoded using signals, the following two implementations are assumed. (Means 1) In Means 1, the signals become vertex_idx1, vertex_idx2, and intra_flag.

[0182] Here, vertex_idx1 and vertex_idx2 are indices of the decoding order of the vertices (vertex indices), and intra_flag is a flag indicating whether it is the above-described inter-decoding method or intra-decoding method. There may be a plurality of such signals.

[0183] That is, vertex_idx1 and vertex_idx2 are vertex indices that define the start position and end position of the above-described partial regions (the first region and the second region). (Means 2) In Means 2, there is a premise that the connection information of the basic mesh of the reference frame is decoded by Edgebreaker, and the decoding order of the vertex coordinates is set to the order determined by Edgebreaker.

[0184] FIG. 19 is a diagram showing an example of an operation for determining connection information and the order of vertices using Edgebreaker.

[0185] In FIG. 19, the arrows indicate the decoding order of the connection information, the numbers indicate the decoding order of the vertices, and the same region is defined by arrows of the same line type.

[0186] In Means 2, the signal becomes only intra_flag, which is a flag indicating whether it is the inter-decoding method or intra-decoding method.

[0187] That is, in Means 2, the inter-decoding unit 202E is configured to be divided into a first region and a second region using Edgebreaker.

[0188] <Subdivision section 203> The subdivision section 203 is configured to generate and output additional subdivision vertices and their connection information from the basic mesh decoded by the basic mesh decoder 202 according to the subdivision method indicated by the control information.

[0189] Here, the basic mesh, the additional subdivision vertices, and their connection information are collectively referred to as the "subdivision mesh".

[0190] The subdivision section 202 is configured to identify the type of subdivision method from the subdivision_method_id, which is the control information generated by decoding the basic mesh bitstream.

[0191] Hereinafter, the subdivision section 202 will be described with reference to FIGS. 3A and 3B.

[0192] FIGS. 3A and 3B are diagrams for explaining an example of the operation of generating subdivision vertices from a basic mesh.

[0193] FIG. 3A shows an example of a basic mesh composed of five vertices.

[0194] Here, for example, the Mid-edge subdivision method of connecting the midpoints of each side on each basic face may be used for subdivision. As a result, a certain basic face will be divided into four faces.

[0195] FIG. 3B shows an example of a subdivision mesh obtained by dividing the basic mesh composed of five vertices. In the subdivision mesh shown in FIG. 3B, eight subdivision vertices (white circles) are generated in addition to the original five vertices (black circles).

[0196] By decoding the displacement amount for each of the thus-generated subdivision vertices by the displacement amount decoder 206, an improvement in coding performance can be expected.

[0197] Also, different subdivision methods may be applied to each patch. By doing so, the displacement amount decoded by the displacement amount decoding unit 206 can be adaptively changed for each patch, and an improvement in coding performance can be expected. The information of the divided patches is received as patch_id which is control information.

[0198] Hereinafter, the subdivision unit 203 will be described with reference to FIG. 20. FIG. 21 is a diagram showing an example of the functional blocks of the subdivision unit 203.

[0199] As shown in FIG. 21, the subdivision unit 203 includes a basic mesh subdivision unit 203A and a subdivided mesh adjustment unit 203B.

[0200] (Basic Mesh Subdivision Unit 203A) The basic mesh subdivision unit 203A is configured to calculate the number of divisions (subdivision number) for each basic surface and basic patch based on the input basic mesh and the division information of the basic mesh, and to subdivide the basic mesh based on such a division number and output a subdivided surface.

[0201] That is, the basic mesh subdivision unit 203A may be configured to be able to change the above-described division number in units of basic surfaces and basic patches.

[0202] Here, the basic surface is a surface constituting the basic mesh, and the basic patch is a set of several basic surfaces.

[0203] Also, the basic mesh subdivision unit 203A may be configured to predict the number of fine subdivisions of the basic surface and calculate the number of subdivisions of the basic surface by adding a predicted division number residual to the predicted number of subdivisions of the basic surface.

[0204] Also, the basic mesh subdivision unit 203A may be configured to calculate the number of subdivisions of the basic surface based on the number of subdivisions of the adjacent basic surfaces of the basic surface.

[0205] Further, the basic mesh subdivision unit 203A may be configured to calculate the subdivision number of the basic surface based on the subdivision number of the basic surface accumulated immediately before.

[0206] Alternatively, the basic mesh subdivision unit 203A may be configured to generate vertices that divide the three sides constituting the basic surface, and divide the basic surface by connecting the generated vertices.

[0207] As shown in FIG. 21, 203B including a subdivision mesh adjustment unit described later is provided at the subsequent stage of the basic mesh subdivision unit 203A.

[0208] Hereinafter, an example of the process of the basic mesh subdivision unit 203A will be described with reference to FIGS. 21 to 23.

[0209] FIG. 21 is a diagram showing an example of the functional block of the basic mesh subdivision unit 203A, and FIG. 23 is a flowchart showing an example of the operation of the basic mesh subdivision unit 203A.

[0210] As shown in FIG. 21, the basic mesh subdivision unit 203A includes a basic surface division number buffer unit 203A1, a basic surface division number reference unit 203A2, a basic surface division number prediction unit 203A3, an addition unit 203A4, and a basic surface division unit 203A5.

[0211] The basic surface division number buffer unit 203A1 stores the division information of the basic surface including the division number of the basic surface, and is configured to output the division information of the basic surface to the basic surface division number reference unit 203A2.

[0212] Here, the size of the basic surface division number buffer unit 203A1 may be set to 1, and it may be configured to output the division number of the basic surface accumulated immediately before to the basic surface division number reference unit 203A2.

[0213] That is, by setting the size of the basic surface division number buffer unit 203A1 to 1, it may be configured to refer only to the last decoded fine division number (the immediately preceding decoded subdivision number).

[0214] When there is no fundamental plane adjacent to the fundamental plane to be decoded, or when there is a fundamental plane adjacent to the fundamental plane to be decoded but the number of divisions has not been determined, the fundamental plane division number reference unit 203A2 is configured to output non-referenceable to the fundamental plane division number prediction unit 203A3.

[0215] On the other hand, when there is a fundamental plane adjacent to the fundamental plane to be decoded and the number of divisions is determined, the fundamental plane division number reference unit 203A2 is configured to output such a number of divisions to the fundamental plane division number prediction unit 203A3.

[0216] The fundamental plane division number prediction unit 203A3 is configured to predict the division number (fine division number) of the fundamental plane based on one or more input division numbers, and output the predicted division number (predicted division number) to the addition unit 203A4.

[0217] Here, when only non-referenceable is input from the fundamental plane division number reference unit 203A2, the fundamental plane division number prediction unit 203A3 is configured to output 0 to the addition unit 203A4.

[0218] Note that when one or more division numbers are input, the fundamental plane division number prediction unit 203A3 may be configured to generate a predicted division number using any of statistical values such as the average value, maximum value, minimum value, and most frequent value of the input division numbers.

[0219] Note that when one or more division numbers are input, the fundamental plane division number prediction unit 203A3 may be configured to generate the predicted division number as the division number of the most adjacent plane.

[0220] The addition unit 203A4 is configured to output the division number obtained by adding the predicted division number residual decoded from the predicted residual bit stream and the predicted division number obtained from the fundamental plane division number prediction unit 203A3 to the fundamental plane division unit 203A5.

[0221] The basic surface division unit 203A5 is configured to finely divide the basic surface based on the number of divisions input from the addition unit 203A4.

[0222] FIG. 22 shows an example of a case where the basic surface is divided into nine parts. With reference to FIG. 22, the method of dividing the basic surface by the basic surface division unit 203A5 will be described.

[0223] The basic surface division unit 203A5 generates points A_1,…,A_(N - 1) that divide the side AB constituting the basic surface into N equal parts (N = 3).

[0224] Similarly, the basic surface division unit 203A5 also divides the sides BC and CA into N equal parts, and generates points B_1,…,B_(N - 1) and C_1,…,C_(N - 1) respectively.

[0225] Hereinafter, the points on the sides AB, BC, and CA are referred to as "side division points".

[0226] For all i (i = 1, 2,…, N - 1), the basic surface division unit 203A5 generates the sides A_i B_(N - i), B_i C_(N - i), C_i A_(N - i), and generates N 2 fine - divided surfaces.

[0227] Next, with reference to FIG. 23, the processing procedure of the basic mesh subdivision unit 203A will be described.

[0228] In step S2201, it is determined whether the re - division process for the last basic surface has been completed. If the process is completed, it ends; otherwise, it proceeds to step S2202.

[0229] In step S2202, the basic mesh subdivision unit 203A determines whether Depth < mdu_max_depth.

[0230] Here, Depth is a variable representing the current depth, with an initial value of 0, and mdu_max_depth represents the maximum depth determined for each basic surface.

[0231] If the conditions in step S2202 are met, this processing procedure proceeds to step S2203; if not, this processing procedure returns to step S2201.

[0232] In step S2203, the basic mesh subdivision unit 203A determines whether the mdu_subdivision_flag at the current depth is 1.

[0233] If Yes, this processing procedure returns to step S2201; if No, this processing procedure proceeds to step S2204.

[0234] In step S2204, the basic mesh subdivision unit 203A further subdivides all the subdivision surfaces within the basic plane.

[0235] Here, if the basic mesh subdivision unit 203A has never performed a subdivision process on the basic plane, it subdivides the basic plane.

[0236] Note that the method of subdivision is the same as the method described in step S2204.

[0237] Specifically, if the basic plane has never been subdivided, it is subdivided as shown in Fig. 22. If it has been subdivided at least once, the subdivision surface is subdivided into N 2 pieces. Taking Fig. 22 as an example, the surface composed of vertex A_2, vertex B, and vertex B_1 is further divided in the same way as when dividing the basic plane to generate N 2 surfaces.

[0238] When the subdivision process is completed, this processing procedure proceeds to step S2205.

[0239] In step S2205, the basic mesh subdivision unit 203A adds 1 to Depth, and this processing procedure returns to step S2202.

[0240] (Subdivision Mesh Adjustment Unit 203B) Next, a specific example of the process performed by the sub-division mesh adjustment unit 203B will be described. Hereinafter, an example of the process performed by the sub-division mesh adjustment unit 203B will be described with reference to FIGS. 24 to 28.

[0241] FIG. 24 is a diagram showing an example of the functional blocks of the sub-division mesh adjustment unit 203B.

[0242] As shown in FIG. 24, the sub-division mesh adjustment unit 203B includes an edge division point movement unit 701 and a sub-division surface division unit 702.

[0243] (Edge division point movement unit 701) The edge division point movement unit 701 is configured to move the edge division points of the basic surface to either the edge division points of the adjacent basic surfaces with respect to the input initial sub-division surface, and output the sub-division surface.

[0244] FIG. 25 is an example of moving the edge division points on the basic surface ABC. For example, as shown in FIG. 25, the edge division point movement unit 701 may be configured to move the edge division points of the basic surface ABC to the edge division points of the nearest adjacent basic surface.

[0245] (Sub-division surface division unit 702) The sub-division surface division unit 702 is configured to re-sub-divide the input sub-division surface and output the decoded sub-division surface.

[0246] FIG. 26 is a diagram showing an example of a case where the sub-division surface X within the basic surface is re-sub-divided.

[0247] As shown in FIG. 26, the sub-division surface division unit 702 may be configured to generate a new sub-division surface within the basic surface by connecting the vertices constituting the sub-division surface and the edge division points of the adjacent basic surfaces.

[0248] FIG. 27 is a diagram showing an example of a case where the above-described sub-division process is performed on all sub-division surfaces.

[0249] The mesh decoding unit 204 is configured to generate and output a decoded mesh using the sub-divided mesh generated by the sub-division unit 203 and the displacement amount decoded by the displacement amount decoding unit 206.

[0250] Specifically, the mesh decoding unit 204 is configured to generate a decoded mesh by adding the corresponding displacement amount to each sub-divided vertex. Here, information indicating which sub-divided vertex each displacement amount corresponds to is indicated by control information.

[0251] The patch integration unit 205 is configured to integrate and output the decoded mesh generated by the mesh decoding unit 206 for a plurality of patches.

[0252] Here, the patch division method is defined by the mesh encoding device 100. For example, the patch division method may be configured to calculate the normal vector for each basic surface, select the basic surface with the most similar normal vector among adjacent basic surfaces, group both basic surfaces as the same patch, and sequentially repeat such a procedure for the next basic surface.

[0253] The video decoding unit 207 is configured to decode and output a texture by video encoding. For example, the video decoding unit 207 may use HEVC of Non-Patent Document 1.

[0254] <Displacement amount decoding unit 206> The displacement amount decoding unit 206 is configured to decode the displacement amount bit stream to generate and output a displacement amount.

[0255] FIG. 28 is a diagram showing an example of a displacement amount for a certain sub-divided vertex.

[0256] In the example of FIG. 3B, since there are 8 sub-divided vertices, the displacement amount decoding unit 206 is configured to define 8 displacement amounts represented by a scalar or a vector for each sub-divided vertex.

[0257] Hereinafter, with reference to FIG. 28, the displacement amount decoding unit 206 will be described. FIG. 28 is a diagram showing an example of the functional blocks of the displacement amount decoding unit 206.

[0258] As shown in FIG. 28, the displacement amount decoding unit 206 includes a decoding unit 206A, an inverse quantization unit 206B, an inverse wavelet transform unit 206C, an adder 206D, an inter prediction unit 206E, and a frame buffer 206F.

[0259] The decoding unit 206A is configured to perform variable length decoding on the received displacement amount bit stream to decode and output a level value and control information. Here, the level value obtained by variable length decoding is output to the inverse quantization unit 206B, and the control information is output to the inter prediction unit 206E.

[0260] Hereinafter, with reference to FIG. 29, an example of the configuration of the displacement amount bit stream will be described. FIG. 29 is a diagram showing an example of the configuration of the displacement amount bit stream.

[0261] As shown in FIG. 29, first, the displacement amount bit stream may include a DPS (Displacement Parameter Set), which is a set of control information related to the decoding of the displacement amount.

[0262] Second, the displacement amount bit stream may include a DPH (Displacement Patch Header), which is a set of control information corresponding to a patch.

[0263] Third, the displacement amount bit stream may include, after the DPH, the encoded displacement amounts that make up the patch.

[0264] As described above, the displacement amount bit stream is configured such that one DPH and one DPS correspond to each encoded displacement amount.

[0265] Note that the configuration in Fig. 29 is merely an example. As long as DPH and DPS correspond to each encoded displacement amount, elements other than those described above may be added as components of the displacement amount bitstream.

[0266] For example, as shown in Fig. 29, the displacement amount bitstream may include an SPS (Sequence Parameter Set).

[0267] Fig. 30 is a diagram showing an example of the syntax configuration of DPS.

[0268] In Fig. 30, the Descriptor field indicates how each syntax is encoded.

[0269] Also, in Fig. 30, ue(v) means it is an unsigned zero-order exponential Golomb code, and u(n) means it is an n-bit flag.

[0270] When there are multiple DPSs, DPS includes at least DPS id information (dps_displacement_parameter_set_id) for identifying each DPS.

[0271] Also, DPS may include a flag (interprediction_enabled_flag) for controlling whether to perform inter prediction.

[0272] For example, when interprediction_enabled_flag is 0, it may be defined that inter prediction is not performed, and when interprediction_enabled_flag is 1, it may be defined that inter prediction is performed. When interprediction_enabled_flag is not included, it may be defined that inter prediction is not performed.

[0273] DPS may include a flag (dct_enabled_flag) for controlling whether to perform inverse DCT.

[0274] For example, when the dct_enabled_flag is 0, it may be defined that the inverse DCT is not performed, and when the dct_enabled_flag is 1, it may be defined that the inverse DCT is performed. When the dct_enabled_flag is not included, it may be defined that the inverse DCT is not performed.

[0275] FIG. 31 is a diagram showing an example of the syntax configuration of DPH.

[0276] As shown in FIG. 31, DPH at least includes DPS id information for specifying a DPS corresponding to each DPH.

[0277] The inverse quantization unit 206B is configured to generate and output a transform coefficient by inverse quantizing the level value decoded by the decoding unit 206A.

[0278] The inverse wavelet transform unit 206C is configured to generate and output a prediction residue by performing an inverse wavelet transform on the transform coefficient generated by the inverse quantization unit 206B.

[0279] (Inter prediction unit 206E) The inter prediction unit 206E is configured to generate and output a prediction displacement amount by performing inter prediction using the decoded displacement amount of the reference frame read from the frame buffer 206F.

[0280] The inter prediction unit 206E is configured to perform such inter prediction only when the interprediction_enabled_flag is 1.

[0281] The inter prediction unit 206E may perform inter prediction in the spatial domain or in the frequency domain. The inter prediction may perform bidirectional prediction using a reference frame in the past and a reference frame in the future in terms of time.

[0282] FIG. 28 is an example of a functional block of the inter prediction unit 206E when inter prediction is performed in the spatial domain.

[0283] When the inter prediction unit 206E performs inter prediction in the spatial domain, for the predicted displacement amount of the sub - divided vertex in the target frame, the decoded displacement amount of the corresponding sub - divided vertex in the reference frame may be directly referred to for determination.

[0284] Alternatively, the predicted displacement amount of a certain sub - divided vertex in the target frame may be probabilistically determined according to a normal distribution in which the mean and variance are estimated using the decoded displacement amounts of the corresponding sub - divided vertices in a plurality of reference frames. In that case, the variance may be set to zero and determined uniquely by the mean only.

[0285] Alternatively, the predicted displacement amount of a certain sub - divided vertex in the target frame may be determined based on a regression curve estimated using the decoded displacement amounts of the corresponding sub - divided vertices in a plurality of reference frames, with time as the explanatory variable and the displacement amount as the objective variable.

[0286] In the mesh encoding device 100, for each frame, in order to improve the encoding efficiency, the order of such decoded displacement amounts may be rearranged.

[0287] In such a case, the inter prediction unit 206E may be configured to perform inter prediction on the rearranged decoded displacement amounts.

[0288] The correspondence relationship of the sub - divided vertices between the reference frame and the frame to be decoded is indicated by control information.

[0289] FIG. 32 is a diagram for explaining an example of the correspondence relationship of the sub - divided vertices between the reference frame and the frame to be decoded when inter prediction is performed in the spatial domain.

[0290] FIG. 33 is an example of a functional block of the inter prediction unit 206E when inter prediction is performed in the frequency domain.

[0291] When the inter prediction unit 206E performs inter prediction in the frequency domain, for the predicted wavelet transform coefficients of the frequency in the frame to be decoded, the decoded wavelet transform coefficients of the corresponding frequency in the reference frame may be directly referred to and determined.

[0292] The inter prediction unit 206E may perform probabilistic inter prediction according to a normal distribution in which the mean and variance are estimated by using the decoded displacement amounts or decoded wavelet transform coefficients of the sub-division vertices in a plurality of reference frames.

[0293] The inter prediction unit 206E may perform inter prediction based on a regression curve estimated with time as an explanatory variable and the displacement amount as an objective variable by using the decoded displacement amounts or decoded wavelet transform coefficients of the sub-division vertices in a plurality of reference frames.

[0294] The inter prediction unit 206E may be configured to perform bidirectional inter prediction by using a reference frame in the past and a reference frame in the future in terms of time.

[0295] In the mesh encoding device 100, for each frame, in order to improve the encoding efficiency, the order of such decoded wavelet transform coefficients may be rearranged.

[0296] The correspondence relationship of the frequency between the reference frame and the frame to be decoded is indicated by control information.

[0297] FIG. 34 is a diagram for explaining an example of the correspondence relationship of the frequency between the reference frame and the frame to be decoded when inter prediction is performed in the frequency domain.

[0298] Also, when the sub-division unit 203 divides the basic mesh into a plurality of patches, the inter prediction unit 206E is also configured to perform inter prediction for each divided patch. Thereby, the temporal correlation between frames is increased, and an improvement in encoding performance can be expected.

[0299] The adder 206D receives a prediction residual from the inverse wavelet transform unit 206C and a prediction displacement amount from the inter prediction unit 206E.

[0300] The adder 206D is configured to calculate and output a decoding displacement amount by adding such a prediction residual and a prediction displacement amount.

[0301] The decoding displacement amount calculated by the adder 206D is also output to the frame buffer 206F.

[0302] The frame buffer 206F is configured to acquire and store the decoding displacement amount from the adder 206D.

[0303] Here, the frame buffer 206F outputs the decoding displacement amount at the corresponding vertex in the reference frame according to control information (not shown).

[0304] FIG. 35 is a flowchart showing an example of the operation of the displacement amount decoding unit 206.

[0305] As shown in FIG. 35, in step S3501, the displacement amount decoding unit 206 determines whether this process has been completed for all patches.

[0306] If Yes, this operation ends; if No, this operation proceeds to step S3502.

[0307] In step S3502, the displacement amount decoding unit 206 performs inverse DCT, inverse quantization, and inverse wavelet transform on the patch to be decoded.

[0308] In step S3503, the displacement amount decoding unit 206 determines whether the interprediction_enabled_flag is 1.

[0309] If Yes, this operation proceeds to step S3504; if No, this operation returns to step S3501.

[0310] In step S3504, the displacement amount decoding unit 206 performs the above-described inter prediction and addition.

[0311] <Modification Example 1> Hereinafter, with reference to FIG. 36, Modification Example 1 of the above-described first embodiment will be described focusing on the differences from the above-described first embodiment.

[0312] FIG. 36 is a diagram showing an example of a functional block of the displacement amount decoding unit 206 according to this Modification Example 1.

[0313] As shown in FIG. 36, the displacement amount decoding unit 206 according to this Modification Example 1 includes an inverse DCT unit 206G at the subsequent stage of the decoding unit 206A, that is, between the decoding unit 206A and the inverse quantization unit 206B.

[0314] That is, in this Modification Example 1, the inverse quantization unit 206B is configured to generate a prediction residual by performing an inverse wavelet transform on the level value output from the inverse DCT unit 202G.

[0315] <Modification Example 2> Hereinafter, with reference to FIG. 37, Modification Example 2 of the above-described first embodiment will be described focusing on the differences from the above-described first embodiment.

[0316] As shown in FIG. 37, the displacement amount decoding unit 206 according to this Modification Example 2 includes a video decoding unit 2061, an image expansion unit 2062, an inverse quantization unit 2063, and an inverse wavelet transform unit 2064.

[0317] The video decoding unit 2061 is configured to output a video by decoding the received displacement amount bit stream by video encoding.

[0318] For example, the video decoding unit 2061 may use HEVC of Non-Patent Document 1.

[0319] Alternatively, the video decoding unit 2061 may use a video encoding method in which motion vectors are always set to zero. For example, the video decoding unit 2061 may always set the motion vectors of HEVC to zero and use inter prediction at the same position all the time.

[0320] Alternatively, the video decoding unit 2061 may use a video encoding method in which transformation is always skipped. For example, the video decoding unit 2061 may always set the transformation of HEVC to the transformation skip mode and use a video encoding method without performing transformation.

[0321] The image expansion unit 2062 is configured to expand and output the video decoded by the video decoding unit 2061 as level values for each image (frame).

[0322] In such an expansion method, the image expansion unit 2062 can be specified by inverse calculation from the arrangement order of the level values for the image indicated by the control information.

[0323] As the arrangement order of the level values, for example, the level values from the high-frequency components to the low-frequency components may be arranged in the raster operation order in the image.

[0324] The inverse quantization unit 2063 is configured to generate and output transform coefficients by inverse quantizing the level values generated by the image expansion unit 2062.

[0325] The inverse wavelet transform unit 2064 is configured to generate and output a decoded displacement amount by performing an inverse wavelet transform on the transform coefficients generated by the inverse quantization unit 2063.

[0326] The above-described mesh encoding device 100 and mesh decoding device 200 may be realized by a program that causes a computer to execute each function (each process).

Industrial Applicability

[0327] According to this embodiment, for example, in video communication, since an improvement in overall service quality can be realized, it becomes possible to contribute to Goal 9 of the Sustainable Development Goals (SDGs) led by the United Nations, which is to "build resilient infrastructure, promote sustainable industrialization, and foster innovation."

Explanation of Reference Numerals

[0328] 1…Mesh processing system 100…Mesh encoding device 200…Mesh decoding unit 201…Multiplex separation unit 202…Basic mesh decoding unit 202A…Separation unit 202B…Intra decoding unit 202B1…Optional intra decoding unit 202B2…Alignment unit 202C…Mesh buffer unit 202D…Connection information decoding unit 202E…Inter decoding unit 202E1…Motion vector decoding unit 202E2…Motion vector buffer unit 202E3…Motion vector prediction unit 202E4…Motion vector calculation unit 202E5…Adder 202E6…Duplicate vertex search unit 202E7…Duplicate vertex discrimination unit 202E8…Motion vector acquisition unit 203…Subdivision unit 203A…Basic mesh subdivision unit 203A1…Basic surface division number buffer unit 203A2…Basic surface division number reference unit 203A3…Basic surface division number prediction unit 203A4…Addition unit 203A5…Basic surface division unit 203B…Subdivision mesh adjustment unit 204…Mesh decoding unit 205…Patch integration unit 206…Displacement amount decoding unit 206A…Decoder 206B, 2063…Inverse quantization section 206C, 2064…Inverse wavelet transform section 206D…Adder 206E…Inter prediction section 206F…Frame buffer 206G…Inverse DCT section 2062…Image expansion section 207, 2061…Video decoder

Claims

1. A mesh decoding device, comprising: a displacement decoding unit configured to decode a displacement amount bit stream to generate and output a displacement amount; wherein the displacement decoding unit includes: a decoding unit configured to decode a level value from the displacement amount bit stream; an inverse quantization unit configured to generate a transform coefficient by inverse quantizing the level value; an inverse wavelet transform unit configured to generate a prediction residual by performing an inverse wavelet transform on the transform coefficient; an adder configured to calculate the displacement amount by adding a predicted displacement amount and the prediction residual.

2. The mesh decoding device according to claim 1, further comprising an inverse DCT unit configured to perform an inverse DCT on the level value, between the decoding unit and the inverse quantization unit; wherein the inverse quantization unit is configured to generate a prediction residual by performing the inverse wavelet transform on the level value output from the inverse DCT unit.

3. The mesh decoding device according to claim 1, further comprising: a frame buffer configured to acquire and store the displacement amount output from the adder; and an inter prediction unit configured to generate the predicted displacement amount by performing inter prediction using the displacement amount of a reference frame read from the frame buffer.

4. The mesh decoding device according to claim 3, wherein the inter prediction unit is configured to perform the inter prediction for each patch.

5. The mesh decoding device according to claim 3, wherein the inter prediction unit is configured to perform the inter prediction in a spatial domain.

6. The mesh decoding device according to claim 3, wherein the inter prediction unit is configured to perform the inter prediction in a frequency domain.

7. The mesh decoding device according to claim 5 or 6, wherein the inter prediction unit is configured to perform the inter prediction on the rearranged displacement amount.

8. A mesh decoding method, comprising: a step of decoding a level value from a displacement amount bit stream; a step of generating a transform coefficient by inverse quantizing the level value; A step of generating a prediction residual by performing an inverse wavelet transform on the conversion coefficient; A step of calculating a displacement amount by adding a predicted displacement amount and the prediction residual, wherein the method for mesh decoding is characterized by comprising the steps.

9. A program for causing a computer to function as a mesh decoding device, wherein the mesh decoding device comprises a displacement amount decoding unit configured to decode a displacement amount bit stream to generate and output a displacement amount, wherein the displacement amount decoding unit comprises a decoding unit configured to decode a level value from the displacement amount bit stream, an inverse quantization unit configured to generate a conversion coefficient by inverse quantizing the level value, an inverse wavelet transform unit configured to generate a prediction residual by performing an inverse wavelet transform on the conversion coefficient, and an adder configured to calculate the displacement amount by adding a predicted displacement amount and the prediction residual, wherein the program is characterized by comprising the units.

Images