Mesh decoding device, mesh decoding method, and program product

Abstract

A mesh decoding device (200) according to the present invention is provided with: a motion vector residual decoding unit (202E1) that generates a prediction mode for a motion vector residual and a motion vector from a bitstream of an inter-frame; a motion vector prediction unit (202E3) that predicts a motion vector of a vertex within a reference frame using a motion vector of a decoded vertex around a vertex to be decoded, a motion vector of a vertex within the reference frame corresponding to the vertex to be decoded, and a motion vector of a vertex within the reference frame corresponding to a decoded vertex around the vertex to be decoded; calculating a predicted value of the motion vector of the vertex to be decoded by a prediction method specified by the prediction mode from among a plurality of prediction methods; and a motion vector calculation unit (202E4) that adds the predicted value of the motion vector to the motion vector residual.

Description

Technical Field

[0001] The invention relates to a grid decoding device, a grid decoding method and a program product. Background Art

[0002] Non-patent document 1 discloses a technology for encoding a grid using non-patent document 2.

[0003] Prior art literature

[0004] Non-patent literature

[0005] Non-patent literature 1: Khaled Mammou, Jungsun Kim, Alexis M Tourapis, DimitriPodborski, and Krasimir Kolarov, “[V-CG] Apple's Dynamic Mesh Coding CfP Response,” April 2022, ISO / IEC JTC 1 / SC 29 / WG 7.

[0006] Non-patent document 2: Google Draco, accessed on May 26, 2022 [Online], https: / / google.github.io / draco Summary of the invention

[0007] Problems to be solved by the invention

[0008] However, in the prior art, since the coordinates or connection information of all vertices constituting the dynamic mesh are losslessly encoded, the amount of information cannot be reduced even under the condition of allowing loss, and there is a problem of low encoding efficiency.

[0009] In addition, in the prior art, there is also a problem of low coding efficiency of motion vectors due to low prediction accuracy of motion vectors. Therefore, the present invention is made in view of the above-mentioned problems, and its purpose is to provide a grid decoding device, grid decoding method and program product that can improve the coding efficiency of grids.

[0010] Means for solving problems

[0011] The first feature of the present invention is a grid decoding device, the main purpose of which is to have: a motion vector residual decoding unit, which generates a motion vector residual and a prediction mode of a motion vector based on a bit stream of an inter-frame frame; a motion vector prediction unit, which uses the motion vectors of decoded vertices around a decoding target vertex, the motion vectors of vertices in a reference frame corresponding to the decoding target vertex, and the motion vectors of vertices in the reference frame corresponding to the decoded vertices around the decoding target vertex, and calculates a predicted value of the motion vector of the decoding target vertex by a prediction method specified by the prediction mode from among multiple prediction methods; and a motion vector calculation unit, which adds the predicted value of the motion vector to the motion vector residual.

[0012] The second feature of the present invention is a grid decoding method, the main purpose of which is to have: a process of generating a motion vector residual and a prediction mode of a motion vector based on a bit stream of an inter-frame frame; a process of calculating a predicted value of the motion vector of a decoding target vertex by a prediction method specified by the prediction mode from among multiple prediction methods, using the motion vectors of decoded vertices around the decoding target vertex, the motion vectors of vertices in a reference frame corresponding to the decoding target vertex, and the motion vectors of vertices in the reference frame corresponding to the decoded vertices around the decoding target vertex; and a process of adding the predicted value of the motion vector to the motion vector residual.

[0013] The third feature of the present invention is a program product, which includes a computer program, so that when the computer program is executed, it realizes the function of a grid decoding device, and the grid decoding device has: a motion vector residual decoding unit, which generates a motion vector residual and a prediction mode of a motion vector based on a bit stream of an inter-frame frame; a motion vector prediction unit, which uses the motion vectors of decoded vertices around a decoding target vertex, the motion vectors of vertices in a reference frame corresponding to the decoding target vertex, and the motion vectors of vertices in the reference frame corresponding to the decoded vertices around the decoding target vertex, and calculates a predicted value of the motion vector of the decoding target vertex by using a prediction method specified by the prediction mode from among multiple prediction methods; and a motion vector calculation unit, which adds the predicted value of the motion vector to the motion vector residual.

[0014] Effects of the Invention

[0015] According to the present invention, a grid decoding device, a grid decoding method and a program product capable of improving the coding efficiency of the grid can be provided.

[0016] Brief description of the attached figure

[0017] Figure 1 It is a diagram showing an example of the configuration of a mesh processing system 1 according to an embodiment.

[0018] Figure 2 This is a diagram showing an example of functional blocks of a trellis decoding device 200 according to an embodiment.

[0019] Figure 3A This is a diagram showing an example of a basic grid and sub-divided grids.

[0020] Figure 3B This is a diagram showing an example of a basic grid and sub-divided grids.

[0021] Figure 4 This is a diagram showing an example of the syntax structure of the basic grid bit stream.

[0022] Figure 5 This is a diagram showing an example of the syntax structure of BPH.

[0023] Figure 6 This is a diagram showing an example of a functional block of the basic trellis decoding unit 202 of the trellis decoding device 200 according to one embodiment.

[0024] Figure 7 This is a diagram showing an example of a functional block of the intra-frame decoding unit 202B of the base trellis decoding unit 202 of the trellis decoding device 200 according to one embodiment.

[0025] Figure 8 This is a diagram showing an example of the correspondence relationship between the vertices of the base mesh of the P frame and the vertices of the base mesh of the I frame.

[0026] Fig. 9 This is a diagram showing an example of a functional block of the inter-frame decoding unit 202E of the basic trellis decoding unit 202 of the trellis decoding device 200 according to one embodiment.

[0027] Fig.10 This is a diagram for explaining an example of a method for calculating the MVP of a vertex of a decoding target implemented by the motion vector prediction unit 202E3 of the inter decoding unit 202E of the base trellis decoding unit 202 of the trellis decoding device 200 according to one embodiment.

[0028] Fig.11 A flowchart is shown showing an example of the operation of the motion vector prediction unit 202E3 of the inter-frame decoding unit 202E of the trellis decoding device 200 according to one embodiment. Fig.12 This is a diagram showing a modified example of the functional blocks of the inter-frame decoding unit 202E of the basic trellis decoding unit 202 of the trellis decoding device 200 according to one embodiment.

[0029] Fig.13This is a diagram showing an example of the structure of "Basemesh submesh header syntax".

[0030] Fig.14 This is a diagram showing a modified example of the functional blocks of the basic trellis decoding unit 202 of the trellis decoding device 200 according to one embodiment.

[0031] Fig.15 It is a diagram showing an example of a table for explaining the operation of the basic mesh updating unit 202F.

[0032] Fig.16 It is a diagram showing an example of a table for explaining the operation of the basic mesh updating unit 202F.

[0033] Fig.17 It is a diagram showing an example of a table for explaining the operation of the basic mesh updating unit 202F.

[0034] Fig.18 1 is a diagram showing an example of a functional block of the sub-division unit 203 of the trellis decoding device 200 according to an embodiment.

[0035] Fig.19 This is a diagram showing an example of a functional block of a base trellis sub-dividing unit 203A of the sub-dividing unit 203 of the trellis decoding device 200 according to an embodiment.

[0036] Fig. 20 This is a diagram for explaining an example of a base plane division method implemented by a base plane division unit 203A5 of a base mesh sub-division unit 203A of a sub-division unit 203 of a mesh decoding device 200 according to an embodiment.

[0037] Fig.21 This is a flowchart showing an example of the operation of the base grid sub-dividing unit 203A of the sub-dividing unit 203 of the grid decoding device 200 according to one embodiment.

[0038] Fig. 22 1 is a diagram showing an example of a functional block of a sub-division grid adjustment unit 203B of the sub-division unit 203 of the grid decoding device 200 according to an embodiment.

[0039] Fig.23 1 is a diagram showing an example of a case where the side division point moving unit 701 of the sub division grid adjustment unit 203B of the sub division unit 203 of the grid decoding device 200 according to one embodiment moves the side division points on the base plane ABC.

[0040] Fig.241 is a diagram showing an example of a case where the sub-division plane X in the base plane is sub-divided again by the sub-division plane dividing unit 702 of the sub-division grid adjustment unit 203B of the sub-division unit 203 of the grid decoding device 200 according to one embodiment.

[0041] Fig.25 1 is a diagram showing an example of a case where all sub-division planes are sub-divided again by the sub-division plane dividing unit 702 of the sub-division grid adjusting unit 203B of the sub-division unit 203 of the trellis decoding device 200 according to one embodiment.

[0042] Fig.26 This is a diagram showing an example of functional blocks of the displacement amount decoding unit 206 of the trellis decoding device 200 according to one embodiment (a case where inter-frame prediction is performed in a spatial region).

[0043] Fig. 27 This is a diagram showing an example of the structure of a displacement bit stream.

[0044] Fig.28 This is a diagram showing an example of the syntax structure of DPS.

[0045] Fig.29 This is a diagram showing an example of the syntax structure of DPH.

[0046] Fig.28 This is a diagram for explaining an example of the correspondence relationship between sub-division vertices between a reference frame and a decoding target frame when inter-frame prediction is performed in a spatial region.

[0047] Fig.31 This is a diagram showing an example of a functional block of the displacement amount decoding unit 206 of the trellis decoding device 200 according to an embodiment (a case where inter-frame prediction is performed in the frequency domain).

[0048] Fig.32 This is a diagram for explaining an example of the frequency correspondence relationship between a reference frame and a decoding target frame when inter-frame prediction is performed in the frequency domain.

[0049] Fig.33 This is a flowchart showing an example of the operation of the displacement amount decoding unit 206 of the trellis decoding device 200 according to one embodiment.

[0050] Fig.34 This is a diagram showing an example of functional blocks of the displacement amount decoding unit 206 according to the first modification.

[0051] Fig.35 This is a diagram showing an example of functional blocks of the displacement amount decoding unit 206 involved in the second modification. DETAILED DESCRIPTION

[0052] Below, while referring to the attached Figure 1 The embodiments of the present invention are described below. In addition, the structural elements in the following embodiments can be appropriately replaced with existing structural elements, etc., and various changes including combinations with other existing structural elements can be made. Therefore, the content of the invention described in the claims is not limited based on the description of the following embodiments.

[0053] <First Embodiment>

[0054] Below, refer to Figure 1 to Figure 33 , the grid processing system involved in this embodiment is described.

[0055] Figure 1 1 is a diagram showing an example of the structure of the grid processing system 1 according to the present embodiment. Figure 1 As shown, the grid processing system 1 has a grid encoding device 100 and a grid decoding device 200 .

[0056] Figure 2 This is a diagram showing an example of functional blocks of the trellis decoding device 200 according to the present embodiment.

[0057] like Figure 2 As shown, the trellis decoding device 200 includes a demultiplexing unit 201 , a basic trellis decoding unit 202 , a sub-division unit 203 , a trellis decoding unit 204 , a patch integration unit 205 , a displacement decoding unit 206 and a video decoding unit 207 .

[0058] Here, the basic mesh decoding unit 202 , the sub-division unit 203 , the mesh decoding unit 204 , and the displacement decoding unit 206 are configured to perform processing in units of patches obtained by dividing the mesh, and then the patch integration unit 205 may integrate these processing results.

[0059] exist Figure 3A In the example, the mesh is divided into patch 1 consisting of base surfaces 1 and 2, and patch 2 consisting of base surfaces 3 and 4.

[0060] The demultiplexing unit 201 is configured to separate the multiplexed bit stream into a base grid bit stream, a displacement bit stream, and a texture bit stream.

[0061] <Basic grid decoding unit 202>

[0062] The base grid decoding unit 202 is configured to decode the base grid bit stream, generate and output a base grid.

[0063] Here, the basic mesh is composed of a plurality of vertices in a three-dimensional space and edges connecting the plurality of vertices.

[0064] In addition, if Figure 3AAs shown, the basic mesh is constructed by combining basic faces expressed by three vertices.

[0065] The basic grid decoding unit 202 can be configured to decode the basic grid bit stream using Draco as shown in non-patent document 2, for example.

[0066] In addition, the basic grid decoding unit 202 may be configured to generate “subdivision_method_id” described later as control information for controlling the type of the subdivision method.

[0067] Below, refer to Figure 4-5 , the control information decoded by the basic grid decoding unit 202 is explained.

[0068] Figure 4 This is a diagram showing an example of the syntax structure of the basic grid bit stream.

[0069] like Figure 4 As shown, first, the base grid bitstream may include a BPH (Base Patch Header) which is a collection of control information corresponding to the base grid patch. Second, the base grid bitstream may include base grid patch data encoding the base grid patch after the BPH.

[0070] As described above, the basic mesh bitstream has a structure in which each BPH corresponds to each patch data. Figure 4 The structure is just an example, and as long as the structure is such that BPH corresponds to each patch data, elements other than the above can be added as structural elements of the basic mesh bit stream.

[0071] For example, Figure 4 As shown, the basic mesh bitstream may include an SPS (Sequence Parameter Set), an FH (Frame Header) which is a set of control information corresponding to a frame, and an MH (Mesh Header) which is control information corresponding to a mesh.

[0072] Figure 5 is a diagram showing an example of the syntax structure of BPH. Here, as long as the syntax functions are the same, you can use Figure 5 The syntax companions shown have different syntax names.

[0073] exist Figure 5 In the syntax structure of the BPH shown, the descriptor column indicates how each syntax is encoded. In addition, ue(v) indicates an uncoded 0th-order exponential-Golomb code, and u(n) indicates an n-bit flag.

[0074] The BPH includes at least a control signal (mdu_face_count_minus1) that specifies the number of base faces contained in the base mesh patch.

[0075] In addition, the BPH includes at least a control signal (mdu_subdivision_method_id) that specifies the type of subdivision method of the base mesh for each base patch.

[0076] In addition, the BPH may include a control signal (mdu_subdivision_num_method_id) that specifies the type of subdivision number generation method for each base mesh patch. For example, when mdu_subdivision_num_method_id=0, it is defined as generating the subdivision number of the base surface by predicting the segmentation residual, and when mdu_subdivision_num_method_id=1, it is defined as recursively generating the subdivision number of the base surface.

[0077] When the BPH generates the number of subdivisions of the base plane using the prediction partition residual, it may include a control signal (mdu_subdivision_resuiduals) that specifies the prediction partition residual of the base plane for each index i (i=0, . . . , mdu_face_count_minus1).

[0078] When recursively generating the number of sub-divisions of the base surface, the BPH may include a control signal (mdu_max_depth) for identifying an upper limit of the number of sub-divisions to be recursively performed for each base mesh patch.

[0079] The BPH may include a control signal (mdu_subdivision_flag) that specifies whether to recursively sub-divide the base plane for each index i (i=0, ..., mdu_face_count_minus1) and j (j=0, ..., mdu_subdivision_depth_index).

[0080] like Figure 6 As shown, the basic mesh decoding unit 202 includes a separation unit 202A, an intra-frame decoding unit 202B, a mesh buffer unit 202C, a connection information decoding unit 202D, and an inter-frame decoding unit 202E.

[0081] The separation unit 202A is configured to classify the base grid bit stream into an I-frame bit stream and a P-frame bit stream.

[0082] (Intra-frame decoding unit 202B)

[0083] The intra-frame decoding unit 202B is configured to decode the coordinates and connection information of the vertices of the I frame from the bit stream of the I frame using, for example, Draco as described in Non-Patent Document 2.

[0084] Figure 7 This is a diagram showing an example of functional blocks of the intra-frame decoding unit 202B.

[0085] like Figure 7 As shown, the intra-frame decoding unit 202B includes an arbitrary intra-frame decoding unit 202B1 and a sorting unit 202B2.

[0086] The arbitrary intra-frame decoding unit 202B1 is configured to decode the coordinates and connection information of the non-sequential vertices of the I frame from the bit stream of the I frame using an arbitrary method including Draco described in Non-Patent Document 2.

[0087] The sorting unit 202B2 is configured to sort out the out-of-order vertices into a predetermined order and output the vertices.

[0088] As the prescribed order, for example, a Morton code order or a raster scan order may be used.

[0089] In addition, the sorting unit 202B2 may collect a plurality of vertices having the same coordinates in the decoded basic mesh, that is, duplicate vertices, into one vertex, and then rearrange the vertices into a predetermined order.

[0090] The mesh buffer unit 202C is configured to accumulate the coordinates and connection information of the vertices of the I frame decoded by the intra-frame decoding unit 202B. Here, a specific buffer may be provided to store pairs of indexes A(k) and B(k) of vertices existing as duplicate vertices in a predetermined order.

[0091] The connection information decoding unit 202D is configured to use the connection information of the I frame extracted from the mesh buffer unit 202C as the connection information of the P frame.

[0092] The inter-frame decoding unit 202E is configured to decode the coordinates of the vertices of the P frame by adding the coordinates of the vertices of the I frame taken out from the mesh buffer unit 202C to the motion vectors decoded from the bit stream of the P frame.

[0093] Furthermore, the inter-frame decoding unit 202E can adjust the indices of the vertices of the P frame using the pair of indices A(k) and B(k) of the vertices existing as duplicate vertices stored in the specific buffer.

[0094] In this embodiment, if Figure 8As shown, there is a correspondence between the vertices of the base mesh of the P frame and the vertices of the base mesh of the reference frame (I frame or P frame). Here, the motion vector decoded by the inter-frame decoding unit 202E is a differential vector between the coordinates of the vertices of the base mesh of the P frame and the coordinates of the vertices of the base mesh of the I frame.

[0095] (Inter-frame decoding unit 202E)

[0096] Fig. 9 This is a diagram showing an example of functional blocks of the inter-frame decoding unit 202E.

[0097] like Fig. 9 As shown, the inter-frame decoding unit 202E includes a motion vector residual decoding unit 202E1, a motion vector buffer unit 202E2, a motion vector prediction unit 202E3, a motion vector calculation unit 202E4, and an adder 202E5.

[0098] The motion vector residual decoding unit 202E1 is configured to generate MVR (Motion Vector Residual) from the bit stream of the P frame.

[0099] Here, MVR is a motion vector residual representing the difference between MV (Motion Vector) and MVP (Motion Vector Prediction). MV is a difference vector (motion vector) between the coordinates of the vertex of the corresponding I frame and the coordinates of the vertex of the P frame. MVP is the value of the MV of the vertex of the prediction target using MV (predicted value of the motion vector).

[0100] The motion vector buffer unit 202E2 is configured to sequentially store the MV output by the motion vector calculation unit 202E4.

[0101] The motion vector prediction unit 202E3 is configured to obtain the decoded MV from the motion vector buffer unit 202E2 for the vertex connected to the vertex of the decoding target, such as Fig.10 As shown, the MVP of the vertex of the decoding target is output using all or part of the acquired decoded MV.

[0102] The motion vector calculation unit 202E4 is configured to add the MVR generated by the motion vector residual decoding unit 202E1 to the MVP output from the motion vector prediction unit 202E3, and output the MV of the vertex of the decoding target.

[0103] The adder 202E5 is configured to add the coordinates of the vertex corresponding to the vertex of the decoding target obtained from the decoded basic mesh of the corresponding reference frame (I frame or P frame) and the motion vector MV output from the motion vector calculation unit 202E3, and output the coordinates of the vertex of the decoding target.

[0104] However, when there is no MVR data from the bit stream of the P frame, the inter-frame decoding unit 202E does not implement the processing of the motion vector residual decoding unit 202E1, the motion vector buffer unit 202E2, the motion vector prediction unit 202E3, the motion vector calculation unit 202E4 and the adder 202E5, but directly uses the coordinates of the vertices of the decoded basic mesh of the specified reference frame to decode the coordinates of the vertices of the basic mesh of the decoding target frame.

[0105] The following describes the details of each part of the inter-frame decoding unit 202E.

[0106] exist Fig.11 , a flowchart showing an example of the operation of the motion vector prediction unit 202E3 is shown.

[0107] like Fig.11 As shown, in step S1001, the motion vector prediction unit 202E3 sets MVP and N to 0.

[0108] In step S1002, the motion vector prediction unit 202E3 obtains a set of MVs of vertices around the vertex of the decoding target from the motion vector buffer unit 202E2, identifies vertices for which subsequent processing has not been completed, and switches to No; if subsequent processing has been completed for all vertices, switches to Yes.

[0109] In step S1003 , the motion vector prediction unit 202E3 switches to No when the MV of the vertex of the processing target has not been decoded, and switches to Yes when the MV of the vertex of the processing target has been decoded.

[0110] In step S1004, the motion vector prediction unit 202E3 adds MV and MVP, and increases N by 1.

[0111] In step S1005, the motion vector prediction unit 202E3 divides MVP by N when N is greater than 0 and outputs the result, and outputs 0 and terminates the process when N is 0.

[0112] That is, the motion vector prediction unit 202E3 is configured to output the MVP of the decoding target by averaging the decoded motion vectors of the vertices located around the vertex of the decoding target.

[0113] Furthermore, the motion vector prediction unit 202E3 may be configured to set the MVP to 0 when the set of decoded motion vectors is an empty set.

[0114] The motion vector calculation unit 202E4 may be configured to calculate the MV of the vertex of the decoding target based on the MVP output by the motion vector prediction unit 202E3 and the MVR generated by the motion vector residual decoding unit 202E1 using equation (1).

[0115] MV(k)=MVP(k)+MVR(k) … (1)

[0116] Here, k is the index of the vertex. MV, MVR, and MVP are vectors having an x ​​component, a y component, and a z component.

[0117] According to the above configuration, since MVP is used instead of MV and only MVR is encoded, the effect of improving the encoding efficiency can be expected.

[0118] The adder 202E5 is configured to calculate the coordinates of the vertex by adding the MV of the vertex calculated by the motion vector calculation unit 202E4 and the coordinates of the vertex of the reference frame corresponding to the vertex, thereby maintaining the connection information (Connectivity) in the reference frame.

[0119] Specifically, the adder 202E5 can be configured to use equation (2) to calculate the coordinate v' of the kth vertex. i (k) Perform calculations.

[0120] v' i (k) = v' j (k)+MV(k) … (2)

[0121] Here, v' i (k) is the coordinate of the kth vertex decoded in the decoding target frame, v' j (k) is the coordinate of the k-th vertex after decoding of the reference frame, MV(k) is the k-th MV of the frame of the decoding target, k=1, 2, . . . , K.

[0122] In addition, the connection information of the decoding target frame is the same as the connection information of the reference frame.

[0123] Furthermore, since the motion vector prediction unit 202E3 calculates the MVP using the decoded MV, the decoding order does not affect the MVP.

[0124] The decoding order is the order of decoding the vertices of the base mesh of the reference frame. Generally speaking, if a decoding method is used that uses a constant repetition pattern to increase the base surface one by one from the edge as the starting point, the order of the vertices of the decoded base mesh is determined by the decoding process.

[0125] For example, the motion vector prediction unit 202E3 may use Edgebreaker (compression algorithm) to determine the order of decoding vertices in the base mesh of the reference frame.

[0126] According to the above configuration, since the MV from the reference frame is encoded instead of the coordinates of the vertices, it is possible to expect an effect of improving the encoding efficiency.

[0127] (Example of Modification of Inter-frame Decoding Unit 202E)

[0128] like Fig.12 As shown, the motion vector residual decoding unit 202E1 is configured to generate the MVR and prediction mode of the motion vector of the decoding target vertex based on the bit stream of the P frame.

[0129] (Change Example 1)

[0130] In Modification 1, the motion vector residual decoding unit 202E1 is configured to generate a prediction mode of a decoding target vertex by selecting a context model according to the prediction mode of a motion vector of a decoded vertex and performing arithmetic decoding using the probability of the context model.

[0131] Furthermore, the motion vector residual decoding unit 202E1 is configured to update the probability of the context model according to the prediction mode of the decoding target vertex.

[0132] The motion vector residual decoding unit 202E1 can select the prediction mode of the previous vertex or vertex group in the decoding order as the prediction mode of the decoded motion vector, can select the prediction mode of the vertex closest to the decoding target vertex or the decoding target vertex group among the decoded vertices adjacent to the decoding target vertex or the decoding target vertex group, and can select the most frequently used prediction mode among the decoded vertices adjacent to the decoding target vertex or the decoding target vertex group.

[0133] Since the prediction mode is correlated with the surroundings, by introducing a context model, it is expected that the coding efficiency will be improved.

[0134] The motion vector prediction unit 202E3 is configured to use the motion vectors of the decoded vertices around the decoded target vertex, the motion vectors of the vertices in the reference frame corresponding to the decoded target vertex, and the motion vectors in the reference frame corresponding to the decoded vertices around the decoded target vertex, and calculate the predicted value (MVP) of the motion vector of the decoded target vertex using a prediction method specified by the prediction mode obtained from the motion vector residual decoding unit 202E1 from among multiple prediction methods.

[0135] Hereinafter, three modified examples of the method of calculating the MVP will be described with reference to modified examples 1-1 to 1-3.

[0136] (Change Example 1-1)

[0137] The MVP is a simple average of the MVs of the decoded surrounding vertices, but can be the MV of the nearest vertex.

[0138] That is, in variation 1-1, the motion vector prediction unit 202E3 is configured to calculate the distance between a vertex in a reference frame corresponding to a decoded vertex adjacent to a decoding target vertex (decoded vertices around the decoding target vertex) (a first vertex) and a vertex in a reference frame corresponding to the decoding target vertex (a second vertex), select the vertex in the reference frame (the first vertex) having the smallest distance, and use the motion vector of the vertex in the decoding target frame corresponding to the vertex in the selected reference frame (the first vertex) as the MVP of the decoding target vertex.

[0139] In this case, the motion vector prediction unit 202E3 can calculate the distances between the decoded vertices around the decoding target vertex and the decoding target vertex.

[0140] However, since the decoding of the MV and coordinates of the decoding target vertex has not been completed yet, the motion vector prediction section 202E3 cannot calculate the distance between the decoding target vertex and the decoded vertex.

[0141] Therefore, in the present modification example 1-1, the motion vector prediction unit 202E3 uses the distance between the vertices of the reference frame in which the correspondence relationship between the vertices is known.

[0142] According to the above configuration, since the MVP can be calculated with higher accuracy at the nearest vertex, the value of MVR can be reduced and concentrated near zero, and the effect of improving coding efficiency can be expected.

[0143] (Change Example 1-2)

[0144] In Modification 1-2, the motion vector prediction unit 202E3 is configured to calculate the MVP of the decoding target vertex of the decoding target frame by referring to other decoded inter-frames (P frames) having a one-to-one correspondence relationship with the decoding target frame.

[0145] For example, the motion vector prediction unit 202E3 may be configured to extract the motion vector of the vertex corresponding to the decoding target vertex from the above-mentioned decoded other inter-frames, and use the extracted motion vector of the vertex corresponding to the decoding target vertex as the MVP of the decoding target vertex.

[0146] In addition, the motion vector prediction unit 202E3 can be configured to extract the ratio relationship (first ratio relationship) between the MV and the MVP of the vertex corresponding to the decoding target vertex from other decoded inter-frames that have a one-to-one correspondence with the decoding target frame, and calculate the MVP of the decoding target vertex in the same manner as the ratio relationship between the MV and the MVP of the decoding target vertex and the first ratio relationship.

[0147] According to the above configuration, since the MVP is calculated more accurately using other inter-frames, the value of MVR is reduced and concentrated near zero, and the effect of improving the coding efficiency can be expected.

[0148] The motion vector prediction unit 202E3 may calculate MVPs for all vertices using the same prediction method, or may calculate multiple MVPs using multiple prediction methods.

[0149] In addition, when using a plurality of prediction methods, the motion vector prediction unit 202E3 may use a prediction method other than the above-described embodiment or modification example.

[0150] For example, the motion vector prediction unit 202E3 may use a method of always setting MVP to zero as the prediction method, or may use a method of not predicting MVP.

[0151] (Change Example 1-3)

[0152] In Modification 1-3, first, when the motion vector prediction unit 202E3 calculates and outputs a plurality of MVPs at each decoding target vertex using a plurality of prediction methods, the motion vector calculation unit 202E4 is configured to use a syntax specified in bitstream decoding as a prediction mode.

[0153] For example, the motion vector prediction unit 202E3 can decode the specified syntax. Fig.13 The "header syntax of the base grid sub-grid" is a syntax indicating which prediction method is the best and that the MVP calculated using the prediction method should be selected.

[0154] Secondly, the motion vector calculation unit 202E4 is configured to select a prediction method indicated by the above-mentioned predetermined syntax, and select an MVP indicated by the above-mentioned predetermined syntax from among a plurality of MVPs calculated at each decoding target vertex.

[0155] Thirdly, the motion vector calculation unit 202E4 is configured to calculate the MV of the decoding target vertex based on the MVP selected by the motion vector prediction unit 202E3 using the above-mentioned equation (1).

[0156] According to the above configuration, since the MVP can be calculated with higher accuracy by switching between a plurality of prediction methods, the value of MVR is reduced and concentrated near zero, thereby being expected to improve the coding efficiency.

[0157] Two modification examples of the method of switching between the plurality of prediction methods will be described below with reference to modification examples 1-3-1 to 1-3-2.

[0158] (Change Example 1-3-1)

[0159] In Modification 1-3-1, the motion vector calculation unit 202E4 is configured to use the same prediction mode (prediction mode indicating an optimal prediction method) in N consecutive decoding target vertices.

[0160] Therefore, unlike Modification 1-3 in which a prediction mode is selected for each decoding target vertex, in Modification 1-3-1, N consecutive decoding target vertices are regarded as one group, and a prediction mode is selected for each group.

[0161] According to the above configuration, by using the same prediction mode in groups (N consecutive decoding target vertices), the amount of code for the prediction mode can be reduced, and the effect of improving the encoding efficiency of the motion vector can be expected.

[0162] (Change Example 1-3-2)

[0163] In variation example 1-3-2, first, the motion vector calculation unit 202E4 is configured to use, based on the above-mentioned prescribed syntax decoded according to the above-mentioned bit stream, a prediction mode of a vertex corresponding to the decoding target vertex in other decoded inter-frames having a one-to-one correspondence with the decoding target frame as a prediction value of the prediction mode of the decoding target vertex.

[0164] Secondly, the motion vector calculation unit 202E4 is configured to calculate the difference between the above-mentioned bit stream decoding and the prediction value of the above-mentioned prediction mode.

[0165] Thirdly, the motion vector calculation unit 202E4 is configured to calculate the prediction mode of the decoding target vertex by adding the prediction mode prediction value and the difference.

[0166] (Example of Modification of Basic Grid Decoding Unit 202)

[0167] Hereinafter, a modification example of the basic grid decoding unit 202 will be described.

[0168] like Fig.14 As shown, the basic mesh decoding unit 202 includes a separation unit 202A, an intra-frame decoding unit 202B, a mesh buffer unit 202C, a connection information decoding unit 202D, an inter-frame decoding unit 202E, a basic mesh updating unit 202F and a skip decoding unit 202G.

[0169] (Jump decoding unit 202G)

[0170] The skip decoding unit 202G is configured to have at least one reference frame, extract a designated reference base mesh from a mesh buffer unit 202C that stores at least one base mesh for each reference frame, and directly use the coordinates of the vertices of the extracted reference base mesh to decode the coordinates of the vertices of the base mesh of the decoding target frame.

[0171] The skip decoding unit 202G specifies the designated reference basic grid using a control signal (smh_ref_index) having a syntax described later.

[0172] In the present embodiment, a frame in which the coordinates of the vertices of the reference base mesh are decoded directly using the coordinates of the vertices of the reference base mesh is referred to as an S frame.

[0173] According to the above configuration, since motion vectors are not required in the skip decoding unit 202G, a significant reduction in the amount of code and a significant reduction in the amount of calculation can be expected.

[0174] (Basic mesh updating unit 202F)

[0175] The base mesh update unit 202F is configured to update the vertex coordinates of the base mesh obtained from the decoded mesh using the value obtained by adding the vertex coordinates of the base mesh obtained from the decoded mesh to the motion vector and the displacement of the vertices of the base mesh, save the updated base mesh in the mesh buffer unit 202C, and update the reference frame list.

[0176] Here, the reference frame list is a list of indexes that identify all reference base meshes stored in the mesh buffer unit 202C.

[0177] Furthermore, when a plurality of base meshes are stored in the mesh buffer unit 202C in a certain frame, the base mesh updating unit 202F updates the reference frame list in the order in which the base meshes are stored.

[0178] For example, if the base grid 1 of the j-th frame is the k-th base grid in the grid buffer section 202C, the base grid 1 is specified as k in the reference frame list.

[0179] In addition, if the base grid 2 of the j-th frame is the (k+1)-th base grid in the grid buffer section 202C, the base grid 1 uses the (k+1)-th reference grid in the reference frame list.

[0180] The base mesh updating unit 202F updates the reference frame list by adding a base mesh to the mesh buffer unit 202C or deleting the base mesh from the mesh buffer unit 202C.

[0181] However, if there is only one base grid for a certain frame stored in the grid buffer unit 202C, the reference frame list may only include a frame index.

[0182] In addition, the basic mesh updating unit 202F Fig.15 The table shown or Fig.16 The table shown acquires the vertex coordinates and motion vectors of the base mesh and the displacement amounts of the vertices of the base mesh from the decoded mesh described above.

[0183] Here, Fig.15 The table shown is similar to Fig.16 The difference between the tables shown is only the displacement of the vertices of the base mesh. In addition, the displacement of the vertices of the base mesh is preferably Fig.15 The values ​​listed in the table shown.

[0184] In addition, the basic grid updating unit 202F can use the bit stream decoding representation to Fig.15 The table shown or Fig.16 Which one of the tables shown is marked?

[0185] However, the basic mesh updating unit 202F selects Fig.15 The table and Fig.16 In the case of any of the tables shown, both tables are stored in the mesh buffer unit 202C, and both are updated in the reference frame list.

[0186] Furthermore, the basic mesh updating unit 202F may or may not perform the above-mentioned updating when it is an S frame.

[0187] According to the above structure, since the base mesh updating unit 202F can calculate the base mesh with high quality, the amount of code of the motion vector and the amount of code of the displacement can be reduced in the next frame, and the decoded mesh is expected to be close to the original mesh.

[0188] (syntax)

[0189] exist Fig.13 In the syntax of the basemesh submesh header (Basemesh submesh headersyntax, BSH) shown, the descriptor column indicates how each syntax is encoded. In addition, ue(v) indicates an uncoded 0th order exponential Golomb code, and u(n) indicates an n-bit flag.

[0190] The BSH contains at least a control signal (smh_type) for specifying the type of the base grid subgrid contained in the base grid subgrid. The type of the base grid subgrid contains at least Fig.17 The values ​​and names are shown in Table 3.

[0191] In addition, when the type of the base mesh sub-mesh is SKIP_SUBMESH or P_SUBMESH, the BSH includes at least a control signal (smh_num_ref_idx_active_override_flag) indicating whether the control signal (smh_num_ref_idx_active_minus1) exists.

[0192] In addition, BSH includes at least a control signal (smh_num_ref_idx_active_minus1) for calculating a control signal (NumRefIdxActive) when the control signal (smh_num_ref_idx_active_override_flag) is 1. In addition, the control signal (NumRefIdxActive) is obtained by the following equation.

[0193]

[0194] In addition, the smh_type descriptor can be Fig.13 The ue(v) recorded in can also be U(8). Figure 6 and Fig.14 The frames in can be either mesh or submesh.

[0195] For example, Fig.17 The P_SUBMESH shown in Table 3 may be referred to as a P frame, the I_SUBMESH may be referred to as an I frame, and the SKIP_SUBMESHS may be referred to as an S frame.

[0196] In addition, when the decoding target frame is a P frame or an S frame, the BSH includes at least a control signal (smh_ref_index) for specifying a reference base grid.

[0197] However, in Fig.13 The "Syntax of the header of the base grid sub-grid, BSH" shown may include the condition of smh_num_ref_idx_active_minus1>0, or may not include the condition.

[0198] When the condition smh_num_ref_idx_active_minus1>0 is included, if smh_num_ref_idx_active_minus1 is zero, the control signal (smh_ref_index) is made zero.

[0199] <Sub-division Unit 203>

[0200] The sub-division unit 203 is configured to generate and output the added sub-division vertices and connection information thereof based on the basic mesh decoded by the basic mesh decoding unit 202 according to the sub-division method indicated by the control information.

[0201] Here, the base mesh, the added sub-division vertices, and the connection information thereof are collectively referred to as a “sub-division mesh”.

[0202] The sub-division unit 202 is configured to specify the type of sub-division method based on subdivision_method_id, which is control information generated by decoding the basic grid bit stream.

[0203] Below, refer to Figure 3A and Figure 3B The sub-dividing unit 202 will be described.

[0204] Figure 3A and Figure 3B This is a diagram for explaining an example of an operation of generating sub-division vertices from a base mesh.

[0205] Figure 3A This is a diagram showing an example of a basic mesh composed of five vertices.

[0206] Here, the subdivision may be performed using, for example, a mid-edge segmentation method that connects the midpoints of the edges of each base surface to each other, thereby segmenting one base surface into four surfaces.

[0207] Figure 3B An example of a sub-divided mesh formed by dividing a base mesh consisting of five vertices is shown. Figure 3B In the sub-division mesh shown, eight sub-division vertices (hollow circles) are generated in addition to the original five vertices (solid circles).

[0208] By decoding the displacement amount for each of the sub-divided vertices generated in this way by the displacement amount decoding unit 206 , it is expected that the encoding performance will be improved.

[0209] In addition, different sub-division methods may be applied to each patch. In this way, the displacement decoded by the displacement decoding unit 206 is adaptively changed for each patch, and improvement in encoding performance can be expected. Information on the divided patch is received as control information, namely, patch_id.

[0210] Below, refer to Fig.18 The sub-dividing unit 203 will be described. Fig.18 2 is a diagram showing an example of functional blocks of the sub-dividing unit 203 .

[0211] like Fig.18 As shown, the sub-division unit 203 includes a basic grid sub-division unit 203A and a sub-division grid adjustment unit 203B.

[0212] (Basic mesh sub-division unit 203A)

[0213] The base mesh sub-division unit 203A is configured to calculate the number of divisions (sub-division number) for each base surface and each base patch based on the input base mesh and the division information of the base mesh, sub-divide the base mesh according to the division number, and output the sub-divided surfaces.

[0214] That is, the base mesh sub-division unit 203A may be configured to be able to change the above-mentioned number of divisions in units of base surfaces and base patches.

[0215] Here, a base face is a face that constitutes a base mesh, and a base patch is a collection of several base faces.

[0216] In addition, the basic mesh sub-division unit 203A may be configured to calculate the number of sub-divisions of the basic surface by predicting the number of sub-re-divisions of the basic surface and adding the predicted number of sub-divisions of the basic surface to the residual of the predicted number of sub-divisions of the basic surface.

[0217] In addition, the basic mesh sub-division unit 203A may be configured to calculate the number of sub-divisions of the basic surface based on the number of sub-divisions of adjacent basic surfaces of the basic surface.

[0218] In addition, the basic mesh sub-division unit 203A may be configured to calculate the number of sub-divisions of the basic surface based on the previously accumulated number of sub-divisions of the basic surface.

[0219] In addition, the basic mesh sub-division unit 203A may be configured to generate vertices for dividing three sides constituting the basic surface and connect the generated vertices, thereby performing sub-division on the basic surface.

[0220] like Fig.18 As shown, the basic mesh sub-division unit 203A has 203B at the rear stage, which has a sub-division mesh adjustment unit described later.

[0221] Below, use Figure 19 to Figure 21 An example of the processing performed by the basic mesh sub-division unit 203A will be described.

[0222] Fig.19 is a diagram showing an example of a functional block of the basic mesh sub-division unit 203A. Fig.21 This is a flowchart showing an example of the operation of the basic mesh sub-division unit 203A.

[0223] like Fig.19 As shown, the basic mesh sub-division unit 203A includes a basic surface division number buffer unit 203A1, a basic surface division number reference unit 203A2, a basic surface division number prediction unit 203A3, an addition unit 203A4, and a basic surface division unit 203A5.

[0224] The base surface division number buffer unit 203A1 is configured to store the base surface division information including the division number of the base surface, and output the base surface division information to the base surface division number reference unit 203A2.

[0225] Here, the size of the base surface division number buffer unit 203A1 may be set to 1, and the previously accumulated base surface division number may be output to the base surface division number reference unit 203A2.

[0226] That is, the size of the base face division number buffer unit 203A1 may be set to 1 so that only the last decoded sub-division number (the sub-division number decoded before) is referenced.

[0227] The base plane division number reference unit 203A2 is configured to output "cannot be referenced" to the base plane division number prediction unit 203A3 when there is no adjacent base plane to the decoding target base plane or when there is an adjacent base plane to the decoding target base plane but the division number is uncertain.

[0228] On the other hand, when there is an adjacent base plane to the decoding target base plane and the number of divisions is determined, the base plane division number reference unit 203A2 is configured to output the number of divisions to the base plane division number prediction unit 203A3.

[0229] The base surface division number prediction unit 203A3 is configured to predict the number of divisions of the base surface (the number of sub-divisions) based on one or more input division numbers, and output the predicted number of divisions (predicted number of divisions) to the addition unit 203A4.

[0230] Here, the base surface division number prediction unit 203A3 is configured to output 0 to the addition unit 203A4 when only the input of "cannot be referenced" is input from the base surface division number reference unit 203A2.

[0231] Furthermore, the base surface division number prediction unit 203A3 may be configured to generate a predicted division number using any statistical value such as an average value, maximum value, minimum value, mode, etc. of the input division numbers when more than one division number is input.

[0232] Furthermore, the base surface division number prediction unit 203A3 may be configured to generate the number of divisions of the most adjacent surface as the predicted number of divisions when one or more numbers of divisions are input.

[0233] The adding unit 203A4 is configured to output, to the base plane partitioning unit 203A5, a number of divisions obtained by adding the prediction number of divisions residual decoded from the prediction residual bit stream and the prediction number of divisions acquired from the base plane partitioning number predicting unit 203A3.

[0234] The base surface dividing unit 203A5 is configured to divide the base surface into sub-divisions according to the number of divisions input from the adding unit 203A4.

[0235] Fig. 20 This is an example of dividing the base surface into nine parts. Fig. 20Describe the method of dividing the base surface of the base surface division unit 203A5.

[0236] The base surface division unit 203A5 generates points A_1, …, A_(N-1) that divide the side AB forming the base surface into N equal parts (N = 3).

[0237] Similarly, the base surface division unit 203A5 also divides the side BC or the side CA into N equal parts, and generates points B_1, …, B_(N-1), C_1, …, C_(N-1) respectively.

[0238] After that, the points on the sides AB, BC, and CA are called "side division points".

[0239] The base surface division unit 203A5 generates the sides A_i B_(N-i), B_i C_(N-i), C_i A_(N-i) for all i (i = 1, 2, …, N-1), and generates N 2 sub-division surfaces.

[0240] Next, refer to Fig.19 Describe the processing sequence of the base grid sub-division unit 203A.

[0241] In step S2201, the base grid sub-division unit 203A determines whether the re-division processing of the last base surface is completed. If the processing is completed, it ends; if not, it proceeds to step S2202.

[0242] In step S2202, the base grid sub-division unit 203A determines whether Depth < mdu_max_depth.

[0243] Here, Depth is a variable representing the current depth, with an initial value of 0, and mdu_max_depth represents the maximum depth determined for each base surface.

[0244] If the condition of step S2202 is satisfied, the processing sequence proceeds to step S2203; if the condition is not satisfied, the processing sequence returns to step S2201.

[0245] In step S2203, the base grid sub-division unit 203A determines whether the mdu_subdivision_flag of the current depth is 1.

[0246] If so, the processing sequence returns to step S2201; if not, the processing sequence proceeds to step S2204.

[0247] In step S2204, the base grid sub-division unit 203A further divides all the sub-division surfaces within the base surface.

[0248] Here, the basic mesh sub-dividing unit 203A sub-divides the basic surface when the basic surface has not been sub-divided once.

[0249] In addition, the sub-division method is the same as the method described in step S2204.

[0250] Specifically, when the base surface is not subdivided once, the base surface is Fig.18 When at least one sub-division is performed, the sub-division face is divided into N 2 In Fig.18 For example, the face composed of vertices A_2, B, and B_1 is further segmented to generate N 2 Noodles.

[0251] When the sub-division processing is completed, the processing sequence proceeds to step S2205.

[0252] In step S2205, the basic mesh sub-division unit 203A adds 1 to Depth, and the processing sequence returns to step S2202.

[0253] (Sub-division grid adjustment unit 203B)

[0254] Next, a specific example of the processing performed by the sub-division grid adjustment unit 203B will be described. Figures 22 to 25 An example of the processing performed by the sub-division grid adjustment unit 203B will be described.

[0255] Fig. 22 2 is a diagram showing an example of the functional blocks of the sub-division grid adjustment unit 203B.

[0256] like Fig. 22 As shown, the sub-division grid adjustment unit 203B includes an edge division point moving unit 701 and a sub-division surface dividing unit 702 .

[0257] (Edge Division Point Moving Unit 701)

[0258] The edge division point moving unit 701 is configured to move the edge division point of the base surface to any one of the edge division points of the adjacent base surface with respect to the input initial sub-division surface, and output the sub-division surface.

[0259] Fig.23 is an example of moving the edge splitting point on the base surface ABC. Fig.23 As shown, the side division point moving unit 701 may be configured to move the side division point of the base surface ABC to the side division point of the nearest adjacent base surface.

[0260] (Sub-dividing surface dividing unit 702)

[0261] The sub-divided plane dividing unit 702 is configured to sub-divide the input sub-divided plane again and output a decoded sub-divided plane.

[0262] Fig.24 1 is a diagram showing an example of a case where the sub-divided surface X in the base surface is further sub-divided.

[0263] like Fig.24 As shown, the sub-divided surface dividing unit 702 may be configured to generate new sub-divided surfaces within the base surface by connecting vertices constituting the sub-divided surfaces with edge division points of the adjacent base surface.

[0264] Fig.25 1 is a diagram showing an example of a case where the above-mentioned sub-division process is performed on all sub-division surfaces.

[0265] The mesh decoding unit 204 is configured to generate and output a decoded mesh using the sub-division mesh generated by the sub-division unit 203 and the displacement amount decoded by the displacement amount decoding unit 206 .

[0266] Specifically, the mesh decoding unit 204 is configured to generate a decoded mesh by adding the corresponding displacement amount to each sub-division vertex. Here, information on which sub-division vertex each displacement amount corresponds to is indicated by control information.

[0267] The patch integration unit 205 is configured to integrate and output a plurality of patches of the decoded meshes generated by the mesh decoding unit 206 .

[0268] Here, the patch segmentation method is defined by the grid encoding device 100. For example, the patch segmentation method may be configured to calculate the normal vector for each base surface in advance, select the base surface with the most similar normal vector from adjacent base surfaces, aggregate the two base surfaces into the same patch, and repeat the above sequence for the next base surface.

[0269] The video decoding unit 207 is configured to decode and output textures by video coding. For example, the video decoding unit 207 can use HEVC of Non-Patent Document 1.

[0270] <Displacement Decoding Unit 206>

[0271] The displacement amount decoding unit 206 is configured to decode the displacement amount bit stream and generate and output the displacement amount.

[0272] Figure 3B is a diagram showing an example of the displacement amount relative to a sub-division vertex. Figure 3B In the example of , since there are 8 sub-division vertices, the displacement amount decoding unit 206 is configured to define 8 displacement amounts expressed by scalars or vectors for each sub-division vertex.

[0273] Below, refer to Fig.26 , the displacement decoding unit 206 is explained. Fig.26 2 is a diagram showing an example of functional blocks of the displacement amount decoding unit 206 .

[0274] like Fig.26 As shown, the displacement amount decoding unit 206 includes a decoding unit 206A, an inverse quantization unit 206B, an inverse wavelet transform unit 206C, an adder 206D, an inter-frame prediction unit 206E, and a frame buffer 206F.

[0275] The decoding unit 206A is configured to decode and output the level value and control information by performing variable length decoding on the received displacement bit stream. Here, the level value obtained by variable length decoding is output to the inverse quantization unit 206B, and the control information is output to the inter-frame prediction unit 206E.

[0276] Below, refer to Fig. 27 , an example of the structure of the displacement bit stream is explained. Fig. 27 This is a diagram showing an example of the structure of a displacement bit stream.

[0277] like Fig. 27 As shown, first, the displacement bit stream may include a set of control information related to the decoding of the displacement, namely, DPS (Displacement Parameter Set).

[0278] Second, the displacement bitstream may include a set of control information corresponding to the patch, namely, DPH (DisplacementPatch Header).

[0279] Third, the displacement bitstream may contain the encoded displacements that make up the patch after the DPH.

[0280] As described above, the structure of the displacement bit stream is that DPH and DPS correspond to each encoded displacement.

[0281] also, Fig. 27 The structure is just an example. As long as DPH and DPS correspond to the encoded displacement amounts, elements other than the above may be added as structural elements of the displacement amount bit stream.

[0282] For example, Fig. 27 As shown, the displacement bit stream may include an SPS (Sequence Parameter Set).

[0283] Fig.28 This is a diagram showing an example of the syntax structure of DPS.

[0284] exist Fig.28In the descriptor column, the descriptor column indicates how each syntax is encoded.

[0285] In addition, Fig.28 In it, ue(v) represents the codeless 0th order exponential-Golomb code, and u(n) represents the n-bit tag.

[0286] When there are a plurality of DPSs, the DPS includes at least DPS id information (dps_displacement_parameter_set_id) for identifying each DPS.

[0287] In addition, the DPS may include a flag (interprediction_enabled_flag) for controlling whether to perform inter-frame prediction.

[0288] For example, when the interpretation_enabled_flag is 0, it can be defined as not performing inter-frame prediction, and when the interpretation_enabled_flag is 1, it can be defined as performing inter-frame prediction. When the interpretation_enabled_flag is not included, it can be defined as not performing inter-frame prediction.

[0289] DPS may include a flag (dct_enabled_flag) that controls whether to perform inverse DCT.

[0290] For example, when dct_enabled_flag is 0, it can be defined as not performing inverse DCT, and when dct_enabled_flag is 1, it can be defined as performing inverse DCT. When dct_enabled_flag is not included, it can be defined as not performing inverse DCT.

[0291] Fig.29 This is a diagram showing an example of the syntax structure of DPH.

[0292] like Fig.29 As shown, the DPH includes at least DPS id information for specifying the DPS corresponding to each DPH.

[0293] The inverse quantization unit 206B is configured to generate and output a transform coefficient by inverse quantizing the level value decoded by the decoding unit 206A.

[0294] The inverse wavelet transform unit 206C is configured to generate and output a prediction residual by performing an inverse wavelet transform on the transform coefficients generated by the inverse quantization unit 206B.

[0295] (Inter-frame prediction unit 206E)

[0296] The inter-frame prediction unit 206E is configured to perform inter-frame prediction using the decoded displacement of the reference frame read from the frame buffer 206F, and to generate and output a predicted displacement.

[0297] The inter-frame prediction unit 206E is configured to perform the inter-frame prediction only when interprediction_enabled_flag is 1.

[0298] The inter-frame prediction unit 206E can perform inter-frame prediction in the spatial domain or in the frequency domain. The inter-frame prediction can use a reference frame in the past and a reference frame in the future to perform bidirectional prediction.

[0299] When the inter-frame prediction unit 206E performs inter-frame prediction in the spatial region, it may directly refer to the decoded displacement of the corresponding sub-divided vertex in the reference frame to determine the predicted displacement of the sub-divided vertex in the target frame.

[0300] Alternatively, the predicted displacement of a sub-divided vertex in the target frame can be determined probabilistically using the decoded displacements of the corresponding sub-divided vertices in multiple reference frames according to a normal distribution with an estimated mean and dispersion. In this case, the dispersion can be uniquely determined to be zero only by the mean.

[0301] Alternatively, the predicted displacement of a sub-divided vertex in the target frame can be determined using decoded displacements of corresponding sub-divided vertices in multiple reference frames, based on a regression curve that estimates time using explanatory variables and displacement as target variables.

[0302] In the trellis coding apparatus 100, in order to improve the coding efficiency of each frame, the order of the decoding displacements may be rearranged.

[0303] In this case, the inter-frame prediction unit 206E may be configured to perform inter-frame prediction on the rearranged decoded displacement.

[0304] The correspondence relationship between the sub-division vertices between the reference frame and the decoding target frame is represented by the control information.

[0305] Fig.30 This is a diagram for explaining an example of the correspondence relationship between sub-division vertices between a reference frame and a decoding target frame when inter-frame prediction is performed in a spatial region.

[0306] Fig.31 This is an example of a functional block of the displacement amount decoding unit 206 when inter-frame prediction is performed in the frequency domain.

[0307] When the inter-frame prediction unit 206E performs inter-frame prediction in the frequency domain, it may directly refer to the decoded wavelet transform coefficient of the corresponding frequency in the reference frame to determine the predicted wavelet transform coefficient of the frequency in the decoding target frame.

[0308] The inter prediction unit 206E may perform inter prediction according to a normal distribution probability in which the average and the dispersion are estimated, using the decoded displacement amounts or decoded wavelet transform coefficients of the sub-division vertices in the plurality of reference frames.

[0309] The inter prediction unit 206E may perform inter prediction based on a regression curve that estimates time using explanatory variables and displacement amounts as target variables, using decoded displacement amounts or decoded wavelet transform coefficients of sub-divided vertices in a plurality of reference frames.

[0310] The inter-frame prediction unit 206E may be configured to perform inter-frame prediction bidirectionally using a temporally past reference frame and a future reference frame.

[0311] In the trellis encoding device 100, in order to improve the encoding efficiency of each frame, the order of the decoded wavelet transform coefficients may be rearranged.

[0312] The frequency correspondence between the reference frame and the decoding target frame is represented by the control information.

[0313] Fig.32 This is a diagram for explaining an example of the frequency correspondence relationship between a reference frame and a decoding target frame when inter-frame prediction is performed in the frequency domain.

[0314] Furthermore, when the sub-dividing unit 203 divides the base grid into a plurality of patches, the inter-frame prediction unit 206E also performs inter-frame prediction for each of the divided patches. This increases the temporal correlation between frames and can be expected to improve the encoding performance.

[0315] The adder 206D receives an input of a prediction residual from the inverse wavelet transform unit 206C and an input of a prediction displacement from the inter-frame prediction unit 206E.

[0316] The adder 206D is configured to calculate and output a decoded displacement by adding the prediction residual to the prediction displacement.

[0317] The decoded displacement calculated by the adder 206D is also output to the frame buffer 206F.

[0318] The frame buffer 206F is configured to acquire and accumulate the decoded displacement amount from the adder 206D.

[0319] Here, the frame buffer 206F outputs the decoded displacement of the corresponding vertex in the reference frame in response to control information (not shown).

[0320] Fig.33 This is a flowchart showing an example of the operation of the displacement amount decoding unit 206.

[0321] like Fig.33As shown, in step S3501, the displacement amount decoding unit 206 determines whether the processing is completed for all patches.

[0322] If yes, the action ends, and if no, the action proceeds to step S3502.

[0323] In step S3502 , the displacement decoding unit 206 performs inverse DCT on the patch of the decoding target, and then performs inverse quantization and inverse wavelet transform.

[0324] In step S3503 , the displacement amount decoding unit 206 determines whether interpretation_enabled_flag is 1.

[0325] In the case of yes, the action proceeds to step S3504, and in the case of no, the action returns to step S3501.

[0326] In step S3504, the displacement decoding unit 206 performs the above-mentioned inter-frame prediction and addition.

[0327] <Modification 1>

[0328] Below, refer to Fig.34 , focusing on the differences between the above-mentioned modification example 1 of the first embodiment and the above-mentioned first embodiment.

[0329] Fig.34 This is a diagram showing an example of functional blocks of the displacement amount decoding unit 206 according to the first modification.

[0330] like Fig.34 As shown, the displacement decoding unit 206 according to the first modification includes an inverse DCT unit 206G after the decoding unit 206A, that is, between the decoding unit 206A and the inverse quantization unit 206B.

[0331] That is, in the present modification example 1, the inverse quantization unit 206B is configured to generate a prediction residual by performing an inverse wavelet transform on the level value output from the inverse DCT unit 202G.

[0332] <Modification 2>

[0333] Below, refer to Fig.35 , focusing on the differences between the modified example 2 of the above-mentioned first embodiment and the above-mentioned first embodiment.

[0334] like Fig.35 As shown, the displacement amount decoding unit 206 according to the second modification includes a video decoding unit 2061 , an image development unit 2062 , an inverse quantization unit 2063 , and an inverse wavelet transformation unit 2064 .

[0335] The video decoding unit 2061 is configured to output a video by decoding the displacement bit stream received by the video encoding.

[0336] For example, the video decoding unit 2061 can use HEVC of Non-Patent Document 1.

[0337] In addition, the video decoding unit 2061 may use a video encoding method that always sets the motion vector to 0. For example, the video decoding unit 2061 may always set the motion vector of HEVC to 0 and use inter-frame prediction that is always at the same position.

[0338] In addition, the video decoding unit 2061 may use a video coding method that always skips transformation. For example, the video decoding unit 2061 may always set the transformation of HEVC to the transformation skip mode and use the video coding method without performing transformation.

[0339] The image expansion unit 2062 is configured to expand the image decoded by the image decoding unit 2061 as a level value for each image (frame) and output the expanded image.

[0340] In the above-described expansion method, the image expansion unit 2062 can perform back-calculation and identification based on the arrangement method of the level values ​​with respect to the image indicated by the control information.

[0341] The image expansion unit 2062 may, for example, arrange the level values ​​from high-frequency components to low-frequency components in the image in a raster operation order as a method of arranging the level values.

[0342] The inverse quantization unit 2063 is configured to generate and output a transform coefficient by inverse quantizing the level value generated by the image development unit 2062 .

[0343] The inverse wavelet transform unit 2064 is configured to generate and output a decoded displacement by performing an inverse wavelet transform on the transform coefficients generated by the inverse quantization unit 2063 .

[0344] The above-mentioned trellis encoding device 100 and trellis decoding device 200 can be implemented as a program product that causes a computer to execute each function (each process).

[0345] Industrial Applicability

[0346] In addition, according to this embodiment, for example, since comprehensive service quality can be improved in moving image communications, it can contribute to Goal 9 of the Sustainable Development Goals (SDGs) led by the United Nations, "Improve the infrastructure of the information environment, promote sustainable industrialization and achieve expansion of innovation."

[0347] Explanation of symbols

[0348] 1: Grid processing system;

[0349] 100: grid encoding device;

[0350] 200: grid decoding unit;

[0351] 201: multiplexing and separation unit;

[0352] 202: basic grid decoding unit;

[0353] 202A: separation part;

[0354] 202B: intra-frame decoding unit;

[0355] 202B1: arbitrary intra-frame decoding unit;

[0356] 202B2: Sorting Department;

[0357] 202C: grid buffer section;

[0358] 202D: connection information decoding unit;

[0359] 202E: inter-frame decoding unit;

[0360] 202E1: motion vector decoding unit;

[0361] 202E2: motion vector buffer unit;

[0362] 202E3: Motion vector prediction unit;

[0363] 202E4: motion vector calculation unit;

[0364] 202E5: adder;

[0365] 202F: jump decoding unit;

[0366] 202G: Basic Grid Update Department;

[0367] 203: sub-division part;

[0368] 203A: basic grid subdivision part;

[0369] 203A1: basic surface division number buffer unit;

[0370] 203A2: Reference to the number of divisions of the base surface;

[0371] 203A3: Basic surface segmentation number prediction unit;

[0372] 203A4: Addition department;

[0373] 203A5: basic surface segmentation;

[0374] 203B: sub-division grid adjustment unit;

[0375] 701: edge segmentation point moving unit;

[0376] 702: sub-division surface division part;

[0377] 204: grid decoding unit;

[0378] 205: Patch Integration Department;

[0379] 206: displacement decoding unit;

[0380] 206A: decoding unit;

[0381] 206B, 2063: Dequantization Department;

[0382] 206C, 2064: inverse wavelet transform unit;

[0383] 206D: Adder;

[0384] 206E: inter-frame prediction unit;

[0385] 206F: frame buffer;

[0386] 2062: Image development unit;

[0387] 207, 2061: Image decoding unit.

Claims

1. A grid decoding device, characterized in that: have: A motion vector residual decoding unit, which generates a motion vector residual and a prediction mode of the motion vector according to a bit stream of an inter-frame; a motion vector prediction unit that calculates a predicted value of the motion vector of the decoding target vertex by a prediction method specified by the prediction mode from among a plurality of prediction methods, using motion vectors of decoded vertices around the decoding target vertex, motion vectors of vertices in a reference frame corresponding to the decoding target vertex, and motion vectors of vertices in the reference frame corresponding to the decoded vertices around the decoding target vertex; as well as A motion vector calculation unit adds the predicted value of the motion vector to the motion vector residual.

2. The grid decoding device according to claim 1, characterized in that: The motion vector prediction section calculates a distance between a first vertex in the reference frame corresponding to a decoded vertex around the decoding target vertex and a second vertex in the reference frame corresponding to the decoding target vertex, Select the first vertex with the smallest distance, The motion vector of the vertex in the decoding target frame corresponding to the selected first vertex is used as a predicted value of the motion vector of the decoding target vertex.

3. The grid decoding device according to claim 1, characterized in that: The motion vector prediction unit extracts a motion vector of a vertex corresponding to the decoding target vertex from another decoded inter-frame having a one-to-one correspondence relationship with the decoding target frame. The extracted motion vector is used as a predicted value of the motion vector of the decoding target vertex.

4. The grid decoding device according to claim 1, characterized in that: The motion vector prediction unit extracts a first ratio relationship between a motion vector of a vertex corresponding to the decoding target vertex and a predicted value of the motion vector of the vertex corresponding to the decoding target vertex from other decoded inter-frames having a one-to-one correspondence relationship with the decoding target frame, The predicted value of the motion vector of the decoding target vertex is calculated in such a manner that a ratio relationship between the motion vector of the decoding target vertex and the predicted value of the motion vector of the decoding target vertex is the same as the first ratio relationship.

5. The grid decoding device according to claim 1, characterized in that: The motion vector calculation section uses the same prediction mode in consecutive N decoding target vertices.

6. The grid decoding device according to claim 1, characterized in that: The motion vector calculation unit uses the prediction mode of the vertex corresponding to the decoding target vertex in other decoded inter-frames having a one-to-one correspondence with the decoding target frame as the prediction value of the prediction mode of the decoding target vertex based on the specified syntax decoded according to the bit stream, Decoding the bitstream based on the difference between the prediction value and the prediction mode, The prediction mode of the decoding target vertex is calculated by adding the prediction value of the prediction mode to the difference.

7. The trellis decoding device according to claim 1, characterized in that: The motion vector residual decoding unit selects a context model according to the prediction mode of the decoded vertex motion vector, By performing arithmetic decoding using the probability of the context model, a prediction mode of the decoding target vertex is generated, The probability of the context model is updated by the prediction mode of the decoded target vertex.

8. A grid decoding method, characterized in that: have: The step of generating a motion vector residual and a prediction mode of the motion vector according to a bit stream of an inter-frame; A step of calculating a predicted value of the motion vector of the decoding target vertex by a prediction method specified by the prediction mode from among a plurality of prediction methods, using motion vectors of decoded vertices around the decoding target vertex, motion vectors of vertices in a reference frame corresponding to the decoding target vertex, and motion vectors of vertices in the reference frame corresponding to the decoded vertices around the decoding target vertex; as well as A step of adding the predicted value of the motion vector to the motion vector residual.

9. A program product, comprising a computer program, which, when executed, implements the function of a trellis decoding device, characterized in that: The grid decoding device comprises: A motion vector residual decoding unit, which generates a motion vector residual and a prediction mode of the motion vector according to a bit stream of an inter-frame; a motion vector prediction unit that calculates a predicted value of the motion vector of the decoding target vertex by a prediction method specified by the prediction mode from among a plurality of prediction methods, using motion vectors of decoded vertices around the decoding target vertex, motion vectors of vertices in a reference frame corresponding to the decoding target vertex, and motion vectors of vertices in the reference frame corresponding to the decoded vertices around the decoding target vertex; as well as A motion vector calculation unit adds the predicted value of the motion vector to the motion vector residual.

Images