Mesh decoding device, mesh encoding device, mesh decoding method, and program product

Abstract

In a mesh decoding device (200) according to the present invention, a displacement amount decoding unit (206) has: a decoding unit (206A) that decodes a level value on the basis of a displacement amount bit stream; an inverse quantization unit (206B) that generates a transform coefficient by inverse quantization of the level value; a reverse wavelet transform unit (206C) that generates a prediction residual by performing a reverse wavelet transform on the transform coefficient; and an adder (206D) that calculates the amount of displacement by adding the predicted amount of displacement and the predicted residual.

Description

Technical Field

[0001] The present invention relates to a grid decoding device, a grid encoding device, a grid decoding method and a program product. Background Art

[0002] Non-patent document 1 discloses a technology for encoding a grid using non-patent document 2.

[0003] Prior art literature

[0004] Patent Literature

[0005] Non-patent document 1: Cfp for Dynamic Mesh Coding, ISO / IEC JTC1 / SC29 / WG7 N00231, MPEG136–Online Non-patent document 2: Google Draco, accessed on May 26, 2022 [Online], https: / / google.github.io / draco Summary of the invention

[0006] Problems to be solved by the invention

[0007] However, in the prior art, since the coordinates or connection information of all vertices constituting the dynamic mesh are losslessly encoded, the amount of information cannot be reduced even under the condition of allowing loss, resulting in a problem of low coding efficiency. Therefore, the present invention is made in view of the above-mentioned problems, and its purpose is to provide a mesh decoding device, a mesh encoding device, a mesh decoding method and a program product that can improve the coding efficiency of the mesh.

[0008] Means for solving problems

[0009] The first technical solution of the present invention is a grid decoding device, the main purpose of which is to have: a displacement decoding unit, which is configured to decode the displacement bit stream, generate and output the displacement, and the displacement decoding unit has: a decoding unit, which is configured to decode the level value according to the displacement bit stream; a dequantization unit, which is configured to generate a transform coefficient by dequantizing the level value; an inverse wavelet transform unit, which is configured to generate a prediction residual by performing an inverse wavelet transform on the transform coefficient; and an adder, which is configured to calculate the displacement by adding the predicted displacement to the prediction residual.

[0010] The second technical solution of the present invention is a grid decoding method, which mainly comprises: a process of decoding a level value according to a displacement bit stream; a process of generating a transform coefficient by inverse quantizing the level value; a process of generating a prediction residual by performing an inverse wavelet transform on the transform coefficient; and a process of calculating the displacement by adding the predicted displacement to the prediction residual.

[0011] The third technical solution of the present invention is a program product, the main purpose of which is that it includes a program that enables a computer to function as a grid decoding device, the grid decoding device has a displacement decoding unit, and the displacement decoding unit is configured to decode the displacement bit stream, generate and output the displacement, and the displacement decoding unit has: a decoding unit, which is configured to decode the level value according to the displacement bit stream; a dequantization unit, which is configured to generate a transform coefficient by dequantizing the level value; an inverse wavelet transform unit, which is configured to generate a prediction residual by performing an inverse wavelet transform on the transform coefficient; and an adder, which is configured to calculate the displacement by adding the predicted displacement to the prediction residual.

[0012] Effects of the Invention

[0013] According to the present invention, a grid decoding device, a grid encoding device, a grid decoding method and a program product that can improve the coding efficiency of the grid can be provided. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a diagram showing an example of the configuration of a mesh processing system 1 according to an embodiment.

[0015] Figure 2 This is a diagram showing an example of functional blocks of a trellis decoding device 200 according to an embodiment.

[0016] Figure 3A This is a diagram showing an example of a basic grid and sub-divided grids.

[0017] Figure 3B This is a diagram showing an example of a basic grid and sub-divided grids.

[0018] Figure 4 This is a diagram showing an example of the syntax structure of the basic grid bit stream.

[0019] Figure 5 This is a diagram showing an example of the syntax structure of BPH.

[0020] Figure 6 This is a diagram showing an example of a functional block of the basic trellis decoding unit 202 of the trellis decoding device 200 according to one embodiment.

[0021] Figure 7 This is a diagram showing an example of the correspondence relationship between the vertices of the base mesh of the P frame and the vertices of the base mesh of the I frame.

[0022] Figure 8 This is a diagram showing an example of a functional block of the inter-frame decoding unit 202E of the basic trellis decoding unit 202 of the trellis decoding device 200 according to one embodiment.

[0023] Fig. 9 This is a diagram for explaining an example of a method for calculating the MVP of a vertex of a decoding target implemented by the motion vector prediction unit 202E3 of the inter decoding unit 202E of the base trellis decoding unit 202 of the trellis decoding device 200 according to one embodiment.

[0024] Fig.10 A flowchart is shown which illustrates an example of the operation of the motion vector prediction unit 202E3 of the inter-frame decoding unit 202E of the trellis decoding device 200 according to one embodiment.

[0025] Fig.11 A flowchart is shown which illustrates an example of the operation of calculating the sum Total_D of distances to decoded surrounding vertices in the motion vector prediction unit 202E3 of the inter-frame decoding unit 202 of the trellis decoding device 200 according to one embodiment.

[0026] Fig.12 This is a flowchart showing an example of an operation of calculating MVP using weighted average by the motion vector prediction unit 202E3 of the inter-frame decoding unit 202E of the trellis decoding device 200 according to one embodiment.

[0027] Fig.13 This is a flowchart showing an example of an operation of the motion vector prediction unit 202E3 of the inter decoding unit 202E of the base trellis decoding unit 202 of the trellis decoding device 200 according to one embodiment to select an MV as the MVP from a set of candidate MVs.

[0028] Fig.14 This is a flowchart showing an example of the operation of the motion vector prediction unit 202E3 of the inter-frame decoding unit 202E of the trellis decoding device 200 according to one embodiment to generate a set of candidate MVs.

[0029] Fig.15 This is a diagram for explaining an example of parallelogram prediction.

[0030] Fig.16This is a flowchart showing an example of an operation of returning the precision of MVR to the original precision based on adaptive_mesh_flag, adaptive_bit_flag and precision control parameters, which are control information generated by decoding the base mesh bit stream.

[0031] Fig.17 This is a schematic diagram for explaining an example of MVR encoding.

[0032] Fig.18 This is a diagram showing an example of a functional block of the inter-frame decoding unit 202E of the basic trellis decoding unit 202 of the trellis decoding device 200 according to one embodiment.

[0033] Fig.19 This diagram shows an example of the operation of determining connection information and the order of vertices using Edgebreaker.

[0034] Fig. 20 1 is a diagram showing an example of a functional block of the sub-division unit 203 of the trellis decoding device 200 according to an embodiment.

[0035] Fig.21 This is a diagram showing an example of a functional block of a base trellis sub-dividing unit 203A of the sub-dividing unit 203 of the trellis decoding device 200 according to an embodiment.

[0036] Fig. 22 This is a diagram for explaining an example of a base plane division method implemented by the base plane division unit 203A5 of the base mesh sub-division unit 203A of the sub-division unit 203 of the mesh decoding device 200 according to one embodiment.

[0037] Fig.23 This is a flowchart showing an example of the operation of the base grid sub-dividing unit 203A of the sub-dividing unit 203 of the grid decoding device 200 according to one embodiment.

[0038] Fig.24 1 is a diagram showing an example of a functional block of a sub-division grid adjustment unit 203B of the sub-division unit 203 of the grid decoding device 200 according to an embodiment.

[0039] Fig.25 1 is a diagram showing an example of a case where the side division point moving unit 701 of the sub division grid adjustment unit 203B of the sub division unit 203 of the grid decoding device 200 according to one embodiment moves the side division points on the base plane ABC.

[0040] Fig.261 is a diagram showing an example of a case where the sub-division plane X in the base plane is sub-divided again by the sub-division plane dividing unit 702 of the sub-division grid adjustment unit 203B of the sub-division unit 203 of the grid decoding device 200 according to one embodiment.

[0041] Fig. 27 1 is a diagram showing an example of a case where all sub-division planes are sub-divided again by the sub-division plane dividing unit 702 of the sub-division grid adjusting unit 203B of the sub-division unit 203 of the trellis decoding device 200 according to one embodiment.

[0042] Fig.28 This is a diagram showing an example of functional blocks of the displacement amount decoding unit 206 of the trellis decoding device 200 according to one embodiment (a case where inter-frame prediction is performed in a spatial region).

[0043] Fig.29 This is a diagram showing an example of the structure of a displacement bit stream.

[0044] Fig.30 This is a diagram showing an example of the syntax structure of DPS.

[0045] Fig.31 This is a diagram showing an example of the syntax structure of DPH.

[0046] Fig.32 This is a diagram for explaining an example of the correspondence relationship between sub-division vertices between a reference frame and a decoding target frame when inter-frame prediction is performed in a spatial region.

[0047] Fig.33 This is a diagram showing an example of a functional block of the displacement amount decoding unit 206 of the trellis decoding device 200 according to an embodiment (a case where inter-frame prediction is performed in the frequency domain).

[0048] Fig.34 This is a diagram for explaining an example of the frequency correspondence relationship between a reference frame and a decoding target frame when inter-frame prediction is performed in the frequency domain.

[0049] Fig.35 This is a flowchart showing an example of the operation of the displacement amount decoding unit 206 of the trellis decoding device 200 according to one embodiment.

[0050] Fig.36 This is a diagram showing an example of functional blocks of the displacement amount decoding unit 206 according to the first modification.

[0051] Fig.37 This is a diagram showing an example of functional blocks of the displacement amount decoding unit 206 involved in the second modification.

[0052] Fig.38This is a diagram showing an example of a functional block of the inter-frame decoding unit 202E of the basic trellis decoding unit 202 of the trellis decoding device 200 according to one embodiment.

[0053] Fig.39 This is a diagram showing an example of a functional block of the intra-frame decoding unit 202B of the base trellis decoding unit 202 of the trellis decoding device 200 according to one embodiment.

[0054] Explanation of symbols

[0055] 1: grid processing system; 100: grid coding device; 200: grid decoding unit; 201: multiplexing separation unit; 202: basic grid decoding unit; 202A: separation unit; 202B: intra-frame decoding unit; 202B1: arbitrary intra-frame decoding unit; 202B2: sorting unit; 202C: grid buffer unit; 202D: connection information decoding unit; 202E: inter-frame decoding unit; 202E1: motion vector decoding unit; 202E2: motion vector buffer unit; 202E3: motion vector prediction unit; 202E4: motion vector calculation unit; 202E5: adder; 202E6: duplicate vertex search unit; 202E7: duplicate vertex judgment unit; 202E8: motion vector acquisition unit; 203 : Sub-division unit; 203A: basic grid sub-division unit; 203A1: basic surface division number buffer unit; 203A2: basic surface division number reference unit; 203A3: basic surface division number prediction unit; 203A4: addition unit; 203A5: basic surface division unit; 203B: sub-division grid adjustment unit; 204: grid decoding unit; 205: patch integration unit; 206: displacement decoding unit; 206A: decoding unit; 206B, 2063: inverse quantization unit; 206C, 2064: inverse wavelet transform unit; 206D: adder; 206E: inter-frame prediction unit; 206F: frame buffer; 206G: inverse DCT unit; 2062: image expansion unit; 207, 2061: image decoding unit. DETAILED DESCRIPTION

[0056] Below, while referring to the attached Figure 1 The embodiments of the present invention are described below. In addition, the structural elements in the following embodiments can be appropriately replaced with existing structural elements, etc., and various changes including combinations with other existing structural elements can be made. Therefore, the content of the invention described in the claims is not limited based on the description of the following embodiments.

[0057] <First Embodiment>

[0058] Below, refer to Figure 1 to Figure 35 , the grid processing system involved in this embodiment is described.

[0059] Figure 1 1 is a diagram showing an example of the structure of the grid processing system 1 according to the present embodiment. Figure 1 As shown, the grid processing system 1 has a grid encoding device 100 and a grid decoding device 200 .

[0060] Figure 2 This is a diagram showing an example of functional blocks of the trellis decoding device 200 according to the present embodiment.

[0061] like Figure 2 As shown, the trellis decoding device 200 includes a demultiplexing unit 201 , a basic trellis decoding unit 202 , a sub-division unit 203 , a trellis decoding unit 204 , a patch integration unit 205 , a displacement decoding unit 206 and a video decoding unit 207 .

[0062] Here, the basic mesh decoding unit 202 , the sub-division unit 203 , the mesh decoding unit 204 , and the displacement decoding unit 206 are configured to perform processing in units of patches obtained by dividing the mesh, and then the patch integration unit 205 may integrate the processing results.

[0063] exist Figure 3A In the example, the mesh is divided into patch 1 consisting of base surfaces 1 and 2, and patch 2 consisting of base surfaces 3 and 4.

[0064] The demultiplexing unit 201 is configured to separate the multiplexed bit stream into a base grid bit stream, a displacement bit stream, and a texture bit stream.

[0065] <Basic grid decoding unit 202>

[0066] The base grid decoding unit 202 is configured to decode the base grid bit stream, generate and output a base grid.

[0067] Here, the basic mesh is composed of a plurality of vertices in a three-dimensional space and edges connecting the plurality of vertices.

[0068] In addition, if Figure 3A As shown, the basic mesh is constructed by combining basic faces expressed by three vertices.

[0069] The basic grid decoding unit 202 can be configured to decode the basic grid bit stream using Draco as shown in non-patent document 2, for example.

[0070] In addition, the basic grid decoding unit 202 may be configured to generate “subdivision_method_id” described later as control information for controlling the type of the subdivision method.

[0071] Below, refer to Figure 4-5, the control information decoded by the basic grid decoding unit 202 is explained.

[0072] Figure 4 This is a diagram showing an example of the syntax structure of the basic grid bit stream.

[0073] like Figure 4 As shown, first, the base grid bitstream may include a BPH (Base Patch Header) which is a collection of control information corresponding to the base grid patch. Second, the base grid bitstream may include base grid patch data encoding the base grid patch after the BPH.

[0074] As described above, the basic mesh bitstream has a structure in which each BPH corresponds to each patch data. Figure 4 The structure is just an example, and as long as the structure is such that BPH corresponds to each patch data, elements other than the above-mentioned ones can be added as structural elements of the basic mesh bit stream.

[0075] For example, Figure 4 As shown, the basic mesh bitstream may include an SPS (Sequence Parameter Set), an FH (Frame Header) which is a set of control information corresponding to a frame, and an MH (Mesh Header) which is control information corresponding to a mesh.

[0076] Figure 5 is a diagram showing an example of the syntax structure of BPH. Here, as long as the syntax functions are the same, you can use Figure 5 The syntax names shown are different from the syntax names.

[0077] exist Figure 5 In the syntax structure of the BPH shown, the descriptor column indicates how each syntax is encoded. In addition, ue(v) indicates an uncoded 0th-order exponential-Golomb code, and u(n) indicates an n-bit flag.

[0078] The BPH includes at least a control signal (mdu_face_count_minus1) that specifies the number of base faces contained in the base mesh patch.

[0079] In addition, the BPH includes at least a control signal (mdu_subdivision_method_id) for specifying the type of subdivision method of the base mesh for each base patch.

[0080] In addition, the BPH may include a control signal (mdu_subdivision_num_method_id) that specifies the type of subdivision number generation method for each base mesh patch. For example, when mdu_subdivision_num_method_id=0, it is defined as generating the subdivision number of the base surface by predicting the segmentation residual, and when mdu_subdivision_num_method_id=1, it may be defined as recursively generating the subdivision number of the base surface.

[0081] When the BPH generates the number of subdivisions of the base plane using the prediction partition residual, it may include a control signal (mdu_subdivision_resuiduals) that specifies the prediction partition residual of the base plane for each index i (i=0, . . . , mdu_face_count_minus1).

[0082] When recursively generating the number of sub-divisions of the base surface, the BPH may include a control signal (mdu_max_depth) for identifying an upper limit of the number of sub-divisions to be recursively performed for each base mesh patch.

[0083] The BPH may include a control signal (mdu_subdivision_flag) that specifies whether to recursively sub-divide the base plane for each index i (i=0, ..., mdu_face_count_minus1) and j (j=0, ..., mdu_subdivision_depth_index).

[0084] like Figure 6 As shown, the basic mesh decoding unit 202 includes a separation unit 202A, an intra-frame decoding unit 202B, a mesh buffer unit 202C, a connection information decoding unit 202D, and an inter-frame decoding unit 202E.

[0085] The separation unit 202A is configured to classify the base grid bit stream into a bit stream of an I frame (reference frame) and a bit stream of a P frame.

[0086] (Intra-frame decoding unit 202B)

[0087] The intra-frame decoding unit 202B is configured to decode the coordinates and connection information of the vertices of the I frame from the bit stream of the I frame using, for example, Draco as described in Non-Patent Document 2.

[0088] Fig.39 This is a diagram showing an example of functional blocks of the intra-frame decoding unit 202B.

[0089] like Fig.39 As shown, the intra-frame decoding unit 202B includes a separation unit 202A, an arbitrary intra-frame decoding unit 202B1, and a sorting unit 202B2.

[0090] The arbitrary intra-frame decoding unit 202B1 is configured to decode the coordinates and connection information of the non-sequential vertices of the I frame from the bit stream of the I frame using an arbitrary method including Draco described in Non-Patent Document 2.

[0091] The sorting unit 202B2 is configured to sort out the out-of-order vertices into a predetermined order and output the vertices.

[0092] As the prescribed order, for example, a Morton code order or a raster scan order may be used.

[0093] In addition, multiple vertices with the same coordinates, that is, repeated vertices, can be grouped as one vertex and then rearranged in a predetermined order.

[0094] The mesh buffer unit 202C is configured to accumulate the coordinates and connection information of the vertices of the I frame decoded by the intra-frame decoding unit 202B.

[0095] The connection information decoding unit 202D is configured to use the connection information of the I frame extracted from the mesh buffer unit 202C as the connection information of the P frame.

[0096] The inter-frame decoding unit 202E is configured to decode the coordinates of the vertices of the P frame by adding the coordinates of the vertices of the I frame taken out from the mesh buffer unit 202C to the motion vectors decoded from the bit stream of the P frame.

[0097] In this embodiment, if Figure 7 As shown, there is a correspondence between the vertices of the base mesh of the P frame and the vertices of the base mesh of the I frame (reference frame). Here, the motion vector decoded by the inter-frame decoding unit 202E is a differential vector between the coordinates of the vertices of the base mesh of the P frame and the coordinates of the vertices of the base mesh of the I frame.

[0098] (Inter-frame decoding unit 202E)

[0099] Figure 8 This is a diagram showing an example of functional blocks of the inter-frame decoding unit 202E.

[0100] like Figure 8 As shown, the inter-frame decoding unit 202E includes a motion vector residual decoding unit 202E1, a motion vector buffer unit 202E2, a motion vector prediction unit 202E3, a motion vector calculation unit 202E4, and an adder 202E5.

[0101] The motion vector residual decoding unit 202E1 is configured to generate MVR (Motion Vector Residual) from the bit stream of the P frame.

[0102] Here, MVR is a motion vector residual representing the difference between MV (Motion Vector) and MVP (Motion Vector Prediction). MV is a difference vector (motion vector) between the coordinates of the vertex of the corresponding I frame and the coordinates of the vertex of the P frame. MVP is the value of the MV of the vertex of the prediction target using MV (predicted value of the motion vector).

[0103] The motion vector buffer unit 202E2 is configured to sequentially store the MV output by the motion vector calculation unit 202E4.

[0104] The motion vector prediction unit 202E3 is configured to obtain the decoded MV from the motion vector buffer unit 202E2 for the vertex connected to the vertex of the decoding target, such as Fig. 9 As shown, the MVP of the vertex of the decoding target is output using all or part of the acquired decoded MV.

[0105] The motion vector calculation unit 202E4 is configured to add the MVR generated by the motion vector residual decoding unit 202E1 to the MVP output from the motion vector prediction unit 202E3, and output the MV of the vertex of the decoding target.

[0106] The adder 202E5 is configured to add the coordinates of the vertex corresponding to the vertex of the decoding target obtained from the decoded basic mesh of the corresponding I frame (reference frame) and the motion vector MV output from the motion vector calculation unit 202E3, and output the coordinates of the vertex of the decoding target.

[0107] The following describes the details of each part of the inter-frame decoding unit 202E.

[0108] exist Fig.10 , a flowchart showing an example of the operation of the motion vector prediction unit 202E3 is shown.

[0109] like Fig.10 As shown, in step S1001, the motion vector prediction unit 202E3 sets MVP and N to 0.

[0110] In step S1002, the motion vector prediction unit 202E3 obtains a set of MVs of vertices around the vertex of the decoding target from the motion vector buffer unit 202E2, identifies vertices for which subsequent processing has not been completed, and switches to No; if subsequent processing has been completed for all vertices, switches to Yes.

[0111] In step S1003 , the motion vector prediction unit 202E3 switches to No when the MV of the vertex of the processing target has not been decoded, and switches to Yes when the MV of the vertex of the processing target has been decoded.

[0112] In step S1004, the motion vector prediction unit 202E3 adds MV and MVP, and increases N by 1.

[0113] In step S1005, the motion vector prediction unit 202E3 divides MVP by N when N is greater than 0 and outputs the result, and outputs 0 and terminates the process when N is 0.

[0114] That is, the motion vector prediction unit 202E3 is configured to output the MVP of the decoding target by averaging the decoded motion vectors of the vertices located around the vertex of the decoding target.

[0115] Furthermore, the motion vector prediction unit 202E3 may be configured to set the MVP to 0 when the set of decoded motion vectors is an empty set.

[0116] The motion vector calculation unit 202E4 may be configured to calculate the MV of the vertex of the decoding target based on the MVP output by the motion vector prediction unit 202E3 and the MVR generated by the motion vector residual decoding unit 202E1 using equation (1).

[0117] MV(k)=MVP(k)+MVR(k) … (1)

[0118] Here, k is the index of the vertex. MV, MVR, and MVP are vectors having an x ​​component, a y component, and a z component.

[0119] According to the above configuration, since MVP is used instead of MV and only MVR is encoded, the effect of improving the encoding efficiency can be expected.

[0120] The adder 202E5 is configured to calculate the coordinates of the vertex by adding the MV of the vertex calculated by the motion vector calculation unit 202E4 and the coordinates of the vertex of the reference frame corresponding to the vertex, thereby maintaining the connection information (Connectivity) in the reference frame.

[0121] Specifically, the adder 202E5 can be configured to use equation (2) to calculate the coordinate v' of the kth vertex. i (k) Perform calculations.

[0122] v' i (k) = v' j (k)+MV(k) … (2)

[0123] Here, v' i (k) is the coordinate of the kth vertex decoded in the decoding target frame, v' j (k) is the coordinate of the k-th vertex after decoding of the reference frame, MV(k) is the k-th MV of the frame of the decoding target, k=1, 2, . . . , K.

[0124] In addition, the connection information of the decoding target frame is the same as the connection information of the reference frame.

[0125] Furthermore, since the motion vector prediction unit 202E3 calculates the MVP using the decoded MV, the decoding order does not affect the MVP.

[0126] The decoding order is the order of decoding the vertices of the base mesh of the reference frame. Generally speaking, if a decoding method is used that uses a constant repetition pattern to increase the base surface one by one from the edge as the starting point, the order of the vertices of the decoded base mesh is determined by the decoding process.

[0127] For example, the motion vector prediction unit 202E3 may use Edgebreaker (compression algorithm) to determine the order of decoding vertices in the base mesh of the reference frame.

[0128] According to the above configuration, since the MV from the reference frame is encoded instead of the coordinates of the vertices, it is possible to expect an effect of improving the encoding efficiency.

[0129] (Modification Example 1 of Inter-frame Decoding Unit 202E)

[0130] exist Fig.10 The MVP calculated in the flowchart shown is calculated by a simple average of the decoded surrounding MVs, but may be calculated by a weighted average.

[0131] That is, the motion vector prediction unit 202E3 can be configured to output a predicted value of the motion vector of the decoding target by performing weighted averaging on the decoded motion vectors of vertices located around the vertex of the decoding target with a weight corresponding to the distance between the vertex of the decoding target and the vertices of the reference frame corresponding to the vertices located around the vertex of the decoding target.

[0132] In addition, the motion vector prediction unit 202E3 can be configured to output a predicted value of the motion vector of the decoding target by taking a weighted average of a portion of the decoded motion vectors of vertices located around the vertex of the decoding target with a weight corresponding to the distance between the vertex of the decoding target and the vertices of the reference frame corresponding to the vertices located around the vertex of the decoding target.

[0133] In the present modification example 1, the motion vector prediction unit 202E3 of the inter decoding unit 202E is configured to calculate the MVP in the following procedure.

[0134] First, the motion vector prediction unit 202E3 is configured to calculate weights.

[0135] exist Fig.11, a flowchart showing an example of an operation of calculating the sum Total_D of distances to decoded surrounding vertices.

[0136] like Fig.11 As shown, in step S1101, the motion vector prediction unit 202E3 sets Total_D to 0.

[0137] Step S1102 is the same as step S1002.

[0138] Step S1103 is the same as step S1003.

[0139] In step S1104 , the motion vector prediction unit 202E3 adds e(k) to Total_D.

[0140] That is, the motion vector prediction unit 202E3 refers to a set of vertices located around the vertex of the decoding target, and adds the distances of the decoded vertices.

[0141] In the present modification example 1, the motion vector prediction unit 202E3 is configured to calculate the weight using the distance of the reference frame in which the correspondence relationship between vertices is known.

[0142] Right now, Fig.11 e(k) in step S1104 is the distance between corresponding vertices of the reference frame.

[0143] Furthermore, the motion vector prediction unit 202E3 may be configured to calculate the weight w(k) using equations (3) and (4).

[0144] [Mathematical formula 1]

[0145]

[0146] Here, Θ is the set of decoded vertices in the face of the mesh containing the vertex of the decoding target, e(p / k) is the distance between the vertex of the decoding target in the reference frame and the vertex corresponding to vertex p / k, and w(k) is the weight of vertex k.

[0147] In addition, the motion vector prediction unit 202E3 may be configured to set the weight according to a rule determined in advance based on the distance.

[0148] For example, the motion vector prediction unit 202E3 can be configured to set the weight to 1 when e(k) is less than the threshold TH1, set the weight to 0.5 when e(k) is less than the threshold TH2, and set the weight to 0 (not using the weight) in other cases.

[0149] According to the above configuration, when the distance to the vertex of the decoding target is short, by increasing the weight, it is possible to expect an effect that the MVP can be calculated with higher accuracy.

[0150] Second, the motion vector prediction unit 202E3 is configured to refer to the MVP.

[0151] exist Fig.12 A flowchart showing an example of an operation of calculating MVP using weighted average is shown in FIG.

[0152] like Fig.12 As shown, in step S1201, the motion vector prediction unit 202E3 sets MVP and N to 0.

[0153] Step S1202 is the same as step S1002.

[0154] Step S1203 is the same as step S1003.

[0155] In step S1204 , the motion vector prediction unit 202E3 adds w(k)×MV(k) to MVP, and N increases by 1.

[0156] Step S1205 is the same as step S1005.

[0157] Alternatively, the motion vector prediction unit 202E3 may be configured to calculate the MVP using equation (5).

[0158] [Mathematical formula 2]

[0159] MVP(k)=Σ m∈θ w(m)·MV(m) (5)

[0160] Here, θ is a set of decoded vertices in the surface of the mesh including the vertex of the decoding target.

[0161] According to the above configuration, since a more accurate MVP can be calculated by weighted averaging, the MVR value can be reduced to concentrate near zero, thereby expecting an effect of improving coding efficiency.

[0162] (Modification Example 2 of Inter-frame Decoding Unit 202E)

[0163] In the second modification, the motion vector prediction unit 202E3 is configured to select one MV instead of calculating the MVP using a plurality of surrounding MVs.

[0164] That is, the motion vector prediction unit 202E3 may be configured to select the MV of the nearest vertex among the decoded MVs accumulated in the motion vector buffer unit 202E2 as the MV of the vertex connected to the vertex of the decoding target.

[0165] Here, the motion vector prediction unit 202E3 can be configured to construct a candidate list consisting of MVs of vertices connected to the vertices of the decoding target from the decoded MVs accumulated in the motion vector buffer unit 202E2, and select a motion vector from the candidate list based on an index decoded from the bit stream of the P frame (the frame of the decoding target).

[0166] exist Fig.13 2 shows a flowchart showing an example of an operation of selecting an MV as the MVP from a set of candidate MVs.

[0167] like Fig.13 As shown, in step S1301, the motion vector prediction unit 202E3 decodes the list ID based on the bit stream of the P frame.

[0168] In step S1302, the motion vector prediction unit 202E3 selects the MV with the additional list ID from the candidate MVs as the MVP.

[0169] In addition, Fig.13 In the set of candidate MVs, MVs calculated from the decoded surrounding MVs and their combinations are arranged in a constant order.

[0170] exist Fig.14 A flowchart showing an example of the operation of creating the set of candidate MVs is shown in FIG.

[0171] like Fig.14 As shown, in step S1401, the motion vector prediction unit 202E3 refers to the set of MVs of vertices around the vertex of the decoding target to determine whether the processing performed on all vertices around the vertex of the decoding target is completed.

[0172] If the processing is completed, the action ends, and if the processing is not completed, the action proceeds to step S1402.

[0173] In step S1402 , the motion vector prediction unit 202E3 determines whether the MV of the target vertex has been decoded.

[0174] In the case where the MV is decoded, the action proceeds to step S1403, and in the case where the MV is not decoded, the action returns to step S1401.

[0175] In step S1403, the motion vector prediction unit 202E3 determines whether the MV overlaps with other decoded MVs.

[0176] In case of repetition, the action returns to step S1401, and in case of non-repetition, the action proceeds to step S1404.

[0177] The motion vector prediction unit 202E3 determines the list ID assigned to the MV in step S1404 and includes it in the set of candidate MVs in step S1405.

[0178] In addition, Fig.14 When determining the list ID, the motion vector prediction unit 202E3 may increase the list ID one by one in sequence, or may determine the list ID in the order of the distance between the vertex of the decoding target of the reference frame and the vertex corresponding to vertex k (e(k) in formula (3)).

[0179] According to the above structure, one of the candidate MVs is selected as the MVP, but sometimes it is closer to the MV than the average depending on the situation. Therefore, in this case, the effect of improving the coding efficiency can be expected.

[0180] Furthermore, the motion vector prediction unit 202E3 may be configured to add an MV obtained by averaging consecutive MV0 and MV1 from the candidate MVs as a new candidate MV to the list. As shown in Table 1, the motion vector prediction unit 202E3 sets the MV to be added after MV0 and MV1.

[0181] [Table 1]

[0182] No(index) Candidate MV 0 MV0 1 MV1 2 (MV0+MV1) / 2 3 MV2 4 MV3

[0183] According to the above configuration, it is possible to expect an effect of increasing the possibility that the selected candidate MV is closer to the MV of the vertex of the decoding target.

[0184] Furthermore, the motion vector prediction unit 202E3 may be configured to select the MV of the nearest vertex from the set of candidate MVs without encoding the list ID. According to the above configuration, it is expected that the effect of further improving the encoding efficiency can be achieved.

[0185] (Modification Example 3 of Inter-frame Decoding Unit 202E)

[0186] In the above-described embodiment and Modifications 1 and 2, the surrounding vertices are vertices connected to the vertex of the decoding target.

[0187] On the other hand, in the third modification, the motion vector prediction unit 202E3 is configured to calculate the MVP by parallelogram prediction, that is, also by using vertices that are not directly connected to the vertices of the decoding target.

[0188] like Fig.15 As shown, in parallelogram prediction, a vertex D on the opposite side of the decoded surface having a common edge BC with the vertex A of the decoding target is also used.

[0189] In addition, the common edges of the decoding target vertex A include CE and BG in addition to AB. Therefore, in the parallelogram prediction, the vertices F and H can also be used.

[0190] For example, the motion vector prediction unit 202E3 may be configured to use Fig.15 For the surface BCD shown, the MVP is calculated using equation (6).

[0191] MVP=MV(B)+MV(C)-MV(D)…(6)

[0192] Here, MV(X) is the motion vector of vertex X, and MVP is the predicted value of the motion vector of vertex A of the decoding target.

[0193] In addition, when there are multiple common edges as described above, the motion vector prediction unit 202E3 may average the MVPs or select the surface whose center of gravity is closest.

[0194] (Modification Example 4 of Inter-frame Decoding Unit 202E)

[0195] In the present modification example, the MVR generated by the motion vector residual decoding unit 202E1 is configured so that the quantization width when the MVR is expressed as an integer is controlled instead of being maintained as it is.

[0196] In the present modification example, the motion vector residual decoding unit 202E1 is configured to decode adaptive_mesh_flag, adaptive_bit_flag, and the precision control parameter as control information for controlling the quantization width of MVR.

[0197] That is, the motion vector residual decoding unit 202E1 is configured to decode adaptive_mesh_flag of the entire base mesh and adaptive_bit_flag of each base patch.

[0198] Here, adaptive_mesh_flag and adaptive_bit_flag are flags indicating whether to adjust the quantization width of the MVR described above, and take either a value of 0 or 1.

[0199] Here, the motion vector residual decoding unit 202E1 decodes adaptive_bit_flag only when adaptive_mesh_flag is valid (that is, 1).

[0200] Furthermore, when adaptive_mesh_flag is invalid (ie, 0), the motion vector residual decoding unit 202E1 regards adaptive_bit_flag as invalid (ie, 0).

[0201] exist Fig.16 , a flowchart of an example of an operation of controlling the quantization width of the decoded MVR based on adaptive_mesh_flag, adaptive_bit_flag, and precision control parameters, which are control information generated by decoding the base mesh bit stream.

[0202] like Fig.16 As shown, in step S1601, the motion vector prediction unit 202E3 determines whether adaptive_mesh_flag is 0.

[0203] When it is determined that the adaptive_mesh_flag of the entire mesh is 0, this operation ends.

[0204] On the other hand, when it is determined that the adaptive_mesh_flag of the entire mesh is 1, the operation proceeds to step S1602.

[0205] In step S1602, the motion vector prediction unit 202E3 determines whether or not there is an unprocessed patch in the frame.

[0206] In step S1603 , the motion vector prediction unit 202E3 determines whether adaptive_mesh_flag decoded for each patch is 0.

[0207] When it is determined that adaptive_mesh_flag is 0, the operation returns to step S1601.

[0208] On the other hand, when it is determined that adaptive_mesh_flag is 1, the operation proceeds to step S1604.

[0209] In step S1604, the motion vector prediction unit 202E3 controls the quantization width of MVR based on the accuracy control parameter described later.

[0210] In addition, the value of MVR that controls the quantization width in this way is called "MVRQ (Motion Vector Residual Quantization)".

[0211] Here, the motion vector prediction unit 202E3 may be configured to refer to a table such as Table 2, for example, and use the quantization width of the MVR corresponding to the quantization width control parameter generated by decoding the base grid bit stream.

[0212] [Table 2]

[0213] Quantization width control parameters MVR quantization width 1 2 times 2 4 times 3 8 times

[0214] According to the above structure, the effect of improving coding efficiency by controlling the quantization width of MVR can be expected. In addition, the effect of minimizing useless bits when the quantization width of MVR is not controlled can be expected by the hierarchical structure of adaptive_mesh_flag at the mesh level and adaptive_mesh_flag at the patch level.

[0215] (Modification Example 5 of Inter-frame Decoding Unit 202E)

[0216] If the MVR generated by the motion vector residual decoding unit 202E1 is not encoded, an error will occur. In this modification example 5, in order to correct the error, the discrete motion vector difference is encoded.

[0217] Specifically, if Fig.17 As shown, MVR can be of size 1, 2, 4, and 8 in the six directions of the x-axis, y-axis, and z-axis. Tables 3 and 4 show examples of the above encoding.

[0218] In addition, MVR encoding may be performed in combination of multiple directions. For example, correction may be performed in the order of 2 in the positive direction of the x-axis and 1 in the positive direction of the y-axis.

[0219] [Table 3]

[0220] direction_idx 000 001 010 011 100 101

[0221] [Table 4]

[0222] distance_idx 0 1 2 3 Value 1 2 4 8

[0223] According to the above structure, it can be expected that the encoding efficiency of discrete motion vector differences is higher than that of MVR encoding.

[0224] Next, further modification examples of the inter-frame decoding unit 202E will be described.

[0225] In a further modification example of the above-mentioned inter-frame decoding unit 202E, the following functional blocks are added before the above-mentioned inter-frame decoding unit 202E is implemented.

[0226] Specifically, if Fig.18 As shown, the inter-frame decoding unit 202E is Figure 8 In addition to the structure shown, it also has a duplicate vertex search unit 202E6, a duplicate vertex judgment unit 202E7, a motion vector acquisition unit 202E8, and an all skip mode signal and a skip mode signal.

[0227] Here, all skip mode signals are located at the beginning of the bit stream of the P frame, have at least 2 values, and are 1 bit or more.

[0228] One of them (when all skip mode signals indicate yes, for example, 1) is a signal for not decoding the motion vectors of all repeated vertices of the P frame from the bitstream but copying the motion vectors of the repeated vertices.

[0229] The other (when all skip mode signals indicate no, for example, when they are 0) is a signal for performing different processing on each vertex of a P frame. The other may have other values. For example, the other is to perform the same processing as the motion vector acquisition unit 202E8 on the motion vectors of all repeated vertices without performing the processing. Figure 8 The inter-frame decoding unit 202E shown processes the signal in the same manner.

[0230] Here, when all skip mode signals indicate "no", the skip mode signal has two values ​​for each repeated vertex, and is 1 bit.

[0231] The skip mode signal is a signal for copying the motion vector of a duplicate vertex without decoding the motion vector of the vertex from the bit stream when all the skip mode signals indicate yes (for example, when they are 1).

[0232] The skip mode signal is when all skip mode signals indicate no (for example, when they are 0), the motion vector of the vertex is processed without being processed by the motion vector acquisition unit 202E8. Figure 8 The inter-frame decoding unit 202E shown processes the signal in the same manner.

[0233] In addition, the skip mode signal can be directly decoded according to the bit stream, or can be decoded according to the bit stream and Figure 8 The inter-frame decoding unit 202E shown performs the same processing on the data of the specific repeated vertex (for example, the index of the repeated vertex), and calculates the skip mode signal based on the data.

[0234] Moreover, if Fig.38 As shown, the skip mode signal can be used without calculating the skip mode signal. Figure 8 The inter-frame decoding unit 202E shown performs the same processing on the data of the specific repeated vertex (for example, the index of the repeated vertex), and determines the motion vector decoding method of the vertex in the same way as in the above case.

[0235] The duplicate vertex search unit 202E6 is configured to search for indices of vertices with identical coordinates (hereinafter referred to as duplicate vertices) based on the geometric information of the base mesh of the decoded reference frame and store the indices in a buffer (not shown).

[0236] Specifically, the input of the duplicate vertex search unit 202E6 is the index (decoding order) and position coordinates of each vertex of the decoded base mesh of the reference frame.

[0237] The output of the duplicate vertex search unit 202E6 is a list of pairs of the index (vindex0) of the vertex where the duplicate vertex exists and the index (vindex1) of the duplicate vertex. Here, the list of pairs is stored in the buffer repVert in the order of index0.

[0238] In addition, since the vertex of vindex1 is decoded before vindex0, the relationship of vindex0>vindex1 is established.

[0239] In addition, as a method of knowing duplicate vertices in the base mesh of the reference frame, for vertices where duplicate vertices exist, instead of position coordinates, the index of the duplicate vertices is decoded through a special signal. Through the special signal, pairs of the index of the corresponding vertex and the index of the duplicate vertex can be saved in the decoding order.

[0240] The duplicate vertex determination unit 202E7 is configured to determine whether or not the corresponding vertex is a duplicate vertex among the decoded vertices.

[0241] Here, the duplicate vertex determination unit 202E7 determines that a duplicate vertex exists in the decoded vertices when the index of the corresponding vertex is one of the indices of the vertices where the duplicate vertex exists. In addition, since the corresponding vertex is in place according to the decoding, the above-mentioned search is not necessary.

[0242] Here, when the duplicate vertex determination unit 202E7 determines that there is no duplicate vertex of the corresponding vertex, Figure 8 The same processing is performed by the inter-frame decoding unit 202E shown.

[0243] The motion vector acquisition unit 202E8 is configured to, when there is a duplicate vertex of the corresponding vertex, when all skip mode signals indicate yes, or when all skip mode signals indicate no and the skip mode signal of the corresponding vertex indicates yes, obtain the motion vector of the vertex having the same index as the duplicate vertex from the motion vector buffer unit 202E2 that stores the decoded motion vector, and set it as the motion vector of the corresponding vertex.

[0244] Here, when all skip mode signals indicate no and the skip mode signal of the corresponding vertex indicates no, the motion vector acquisition unit 202E8 is replaced with Figure 8 The same processing is performed by the inter-frame decoding unit 202E shown.

[0245] According to the above configuration, it is possible to expect the effect of reducing the decoding calculation of motion vectors of vertices having duplicate vertices and the amount of code.

[0246] In a further modification example of the above-mentioned inter-frame decoding unit 202E, the inter-frame decoding unit 202E acquires the correspondence relationship between the vertices of the reference frame and the vertices of the decoding target frame from the decoded basic mesh of the reference frame.

[0247] Furthermore, the inter-frame decoding unit 202E is configured to make the connection information of the vertices of the decoding target frame the same as the connection information of the decoded vertices of the reference frame without encoding it based on the above-mentioned correspondence relationship.

[0248] In addition, the inter-frame decoding unit 202E divides the base mesh of the decoding target frame into two regions according to the decoding order of the vertices of the reference frame. The first region is decoded using the inter-frame process, and the second region is decoded using the intra-frame process.

[0249] Furthermore, the above-mentioned region is defined as a region formed by a plurality of vertices that are continuous in decoding order when decoding the base mesh of the reference frame.

[0250] In addition, the mechanism for decoding the coordinates of the vertices of the base mesh of the decoding target frame using the signal can assume the following two methods.

[0251] (Organization 1)

[0252] In mechanism 1, the signals are vertex_idx1, vertex_idx2 and intra_flag.

[0253] Here, vertex_idx1 and vertex_idx2 are indexes of the decoding order of vertices (vertex indexes), and intra_flag is a flag indicating whether the above-mentioned inter-frame decoding method or intra-frame decoding method is used. There may be a plurality of such signals.

[0254] That is, vertex_idx1 and vertex_idx2 are vertex indices that define the start position and the end position of the above-mentioned partial region (the first region and the second region).

[0255] (Organization 2)

[0256] In mechanism 2, it is assumed that Edgebreaker decodes the connection information of the base mesh of the reference frame so that the decoding order of the vertex coordinates is the order determined by Edgebreaker.

[0257] Fig.19 This diagram shows an example of the operation of determining connection information and the order of vertices using Edgebreaker.

[0258] exist Fig.19In the figure, arrows indicate the decoding order of connection information, numbers indicate the decoding order of vertices, and arrows of the same line type define the same area.

[0259] In mechanism 2, the signal is only intra_flag, which is a flag indicating whether the inter-frame decoding method or the intra-frame decoding method is used.

[0260] That is, in mechanism 2, the inter-frame decoding unit 202E is configured to be divided into the first region and the second region using Edgebreaker.

[0261] <Sub-division Unit 203>

[0262] The sub-division unit 203 is configured to generate and output the added sub-division vertices and connection information thereof based on the basic mesh decoded by the basic mesh decoding unit 202 according to the sub-division method indicated by the control information.

[0263] Here, the base mesh, the added sub-division vertices, and the connection information thereof are collectively referred to as a “sub-division mesh”.

[0264] The sub-division unit 202 is configured to specify the type of sub-division method based on subdivision_method_id, which is control information generated by decoding the basic grid bit stream.

[0265] Below, refer to Figure 3A and Figure 3B The sub-dividing unit 202 will be described.

[0266] Figure 3A and Figure 3B This is a diagram for explaining an example of an operation of generating sub-division vertices from a base mesh.

[0267] Figure 3A This is a diagram showing an example of a basic mesh composed of five vertices.

[0268] Here, the subdivision may be performed using, for example, a mid-edge segmentation method that connects the midpoints of the edges of each base surface to each other, thereby segmenting one base surface into four surfaces.

[0269] Figure 3B An example of a sub-divided mesh formed by dividing a base mesh consisting of five vertices is shown. Figure 3B In the sub-division mesh shown, eight sub-division vertices (hollow circles) are generated in addition to the original five vertices (solid circles).

[0270] By decoding the displacement amount for each of the sub-divided vertices generated in this way by the displacement amount decoding unit 206 , it is expected that the encoding performance will be improved.

[0271] In addition, different sub-division methods may be applied to each patch. In this way, the displacement decoded by the displacement decoding unit 206 is adaptively changed for each patch, and improvement in encoding performance can be expected. Information on the divided patch is received as control information, namely, patch_id.

[0272] Below, refer to Fig. 20 The sub-dividing unit 203 will be described. Fig.21 2 is a diagram showing an example of functional blocks of the sub-dividing unit 203 .

[0273] like Fig.21 As shown, the sub-division unit 203 includes a basic grid sub-division unit 203A and a sub-division grid adjustment unit 203B.

[0274] (Basic mesh sub-division unit 203A)

[0275] The base mesh sub-division unit 203A is configured to calculate the number of divisions (sub-division number) for each base surface and each base patch based on the input base mesh and the division information of the base mesh, sub-divide the base mesh according to the division number, and output the sub-divided surfaces.

[0276] That is, the base mesh sub-division unit 203A may be configured to be able to change the above-mentioned number of divisions in units of base surfaces and base patches.

[0277] Here, a base face is a face that constitutes a base mesh, and a base patch is a collection of several base faces.

[0278] In addition, the basic mesh sub-division unit 203A may be configured to calculate the number of sub-divisions of the basic surface by predicting the number of sub-divisions of the basic surface and adding the predicted number of sub-divisions of the basic surface to a residual of the predicted number of sub-divisions of the basic surface.

[0279] In addition, the basic mesh sub-division unit 203A may be configured to calculate the number of sub-divisions of the basic surface based on the number of sub-divisions of adjacent basic surfaces of the basic surface.

[0280] In addition, the basic mesh sub-division unit 203A may be configured to calculate the number of sub-divisions of the basic surface based on the previously accumulated number of sub-divisions of the basic surface.

[0281] In addition, the basic mesh sub-division unit 203A may be configured to generate vertices for dividing three sides constituting the basic surface and connect the generated vertices, thereby performing sub-division on the basic surface.

[0282] like Fig.21 As shown, the basic mesh sub-division unit 203A has 203B at the rear stage, which has a sub-division mesh adjustment unit described later.

[0283] Below, use Figure 21 to Figure 23An example of the processing performed by the basic mesh sub-division unit 203A will be described.

[0284] Fig.21 is a diagram showing an example of a functional block of the basic mesh sub-division unit 203A. Fig.23 This is a flowchart showing an example of the operation of the basic mesh sub-division unit 203A.

[0285] like Fig.21 As shown, the basic mesh sub-division unit 203A includes a basic surface division number buffer unit 203A1, a basic surface division number reference unit 203A2, a basic surface division number prediction unit 203A3, an addition unit 203A4, and a basic surface division unit 203A5.

[0286] The base surface division number buffer unit 203A1 is configured to store the base surface division information including the division number of the base surface, and output the base surface division information to the base surface division number reference unit 203A2.

[0287] Here, the size of the base surface division number buffer unit 203A1 may be set to 1, and the previously accumulated base surface division number may be output to the base surface division number reference unit 203A2.

[0288] That is, the size of the base face division number buffer unit 203A1 may be set to 1 so that only the last decoded sub-division number (the sub-division number decoded before) is referenced.

[0289] The base plane division number reference unit 203A2 is configured to output "cannot be referenced" to the base plane division number prediction unit 203A3 when there is no adjacent base plane to the decoding target base plane or when there is an adjacent base plane to the decoding target base plane but the division number is uncertain.

[0290] On the other hand, when there is an adjacent base plane to the decoding target base plane and the number of divisions is determined, the base plane division number reference unit 203A2 is configured to output the number of divisions to the base plane division number prediction unit 203A3.

[0291] The base surface division number prediction unit 203A3 is configured to predict the number of divisions of the base surface (the number of sub-divisions) based on one or more input division numbers, and output the predicted number of divisions (predicted number of divisions) to the addition unit 203A4.

[0292] Here, the base surface division number prediction unit 203A3 is configured to output 0 to the addition unit 203A4 when only the input of "cannot be referenced" is input from the base surface division number reference unit 203A2.

[0293] In addition, the base surface division number prediction unit 203A3 may be configured to generate a predicted division number by using any one of statistical values such as the average value, maximum value, minimum value, and mode of the input division numbers when one or more division numbers are input.

[0294] In addition, the base surface division number prediction unit 203A3 may be configured to generate the division number of the closest adjacent surface as the predicted division number when one or more division numbers are input.

[0295] The addition unit 203A4 is configured to output the division number obtained by adding the predicted division number residual decoded from the predicted residual bitstream and the predicted division number obtained from the base surface division number prediction unit 203A3 to the base surface division unit 203A5.

[0296] The base surface division unit 203A5 is configured to perform sub-division of the base surface according to the division number input from the addition unit 203A4.

[0297] Fig. 22 This is an example of the case where the base surface is divided into nine parts. Refer to Fig. 22 to illustrate the division method of the base surface of the base surface division unit 203A5.

[0298] The base surface division unit 203A5 generates points A_1,..., A_(N - 1) that equally divide the side AB (N = 3) constituting the base surface.

[0299] Similarly, the base surface division unit 203A5 also equally divides the side BC or the side CA, and generates points B_1,..., B_(N - 1), C_1,..., C_(N - 1) respectively.

[0300] After that, the points on the sides AB, BC, and CA are called "side division points".

[0301] The base surface division unit 203A5 generates sides A_i B_(N - i), B_i C_(N - i), C_i A_(N - i) for all i (i = 1, 2,..., N - 1), and generates N 2 sub-division surfaces.

[0302] Next, refer to Fig.23 to illustrate the processing sequence of the base mesh sub-division unit 203A.

[0303] In step S2201, it is determined whether the re-division process for the last base surface is completed. If the process is completed, it ends; if not, it proceeds to step S2202.

[0304] In step S2202, the base mesh sub-division unit 203A determines whether Depth < mdu_max_depth.

[0305] Here, Depth is a variable indicating the current depth, and its initial value is 0. mdu_max_depth indicates the maximum depth determined for each base surface.

[0306] If the condition of step S2202 is met, the processing sequence enters step S2203, and if the condition is not met, the processing sequence returns to step S2201.

[0307] In step S2203 , the basic mesh sub-division unit 203A determines whether the mdu_subdivision_flag of the current depth is 1.

[0308] If yes, the processing sequence returns to step S2201, and if no, the processing sequence enters step S2204.

[0309] In step S2204, the basic mesh sub-division unit 203A further sub-divides all sub-divided surfaces within the basic surface.

[0310] Here, the basic mesh sub-dividing unit 203A sub-divides the basic surface without performing a sub-division process on the basic surface.

[0311] In addition, the sub-division method is the same as the method described in step S2204.

[0312] Specifically, when the base surface is not subdivided once, the base surface is Fig. 22 When at least one sub-division is performed, the sub-division face is subdivided into N 2 In Fig. 22 For example, the face composed of vertices A_2, B and B_1 is further segmented using the same method as the basic face segmentation to generate N 2 Noodles.

[0313] When the sub-division processing is completed, the processing sequence enters step S2205.

[0314] In step S2205, the basic mesh sub-division unit 203A adds 1 to Depth, and the processing sequence returns to step S2202.

[0315] (Sub-division grid adjustment unit 203B)

[0316] Next, a specific example of the processing performed by the sub-division grid adjustment unit 203B will be described. Figures 24 to 28 An example of the processing performed by the sub-division grid adjustment unit 203B will be described.

[0317] Fig.242 is a diagram showing an example of the functional blocks of the sub-division grid adjustment unit 203B.

[0318] like Fig.24 As shown, the sub-division grid adjustment unit 203B includes an edge division point moving unit 701 and a sub-division surface dividing unit 702 .

[0319] (Edge Division Point Moving Unit 701)

[0320] The edge division point moving unit 701 is configured to move the edge division point of the base surface to any one of the edge division points of the adjacent base surface with respect to the input initial sub-division surface, and output the sub-division surface.

[0321] Fig.25 is an example of moving the edge splitting point on the base surface ABC. Fig.25 As shown, the side division point moving unit 701 may be configured to move the side division point of the base surface ABC to the side division point of the nearest adjacent base surface.

[0322] (Sub-dividing surface dividing unit 702)

[0323] The sub-divided plane dividing unit 702 is configured to sub-divide the input sub-divided plane again and output a decoded sub-divided plane.

[0324] Fig.26 1 is a diagram showing an example of a case where the sub-divided surface X in the base surface is further sub-divided.

[0325] like Fig.26 As shown, the sub-divided surface dividing unit 702 may be configured to generate new sub-divided surfaces within the base surface by connecting vertices constituting the sub-divided surfaces with edge division points of the adjacent base surface.

[0326] Fig. 27 1 is a diagram showing an example of a case where the above-mentioned sub-division process is performed on all sub-division surfaces.

[0327] The mesh decoding unit 204 is configured to generate and output a decoded mesh using the sub-division mesh generated by the sub-division unit 203 and the displacement amount decoded by the displacement amount decoding unit 206 .

[0328] Specifically, the mesh decoding unit 204 is configured to generate a decoded mesh by adding the corresponding displacement amount to each sub-division vertex. Here, information on which sub-division vertex each displacement amount corresponds to is indicated by control information.

[0329] The patch integration unit 205 is configured to integrate and output a plurality of patches of the decoded meshes generated by the mesh decoding unit 206 .

[0330] Here, the patch segmentation method is defined by the grid encoding device 100. For example, the patch segmentation method may be configured to calculate the normal vector for each base surface in advance, select the base surface with the most similar normal vector from adjacent base surfaces, aggregate the two base surfaces into the same patch, and repeat the above sequence for the next base surface.

[0331] The video decoding unit 207 is configured to decode and output textures by video coding. For example, the video decoding unit 207 can use HEVC of Non-Patent Document 1.

[0332] <Displacement Decoding Unit 206>

[0333] The displacement amount decoding unit 206 is configured to decode the displacement amount bit stream and generate and output the displacement amount.

[0334] Fig.28 This is a diagram showing an example of the displacement amount with respect to one sub-division vertex.

[0335] exist Figure 3B In the example of , since there are 8 sub-division vertices, the displacement amount decoding unit 206 is configured to define 8 displacement amounts expressed by scalars or vectors for each sub-division vertex.

[0336] Below, refer to Fig.28 , the displacement decoding unit 206 is explained. Fig.28 2 is a diagram showing an example of functional blocks of the displacement amount decoding unit 206 .

[0337] like Fig.28 As shown, the displacement amount decoding unit 206 includes a decoding unit 206A, an inverse quantization unit 206B, an inverse wavelet transform unit 206C, an adder 206D, an inter-frame prediction unit 206E, and a frame buffer 206F.

[0338] The decoding unit 206A is configured to decode and output the level value and control information by performing variable length decoding on the received displacement bit stream. Here, the level value obtained by variable length decoding is output to the inverse quantization unit 206B, and the control information is output to the inter-frame prediction unit 206E.

[0339] Below, refer to Fig.29 , an example of the structure of the displacement bit stream is explained. Fig.29 This is a diagram showing an example of the structure of a displacement bit stream.

[0340] like Fig.29 As shown, first, the displacement bit stream may include a set of control information related to the decoding of the displacement, namely, DPS (Displacement Parameter Set).

[0341] Second, the displacement bitstream may include a set of control information corresponding to the patch, namely, DPH (DisplacementPatch Header).

[0342] Third, the displacement bitstream may contain the encoded displacements that make up the patch after the DPH.

[0343] As described above, the structure of the displacement bit stream is that DPH and DPS correspond to each encoded displacement.

[0344] also, Fig.29 The structure is just an example. As long as DPH and DPS correspond to the encoded displacement amounts, elements other than the above may be added as structural elements of the displacement amount bit stream.

[0345] For example, Fig.29 As shown, the displacement bit stream may include an SPS (Sequence Parameter Set).

[0346] Fig.30 This is a diagram showing an example of the syntax structure of DPS.

[0347] exist Fig.30 In the descriptor column, the descriptor column indicates how each syntax is encoded.

[0348] In addition, Fig.30 In it, ue(v) represents the codeless 0th order exponential-Golomb code, and u(n) represents the n-bit tag.

[0349] When there are a plurality of DPSs, the DPS includes at least DPS id information (dps_displacement_parameter_set_id) for identifying each DPS.

[0350] In addition, the DPS may include a flag (interprediction_enabled_flag) for controlling whether to perform inter-frame prediction.

[0351] For example, when the interpretation_enabled_flag is 0, it can be defined as not performing inter-frame prediction, and when the interpretation_enabled_flag is 1, it can be defined as performing inter-frame prediction. When the interpretation_enabled_flag is not included, it can be defined as not performing inter-frame prediction.

[0352] DPS may include a flag (dct_enabled_flag) that controls whether to perform inverse DCT.

[0353] For example, when dct_enabled_flag is 0, it can be defined as not performing inverse DCT, and when dct_enabled_flag is 1, it can be defined as performing inverse DCT. When dct_enabled_flag is not included, it can be defined as not performing inverse DCT.

[0354] Fig.31 This is a diagram showing an example of the syntax structure of DPH.

[0355] like Fig.31 As shown, the DPH includes at least DPS id information for specifying the DPS corresponding to each DPH.

[0356] The inverse quantization unit 206B is configured to generate and output a transform coefficient by inverse quantizing the level value decoded by the decoding unit 206A.

[0357] The inverse wavelet transform unit 206C is configured to generate and output a prediction residual by performing an inverse wavelet transform on the transform coefficients generated by the inverse quantization unit 206B.

[0358] (Inter-frame prediction unit 206E)

[0359] The inter-frame prediction unit 206E is configured to perform inter-frame prediction using the decoded displacement of the reference frame read from the frame buffer 206F, and to generate and output a predicted displacement.

[0360] The inter-frame prediction unit 206E is configured to perform the inter-frame prediction only when interprediction_enabled_flag is 1.

[0361] The inter-frame prediction unit 206E can perform inter-frame prediction in the spatial domain or in the frequency domain. The inter-frame prediction can use a reference frame in the past and a reference frame in the future to perform bidirectional prediction.

[0362] Fig.28 This is an example of a functional block of the inter-frame prediction unit 206E when inter-frame prediction is performed in the spatial region.

[0363] When the inter-frame prediction unit 206E performs inter-frame prediction in the spatial region, it may directly refer to the decoded displacement of the corresponding sub-divided vertex in the reference frame to determine the predicted displacement of the sub-divided vertex in the target frame.

[0364] Alternatively, the predicted displacement of a sub-divided vertex in the target frame can be determined probabilistically using the decoded displacements of the corresponding sub-divided vertices in multiple reference frames according to a normal distribution with an estimated mean and dispersion. In this case, the dispersion can be uniquely determined to be zero only by the mean.

[0365] Alternatively, the predicted displacement of a sub-divided vertex in the target frame can be determined using decoded displacements of corresponding sub-divided vertices in multiple reference frames, based on a regression curve that estimates time using explanatory variables and displacement as target variables.

[0366] In the trellis coding apparatus 100, in order to improve the coding efficiency of each frame, the order of the decoding displacements may be rearranged.

[0367] In this case, the inter-frame prediction unit 206E may be configured to perform inter-frame prediction on the rearranged decoded displacement.

[0368] The correspondence relationship between the sub-division vertices between the reference frame and the decoding target frame is represented by the control information.

[0369] Fig.32 This is a diagram for explaining an example of the correspondence relationship between sub-division vertices between a reference frame and a decoding target frame when inter-frame prediction is performed in a spatial region.

[0370] Fig.33 This is an example of a functional block of the inter-frame prediction unit 206E when inter-frame prediction is performed in the frequency domain.

[0371] When the inter-frame prediction unit 206E performs inter-frame prediction in the frequency domain, it may directly refer to the decoded wavelet transform coefficient of the corresponding frequency in the reference frame to determine the predicted wavelet transform coefficient of the frequency in the decoding target frame.

[0372] The inter prediction unit 206E may perform inter prediction according to a normal distribution probability in which the average and the dispersion are estimated, using the decoded displacement amounts or decoded wavelet transform coefficients of the sub-division vertices in the plurality of reference frames.

[0373] The inter prediction unit 206E may perform inter prediction based on a regression curve that estimates time using explanatory variables and displacement amounts as target variables, using decoded displacement amounts or decoded wavelet transform coefficients of sub-divided vertices in a plurality of reference frames.

[0374] The inter-frame prediction unit 206E may be configured to perform inter-frame prediction bidirectionally using a temporally past reference frame and a future reference frame.

[0375] In the trellis encoding device 100, in order to improve the encoding efficiency of each frame, the order of the decoded wavelet transform coefficients may be rearranged.

[0376] The frequency correspondence between the reference frame and the decoding target frame is represented by the control information.

[0377] Fig.34 This is a diagram for explaining an example of the frequency correspondence relationship between a reference frame and a decoding target frame when inter-frame prediction is performed in the frequency domain.

[0378] Furthermore, when the sub-dividing unit 203 divides the base grid into a plurality of patches, the inter-frame prediction unit 206E also performs inter-frame prediction for each of the divided patches. This increases the temporal correlation between frames and can be expected to improve the encoding performance.

[0379] The adder 206D receives an input of a prediction residual from the inverse wavelet transform unit 206C and an input of a prediction displacement from the inter-frame prediction unit 206E.

[0380] The adder 206D is configured to calculate and output a decoded displacement by adding the prediction residual to the prediction displacement.

[0381] The decoded displacement calculated by the adder 206D is also output to the frame buffer 206F.

[0382] The frame buffer 206F is configured to acquire and accumulate the decoded displacement amount from the adder 206D.

[0383] Here, the frame buffer 206F outputs the decoded displacement of the corresponding vertex in the reference frame in response to control information (not shown).

[0384] Fig.35 This is a flowchart showing an example of the operation of the displacement amount decoding unit 206.

[0385] like Fig.35 As shown, in step S3501, the displacement amount decoding unit 206 determines whether the processing is completed for all patches.

[0386] If yes, the action ends, and if no, the action proceeds to step S3502.

[0387] In step S3502 , the displacement decoding unit 206 performs inverse DCT on the patch of the decoding target, and then performs inverse quantization and inverse wavelet transform.

[0388] In step S3503 , the displacement amount decoding unit 206 determines whether interpretation_enabled_flag is 1.

[0389] In the case of yes, the action proceeds to step S3504, and in the case of no, the action returns to step S3501.

[0390] In step S3504, the displacement decoding unit 206 performs the above-mentioned inter-frame prediction and addition.

[0391] <Modification 1>

[0392] Below, refer to Fig.36, focusing on the differences between the above-mentioned modification example 1 of the first embodiment and the above-mentioned first embodiment.

[0393] Fig.36 This is a diagram showing an example of functional blocks of the displacement amount decoding unit 206 according to the first modification.

[0394] like Fig.36 As shown, the displacement decoding unit 206 according to the first modification includes an inverse DCT unit 206G after the decoding unit 206A, that is, between the decoding unit 206A and the inverse quantization unit 206B.

[0395] That is, in the present modification example 1, the inverse quantization unit 206B is configured to generate a prediction residual by performing an inverse wavelet transform on the level value output from the inverse DCT unit 202G.

[0396] <Modification 2>

[0397] Below, refer to Fig.37 , focusing on the differences between the modified example 2 of the above-mentioned first embodiment and the above-mentioned first embodiment.

[0398] like Fig.37 As shown, the displacement amount decoding unit 206 according to the second modification includes a video decoding unit 2061 , an image development unit 2062 , an inverse quantization unit 2063 , and an inverse wavelet transformation unit 2064 .

[0399] The video decoding unit 2061 is configured to output a video by decoding the displacement bit stream received by the video encoding.

[0400] For example, the video decoding unit 2061 can use HEVC of Non-Patent Document 1.

[0401] In addition, the video decoding unit 2061 may use a video encoding method that always sets the motion vector to 0. For example, the video decoding unit 2061 may always set the motion vector of HEVC to 0 and use inter-frame prediction that is always at the same position.

[0402] In addition, the video decoding unit 2061 may use a video coding method that always skips transformation. For example, the video decoding unit 2061 may always set the transformation of HEVC to the transformation skip mode and use the video coding method without performing transformation.

[0403] The image expansion unit 2062 is configured to expand the image decoded by the image decoding unit 2061 as a level value for each image (frame) and output the expanded image.

[0404] In the above-described expansion method, the image expansion unit 2062 can perform back-calculation and identification based on the arrangement method of the level values ​​with respect to the image indicated by the control information.

[0405] The image expansion unit 2062 may, for example, arrange the level values ​​from high-frequency components to low-frequency components in the image in a raster operation order as a method of arranging the level values.

[0406] The inverse quantization unit 2063 is configured to generate and output a transform coefficient by inverse quantizing the level value generated by the image development unit 2062 .

[0407] The inverse wavelet transform unit 2064 is configured to generate and output a decoded displacement by performing an inverse wavelet transform on the transform coefficients generated by the inverse quantization unit 2063 .

[0408] The above-mentioned trellis encoding device 100 and trellis decoding device 200 can be implemented as a program that causes a computer to execute each function (each process).

[0409] Industrial Applicability

[0410] In addition, according to this embodiment, for example, since comprehensive service quality can be improved in moving image communications, it can contribute to Goal 9 of the Sustainable Development Goals (SDGs) led by the United Nations, "Improve the infrastructure of the information environment, promote sustainable industrialization and achieve expansion of innovation."

Claims

1. A grid decoding device, characterized in that: The grid decoding device has: The displacement decoding unit is configured to decode the displacement bit stream to generate and output the displacement, The displacement decoding unit comprises: a decoding unit configured to decode the level value according to the displacement bit stream; a dequantization unit configured to generate a transform coefficient by dequantizing the level value; an inverse wavelet transform unit configured to generate a prediction residual by performing an inverse wavelet transform on the transform coefficients; as well as The adder is configured to calculate the displacement amount by adding the predicted displacement amount to the prediction residual.

2. The grid decoding device according to claim 1, characterized in that: The grid decoding device further comprises: an inverse DCT unit configured to perform inverse DCT on the level value between the decoding unit and the inverse quantization unit, The inverse quantization unit is configured to generate a prediction residual by performing the inverse wavelet transform on the level value output from the inverse DCT unit.

3. The grid decoding device according to claim 1, characterized in that: The grid decoding device has: a frame buffer configured to acquire and accumulate the displacement output from the adder; and The inter-frame prediction unit is configured to generate the predicted displacement amount by performing inter-frame prediction using the displacement amount of the reference frame read from the frame buffer.

4. The grid decoding device according to claim 3, characterized in that: The inter-frame prediction unit is configured to perform the inter-frame prediction for each patch.

5. The grid decoding device according to claim 3, characterized in that: The inter-frame prediction unit is configured to perform the inter-frame prediction in a spatial domain.

6. The grid decoding device according to claim 3, characterized in that: The inter-frame prediction unit is configured to perform the inter-frame prediction in a frequency domain.

7. The trellis decoding device according to claim 5 or 6, characterized in that: The inter-frame prediction unit is configured to perform the inter-frame prediction on the rearranged displacement amounts.

8. A grid decoding method, characterized in that: The grid decoding method has: A step of decoding the level value according to the displacement bit stream; A step of generating transform coefficients by dequantizing the level values; A step of generating a prediction residual by performing an inverse wavelet transform on the transform coefficients; as well as A step of calculating the displacement amount by adding the predicted displacement amount to the prediction residual.

9. A program product comprising a program for causing a computer to function as a trellis decoding device, the program product being characterized in that: The trellis decoding device includes a displacement decoding unit configured to decode a displacement bit stream, generate and output a displacement, The displacement decoding unit comprises: a decoding unit configured to decode the level value according to the displacement bit stream; a dequantization unit configured to generate a transform coefficient by dequantizing the level value; an inverse wavelet transform unit configured to generate a prediction residual by performing an inverse wavelet transform on the transform coefficients; and The adder is configured to calculate the displacement amount by adding the predicted displacement amount to the prediction residual.

Images