Mesh decoding device, mesh encoding device, mesh decoding method and program

Abstract

To provide a mesh decoding device, a mesh encoding device, a mesh decoding method, and a program that improve the efficiency of encoding.SOLUTION: In a mesh processing system comprising a mesh encoding device and a mesh decoding device, a mesh decoding device 200 includes a basic mesh subdivision unit 203A that is constituted so as to calculate the number of divisions (number of subdivisions) per basic plane and basic patch, on the basis of an inputted basic mesh and division information regarding the basic mesh, and to subdivide the basic mesh on the basis of the calculated number of subdivisions. Here, the basic plane is a plane that constitutes the basic mesh, and the basic patch is a set of several basic planes.SELECTED DRAWING: Figure 21

Description

[Technical Field]

[0001] The present invention relates to a mesh decoding device, a mesh encoding device, a mesh decoding method, and a program. [Background technology]

[0002] Non-Patent Document 1 discloses a technique for encoding a mesh using Non-Patent Document 2. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Cfp for Dynamic Mesh Coding, ISO / IEC JTC1 / SC29 / WG7 N00231, MPEG136 - Online [Non-patent document 2] Google Draco, accessed May 26, 2022 [Online], https: / / google.github.io / draco Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the prior art, because the coordinates and connectivity information of all vertices that make up a dynamic mesh are losslessly encoded, the amount of information cannot be reduced even under conditions where loss is acceptable, resulting in low encoding efficiency. Therefore, the present invention has been made in consideration of the above-mentioned problems, and an object of the present invention is to provide a mesh decoding device, a mesh encoding device, a mesh decoding method, and a program that can improve the encoding efficiency of meshes. [Means for solving the problem]

[0005] A first feature of the present invention is that it provides a mesh decoding device comprising a basic mesh subdivision unit configured to calculate the number of subdivisions of a basic mesh and to perform subdivision of the basic mesh based on the calculated number of subdivisions.

[0006] A second feature of the present invention is a mesh decoding method comprising the steps of: calculating the number of subdivisions of a basic mesh; and subdividing the basic mesh based on the calculated number of subdivisions.

[0007] A third feature of the present invention is a program that causes a computer to function as a mesh decoding device, wherein the mesh decoding device includes a basic mesh subdivision unit that is configured to calculate the number of subdivisions of a basic mesh and to perform subdivision of the basic mesh based on the calculated number of subdivisions. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide a mesh decoding device, a mesh encoding device, a mesh decoding method and a program that can improve mesh encoding efficiency. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a mesh processing system 1 according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of functional blocks of a mesh decoding device 200 according to an embodiment. [Figure 3A] FIG. 3A is a diagram showing an example of a basic mesh and a subdivision mesh. [Figure 3B] FIG. 3B is a diagram showing an example of a basic mesh and a subdivision mesh. [Figure 4] FIG. 4 is a diagram showing an example of the syntax configuration of a basic mesh bit stream. [Figure 5] FIG. 5 is a diagram showing an example of the syntax configuration of the BPH. [Figure 6] FIG. 6 is a diagram showing an example of functional blocks of the basic mesh decoding unit 202 of the mesh decoding device 200 according to an embodiment. [Figure 7] FIG. 7 is a diagram showing an example of the correspondence between the vertices of the basic mesh of the P frame and the vertices of the basic mesh of the I frame. [Figure 8] FIG. 8 is a diagram showing an example of functional blocks of the inter decoding unit 202E of the basic mesh decoding unit 202 of the mesh decoding device 200 according to an embodiment. [Figure 9] FIG. 9 is a diagram illustrating an example of a method for calculating the MVP of a vertex to be decoded by the motion vector prediction unit 202E3 of the inter decoding unit 202E of the basic mesh decoding unit 202 of the mesh decoding device 200 according to one embodiment. [Figure 10] FIG. 10 shows a flowchart illustrating an example of the operation of the motion vector prediction unit 202E3 of the inter decoding unit 202E of the basic mesh decoding unit 202 of the mesh decoding device 200 according to one embodiment. [Figure 11] Figure 11 shows a flowchart illustrating an example of the operation of the motion vector prediction unit 202E3 of the inter decoding unit 202E of the basic mesh decoding unit 202 of the mesh decoding device 200 according to one embodiment, calculating the sum Total_D of distances to surrounding vertices that have already been decoded. [Figure 12] FIG. 12 is a flowchart showing an example of the operation of the motion vector prediction unit 202E3 of the inter decoding unit 202E of the basic mesh decoding unit 202 of the mesh decoding device 200 according to one embodiment, which calculates the MVP using a weighted average. [Figure 13] FIG. 13 is a flowchart showing an example of the operation of the motion vector prediction unit 202E3 of the inter decoding unit 202E of the basic mesh decoding unit 202 of the mesh decoding device 200 according to one embodiment, in selecting an MV from a set of candidate MVs as an MVP. [Figure 14] FIG. 14 is a flowchart showing an example of the operation of the motion vector prediction unit 202E3 of the inter decoding unit 202E of the basic mesh decoding unit 202 of the mesh decoding device 200 according to one embodiment, in creating a set of candidate MVs. [Figure 15] FIG. 15 is a diagram illustrating an example of parallelogram prediction. [Figure 16] FIG. 16 is a flowchart showing an example of an operation for restoring the MVR precision to the original bit precision from the control information adaptive_mesh_flag, adaptive_bit_flag, and precision control parameters generated by decoding the basic mesh bit stream. [Figure 17] FIG. 17 is intended to illustrate an example of MVR encoding. [Figure 18] FIG. 18 is a diagram showing an example of functional blocks of an inter decoding unit 202E of the basic mesh decoding unit 202 of the mesh decoding device 200 according to an embodiment. [Figure 19] FIG. 19 is a diagram showing an example of an operation for determining connection information and the order of vertices using Edgebreaker. [Figure 20] FIG. 20 is a diagram showing an example of functional blocks of the subdivision unit 203 of the mesh decoding device 200 according to an embodiment. [Figure 21] FIG. 21 is a diagram showing an example of functional blocks of a basic mesh subdivision unit 203A of the subdivision unit 203 of the mesh decoding device 200 according to an embodiment. [Figure 22] FIG. 22 is a diagram illustrating an example of a method for dividing a basic surface by the basic surface dividing unit 203A5 of the basic mesh subdivision unit 203A of the subdivision unit 203 of the mesh decoding device 200 according to an embodiment. [Figure 23] FIG. 23 is a flowchart showing an example of the operation of the basic mesh subdivision unit 203A of the subdivision unit 203 of the mesh decoding device 200 according to an embodiment. [Figure 24] FIG. 24 is a diagram showing an example of functional blocks of the subdivision mesh adjustment unit 203B of the subdivision unit 203 of the mesh decoding device 200 according to an embodiment. [Figure 25]FIG. 25 is a diagram showing an example of a case in which an edge division point on a basic surface ABC is moved by the edge division point moving unit 701 of the subdivision mesh adjustment unit 203B of the subdivision unit 203 of the mesh decoding device 200 according to one embodiment. [Figure 26] Figure 26 is a diagram showing an example of a case in which subdivision surface X within a basic surface is re-subdivided by the subdivision surface division unit 702 of the subdivision mesh adjustment unit 203B of the subdivision unit 203 of the mesh decoding device 200 according to one embodiment. [Figure 27] Figure 27 is a diagram showing an example of a case in which all subdivision surfaces are re-subdivided by the subdivision surface division unit 702 of the subdivision mesh adjustment unit 203B of the subdivision unit 203 of the mesh decoding device 200 according to one embodiment. [Figure 28] FIG. 28 is a diagram showing an example of functional blocks of the displacement decoding unit 206 of the mesh decoding device 200 according to one embodiment (when inter prediction is performed in the spatial domain). [Figure 29] FIG. 29 is a diagram showing an example of the configuration of a displacement amount bit stream. [Figure 30] FIG. 30 is a diagram illustrating an example of a syntax configuration of a DPS. [Figure 31] FIG. 31 is a diagram illustrating an example of a syntax configuration of the DPH. [Figure 32] FIG. 32 is a diagram illustrating an example of a correspondence relationship between subdivision vertices between a reference frame and a current frame for decoding when inter prediction is performed in the spatial domain. [Figure 33] FIG. 33 is a diagram showing an example of functional blocks of the displacement decoding unit 206 of the mesh decoding device 200 according to one embodiment (when inter prediction is performed in the frequency domain). [Figure 34] FIG. 34 is a diagram illustrating an example of a correspondence relationship in frequency between a reference frame and a current frame to be decoded when inter prediction is performed in the frequency domain. [Figure 35] FIG. 35 is a flowchart showing an example of the operation of the displacement amount decoding unit 206 of the mesh decoding device 200 according to an embodiment. [Figure 36] FIG. 36 is a diagram illustrating an example of functional blocks of the displacement amount decoding unit 206 according to the first modification. [Figure 37] FIG. 37 is a diagram illustrating an example of functional blocks of the displacement amount decoding unit 206 according to the second modification. [Figure 38] FIG. 38 is a diagram showing an example of functional blocks of an inter decoding unit 202E of the basic mesh decoding unit 202 of the mesh decoding device 200 according to an embodiment. [Figure 39] FIG. 39 is a diagram showing an example of functional blocks of the intra decoding unit 202B of the basic mesh decoding unit 202 of the mesh decoding device 200 according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the components in the following embodiments can be appropriately replaced with existing components, etc., and various variations, including combinations with other existing components, are possible. Therefore, the description of the following embodiments does not limit the content of the invention described in the claims.

[0011] First Embodiment The mesh processing system according to this embodiment will be described below with reference to FIGS.

[0012] 1 is a diagram showing an example of the configuration of a mesh processing system 1 according to this embodiment. As shown in FIG. 1, the mesh processing system 1 includes a mesh encoding device 100 and a mesh decoding device 200.

[0013] FIG. 2 is a diagram showing an example of functional blocks of the mesh decoding device 200 according to this embodiment.

[0014] As shown in Figure 2, the mesh decoding device 200 has a demultiplexing unit 201, a basic mesh decoding unit 202, a subdivision unit 203, a mesh decoding unit 204, a patch integration unit 205, a displacement amount decoding unit 206, and a video decoding unit 207.

[0015] Here, the basic mesh decoding unit 202, the subdivision unit 203, the mesh decoding unit 204, and the displacement amount decoding unit 206 are configured to perform processing in units of patches into which the mesh is divided, and the results of this processing may then be integrated by the patch integration unit 205.

[0016] In the example of FIG. 3A, the mesh is divided into patch 1 consisting of basic faces 1 and 2, and patch 2 consisting of basic faces 3 and 4.

[0017] The demultiplexing unit 201 is configured to separate the multiplexed bit stream into a basic mesh bit stream, a displacement amount bit stream, and a texture bit stream.

[0018] <Basic mesh decoding unit 202> The base mesh decoding unit 202 is configured to decode the base mesh bitstream, generate and output a base mesh.

[0019] Here, the basic mesh is made up of a plurality of vertices in a three-dimensional space and edges connecting these vertices.

[0020] As shown in FIG. 3A, the basic mesh is formed by combining basic faces each represented by three vertices.

[0021] The base mesh decoder 202 may be configured to decode the base mesh bitstream using, for example, Draco as described in Non-Patent Document 2.

[0022] The basic mesh decoding unit 202 may also be configured to generate "subdivision_method_id," which will be described later, as control information for controlling the type of subdivision method.

[0023] Hereinafter, the control information decoded by the basic mesh decoding unit 202 will be described with reference to FIGS.

[0024] FIG. 4 is a diagram showing an example of the syntax configuration of a basic mesh bit stream.

[0025] As shown in Figure 4, first, the base mesh bitstream may include a BPH (Base Patch Header), which is a collection of control information corresponding to the base mesh patch. Second, the base mesh bitstream may include, following the BPH, base mesh patch data that encodes the base mesh patch.

[0026] As described above, the basic mesh bitstream is configured such that each patch data corresponds to one BPH. Note that the configuration of Figure 4 is only an example, and as long as each patch data corresponds to a BPH, elements other than those described above may be added as components of the basic mesh bitstream.

[0027] For example, as shown in Figure 4, the basic mesh bitstream may include an SPS (Sequence Parameter Set), an FH (Frame Header) which is a set of control information corresponding to a frame, or an MH (Mesh Header) which is control information corresponding to a mesh.

[0028] 5 is a diagram showing an example of the syntax configuration of the BPH. Here, if the syntax functions are similar, different syntax names may be used in the syntax mate shown in FIG.

[0029] In the BPH syntax structure shown in Figure 5, the Description column indicates how each syntax is coded. Also, ue(v) indicates an unsigned zeroth-order exponential-Golomb code, and u(n) indicates an n-bit flag.

[0030] The BPH includes at least a control signal (mdu_face_count_minus1) that specifies the number of elementary faces included in the basic mesh patch.

[0031] The BPH also includes at least a control signal (mdu_subdivision_method_id) that specifies the type of subdivision method for the basic mesh for each basic patch.

[0032] The BPH may also include a control signal (mdu_subdivision_num_method_id) that specifies the type of subdivision number generation method for each basic mesh patch. For example, when mdu_subdivision_num_method_id=0, it may be defined that the subdivision number of the basic surface is generated based on the predicted division residual, and when mdu_subdivision_num_method_id=1, it may be defined that the subdivision number of the basic surface is generated recursively.

[0033] When generating the number of subdivisions of the base plane by the prediction division residuals, the BPH may include a control signal (mdu_subdivision_residuals) that specifies the prediction division residuals of the base plane for each index i (i=0, . . . , mdu_face_count_minus1).

[0034] The BPH may include a control signal (mdu_max_depth) for identifying an upper limit on the number of recursive subdivisions to perform for each base mesh patch when recursively generating the number of subdivisions of the base surface.

[0035] The BPH may include a control signal (mdu_subdivision_flag) that specifies whether or not to recursively sub-divide the elementary faces for each index i (i=0,...,mdu_face_count_minus1) and j (j=0,...,mdu_subdivision_depth_index).

[0036] As shown in FIG. 6, the basic mesh decoding unit 202 includes a separating unit 202A, an intra decoding unit 202B, a mesh buffer unit 202C, a connection information decoding unit 202D, and an inter decoding unit 202E.

[0037] The separating unit 202A is configured to classify the basic mesh bitstream into a bitstream of I frames (reference frames) and a bitstream of P frames.

[0038] (Intra decoding unit 202B) The intra decoder 202B is configured to decode the coordinates and connection information of the vertices of the I frame from the bit stream of the I frame using, for example, Draco as disclosed in Non-Patent Document 2.

[0039] FIG. 39 is a diagram illustrating an example of functional blocks of the intra decoding unit 202B.

[0040] As shown in FIG. 39, the intra decoding unit 202B includes a separating unit 202A, an arbitrary intra decoding unit 202B1, and an aligning unit 202B2.

[0041] The arbitrary intra decoder 202B1 is configured to decode the coordinates and connectivity information of unordered vertices of an I frame from the bitstream of the I frame using any method including Draco described in Non-Patent Document 2.

[0042] The sorting unit 202B2 is configured to output the vertices by sorting the unordered vertices into a predetermined order.

[0043] The predetermined order may be, for example, a Morton code order or a raster scan order.

[0044] Furthermore, multiple vertices with the same coordinates, i.e., overlapping vertices, may be combined into a single vertex and then rearranged in a predetermined order.

[0045] The mesh buffer unit 202C is configured to store the vertex coordinates and connection information of the I frame decoded by the intra decoding unit 202B.

[0046] The connection information decoding unit 202D is configured to convert the connection information of the I frame extracted from the mesh buffer unit 202C into connection information of the P frame.

[0047] The inter-decoding unit 202E is configured to decode the coordinates of the vertices of the P frame by adding the coordinates of the vertices of the I frame retrieved from the mesh buffer unit 202C to the motion vectors decoded from the bitstream of the P frame.

[0048] In this embodiment, there is a correspondence between the vertices of the base mesh of the P frame and the vertices of the base mesh of the I frame (reference frame), as shown in Fig. 7. Here, the motion vector decoded by the inter decoding unit 202E is a difference vector between the coordinates of the vertices of the base mesh of the P frame and the coordinates of the vertices of the base mesh of the I frame.

[0049] (Inter-decoding unit 202E) FIG. 8 is a diagram illustrating an example of functional blocks of the inter decoding unit 202E.

[0050] As shown in FIG. 8, the inter decoding unit 202E includes a motion vector residual decoding unit 202E1, a motion vector buffer unit 202E2, a motion vector prediction unit 202E3, a motion vector calculation unit 202E4, and an adder 202E5.

[0051] The motion vector residual decoding unit 202E1 is configured to generate a motion vector residual (MVR) from a P frame bitstream.

[0052] Here, MVR is a motion vector residual indicating the difference between MV (Motion Vector) and MVP (Motion Vector Prediction). MV is a difference vector (motion vector) between the coordinates of the corresponding vertex in the I frame and the coordinates of the vertex in the P frame. MVP is a predicted value (predicted value of the motion vector) of the MV of the target vertex using MV.

[0053] The motion vector buffer unit 202E2 is configured to sequentially store the MVs output by the motion vector calculation unit 202E4.

[0054] The motion vector prediction unit 202E3 is configured to obtain the decoded MV for the vertex connected to the vertex to be decoded from the motion vector buffer unit 202E2, and output the MVP of the vertex to be decoded using all or part of the obtained decoded MV, as shown in Figure 9.

[0055] The motion vector calculation unit 202E4 is configured to add the MVR generated by the motion vector residual decoding unit 202E1 and the MVP output from the motion vector prediction unit 202E3, and output the MV of the vertex to be decoded.

[0056] The adder 202E5 is configured to add the coordinates of the vertex corresponding to the vertex to be decoded, obtained from the decoded basic mesh of the corresponding I frame (reference frame), to the motion vector MV output from the motion vector calculation unit 202E3, and output the coordinates of the vertex to be decoded.

[0057] The individual components of inter decoding unit 202E will be described in detail below.

[0058] FIG. 10 is a flowchart showing an example of the operation of the motion vector prediction unit 202E3.

[0059] As shown in FIG. 10, in step S1001, the motion vector prediction unit 202E3 sets MVP and N to 0.

[0060] In step S1002, the motion vector prediction unit 202E3 obtains a set of MVs of vertices around the vertex to be decoded from the motion vector buffer unit 202E2, identifies vertices for which subsequent processing has not been completed, and transitions to No. If subsequent processing has been completed for all vertices, it transitions to Yes.

[0061] In step S1003, the motion vector prediction unit 202E3 transitions to No if the MV of the vertex to be processed has not been decoded, and transitions to Yes if the MV of the vertex to be processed has been decoded.

[0062] In step S1004, the motion vector prediction unit 202E3 adds MV to MVP and adds 1 to N.

[0063] In step S1005, if N is greater than 0, the motion vector prediction unit 202E3 outputs the result of dividing MVP by N, and if N is 0, outputs 0, and ends the process.

[0064] That is, the motion vector prediction unit 202E3 is configured to output the MVP to be decoded by averaging the decoded motion vectors of the vertices around the vertex to be decoded.

[0065] The motion vector prediction unit 202E3 may be configured to set MVP to 0 when the set of decoded motion vectors is an empty set.

[0066] The motion vector calculation unit 202E4 may be configured to calculate the MV of the vertex to be decoded from the MVP output by the motion vector prediction unit 202E3 and the MVR generated by the motion vector residual decoding unit 202E1 using equation (1).

[0067] MV(k)=MVP(k)+MVR(k) … (1) where k is the index of the vertex. MV, MVR and MVP are vectors with x, y and z components.

[0068] According to this configuration, since only the MVR is coded instead of the MV using the MVP, it is expected that the coding efficiency will be improved.

[0069] The adder 202E5 is configured to calculate the coordinates of a vertex by adding the MV of the vertex calculated by the motion vector calculation unit 202E4 to the coordinates of the vertex in the reference frame corresponding to the vertex, and to leave the connectivity information (Connectivity) as it is in the reference frame.

[0070] Specifically, the adder 202E5 calculates the coordinate v' of the kth vertex using equation (2). i (k) may be calculated.

[0071] v' i (k)=v' j (k)+MV(k) … (2) where v' i (k) is the coordinate of the kth vertex to be decoded in the frame to be decoded, and v' j (k) is the coordinate of the decoded k-th vertex of the reference frame, and MV(k) is the k-th MV of the frame to be decoded, where k=1, 2, . . . , K.

[0072] Furthermore, the connection information of the frame to be decoded is made the same as the connection information of the reference frame.

[0073] Note that the motion vector prediction unit 202E3 calculates the MVP using the decoded MV, and therefore the order of decoding affects the MVP.

[0074] The decoding order is the order in which the vertices of the base mesh in the reference frame are decoded. Generally, if a decoding method is used that uses a fixed repetition pattern to increase the number of base faces one by one from the starting edge, the order of the vertices of the decoded base mesh is determined during the decoding process.

[0075] For example, the motion vector prediction unit 202E3 may use an Edgebreaker to determine the order of decoding vertices in the base mesh of the reference frame.

[0076] According to this configuration, since MVs from a reference frame are coded instead of vertex coordinates, it is expected that coding efficiency will be improved.

[0077] (Modification 1 of the inter-decoding unit 202E) The MVP calculated in the flowchart shown in FIG. 10 is calculated as a simple average of surrounding decoded MVs, but it may also be calculated as a weighted average.

[0078] That is, the motion vector prediction unit 202E3 may be configured to output a predicted value of the motion vector to be decoded by weighting the decoded motion vectors of the vertices around the vertex to be decoded with a weight according to the distance between the vertex to be decoded and the vertices of the reference frame corresponding to the vertex to be decoded and the vertices around the vertex to be decoded.

[0079] In addition, the motion vector prediction unit 202E3 may be configured to output a predicted value of the motion vector to be decoded by taking a weighted average of some of the decoded motion vectors of the vertices around the vertex to be decoded, with a weight corresponding to the distance between the vertex to be decoded and the vertices in the reference frame corresponding to the vertex to be decoded and the vertices around the vertex to be decoded.

[0080] In this first modification, the motion vector prediction unit 202E3 of the inter decoding unit 202E is configured to calculate the MVP in the following procedure.

[0081] First, the motion vector predictor 202E3 is configured to calculate weights.

[0082] FIG. 11 shows a flowchart illustrating an example of the operation of calculating the sum Total_D of the distances to the surrounding decoded vertices.

[0083] As shown in FIG. 11, in step S1101, the motion vector prediction unit 202E3 sets 0 to Total_D.

[0084] Step S1102 is the same as step S1002.

[0085] Step S1103 is the same as step S1003.

[0086] In step S1104, the motion vector prediction unit 202E3 adds e(k) to Total_D.

[0087] That is, the motion vector prediction unit 202E3 refers to a set of vertices around the vertex to be decoded, and adds the distances of the vertices that have already been decoded.

[0088] In this first modification, the motion vector prediction unit 202E3 is configured to calculate the weight using the distance in a reference frame in which the correspondence between vertices is known.

[0089] That is, e(k) in step S1104 of FIG. 11 is the distance between corresponding vertices in the reference frame.

[0090] The motion vector prediction unit 202E3 may be configured to calculate the weight w(k) using equations (3) and (4).

[0091]

number

[0092] The motion vector prediction unit 202E3 may be configured to set the weight according to a predetermined rule depending on the distance.

[0093] For example, the motion vector prediction unit 202E3 may be configured to set the weight to 1 if e(k) is smaller than a threshold TH1, to set the weight to 0.5 if e(k) is smaller than a threshold TH2, and to set the weight to 0 (no weight is used) otherwise.

[0094] According to this configuration, by increasing the weight when the distance to the vertex to be decoded is short, it is possible to expect the effect of being able to calculate the MVP with higher accuracy.

[0095] Second, the motion vector prediction unit 202E3 is configured to refer to the MVP.

[0096] FIG. 12 is a flowchart showing an example of the operation of calculating the MVP using the weighted average.

[0097] As shown in FIG. 12, in step S1201, the motion vector prediction unit 202E3 sets MVP and N to 0.

[0098] Step S1202 is the same as step S1002.

[0099] Step S1203 is the same as step S1003.

[0100] In step S1204, the motion vector prediction unit 202E3 adds w(k)×MV(k) to MVP, and adds 1 to N.

[0101] Step S1205 is the same as step S1005.

[0102] Alternatively, the motion vector prediction unit 202E3 may be configured to calculate the MVP using equation (5).

[0103]

number

[0104] According to this configuration, it is possible to calculate a more accurate MVP using a weighted average, and therefore it is expected that the coding efficiency will be improved by reducing the MVR value and concentrating it around zero.

[0105] (Modification 2 of the inter-decoding unit 202E) In this second modification, the motion vector prediction unit 202E3 is configured to select one MV rather than calculating an MVP using multiple surrounding MVs.

[0106] That is, the motion vector prediction unit 202E3 may be configured to select the MV of the nearest vertex from among the decoded MVs stored in the motion vector buffer unit 202E2 as the MV of the vertex connected to the vertex to be decoded.

[0107] Here, the motion vector prediction unit 202E3 may be configured to construct a candidate list consisting of MVs of vertices connected to the vertex to be decoded from the decoded MVs stored in the motion vector buffer unit 202E2, and select a motion vector from the candidate list based on an index decoded from the bitstream of the P frame (the frame to be decoded).

[0108] FIG. 13 shows a flowchart illustrating an example of the operation of selecting an MV as an MVP from a set of candidate MVs.

[0109] As shown in FIG. 13, in step S1301, the motion vector prediction unit 202E3 decodes the list ID from the bitstream of the P frame.

[0110] In step S1302, the motion vector prediction unit 202E3 selects, from among the candidate MVs, an MV to which a list ID is to be assigned as an MVP.

[0111] In the set of candidate MVs in FIG. 13, the surrounding MVs that have already been decoded and the MVs calculated from their combinations are arranged in a certain order.

[0112] FIG. 14 shows a flowchart showing an example of the operation of creating such a set of candidate MVs.

[0113] As shown in FIG. 14, in step S1401, the motion vector prediction unit 202E3 refers to the set of MVs of vertices around the vertex to be decoded and determines whether processing for all vertices around the vertex to be decoded has been completed.

[0114] If the process is complete, the operation ends; if the process is not complete, the operation proceeds to step S1402.

[0115] In step S1402, the motion vector prediction unit 202E3 determines whether the MV of the target vertex has been decoded.

[0116] If the MV has been decoded, the operation proceeds to step S1403; if the MV has not been decoded, the operation returns to step S1401.

[0117] In step S1403, the motion vector prediction unit 202E3 determines whether or not the MV overlaps with another decoded MV.

[0118] If there is an overlap, the operation returns to step S1401; if there is no overlap, the operation proceeds to step S1404.

[0119] In step S1404, the motion vector prediction unit 202E3 determines a list ID to be assigned to the MV, and in step S1405, includes the MV in a set of candidate MVs.

[0120] In addition, in Figure 14, when determining the list ID, the motion vector prediction unit 202E3 may increase the list ID by one in order, or may determine the list ID in the order of the distance between the vertex to be decoded in the reference frame and the vertex corresponding to vertex k (e(k) in equation (3)).

[0121] According to this configuration, selecting one of the candidate MVs as the MVP may be closer to the MV than the average in some cases, and in such cases, it is expected to have the effect of improving coding efficiency.

[0122] Furthermore, the motion vector prediction unit 202E3 may be configured to add an MV obtained by averaging consecutive MV0 and MV1 from among the candidate MVs described above as a new candidate MV to the list. The motion vector prediction unit 202E3 adds this MV after MV0 and MV1, as shown in Table 1.

[0123] [Table 1] According to this configuration, it is expected that the effect of increasing the possibility that the selected candidate MV is closer to the MV of the vertex to be decoded can be achieved.

[0124] Furthermore, the motion vector prediction unit 202E3 may be configured to select the MV of the nearest vertex from the set of candidate MVs without encoding the list ID. This configuration is expected to further improve encoding efficiency.

[0125] (Modification 3 of the inter-decoding unit 202E) In the above-described embodiment and modifications 1 and 2, the surrounding vertices are vertices connected to the vertex to be decoded.

[0126] In contrast to this, in the present modified example 3, the motion vector predictor 202E3 is configured to calculate the MVP by parallelogram prediction, that is, by using vertices that are not directly connected to the vertex to be decoded.

[0127] As shown in FIG. 15, parallelogram prediction also uses a vertex D on the opposite side of a decoded face that has a shared edge BC with the vertex A to be decoded.

[0128] Furthermore, the shared edges of the vertex A to be decoded are CE and BG in addition to AB. Therefore, in parallelogram prediction, vertices F and H can also be used in a similar manner.

[0129] For example, the motion vector prediction unit 202E3 may be configured to calculate MVP using the plane BCD shown in FIG. 15 according to equation (6).

[0130] MVP = MV(B) + MV(C) - MV(D) … (6) Here, MV(X) is the motion vector of vertex X, and MVP is the motion vector prediction value of vertex A to be decoded.

[0131] Furthermore, when there are multiple shared edges as described above, the motion vector prediction unit 202E3 may average the MVPs of each edge, or may select the face whose center of gravity is closest to the edge.

[0132] (Modification 4 of the inter-decoding unit 202E) In this modification, the MVR generated by the motion vector residual decoding unit 202E1 is not used as is, but the quantization width when expressing the MVR as an integer is controlled.

[0133] In this modification, the motion vector residual decoding unit 202E1 is configured to decode adaptive_mesh_flag, adaptive_bit_flag, and a precision control parameter as control information for controlling the quantization width of MVR.

[0134] That is, the motion vector residual decoding unit 202E1 is configured to decode the adaptive_mesh_flag of the entire basic mesh and the adaptive_bit_flag of each basic patch.

[0135] Here, adaptive_mesh_flag and adaptive_bit_flag are flags that indicate whether or not the quantization width of the MVR described above is to be adjusted, and take the value of either 0 or 1.

[0136] Here, the motion vector residual decoding unit 202E1 decodes adaptive_bit_flag only when adaptive_mesh_flag is valid (ie, 1).

[0137] Furthermore, when adaptive_mesh_flag is invalid (ie, 0), the motion vector residual decoding unit 202E1 regards adaptive_bit_flag as invalid (ie, 0).

[0138] FIG. 16 shows a flowchart illustrating an example of an operation for controlling the quantization width of the decoded MVR from the control information adaptive_mesh_flag, adaptive_bit_flag, and precision control parameters generated by decoding the basic mesh bit stream.

[0139] As shown in FIG. 16, in step S1601, the motion vector prediction unit 202E3 determines whether adaptive_mesh_flag is 0 or not.

[0140] If it is determined that the adaptive_mesh_flag of the entire mesh is 0, this operation ends.

[0141] On the other hand, if it is determined that adaptive_mesh_flag is 1 for the entire mesh, the operation proceeds to step S1602.

[0142] In step S1602, the motion vector prediction unit 202E3 determines whether or not there is an unprocessed patch in the frame.

[0143] In step S1603, the motion vector prediction unit 202E3 determines whether adaptive_mesh_flag decoded for each patch is 0 or not.

[0144] If it is determined that adaptive_mesh_flag is 0, the operation returns to step S1601.

[0145] On the other hand, if it is determined that adaptive_mesh_flag is 1, the operation proceeds to step S1604.

[0146] In step S1604, the motion vector prediction unit 202E3 controls the quantization step size of the MVR based on a precision control parameter, which will be described later.

[0147] The MVR value with the quantization width controlled in this way is called "MVRQ (Motion Vector Residual Quantization)."

[0148] Here, the motion vector prediction unit 202E3 may be configured to use the quantization width of the MVR corresponding to the quantization width control parameter generated by decoding the basic mesh bitstream, for example, by referring to a table such as Table 2.

[0149] [Table 2] This configuration is expected to improve coding efficiency by controlling the quantization width of MVR. Furthermore, the hierarchical structure of adaptive_mesh_flag at the mesh level and adaptive_mesh_flag at the patch level is expected to minimize wasted bits when the quantization width of MVR is not controlled.

[0150] (Modification 5 of the inter-decoding unit 202E) If the MVR generated by the motion vector residual decoding unit 202E1 is not coded, an error occurs. In this fifth modification, discrete motion vector differences are coded to correct such errors.

[0151] 17, MVR can take on magnitudes of 1, 2, 4, and 8 in six directions of the x-axis, y-axis, and z-axis. An example of such encoding is shown in Tables 3 and 4.

[0152] Furthermore, MVR encoding may be performed using a combination of multiple directions. For example, correction may be performed in the order of 2 in the positive direction of the x-axis and 1 in the positive direction of the y-axis.

[0153] [Table 3]

[0154] [Table 4] According to this configuration, it is expected that the coding efficiency of discrete motion vector differences will be higher than that of MVR coding.

[0155] Further modifications of the inter decoding unit 202E will be described below.

[0156] In a further modification of the above-described inter decoding unit 202E, the following functional blocks are added before implementing the above-described inter decoding unit 202E.

[0157] Specifically, as shown in FIG. 18, the inter-decoding unit 202E includes, in addition to the configuration shown in FIG. 8, an overlapping vertex searching unit 202E6, an overlapping vertex determining unit 202E7, a motion vector obtaining unit 202E8, an all skip mode single, and a skip mode single.

[0158] Here, the All skip mode signal is at the beginning of the P frame bitstream, has at least two values, and is one bit or more than one bit.

[0159] One of them (when the All skip mode signal indicates Yes, e.g., 1) is a signal to not decode the motion vectors of all duplicated vertices of the P frame from the bitstream, but to copy the motion vectors of the duplicated vertices.

[0160] The other signal (when the All skip mode signal indicates No, for example, 0) is a signal that performs different processing at each vertex of the P frame. Furthermore, the other signal may have another value. For example, the other signal is Single, which does not perform processing in the motion vector acquisition unit 202E8 for the motion vectors of all overlapping vertices, but performs processing similar to that of the inter decoding unit 202E shown in FIG. 8.

[0161] Here, if the All skip mode signals indicate No, the Skip mode signal has two values ​​for each overlapping vertex, and is one bit.

[0162] The Skip mode signal is a signal that, if the All skip mode signal indicates Yes (for example, if it is 1), the motion vector of the vertex is not decoded from the bitstream, and the motion vector of the overlapped vertex is copied.

[0163] When the All skip mode signal indicates No (for example, 0), the Skip mode signal is Single, in which processing is not performed in the motion vector acquisition unit 202E8 for the motion vector of the vertex, and processing similar to that of the inter decoding unit 202E shown in Figure 8 is performed.

[0164] The above-mentioned Skip mode signal may be decoded directly from the bitstream, or data identifying the overlapped vertex (e.g., the index of the overlapped vertex) that performs processing similar to that of the inter-decoding unit 202E shown in Figure 8 may be decoded from the bitstream, and the Skip mode signal may be calculated from such data.

[0165] Furthermore, as shown in Figure 38, instead of calculating the Skip mode signal, the motion vector decoding method for the vertex may be determined in the same manner as described above, using data identifying the overlapping vertex (e.g., the index of the overlapping vertex) that performs processing similar to that of the inter decoding unit 202E shown in Figure 8 above.

[0166] The duplicated vertex search unit 202E6 is configured to search for the index of a vertex (hereinafter referred to as a duplicated vertex) with matching coordinates from the geometric information of the basic mesh of the decoded reference frame, and store the index in a buffer (not shown).

[0167] Specifically, the input to the overlapping vertex search unit 202E6 is the index (in decoding order) and position coordinates of each vertex of the basic mesh of the decoded reference frame.

[0168] The output of the duplicate vertex search unit 202E6 is a list of pairs of the index (vindex0) of a vertex where a duplicate vertex exists and the index (vindex1) of such duplicate vertex. Here, the list of such pairs is saved in the buffer repVert in the order of index0.

[0169] Furthermore, since the vertex of vindex1 is decoded before vindex0, the relationship vindex0>vindex1 holds.

[0170] In order to identify duplicate vertices in the base mesh of the reference frame, a special signal is used to decode the index of the duplicate vertex, instead of the position coordinate, for the vertex where the duplicate vertex exists. This special signal allows the pair of the index of the relevant vertex and the index of the duplicate vertex to be stored in decoding order.

[0171] The duplicated vertex determination unit 202E7 is configured to determine whether or not the relevant vertex has a duplicated vertex among the decoded vertices.

[0172] Here, if the index of the relevant vertex is among the indexes of vertices where a duplicate vertex exists, the duplicate vertex determination unit 202E7 determines that there is a duplicate vertex among the decoded vertices. Note that since the relevant vertex comes in the decoding order, the above search is not necessary.

[0173] Here, if the overlapping vertex determination unit 202E7 determines that there is no overlapping vertex for the vertex in question, the same processing as that of the inter decoding unit 202E shown in FIG. 8 is performed.

[0174] The motion vector acquisition unit 202E8 is configured to acquire the motion vector of a vertex having the same index as the overlapping vertex from the motion vector buffer unit 202E2 that stores decoded motion vectors when there is an overlapping vertex of the corresponding vertex, when the All skip mode signal indicates Yes, or when the All skip mode signal indicates No and the Skip mode signal of the corresponding vertex indicates Yes, and set it as the motion vector of the corresponding vertex.

[0175] Here, if the All skip mode signal indicates No and the Skip mode signal of the vertex indicates No, the same processing as that of the inter decoding unit 202E shown in FIG. 8 is performed instead of the motion vector acquisition unit 202E8.

[0176] According to this configuration, it is possible to expect an effect of reducing the decoding calculation and the amount of code for the motion vector for the vertices where overlapping vertices exist.

[0177] In a further modification of the inter decoding unit 202E described above, the inter decoding unit 202E obtains, from the decoded base mesh of the reference frame, the correspondence between the vertices of the reference frame and the vertices of the frame to be decoded.

[0178] Then, based on this correspondence, the inter-decoding unit 202E is configured to make the connectivity information of the vertices of the frame to be decoded the same as the connectivity information of the decoded vertices of the reference frame without encoding the connectivity information of the vertices of the frame to be decoded.

[0179] The inter-decoding unit 202E divides the base mesh of the frame to be decoded into two types of regions based on the signal in the decoding order of the vertices of the reference frame. The first region is decoded using inter processing, and the second region is decoded using intra processing.

[0180] The above-mentioned region is defined as a region formed by a plurality of vertices that are consecutive in decoding order when decoding the basic mesh of the reference frame.

[0181] In addition, the following two implementations are assumed for the means of decoding the coordinates of the vertices of the base mesh of the frame to be decoded using a signal. (Measure 1) In means 1, the signals are vertex_idx1, vertex_idx2 and intra_flag.

[0182] Here, vertex_idx1 and vertex_idx2 are indices (vertex indices) of the vertices in the decoding order, and intra_flag is a flag indicating whether the above-mentioned inter-decoding method or intra-decoding method is used. There may be multiple such signals.

[0183] That is, vertex_idx1 and vertex_idx2 are vertex indices that define the start position and end position of the above-mentioned partial region (first region and second region). (Measure 2) In the method 2, the Edgebreaker decodes the connection information of the basic mesh of the reference frame, and the decoding order of the vertex coordinates is assumed to be the order determined by the Edgebreaker.

[0184] FIG. 19 is a diagram showing an example of an operation for determining connection information and the order of vertices using Edgebreaker.

[0185] In FIG. 19, arrows indicate the decoding order of the connection information, numbers indicate the decoding order of the vertices, and arrows of the same line type define the same area.

[0186] In the means 2, the signal is only intra_flag, which is a flag indicating whether the decoding method is an inter-decoding method or an intra-decoding method.

[0187] That is, in the means 2, the inter decoding unit 202E is configured to divide into a first region and a second region using an Edgebreaker.

[0188] <Subdivision section 203> The subdivision unit 203 is configured to generate and output the added subdivision vertices and their connection information from the basic mesh decoded by the basic mesh decoding unit 202 using the subdivision method indicated by the control information.

[0189] Here, the basic mesh, the added subdivision vertices, and the connection information therebetween are collectively referred to as a "subdivision mesh."

[0190] The subdivision unit 202 is configured to identify the type of subdivision method from subdivision_method_id, which is control information generated by decoding the basic mesh bitstream.

[0191] The subdivision unit 202 will be described below with reference to FIGS. 3A and 3B.

[0192] 3A and 3B are diagrams for explaining an example of the operation of generating subdivision vertices from a basic mesh.

[0193] FIG. 3A is a diagram showing an example of a basic mesh made up of five vertices.

[0194] Here, the subdivision may be performed using, for example, a mid-edge division method in which the midpoints of the sides of each basic face are connected to each other, resulting in a basic face being divided into four faces.

[0195] Figure 3B shows an example of a subdivision mesh obtained by dividing a base mesh consisting of five vertices. In the subdivision mesh shown in Figure 3B, eight subdivision vertices (white circles) are generated in addition to the original five vertices (black circles).

[0196] By decoding the displacement amount for each subdivision vertex thus generated by the displacement amount decoder 206, it is possible to expect improvement in coding performance.

[0197] Also, different subdivision methods may be applied to each patch. This allows the displacement decoded by the displacement decoder 206 to be adaptively changed for each patch, which is expected to improve coding performance. Information about the divided patches is received as patch_id, which is control information.

[0198] The subdivision section 203 will be described below with reference to Fig. 20. Fig. 21 is a diagram showing an example of functional blocks of the subdivision section 203.

[0199] As shown in FIG. 21, the subdivision unit 203 has a basic mesh subdivision unit 203A and a subdivision mesh adjustment unit 203B.

[0200] (Basic mesh refinement section 203A) The basic mesh subdivision unit 203A is configured to calculate the number of divisions (number of subdivisions) for each basic surface and basic patch based on the input basic mesh and division information of the basic mesh, subdivide the basic mesh based on this number of divisions, and output the subdivision surface.

[0201] That is, the basic mesh subdivision unit 203A may be configured so that the number of divisions can be changed for each basic surface and each basic patch.

[0202] Here, a basic face is a face that constitutes a basic mesh, and a basic patch is a set of several basic faces.

[0203] In addition, the basic mesh subdivision unit 203A may be configured to predict the number of fine subdivisions of the basic surface and calculate the number of subdivisions of the basic surface by adding the predicted subdivision number residual to the predicted number of subdivisions of the basic surface.

[0204] The basic mesh subdivision unit 203A may also be configured to calculate the number of subdivisions of a basic surface based on the number of subdivisions of adjacent basic surfaces of the basic surface.

[0205] The basic mesh subdivision unit 203A may also be configured to calculate the number of subdivisions of a basic surface based on the number of subdivisions of the basic surface that was accumulated immediately before.

[0206] The basic mesh subdivision unit 203A may also be configured to generate vertices that divide the three sides that make up the basic surface, and to connect the generated vertices to subdivide the basic surface.

[0207] As shown in FIG. 21, a basic mesh subdivision section 203A is followed by a subdivision mesh adjustment section 203B, which will be described later.

[0208] An example of the processing of the basic mesh subdivision unit 203A will be described below with reference to FIGS.

[0209] FIG. 21 is a diagram showing an example of functional blocks of the basic mesh subdivision unit 203A, and FIG. 23 is a flowchart showing an example of the operation of the basic mesh subdivision unit 203A.

[0210] As shown in FIG. 21, the basic mesh subdivision unit 203A has a basic surface division number buffer unit 203A1, a basic surface division number reference unit 203A2, a basic surface division number prediction unit 203A3, an addition unit 203A4, and a basic surface division unit 203A5.

[0211] The basic surface division number buffer unit 203A1 stores division information of basic surfaces including the division number of the basic surface, and is configured to output the division information of the basic surface to the basic surface division number reference unit 203A2.

[0212] Here, the size of the basic surface division number buffer unit 203A1 may be set to 1, and the unit may be configured to output the division number of the basic surface that was stored immediately before to the basic surface division number reference unit 203A2.

[0213] That is, by setting the size of the basic surface division number buffer unit 203A1 to 1, it may be configured to refer to only the fine division number decoded last (the subdivision number decoded immediately before).

[0214] The basic surface division number reference unit 203A2 is configured to output a reference not possible to the basic surface division number prediction unit 203A3 when there is no adjacent basic surface to the basic surface to be decoded, or when there is an adjacent basic surface to the basic surface to be decoded but the division number has not been determined.

[0215] On the other hand, if there is a basic surface adjacent to the basic surface to be decoded and the number of divisions has been determined, the basic surface division number reference unit 203A2 is configured to output the number of divisions to the basic surface division number prediction unit 203A3.

[0216] The basic surface division number prediction unit 203A3 is configured to predict the division number (number of subdivisions) of the basic surface based on one or more input division numbers, and output the predicted division number (predicted division number) to the addition unit 203A4.

[0217] Here, the basic surface division number prediction unit 203A3 is configured to output 0 to the addition unit 203A4 when only reference impossible is input from the basic surface division number reference unit 203A2.

[0218] In addition, when one or more division numbers are input, the basic surface division number prediction unit 203A3 may be configured to generate a predicted division number using any of statistical values ​​such as the average value, maximum value, minimum value, or mode of the input division numbers.

[0219] The basic surface division number prediction unit 203A3 may be configured to generate the division number of the most adjacent surface as the predicted division number when one or more division numbers are input.

[0220] The addition unit 203A4 is configured to output the division number obtained by adding the prediction division number residual decoded from the prediction residual bitstream and the prediction division number obtained from the basic plane division number prediction unit 203A3 to the basic plane division unit 203A5.

[0221] The basic surface dividing unit 203A5 is configured to subdivide the basic surface based on the division number input from the addition unit 203A4.

[0222] Fig. 22 shows an example of a case where a basic surface is divided into 9. The division method of the basic surface by the basic surface dividing section 203A5 will be described with reference to Fig. 22.

[0223] The basic surface dividing unit 203A5 generates points A_1, . . . , A_(N-1) that divide the side AB that constitutes the basic surface into N equal parts (N=3).

[0224] Similarly, the basic surface division unit 203A5 also divides the sides BC and CA into N equal parts, generating points B_1, …, B_(N-1) and C_1, …, C_(N-1) respectively.

[0225] Hereafter, the points on sides AB, BC, and CA are referred to as "side division points".

[0226] For all i (i = 1, 2, …, N-1), the basic surface division unit 203A5 generates sides A_i B_(N-i), B_i C_(N-i), and C_i A_(N-i), generating N 2 subdivision surfaces.

[0227] Next, referring to FIG. 23, the processing procedure of the basic mesh subdivision unit 203A will be described.

[0228] In step S2201, it is determined whether the subdivision process for the last basic surface has been completed. If the process is completed, it ends; otherwise, it proceeds to step S2202.

[0229] In step S2202, the basic mesh subdivision unit 203A determines whether Depth < mdu_max_depth.

[0230] Here, Depth is a variable representing the current depth, with an initial value of 0, and mdu_max_depth represents the maximum depth determined for each basic surface.

[0231] If the condition in step S2202 is satisfied, this processing procedure proceeds to step S2203; otherwise, this processing procedure returns to step S2201.

[0232] In step S2203, the basic mesh subdivision unit 203A determines whether mdu_subdivision_flag at the current depth is 1.

[0233] If Yes, this processing procedure returns to step S2201; if No, this processing procedure proceeds to step S2204.

[0234] In step S2204, the basic mesh subdivision unit 203A further subdivides all subdivision surfaces within the basic surface.

[0235] Here, if the subdivision process has never been performed on the basic surface, the basic mesh subdivision unit 203A subdivides the basic surface.

[0236] The subdivision method is the same as that explained in step S2204.

[0237] Specifically, if the basic surface has never been subdivided, the basic surface is subdivided as shown in Figure 22. If the basic surface has been subdivided at least once, the subdivision surface is 2 In the example of Figure 22, the face consisting of vertices A_2, B, and B_1 is further divided into N 2 Generate faces.

[0238] When the subdivision process is completed, the process proceeds to step S2205.

[0239] In step S2205, the basic mesh subdivision unit 203A adds 1 to Depth, and the processing procedure returns to step S2202.

[0240] (Fine division mesh adjustment unit 203B) Next, a specific example of the processing performed by the finely divided mesh adjustment unit 203B will be described. An example of the processing performed by the finely divided mesh adjustment unit 203B will be described below with reference to FIGS.

[0241] FIG. 24 is a diagram showing an example of functional blocks of the finely divided mesh adjustment unit 203B.

[0242] As shown in FIG. 24, the subdivision mesh adjustment unit 203B has an edge division point moving unit 701 and a subdivision surface division unit 702. (Edge division point moving unit 701) The edge division point moving unit 701 is configured to move an edge division point of a basic face to any of the edge division points of an adjacent basic face for an input initial subdivision face, and output a subdivision face.

[0243] Fig. 25 shows an example in which an edge division point on a basic face ABC is moved. For example, as shown in Fig. 25, the edge division point moving unit 701 may be configured to move an edge division point of the basic face ABC to an edge division point of the nearest adjacent basic face.

[0244] (Subdivision surface division part 702) The subdivision surface division unit 702 is configured to again subdivide the input subdivision surface and output a decoded subdivision surface.

[0245] FIG. 26 is a diagram showing an example of a case where subdivision is performed again on a subdivision surface X within a basic surface.

[0246] As shown in FIG. 26, the subdivision surface division unit 702 may be configured to generate a new subdivision surface within a basic surface by connecting the vertices that make up the subdivision surface with the edge division points of an adjacent basic surface.

[0247] FIG. 27 is a diagram showing an example of a case where the above-described subdivision processing is performed on all subdivision surfaces.

[0248] The mesh decoding unit 204 is configured to generate and output a decoded mesh using the subdivision mesh generated by the subdivision unit 203 and the displacement amount decoded by the displacement amount decoding unit 206 .

[0249] Specifically, the mesh decoding unit 204 is configured to generate a decoded mesh by adding a corresponding displacement amount to each subdivision vertex, where information about which subdivision vertex each displacement amount corresponds to is indicated by control information.

[0250] The patch integration unit 205 is configured to integrate the decoded mesh generated by the mesh decoding unit 206 by the number of patches and output the integrated mesh.

[0251] Here, the patch division method is defined by the mesh encoding device 100. For example, the patch division method may be configured to calculate a normal vector for each basic face, select the basic face with the most similar normal vector among adjacent basic faces, combine both basic faces into the same patch, and then repeat this procedure for the next basic face in sequence.

[0252] The video decoding unit 207 is configured to decode and output the texture by video encoding. For example, the video decoding unit 207 may use HEVC in Non-Patent Document 1.

[0253] <Displacement Amount Decoding Unit 206> The displacement decoding unit 206 is configured to decode the displacement bitstream to generate and output a displacement.

[0254] FIG. 28 is a diagram showing an example of the displacement amount for a certain subdivision vertex.

[0255] In the example of FIG. 3B, there are eight subdivision vertices, and therefore the displacement decoding unit 206 is configured to define eight displacements, each expressed as a scalar or vector, for each subdivision vertex.

[0256] The displacement amount decoding unit 206 will be described below with reference to Fig. 28. Fig. 28 is a diagram showing an example of functional blocks of the displacement amount decoding unit 206.

[0257] As shown in FIG. 28, the displacement decoding unit 206 includes a decoding unit 206A, an inverse quantization unit 206B, an inverse wavelet transformation unit 206C, an adder 206D, an inter prediction unit 206E, and a frame buffer 206F.

[0258] The decoding unit 206A is configured to decode and output level values ​​and control information by performing variable length decoding on the received displacement bitstream. Here, the level values ​​obtained by variable length decoding are output to the inverse quantization unit 206B, and the control information is output to the inter prediction unit 206E.

[0259] An example of the configuration of a displacement amount bit stream will be described below with reference to Fig. 29. Fig. 29 is a diagram showing an example of the configuration of a displacement amount bit stream.

[0260] As shown in FIG. 29, first, the displacement bitstream may include a DPS (Displacement Parameter Set) which is a set of control information related to decoding of the displacements.

[0261] Second, the displacement bitstream may include a DPH (Displacement Patch Header), which is a collection of control information corresponding to a patch.

[0262] Third, the displacement bitstream may contain, next to the DPH, the coded displacements that make up the patch.

[0263] As described above, the displacement bitstream has a configuration in which each coded displacement corresponds to one DPH and one DPS.

[0264] Note that the configuration in Fig. 29 is merely an example. As long as the DPH and DPS correspond to each coded displacement, elements other than those described above may be added as components of the displacement bitstream.

[0265] For example, as shown in FIG. 29, the displacement bitstream may include an SPS (Sequence Parameter Set).

[0266] FIG. 30 is a diagram illustrating an example of a syntax configuration of a DPS.

[0267] In FIG. 30, the Descriptor column indicates how each syntax is coded.

[0268] In addition, in FIG. 30, ue(v) means an unsigned zeroth-order exponential-Golomb code, and u(n) means an n-bit flag.

[0269] When there are multiple DPSs, the DPS includes at least DPS id information (dps_displacement_parameter_set_id) for identifying each DPS.

[0270] Furthermore, the DPS may include a flag (interprediction_enabled_flag) that controls whether or not inter prediction is performed.

[0271] For example, it may be defined that inter prediction is not performed when interprediction_enabled_flag is 0, and that inter prediction is performed when interprediction_enabled_flag is 1. It may be defined that inter prediction is not performed when interprediction_enabled_flag is not included.

[0272] The DPS may include a flag (dct_enabled_flag) that controls whether or not to perform the inverse DCT.

[0273] For example, it may be defined that the inverse DCT is not performed when dct_enabled_flag is 0, and that the inverse DCT is performed when dct_enabled_flag is 1. It may also be defined that the inverse DCT is not performed when dct_enabled_flag is not included.

[0274] FIG. 31 is a diagram illustrating an example of a syntax configuration of the DPH.

[0275] As shown in FIG. 31, a DPH includes at least DPS id information for specifying a DPS corresponding to each DPH.

[0276] The inverse quantization unit 206B is configured to inversely quantize the level values ​​decoded by the decoding unit 206A to generate and output transform coefficients.

[0277] The inverse wavelet transform unit 206C is configured to perform an inverse wavelet transform on the transform coefficients generated by the inverse quantization unit 206B to generate and output prediction residuals.

[0278] (Inter prediction unit 206E) The inter prediction unit 206E is configured to perform inter prediction using the decoding displacement amount of the reference frame read from the frame buffer 206F, thereby generating and outputting a predicted displacement amount.

[0279] The inter prediction unit 206E is configured to perform such inter prediction only when interprediction_enabled_flag is set to one.

[0280] The inter prediction unit 206E may perform inter prediction in the spatial domain or in the frequency domain. The inter prediction may be bidirectional prediction using a past reference frame and a future reference frame in terms of time.

[0281] FIG. 28 shows an example of functional blocks of the inter prediction unit 206E when performing inter prediction in the spatial domain.

[0282] When performing inter prediction in the spatial domain, the inter prediction unit 206E may determine the predicted displacement amount of a subdivision vertex in the target frame by directly referring to the decoded displacement amount of the corresponding subdivision vertex in the reference frame.

[0283] Alternatively, the predicted displacement of a certain subdivision vertex in the target frame may be determined probabilistically according to a normal distribution with estimated mean and variance using decoded displacements of corresponding subdivision vertices in multiple reference frames. In this case, the variance may be set to zero and the predicted displacement may be uniquely determined using only the mean.

[0284] Alternatively, the predicted displacement of a subdivision vertex in the target frame may be determined based on a regression curve estimated using the decoded displacement of corresponding subdivision vertices in multiple reference frames, with time as the explanatory variable and displacement as the objective variable.

[0285] In the mesh coding device 100, the order of the decoded displacement amounts may be rearranged for each frame in order to improve coding efficiency.

[0286] In such a case, the inter prediction unit 206E may be configured to perform inter prediction on the reordered decoding displacement amounts.

[0287] The correspondence between the subdivision vertices of the reference frame and the frame to be decoded is indicated by control information.

[0288] FIG. 32 is a diagram illustrating an example of a correspondence relationship between subdivision vertices between a reference frame and a current frame for decoding when inter prediction is performed in the spatial domain.

[0289] FIG. 33 shows an example of functional blocks of the inter prediction unit 206E when performing inter prediction in the frequency domain.

[0290] When performing inter prediction in the frequency domain, the inter prediction unit 206E may determine predicted wavelet transform coefficients of a frequency in the current frame by directly referring to decoded wavelet transform coefficients of a corresponding frequency in the reference frame.

[0291] The inter prediction unit 206E may perform stochastic inter prediction according to a normal distribution with estimated mean and variance, using the decoded displacement amounts or decoded wavelet transform coefficients of the subdivision vertices in a plurality of reference frames.

[0292] The inter prediction unit 206E may perform inter prediction based on a regression curve estimated using the decoded displacement amounts or decoded wavelet transform coefficients of the subdivision vertices in multiple reference frames, with time as the explanatory variable and the displacement amount as the objective variable.

[0293] The inter prediction unit 206E may be configured to perform bidirectional inter prediction using a temporally past reference frame and a temporally future reference frame.

[0294] In the mesh coding device 100, the order of the decoded wavelet transform coefficients may be rearranged for each frame in order to improve coding efficiency.

[0295] The frequency correspondence between the reference frame and the frame to be decoded is indicated by control information.

[0296] FIG. 34 is a diagram illustrating an example of a correspondence relationship in frequency between a reference frame and a current frame to be decoded when inter prediction is performed in the frequency domain.

[0297] Furthermore, when the subdivision unit 203 divides the basic mesh into multiple patches, the inter prediction unit 206E is also configured to perform inter prediction for each divided patch, which increases the temporal correlation between frames and is expected to improve coding performance.

[0298] The adder 206D receives the prediction residual from the inverse wavelet transform unit 206C and the prediction displacement from the inter prediction unit 206E.

[0299] The adder 206D is configured to calculate and output a decoded displacement by adding the prediction residual and the predicted displacement.

[0300] The decoded displacement calculated by the adder 206D is also output to the frame buffer 206F.

[0301] The frame buffer 206F is configured to receive and store the decoded displacement from the adder 206D.

[0302] Here, the frame buffer 206F outputs the decoded displacement amount at the corresponding vertex in the reference frame in accordance with control information (not shown).

[0303] FIG. 35 is a flowchart showing an example of the operation of the displacement amount decoding unit 206.

[0304] As shown in FIG. 35, in step S3501, the displacement decoding unit 206 determines whether or not this process has been completed for all patches.

[0305] If yes, the operation ends, and if no, the operation proceeds to step S3502.

[0306] In step S3502, the displacement decoding unit 206 performs inverse DCT on the patch to be decoded, and then performs inverse quantization and inverse wavelet transform.

[0307] In step S3503, the displacement decoding unit 206 determines whether interprediction_enabled_flag is 1 or not.

[0308] If yes, the operation proceeds to step S3504; if no, the operation returns to step S3501.

[0309] In step S3504, the displacement decoding unit 206 performs the above-described inter prediction and addition.

[0310] <Variation 1> Hereinafter, with reference to FIG. 36, Modification 1 of the first embodiment will be described, focusing on the differences from the first embodiment.

[0311] FIG. 36 is a diagram showing an example of functional blocks of the displacement amount decoding unit 206 according to the first modification.

[0312] As shown in FIG. 36, the displacement decoding unit 206 according to the present first modification includes an inverse DCT unit 206G after the decoding unit 206A, that is, between the decoding unit 206A and the inverse quantization unit 206B.

[0313] That is, in the present first modification, the inverse quantization unit 206B is configured to generate a prediction residual by performing an inverse wavelet transform on the level values ​​output from the inverse DCT unit 202G.

[0314] <Variation 2> Hereinafter, with reference to FIG. 37, Modification 2 of the first embodiment will be described, focusing on the differences from the first embodiment.

[0315] As shown in FIG. 37, the displacement decoding unit 206 according to the second modification includes a video decoding unit 2061, an image developing unit 2062, an inverse quantization unit 2063, and an inverse wavelet transformation unit 2064.

[0316] The video decoding unit 2061 is configured to decode the received displacement amount bitstream by video coding, and output video.

[0317] For example, the video decoding unit 2061 may use HEVC in Non-Patent Document 1.

[0318] Furthermore, the video decoding unit 2061 may use a video encoding method in which the motion vector is always set to 0. For example, the video decoding unit 2061 may always set the motion vector of HEVC to 0 and always use inter prediction at the same position.

[0319] Alternatively, the video decoding unit 2061 may use a video encoding method in which conversion is always skipped. For example, the video decoding unit 2061 may always perform HEVC conversion in a conversion skip mode and use a video encoding method without conversion.

[0320] The image development unit 2062 is configured to develop the video decoded by the video decoding unit 2061 as a level value for each image (frame) and output it.

[0321] In this expansion method, the image expansion unit 2062 can specify the level values ​​by working backward from the arrangement of the level values ​​in the image indicated by the control information.

[0322] The image development unit 2062 may arrange the level values ​​in the image in the order of raster operation, for example, from high frequency components to low frequency components.

[0323] The inverse quantization unit 2063 is configured to inversely quantize the level values ​​generated by the image development unit 2062 to generate and output transformation coefficients.

[0324] The inverse wavelet transform unit 2064 is configured to perform an inverse wavelet transform on the transform coefficients generated by the inverse quantization unit 2063, thereby generating and outputting a decoded displacement.

[0325] The mesh encoding device 100 and the mesh decoding device 200 described above may be realized as a program that causes a computer to execute each function (each step). [Industrial Applicability]

[0326] According to this embodiment, for example, it is possible to improve the overall service quality in video communication, which will contribute to the achievement of Goal 9 of the United Nations-led Sustainable Development Goals (SDGs), which is to "Develop resilient infrastructure, promote sustainable industrialization and foster innovation." [Explanation of symbols]

[0327] 1...Mesh processing system 100...Mesh coding device 200...Mesh decoding unit 201…Demultiplexer 202...Basic mesh decoding unit 202A…Separation part 202B... Intra decoding unit 202B1...Optional intra-decoding unit 202B2...Alignment section 202C...Mesh buffer section 202D...Connection information decoding unit 202E...Inter-decoding unit 202E1...Motion vector decoding unit 202E2...Motion vector buffer unit 202E3...Motion vector prediction unit 202E4...Motion vector calculation unit 202E5...adder 202E6...Duplicate vertex search unit 202E7...Duplicate vertex discrimination unit 202E8...Motion vector acquisition unit 203…Subdivision part 203A...Basic mesh refinement section 203A1...Basic face division number buffer section 203A2…Basic surface division number reference section 203A3...Basic surface division number prediction unit 203A4...Addition section 203A5…Basic surface division part 203B…Fine division mesh adjustment section 204...Mesh decoding unit 205…Patch Integration Department 206...Displacement amount decoding unit 206A...Decoding unit 206B, 2063...Dequantization section 206C, 2064...inverse wavelet transform section 206D...adder 206E…Inter prediction section 206F...frame buffer 206G…Reverse DCT section 2062...Image development section 207, 2061...Video decoding section

Claims

1. A mesh decoding device, comprising: a subdivision unit configured to calculate a number of subdivisions of a basic mesh defined three-dimensionally in units of basic faces or basic patches, and to generate a subdivision mesh by subdividing the basic mesh based on the calculated number of subdivisions; a displacement decoder configured to decode the displacement bitstream to generate a displacement; a mesh decoding unit configured to generate a decoded mesh using the subdivision mesh generated by the subdivision unit and the displacement amount decoded by the displacement amount decoding unit.

2. The mesh decoding device according to claim 1, characterized in that the subdivision unit is configured to predict the number of subdivisions of a base surface and decode the number of subdivisions of the base surface by adding a predicted subdivision number residual to the predicted number of subdivisions of the base surface.

3. 2. The mesh decoding device according to claim 1, wherein the subdivision unit is configured to recursively decode the number of subdivisions of the base surface based on flag information that controls the subdivision.

4. The mesh decoding device according to claim 1, characterized in that the subdivision unit is configured to generate vertices that divide the three sides that constitute the basic surface by any number, and to subdivide the basic surface by connecting the generated vertices.

5. A mesh decoding device, comprising: a subdivision unit configured to subdivide a base mesh defined in three dimensions to generate a subdivision mesh; a displacement decoder configured to decode the displacement bitstream to generate a displacement; a mesh decoding unit configured to generate a decoded mesh using the subdivision mesh generated by the subdivision unit and the displacement amount decoded by the displacement amount decoding unit, The mesh decoding device is characterized in that the subdivision unit includes a subdivision mesh adjustment unit configured to adjust positions of edge division points of a base face that is a face of the base mesh.

6. 6. The mesh decoding device according to claim 5, wherein the subdivision unit is configured to move the position of an edge division point of the basic face to any edge division point of an adjacent basic face, and then output the subdivision mesh.

7. 1. A mesh decoding method, comprising: a step A of calculating the number of subdivisions of a basic mesh defined in three dimensions in units of basic faces or basic patches, and subdividing the basic mesh based on the calculated number of subdivisions to generate a subdivision mesh; a step B of decoding the displacement bitstream to generate a displacement; a step C of generating a decoded mesh using the subdivision mesh generated in the step A and the displacement amount decoded in the step B.

8. A program that causes a computer to function as a mesh decoding device, The mesh decoding device comprises: a subdivision unit configured to calculate a number of subdivisions of a basic mesh defined three-dimensionally in units of basic faces or basic patches, and to generate a subdivision mesh by subdividing the basic mesh based on the calculated number of subdivisions; a displacement decoder configured to decode the displacement bitstream to generate a displacement; a mesh decoding unit configured to generate a decoded mesh using the subdivision mesh generated by the subdivision unit and the displacement amount decoded by the displacement amount decoding unit.

9. 1. A mesh decoding method, comprising: generating a subdivision mesh by subdividing a base mesh defined in three dimensions; decoding the displacement bitstream to generate displacements; generating a composite mesh using the subdivision mesh and the displacement; and adjusting the positions of the edge division points of the basic faces.

10. A program that causes a computer to function as a mesh decoding device, The mesh decoding device comprises: a subdivision unit configured to subdivide a base mesh defined in three dimensions to generate a subdivision mesh; a displacement decoder configured to decode the displacement bitstream to generate a displacement; a mesh decoding unit configured to generate a decoded mesh using the subdivision mesh generated by the subdivision unit and the displacement amount decoded by the displacement amount decoding unit, The subdivision unit adjusts the positions of the edge division points of the basic faces.

Images