Mesh decoding device, mesh decoding method and program

Abstract

To enhance a balance between mesh coding efficiency and a degree of complexity.SOLUTION: A basic mesh decoding unit 202 of a mesh decoding device 200 according to the present invention comprises: an intra decoding unit 202B that decodes a coordinate of an apex of an I frame from a bit stream of a frame, and connection information; and an intra decoding unit 202E that adds a coordinate of an apex of a reference frame and a motion vector decoded from a bit stream of a P frame to thereby decode the coordinate of the apex of the P frame. By a control signal, the basic mesh decoding unit is configured to integrate a CABAC coding method of a residual difference of the motion vector in the intra decoding unit 202E and a CABAC coding method of a residual difference between an apex coordinate and a UV coordinate in the intra decoding unit 202B, or the intra decoding unit 202E is configured to make a context value always stationary regarding prescribed syntax, and thereby bypass an update of the context value to perform arithmetic decoding.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a mesh decoding device, a mesh decoding method, and a program. [Background technology]

[0002] Non-Patent Document 1 discloses a technique for encoding a mesh using Non-Patent Document 2 or 3. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Khaled Mammou, Jungsun Kim, Alexis M Tourapis, Dimitri Podborski, and Krasimir Kolarov, “[V-CG] Apple's Dynamic Mesh Coding CfP Response,” April 2022, ISO / IEC JTC 1 / SC 29 / WG 7. [Non-Patent Document 2] Google Draco, accessed May 26, 2022 [Online], https: / / google.github.io / draco [Non-Patent Document 3] Jean-Eudes Marvie, Olivier Mocquard, “[V-DMC][EE4.4-related] An efficient EdgeBreaker implementation,” April 2023, ISO / IEC JTC 1 / SC 29 / WG 7 m63344. Summary of the Invention [Problem to be solved by the invention]

[0004] However, the conventional technology has a problem that the balance between encoding efficiency and complexity is poor. Therefore, the present invention has been made in consideration of the above-mentioned problems, and has an object to provide a mesh decoding device, a mesh decoding method, and a program that can improve the balance between mesh encoding efficiency and complexity. [Means for solving the problem]

[0005] A first feature of the present invention is a mesh decoding device comprising a base mesh decoding unit configured to decode a base mesh bitstream and generate and output a base mesh, the base mesh decoding unit comprising an intra decoding unit configured to decode vertex coordinates and connection information of the I frame from the I frame bitstream, and an inter decoding unit configured to decode vertex coordinates of the P frame by adding vertex coordinates of a reference frame and a motion vector decoded from the P frame bitstream, and configured to integrate a CABAC encoding method of motion vector residuals in the inter decoding unit and a CABAC encoding method of vertex coordinates and UV coordinate residuals in the intra decoding unit by a control signal, or the inter decoding unit is configured to perform arithmetic decoding by always fixing context values ​​for a predetermined syntax, bypassing updating of the context values.

[0006] A second feature of the present invention is a mesh decoding method comprising the steps of: decoding a base mesh bitstream to generate and output a base mesh; the steps including: a step A of decoding vertex coordinates and connection information of an I frame from an I frame bitstream; and a step B of decoding vertex coordinates of a P frame by adding vertex coordinates of a reference frame and a motion vector decoded from a P frame bitstream; and a control signal is used to integrate a CABAC encoding method of a motion vector residual in the step A and a CABAC encoding method of a vertex coordinate and a UV coordinate residual in the step B; or, in the step B, for a predetermined syntax, a context value is always fixed, thereby bypassing updating of the context value and performing arithmetic decoding.

[0007] A third feature of the present invention is a program for causing a computer to function as a mesh decoding device, the mesh decoding device comprising: a basic mesh decoding unit configured to decode a basic mesh bitstream and generate and output a basic mesh; the basic mesh decoding unit comprising: an intra decoding unit configured to decode vertex coordinates and connection information of the I frame from the I frame bitstream; and an inter decoding unit configured to decode vertex coordinates of the P frame by adding vertex coordinates of a reference frame and motion vectors decoded from the P frame bitstream; and configured to integrate a CABAC encoding method of motion vector residuals in the inter decoding unit and a CABAC encoding method of vertex coordinates and UV coordinate residuals in the intra decoding unit by a control signal; or, the inter decoding unit is configured to always fix a context value for a predetermined syntax, thereby bypassing updating of the context value and performing arithmetic decoding. Effect of the Invention

[0008] According to the present invention, it is possible to provide a mesh decoding device, a mesh decoding method, and a program that can improve the balance between mesh encoding efficiency and complexity. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a mesh processing system 1 according to an embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of functional blocks of a mesh decoding device 200 according to an embodiment. [Figure 3A] FIG. 3A is a diagram showing an example of a base mesh and a subdivision mesh. [Figure 3B] FIG. 3B is a diagram showing an example of a base mesh and a subdivision mesh. [Figure 4] FIG. 4 is a diagram showing an example of functional blocks of the basic mesh decoding unit 202 of the mesh decoding device 200 according to an embodiment. [Diagram 5] FIG. 5 is a diagram showing an example of functional blocks of the intra-decoding unit 202B of the basic mesh decoding unit 202 of the mesh decoding device 200 according to an embodiment. [Figure 6] FIG. 6 is a diagram showing an example of the correspondence between the vertices of the basic mesh of a P frame and the vertices of the basic mesh of an I frame. [Figure 7] FIG. 7 is a diagram showing an example of functional blocks of an inter decoding unit 202E of the basic mesh decoding unit 202 of the mesh decoding device 200 according to an embodiment. [Figure 8] FIG. 8 is a diagram illustrating an example of a method for calculating the MVP of a vertex to be decoded by the motion vector prediction unit 202E3 of the inter decoding unit 202E of the basic mesh decoding unit 202 of the mesh decoding device 200 according to one embodiment. [Figure 9] FIG. 9 shows a flowchart showing an example of the operation of the motion vector prediction unit 202E3 of the inter decoding unit 202E of the basic mesh decoding unit 202 of the mesh decoding device 200 according to one embodiment. [Figure 10]FIG. 10 is a diagram for explaining an example of an MVR coding method in the current inter decoding unit 202E and an example of a coding method for residuals of vertex coordinates and UV coordinates in the intra decoding unit 202B. [Figure 11] FIG. 11 is a diagram for explaining an example of an MVR encoding method in the inter decoding unit 202E of one modified example, and an example of an encoding method of residuals of vertex coordinates and UV coordinates in the intra decoding unit 202B. [Figure 12] FIG. 12 is a diagram showing an example of functional blocks of the subdivision unit 203 of the mesh decoding device 200 according to an embodiment. [Figure 13] FIG. 13 is a diagram showing an example of functional blocks of the basic mesh subdivision unit 203A of the subdivision unit 203 of the mesh decoding device 200 according to an embodiment. [Figure 14] FIG. 14 is a diagram illustrating an example of a method of dividing a base surface by the basic surface dividing unit 203A5 of the basic mesh subdivision unit 203A of the subdivision unit 203 in the mesh decoding device 200 according to an embodiment. [Figure 15] FIG. 15 is a flowchart showing an example of the operation of the basic mesh subdivision unit 203A of the subdivision unit 203 of the mesh decoding device 200 according to an embodiment. [Figure 16] FIG. 16 is a diagram showing an example of functional blocks of the subdivision mesh adjustment unit 203B of the subdivision unit 203 of the mesh decoding device 200 according to an embodiment. [Figure 17] FIG. 17 is a diagram showing an example of a case in which an edge division point on a basic surface ABC is moved by the edge division point moving unit 701 of the subdivision mesh adjustment unit 203B of the subdivision unit 203 of the mesh decoding device 200 according to an embodiment. [Figure 18] FIG. 18 is a diagram showing an example of a case in which subdivision surface X in a base surface is re-performed by the subdivision surface division unit 702 of the subdivision mesh adjustment unit 203B of the subdivision unit 203 of the mesh decoding device 200 according to one embodiment. [Figure 19]FIG. 19 is a diagram showing an example of a case in which all subdivision surfaces are re-subdivided by the subdivision surface division unit 702 of the subdivision mesh adjustment unit 203B of the subdivision unit 203 of the mesh decoding device 200 according to one embodiment. [Figure 20] FIG. 20 is a diagram showing an example of functional blocks of the displacement amount decoding unit 206 of the mesh decoding device 200 according to one embodiment (when inter prediction is performed in the spatial domain). [Figure 21] FIG. 21 is a diagram showing an example of the configuration of a displacement amount bit stream. [Figure 22] FIG. 22 is a diagram showing an example of a syntax configuration of a DPS. [Diagram 23] FIG. 23 is a diagram illustrating an example of a syntax configuration of the DPH. [Figure 24] FIG. 24 is a diagram illustrating an example of a correspondence relationship in frequency between a reference frame and a current frame to be decoded when inter prediction is performed in the spatial domain. [Diagram 25] FIG. 25 is a diagram showing an example of functional blocks of the displacement amount decoding unit 206 of the mesh decoding device 200 according to one embodiment (when inter prediction is performed in the frequency domain). [Figure 26] FIG. 26 is a diagram illustrating an example of a correspondence relationship in frequency between a reference frame and a current frame to be decoded when inter prediction is performed in the frequency domain. [Figure 27] FIG. 27 is a flowchart showing an example of the operation of the displacement amount decoding unit 206 of the mesh decoding device 200 according to an embodiment. [Figure 28] FIG. 28 is a diagram illustrating an example of functional blocks of the displacement amount decoding unit 206 according to the first modification. [Figure 29] FIG. 29 is a diagram illustrating an example of functional blocks of the displacement amount decoding unit 206 according to the second modification. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments of the present invention will be described with reference to the drawings. Note that the components in the following embodiments can be appropriately replaced with existing components, and various variations including combinations with other existing components are possible. Therefore, the description of the following embodiments does not limit the content of the invention described in the claims.

[0011] First Embodiment Hereinafter, the mesh processing system according to this embodiment will be described with reference to FIGS.

[0012] 1 is a diagram showing an example of the configuration of a mesh processing system 1 according to this embodiment. As shown in FIG.

[0013] FIG. 2 is a diagram showing an example of functional blocks of the mesh decoding device 200 according to this embodiment.

[0014] As shown in FIG. 2, the mesh decoding device 200 includes a demultiplexing unit 201, a basic mesh decoding unit 202, a subdivision unit 203, a mesh decoding unit 204, a patch integration unit 205, a displacement amount decoding unit 206, and a video decoding unit 207.

[0015] Here, the basic mesh decoding unit 202, the subdivision unit 203, the mesh decoding unit 204 and the displacement amount decoding unit 206 are configured to perform processing on a patch basis into which the mesh is divided, and the results of these processes may then be configured to be integrated by the patch integration unit 205.

[0016] In the example of FIG. 3A, the mesh is divided into patch 1 consisting of basic faces 1 and 2, and patch 2 consisting of basic faces 3 and 4.

[0017] The demultiplexing unit 201 is configured to separate the multiplexed bit stream into a base mesh bit stream, a displacement amount bit stream, and a texture bit stream.

[0018] <Basic mesh decoding unit 202> The base mesh decoding unit 202 is configured to decode the base mesh bitstream and generate and output a base mesh.

[0019] Here, the base mesh is composed of a plurality of vertices in a three-dimensional space and edges connecting these vertices.

[0020] As shown in FIG. 3A, the basic mesh is formed by combining basic faces each represented by three vertices.

[0021] The base mesh decoder 202 may be configured to decode the base mesh bitstream, for example using the techniques described in Draco in Non-Patent Document 2 or in Non-Patent Document 3.

[0022] Furthermore, the basic mesh decoding unit 202 may be configured to generate "subdivision_method_id" (described later) as control information for controlling the type of subdivision method.

[0023] As shown in FIG. 4, the basic mesh decoding unit 202 includes a separation unit 202A, an intra decoding unit 202B, a mesh buffer unit 202C, a connection information decoding unit 202D, and an inter decoding unit 202E.

[0024] The separator 202A is configured to classify the base mesh bitstream into a bitstream of I frames and a bitstream of P frames.

[0025] (Intra decoding unit 202B) The intra decoder 202B is configured to decode the coordinates and connection information of the vertices of the I-frame from the bit stream of the I-frame, for example, using the technique Draco shown in Non-Patent Document 2 or the technique described in Non-Patent Document 3.

[0026] FIG. 5 is a diagram showing an example of functional blocks of the intra decoding unit 202B.

[0027] As shown in FIG. 5, the intra decoding unit 202B includes an arbitrary intra decoding unit 202B1 and an alignment unit 202B2.

[0028] The arbitrary intra decoder 202B1 is configured to decode the coordinates and connectivity information of the unordered vertices of the I-frame from the bitstream of the I-frame using any method including Draco shown in Non-Patent Document 2 or the technique described in Non-Patent Document 3.

[0029] The sorting unit 202B2 is configured to output the vertices by sorting the unordered vertices into a predetermined order.

[0030] As the predetermined order, for example, the Morton code order or the raster scan order may be used.

[0031] Furthermore, the alignment unit 202B2 may combine overlapping vertices, which are multiple vertices with the same coordinates in the decoded basic mesh, into a single vertex, and then rearrange the vertices in a predetermined order.

[0032] The mesh buffer unit 202C is configured to accumulate the coordinates and connection information of the vertices of the I frame decoded by the intra decoder 202B. Here, a specific buffer may be provided to store pairs of vertex indexes A(k) and B(k) of overlapping vertices in a predetermined order.

[0033] The connection information decoding unit 202D is configured to convert the connection information of the I frame or the reference frame extracted from the mesh buffer unit 202C into connection information of a P frame.

[0034] The inter decoding unit 202E is configured to decode the coordinates of the vertices of the P frame by adding the coordinates of the vertices of the reference frame retrieved from the mesh buffer unit 202C to the motion vectors decoded from the bit stream of the P frame.

[0035] Furthermore, the inter decoding unit 202E can adjust the index of the vertex of the P frame by the pair of vertex indexes A(k) and B(k) existing as overlapping vertices stored in the specific buffer.

[0036] Here, all or part of the above-mentioned indexes are decoded from the bitstream, and such a decoding method may be arithmetic coding.

[0037] In this embodiment, as shown in Fig. 6, there is a correspondence between the vertices of the basic mesh of the P frame and the vertices of the basic mesh of the reference frame (I frame or P frame). Here, the motion vector decoded by the inter decoding unit 202E is a difference vector between the coordinates of the vertices of the basic mesh of the P frame and the coordinates of the vertices of the basic mesh of the I frame.

[0038] (Inter-decoding unit 202E) FIG. 7 is a diagram illustrating an example of functional blocks of the inter decoding unit 202E.

[0039] As shown in FIG. 7, the inter decoding unit 202E includes a motion vector residual decoding unit 202E1, a motion vector buffer unit 202E2, a motion vector prediction unit 202E3, a motion vector calculation unit 202E4, and an adder 202E5.

[0040] The motion vector residual decoding unit 202E1 is configured to generate a motion vector residual (MVR) from a bit stream of a P frame.

[0041] Here, MVR is a motion vector residual indicating the difference between MV (Motion Vector) and MVP (Motion Vector Prediction). MV is a difference vector (motion vector) between the coordinates of the vertex of the corresponding I frame and the coordinates of the vertex of the corresponding P frame. MVP is a predicted value of the MV of the target vertex using MV (predicted value of the motion vector).

[0042] The motion vector buffer unit 202E2 is configured to sequentially store the MVs output by the motion vector calculation unit 202E4.

[0043] The motion vector prediction unit 202E3 is configured to obtain a decoded MV from the motion vector buffer unit 202E2 for a vertex connected to the vertex to be decoded, and output an MVP of the vertex to be decoded using all or part of the obtained decoded MV, as shown in Figure 8.

[0044] The motion vector calculation unit 202E4 is configured to add the MVR generated by the motion vector residual decoding unit 202E1 and the MVP output from the motion vector prediction unit 202E3, and output the MV of the vertex to be decoded.

[0045] The adder 202E5 is configured to add the coordinates of a vertex corresponding to the vertex to be decoded obtained from the decoded basic mesh of a corresponding reference frame (I frame or P frame) to the motion vector MV output from the motion vector calculation unit 202E3, and output the coordinates of the vertex to be decoded.

[0046] Hereinafter, each component of inter decoding unit 202E will be described in detail.

[0047] 9 shows a flowchart illustrating an example of the operation of the motion vector prediction unit 202E3. Hereinafter, the operation of the motion vector prediction unit 202E3 will be referred to as an "average prediction method."

[0048] As shown in FIG. 9, in step S1001, the motion vector prediction unit 202E3 sets MVP and N to 0.

[0049] In step S1002, the motion vector prediction unit 202E3 obtains a set of MVs of vertices around the vertex to be decoded from the motion vector buffer unit 202E2, identifies vertices for which subsequent processing has not been completed, and transitions to No, or transitions to Yes if subsequent processing has been completed for all vertices.

[0050] In step S1003, if the MV of the vertex to be processed has not been decoded, the motion vector prediction unit 202E3 transitions to No, and if the MV of the vertex to be processed has been decoded, the motion vector prediction unit 202E3 transitions to Yes.

[0051] In step S1004, the motion vector prediction unit 202E3 adds MV to MVP and adds 1 to N.

[0052] In step S1005, if N is greater than 0, the motion vector prediction unit 202E3 outputs the result of dividing MVP by N, and if N is 0, outputs 0, and the process ends.

[0053] That is, the motion vector prediction unit 202E3 is configured to output the MVP to be decoded by averaging the decoded motion vectors of vertices around the vertex to be decoded.

[0054] The motion vector prediction unit 202E3 may be configured to set the MVP to 0 when the set of decoded motion vectors is an empty set.

[0055] The motion vector calculation unit 202E4 may be configured to calculate the MV of the vertex to be decoded from the MVP output by the motion vector prediction unit 202E3 and the MVR generated by the motion vector residual decoding unit 202E1, using equation (1).

[0056] MV(k)=MVP(k)+MVR(k) … (1) where k is the index of the vertex. MV, MVR and MVP are vectors with x, y and z components.

[0057] According to this configuration, since only the MVR is encoded instead of the MV using the MVP, it is expected to have the effect of improving the encoding efficiency.

[0058] The adder 202E5 is configured to calculate the coordinates of a vertex by adding the MV of the vertex calculated by the motion vector calculation unit 202E4 to the coordinates of the vertex in the reference frame corresponding to the vertex, and to leave the connectivity information (Connectivity) as it is in the reference frame.

[0059] Specifically, the adder 202E5 calculates the coordinate v' of the kth vertex using the formula (2). i (k) may be calculated.

[0060] v' i (k)=v' j (k) + MV(k) … (2) where v' i (k) is the coordinate of the kth vertex to be decoded in the frame to be decoded, and v' j (k) is the coordinate of the k-th decoded vertex of the reference frame, and MV(k) is the k-th MV of the frame to be decoded, where k=1, 2, . . . , K.

[0061] Moreover, the connection information of the frame to be decoded is made the same as the connection information of the reference frame.

[0062] Note that since the motion vector prediction unit 202E3 calculates the MVP using a decoded MV, the order of decoding affects the MVP.

[0063] The order of the decoding is the order of the vertices of the base mesh in the reference frame. In general, if a decoding method is used that uses a certain repetitive pattern to increase the number of base faces one by one from the starting edge, the order of the vertices of the decoded base mesh is determined during the decoding process.

[0064] For example, the motion vector prediction unit 202E3 may use an Edgebreaker to determine the order of decoding vertices in the base mesh of the reference frame.

[0065] According to this configuration, since MVs from a reference frame are encoded instead of vertex coordinates, it is expected that the encoding efficiency can be improved.

[0066] (Modification of the inter-decoding unit 202E) A modification of the inter decoding unit 202E will be described below.

[0067] The motion vector residual decoding unit 202E1 may decode the motion vector residual by arithmetic decoding.

[0068] In arithmetic decoding, the original value is decoded based on which interval on a number line the input binary decimal falls within. In arithmetic decoding, a number line from 0 to 1 is defined, and the interval is divided according to the probability of occurrence of two values ​​(hereafter referred to as context value).

[0069] The motion vector residual decoding unit 202E1 may perform arithmetic decoding using a plurality of types of context values. The context values ​​may be defined separately for each syntax.

[0070] The motion vector residual decoding unit 202E1 may update the context value every time one bit is decoded.

[0071] The motion vector residual decoding unit 202E1 may use an update table that updates the context value slightly when a symbol with a high probability of occurrence of either 0 or 1 occurs, and updates the context value significantly when a symbol with a low probability of occurrence occurs.

[0072] Further, the motion vector residual decoding unit 202E1 may bypass the update of the context value by always fixing the context value to 0.5.

[0073] By bypassing the update of the context value in this way, high-speed decoding becomes possible.

[0074] The motion vector residual decoding unit 202E1 may determine an upper limit L (<K) on the number of motion vector residuals for which the update of the context value is bypassed, and perform the update of the context value only when the upper limit is exceeded.

[0075] Thereby, an effect of suppressing a decrease in coding efficiency due to bypassing the update of the context value can be expected.

[0076] The motion vector residual decoding unit 202E1 may set whether to bypass the update of the context value for each syntax.

[0077] Note that, as described above, the motion vector residual decoding unit 202E1 may perform decoding based on binary arithmetic decoding, or may perform decoding based on multi-value arithmetic decoding such as RangeCoder.

[0078] (Modification example of the basic mesh decoding unit 202) Hereinafter, a modification example of the basic mesh decoding unit 202 will be described with reference to FIGS. 10 and 11.

[0079] As shown in FIG. 10, currently, the same CABAC is used for the coding method of MVR in the inter decoding unit 202E and the coding method of the residuals of the vertex coordinates and UV coordinates in the intra decoding unit 202B, but the detailed flags and procedures are different between the two.

[0080] As shown in FIG. 11, the basic mesh decoding unit 202 according to this modification integrates the MVR coding method in the inter decoding unit 202E and the coding method of the residuals of the vertex coordinates and UV coordinates in the intra decoding unit 202B in response to a control signal.

[0081] The only difference between the MVR coding method in inter decoding unit 202E and the coding method for residuals of vertex coordinates and UV coordinates in intra decoding unit 202B is the number of flags indicating whether the absolute values ​​of coefficients are equal to or greater than a certain value.

[0082] Therefore, the basic mesh decoding unit 202 provides a control signal for calculating the number of flags indicating whether the absolute value of the coefficient is equal to or greater than a certain value for each frame or sequence, and decodes the control signal from the bit stream.

[0083] For example, the basic mesh decoding unit 202 decodes the control signal smh_ae_bias_minus1 from the frame header, and adds 1 to the value of the control signal smh_ae_bias_minus1 to obtain the number of flags indicating whether the absolute value of the above-mentioned coefficient is equal to or greater than a certain value.

[0084] In the case of an intra-frame, the mesh encoding device 100 sets the value of the control signal smh_ae_bias_minus1 to 2 and writes such control signal smh_ae_bias_minus1 into the bitstream at u(2).

[0085] On the other hand, in the case of an interframe, the mesh encoding device 100 sets the value of the control signal smh_ae_bias_minus1 to 0 or 1, and writes the control signal smh_ae_bias_minus1 into the bitstream at u(2).

[0086] As a result, the mesh decoding device 200 can decode the control signal smh_ae_bias_minus1 from the bitstream, whether it is an intraframe or an interframe.

[0087] Furthermore, the mesh decoding device 200 decodes the control signal smh_ae_bias_type from the frame header, and calculates the number of flags indicating whether the absolute value of the residual is equal to or greater than a certain value, from the mapping relationship with smh_ae_bias_type.

[0088] Table 1 shows an example of the mapping relationship between smh_ae_bias_type and ae_bias.

[0089] [Table 1] Furthermore, the motion vector residual decoding unit 202E1 may calculate the number of flags indicating whether the absolute value of the above-mentioned residual is equal to or greater than a certain value from other control signals or data, without using the above-mentioned dedicated control signal.

[0090] For example, the motion vector residual decoding unit 202E1 may calculate the number of flags indicating whether the absolute value of the residual is equal to or greater than a certain value, based on a control signal indicating whether the frame is an intraframe or an interframe.

[0091] According to this configuration, the coefficient-based CABAC coding method is integrated in intraframes and interframes, and therefore it is expected that the standard document will be refined and the coding efficiency will be improved.

[0092] <Subdivision section 203> The subdivision unit 203 is configured to generate and output added subdivision vertices and their connection information from the basic mesh decoded by the basic mesh decoding unit 202 using a subdivision method indicated by the control information.

[0093] Here, the basic mesh, the added subdivision vertices, and their connection information are collectively referred to as a "subdivision mesh."

[0094] The subdivision unit 202 is configured to identify the type of subdivision method from subdivision_method_id, which is control information generated by decoding the basic mesh bit stream.

[0095] The subdivision unit 202 will now be described with reference to FIGS. 3A and 3B.

[0096] 3A and 3B are diagrams for explaining an example of an operation for generating subdivision vertices from a basic mesh.

[0097] FIG. 3A is a diagram showing an example of a base mesh composed of five vertices.

[0098] Here, the subdivision may be performed, for example, by using a mid-edge division method in which the midpoints of the sides of each basic face are connected to each other, resulting in a basic face being divided into four faces.

[0099] Figure 3B shows an example of a subdivision mesh obtained by dividing a base mesh consisting of five vertices, in which eight subdivision vertices (white circles) are generated in addition to the original five vertices (black circles).

[0100] By decoding the displacement amount for each subdivision vertex thus generated by the displacement amount decoder 206, it is possible to expect improvement in coding performance.

[0101] Also, a different subdivision method may be applied to each patch. This allows the displacement decoded by the displacement decoder 206 to be adaptively changed for each patch, which is expected to improve coding performance. Information on the divided patches is received as patch_id, which is control information.

[0102] The subdivision section 203 will be described below with reference to Fig. 12. Fig. 12 is a diagram showing an example of functional blocks of the subdivision section 203.

[0103] As shown in FIG. 12, the subdivision section 203 has a basic mesh subdivision section 203A and a subdivision mesh adjustment section 203B.

[0104] (Basic mesh refinement section 203A) The basic mesh subdivision unit 203A is configured to calculate the number of divisions (number of subdivisions) for each basic surface and basic patch based on the input basic mesh and division information of the basic mesh, subdivide the basic mesh based on the number of divisions, and output the subdivision surface.

[0105] In other words, the basic mesh subdivision unit 203A may be configured to be able to change the above-mentioned number of divisions for each basic surface and basic patch.

[0106] Here, a base face is a face that constitutes a base mesh, and a base patch is a collection of several base faces.

[0107] In addition, the basic mesh subdivision unit 203A may be configured to predict the number of subdivisions of the base surface and calculate the number of subdivisions of the base surface by adding a predicted subdivision number residual to the predicted number of subdivisions of the base surface.

[0108] The basic mesh subdivision unit 203A may be configured to calculate the number of subdivisions of a basic surface based on the number of subdivisions of an adjacent basic surface of the basic surface.

[0109] The basic mesh subdivision unit 203A may be configured to calculate the number of subdivisions of a base surface based on the number of subdivisions of the base surface that has been stored immediately before.

[0110] The basic mesh subdivision unit 203A may be configured to generate vertices that divide the three sides that constitute the basic surface, and to connect the generated vertices to subdivide the basic surface.

[0111] As shown in FIG. 12, a basic mesh subdivision section 203A is followed by a subdivision mesh adjustment section 203B, which will be described later.

[0112] An example of the processing of the basic mesh subdivision section 203A will be described below with reference to FIGS.

[0113] FIG. 13 is a diagram showing an example of functional blocks of the basic mesh subdivision unit 203A, and FIG. 15 is a flowchart showing an example of the operation of the basic mesh subdivision unit 203A.

[0114] As shown in FIG. 13, the basic mesh subdivision unit 203A has a basic surface division number buffer unit 203A1, a basic surface division number reference unit 203A2, a basic surface division number prediction unit 203A3, an addition unit 203A4, and a basic surface division unit 203A5.

[0115] The basic surface division number buffer unit 203A1 stores division information of basic surfaces including the division number of the basic surface, and is configured to output the division information of the basic surface to the basic surface division number reference unit 203A2.

[0116] Here, the size of the basic surface division number buffer unit 203A1 may be set to 1, and the unit may be configured to output the division number of the basic surface that was stored immediately before to the basic surface division number reference unit 203A2.

[0117] That is, by setting the size of the basic surface division number buffer unit 203A1 to 1, it may be configured to refer to only the last decoded fine division number (the subdivision number decoded immediately before).

[0118] The basic surface division number reference unit 203A2 is configured to output a reference not possible to the basic surface division number prediction unit 203A3 when there is no adjacent basic surface to the basic surface to be decoded, or when there is an adjacent basic surface to the basic surface to be decoded but the number of divisions has not been determined.

[0119] On the other hand, when there is a base face adjacent to the base face to be decoded and the number of divisions is determined, the basic face division number reference unit 203A2 is configured to output the number of divisions to the basic face division number prediction unit 203A3.

[0120] The basic surface division number prediction unit 203A3 is configured to predict the division number (subdivision number) of a basic surface based on one or more input division numbers, and output the predicted division number (predicted division number) to the addition unit 203A4.

[0121] Here, the basic surface division number prediction unit 203A3 is configured to output 0 to the addition unit 203A4 when only reference impossible is input from the basic surface division number reference unit 203A2.

[0122] In addition, when one or more division numbers are input, the basic surface division number prediction unit 203A3 may be configured to generate a predicted division number using any of the statistical values ​​such as the average value, maximum value, minimum value, or mode of the input division numbers.

[0123] The basic face division number prediction unit 203A3 may be configured to generate the division number of the most adjacent face as the predicted division number when one or more division numbers are input.

[0124] The addition unit 203A4 is configured to output the division number obtained by adding the prediction division number residual decoded from the prediction residual bitstream and the prediction division number obtained from the basic surface division number prediction unit 203A3 to the basic surface division unit 203A5.

[0125] The basic surface division unit 203A5 is configured to subdivide the basic surface based on the division number input from the addition unit 203A4.

[0126] Fig. 14 shows an example of a case where a basic surface is divided into 9. The method of dividing a basic surface by the basic surface dividing section 203A5 will be described with reference to Fig. 14.

[0127] The basic surface dividing section 203A5 generates points A_1, . . . , A_(N-1) that divide the side AB constituting the basic surface into N equal parts (N=3).

[0128] Similarly, the basic surface division unit 203A5 also divides the sides BC and CA into N equal parts, generating points B_1, …, B_(N-1) and C_1, …, C_(N-1) respectively.

[0129] Hereinafter, the points on the sides AB, BC, and CA are referred to as "side division points".

[0130] For all i (i = 1, 2, …, N-1), the basic surface division unit 203A5 generates the sides A_i B_(N-i), B_i C_(N-i), and C_i A_(N-i), generating N 2 subdivision surfaces.

[0131] Next, with reference to FIG. 15, the processing procedure of the basic mesh subdivision unit 203A will be described.

[0132] In step S2201, the basic mesh subdivision unit 203A determines whether the subdivision process for the last basic surface has been completed. If the process is completed, it ends; otherwise, it proceeds to step S2202.

[0133] In step S2202, the basic mesh subdivision unit 203A determines whether Depth < mdu_max_depth.

[0134] Here, Depth is a variable representing the current depth, with an initial value of 0, and mdu_max_depth represents the maximum depth determined for each basic surface.

[0135] If the condition in step S2202 is satisfied, this processing procedure proceeds to step S2203; otherwise, this processing procedure returns to step S2201.

[0136] In step S2203, the basic mesh subdivision unit 203A determines whether mdu_subdivision_flag at the current depth is 1.

[0137] If the answer is Yes, the process returns to step S2201; if the answer is No, the process proceeds to step S2204.

[0138] In step S2204, the basic mesh subdivision unit 203A further subdivides all the subdivision surfaces within the basic surface.

[0139] Here, if the subdivision process has never been performed on the basic surface, the basic mesh subdivision unit 203A subdivides the basic surface.

[0140] The subdivision method is the same as that described in step S2204.

[0141] Specifically, if a basic surface has never been subdivided, the basic surface is subdivided as shown in FIG. 14. If the basic surface has been subdivided at least once, the subdivision surface is 2 In the example of FIG. 14, the face consisting of vertices A_2, B, and B_1 is further divided into N 2 Generate faces.

[0142] When the subdivision process is completed, the process proceeds to step S2205.

[0143] In step S2205, the basic mesh subdivision unit 203A adds 1 to Depth, and the processing procedure returns to step S2202.

[0144] (Fine division mesh adjustment unit 203B) Next, a specific example of the process performed by the finely divided mesh adjustment unit 203B will be described. An example of the process performed by the finely divided mesh adjustment unit 203B will be described below with reference to Figs.

[0145] FIG. 16 is a diagram showing an example of functional blocks of the finely divided mesh adjustment unit 203B.

[0146] As shown in FIG. 16, the subdivision mesh adjustment unit 203B has an edge division point moving unit 701 and a subdivision surface division unit 702.

[0147] (Edge division point moving unit 701) The edge division point moving unit 701 is configured to move an edge division point of a basic face to any of the edge division points of an adjacent basic face for an input initial subdivision face, and output a subdivision face.

[0148] Fig. 17 shows an example in which an edge division point on a basic face ABC is moved. For example, as shown in Fig. 17, the edge division point moving unit 701 may be configured to move an edge division point of the basic face ABC to an edge division point of the nearest adjacent basic face.

[0149] (Subdivision surface division part 702) The subdivision surface division unit 702 is configured to re-subdivide the input subdivision surface and output a decomposed subdivision surface.

[0150] FIG. 18 is a diagram showing an example of a case where subdivision is performed again on a subdivision surface X in a base surface.

[0151] As shown in FIG. 18, the subdivision surface division unit 702 may be configured to generate a new subdivision surface within a base surface by connecting vertices constituting the subdivision surface to edge division points of an adjacent base surface.

[0152] FIG. 19 is a diagram showing an example of a case where the above-mentioned subdivision processing is performed on all subdivision surfaces.

[0153] The mesh decoding unit 204 is configured to generate and output a decoded mesh using the subdivision mesh generated by the subdivision unit 203 and the displacement amount decoded by the displacement amount decoding unit 206 .

[0154] Specifically, the mesh decoding unit 204 is configured to generate a decoded mesh by adding, to each subdivision vertex, a corresponding displacement amount, where information about which subdivision vertex each displacement amount corresponds to is indicated by control information.

[0155] The patch integration unit 205 is configured to integrate the decoded mesh generated by the mesh decoding unit 206 by the amount of a plurality of patches and output the integrated mesh.

[0156] Here, the patch division method is defined by the mesh encoding device 100. For example, the patch division method may be configured to calculate a normal vector for each basic face, select a basic face having the most similar normal vector among adjacent basic faces, combine both basic faces into the same patch, and repeat such a procedure for the next basic face in sequence.

[0157] The video decoding unit 207 is configured to decode the texture by video coding and output the texture. For example, the video decoding unit 207 may use HEVC in Non-Patent Document 1.

[0158] <Displacement Amount Decoding Unit 206> The displacement amount decoding unit 206 is configured to decode the displacement amount bitstream to generate and output the displacement amount.

[0159] 3B is a diagram showing an example of a displacement amount for a certain subdivision vertex. In the example of FIG. 3B, since there are eight subdivision vertices, the displacement amount decoding unit 206 is configured to define eight displacement amounts expressed as scalars or vectors for each subdivision vertex.

[0160] The displacement amount decoding unit 206 will be described below with reference to Fig. 20. Fig. 20 is a diagram showing an example of functional blocks of the displacement amount decoding unit 206.

[0161] As shown in FIG. 20, the displacement amount decoding unit 206 includes a decoding unit 206A, an inverse quantization unit 206B, an inverse wavelet transform unit 206C, an adder 206D, an inter prediction unit 206E, and a frame buffer 206F.

[0162] The decoding unit 206A is configured to decode and output the level value and the control information by performing variable length decoding on the received displacement bit stream. Here, the level value obtained by the variable length decoding is output to the inverse quantization unit 206B, and the control information is output to the inter prediction unit 206E.

[0163] An example of the configuration of a displacement amount bit stream will be described below with reference to Fig. 21. Fig. 21 is a diagram showing an example of the configuration of a displacement amount bit stream.

[0164] As shown in FIG. 21, first, the displacement bit stream may include a DPS (Displacement Parameter Set) which is a set of control information related to decoding of the displacement.

[0165] Secondly, the displacement bitstream may include a Displacement Patch Header (DPH), which is a collection of control information corresponding to a patch.

[0166] Third, the displacement bitstream may contain, next to the DPH, the encoded displacements that make up the patch.

[0167] As described above, the displacement bit stream has a configuration in which each coded displacement corresponds to one DPH and one DPS.

[0168] 21 is merely an example. As long as the DPH and DPS correspond to each coded displacement, elements other than those described above may be added as components of the displacement bit stream.

[0169] For example, as shown in FIG. 21, the displacement bit stream may include a sequence parameter set (SPS).

[0170] FIG. 22 is a diagram showing an example of a syntax configuration of a DPS.

[0171] In FIG. 22, the Descriptor column indicates how each syntax element is coded.

[0172] In addition, in FIG. 22, ue(v) denotes an unsigned zeroth-order exponential Golomb code, and u(n) denotes an n-bit flag.

[0173] When multiple DPSs exist, the DPS includes at least DPS id information (dps_displacement_parameter_set_id) for identifying each DPS.

[0174] Furthermore, the DPS may include a flag (interprediction_enabled_flag) that controls whether or not inter prediction is performed.

[0175] For example, it may be defined that inter prediction is not performed when interprediction_enabled_flag is 0, and that inter prediction is performed when interprediction_enabled_flag is 1. It may be defined that inter prediction is not performed when interprediction_enabled_flag is not included.

[0176] The DPS may include a flag (dct_enabled_flag) that controls whether or not to perform an inverse DCT.

[0177] For example, it may be defined that the inverse DCT is not performed when dct_enabled_flag is 0, and that the inverse DCT is performed when dct_enabled_flag is 1. It may be defined that the inverse DCT is not performed when dct_enabled_flag is not included.

[0178] FIG. 23 is a diagram illustrating an example of a syntax configuration of the DPH.

[0179] As shown in FIG. 23, a DPH includes at least DPS id information for specifying a DPS corresponding to each DPH.

[0180] The inverse quantization unit 206B is configured to inversely quantize the level values ​​decoded by the decoding unit 206A to generate and output transform coefficients.

[0181] The inverse wavelet transform unit 206C is configured to perform an inverse wavelet transform on the transform coefficients generated by the inverse quantization unit 206B to generate and output prediction residuals.

[0182] (Inter prediction unit 206E) The inter prediction unit 206E is configured to perform inter prediction using the decoded displacement amount of the reference frame read from the frame buffer 206F, thereby generating and outputting a predicted displacement amount.

[0183] The inter prediction unit 206E is configured to perform such inter prediction only when interprediction_enabled_flag is set to one.

[0184] The inter prediction unit 206E may perform inter prediction in the spatial domain or may perform inter prediction in the frequency domain. The inter prediction may be bidirectional prediction using a past reference frame and a future reference frame in terms of time.

[0185] FIG. 24 is a diagram illustrating an example of a correspondence relationship between subdivision vertices between a reference frame and a current frame for decoding when inter prediction is performed in the spatial domain.

[0186] FIG. 25 is an example of a functional block diagram of the inter prediction unit 206E when performing inter prediction in the frequency domain.

[0187] When performing inter prediction in the frequency domain, the inter prediction unit 206E may determine predicted wavelet transform coefficients of a frequency in a current frame to be decoded by directly referring to decoded wavelet transform coefficients of a corresponding frequency in a reference frame.

[0188] The inter prediction unit 206E may perform stochastic inter prediction according to a normal distribution with estimated mean and variance, using the decoded displacement amounts or decoded wavelet transform coefficients of the subdivision vertices in a plurality of reference frames.

[0189] The inter prediction unit 206E may perform inter prediction based on a regression curve estimated using the decoded displacement amounts or decoded wavelet transform coefficients of the subdivision vertices in a plurality of reference frames, with time as an explanatory variable and the displacement amount as a response variable.

[0190] The inter prediction unit 206E may be configured to perform bidirectional inter prediction using a temporally past reference frame and a future reference frame.

[0191] In the mesh coding device 100, the order of the decoded wavelet transform coefficients may be rearranged in order to improve the coding efficiency for each frame.

[0192] The frequency correspondence between the reference frame and the frame to be decoded is indicated by control information.

[0193] FIG. 26 is a diagram illustrating an example of a correspondence relationship in frequency between a reference frame and a current frame to be decoded when inter prediction is performed in the frequency domain.

[0194] In addition, when the subdivision unit 203 divides the base mesh into a plurality of patches, the inter prediction unit 206E is also configured to perform inter prediction for each divided patch. This increases the time correlation between frames, and is expected to improve coding performance.

[0195] The adder 206D receives the prediction residual from the inverse wavelet transform unit 206C and the prediction displacement amount from the inter prediction unit 206E.

[0196] The adder 206D is configured to add the prediction residual and the prediction displacement amount to calculate and output the decoded displacement amount.

[0197] The decoded displacement calculated by the adder 206D is also output to the frame buffer 206F.

[0198] The frame buffer 206F is configured to receive and store the decoded displacement amount from the adder 206D.

[0199] Here, the frame buffer 206F outputs the decoded displacement amount at the corresponding vertex in the reference frame in response to control information (not shown).

[0200] FIG. 27 is a flowchart showing an example of the operation of the displacement amount decoding unit 206.

[0201] As shown in FIG. 27, in step S3501, the displacement decoding unit 206 determines whether or not this process has been completed for all patches.

[0202] If Yes, the operation ends, and if No, the operation proceeds to step S3502.

[0203] In step S3502, the displacement decoding unit 206 performs inverse DCT, and then performs inverse quantization and inverse wavelet transform on the patch to be decoded.

[0204] In step S3503, the displacement decoding unit 206 determines whether or not the interpretation_enabled flag is 1.

[0205] If yes, the operation proceeds to step S3504; if no, the operation returns to step S3501.

[0206] In step S3504, the displacement decoding unit 206 performs the above-mentioned inter prediction and addition.

[0207] <Variation 1> Hereinafter, with reference to FIG. 28, a first modification of the first embodiment will be described, focusing on differences from the first embodiment.

[0208] FIG. 28 is a diagram showing an example of functional blocks of the displacement amount decoding unit 206 according to the present first modification.

[0209] As shown in FIG. 28, the displacement decoding unit 206 according to the present first modification includes an inverse DCT unit 206G subsequent to the decoding unit 206A, that is, between the decoding unit 206A and the inverse quantization unit 206B.

[0210] That is, in the present first modification, the inverse quantization unit 206B is configured to generate a prediction residual by performing an inverse wavelet transform on the level values ​​output from the inverse DCT unit 202G.

[0211] <Variation 2> Hereinafter, with reference to FIG. 29, a second modification of the first embodiment will be described, focusing on the differences from the first embodiment.

[0212] As shown in FIG. 29, the displacement amount decoding unit 206 according to the present second modification includes a video decoding unit 2061, an image developing unit 2062, an inverse quantization unit 2063, and an inverse wavelet transformation unit 2064.

[0213] The video decoding unit 2061 is configured to decode the received displacement amount bit stream by video coding, and output a video.

[0214] For example, the video decoding unit 2061 may use HEVC in Non-Patent Document 1.

[0215] Furthermore, the video decoding unit 2061 may use a video encoding method in which a motion vector is always set to 0. For example, the video decoding unit 2061 may always set a motion vector of HEVC to 0 and always use inter prediction at the same position.

[0216] Furthermore, the video decoding unit 2061 may use a video encoding method in which conversion is always skipped. For example, the video decoding unit 2061 may always perform HEVC conversion in a conversion skip mode and use a video encoding method without conversion.

[0217] The image development unit 2062 is configured to develop the video decoded by the video decoding unit 2061 as a level value for each image (frame) and output it.

[0218] In such a development method, the image development unit 2062 can specify it by calculating backwards from the arrangement of the level values ​​in the image indicated by the control information.

[0219] The image development unit 2062 may arrange the level values ​​in the image, for example, in the order of raster operation, from high frequency components to low frequency components.

[0220] The inverse quantization unit 2063 is configured to inversely quantize the level values ​​generated by the image development unit 2062 to generate and output transformation coefficients.

[0221] The inverse wavelet transform unit 2064 is configured to perform an inverse wavelet transform on the transform coefficients generated by the inverse quantization unit 2063, thereby generating and outputting a decoded displacement amount.

[0222] The above-mentioned mesh encoding device 100 and mesh decoding device 200 may be realized as a program that causes a computer to execute each function (each step). [Industrial Applicability]

[0223] According to this embodiment, for example, it is possible to improve the overall service quality in video communication, which makes it possible to contribute to Goal 9 of the Sustainable Development Goals (SDGs) led by the United Nations, which is to "build resilient infrastructure, promote sustainable industrialization and foster innovation." [Explanation of symbols]

[0224] 1. Mesh processing system 100...Mesh coding device 200…Mesh decoding unit 201…Demultiplexer 202...Basic mesh decoding unit 202A…Separation part 202B... Intra decoding unit 202B1...Optional intra-decoding unit 202B2…Alignment section 202C…Mesh buffer section 202D…Connection information decoding unit 202E…Inter-decoding section 202E1...Motion vector decoding unit 202E2...Motion vector buffer unit 202E3...Motion vector prediction unit 202E4...Motion vector calculation unit 202E5…Adder 203…Subdivision part 203A…Basic mesh refinement section 203A1...Basic face division number buffer section 203A2…Basic surface division number reference section 203A3...Basic surface division number prediction unit 203A4…Addition section 203A5…Basic surface division part 203B…Subdivision mesh adjustment section 701...Edge division point moving part 702…Subdivision surface division part 204…Mesh decoding unit 205…Patch Integration Department 206...Displacement amount decoding unit 206A…Decoding section 206B, 2063...Dequantization section 206C, 2064...Inverse wavelet transform section 206D…Adder 206E…Inter prediction section 206F…Frame buffer 206G…Reverse DCT section 2062…Image development section 207, 2061...Video decoding section

Claims

1. A mesh decoding device, comprising: a base mesh decoder configured to decode the base mesh bitstream and generate and output a base mesh; The basic mesh decoding unit an intra decoder configured to decode, from a bitstream of an I-frame, coordinates and connectivity information of vertices of the I-frame; an inter-decoding unit configured to decode coordinates of vertices of a P frame by adding coordinates of the vertices of a reference frame and motion vectors decoded from a bitstream of the P frame; The control signal is configured to integrate the CABAC coding method of the motion vector residual in the inter decoding unit and the CABAC coding method of the vertex coordinates and UV coordinate residual in the intra decoding unit; or The mesh decoding device, wherein the inter-decoding unit is configured to perform arithmetic decoding by always fixing a context value for a predetermined syntax, thereby bypassing updating of the context value.

2. The control signal is configured to calculate a number of flags indicating whether the absolute value of the residual is equal to or greater than a certain value in the CABAC encoding method for each frame or each sequence. The mesh decoding device according to claim 1.

3. The control signal is smh_ae_bias_minus1, The mesh decoding device according to claim 2, characterized in that the basic mesh decoding unit is configured to obtain the number of flags by decoding the control signal from a frame header and adding 1 to the value of the control signal.

4. The control signal is smh_ae_bias_type, The mesh decoding device according to claim 2, characterized in that the basic mesh decoding unit is configured to decode the control signal from a frame header and calculate the number of flags from a mapping relationship with the control signal.

5. 2. The mesh decoding device according to claim 1, characterized in that the control signal is configured to be calculated from other control signals or data.

6. A mesh decoding method, comprising: Decoding the base mesh bitstream to generate and output a base mesh; The process comprises: A step A of decoding, from a bitstream of an I-frame, coordinates and connection information of vertices of said I-frame; B. decoding coordinates of vertices of the P frame by adding coordinates of the vertices of the reference frame and motion vectors decoded from a bitstream of the P frame; or configured to combine the CABAC coding of the motion vector residuals in step A with the CABAC coding of the vertex and UV coordinate residuals in step B by a control signal; or A mesh decoding method, characterized in that in the step B, for a predetermined syntax, a context value is always fixed, thereby bypassing update of the context value and performing arithmetic decoding.

7. A program for causing a computer to function as a mesh decoding device, comprising: The mesh decoding device comprises: a base mesh decoder configured to decode the base mesh bitstream and generate and output a base mesh; The basic mesh decoding unit an intra decoder configured to decode, from a bitstream of an I-frame, coordinates and connectivity information of vertices of the I-frame; an inter-decoding unit configured to decode coordinates of vertices of a P frame by adding coordinates of the vertices of a reference frame and motion vectors decoded from a bitstream of the P frame; The control signal is configured to integrate the CABAC coding method of the motion vector residual in the inter decoding unit and the CABAC coding method of the vertex coordinates and UV coordinate residual in the intra decoding unit; or the inter-decoding unit is configured to always fix a context value for a predetermined syntax, thereby bypassing updating of the context value and performing arithmetic decoding.

Images