Encoder, decoder, encoding method, decoding method, and program

Abstract

To improve the efficiency of encoding processing.SOLUTION: An encoder, when encoding an image, classifies a plurality of vertices included in a mesh into groups, calculates a motion vector indicating the motion between a reference point that is a vertex of a reference frame and a vertex of a frame to be encoded, and encodes the motion vector. In the encoding of the motion vector, when skip group information indicating whether the vertices belonging to the same group are encoded by a representative motion vector indicates that the vertices are encoded by the representative motion vector, the encoder encodes skip vector information indicating whether the representative vector of the same group is a 0 vector, and when the skip vector information indicates that the representative vector is not the 0 vector and at least one of the group to be encoded and a group encoded immediately before the group satisfies a predetermined condition, encodes skip prediction information indicating whether the representative motion vector is encoded using a prediction vector.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an encoding device, a decoding device, an encoding method, a decoding method, and a program. [Background technology]

[0002] For example, there is an ongoing MPEG project called V-DMC (Video-based Dynamic Mesh Compression) or ISO 23090-29:2023(E). V-DMC deals with 3D (three-dimensional) meshes in which the positional information of each vertex and the connectivity between vertices can change for each frame of a live-action image.

[0003] In V-DMC, a base mesh, which is a simplified version of the input mesh, is compressed using a static method that uses only information within the frame, and then the base mesh of the subsequent frame is subjected to inter-frame prediction using an already coded frame as a reference frame. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] US Patent Application Publication No. 2023 / 0290008 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the currently studied base mesh coding process has some redundancies, and there is a need to improve the efficiency of the coding process.

[0006] Therefore, the present invention reduces the redundancy in the encoding process of the base mesh, thereby improving the efficiency of the encoding process and the like. [Means for solving the problem]

[0007] In order to solve this problem, for example, the encoding device of the present invention has the following configuration: 1. An encoding device that encodes an image including a mesh having vertices whose positions change dynamically, on a frame-by-frame basis, a grouping means for grouping a plurality of vertices included in the mesh; a motion vector calculation means for calculating a motion vector indicating a motion between a reference point, which is a vertex of a reference frame, and a vertex of a frame to be coded; a motion encoding means for encoding the motion vector; and The motion encoding means encoding skip group information indicating whether vertices belonging to the same group are encoded by the same representative motion vector; If the skip group information indicates that the representative motion vector is coded, coding skip vector information indicating whether the representative motion vector of the same group is a zero vector; If the skip vector information indicates that the representative motion vector is not a 0 vector and at least one of the group to be coded and the group coded immediately before the group satisfies a predetermined condition, skip prediction information indicating whether the representative motion vector has been coded using a predictive vector is coded. [Effects of the Invention]

[0008] According to the present invention, it is possible to reduce redundant parts in the encoding process of the base mesh and improve the efficiency of the encoding process and the like. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing the functional configuration of a 3D mesh encoding device according to a first embodiment. [Figure 2] FIG. 3 is a block diagram showing the functional configuration of a base mesh encoding unit in the first embodiment. [Figure 3]FIG. 2 is a flowchart showing a 3D mesh encoding process in the 3D mesh encoding device according to the first embodiment. [Figure 4] FIG. 3 is a flowchart showing a motion encoding process in the motion encoding unit according to the first embodiment. [Figure 5] FIG. 10 is a block diagram showing the functional configuration of a 3D mesh decoding device according to a second embodiment. [Figure 6] FIG. 10 is a block diagram showing the functional configuration of a base mesh decoding unit in the second embodiment. [Figure 7] FIG. 10 is a flowchart showing the decoding process in the 3D mesh decoding device according to the second embodiment. [Figure 8] FIG. 10 is a flowchart showing a motion decoding process in a motion decoding unit according to the second embodiment. [Figure 9] FIG. 1 is a block diagram showing an example of the hardware configuration of a computer applicable to an encoding device and a decoding device. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0011] (First embodiment) An example of an embodiment of the present invention will be described below with reference to the drawings. Fig. 1 is a block diagram showing the functional configuration of a 3D mesh coding device (hereinafter also simply referred to as a coding device) according to this embodiment. Each functional unit of the coding device shown in Fig. 1 is realized by either an LSI (Large Scale Integration) or an ASIC (Application Specific Integrated Circuit) configured by hardware. The coding device of this embodiment generates a bitstream that complies with V-DMC, but is not limited to this and may generate bitstreams of other formats.

[0012] The encoding device encodes an image including a mesh having vertices whose positions change dynamically, on a frame-by-frame basis. As shown in Fig. 1, the encoding device of this embodiment includes a preprocessing unit 101, a base mesh encoding unit 102, a base mesh decoding unit 103, a displacement vector encoding unit 104, a texture encoding unit 105, an atlas encoding unit 106, and an integration unit 107.

[0013] The preprocessing unit 101 receives a 3D input mesh, converts the input mesh into a base mesh and a displacement vector, and outputs the resulting base mesh. Specifically, the preprocessing unit 101 thins out the vertices of the input mesh, parameterizes it, and generates a base mesh, which it then outputs to the base mesh encoding unit 102. The preprocessing unit 101 repeatedly divides each triangle of the base mesh into four sub-triangles using the midpoints of the edges between the vertices of the triangle. The preprocessing unit 101 searches for the closest position on the surface of the original input mesh to the vertex of each divided sub-triangle, and determines the difference between the vertex and the closest position as the vertex's displacement vector. The preprocessing unit 101 outputs the determined displacement vector to the displacement vector encoding unit 104.

[0014] The base mesh encoding unit 102 encodes the input base mesh using intraframe prediction or interframe prediction using a frame of the encoded base mesh as a reference frame to generate a base mesh bitstream. The base mesh encoding unit 102 outputs the generated base mesh bitstream to the base mesh decoding unit 103 and the integration unit 107.

[0015] The base mesh decoding unit 103 decodes the input base mesh stream and the displacement vector bit stream described later, generates a reconstructed base mesh, and outputs it to the base mesh encoding unit 102 and the displacement vector encoding unit 104.

[0016] The displacement vector encoding unit 104 updates the displacement vector based on the input reconstructed base mesh. The displacement vector encoding unit 104 performs a wavelet transform on the updated displacement vector and further packs the wavelet transform coefficients into a two-dimensional image. After packing, the displacement vector encoding unit 104 performs encoding using a two-dimensional video encoding method such as HEVC, generates a displacement vector bitstream, and outputs it to the base mesh decoding unit 103 and the integration unit 107.

[0017] The texture encoding unit 105 generates a texture bitstream by encoding the texture corresponding to the surface of the input mesh using a two-dimensional video encoding method such as HEVC, and outputs the generated texture bitstream to the integration unit 107.

[0018] The atlas encoding unit 106 encodes the atlas data including the metadata necessary for decoding the meshes to generate an atlas bitstream, and outputs the atlas bitstream to the integration unit 107.

[0019] The integration unit 107 receives a base mesh bitstream from the base mesh encoding unit 102, a displacement vector bitstream from the displacement vector encoding unit 104, a texture bitstream from the texture encoding unit 105, and an atlas bitstream from the atlas encoding unit 106. The integration unit 107 multiplexes and integrates these bitstreams to generate and output a 3D mesh bitstream.

[0020] 2 is a block diagram showing the functional configuration of the base mesh encoding unit 102. The base mesh encoding unit 102 will be described in detail with reference to FIG.

[0021] As shown in FIG. 2, the base mesh encoding unit 102 includes a static mesh encoding unit 201 , a grouping unit 202 , a motion vector calculation unit 203 , a motion encoding unit 204 , and a base mesh integration unit 205 .

[0022] The static mesh encoding unit 201 receives a base mesh and encodes it using only information within the frame, without using information from other frames, to generate a static mesh bitstream, which is then output to the base mesh integration unit 205 .

[0023] The grouping unit 202 receives the base mesh and the reconstructed base mesh, divides the vertices of the base mesh into multiple groups, and outputs the groups to the motion vector calculation unit 203. The grouping unit 202 divides the vertices of the base mesh into groups each containing N vertices.

[0024] The motion vector calculation unit 203 receives the vertices of the grouped base meshes and the reconstructed base mesh, and derives motion vectors indicating the movement of corresponding vertices of the base mesh (current frame) based on the vertices of the reference frame (an example of a reference point) using the reconstructed base mesh as a reference frame. The motion vector calculation unit 203 selects either skip mode, which applies a uniform motion vector to the vertices grouped by the grouping unit 202, or inter mode, which encodes a separate motion vector for each vertex, based on the degree of image quality degradation and the amount of data increase or decrease. If skip mode is selected, the motion vector calculation unit 203 derives only one representative motion vector, which is a motion vector shared by the vertices of the group. On the other hand, if skip mode is not selected, i.e., if inter mode is selected, the motion vector calculation unit 203 derives a motion vector for each vertex in the group. The motion vector calculation unit 203 outputs the motion vector and information related to the selected mode (hereinafter also referred to as mode information) to the motion coding unit 204.

[0025] The motion coding unit 204 performs arithmetic coding and entropy coding using Golomb coding or the like on the derived motion vectors and flags indicating mode information (also called skip group information), etc., to generate a motion bitstream and output it to the base mesh integration unit 205.

[0026] The base mesh synthesis unit 205 receives the static base mesh bitstream and the motion bitstream, and generates a base mesh bitstream. The base mesh synthesis unit 205 also synthesizes the header information of the base mesh into the base mesh bitstream.

[0027] The 3D mesh encoding operation in the 3D mesh encoding device of the first embodiment will be described below.

[0028] In FIG. 1, uncompressed 3D meshes are input to the preprocessing unit 101 in frame units with different time intervals. That is, the encoding device of this embodiment receives three-dimensional mesh data (a three-dimensional model composed of meshes) whose position may change over time. The mesh is a polygon containing multiple vertices, and each vertex contains three-dimensional (XYZ) position information, connection information between the vertices, texture coordinates, and texture connection information. In this embodiment, the input 3D meshes are all composed of triangles, but this is not limiting, and meshes containing connections of polygons other than triangles may also be input. Furthermore, in this embodiment, the 3D meshes are input in frame units, but they may also be input in sub-meshes, for example, by dividing a mesh for one frame into multiple parts.

[0029] The preprocessing unit 101 thins out the vertices of the input mesh to reduce the number of vertices. Next, the preprocessing unit 101 parameterizes the thinned mesh and generates a base mesh by associating each vertex of the mesh with texture coordinates. During this process, the preprocessing unit 101 performs processing so that the vertex connections, vertex texture coordinates, and texture connections remain unchanged from the previous frame. This allows only the vertex positions to change between frames, enabling efficient inter-frame prediction in the subsequent base mesh encoding unit 102. The preprocessing unit 101 then repeatedly divides each triangle of the base mesh into four sub-triangles using the midpoints of the edges between the vertices. In this embodiment, each triangle of the base mesh is divided twice into 16 sub-triangles, but this is not limiting; other division methods may be used, or the triangles may be further divided. Furthermore, triangle division may not be performed at all. The preprocessing unit 101 searches for the closest position to each vertex of the divided triangle on the surface of the original input mesh and outputs the difference between each vertex and the closest position as a displacement vector.

[0030] Next, the operation of the base mesh encoding unit 102 will be described using the block diagram of Figure 2. The static mesh encoding unit 201 receives the base mesh frame output from the pre-processing unit 101, encodes it using only the information in the frame, and generates a static base mesh bitstream. For example, the static mesh encoding unit 201 encodes the base mesh using the Edgebreaker algorithm and generates a static base mesh bitstream. In this embodiment, the static mesh encoding unit 201 uses Edgebreaker as the encoding algorithm, but this is not limited to this and other algorithms may be used.

[0031] The grouping unit 202 receives a base mesh frame from the preprocessing unit 101 and a reconstructed base mesh from the base mesh decoding unit 103, and generates groups of 16 vertices each. The grouping unit 202 groups the vertices in groups of 16 (indexes 0 to 15 (0th group), 16 to 31 (1st group), 32 to 47 (2nd group), etc.) in the order of the indices of the vertices that make up the reconstructed base mesh. In this embodiment, the order of the indices of the vertices of the reconstructed base mesh follows the order searched by Edgebreaker, but this is not limiting and other methods of grouping may also be used. For example, the grouping unit 202 may rearrange the order of the vertices to improve coding efficiency. Adjacent groups are often located close to each other in three-dimensional space and often have similar motion vectors. Therefore, compression efficiency can be improved by having the grouping unit 202 perform predictions from nearby groups.

[0032] Next, the motion vector calculation unit 203 searches for three-dimensional (XYZ) motion between the vertices of the base mesh and the corresponding vertices of the reconstructed base mesh, and calculates the resulting motion vector. The motion vector calculation unit 203 selects either a skip mode, in which the motion vectors of all vertices belonging to the group grouped by the grouping unit 202 are represented and coded using a single representative motion vector for the group, or an inter mode, in which the motion vectors of all vertices belonging to the group are coded separately. Lossless coding is possible when the motion vectors of each vertex are coded separately. On the other hand, coding only the representative motion vector reduces the amount of data for the motion vector, although it may result in some degradation in image quality. Therefore, the motion vector calculation unit 203 estimates the amount of data for the motion vector and the degree of image quality degradation, selects one of the modes, and outputs mode information indicating the selected mode along with the motion vector.

[0033] Next, the motion coding unit 204 receives the motion vector and mode information from the motion vector calculation unit 203, and arithmetically codes the motion vector and flags indicating the mode information to generate and output a motion bitstream.

[0034] The motion coding unit 204 codes sismu_skip_group_flag[g], which is a flag (skip group information) indicating whether the gth group of frames is to be coded in skip mode. Here, g is counted up in the order of processing, with the first group being 0. If the value of sismu_skip_group_flag[g] is 0 (FALSE), it indicates that the gth group is to be coded in inter mode, and if the value is 1 (TRUE), it indicates that the gth group is to be coded in skip mode.

[0035] If the gth group is coded in skip mode, the values ​​of the motion vectors of all vertices in the gth group will be the same as the group's representative motion vector, decodedSubmeshSkipVector[g]. decodedSubmeshSkipVector[g] contains three-dimensional motion information (X,Y,Z). If the coordinates of a vertex in the reference frame are (1,1,1) and decodedSubmeshSkipVector[g] is (0,1,2), then the coordinates of the corresponding vertex in the frame to be coded will be (1,2,3).

[0036] Next, the motion encoding unit 204 encodes sismu_skip_vector_flag[g], which is a flag (skip vector information) indicating whether the representative motion vector of the gth group is to be a zero vector, only if sismu_skip_group_flag[g] is 1. A value of 0 for sismu_skip_vector_flag[g] indicates that the representative motion vector of the gth group is to be a zero vector (the value of decodedSubmeshSkipVector[g] is (0,0,0)). This is equivalent to using the position information of the vertices of the corresponding group (gth group) in the reference frame as is. A value of 1 for sismu_skip_vector_flag[g] indicates that a non-zero motion vector is to be the representative motion vector of the gth group.

[0037] Next, the motion coding unit 204 uses the prediction vector submeshPredictionVector[g] to code sismu_skip_prediction_flag[g], which is a flag (skip prediction information) indicating whether to derive a representative motion vector for the gth group. The motion coding unit 204 codes sismu_skip_prediction_flag[g] only if all of the following three conditions are satisfied. Note that conditions 2 and 3 are examples of predetermined conditions. Condition 1: The value of sismu_skip_vector_flag[g] is 1 Condition 2: The group to be coded is not the first group in the frame (g==0). Condition 3: The immediately preceding coded group (the g-1th group) is coded in skip mode, and the representative motion vector is not coded as a zero vector (this is equivalent to the value of sismu_skip_group_flag[g-1] being 1 and the value of sismu_skip_vector_flag[g-1] being 1). Note that "the representative motion vector is not coded as a zero vector" means "the representative motion vector is coded as a non-zero vector."

[0038] The motion encoding unit 204 encodes the value obtained by subtracting submeshPredictionVector from the value of decodedSubmeshSkipVector[g] as a representative motion vector syntax group when sismu_skip_prediction_flag[g] is 1. When the value of sismu_skip_prediction_flag[g] is 0, the motion encoding unit 204 encodes the value of decodedSubmeshSkipVector[g] as a representative motion vector syntax group without modification.

[0039] The value of submeshPredictionVector[g] depends on the mode of the last coded group (the g-1th group). If the last coded group is not coded using skip mode (i.e., the motion vectors of each vertex are coded individually), the motion coding unit 204 uses the average value of the motion vectors of the vertices in the group as submeshPredictionVector[g]. If the last coded group is coded using skip mode, the motion coding unit 204 uses the value of decodedSubmeshSkipVector[g-1], which is the representative motion vector of the last group, as submeshPredictionVector[g].

[0040] Next, the motion encoding unit 204 encodes the following representative motion vector syntax group only when the value of sismu_skip_vector_flag[g] is 1. Here, when k is 0, it indicates a vector of the X component, and when it is 1 or 2, it indicates a vector of the Y and Z components, respectively.

[0041] sismu_mv_skip_abs_gt0[g][k]: Flag indicating whether the absolute value of the vector of the kth component of the gth group is greater than 0. If the flag value is 0, it indicates that the absolute value is 0, and if the flag value is 1, it indicates that the absolute value is 1 or greater.

[0042] sismu_mv_skip_sign[g][k]: A flag indicating the positive or negative sign of the k-component vector, which is encoded only if the absolute value of the k-component vector is greater than 0. If the flag value is 0, it indicates that the value is greater than or equal to 0, and if the flag value is 1, it indicates that the value is negative.

[0043] sismu_mv_skip_abs_gt1[g][k]: A flag that indicates whether the absolute value of the k-component vector is greater than 1 or not, and is coded only if the absolute value of the k-component vector is greater than 0. If the flag value is 0, it indicates that the absolute value is 1, and if the flag value is 1, it indicates that the absolute value is 2 or greater.

[0044] sismu_mv_skip_abs_rem[g][k]: Encoded only if the absolute value of the k-component vector is greater than 1, and indicates the absolute value of the k-component vector -2.

[0045] In the technology currently being discussed in MPEG, only the above-mentioned condition 1 is used as a condition for encoding sismu_skip_prediction_flag[g]. However, if condition 2 is not satisfied (g == 0), there is no immediately preceding group, and therefore the value of the prediction vector, submeshPredictionVector[g], cannot be correctly derived. Furthermore, if condition 3 is not satisfied, the prediction vector, submeshPredictionVector[g], becomes a 0 vector, and the processing remains the same regardless of the value of sismu_skip_prediction_flag[g]. Therefore, in the 3D mesh encoding device according to this embodiment, the motion encoding unit 204 does not encode sismu_skip_prediction_flag[g] even when conditions 2 and 3 are not satisfied, thereby improving encoding efficiency compared to conventional techniques. In other words, if the above conditions are not satisfied, this embodiment can reduce the amount of unnecessary data (improve the compression rate) and avoid the need for the encoding process. In this embodiment, the motion encoding unit 204 encodes sismu_skip_prediction_flag[g] only when all three conditions are satisfied, but this is not limited to this. For example, the motion encoding unit 204 may encode when only two or only one of the three conditions are satisfied. Alternatively, the motion encoding unit 204 may first calculate submeshPredictionVector[g] and encode sismu_skip_prediction_flag[g] only when the result is not a zero vector. Furthermore, instead of or in addition to condition 2 or 3, the motion encoding unit 204 may encode sismu_skip_prediction_flag[g] when the average of the motion vectors of the group encoded immediately before is not a zero vector, assuming that the condition is satisfied.

[0046] Next, the base mesh integration unit 205 receives the static base mesh bitstream from the static mesh encoding unit 201 and the motion bitstream from the motion encoding unit 204. The base mesh integration unit 205 integrates the static base mesh bitstream and the motion bitstream to generate a base mesh bitstream. The base mesh integration unit 205 may also integrate base mesh header information into the base mesh bitstream. The base mesh header information may include, for example, a parameter such as bmsps_inter_mesh_motion_group_size_minus1, which is the number of vertices belonging to one group minus 1 used in the grouping unit 202. In this embodiment, since the number of vertices belonging to a group is 16, a value of 15 is encoded using Golomb coding as bmsps_inter_mesh_motion_group_size_minus1. Next, the base mesh integration unit 205 outputs the generated base mesh bitstream.

[0047] Returning to Figure 1, the base mesh decoding unit 103 receives the base mesh bit stream output from the base mesh encoding unit 102, receives the displacement vector bit stream from the displacement vector encoding unit 104, decodes and reconstructs the base mesh to generate a reconstructed base mesh.

[0048] Next, the displacement vector encoding unit 104 receives the displacement vector from the preprocessing unit 101 and the reconstructed base mesh from the base mesh decoding unit 103, and updates the displacement vector. The displacement vector encoding unit 104 first divides the reconstructed base mesh in the same way as the preprocessing unit 101. The displacement vector encoding unit 104 repeatedly divides each triangle of the reconstructed base mesh into four sub-triangles using the midpoints of the edges between the vertices. In this embodiment, each triangle of the reconstructed base mesh is divided twice into 16 sub-triangles, but this is not limiting and the division may be performed using the same method as that used in the preprocessing. The division method is encoded as header information by the base mesh integration unit 205.

[0049] The displacement vector encoding unit 104 searches for the position closest to each vertex of the divided triangle on the surface of the original input mesh and generates a displacement vector based on the difference between the vertex and the closest position. The reconstructed base mesh is lossy encoded, and its position information may have changed from that of the original input base mesh. Therefore, the displacement vector encoding unit 104 adjusts the displacement vector based on the reconstructed base mesh. The displacement vector encoding unit 104 performs a wavelet transform on the updated displacement vector and further packs the wavelet transform coefficients into a two-dimensional image. After packing, the displacement vector encoding unit 104 performs video encoding using the HEVC standard, generates a displacement vector bitstream, and outputs it to the integrating unit 107. In this embodiment, the displacement vector is encoded using the HEVC standard, but this is not limiting. The displacement vector encoding unit 104 may also use standards such as H.264, VVC, and AV1.

[0050] Next, the texture encoding unit 105 encodes the texture corresponding to the surface of the input mesh using a video encoding method such as HEVC, generates a texture bitstream, and outputs it to the integration unit 107. In this embodiment, the texture is encoded using the HEVC method, but this is not limiting, and the texture encoding unit 105 may also use methods such as H.264, VVC, and AV1.

[0051] Next, the atlas encoding unit 106 generates an atlas bitstream including metadata necessary for decoding the meshes, and outputs it to the integration unit 107.

[0052] Next, the integration unit 107 receives various bitstreams from the base mesh encoding unit 102, the displacement vector encoding unit 104, the texture encoding unit 105, and the atlas encoding unit 106. The integration unit 107 multiplexes and integrates these bitstreams to generate a 3D mesh bitstream.

[0053] 3 is a diagram showing a flowchart of the encoding process in the 3D mesh encoding device according to the first embodiment. The flow of the process by the 3D mesh encoding device according to the first embodiment will be described using the flowchart in FIG.

[0054] First, in S301, the preprocessing unit 101 performs preprocessing such as thinning, parameterization, and division on the frames of the input mesh as described above to generate a base mesh. The preprocessing unit 101 outputs the generated base mesh to the static mesh encoding unit 201 and grouping unit 202 of the base mesh encoding unit 102. The preprocessing unit 101 also determines displacement vectors from the vertices of sub-triangles generated by dividing each triangle of the base mesh of the input mesh as described above and the closest positions on the surface of the original input mesh, and outputs the displacement vectors to the displacement vector encoding unit 104.

[0055] Next, in S302, the base mesh encoding unit 102 selects whether or not to encode the frame of the base mesh to be processed using inter-frame prediction. If the base mesh encoding unit 102 selects to encode using inter-frame prediction (Yes in S302), the process proceeds to S304. On the other hand, if the base mesh encoding unit 102 selects not to encode using inter-frame prediction (No in S302), the process proceeds to S303. In a series of processes, the base mesh encoding unit 102 cannot perform inter-frame prediction on the first frame to be processed, so the process always proceeds to S303.

[0056] Next, in S303, the static mesh encoding unit 201 encodes the base mesh frame received from the pre-processing unit 101 using only the information within the frame to generate a static mesh bit stream and output it to the base mesh integration unit 205.

[0057] Next, in S304, the grouping unit 202 receives the base mesh from the preprocessing unit 101 and the reconstructed base mesh from the base mesh decoding unit 103, and performs grouping of the vertices as described above.

[0058] Next, in S305, the motion vector calculation unit 203 receives the vertices of the grouped base mesh from the grouping unit 202 and receives the reconstructed base mesh from the base mesh decoding unit 103. The motion vector calculation unit 203 selects either skip mode or inter mode, which is a mode for encoding the motion vector of the group. Based on the selected mode, the motion vector calculation unit 203 calculates a motion vector from the vertices and the reconstructed base mesh as described above. The motion vector calculation unit 203 generates mode information indicating the selected mode. The motion vector calculation unit 203 outputs the mode information and the motion vector to the motion encoding unit 204.

[0059] Next, in S306, the motion encoding unit 204 receives the group mode information and the motion vector from the motion vector calculation unit 203 and encodes them.

[0060] 4 is a flowchart of the motion encoding process in the motion encoding unit 204. The process of S306 will be described in detail with reference to the flowchart in FIG.

[0061] First, in S401, the motion coding unit 204 determines whether to code the current (g-th) group in skip mode or inter mode based on the mode information. If the motion coding unit 204 determines to code in skip mode, it proceeds to the process of S404. If the motion coding unit 204 determines to code in inter mode, it proceeds to the process of S402.

[0062] Next, in S402, the motion coding unit 204 sets the value of sismu_skip_group_flag[g] to 0 and performs arithmetic coding.

[0063] Next, in S403, the motion encoding unit 204 individually encodes the motion vectors of the vertices belonging to the group to generate a motion bitstream. The motion encoding unit 204 outputs the motion bitstream to the base mesh integration unit 205, and proceeds to the process of S403.

[0064] On the other hand, in S404, the motion coding unit 204 sets the value of sismu_skip_group_flag[g] to 1 and performs arithmetic coding.

[0065] Next, in S405, the motion encoding unit 204 determines whether the representative motion vector of the gth group is a 0 vector.

[0066] In S405, if the motion encoding unit 204 determines that the representative motion vector is a 0 vector (Yes in S405), the process proceeds to S406. In S406, the motion encoding unit 204 performs arithmetic encoding by setting the value of sismu_skip_vector_flag[g] to 0, and proceeds to the process of S413 without encoding sismu_skip_prediction_flag[g] and the representative motion vector.

[0067] In S405, if the motion encoding unit 204 determines that the representative motion vector is a non-zero vector (No in S405), the process proceeds to S407. In S407, the motion encoding unit 204 performs arithmetic encoding with the value of sismu_skip_vector_flag[g] set to 1, and then proceeds to the process of S408.

[0068] Next, in S408, the motion coding unit 204 determines whether all of the following three conditions are met. Condition 1: The value of sismu_skip_vector_flag[g] is 1 Condition 2: The group to be coded is not the first group (g==0). Condition 3: The immediately preceding coded group (the g-1th group) was coded in skip mode, and the representative motion vector was not coded as a 0 vector. (This is equivalent to the value of sismu_skip_group_flag[g-1] being 1 and the value of sismu_skip_vector_flag[g-1] being 1.)

[0069] If the motion coding unit 204 determines that any one of the three conditions is not satisfied (No in S408), the process proceeds to S412. In this case, in S412, the motion coding unit 204 encodes the motion vectors and generates a motion bitstream without encoding sismu_skip_prediction_flag[g], which is a flag (skip prediction information) indicating whether or not to derive a representative motion vector of the gth group. The motion coding unit 204 may also encode skip group information indicating mode information together with the motion vectors. The motion coding unit 204 outputs the motion bitstream to the base mesh integration unit 205 and proceeds to S413.

[0070] On the other hand, if the motion coding unit 204 determines that all of the above three conditions are met (Yes in S408), the process proceeds to S409.

[0071] Next, in S409, the motion encoding unit 204 determines whether or not to encode the value obtained by subtracting the predicted vector as the representative motion vector syntax group of the gth group.

[0072] If the motion coding unit 204 determines in S409 to encode the value obtained by subtracting the prediction vector (Yes in S409), the process proceeds to S411. In S411, the motion coding unit 204 performs arithmetic coding by setting the value of sismu_skip_prediction_flag[g] to 1, and then proceeds to S412. In this case, in S412, the motion coding unit 204 performs arithmetic coding on the value of the motion vector obtained by subtracting submeshPredictionVector[g] as a representative motion vector syntax group, thereby generating a motion bitstream. The motion coding unit 204 may also encode skip group information indicating mode information together with the motion vector. The motion coding unit 204 outputs the motion bitstream to the base mesh integration unit 205, and then proceeds to S413.

[0073] On the other hand, if the motion coding unit 204 determines in S409 to encode a value that does not subtract the prediction vector (No in S409), the process proceeds to S410. In S410, the motion coding unit 204 performs arithmetic coding by setting the value of sismu_skip_prediction_flag[g] to 0, and then proceeds to S412. In this case, in S412, the motion coding unit 204 does not subtract submeshPredictionVector[g], but instead performs arithmetic coding on the motion vector value as a representative motion vector syntax group, thereby generating a motion bitstream. The motion coding unit 204 may also encode skip group information indicating mode information together with the motion vector. The motion coding unit 204 outputs the motion bitstream to the base mesh integration unit 205, and then proceeds to S413.

[0074] Next, in the process of S413, the motion encoding unit 204 determines whether or not the processing of all groups in the frame has been completed. If the motion encoding unit 204 determines that the processing has not been completed, it returns to the process of S401, and if it determines that the processing of all groups in the frame has been completed, it ends the motion encoding process.

[0075] Returning to the explanation of Fig. 3, in S307, the base mesh decoding unit 103 receives the base mesh bitstream from the base mesh encoding unit 102 and the displacement vector bitstream from the displacement vector encoding unit 104, decodes and reconstructs the base mesh, and generates a reconstructed base mesh. The base mesh decoding unit 103 outputs the generated reconstructed base mesh to the base mesh encoding unit 102 and the displacement vector encoding unit 104.

[0076] Next, in S308, the base mesh integration unit 205 receives the static base mesh bitstream from the static mesh encoding unit 201 and the motion bitstream from the motion encoding unit 204. The base mesh integration unit 205 multiplexes and integrates the static base mesh bitstream and the motion bitstream to generate a base mesh bitstream. The base mesh integration unit 205 may receive parameters including header information such as bmsps_inter_mesh_motion_group_size_minus1, multiplex them, and integrate them into the base mesh bitstream.

[0077] Next, in S309, the displacement vector encoding unit 104 receives the displacement vector from the preprocessing unit 101 and the reconstructed base mesh from the base mesh decoding unit 103, and updates and encodes the displacement vector. For example, the displacement vector encoding unit 104 performs a wavelet transform on the updated displacement vector and further packs the wavelet transform coefficients into a two-dimensional image. After packing, the displacement vector encoding unit 104 performs video encoding using the HEVC standard to generate a displacement vector bitstream and outputs it to the base mesh decoding unit 103 and the integration unit 107.

[0078] Next, in S310, the texture encoding unit 105 encodes the texture corresponding to the surface of the received input mesh using a video encoding method such as HEVC, and generates a texture bitstream.

[0079] Next, in S311, the atlas encoding unit 106 encodes the atlas data, which includes metadata necessary for decoding the encoded meshes, to generate an atlas bitstream.

[0080] Next, in S312, the integration unit 107 receives the base mesh bitstream from the base mesh encoding unit 102, the displacement vector bitstream from the displacement vector encoding unit 104, the texture bitstream from the texture encoding unit 105, and the atlas bitstream from the atlas encoding unit 106. The integration unit 107 multiplexes and integrates these bitstreams to generate a 3D mesh bitstream.

[0081] Next, in S313, the integrating unit 107 determines whether or not the processing of all frames has been completed. If the integrating unit 107 determines that the processing of all frames has been completed (Yes in S313), it ends the encoding process. If the integrating unit 107 determines that the processing of all frames has not been completed (No in S313), it returns to the process of S301 and processes the subsequent frame.

[0082] With the above configuration and operation, particularly in S306, when encoding the representative motion vector of the group, it is possible to reduce the encoding of redundant sismu_skip_prediction_flag based on conditions 1 to 3, thereby improving the efficiency of the encoding process.

[0083] In this embodiment, the representative motion vector of the group is determined by combining the three-dimensional motion vectors of X, Y, and Z, but this is not limited to this. For example, it may be determined whether to use skip mode or inter mode for encoding each of the X, Y, and Z components individually. In this case, sismu_skip_vector_flag, sismu_skip_prediction_flag, and sismu_skip_vector_flag are also encoded for each component, and conditions 1 to 3 are similarly applicable.

[0084] (Second embodiment) Fig. 5 is a block diagram showing the functional configuration of a 3D mesh decoding device in the second embodiment. Each functional unit of the decoding device shown in Fig. 5 is realized by an LSI, ASIC, or the like configured by hardware. In this embodiment, the decoding of encoded data generated by the same process as in the first embodiment will be described as an example.

[0085] The decoding device of the second embodiment includes a mesh having multiple vertices whose positions change dynamically, and decodes a bitstream of an image in which multiple vertices are grouped, frame by frame. As shown in Fig. 5, the decoding device of the second embodiment includes a separation unit 501, an atlas decoding unit 502, a base mesh decoding unit 503, a displacement vector decoding unit 504, a texture decoding unit 505, and a reconstruction unit 506.

[0086] The separation unit 501 receives a 3D mesh bitstream, separates the 3D mesh bitstream into the following four sub-bitstreams, and outputs them to the atlas decoding unit 502, the base mesh decoding unit 503, the displacement vector decoding unit 504, and the texture decoding unit 505.

[0087] Sub-bitstream 1. Atlas bitstream (output to the atlas decoding unit 502) Sub-bitstream 2. Base mesh bitstream (output to base mesh decoding unit 503) Sub-bitstream 3: Displacement vector bitstream (output to the displacement vector decoding unit 504) Sub-bitstream 4: Texture bitstream (output to texture decoding unit 505)

[0088] The atlas decoding unit 502 decodes the atlas bitstream received from the separation unit 501, generates atlas data, and outputs it to the reconstruction unit 506. The atlas data is data including metadata used for decoding and reconstruction, such as patch information.

[0089] The base mesh decoding unit 503 decodes and reconstructs the base mesh bitstream received from the separation unit 501 to generate a reconstructed base mesh and outputs it to the reconstruction unit 506 .

[0090] The disparity vector decoding unit 504 receives the disparity vector bit stream from the separating unit 501 and the reconstructed base mesh from the base mesh decoding unit 503 , generates a disparity vector, and outputs it to the reconstructing unit 506 .

[0091] The texture decoding unit 505 decodes the texture bitstream received from the separation unit 501 to generate texture and outputs it to the reconstruction unit 506 .

[0092] The reconstruction unit 506 receives the atlas data, the reconstructed base mesh, the displacement vector, and the texture, reconstructs the 3D mesh, and generates and outputs a reconstructed 3D mesh.

[0093] Next, the configuration of the base mesh decoding unit 503 will be described with reference to Fig. 6. As shown in Fig. 6, the base mesh decoding unit 503 has a base mesh separation unit 601, a static mesh decoding unit 602, a motion decoding unit 603, and a control unit 604.

[0094] The base mesh separator 601 receives the base mesh bitstream and separates it into a static mesh bitstream and a motion bitstream. The base mesh separator 601 outputs the static mesh bitstream to the static mesh decoder 602 and the motion bitstream to the motion decoder 603. Output.

[0095] The static mesh decoder 602 receives the static mesh bitstream, decodes it, generates a reconstructed intra-base mesh, and outputs it to the controller 604 .

[0096] The motion decoding unit 603 receives the motion bitstream, decodes it, generates a reconstructed inter-base mesh, and outputs it to the control unit 604 .

[0097] If the frame to be decoded is coded using intra-frame prediction, the control unit 604 outputs the reconstructed intra-base mesh received from the static mesh decoding unit 602 as the reconstructed base mesh. If the frame to be decoded is coded using inter-frame prediction, the control unit 604 outputs the reconstructed inter-base mesh received from the motion decoding unit 603 as the reconstructed base mesh.

[0098] The 3D mesh decoding operation of the 3D mesh decoding device in the second embodiment will be described with reference to Figures 5 and 6. In this embodiment, the 3D mesh bit stream generated in the first embodiment is received in frame units, but it may also be input in sub-meshes, for example, by dividing one frame's worth of mesh into multiple units.

[0099] 5 receives a 3D mesh bitstream in units of frames, and separates the 3D mesh bitstream into the following four sub-bitstreams and outputs them to the respective units.

[0100] Sub-bitstream 1. Atlas bitstream (output to the atlas decoding unit 502) Sub-bitstream 2. Base mesh bitstream (output to base mesh decoding unit 503) Sub-bitstream 3: Displacement vector bitstream (output to the displacement vector decoding unit 504) Sub-bitstream 4: Texture bitstream (output to texture decoding unit 505)

[0101] The atlas decoding unit 502 decodes the atlas bitstream input from the separation unit 501 and outputs the atlas data to the reconstruction unit 506. The atlas data is data including metadata used for decoding and reconstruction, such as patch information.

[0102] 6 is a block diagram showing the functional configuration of the base mesh decoding unit in the second embodiment. The operation of the base mesh decoding unit 503 will be described with reference to the block diagram of FIG.

[0103] The base mesh separation unit 601 receives the base mesh bitstream from the separation unit 501. The base mesh separation unit 601 separates the base mesh bitstream into a static mesh bitstream, which is data encoded using intra-frame prediction, and a motion bitstream, which is data encoded using inter-frame prediction. The base mesh separation unit 601 outputs the static mesh bitstream to the static mesh decoding unit 602 and outputs the motion bitstream to the motion decoding unit 603. The base mesh separation unit 601 also decodes header information related to the base mesh. For example, the base mesh separation unit 601 decodes parameters such as bmsps_inter_mesh_motion_group_size_minus1, which is the number of vertices belonging to one group minus 1, and outputs this to the subsequent stage. In this embodiment, the number of vertices belonging to a group is 16, so a value of 15 is encoded using Golomb coding as bmsps_inter_mesh_motion_group_size_minus1. The base mesh separation unit 601 outputs the header information to the static mesh decoding unit 602 , the motion decoding unit 603 , and the control unit 604 .

[0104] The static mesh decoder 602 receives the static mesh bitstream from the base mesh separator 601, decodes it to generate a reconstructed intra-base mesh, and outputs it to the motion decoder 603 and the controller 604. Here, the static mesh bitstream is assumed to have been coded using the Edgebreaker algorithm, but may also be coded using other algorithms. Here, the decoded vertices are assumed to be indexed in the order in which the Edgebreaker algorithm traverses the triangles.

[0105] The motion decoding unit 603 receives the motion bitstream from the base mesh separation unit 601 and the reconstructed intra-base mesh from the static mesh decoding unit 602 , and decodes and outputs the reconstructed intra-base mesh to the control unit 604 .

[0106] The motion decoding unit 603 decodes sismu_skip_group_flag[g], which is a flag indicating whether the gth group of a frame is coded in skip mode. Here, g is counted up in the order of processing, starting with 0 for the first group. The number of vertices belonging to each group is calculated by bmsps_inter_mesh_motion_group_size_minus1 decoded by the base mesh separation unit 601, and is 16 in this embodiment. Each group contains 16 vertices, ordered by the smallest vertex index. If the value of sismu_skip_group_flag[g] is 0 (FALSE), this indicates that the gth group is coded in inter mode, and if the value is 1 (TRUE), this indicates that the gth group is coded in skip mode. If the gth group is coded in skip mode, the values ​​of the motion vectors of all vertices belonging to the gth group will be the same as decodedSubmeshSkipVector[g], which is the representative motion vector of the group.

[0107] decodedSubmeshSkipVector[g] holds three-dimensional motion information (X, Y, Z). If the coordinates of a vertex in the reference frame are (1, 1, 1) and decodedSubmeshSkipVector[g] is (0, 1, 2), the coordinates of the corresponding vertex in the frame to be decoded will be (1, 2, 3).

[0108] Next, the motion decoding unit 603 decodes sismu_skip_vector_flag[g], a flag indicating whether the representative motion vector of the gth group is to be a zero vector, only if sismu_skip_group_flag[g] is 1. If the value of sismu_skip_vector_flag[g] is 0, it indicates that the representative motion vector of the gth group is to be a zero vector (the value of decodedSubmeshSkipVector[g] is (0,0,0)). This is equivalent to using the position information of the vertices of the corresponding group (gth group) in the reference frame as is. If the value of sismu_skip_vector_flag[g] is 1, it indicates that a non-zero motion vector is to be the representative motion vector of the gth group.

[0109] Next, the motion decoding unit 603 decodes sismu_skip_prediction_flag[g], which is a flag indicating whether to derive a representative motion vector for the gth group using the predicted vector submeshPredictionVector[g]. The motion decoding unit 603 decodes sismu_skip_prediction_flag[g] only if all of the following three conditions are satisfied:

[0110] Condition 1: The value of sismu_skip_vector_flag[g] is 1 Condition 2: The group to be decoded is not the first group of the frame (g==0). Condition 3: The immediately preceding coded group (the g-1th group) was coded in skip mode, and the representative motion vector was not coded as a 0 vector. (This is equivalent to the value of sismu_skip_group_flag[g-1] being 1 and the value of sismu_skip_vector_flag[g-1] being 1.)

[0111] The motion decoding unit 603 decodes the value obtained by subtracting submeshPredictionVector from the value of decodedSubmeshSkipVector[g] as a representative motion vector syntax group when sismu_skip_prediction_flag[g] is 1. When the value of sismu_skip_prediction_flag[g] is 0, the motion decoding unit 603 decodes the value of decodedSubmeshSkipVector[g] as a representative motion vector syntax group.

[0112] The value of submeshPredictionVector[g] depends on the mode of the last decoded group (the g-1th group). If the last decoded group was not coded using skip mode (i.e., the motion vectors of each vertex were coded individually), the average value of the motion vectors of the vertices in the group is used as submeshPredictionVector[g]. If the last decoded group was coded using skip mode, the value of decodedSubmeshSkipVector[g-1], which is the representative motion vector of the last group, is used as submeshPredictionVector[g].

[0113] Next, the motion decoding unit 603 decodes the following representative motion vector syntax group only when the value of sismu_skip_vector_flag[g] is 1. Here, k = 0 indicates a vector of the X component, and k = 1 or 2 indicates vectors of the Y and Z components, respectively.

[0114] sismu_mv_skip_abs_gt0[g][k]: Flag indicating whether the absolute value of the vector of the kth component of the gth group is greater than 0. If the flag value is 0, it indicates that the absolute value is 0, and if the flag value is 1, it indicates that the absolute value is 1 or greater.

[0115] sismu_mv_skip_sign[g][k]: A flag indicating the positive or negative sign of the k-component vector, which is encoded only if the absolute value of the k-component vector is greater than 0. If the flag value is 0, it indicates that the value is greater than or equal to 0, and if the flag value is 1, it indicates that the value is negative.

[0116] sismu_mv_skip_abs_gt1[g][k]: A flag that indicates whether the absolute value of the k-component vector is greater than 1 or not, and is coded only if the absolute value of the k-component vector is greater than 0. If the flag value is 0, it indicates that the absolute value is 1, and if the flag value is 1, it indicates that the absolute value is 2 or greater.

[0117] sismu_mv_skip_abs_rem[g][k]: Encoded only if the absolute value of the k-component vector is greater than 1, and indicates the absolute value of the k-component vector -2.

[0118] In the technology currently being discussed in MPEG, only the above-mentioned condition 1 is used as a condition for decoding sismu_skip_prediction_flag[g]. However, if condition 2 is not satisfied (g == 0), there is no immediately preceding group, and therefore the value of the predicted vector, submeshPredictionVector[g], cannot be correctly derived. Furthermore, if condition 3 is not satisfied, the predicted vector, submeshPredictionVector[g], becomes a 0 vector, and the processing remains the same regardless of the value of sismu_skip_prediction_flag[g]. Therefore, the motion decoding unit 603 of the 3D mesh decoding device according to this embodiment does not decode sismu_skip_prediction_flag[g] even when conditions 2 and 3 are not satisfied, thereby improving coding efficiency compared to conventional techniques. Furthermore, in this embodiment, sismu_skip_prediction_flag[g] is decoded only when all three conditions are satisfied, but this is not limited to this. For example, the motion decoding unit 603 may decode sismu_skip_prediction_flag[g] when only two of the three conditions are satisfied, or may decode when only one of the three conditions is satisfied. Alternatively, the motion decoding unit 603 may first calculate the predicted vector, submeshPredictionVector[g], and decode sismu_skip_prediction_flag[g] only if the result is not a zero vector. Furthermore, instead of or in addition to conditions 2 and 3, the motion encoding unit 204 may encode sismu_skip_prediction_flag[g] by determining that the condition is satisfied when the average of the motion vectors of the immediately preceding encoded group is not a zero vector.

[0119] Next, if the frame to be decoded is coded using intra-frame prediction, the control unit 604 outputs the reconstructed intra-base mesh received from the static mesh decoding unit 602 as the reconstructed base mesh. If the frame to be decoded is coded using inter-frame prediction, the control unit 604 outputs the reconstructed inter-base mesh received from the motion decoding unit 603 as the reconstructed base mesh. Note that the control unit 604 may receive header information from the base mesh separation unit 601 and determine whether the frame to be decoded is coded using intra-frame prediction or inter-frame prediction.

[0120] Returning to Figure 5, the disparity vector decoding unit 504 receives the disparity vector bitstream from the separation unit 501, decodes it to generate disparity vectors, and outputs them to the reconstruction unit 506. The disparity vector bitstream is first decoded using a video decoding method such as HEVC, and then undergoes an inverse wavelet transform to derive the disparity vectors for each vertex.

[0121] Next, the texture decoding unit 505 decodes the texture corresponding to the surface of the input mesh using a video decoding method such as HEVC, and generates a texture.

[0122] Next, the reconstruction unit 506 receives the atlas data, the reconstructed base mesh, the displacement vector, and the texture, and generates and outputs a reconstructed 3D mesh. The reconstruction unit 506 repeatedly divides each triangle of the reconstructed base mesh into four sub-triangles using the midpoints of the edges between the vertices. In this embodiment, the reconstruction unit 506 divides each triangle of the reconstructed base mesh twice into 16 sub-triangles, but this is not limited to this, and any division method specified in the header information may be used. Next, the reconstruction unit 506 adds the decoded displacement vector to the position of each divided vertex to reconstruct the divided vertex positions.

[0123] 7 is a diagram showing a flowchart of the decoding process in the 3D mesh decoding device according to the second embodiment. The flow of the decoding process by the 3D mesh decoding device according to the second embodiment will be described with reference to FIG.

[0124] In S701, the separation unit 501 separates the input 3D mesh bitstream into the following four sub-bitstreams and outputs them to the atlas decoding unit 502, the base mesh decoding unit 503, the displacement vector decoding unit 504, and the texture decoding unit 505.

[0125] Sub-bitstream 1. Atlas bitstream (output to the atlas decoding unit 502) Sub-bitstream 2. Base mesh bitstream (output to base mesh decoding unit 503) Sub-bitstream 3: Displacement vector bitstream (output to the displacement vector decoding unit 504) Sub-bitstream 4: Texture bitstream (output to texture decoding unit 505)

[0126] Next, in S702, the atlas decoding unit 502 decodes the input atlas bitstream to generate atlas data, and outputs it to the reconstruction unit 506.

[0127] Next, in S703, the texture decoding unit 505 decodes the input texture bitstream to generate texture and outputs it to the reconstruction unit 506.

[0128] Next, in S704, the base mesh separation unit 601 of the base mesh decoding unit 503 receives the base mesh bitstream, separates it into a static mesh bitstream and a motion bitstream, and reads header information such as bmsps_inter_mesh_motion_group_size_minus1. The base mesh separation unit 601 outputs the static mesh bitstream together with the associated header information to the static mesh decoding unit 602. The base mesh separation unit 601 outputs the motion bitstream together with the associated header information to the motion decoding unit 603. The base mesh separation unit 601 also outputs the read header information to the control unit 604.

[0129] Next, in S705, the base mesh separation unit 601 determines whether the frame to be decoded is coded using intra-frame prediction or inter-frame prediction based on the header information. If the base mesh separation unit 601 determines that the frame is coded using intra-frame prediction (No in S705), the process proceeds to S706. If the control unit 604 determines that the frame is coded using inter-frame prediction (Yes in S705), the process proceeds to S707.

[0130] Next, in S706, the static mesh decoding unit 602 decodes the static mesh bitstream from the received static mesh bitstream and outputs the reconstructed intra-base mesh to the control unit 604 and the motion decoding unit 603.

[0131] Next, in S707, the motion decoding unit 603 receives the reconstructed intra-base mesh and the motion bitstream. The motion decoding unit 603 decodes the motion bitstream and generates motion vectors for the vertices. The motion decoding unit 603 uses the decoded motion vectors to add the reconstructed intra-base mesh or the reconstructed inter-base mesh that it reconstructed itself to the corresponding vertices as a reference frame, thereby generating a reconstructed inter-base mesh for the frame to be processed. The motion decoding unit 603 outputs the generated reconstructed inter-base mesh to the control unit 604.

[0132] 8 is a flowchart of the motion decoding process of the motion bitstream in the motion decoding unit. The process of S707 will be described in detail with reference to FIG.

[0133] In S801, the motion decoding unit 603 decodes the sismu_skip_group_flag[g] indicating the coding mode from the motion bitstream.

[0134] In S802, the motion decoding unit 603 determines whether the (gth) group to be processed has been coded in skip mode or inter mode based on the value of sismu_skip_group_flag[g]. If the motion decoding unit 603 determines that the group has been coded in skip mode (Yes in S802), the process proceeds to S804. If the motion decoding unit 603 determines that the group has been coded in inter mode (No in S802), the process proceeds to S803.

[0135] Next, in S803, the motion decoding unit 603 individually decodes the motion vectors of the vertices of the group and adds them to the coordinates of the corresponding vertices in the reference frame to find the coordinates of each vertex in the frame to be processed.

[0136] Next, in S804, the motion decoding unit 603 decodes sismu_skip_vector_flag[g], which indicates whether or not the representative motion vector of the group is to be set to a 0 vector.

[0137] Next, in S8040, the motion decoding unit 603 determines whether the representative motion vector of the gth group is a 0 vector based on sismu_skip_vector_flag[g].

[0138] In S8040, if the motion decoding unit 603 determines that the representative motion vector is a 0 vector (Yes in S8040), the motion decoding unit 603 proceeds to the processing of S812 without decoding sismu_skip_prediction_flag[g] and the representative motion vector.

[0139] In S8040, if the motion decoding unit 603 determines that the representative motion vector is a non-zero vector (No in S8040), the process proceeds to S805.

[0140] Next, in S805, the motion decoding unit 603 determines whether or not all of the following conditions 1 to 3 are satisfied. Condition 1: The value of sismu_skip_vector_flag[g] is 1 Condition 2: The group to be decoded is not the first group of the frame (g==0). Condition 3: The most recently decoded group (the g-1th group) is coded in skip mode, and the representative motion vector is not coded as a 0 vector. (This is equivalent to the value of sismu_skip_group_flag[g-1] being 1 and the value of sismu_skip_vector_flag[g-1] being 1.)

[0141] If the motion decoding unit 603 determines that all of the conditions 1 to 3 are satisfied (Yes in S805), the process proceeds to S806. If the motion decoding unit 603 determines that any one of the conditions is not satisfied (No in S805), the process proceeds directly to S807 without executing S806, i.e., without decoding sismu_skip_prediction_flag[g].

[0142] In S806, the motion decoding unit 603 decodes sismu_skip_prediction_flag[g], which indicates whether or not to derive a representative motion vector of the group using a predicted vector, and the process proceeds to step S807.

[0143] In this embodiment, the condition for decoding sismu_skip_prediction_flag[g] is that all of conditions 1 to 3 are satisfied, but this is not limiting. For example, the motion decoding unit 603 may decode when two or less conditions are satisfied, as long as it determines whether to decode based on the same determination process as on the encoding side. These conditions are based on information that has already been decoded, so the same process can be performed on the encoding side and the decoding side.

[0144] Next, in S807, the motion decoding unit 603 decodes the representative motion vector syntax group.

[0145] Next, in S808, the motion decoding unit 603 determines whether to derive a representative motion vector using a predicted vector based on the value of sismu_skip_prediction_flag[g]. If the motion decoding unit 603 determines that a representative motion vector is to be derived using a predicted vector (Yes in S808), the process proceeds to S809. If the motion decoding unit 603 determines that a representative motion vector is to be derived without using a predicted vector, the process proceeds to S810.

[0146] Next, in S809, the motion decoding unit 603 calculates decodedSubmeshSkipVector[g], which is the representative motion vector of the gth group, using the following formula: Note that submeshPredictionVector[g] is a prediction vector. decodedSubmeshSkipVector[g] = submeshPredictionVector[g] + Decoded result of representative motion vector syntax group

[0147] Next, in S810, the motion decoding unit 603 calculates decodedSubmeshSkipVector[g], which is the representative motion vector of the gth group, using the following formula. decodedSubmeshSkipVector[g] = Decoded result of representative motion vector syntax group

[0148] Next, in S811, the motion decoding unit 603 calculates the coordinates of each vertex in the group of the current frame of the base mesh. For example, the motion decoding unit 603 calculates the coordinates of each vertex by adding the decodedSubmeshSkipVector[g] to the coordinates of the corresponding vertex (having the same index) in the reference frame, generates a reconstructed inter-base mesh, and outputs it to the control unit 604.

[0149] Next, in S812, the motion decoding unit 603 determines whether or not processing of all groups in the frame has been completed. If the motion decoding unit 603 determines that processing of all groups has not been completed (No in S812), it returns to S801 and processes the subsequent group. If the motion decoding unit 603 determines that processing of all groups has been completed (Yes in S812), it ends the motion decoding process.

[0150] Returning to Figure 7, in S7080, the control unit 604 of the base mesh decoding unit 503 outputs a reconstructed base mesh. Here, if the frame to be decoded is coded using intra-frame prediction, the control unit 604 outputs the reconstructed intra-base mesh received from the static mesh decoding unit 602 as the reconstructed base mesh. If the frame to be decoded is coded using inter-frame prediction, the control unit 604 outputs the reconstructed inter-base mesh received from the motion decoding unit 603 as the reconstructed base mesh. Note that the control unit 604 may determine the coding method based on header information.

[0151] In S708, the disparity vector decoding unit 504 decodes the received disparity vector bitstream to generate disparity vectors, and outputs the generated disparity vectors to the reconstruction unit 506. The disparity vector decoding unit 504 first decodes the disparity vector bitstream using a video decoding method such as HEVC, and then performs an inverse wavelet transform to derive disparity vectors for each vertex.

[0152] Next, in S709, the reconstruction unit 506 reconstructs and outputs a 3D mesh from the received atlas data, reconstructed base mesh, displacement vector, and texture. For example, the reconstruction unit 506 repeatedly divides each triangle of the reconstructed base mesh into four sub-triangles at the midpoints of the edges between vertices. In this embodiment, each triangle of the reconstructed base mesh is divided twice into 16 sub-triangles, but this is not limiting and any division method specified in the header information may be used. Next, the decoded displacement vector is added to the position of each divided vertex, and the divided vertex positions are reconstructed to generate a 3D mesh.

[0153] With the above configuration and operation, the mesh decoding device of the second embodiment no longer decodes the redundant sismu_skip_prediction_flag when decoding the representative motion vector of a group, and can decode a bitstream with improved coding efficiency.

[0154] In this embodiment, the mode of the representative motion vector of the group is determined by combining the three-dimensional motion vectors of X, Y, and Z, but this is not limited to this. For example, it may be determined whether each of the X, Y, and Z components is coded in skip mode or inter mode. In this case, sismu_skip_vector_flag, sismu_skip_prediction_flag, and sismu_skip_vector_flag are coded for each component, but conditions 1 to 3 are also applicable, so decoding can be performed without any problems.

[0155] (Third embodiment) In the above embodiment, each processing unit shown in Figures 1 and 5 has been described as being configured by hardware. However, the processing performed by each processing unit shown in these figures may be configured by a computer program.

[0156] 9 is a block diagram showing an example of the hardware configuration of a computer applicable to the encoding device and / or decoding device according to each of the above embodiments. The computer has a CPU 901, RAM 902, ROM 903, an operation unit 904, a display unit 905, an external storage device 906, an I / F 907, and a bus 908. The CPU 901, RAM 902, ROM 903, operation unit 904, display unit 905, external storage device 906, and I / F 907 are connected via the bus 908 so as to be able to transmit and receive signals to and from each other.

[0157] The CPU 901 stands for Central Processing Unit and is an arithmetic processing device. The CPU 901 reads computer programs and data stored in the ROM 903 and the external storage device 906 and expands them in the RAM 902, thereby realizing various functions and controlling the entire computer. The CPU 901 executes the processes described above performed by the encoding device and decoding device according to each of the above-mentioned embodiments. That is, the CPU 901 executes computer programs to realize the functions of each processing unit shown in FIGS. 1 and 5. The computer may have other processors, such as an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), and a QPU (Quantum Processing Unit), instead of or in addition to the CPU 901. In this case, one or more processors may function as each processing unit shown in FIGS. 1 and 5.

[0158] RAM 902, an abbreviation for Random Access Memory, is a memory from which data can be read and written at high speed. RAM 902 has areas for temporarily storing computer programs and data loaded from ROM 903 and external storage device 906, and data acquired from the outside via I / F 907. RAM 902 also has a work area used by CPU 901 when executing various processes. That is, RAM 902 can be allocated as a frame memory, for example, or can provide various other areas as needed.

[0159] ROM 903 is an abbreviation for Read Only Memory, and is a non-volatile memory that can retain data even when power is not supplied. ROM 903 stores the setting data and boot program of this computer.

[0160] The operation unit 904 is an input device such as a keyboard and a mouse that can input data, instructions, etc. A user of this computer can input various instructions to the CPU 901 by operating the operation unit 904.

[0161] The display unit 905 displays images such as the results of processing by the CPU 901. The display unit 905 is configured with an image display device such as a liquid crystal display or an organic EL (Electro Luminescence) display.

[0162] The external storage device 906 is a non-volatile large-capacity information storage device, such as a hard disk drive or a solid-state drive (SSD). The external storage device 906 stores an operating system (OS), computer programs for causing the CPU 901 to implement the functions of the various components shown in FIGS. 1 and 5, and other information. The external storage device 906 also stores image data to be processed. The computer programs and data stored in the external storage device 906 are loaded into the RAM 902 as appropriate under the control of the CPU 901, and are then processed by the CPU 901.

[0163] The I / F 907 is an interface for external devices. The I / F 907 can be connected to networks such as a LAN or the Internet, or to other devices such as a projection device or display device. This computer can obtain various information from external devices and send it to external devices via the I / F 907.

[0164] In a computer having the above configuration, the CPU 901 plays a central role in executing the processes described in the flowcharts above.

[0165] (Other embodiments) The present invention can also be achieved by providing a storage medium containing computer program code that implements the above-described functions to a system, and having the system read and execute the computer program code. In this case, the computer program code itself read from the storage medium implements the functions of the above-described embodiments, and the storage medium containing the computer program code constitutes the present invention. It also includes cases where an operating system (OS) running on a computer performs some or all of the actual processing based on the instructions of the program code, thereby implementing the above-described functions. In other words, the present invention can also be realized by a combination of software and hardware.

[0166] Furthermore, the present invention may be realized in the following form: That is, computer program code read from a storage medium is written to memory in a function expansion card inserted into a computer or in a function expansion unit connected to the computer, and the CPU in the function expansion card or function expansion unit then performs some or all of the actual processing based on the instructions of the computer program code, thereby realizing the above-mentioned functions.

[0167] When the present invention is applied to the storage medium, the storage medium stores computer program code corresponding to the flowcharts described above.

[0168] (Other Examples) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0169] The disclosure of this specification includes the following encoding device, decoding device, encoding method, decoding method, and program. (Item 1) 1. An encoding device that encodes an image including a mesh having vertices whose positions change dynamically, on a frame-by-frame basis, a grouping means for grouping a plurality of vertices included in the mesh; a motion vector calculation means for calculating a motion vector indicating a motion between a reference point, which is a vertex of a reference frame, and a vertex of a frame to be coded; a motion encoding means for encoding the motion vector; and The motion encoding means encoding skip group information indicating whether vertices belonging to the same group are encoded by the same representative motion vector; If the skip group information indicates that the representative motion vector is coded, coding skip vector information indicating whether the representative motion vector of the same group is a zero vector; If the skip vector information indicates that the representative motion vector is not a 0 vector and at least one of the group to be coded and the group coded immediately before the group satisfies a predetermined condition, coding skip prediction information indicating whether the representative motion vector is coded using a predictive vector. 1. An encoding device comprising: (Item 2) The predetermined condition is met when the representative motion vector of the immediately preceding coded group indicates that it is not coded as a zero vector. 2. The encoding device according to item 1, (Item 3) The predetermined condition is met when the average of the motion vectors of the immediately preceding coded group is not a zero vector. 2. The encoding device according to item 1, (Item 4) The predetermined condition is met if the group to be coded is not the first group in the frame to be coded. 2. The encoding device according to item 1, (Item 5) The predetermined condition is satisfied when the immediately preceding coded group is coded with the representative motion vector and the representative motion vector is not coded with a zero vector. 2. The encoding device according to item 1, (Item 6) The motion encoding means does not encode the skip prediction information if the predetermined condition is not satisfied. 2. The encoding device according to item 1, (Item 7) When the representative motion vector is a 0 vector, the motion encoding means does not encode at least one of the representative motion vector and the skip prediction information. 2. The encoding device according to item 1, (Item 8) The motion encoding means does not encode the skip prediction information when the prediction vector is a 0 vector. 2. The encoding device according to item 1, (Item 9) 1. A decoding device that decodes a bitstream to be decoded on a frame-by-frame basis, the bitstream including a mesh having a plurality of vertices whose positions change dynamically, and an image in which the plurality of vertices are grouped, the decoding device comprising: a motion decoding means for decoding a motion vector indicating a motion of a vertex from the bitstream; The motion decoding means decode skip group information from the bitstream, which indicates whether vertices belonging to the same group are coded using the same representative motion vector; If the vertices belonging to the same group are coded using the representative motion vector, decoding skip vector information from the bitstream that indicates whether the representative motion vector of the group is a 0 vector; When the skip vector information indicates that the group is coded using a representative motion vector that is not a zero vector, and at least one of the group to be decoded and the group coded immediately before the group satisfies a predetermined condition, decode skip prediction information that indicates whether the representative motion vector that is not a zero vector is coded using a predictive vector. A decoding device characterized by: (Item 10) The predetermined condition is met when the representative motion vector of the immediately preceding decoded group indicates that it is not coded with a zero vector. 10. The decoding device according to item 9, (Item 11) The predetermined condition is met when the average of the motion vectors of the immediately preceding decoded group is not a zero vector. 11. The decoding device according to item 9 or 10, (Item 12) The predetermined condition is met when the group to be decoded is not the first group of the frame to be decoded. 12. The decoding device according to claim 9, wherein: (Item 13) The predetermined condition is satisfied when the previously decoded group is coded with the representative motion vector and the representative motion vector is not coded with a zero vector. 13. The decoding device according to any one of items 9 to 12, wherein: (Item 14) The motion decoding means does not decode the skip prediction information if the predetermined condition is not satisfied. 14. The decoding device according to claim 9, wherein: (Item 15) When the representative motion vector is a 0 vector, the motion decoding means does not decode at least one of the representative motion vector and the skip prediction information. 15. The decoding device according to claim 9, wherein: (Item 16) The motion decoding means does not decode the skip prediction information when the prediction vector is a 0 vector. 16. The decoding device according to claim 9, wherein: (Item 17) 1. A coding method for coding, on a frame-by-frame basis, an image including a mesh having vertices whose positions change dynamically, comprising: a grouping step of grouping a plurality of vertices included in the mesh; a motion vector calculation step of calculating a motion vector indicating a motion between a reference point, which is a vertex of a reference frame, and a vertex of a frame to be coded; a motion encoding step of encoding the motion vector; and In the motion encoding step, encoding skip group information indicating whether vertices belonging to the same group are encoded by the same representative motion vector; If the skip group information indicates that the representative motion vector is coded, coding skip vector information indicating whether the representative motion vector of the same group is a zero vector; If the skip vector information indicates that the representative motion vector is not a 0 vector and at least one of the group to be coded and the group coded immediately before the group satisfies a predetermined condition, coding skip prediction information indicating whether the representative motion vector has been coded using a predictive vector. 10. A coding method comprising: (Item 18) 1. A decoding method for decoding a bitstream to be decoded in units of frames of an image including a mesh having a plurality of vertices whose positions change dynamically, the image being a group of the plurality of vertices, the method comprising: a motion decoding step of decoding motion vectors indicating vertex motion from the bitstream, In the motion decoding step, decode skip group information from the bitstream, which indicates whether vertices belonging to the same group are coded using the same representative motion vector; If the vertices belonging to the same group are coded using the representative motion vector, decode skip vector information from the bitstream, which indicates whether the representative motion vector of the group is a 0 vector; When the skip vector information indicates that the group is coded using a representative motion vector that is not a zero vector, and at least one of the group to be decoded and the group coded immediately before the group satisfies a predetermined condition, decode skip prediction information that indicates whether the representative motion vector that is not a zero vector is coded using a predictive vector. A decoding method comprising: (Item 19) A program for causing a computer to function as each means of the encoding device according to any one of items 1 to 8. (Item 20) A program for causing a computer to function as a means for the decoding device according to any one of items 9 to 16.

[0170] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0171] 202: Grouping unit; 203: Motion vector calculation unit; 204: Motion coding unit; 601: Base mesh separation unit; 603: Motion decoding unit.

Claims

1. 1. An encoding device that encodes an image including a mesh having vertices whose positions change dynamically, on a frame-by-frame basis, a grouping means for grouping a plurality of vertices included in the mesh; a motion vector calculation means for calculating a motion vector indicating a motion between a reference point, which is a vertex of a reference frame, and a vertex of a frame to be coded; a motion encoding means for encoding the motion vector; and The motion encoding means encoding skip group information indicating whether vertices belonging to the same group are encoded by the same representative motion vector; If the skip group information indicates that the representative motion vector is coded, coding skip vector information indicating whether the representative motion vector of the same group is a zero vector; If the skip vector information indicates that the representative motion vector is not a 0 vector and at least one of the group to be coded and the group coded immediately before the group satisfies a predetermined condition, skip prediction information indicating whether the representative motion vector has been coded using a prediction vector is coded.

1. An encoding device comprising:

2. The predetermined condition is met when the representative motion vector of the immediately preceding coded group indicates that it is not coded as a zero vector.

2. The encoding device according to claim 1.

3. The predetermined condition is met when the average of the motion vectors of the immediately preceding coded group is not a zero vector.

2. The encoding device according to claim 1.

4. The predetermined condition is met if the group to be coded is not the first group in the frame to be coded.

2. The encoding device according to claim 1.

5. The predetermined condition is satisfied when the immediately preceding coded group is coded with the representative motion vector and the representative motion vector is not coded with a zero vector.

2. The encoding device according to claim 1.

6. The motion encoding means does not encode the skip prediction information if the predetermined condition is not satisfied.

2. The encoding device according to claim 1.

7. When the representative motion vector is a 0 vector, the motion encoding means does not encode at least one of the representative motion vector and the skip prediction information.

2. The encoding device according to claim 1.

8. The motion encoding means does not encode the skip prediction information when the prediction vector is a 0 vector.

2. The encoding device according to claim 1.

9. 1. A decoding device that decodes a bitstream to be decoded in units of frames of an image in which the mesh has a plurality of vertices whose positions change dynamically, the image being a group of the plurality of vertices, a motion decoding means for decoding a motion vector indicating a motion of a vertex from the bitstream; The motion decoding means decode skip group information from the bitstream, which indicates whether vertices belonging to the same group are coded using the same representative motion vector; If the vertices belonging to the same group are coded using the representative motion vector, decoding skip vector information from the bitstream that indicates whether the representative motion vector of the group is a 0 vector; When the skip vector information indicates that the group is coded using a representative motion vector that is not a zero vector, and at least one of the group to be decoded and the group coded immediately before the group satisfies a predetermined condition, decode skip prediction information that indicates whether the representative motion vector that is not a zero vector is coded using a predictive vector. A decoding device characterized by:

10. The predetermined condition is met when the representative motion vector of the immediately preceding decoded group indicates that it is not coded with a zero vector.

10. The decoding device according to claim 9.

11. The predetermined condition is met when the average of the motion vectors of the immediately preceding decoded group is not a zero vector.

10. The decoding device according to claim 9.

12. The predetermined condition is met when the group to be decoded is not the first group of the frame to be decoded.

10. The decoding device according to claim 9.

13. The predetermined condition is satisfied when the previously decoded group is coded with the representative motion vector and the representative motion vector is not coded with a zero vector.

10. The decoding device according to claim 9.

14. The motion decoding means does not decode the skip prediction information if the predetermined condition is not satisfied.

10. The decoding device according to claim 9.

15. When the representative motion vector is a 0 vector, the motion decoding means does not decode at least one of the representative motion vector and the skip prediction information.

10. The decoding device according to claim 9.

16. The motion decoding means does not decode the skip prediction information when the prediction vector is a 0 vector.

10. The decoding device according to claim 9.

17. 1. A coding method for coding, on a frame-by-frame basis, an image including a mesh having vertices whose positions change dynamically, comprising: a grouping step of grouping a plurality of vertices included in the mesh; a motion vector calculation step of calculating a motion vector indicating a motion between a reference point, which is a vertex of a reference frame, and a vertex of a frame to be coded; a motion encoding step of encoding the motion vector; and In the motion encoding step, encoding skip group information indicating whether vertices belonging to the same group are encoded by the same representative motion vector; If the skip group information indicates that the representative motion vector is coded, coding skip vector information indicating whether the representative motion vector of the same group is a zero vector; If the skip vector information indicates that the representative motion vector is not a 0 vector and at least one of the group to be coded and the group coded immediately before the group satisfies a predetermined condition, skip prediction information indicating whether the representative motion vector has been coded using a prediction vector is coded.

10. A coding method comprising:

18. 1. A decoding method for decoding a bitstream to be decoded in units of frames of an image including a mesh having a plurality of vertices whose positions change dynamically, the image being a group of the plurality of vertices, the method comprising: a motion decoding step of decoding motion vectors indicating vertex motion from the bitstream, In the motion decoding step, decode skip group information from the bitstream, which indicates whether vertices belonging to the same group are coded using the same representative motion vector; If the vertices belonging to the same group are coded using the representative motion vector, decoding skip vector information from the bitstream that indicates whether the representative motion vector of the group is a 0 vector; When the skip vector information indicates that the group is coded using a representative motion vector that is not a zero vector, and at least one of the group to be decoded and the group coded immediately before the group satisfies a predetermined condition, decode skip prediction information that indicates whether the representative motion vector that is not a zero vector is coded using a predictive vector. A decoding method comprising:

19. A program for causing a computer to function as each means of the encoding device according to any one of claims 1 to 8.

20. A program for causing a computer to function as the means of the decoding device according to any one of claims 9 to 16.

Images