Information processing apparatus and method

Abstract

The present disclosure pertains to an information processing device and method that make it possible to suppress a decrease in encoding efficiency. The method comprises: decoding encoded data; generating a motion vector indicating a position difference between a vertex of the reference base grid and a vertex of the current base grid; reconstructing the current base grid by applying the motion vector to the vertices of the reference base grid; and reconstructing the current grid by applying the reference displacement vector to the vertices of the current base grid. In addition, a motion vector indicating a position difference between the vertices of the reference base grid and the vertices of the current base grid is encoded; and encoding control information related to applying the reference displacement vector to the vertices of the current base grid when reconstructing the current grid. The present disclosure is applicable, for example, to an information processing device, an electronic instrument, an information processing method, and a program.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to an information processing apparatus and method, and particularly relates to an information processing apparatus and method in which a decrease in coding efficiency can be suppressed. BACKGROUND

[0002] Video-based dynamic mesh coding (V-DMC) is known as an encoding method for a mesh that is 3D data representing a three-dimensional structure of an object using vertices and connections (see, for example, Non-Patent Literature 1). In V-DMC, a mesh to be encoded is represented by a coarse base mesh and displacement vectors of partition points obtained by subdividing the base mesh, and the base mesh and the displacement vectors are encoded. The displacement vectors are stored (packed) in a two-dimensional image, and then encoded as a moving image (displacement video) with the two-dimensional image as a frame.

[0003] Various other encoding methods that apply V-DMC have been proposed. For example, a method of arithmetic encoding displacement coefficients or prediction residuals thereof has been proposed (see, for example, Non-Patent Literature 2). In addition, a method of skipping transmission of a current motion vector has been proposed (see, for example, Non-Patent Literature 3). LIST OF REFERENCES NON-PATENT LITERATURE

[0004] Non-Patent Literature 1: Khaled Mammou, Jungsun Kim, Alexis Tourapis, Dimitri Podborski, Krasimir Kolarov, “[V-CG] Apple’s Dynamic Mesh Coding CfP Response”, ISO / IEC JTC 1 / SC 29 / WG 7 m59281, April 2022 Non-Patent Literature 2: Chao Huang, Xiang Zhang, Xiaozhong Xu, Jun Tian, Shan Liu, “Arithmetic Coding of Displacements for Subdivision-based Mesh Compression”, ISO / IEC JTC 1 / SC 29 / WG 7 m60300, July 2022 Non-Patent Literature 3: Jianfeng Xu, Kei Kawamura, “[V-DMC] [new] On supporting a SKIP type in base mesh coding”, ISO / IEC JTC 1 / SC 29 / WG 7 m62092, January 2023 SUMMARY TECHNICAL PROBLEM

[0005] However, application of these methods does not always result in optimal coding efficiency, and other methods have been sought.

[0006] The present disclosure has been made in view of the above, and aims to suppress a decrease in coding efficiency. SOLUTION TO PROBLEM

[0007] The information processing apparatus according to an aspect of the present technology is an information processing apparatus including: a motion vector decoding unit that decodes coded data and generates a motion vector indicating a positional difference between vertices of a reference base mesh and vertices of a current base mesh; a current base mesh reconstructing unit that reconstructs the current base mesh by applying the motion vector to the vertices of the reference base mesh; and a current mesh reconstructing unit that reconstructs a current mesh by applying a reference displacement vector to the vertices of the current base mesh, wherein the reference base mesh is a base mesh of a reference frame referred to by a current frame, the current base mesh is a base mesh of the current frame, a base mesh is a mesh having a lower degree of fineness than an original mesh to be coded, the original mesh includes vertices and connections representing a three-dimensional structure of an object, the base mesh is generated by removing vertices from the middle of the original mesh, and the reference displacement vector indicates a positional difference between vertices obtained by subdividing the reference base mesh and vertices of a reference mesh.

[0008] The information processing method according to an aspect of the present technology is an information processing method including: decoding coded data and generating a motion vector indicating a positional difference between vertices of a reference base mesh and vertices of a current base mesh; reconstructing the current base mesh by applying the motion vector to the vertices of the reference base mesh; and reconstructing a current mesh by applying a reference displacement vector to the vertices of the current base mesh, wherein the reference base mesh is a base mesh of a reference frame referred to by a current frame, the current base mesh is a base mesh of the current frame, a base mesh is a mesh having a lower degree of fineness than an original mesh to be coded, the original mesh includes vertices and connections representing a three-dimensional structure of an object, the base mesh is generated by removing vertices from the middle of the original mesh, and the reference displacement vector indicates a positional difference between vertices obtained by subdividing the reference base mesh and vertices of a reference mesh.

[0009] An information processing apparatus according to another aspect of the present technology is an information processing apparatus including: an encoding unit that encodes a motion vector that indicates a positional difference between a vertex of a reference base mesh and a vertex of a current base mesh; and a control information encoding unit that encodes control information related to application of a reference displacement vector to the vertex of the current base mesh at the time of reconstruction of a current mesh, wherein the reference base mesh is a base mesh of a reference frame that is referred to by a current frame, the current base mesh is a base mesh of the current frame, a base mesh is a mesh having a lower degree of fineness than an original mesh to be encoded, the original mesh includes a vertex and a connection that represent a three-dimensional structure of an object, the base mesh is generated by removing a vertex from the middle of the original mesh, and the reference displacement vector indicates a positional difference between a vertex obtained by subdividing the reference base mesh and a vertex of a reference mesh.

[0010] An information processing method according to another aspect of the present technology is an information processing method including: encoding a motion vector that indicates a positional difference between a vertex of a reference base mesh and a vertex of a current base mesh; and encoding control information related to application of a reference displacement vector to the vertex of the current base mesh at the time of reconstruction of a current mesh, wherein the reference base mesh is a base mesh of a reference frame that is referred to by a current frame, the current base mesh is a base mesh of the current frame, a base mesh is a mesh having a lower degree of fineness than an original mesh to be encoded, the original mesh includes a vertex and a connection that represent a three-dimensional structure of an object, the base mesh is generated by removing a vertex from the middle of the original mesh, and the reference displacement vector indicates a positional difference between a vertex obtained by subdividing the reference base mesh and a vertex of a reference mesh.

[0011] An information processing apparatus according to still another aspect of the present technology is an information processing apparatus including: a current displacement vector decoding unit that decodes encoded data and generates a current displacement vector that indicates a positional difference between a vertex of a current mesh and a vertex of a reference mesh; and a displacement vector application unit that reconstructs the current mesh by applying the current displacement vector to the vertex of the reference mesh, wherein the current mesh is a mesh that includes a vertex and a connection that represent a three-dimensional structure of an object in a current frame, and the reference mesh is a mesh that includes a vertex and a connection that represent a three-dimensional structure of an object in a reference frame that is referred to by the current frame.

[0012] The information processing method according to a further aspect of the present technology is an information processing method including: decoding encoded data and generating a current displacement vector indicating a positional difference between a vertex of a current mesh and a vertex of a reference mesh; and reconstructing the current mesh by applying the current displacement vector to the vertex of the reference mesh, wherein the current mesh is a mesh including vertices and connections representing a three-dimensional structure of an object in a current frame, and the reference mesh is a mesh including vertices and connections representing a three-dimensional structure of the object in a reference frame referred to by the current frame.

[0013] The information processing apparatus according to a further aspect of the present technology is an information processing apparatus including: a current displacement vector derivation that derives a current displacement vector indicating a positional difference between a vertex of a current mesh and a vertex of a reference mesh; and a current displacement vector encoding unit that encodes the current displacement vector, wherein the current mesh is a mesh including vertices and connections representing a three-dimensional structure of an object in a current frame, and the reference mesh is a mesh including vertices and connections representing a three-dimensional structure of the object in a reference frame referred to by the current frame.

[0014] The information processing method according to a further aspect of the present technology is an information processing method including: deriving a current displacement vector indicating a positional difference between a vertex of a current mesh and a vertex of a reference mesh; and encoding the current displacement vector, wherein the current mesh is a mesh including vertices and connections representing a three-dimensional structure of an object in a current frame, and the reference mesh is a mesh including vertices and connections representing a three-dimensional structure of the object in a reference frame referred to by the current frame.

[0015] The information processing apparatus according to a further aspect of the present technology is an information processing apparatus including: an up-conversion unit that up-converts a current base mesh that is a base mesh of a current frame; a current displacement vector decoding unit that decodes encoded data and generates a current displacement vector indicating a positional difference between a vertex of a current mesh and a vertex of the up-converted current base mesh; and a current mesh reconstruction unit that reconstructs the current mesh by applying the current displacement vector to the vertex of the up-converted current base mesh, wherein the base mesh is a mesh having lower fineness than an original mesh to be encoded, the original mesh including vertices and connections representing a three-dimensional structure of an object, and the base mesh is generated by removing vertices from the middle of the original mesh.

[0016] The information processing method according to still another aspect of the present technology is an information processing method including: up-converting a current base mesh that is a base mesh of a current frame; decoding encoding data and generating a current displacement vector that indicates a positional difference between a vertex of a current mesh and a vertex of the up-converted current base mesh; and reconstructing the current mesh by applying the current displacement vector to the vertex of the up-converted current base mesh, wherein the base mesh is a mesh having a lower degree of fineness than an original mesh to be encoded, the original mesh including a vertex and a connection that represent a three-dimensional structure of an object, and the base mesh is generated by removing the vertex from the middle of the original mesh.

[0017] The information processing apparatus according to still another aspect of the present technology is an information processing apparatus including: an up-conversion unit that up-converts a current base mesh that is a base mesh of a current frame; a current displacement vector derivation unit that derives a current displacement vector that indicates a positional difference between a vertex of a current mesh and a vertex of the up-converted current base mesh; and a current displacement vector encoding unit that encodes the current displacement vector, wherein the base mesh is a mesh having a lower degree of fineness than an original mesh to be encoded, the original mesh including a vertex and a connection that represent a three-dimensional structure of an object, and the base mesh is generated by removing the vertex from the middle of the original mesh.

[0018] The information processing method according to still another aspect of the present technology is an information processing method including: up-converting a current base mesh that is a base mesh of a current frame; deriving a current displacement vector that indicates a positional difference between a vertex of a current mesh and a vertex of the up-converted current base mesh; and encoding the current displacement vector, wherein the base mesh is a mesh having a lower degree of fineness than an original mesh to be encoded, the original mesh including a vertex and a connection that represent a three-dimensional structure of an object, and the base mesh is generated by removing the vertex from the middle of the original mesh.

[0019] In the information processing apparatus and method according to an aspect of the present technology, encoding data is decoded, a motion vector that indicates a positional difference between a vertex of a reference base mesh and a vertex of a current base mesh is generated, the current base mesh is reconstructed by applying the motion vector to the vertex of the reference base mesh, and a current mesh is reconstructed by applying a reference displacement vector to the vertex of the current base mesh.

[0020] In the information processing apparatus and method according to another aspect of the present technology, a motion vector that indicates a positional difference between a vertex of a reference base mesh and a vertex of a current base mesh is encoded, and control information related to application of a reference displacement vector to the vertex of the current base mesh at the time of reconstruction of a current mesh is encoded.

[0021] In the information processing apparatus and method according to yet another aspect of the present technology, the encoded data is decoded, a current displacement vector indicating a positional difference between a vertex of a current mesh and a vertex of a reference mesh is generated, and the current mesh is reconstructed by applying the current displacement vector to the vertex of the reference mesh.

[0022] In the information processing apparatus and method according to yet another aspect of the present technology, a current displacement vector indicating a positional difference between a vertex of a current mesh and a vertex of a reference mesh is derived, and the current displacement vector is encoded.

[0023] In the information processing apparatus and method according to yet another aspect of the present technology, an up-conversion is performed on a current base mesh that is a base mesh of a current frame, the encoded data is decoded, and a current displacement vector indicating a positional difference between a vertex of a current mesh and a vertex of the up-converted current base mesh is generated, and the current mesh is reconstructed by applying the current displacement vector to the vertex of the up-converted current base mesh.

[0024] In the information processing apparatus and method according to yet another aspect of the present technology, an up-conversion is performed on a current base mesh that is a base mesh of a current frame, a current displacement vector indicating a positional difference between a vertex of a current mesh and a vertex of the up-converted current base mesh is derived, and the current displacement vector is encoded. BRIEF DESCRIPTION OF DRAWINGS

[0025] [ Figure 1 ] Figure 1 is a diagram for explaining a mesh. [ Figure 2 ] Figure 2 is a diagram for explaining V-DMC. [ Figure 3 ] Figure 3 is a diagram showing an example of a state of reconstruction. [ Figure 4 ] Figure 4 is a diagram showing an example of a state of reconstruction. [ Figure 5 ] Figure 5 is a diagram showing an example of a mesh encoding and decoding method. [ Figure 6 ] Figure 6 is a diagram showing an example of a state of reconstruction. [ Figure 7 ] Figure 7 is a diagram showing an example of a state of reconstruction. [ Figure 8 ] Figure 8 is a diagram for explaining a key frame. [ Figure 9 ] Figure 9FIG. 1 is a block diagram showing a main configuration example of an encoding apparatus. Figure 10 Figure 10 FIG. 2 is a flowchart showing an example of the flow of an interframe encoding process. Figure 11 Figure 11 FIG. 1 is a block diagram showing a main configuration example of an encoding apparatus. Figure 12 Figure 12 FIG. 2 is a flowchart showing an example of the flow of an interframe encoding process. Figure 13 Figure 13 FIG. 3 is a diagram showing an example of a state of reconstruction. Figure 14 Figure 14 FIG. 4 is a diagram showing an example of a state of reconstruction. Figure 15 Figure 15 FIG. 1 is a block diagram showing a main configuration example of an encoding apparatus. Figure 16 Figure 16 FIG. 2 is a flowchart showing an example of the flow of an interframe encoding process. Figure 17 Figure 17 FIG. 1 is a block diagram showing a main configuration example of an encoding apparatus. Figure 18 Figure 18 FIG. 2 is a flowchart showing an example of the flow of an interframe encoding process. Figure 19 Figure 19 FIG. 3 is a diagram showing an example of a state of reconstruction. Figure 20 Figure 20 FIG. 4 is a diagram showing an example of a state of reconstruction. Figure 21 Figure 21 FIG. 1 is a block diagram showing a main configuration example of an encoding apparatus. Figure 22 Figure 22 FIG. 2 is a flowchart showing an example of the flow of an interframe encoding process. Figure 23 Figure 23 FIG. 1 is a block diagram showing a main configuration example of an encoding apparatus. Figure 24 Figure 24 FIG. 2 is a flowchart showing an example of the flow of an interframe encoding process. Figure 25 Figure 25 FIG. 3 is a diagram showing an example of a state of reconstruction. Figure 26 Figure 26 ​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​FIG. 1 is a block diagram showing a main configuration example of an encoding apparatus. [ Figure 27 ] Figure 27 FIG. 4 is a flowchart illustrating an example of the flow of an intra encoding process. [ Figure 28 ] Figure 28 FIG. 5 is a block diagram showing a main configuration example of a decoding apparatus. [ Figure 29 ] Figure 29 FIG. 6 is a flowchart illustrating an example of the flow of an intra decoding process. [ Figure 30 ] Figure 30 FIG. 7 is a block diagram showing a main configuration example of an encoding apparatus. [ Figure 31 ] Figure 31 FIG. 8 is a flowchart illustrating an example of the flow of an inter encoding process. [ Figure 32 ] Figure 32 FIG. 9 is a block diagram showing a main configuration example of a decoding apparatus. [ Figure 33 ] Figure 33 FIG. 10 is a flowchart illustrating an example of the flow of an inter decoding process. [ Figure 34 ] Figure 34 FIG. 11 is a block diagram showing a main configuration example of an encoding apparatus. [ Figure 35 ] Figure 35 FIG. 12 is a flowchart illustrating an example of an encoding process. [ Figure 36 ] Figure 36 FIG. 13 is a block diagram showing a main configuration example of a computer. DETAILED DESCRIPTION

[0026] Hereinafter, modes for implementing the present disclosure (hereinafter referred to as embodiments) will be described. The description will be given in the following order. 1. Literature and other information supporting technical contents and terms 2. Encoding of current displacement vectors 3. Reducing current displacement vector transmission using reference displacement vectors 4. Reducing the amount of displacement vector information using up-conversion 5. Mode selection 6. Supplementary explanation

[0027] 1. Literature and other information supporting technical contents and terms The scope disclosed in the present technology includes not only the contents described in the embodiments, but also the contents described in the following non-patent literature and the like known at the time of filing, and the contents of other documents cited in the following non-patent literature and the like.

[0028] Non-patent literature 1: (described above) Non-patent literature 2: (described above) Non-patent literature 3: (described above)

[0029] That is, the content described in the aforementioned non-patent documents, as well as the content of other documents cited in the aforementioned non-patent documents, are also used as the basis for determining the supporting claims.

[0030] 2. Encoding the current displacement vector V-DMC Meshes are known as 3D data representing the three-dimensional structure of a solid structure (an object with a three-dimensional shape). Meshes represent the three-dimensional shape of an object's surface by forming polygons using vertices and connections.

[0031] As in Figure 1 As shown in the upper left corner, in the mesh, vertices 11 and connections 12 that connect vertices 11 form polygonal planes (polygons). These polygons (also called faces) represent the surfaces of objects with a three-dimensional structure, i.e., the three-dimensional shape of the object. Note that texture 13 can be attached to each face of the mesh.

[0032] Grid data includes information, such as, for example, in Figure 1 As shown in the lower part. Figure 1 The first vertex information 14 shown from the left in the lower part is information indicating the three-dimensional position (three-dimensional coordinates (X, Y, Z)) of each vertex in the vertices 11 contained in the mesh. Figure 1 The second connection information shown from the left in the lower part, 15, indicates information about each connection in the connection 12 contained in the grid. Figure 1 The third texture image 16 shown from the left in the lower part is the mapping information of the texture 13 attached to each face in the face. Figure 1 The UV map 17 shown fourth from the left in the lower part is information indicating the correspondence between vertex 11 and texture 13. In UV map 17, the coordinates (UV coordinates) of each vertex of vertex 11 in texture image 16 are shown.

[0033] As an encoding method for such grids, there exists, for example, video-based dynamic grid coding (V-DMC) as described in Non-Patent Document 1.

[0034] In V-DMC, the mesh to be encoded is represented by a coarse base mesh and displacement vectors of the subdivision points obtained by subdividing the base mesh, and the base mesh and displacement vectors are encoded.

[0035] For example, suppose there exists such as Figure 2The original mesh is shown in the top row. In this specification, the original mesh refers to the mesh to be encoded. Figure 2 In this diagram, black dots represent vertices, and lines connecting the black dots represent connections. As described above, a mesh is essentially a polygon formed by vertices and connections, but for convenience, the mesh is described here as a set of linearly connected (continuous) vertices.

[0036] By removing some vertices from the original mesh, the number of polygons was reduced, resulting in a shape resembling that from... Figure 2 The second row from the top shows a coarse mesh. In this specification, such a coarse mesh, obtained by removing some vertex connections from the original mesh, is called the base mesh. That is, the base mesh can be considered as a mesh with fewer polygons than the original mesh.

[0037] By subdividing the base grid, a subdivided base grid was obtained, such as from... Figure 2 As shown in the third line from the top. In this specification, subdivision is the process of dividing the polygons of a mesh into multiple polygons. That is, subdividing the base mesh increases the number of polygons in the base mesh. In other words, subdividing the base mesh adds vertices and connections to the base mesh. For example, by subdividing the base mesh and adding the same number of vertices as were removed from the original mesh when generating the base mesh, a mesh with the same number of vertices as the original mesh can be obtained. In this specification, these added vertices are also referred to as subdivision points.

[0038] However, during the generation of the base mesh, i.e., when vertices are removed from the middle of the original mesh, the connections are updated. Therefore, split points are formed on these updated connections. Consequently, the shape of the subdivided base mesh may not match the shape of the original mesh. More specifically, as... Figure 2 As shown in the bottom row, the positions of the split points (on the dashed lines) may not match the positions of the vertices of the original mesh. In this specification, the difference between the positions of the vertices (split points) of the subdivided base mesh and the positions of the vertices of the original mesh is called a displacement vector. That is, the displacement vector indicates the difference between the positions of the vertices of the subdivided base mesh and the positions of the corresponding vertices of the original mesh. In other words, the original mesh is obtained by updating the position of each vertex in the subdivided base mesh according to these displacement vectors.

[0039] That is, by using such a conversion process, ideally, the original mesh can be obtained at the transmission destination by transmitting the base mesh and the displacement vector. In other words, the base mesh and the displacement vector can be transmitted instead of the original mesh. In data transmission, data is generally encoded and transmitted, and then the encoded data is decoded at the transmission destination to generate (restore) the original data. In the encoder (source device), the original mesh is known, so the encoder can generate the base mesh and further derive such a displacement vector. That is, the encoder can encode the base mesh and the displacement vector. The decoder (source device) can decode the transmitted encoded data to generate (restore) the base mesh and the displacement vector. The decoder can then generate (restore) the original mesh by subdividing the base mesh and applying the displacement vector to the corresponding vertex. Note that in this specification, the original mesh generated in the decoder is also referred to as the decoded original mesh. In contrast, the original mesh before encoding by the encoder is also referred to as the pre-encoding original mesh. The decoded original mesh is a mesh that represents the three-dimensional structure of the object with the three-dimensional structure represented by the pre-encoding original mesh, and can be said to be substantially the same as the pre-encoding original mesh. However, in reality, the decoded original mesh is not necessarily exactly the same as the pre-encoding original mesh in terms of information, because the configuration of the polygon and the like can change.

[0040] As described above, the base mesh is coarser than the original mesh, so the amount of data thereof is smaller than that of the original mesh. In addition, while the vertex positions are represented by absolute values in the original mesh, the displacement vector is differential information between the vertex positions, so the amount of data thereof is smaller than that of the original mesh. Furthermore, the displacement vector can be stored in a two-dimensional image and encoded using 2D encoding, thereby improving the encoding efficiency. Therefore, the amount of code required to encode the base mesh and the displacement vector is smaller than that required to encode the original mesh. That is, by using the conversion process described above to encode and decode the base mesh and the displacement vector, the encoding efficiency can be improved compared to encoding and decoding the original mesh. In other words, by using the conversion process described above to transmit the base mesh and the displacement vector, the amount of data transmitted can be reduced and the transmission efficiency can be improved compared to transmitting the original mesh. Note that in this specification, storing data in a two-dimensional image is also referred to as packing.

[0041] Note that, in the present specification, being variable in the time direction is also referred to as "dynamic". Being constant in the time direction or having no concept of the time direction is also referred to as "static". For example, a mesh that represents a three-dimensional shape of an object at one time is referred to as static. When a sequence includes a plurality of meshes that represent three-dimensional shapes of an object at different times, the meshes are referred to as dynamic in the sequence. That is, a dynamic mesh includes static meshes at a plurality of times. In the present specification, information at each time is also referred to as a frame. That is, a dynamic mesh includes meshes of a plurality of frames.

[0042] In the present specification, a three-dimensional shape of an object is dynamic, and a mesh that is 3D data of the object is also dynamic. That is, the target to be encoded is a dynamic mesh. In other words, the base mesh and the displacement vector are also dynamic, and the displacement vector is encoded as a moving image that includes two-dimensional images as frames. In the present specification, the moving image is also referred to as a displacement video.

[0043] Note that the displacement vector can be packed into a two-dimensional image as a transform coefficient using a coefficient transform such as a wavelet transform. The displacement vector can be quantized. For example, the displacement vector can be transformed into a transform coefficient by a wavelet transform, the transform coefficient can be quantized, and the quantized transform coefficient (quantized coefficient) can be packed.

[0044] Redundancy in displacement vector encoding As a method for encoding a base mesh, there is intra-frame encoding that independently encodes each frame and inter-frame encoding that encodes using correlation between frames. In general, inter-frame encoding has higher encoding efficiency than intra-frame encoding.

[0045] In contrast, the displacement vector (displacement video) is transmitted per frame regardless of whether the base mesh is intra-frame encoded or inter-frame encoded.

[0046] For example, described is an example of transmission of a current frame and a reference frame, as Figure 3The reference base mesh 51 shown at the upper side of FIG. 6 is subdivided to generate a subdivided reference base mesh 52. Then, a reference displacement vector (a dotted arrow) is applied to each of the vertices of the subdivided reference base mesh 53 (the same mesh as the subdivided reference base mesh 52) to reconstruct a reference mesh 54. Figure 4 The reference base mesh 51 shown at the upper side of FIG. 6 is subdivided to generate a subdivided reference base mesh 52. Then, a reference displacement vector (a dotted arrow) is applied to each of the vertices of the subdivided reference base mesh 53 (the same mesh as the subdivided reference base mesh 52) to reconstruct a reference mesh 54.

[0047] Suppose Figure 3 The base mesh of the current frame shown in FIG. 5 is interframe-encoded. In this case, a motion vector and a displacement vector are each encoded and transmitted. Note that, in the present specification, the base mesh of the current frame is also referred to as a current base mesh. The displacement vector of the current frame is also referred to as a current displacement vector. The motion vector is information indicating a difference in position between a vertex of a reference base mesh and a vertex of the current base mesh. At the transmission destination, the encoded data is decoded to generate (restore) the motion vector and the current displacement vector. Then, the motion vector is applied to the reference base mesh to reconstruct the current base mesh. The current base mesh is then subdivided. Then, the subdivided current base mesh and the current displacement vector are used to reconstruct a mesh. In the present specification, the mesh of the current frame reconstructed using the current base mesh and the current displacement vector is also referred to as a current mesh. That is, the current displacement vector indicates a difference between a position of a vertex of the subdivided current base mesh and a position of a corresponding vertex of the current mesh. For example, the motion vector (a dotted arrow) is applied to the reference base mesh 61 shown at the lower side of FIG. 5 to reconstruct a current base mesh 62. Figure 4 The reference base mesh 51 shown at the upper side of FIG. 6 is subdivided to generate a subdivided reference base mesh 52. Then, a reference displacement vector (a dotted arrow) is applied to each of the vertices of the subdivided reference base mesh 53 (the same mesh as the subdivided reference base mesh 52) to reconstruct a reference mesh 54.

[0048] Thus, the displacement vector is transmitted and used to reconstruct a mesh on a frame-by-frame basis. However, as Figure 4As shown, the displacement vector is the difference between the vertex positions of the subdivided base mesh and the original mesh, and is therefore a parameter representing the detail of the mesh (3D model). Thus, the displacement vector does not change when the shape of the 3D model's details remains unchanged, and the transmission of the displacement vector in each frame may be redundant.

[0049] For example, when a person moves their entire arm, the shape of the arm's details (skin, clothing, etc.) may not change in some cases (the position of the entire arm changes, but the relative positions of each vertex forming the arm remain the same). In such cases, the displacement vector does not change or changes very little in the time direction. That is, there is a risk that transmitting the displacement vector will reduce coding efficiency.

[0050] In the methods described in Non-Patent Document 1 and Non-Patent Document 2, it is difficult to suppress the reduction in coding efficiency under such circumstances because a displacement vector is transmitted. Therefore, the various methods proposed to date do not always produce optimal coding efficiency, and other methods have been sought.

[0051] 3. Use a reference displacement vector to reduce the current displacement vector transmission. Method 1 Therefore, as Figure 5 As shown in the top row of the table, the current mesh is reconstructed using a reference displacement vector (Method 1). That is, as referenced... Figure 3 As explained, in the case of existing technology, the current displacement vector is applied to the reconstruction of the current mesh, but the reference displacement vector of the reference frame is applied instead of the current displacement vector.

[0052] When the shape of the details has no (or very small) changes, the difference between the reference displacement vector and the current displacement vector can be estimated to be very small. Therefore, by applying the transmitted reference displacement vector instead of the current displacement vector, the transmission of the current displacement vector can be skipped (omitted). That is, the amount of displacement vector to be transmitted can be reduced. Therefore, the reduction in coding efficiency can be suppressed.

[0053] Method 1-1 When applying method 1, for example, as Figure 5 As shown in the second row from the top of the table, a reference displacement vector can be applied to the current base mesh (Method 1-1). For example, as... Figure 6 As shown, in the current frame, the current base mesh is reconstructed by applying motion vectors to the vertices of the reference base mesh. The current base mesh is then subdivided. Then, instead of the current displacement vector, the reference displacement vector from the reference frame is applied to the subdivided current base mesh to reconstruct the current mesh. For example, the motion vector (dashed arrow) is applied to... Figure 7The vertices of the reference base mesh 111 shown in FIG. 11 are used to reconstruct the current base mesh 112. Then, the current base mesh 112 is subdivided to generate the subdivided current base mesh 113. Then, the reference displacement vector (dashed arrow) is applied to each of the vertices (each of the subdivision points) of the subdivided current base mesh 114 (the same mesh as the current base mesh 113) to reconstruct the current mesh 115.

[0054] In this way, the transmission of the current displacement vector for reconstructing the current mesh can be skipped (omitted). That is, the amount of transmission for the displacement vector can be reduced. Therefore, the reduction in coding efficiency can be suppressed.

[0055] Note that the reference displacement vector can be applied to all of the vertices (all of the subdivision points) of the current base mesh. For example, in the case of the subdivided current base mesh 114 in FIG. 11, the reference displacement vector (dashed arrow) is applied to all of the subdivision points. Alternatively, the reference displacement vector can be applied to some of the vertices (some of the subdivision points) of the current base mesh. That is, in this case, the current displacement vector is applied to the remaining vertices (remaining subdivision points). For example, in the case of the subdivided current base mesh 116 (the same mesh as the current base mesh 113) in FIG. 11, the reference displacement vector (dashed arrow) is applied to some of the subdivision points, and the current displacement vector (solid arrow) is applied to the remaining subdivision points. For example, by applying the current displacement vector to the vertices at which the displacement vector greatly changes in the time direction and applying the reference displacement vector to the vertices at which the displacement vector little changes in the spatial direction, flexible application can be achieved. That is, the current displacement vector can be applied as needed. In this way, comprehensive control, such as suppressing the reduction in coding efficiency while suppressing the reduction in the subjective quality of the mesh, can be achieved. Figure 7 Figure 7 In the reconstruction of the current mesh, when and where the reference displacement vector is applied to the vertices of the current base mesh can be determined in any manner. For example, the application of the reference displacement vector can be determined in advance. The encoder can control the application of the reference displacement vector by providing control information to the decoder using a bitstream. That is, the decoder can apply the reference displacement vector when reconstructing the current mesh according to the control information.

[0056] Such control information related to the application of the reference displacement vector to the vertices of the current base mesh can include any content.

[0057]

[0058] ​​For example, the control information can include a parameter that specifies a target to be reused, a range to be reused, and the like (i.e., information that indicates where to apply the reference displacement vector). That is, the encoder can provide the decoder with control information that includes such a parameter. The decoder can apply the reference displacement vector to the partition points of the current base mesh that are specified as the target by the parameter, or to the partition points of the current base mesh within the range specified by the parameter.

[0059] For example, the parameter can be flag information on a sequence-by-sequence basis. For example, when the value of the flag information is true (e.g., 1), the reference displacement vector can be applied to the vertices of the current base mesh in all inter frames (frames to which inter-frame encoding is applied) in the sequence. An enabling flag can be provided. That is, the encoder can provide the decoder with control information that includes such flag information. The decoder can determine whether to apply the reference displacement vector in accordance with the flag information.

[0060] The flag information can be frame-by-frame information. For example, the flag information can be provided on a frame-by-frame basis, and the decoder can apply the reference displacement vector to the vertices of the current base mesh in frames in which the value of the flag information is true (e.g., 1). A list of identification information of frames (frame ID list) to which the reference displacement vector is applied to the vertices of the current base mesh can be provided on a sequence-by-sequence basis.

[0061] The control can be performed on a sub-region-by-sub-region basis in frames. For example, the control information can include a parameter that specifies a sub-region to which the reference displacement vector is applied to the vertices of the current base mesh. For example, the sub-region to which the reference displacement vector is applied to the vertices of the current base mesh can be specified on a frame-by-frame basis. That is, the encoder can provide the decoder with control information that includes such a parameter. The decoder can apply the reference displacement vector to the partition points of the current base mesh in the sub-region specified by the parameter. Note that the control can be performed on any information unit (sub-region). For example, the control can be performed per sub-mesh, per face, per level of detail (LoD), or a combination of controls on multiple information units can be applied. In either case, the relevant position can be indicated by using identification information for the information unit or the like. Note that the target or range can be specified using a conditional expression or the like.

[0062] The control information can include a parameter that specifies a reference target. That is, the control information can specify which frames are key frames (frames that can be reference frames) for the displacement vector. That is, the encoder can provide the decoder with control information that includes such a parameter. The decoder can use the key frames specified by the parameter as reference frames. In other words, the frames to which the displacement vector is to be transmitted can be specified.

[0063] Any method can be used to specify the key frames (intra frames) for the displacement vectors. For example, the control information can include a list of the key frames (intra frames) for the displacement vectors. When a frame other than the key frames is the current frame, the displacement vector of the key frame immediately before the current frame can be used as a reference displacement vector for mesh reconstruction.

[0064] The reference frames can be specified on a frame-by-frame basis. For example, the control information can include a list of the identification information of the reference frames on a frame-by-frame basis. In this case, the reference frame for each frame can be freely configured. That is, the encoder can provide the decoder with the control information including the list. The decoder can use the frames specified by the list as the reference frames.

[0065] In this way, the key frames for the displacement vectors can be configured independently of the key frames for the base mesh. For example, in Figure 8 , the horizontal axis represents the sequence (frame direction), the vertical lines represent the frames, and the thick lines represent the key frames. As Figure 8 indicated, frames other than the key frames for the base mesh can also be used as the key frames for the displacement vectors.

[0066] The control information can include information that specifies the key frames for the displacement vectors at a ratio to the interval of the key frames for the base mesh. For example, the displacement vectors can be transmitted at an interval twice that of the key frames for the base mesh.

[0067] Note that, needless to say, the reference frames for the displacement vectors can coincide with the reference frames for the base mesh. In this case, these parameters can be omitted.

[0068] Encoding apparatus Figure 9 is a block diagram showing an example of the configuration of an encoding apparatus, which is an aspect of the information processing apparatus to which the present technology (the method 1-1 described above) is applied. Figure 9 The encoding apparatus 200 shown in

[0069] Note that, Figure 9 The main processing units, data flows, and the like are shown, but all the processing units, data flows, and the like are not necessarily shown. That is, in the encoding apparatus 200, there can be processing units that are not shown as blocks in Figure 9 , or there can be processing or data flows that are not shown as arrows and the like in Figure 9 .

[0070] As Figure 9As shown in FIG. 2, the encoding apparatus 200 includes a control unit 201, a transmission data generating unit 202, and a bitstream generating unit 203. The bitstream generating unit 203 includes a motion vector encoding unit 211, a displacement video encoding unit 212, a control information encoding unit 213, and a multiplexer 214.

[0071] The control unit 201 controls the processing units (each of the processing units in the transmission data generating unit 202 and the bitstream generating unit 203) in the encoding apparatus 200, and causes the decoder to perform a process for applying a reference displacement vector to a current base mesh. For example, the control unit 201 controls the processing units so that a current displacement vector is not encoded (or only some of the current displacement vectors are limited) in frames other than a key frame for the displacement vector. For example, the control unit 201 can control the transmission data generating unit 202 not to generate a displacement vector (or to generate only some of the displacement vectors). The other processing units operate under the control of the control unit 201. The control unit 201 can generate control information related to application of a reference displacement vector to a vertex of a current base mesh. The control information is transmitted to the decoder and used by the decoder. Therefore, the control unit 201 can provide the generated control information to the control information encoding unit 213.

[0072] The transmission data generating unit 202 converts the provided mesh into a format for transmission. For example, the transmission data generating unit 202 can generate a current base mesh from a provided current mesh, and provide the current base mesh to the motion vector encoding unit 211. The transmission data generating unit 202 can skip (omit) generation of a current displacement vector under the control of the control unit 201. The transmission data generating unit 202 can generate a current displacement vector for a part of the mesh under the control of the control unit 201, and provide the current displacement vector to the displacement video encoding unit 212.

[0073] The bitstream generating unit 203 performs a process related to bitstream generation. The motion vector encoding unit 211 obtains a reference base mesh and a current base mesh, and generates a motion vector using the reference base mesh and the current base mesh. That is, the motion vector encoding unit 211 sets a position difference between vertices of the reference base mesh and the current base mesh as a motion vector. The motion vector encoding unit 211 encodes the motion vector, and provides the encoded data to the multiplexer 214.

[0074] The displacement video encoding unit 212 packs some of the current displacement vectors provided from the transmission data generating unit 202 into a displacement video frame, and encodes the current displacement vectors as a displacement video. The displacement video encoding unit 212 provides the encoded data to the multiplexer 214.

[0075] The control information encoding unit 213 obtains the control information supplied from the control unit 201, and encodes the control information, and supplies the encoded data to the multiplexer 214.

[0076] The multiplexer 214 obtains the encoded data of the motion vector supplied from the motion vector encoding unit 211, the encoded data of the displacement video supplied from the displacement video encoding unit 212, and the encoded data of the control information supplied from the control information encoding unit 213, and multiplexes these encoded data to generate a bitstream. The multiplexer 214 outputs the generated bitstream to the outside of the encoding apparatus 200. The bitstream is transmitted to a decoder (the decoding apparatus 250 described below) via any storage medium, communication medium, or the like.

[0077] The present technology (the method 1-1 described above) can be applied to the encoding apparatus 200 having such a configuration. For example, the motion vector encoding unit 211 of the encoding apparatus 200 can encode the motion vector indicating the position difference between the vertices of the reference base mesh and the vertices of the current base mesh. The control information encoding unit 213 can encode the control information related to the application of the reference displacement vector to the vertices of the current base mesh at the time of reconstructing the current mesh.

[0078] With such a configuration, the encoding apparatus 200 can control the application of the reference displacement vector to the vertices of the current base mesh at the time of the reconstruction of the current mesh by the decoder. Therefore, the encoding apparatus 200 can suppress the reduction of the encoding efficiency.

[0079] Note that, as described above, the encoding apparatus 200 can include the displacement video encoding unit 212. The displacement video encoding unit 212 can encode the current displacement vector indicating the position difference between some vertices of the current mesh and the corresponding vertices of the current base mesh. That is, the displacement video encoding unit 212 can also be referred to as a current displacement vector encoding unit.

[0080] Flow of inter-frame encoding process Figure 9 The encoding apparatus 200 in the above-described Figure 10 An example of the flow of the inter-frame encoding process will be described with reference to the flowchart in

[0081] When the inter-frame encoding process starts, in step S201, the transmission data generating unit 202 generates the current transmission data (for example, the current base mesh) using the current mesh.

[0082] In step S202, the motion vector encoding unit 211 generates the motion vector using the reference base mesh and the current base mesh, and encodes the motion vector.

[0083] In step S203, the control unit 201 determines whether or not to transmit some of the current displacement vectors of the current mesh. If it is determined to transmit the current displacement vectors, the processing proceeds to step S204.

[0084] In step S204, the displacement video encoding unit 212 generates a displacement video using some of the current displacement vectors of the current mesh, and encodes the displacement video. When the processing in step S204 is completed, the processing proceeds to step S205. If it is determined in step S203 not to transmit the current displacement vectors, the processing in step S204 is skipped (omitted), and the processing proceeds to step S205.

[0085] In step S205, the control unit 201 generates control information related to application of the reference displacement vectors to the vertices of the current base mesh at the time of reconstruction of the current mesh. In step S206, the control information encoding unit 213 encodes the control information.

[0086] In step S207, the multiplexer 214 multiplexes the encoded data of the motion vectors and the encoded data of the control information (and the encoded data of the displacement video, if the displacement video (some of the displacement vectors) is encoded), and generates a bitstream. When the processing in step S207 is completed, the inter-frame encoding processing ends.

[0087] The present technology (the above-described method 1-1) can be applied to such inter-frame encoding processing. For example, in step S202, the motion vector encoding unit 211 can encode the motion vectors indicating the position differences between the vertices of the reference base mesh and the vertices of the current base mesh. In step S206, the control information encoding unit 213 can encode the control information related to application of the reference displacement vectors to the vertices of the current base mesh at the time of reconstruction of the current mesh.

[0088] By performing the inter-frame encoding processing in this way, the encoding apparatus 200 can control the application of the reference displacement vectors to the vertices of the current base mesh at the time of reconstruction of the current mesh by the decoder. Therefore, the encoding apparatus 200 can suppress a decrease in encoding efficiency.

[0089] Decoding Apparatus Figure 11 is a block diagram illustrating an example of a configuration of a decoding apparatus that is an aspect of the information processing apparatus to which the present technology (the above-described method 1) is applied. Figure 11 The decoding apparatus 250 illustrated in FIG. 12 is an apparatus that inter-frame decodes encoded data (a bitstream) to generate (restore) a dynamic mesh. The decoding apparatus 250 obtains a bitstream and decodes the bitstream to generate a dynamic mesh, and outputs the generated dynamic mesh.

[0090] Note that, Figure 11The main processing units, data flows, and the like are shown, but all the processing units, data flows, and the like are not necessarily shown. That is, in the decoding apparatus 250, there can be processing units shown as blocks in Figure 11 , or there can be processing or data flows shown as arrows and the like in Figure 11 , which are not shown.

[0091] As shown in Figure 11 , the decoding apparatus 250 includes a demultiplexer 251, a decoding unit 252, a control unit 253, and a reconstruction unit 254. The decoding unit 252 includes a control information decoding unit 261, a motion vector decoding unit 262, and a displacement video decoding unit 263. The reconstruction unit 254 includes a current base mesh reconstruction unit 271 and a current mesh reconstruction unit 272.

[0092] The demultiplexer 251 obtains a bitstream supplied from outside the decoding apparatus 250 (e.g., the encoding apparatus 200). The demultiplexer 251 demultiplexes the bitstream, and extracts encoded data of control information and encoded data of motion vectors. In a case where the bitstream includes encoded data of a displacement video, the demultiplexer 251 further extracts encoded data of the displacement video.

[0093] The decoding unit 252 performs processing related to decoding. The control information decoding unit 261 obtains encoded data of control information supplied from the demultiplexer 251. The control information decoding unit 261 decodes the encoded data to generate (restore) control information. The control information decoding unit 261 supplies the control information to the control unit 253. The motion vector decoding unit 262 obtains encoded data of motion vectors supplied from the demultiplexer 251. The motion vector decoding unit 262 decodes the encoded data to generate (restore) motion vectors. The motion vector decoding unit 262 supplies the motion vectors to the current base mesh reconstruction unit 271. In a case where encoded data of a displacement video is supplied from the demultiplexer 251, the displacement video decoding unit 263 obtains the encoded data. The displacement video decoding unit 263 decodes the encoded data to generate (restore) a displacement video. The displacement video decoding unit 263 unpacks the displacement video and extracts a current displacement vector. The displacement video decoding unit 263 supplies the current displacement vector to the current mesh reconstruction unit 272.

[0094] The control unit 253 controls processing units in the decoding apparatus 250 based on control information supplied from the control information decoding unit 261, and causes the processing units to perform processing related to application of a reference displacement vector when reconstructing a current mesh.

[0095] The reconstruction unit 254 performs processing related to mesh reconstruction. The current base mesh reconstruction unit 271 applies the motion vector supplied from the motion vector decoding unit 262 to the vertices of the reference base mesh to reconstruct the current base mesh. The current base mesh reconstruction unit 271 supplies the reconstructed current base mesh to the current mesh reconstruction unit 272. The current mesh reconstruction unit 272 applies the reference displacement vector to the vertices of the current base mesh to reconstruct the current mesh. Note that when the displacement video decoding unit 263 supplies some of the current displacement vectors of the mesh, the current mesh reconstruction unit 272 reconstructs the current mesh by applying the current displacement vectors to some of the vertices of the current base mesh and applying the reference displacement vector to the remaining vertices. The current mesh reconstruction unit 272 outputs the reconstructed current mesh to the outside of the decoding apparatus 250.

[0096] The present technology (the method 1-1 described above) can be applied to the decoding apparatus 250 having such a configuration. For example, the motion vector decoding unit 262 of the decoding apparatus 250 can decode the encoded data to generate the motion vector indicating the positional difference between the vertices of the reference base mesh and the vertices of the current base mesh. The current base mesh reconstruction unit 271 can reconstruct the current base mesh by applying the motion vector to the vertices of the reference base mesh. The current mesh reconstruction unit 272 can reconstruct the current mesh by applying the reference displacement vector to the vertices of the current base mesh.

[0097] With such a configuration, the decoding apparatus 250 can apply the reference displacement vector to the vertices of the current base mesh when reconstructing the current mesh. Therefore, the decoding apparatus 250 can suppress a decrease in encoding efficiency.

[0098] Note that the current mesh reconstruction unit 272 can apply the reference displacement vector to all of the vertices generated by subdividing the current base mesh. The current mesh reconstruction unit 272 can apply the reference displacement vector to some of the vertices generated by subdividing the current base mesh, and can apply the current displacement vector indicating the positional difference between the vertices obtained by subdividing the current base mesh and the vertices of the current mesh to the remaining vertices.

[0099] As described above, the decoding apparatus 250 can include the control information decoding unit 261 and the control unit 253. The control information decoding unit 261 can decode the encoded data to generate the control information related to the application of the reference displacement vector to the vertices of the current base mesh. The control unit 253 can control the reconstruction of the current mesh on the basis of the control information.

[0100] Flow of inter-decoding processing Figure 11 The decoding apparatus 250 in FIG. 25 performs inter-decoding on a bitstream. Reference will be made to FIG. 26. Figure 12The flowchart in FIG. 1 describes an example of the flow of the inter-decoding process.

[0101] When the inter-decoding process starts, in step S251, the demultiplexer 251 demultiplexes the bitstream and extracts the encoded data of the control information and the encoded data of the motion vector. In the case where the bitstream includes the encoded data of the displacement video, the demultiplexer 251 also extracts the encoded data of the displacement video. In step S252, the control information decoding unit 261 decodes the encoded data of the control information extracted from the bitstream to generate (restore) the control information. In step S253, the motion vector decoding unit 262 decodes the encoded data of the motion vector extracted from the bitstream to generate (restore) the motion vector. In step S254, the current base mesh reconstruction unit 271 applies the motion vector to the vertices of the reference base mesh to reconstruct the current base mesh.

[0102] In step S255, the control unit 253 determines whether or not the current displacement vector exists. If it is determined that the current displacement vector exists (the current displacement vector has been provided from the encoding apparatus 200), the process proceeds to step S256. In step S256, the displacement video decoding unit 263 decodes the encoded data of the displacement video to generate (restore) the displacement video. The displacement video decoding unit 263 unpacks the displacement video and extracts the current displacement vector from the displacement video.

[0103] In step S257, the current mesh reconstruction unit 272 reconstructs the current mesh using the current base mesh, the reference displacement vector, and the current displacement vector. When the process in step S257 is completed, the inter-decoding process ends.

[0104] If it is determined in step S255 that the current displacement vector does not exist, the process proceeds to step S258. In step S258, the current mesh reconstruction unit 272 reconstructs the current mesh using the current base mesh and the reference displacement vector. When the process in step S258 is completed, the inter-decoding process ends.

[0105] The present technology (the method 1-1 described above) can be applied to such an inter-decoding process. For example, in step S253, the motion vector decoding unit 262 can decode the encoded data to generate the motion vector indicating the positional difference between the vertices of the reference base mesh and the vertices of the current base mesh. In step S254, the current base mesh reconstruction unit 271 can reconstruct the current base mesh by applying the motion vector to the vertices of the reference base mesh. In step S257 or step S258, the current mesh reconstruction unit 272 can reconstruct the current mesh by applying the reference displacement vector to the vertices of the current base mesh.

[0106] By performing the inter-frame decoding process in this way, the decoding apparatus 250 can apply the reference displacement vector to the vertices of the current base mesh when reconstructing the current mesh. Therefore, the decoding apparatus 250 can suppress a decrease in coding efficiency.

[0107] Method 1-1-1 For example, when the method 1-1 is applied, as shown in the third row from the top of the table in Figure 5 , the current displacement vector can be further applied to the current base mesh to which the reference displacement vector has been applied (method 1-1-1). For example, as shown in Figure 13 , in the current frame, the current displacement vector can be further applied to the mesh reconstructed in the same manner as the example in Figure 6 . For example, as shown in Figure 14 , the reference displacement vector (dashed arrow) is applied to each of the vertices of the subdivided current mesh 301 to reconstruct the mesh 302, and the current displacement vector (solid arrow) is further applied to each of the vertices of the mesh 302 to reconstruct the current mesh 303. That is, in this case, as shown in Figure 14 , both the reference displacement vector and the current displacement vector are applied to one vertex.

[0108] When the reference displacement vector is applied to the reconstruction of the current mesh, the subjective quality of the mesh theoretically decreases unless the displacement vectors match completely between the current frame and the reference frame. On the other hand, transmitting the displacement vector every frame increases the code amount. Therefore, the reference displacement vector is applied to the reconstruction of the current mesh, and the difference between the displacement vectors in the current frame and the reference frame is set as the current displacement vector. In this way, the amount of information in the current displacement vector to be transmitted can be reduced, and the difference between the displacement vectors in the current frame and the reference frame can be reduced. Therefore, a decrease in coding efficiency can be suppressed while a decrease in the subjective quality of the mesh is suppressed.

[0109] Encoding Apparatus Figure 15 is a block diagram illustrating an example of a configuration of an encoding apparatus that is an aspect of the information processing apparatus to which the present technology (the method 1-1-1 described above) is applied. Figure 15 The encoding apparatus 400 shown in

[0110] Note that Figure 15The main processing units, data flows, and the like are shown, but all the processing units, data flows, and the like are not necessarily shown. That is, in the encoding apparatus 400, there can be processing units shown as blocks in Figure 15 , or there can be processing or data flows shown as arrows and the like in Figure 15 , which are not shown in .

[0111] As shown in Figure 15 , the encoding apparatus 400 includes a control unit 401, a transmission data generating unit 402, and a bitstream generating unit 403. The bitstream generating unit 403 includes a motion vector encoding unit 411, a current displacement vector deriving unit 412, a displacement video encoding unit 413, a control information encoding unit 414, and a multiplexer 415.

[0112] The control unit 401 is a processing unit similar to the control unit 201 ( Figure 9 ) and performs similar processing. The transmission data generating unit 402 is a processing unit similar to the transmission data generating unit 202 ( Figure 9 ) and performs similar processing. The motion vector encoding unit 411 is a processing unit similar to the motion vector encoding unit 211 ( Figure 9 ) and performs similar processing.

[0113] The current displacement vector deriving unit 412 obtains the current base mesh and the current mesh provided from the transmission data generating unit 402. The current displacement vector deriving unit 412 also obtains the reference displacement vector. The current displacement vector deriving unit 412 compares the mesh obtained by applying the reference displacement vector to the vertices of the current base mesh with the current mesh, and sets the difference (the position difference of each of the vertices) as the current displacement vector for additional correction, and provides the current displacement vector to the displacement video encoding unit 413.

[0114] The displacement video encoding unit 413 packs the current displacement vector for additional correction provided from the current displacement vector deriving unit 412 into a displacement video, and encodes the displacement video. The displacement video encoding unit 413 provides the encoded data of the displacement video to the multiplexer 415.

[0115] The control information encoding unit 414 is a processing unit similar to the control information encoding unit 213 ( Figure 9 ) and performs similar processing. Similar to the multiplexer 214 ( Figure 9), the multiplexer 415 obtains the encoded data of the motion vector provided from the motion vector encoding unit 411, the encoded data of the displacement video provided from the displacement video encoding unit 413, and the encoded data of the control information provided from the control information encoding unit 414, and multiplexes these encoded data to generate a bitstream. However, in the encoded data of the displacement video, the current displacement vector for the additional correction described above is packed.

[0116] The present technology (the method 1-1-1 described above) can be applied to the encoding apparatus 400 having such a configuration. For example, the encoding apparatus 400 can include the current displacement vector derivation unit 412 and the displacement video encoding unit 413 as described above. The current displacement vector derivation unit 412 can then derive the current displacement vector indicating a positional difference between the vertex of the current mesh and the vertex of the mesh obtained by applying the reference displacement vector to the vertex of the current base mesh. The displacement video encoding unit 413 can encode the current displacement vector. That is, the displacement video encoding unit 413 can also be referred to as a current displacement vector encoding unit.

[0117] With such a configuration, the encoding apparatus 400 can provide the decoder with the current displacement vector for the additional correction. Therefore, the encoding apparatus 400 can suppress a decrease in encoding efficiency while suppressing a decrease in subjective quality of the mesh.

[0118] Flow of inter-frame encoding process Figure 15 The encoding apparatus 400 in the flowchart of FIG. 20 performs inter-frame encoding on the current mesh. The flow of the inter-frame encoding process will be described with reference to the flowchart of FIG. 21. Figure 16

[0119] The processes in steps S401 and S402 are respectively performed in a similar manner to the processes in steps S201 and S202 (FIG. 20). Figure 10

[0120] In step S403, the current displacement vector derivation unit 412 derives the current displacement vector for the additional correction. In step S404, the displacement video encoding unit 413 generates a displacement video using the current displacement vector for the additional correction, and encodes the displacement video.

[0121] The processes in steps S405 to S407 are respectively performed in a similar manner to the processes in steps S205 to S207 (FIG. 20). However, in the encoded data of the displacement video, the current displacement vector for the additional correction described above is packed. When the process in step S407 is completed, the inter-frame encoding process ends. Figure 10

[0122] ​​​This technique (method 1-1-1 described above) can be applied to such inter-frame coding processes. For example, in step S403, the current displacement vector derivation unit 412 can derive a current displacement vector (a current displacement vector for additional correction), which indicates the positional difference between the vertices of the current mesh and the vertices of the mesh obtained by applying a reference displacement vector to the vertices of the current base mesh. In step S404, the displacement video coding unit 413 can pack the current displacement vector for additional correction into the displacement video and encode the displacement video.

[0123] By performing inter-frame coding in this manner, the coding device 400 can provide the decoder with the current displacement vector for additional correction. Therefore, the coding device 400 can suppress the reduction in coding efficiency while simultaneously suppressing the reduction in the subjective quality of the mesh.

[0124] Decoding device Figure 17 This is a block diagram illustrating an example configuration of a decoding device, which is an aspect of an information processing apparatus to which this technology (method 1-1-1 described above) is applied. Figure 17 The decoding device 450 shown is similar to the decoding device 250, and performs inter-frame decoding on the encoded data (bitstream) to generate (restore) a dynamic mesh. That is, the decoding device 450 obtains the bitstream, decodes the bitstream to generate a dynamic mesh, and outputs the generated dynamic mesh.

[0125] Notice, Figure 17 The main processing units and data streams are shown, but not all processing units and data streams are necessarily shown. That is, within the decoding device 450, there may be components not shown in the diagram. Figure 17 The processing unit shown in the diagram is a block, or there may be units not shown in the diagram. Figure 17 The processing or data flow is shown as arrows, etc.

[0126] like Figure 17 As shown, the decoding device 450 includes a demultiplexer 451, a decoding unit 452, a control unit 453, and a reconstruction unit 454. The decoding unit 452 includes a control information decoding unit 461, a motion vector decoding unit 462, and a displacement video decoding unit 463. The reconstruction unit 454 includes a current base mesh reconstruction unit 471, a current mesh reconstruction unit 472, and an additional correction unit 473.

[0127] Demultiplexer 451 is similar to demultiplexer 251 ( Figure 11 The control unit 453 is a processing unit similar to control unit 253, and performs similar processing. Figure 11The control information decoding unit 461 is a processing unit similar to the control information decoding unit 261, and performs similar processing. Figure 11 The motion vector decoding unit 462 is a processing unit similar to the motion vector decoding unit 262, and performs similar processing. Figure 11 The translation video decoding unit 463 is a processing unit similar to the translation video decoding unit 463 ( Figure 11 The current base mesh reconstruction unit 471 is a processing unit similar to the current base mesh reconstruction unit 271, and performs similar processing. Figure 11 The current mesh reconstruction unit 472 is a processing unit similar to the current mesh reconstruction unit 272, and performs similar processing. Figure 11 It is a processing unit that performs similar processing.

[0128] However, the displacement video is packaged together with the current displacement vector used for additional correction, and the displacement video decoding unit 463 extracts the current displacement vector used for additional correction and provides it to the additional correction unit 473. The current mesh reconstruction unit 472 provides the reconstructed current mesh to the additional correction unit 473.

[0129] The additional correction unit 473 applies the current displacement vector for additional correction to the vertices of the already provided current mesh. The additional correction unit 473 outputs the additionally corrected current mesh to the outside of the decoding device 450.

[0130] This technique (method 1-1-1 described above) can be applied to a decoding device 450 having such a configuration. For example, the decoding device 450 may include an additional correction unit 473 as described above. The additional correction unit 473 may further apply a current displacement vector (i.e., the current displacement vector used for additional correction) to the vertices of the current mesh, the current displacement vector indicating the positional difference between the vertices obtained by subdividing the current base mesh and the vertices of the current mesh. Therefore, the additional correction unit 473 may also be referred to as a displacement vector application unit.

[0131] With this configuration, the decoding device 450 can apply the provided current displacement vector for additional correction to the current mesh. Therefore, the decoding device 450 can suppress the reduction in encoding efficiency while also suppressing the reduction in the subjective quality of the mesh.

[0132] Inter-frame decoding process Figure 17 The decoding device 450 in the middle performs inter-frame decoding on the bitstream. (Refer to...) Figure 18 The flowchart in the image illustrates an example of the inter-frame decoding process.

[0133] The processes in steps S451 to S454 are respectively performed in a similar manner to the processes in steps S251 and S254 ( Figure 12 ). The process in step S455 is performed in a similar manner to the process in step S258 ( Figure 12 ).

[0134] In step S456, the displacement video decoding unit 463 decodes the encoded data of the displacement video to generate the displacement video, and unpacks the displacement video to extract the current displacement vector for additional correction.

[0135] In step S457, the additional correction unit 473 performs additional correction on the current mesh using the current displacement vector for additional correction. That is, the additional correction unit 473 applies the current displacement vector for additional correction to the vertices of the current mesh. When the process in step S457 is completed, the inter-frame decoding process ends.

[0136] The present technology (the method 1-1-1 described above) can be applied to such an inter-frame decoding process. For example, in step S457, the additional correction unit 473 can further apply the current displacement vector (i.e., the current displacement vector for additional correction) to the vertices of the current mesh, the current displacement vector indicating a positional difference between a vertex obtained by subdividing the current base mesh and a vertex of the current mesh.

[0137] By performing the inter-frame decoding process in this way, the decoding apparatus 450 can apply the provided current displacement vector for additional correction to the current mesh. Therefore, the decoding apparatus 450 can suppress a decrease in encoding efficiency while suppressing a decrease in subjective quality of the mesh.

[0138] Method 1-2 For example, in applying the method 1, as shown in the fourth row from the top of the table in Figure 5 , the reference mesh can be reused, and the current displacement vector can be further applied (method 1-2). For example, as shown in Figure 19 , the reference mesh reconstructed in the current frame is reused as the current mesh in the current frame, and the current displacement vector is further applied. For example, as shown in Figure 20 , the reference displacement vector (dashed arrow) is applied to the vertices of the subdivided reference base mesh 501 to reconstruct the reference mesh 502. The current displacement vector (solid arrow) is further applied to the vertices of the reference mesh 502 to reconstruct the current mesh 503. Therefore, as shown in Figure 20 , both the reference displacement vector and the current displacement vector are applied to one vertex.

[0139] Thus, similarly to the method 1-1-1 described above, the amount of information in the current displacement vector to be transmitted can be reduced, and the difference in the displacement vector between the current frame and the reference frame can be reduced. Thus, a decrease in coding efficiency can be suppressed while suppressing a decrease in the subjective quality of the mesh.

[0140] Note that, in the method 1-2, the reference mesh is reused as the current mesh as described above. Thus, there is no need to reconstruct the base mesh or the like in the current frame, thereby suppressing an increase in the burden of the decoding process. However, in order to suppress a decrease in the subjective quality of the mesh, it is desirable that the positional difference between the vertices of the reference mesh and the current mesh (i.e., the reference base mesh and the current base mesh) be as small as possible.

[0141] In the case of the method 1-2, by reusing the reference mesh in the current frame, it can be said that the reference displacement vector is used to reconstruct the current mesh, similarly to the case of the method 1 and the other methods described above.

[0142] Encoding apparatus Figure 21 is a block diagram illustrating an example of a configuration of an encoding apparatus, which is an aspect of the information processing apparatus to which the present technology (the method 1-2 described above) is applied. Figure 21 The encoding apparatus 600 illustrated in FIG. 6 is an apparatus similar to the encoding apparatus 400, and encodes a dynamic mesh. That is, the encoding apparatus 600 obtains a current mesh that is a dynamic mesh, inter-frame encodes the obtained current mesh to generate a bitstream, and outputs the generated bitstream.

[0143] Note that, Figure 21 The main processing units, data flows, and the like are illustrated, but not necessarily all of the processing units, data flows, and the like are illustrated. That is, in the encoding apparatus 600, there can be processing units illustrated as blocks in Figure 21 , or there can be processing or data flows illustrated as arrows or the like in Figure 21

[0144] As illustrated in Figure 21 , the encoding apparatus 600 includes a control unit 601, a transmission data generation unit 602, and a bitstream generation unit 603. The bitstream generation unit 603 includes a current displacement vector derivation unit 611, a displacement video encoding unit 612, a control information encoding unit 613, and a multiplexer 614.

[0145] The control unit 601 is a processing unit similar to the control unit 401 ( Figure 15 ), and performs similar processing. The transmission data generation unit 602 is a processing unit similar to the transmission data generation unit 402 ( Figure 15 ), and performs similar processing. ​

[0146] The current displacement vector derivation unit 611 obtains the current mesh provided by the transmission data generation unit 602. The current displacement vector derivation unit 611 also obtains the reference mesh. The current displacement vector derivation unit 611 compares the current mesh with the reference mesh and provides the difference (the position difference of each vertex in the vertices) as the current displacement vector for additional correction to the displacement video encoding unit 612.

[0147] The displacement video coding unit 612 is similar to the displacement video coding unit 413 ( Figure 15 The control information encoding unit 613 is a processing unit similar to the control information encoding unit 414, and performs similar processing. Figure 15 The multiplexer 614 is a processing unit similar to the multiplexer 415, and performs similar processing. Figure 15 It is a processing unit that performs similar processing.

[0148] However, as described above, the current displacement vector for additional correction, obtained by the displacement video coding unit 612 and packed into the displacement video, indicates the difference between the current grid and the reference grid (the position difference of each vertex in the vertices).

[0149] This technique (methods 1-2 described above) can be applied to an encoding device 600 with such a configuration. For example, the current displacement vector deriving unit 611 of the encoding device 600 can derive a current displacement vector indicating the positional difference between the vertices of the current mesh and the vertices of the reference mesh. The displacement video encoding unit 612 can encode the current displacement vector. Therefore, the displacement video encoding unit 612 can also be referred to as the current displacement vector encoding unit.

[0150] Note that the encoding device 600 may include a control information encoding unit 613. The control information encoding unit 613 may encode control information related to applying the current displacement vector to the vertices of the reference mesh when reconstructing the current mesh.

[0151] With this configuration, the encoding device 600 can provide the decoder with the current displacement vector for additional correction. Therefore, the encoding device 600 can suppress the reduction in encoding efficiency while also suppressing the reduction in the subjective quality of the mesh.

[0152] Inter-frame coding process Figure 21 The encoding device 600 in the middle performs inter-frame coding on the current grid. (Refer to...) Figure 22 The flowchart in the image illustrates an example of the inter-frame coding process.

[0153] The processing in step S601 is consistent with that in step S401 ( Figure 16The processing in steps S602 to S606 is performed in a similar manner to the processing in steps S405 to S407, respectively. When the processing in step S606 is completed, the inter-frame encoding processing ends.

[0154] In step S603, the displacement video encoding unit 612 generates a displacement video using the current displacement vector for additional correction, and encodes the displacement video.

[0155] The processing in steps S602 to S606 is performed in a similar manner to the processing in steps S405 to S407, respectively. When the processing in step S606 is completed, the inter-frame encoding processing ends.

[0156] The present technology (the method 1-2 described above) can be applied to such inter-frame encoding processing. For example, in step S602, the current displacement vector derivation unit 611 can derive a current displacement vector indicating a positional difference between a vertex of the current mesh and a vertex of the reference mesh. In step S603, the displacement video encoding unit 612 can encode the current displacement vector. Thus, the displacement video encoding unit 612 can also be referred to as a current displacement vector encoding unit.

[0157] By performing the inter-frame encoding processing in this way, the encoding device 600 can provide the decoder with the current displacement vector for additional correction. Thus, the encoding device 600 can suppress a decrease in encoding efficiency while suppressing a decrease in subjective quality of the mesh.

[0158] Decoding device Figure 23 is a block diagram illustrating an example of a configuration of a decoding device, which is an aspect of an information processing device to which the present technology (the method 1-2 described above) is applied. Figure 23 The decoding device 650 illustrated in

[0159] Note that, Figure 23 The main processing units, data flows, and the like are illustrated, but all the processing units, data flows, and the like are not necessarily illustrated. That is, in the decoding device 650, there can be processing units not illustrated as blocks in Figure 23 , or there can be processing or data flows not illustrated as arrows and the like in Figure 23

[0160] As Figure 23 ​The decoding apparatus 650 includes a demultiplexer 651, a decoding unit 652, a control unit 653, and an additional correction unit 654, as illustrated in FIG. 6. The decoding unit 652 includes a control information decoding unit 661 and a displacement video decoding unit 662.

[0161] The demultiplexer 651 is a processing unit similar to the demultiplexer 451 Figure 17 and performs similar processing. The control unit 653 is a processing unit similar to the control unit 453 Figure 17 and performs similar processing. The control information decoding unit 661 is a processing unit similar to the control information decoding unit 461 Figure 17 and performs similar processing. The displacement video decoding unit 662 is a processing unit similar to the displacement video decoding unit 463 Figure 17 and performs similar processing. The additional correction unit 654 is a processing unit similar to the additional correction unit 473 Figure 17 and performs similar processing.

[0162] However, in the displacement video, the current displacement vector for additional correction is packed. The current displacement vector for additional correction indicates a difference (a position difference of each of the vertices) between the reference mesh and the current mesh. The displacement video decoding unit 662 extracts the current displacement vector for additional correction, and supplies the current displacement vector for additional correction to the additional correction unit 654. The additional correction unit 654 applies the current displacement vector for additional correction to the vertices of the supplied reference mesh. The additional correction unit 654 outputs the additional correction current mesh to the outside of the decoding apparatus 650.

[0163] The present technology (the methods 1-2 described above) can be applied to the decoding apparatus 650 having such a configuration. For example, the displacement video decoding unit 662 of the decoding apparatus 650 can decode the encoded data to generate a current displacement vector indicating a position difference between the vertices of the current mesh and the vertices of the reference mesh. Thus, the displacement video decoding unit 662 can also be referred to as a current displacement vector decoding unit. The additional correction unit 654 reconstructs the current mesh by applying the current displacement vector to the vertices of the reference mesh. Thus, the additional correction unit 654 can also be referred to as a displacement vector application unit.

[0164] With such a configuration, the decoding apparatus 650 can apply the supplied current displacement vector for additional correction to the reference mesh (which is used as the current mesh). Thus, the decoding apparatus 650 can suppress a decrease in encoding efficiency while suppressing a decrease in subjective quality of the mesh.

[0165] Note that, as described above, the decoding apparatus 650 can include the control information decoding unit 661 and the control unit 653. The control information decoding unit 661 can decode the encoded data to generate the control information related to application of the current displacement vector to the vertices of the reference mesh. The control unit 653 can control the application of the current displacement vector to the vertices of the reference mesh on the basis of the control information. With such a configuration, the decoding apparatus 650 can control the application of the current displacement vector to the vertices of the reference mesh on the basis of the control information (i.e., in accordance with the control of the encoding apparatus 600). Thus, the decoding apparatus 650 can suppress a reduction in encoding efficiency while suppressing a reduction in subjective quality of the mesh.

[0166] Flow of inter-decoding process Figure 23 The decoding apparatus 650 in FIG. 19 performs inter-decoding on the bitstream. The flow of the inter-decoding process will be described with reference to the flowchart in FIG. 20. Figure 24 The flow of the inter-decoding process will be described with reference to the flowchart in FIG. 20.

[0167] The processes in steps S651 and S652 are respectively performed in a similar manner to the processes in steps S451 and S452 (FIG. 18). The process in step S653 is performed in a similar manner to the process in step S456 (FIG. 18). However, in this case, the current displacement vector for additional correction indicates a difference (a position difference of each of the vertices) between the reference mesh and the current mesh. Figure 18 Figure 18 In step S654, the additional correction unit 654 performs additional correction on the reference mesh (which is used as the current mesh) using the current displacement vector for additional correction. When the process in step S654 is completed, the inter-decoding process ends.

[0168] The present technology (the method 1-2 described above) can be applied to such an inter-decoding process. For example, in step S653, the displacement video decoding unit 662 can decode the encoded data to generate the current displacement vector indicating a position difference between the vertices of the current mesh and the vertices of the reference mesh. In step S654, the additional correction unit 654 can reconstruct the current mesh by applying the current displacement vector to the vertices of the reference mesh.

[0169] By performing the inter-decoding process in this way, the decoding apparatus 650 can apply the provided current displacement vector for additional correction to the reference mesh (which is used as the current mesh). Thus, the decoding apparatus 650 can suppress a reduction in encoding efficiency while suppressing a reduction in subjective quality of the mesh.

[0170] By performing the inter-decoding process in this way, the decoding apparatus 650 can apply the provided current displacement vector for additional correction to the reference mesh (which is used as the current mesh). Thus, the decoding apparatus 650 can suppress a reduction in encoding efficiency while suppressing a reduction in subjective quality of the mesh.

[0171] 4. Reducing the amount of information of displacement vectors using up-conversion Method 2​ Up-conversion can be used to reduce the amount of information in the displacement vector to be transmitted. For example, as shown in Figure 5 In the case of intra frames, the current base mesh is up-converted and the current displacement vector is applied to the up-converted current base mesh (Method 2).

[0172] Any method can be used to up-convert the current base mesh. For example, an up-conversion filter that is generally applied to meshes (e.g., butterfly subdivision) can be applied to the current base mesh. Such an up-conversion filter for meshes is generally a filter that increases the fineness of the mesh, and subdivides the faces along the shape of the object so as not to degrade the quality of the mesh. Thus, the up-conversion subdivides the current base mesh to form a more accurate contour than the base mesh that is subdivided by applying the V-DMC. Therefore, by using the difference between the up-converted mesh and the original mesh as the displacement vector, the amount of data of the displacement vector can be reduced more than when using the difference between the base mesh that is subdivided by applying the V-DMC and the original mesh as the displacement vector. That is, the increase in the amount of information in the displacement vector can be suppressed, and the decrease in the encoding efficiency can be suppressed.

[0173] Methods 2-1 and 2-2 When Method 2 is applied, as shown in Figure 5 In the case of intra frames, the current base mesh is up-converted and the current displacement vector is applied to the up-converted current base mesh (Method 2). Figure 5 In the case of inter frames, the current base mesh can be up-converted in inter encoding and inter decoding (Method 2-2), as shown in the sixth row from the top of the table.

[0174] As shown in Figure 25 For intra frames, the transmitted current base mesh is up-converted, and the current displacement vector is applied to the up-converted current base mesh to reconstruct the current mesh. In contrast, for inter frames, the transmitted motion vector and the reference base mesh are used to reconstruct the current base mesh. The reconstructed current base mesh is then up-converted. The current displacement vector is then applied to the up-converted current base mesh to reconstruct the current mesh. In this way, the decrease in the encoding efficiency for inter frames and intra frames can be suppressed.

[0175] Encoding device (intra encoding) Figure 26 is a block diagram showing an example of the configuration of an encoding device, which is an aspect of the information processing device to which the present technology (Method 2-1 described above) is applied. Figure 26The encoding apparatus 800 shown in FIG. 8 is an apparatus that encodes a dynamic mesh. The encoding apparatus 800 obtains a current mesh that is a dynamic mesh, intra-encodes the obtained current mesh to generate a bitstream, and outputs the generated bitstream.

[0176] Note that, Figure 26 The main processing units, data flows, and the like are shown, but all the processing units, data flows, and the like are not necessarily shown. That is, in the encoding apparatus 800, there can be processing units shown as blocks in FIG. 8, or there can be processing or data flows shown as arrows and the like in FIG. 8 that are not shown. Figure 26 Figure 26 In FIG. 8, the processing units shown as blocks in FIG. 8, or the processing or data flows shown as arrows and the like in FIG. 8 can not be shown.

[0177] As shown in FIG. 8, the encoding apparatus 800 includes a control unit 801, a transmission data generation unit 802, and a bitstream generation unit 803. The bitstream generation unit 803 includes a base mesh encoding unit 811, an up-converter 812, a current displacement vector derivation unit 813, a displacement video encoding unit 814, a control information encoding unit 815, and a multiplexer 816. Figure 26 The control unit 801 is a processing unit similar to the control unit 401 (

[0178] ) and performs similar processing. The transmission data generation unit 802 is a processing unit similar to the transmission data generation unit 402 and performs similar processing. Figure 15

[0179] The base mesh encoding unit 811 encodes the current base mesh provided from the transmission data generation unit 802 and provides the encoded data to the multiplexer 816.

[0180] The up-converter 812 up-converts the current base mesh provided from the transmission data generation unit 802 and provides the up-converted current base mesh to the current displacement vector derivation unit 813.

[0181] The current displacement vector derivation unit 813 compares the up-converted current base mesh provided from the up-converter 812 with the current mesh provided from the transmission data generation unit 802 and generates a current displacement vector for up-conversion that indicates a difference (a position difference of a vertex) between the up-converted current base mesh and the current mesh. The current displacement vector derivation unit 813 provides the current displacement vector for up-conversion to the displacement video encoding unit 814.

[0182] The displacement video encoding unit 814 packs the current displacement vector for up-conversion into a frame of a displacement video, encodes the displacement video, and provides the encoded data to the multiplexer 816.

[0183] ​​The control information encoding unit 815 is a processing unit similar to the control information encoding unit 414, and performs similar processing. The multiplexer 816 is a processing unit similar to the multiplexer 415, and performs similar processing. However, the encoded data of the displacement video includes the current displacement vector for up-conversion.

[0184] The present technology (the method 2-1 described above) can be applied to the encoding apparatus 800 having such a configuration. For example, the up-converter 812 of the encoding apparatus 800 can perform up-conversion on the current base mesh that is the base mesh of the current frame. The displacement video encoding unit 814 can derive the current displacement vector that indicates a positional difference between the vertices of the current mesh and the vertices of the up-converted current base mesh. Therefore, the displacement video encoding unit 814 can also be referred to as a current displacement vector deriving unit.

[0185] The encoding apparatus 800 can further include a base mesh encoding unit that encodes the current base mesh.

[0186] With such a configuration, the encoding apparatus 800 can suppress an increase in the amount of information in the displacement vector to be transmitted, and suppress a decrease in encoding efficiency.

[0187] Flow of intra-frame encoding processing Figure 26 The encoding apparatus 800 in the configuration of FIG. 8 performs intra-frame encoding on the current mesh. The flow of the intra-frame encoding processing will be described with reference to the flowchart of FIG. 9. Figure 27

[0188] In step S801, the transmission data generating unit 802 generates the current transmission data.

[0189] In step S802, the base mesh encoding unit 811 encodes the current base mesh.

[0190] In step S803, the up-converter 812 performs up-conversion on the current base mesh.

[0191] In step S804, the current displacement vector deriving unit 813 derives the current displacement vector for up-conversion.

[0192] In step S805, the displacement video encoding unit 814 generates a displacement video using the current displacement vector for up-conversion, and encodes the displacement video.

[0193] The processing in steps S806 to S808 is respectively performed in a manner similar to the processing in steps S406 to S407.

[0194] ​The present technology (the method 2-1 described above) can be applied to such an intra coding process. For example, in step S803, the up-converter 812 of the encoding apparatus 800 can perform up-conversion on a current base mesh that is a base mesh of a current frame. In step S805, the displacement video coding unit 814 can derive a current displacement vector that indicates a positional difference between vertices of a current mesh and vertices of the up-converted current base mesh. Thus, the displacement video coding unit 814 can also be referred to as a current displacement vector derivation unit.

[0195] By performing the intra coding process in this way, the encoding apparatus 800 can suppress an increase in the amount of information in the displacement vector to be transmitted, and suppress a reduction in coding efficiency.

[0196] Decoding apparatus (intra decoding) Figure 28 is a block diagram that shows an example of a configuration of an encoding apparatus, which is an aspect of an information processing apparatus to which the present technology (the method 2-1 described above) is applied. Figure 28 The decoding apparatus 850 shown in FIG. 8 is an apparatus that performs intra decoding on encoded data (a bitstream) to generate (restore) a dynamic mesh. The decoding apparatus 850 obtains a bitstream and decodes the bitstream to generate a dynamic mesh, and outputs the generated dynamic mesh.

[0197] Note that, Figure 28 Main processing units, data flows, and the like are shown, but all processing units, data flows, and the like are not necessarily shown. That is, in the decoding apparatus 850, there can be processing units that are not shown as blocks in Figure 28 , or there can be processing or data flows that are not shown as arrows and the like in Figure 28 .

[0198] As shown in Figure 28 , the decoding apparatus 850 includes a demultiplexer 851, a decoding unit 852, a control unit 853, and a reconstruction unit 854. The decoding unit 852 includes a control information decoding unit 861, a base mesh decoding unit 862, and a displacement video decoding unit 863. The reconstruction unit 854 includes an up-converter 871 and a current mesh reconstruction unit 872.

[0199] The base mesh decoding unit 862 obtains encoded data of a base mesh extracted from a bitstream by the demultiplexer 851, and decodes the encoded data of the base mesh, and provides a current base mesh to the up-converter 871. The displacement video decoding unit 863 decodes encoded data of a displacement video to extract a current displacement vector for up-conversion from the obtained displacement video, and provides the current displacement vector for up-conversion to the current mesh reconstruction unit 872.

[0200] The up-converter 871 up-converts the provided current base mesh, and provides the up-converted current base mesh to the current mesh reconstruction unit 872. The current mesh reconstruction unit 872 applies the current displacement vector for up-conversion to the up-converted current base mesh, reconstructs the current mesh, and outputs the reconstructed current mesh.

[0201] The present technology (the method 2-1 described above) can be applied to the decoding apparatus 850 having such a configuration. For example, the up-converter 812 of the decoding apparatus 850 can up-convert the current base mesh that is the base mesh of the current frame. The displacement video decoding unit 863 can decode the encoded data to generate the current displacement vector indicating the positional difference between the vertices of the current mesh and the vertices of the up-converted current base mesh. Therefore, the displacement video decoding unit 863 is also referred to as a current displacement vector decoding unit. The current mesh reconstruction unit 872 can reconstruct the current mesh by applying the current displacement vector to the vertices of the up-converted current base mesh.

[0202] With such a configuration, the decoding apparatus 850 can suppress an increase in the amount of information in the displacement vector to be transmitted, and suppress a decrease in the encoding efficiency.

[0203] Note that the decoding apparatus 850 can further include a base mesh decoding unit that decodes the encoded data to generate the current base mesh. The up-converter 871 can up-convert the generated current base mesh.

[0204] Flow of intra decoding process Figure 28 The decoding apparatus 850 in FIG. 25 performs intra decoding on the bitstream. The flow of the intra decoding process will be described with reference to the flowchart in FIG. 26. Figure 29

[0205] ​In step S851, the de-multiplexer 851 de-multiplexes the bitstream, and extracts the encoded data of the control information, the encoded data of the current base mesh, and the encoded data of the displacement video. In step S852, the control information decoding unit 861 decodes the encoded data of the control information to generate (restore) the control information. In step S853, the base mesh decoding unit 862 decodes the encoded data of the current base mesh to generate (restore) the current base mesh. In step S854, the up-converter 871 up-converts the current base mesh. In step S855, the displacement video decoding unit 863 decodes the encoded data of the displacement video to generate (restore) the displacement video. The displacement video decoding unit 863 unpacks the displacement video, and derives the current displacement vector for up-conversion from the displacement video. In step S856, the current mesh reconstruction unit 872 reconstructs the current mesh by applying the current displacement vector for up-conversion to the up-converter. When the processing in step S856 is completed, the intra decoding processing ends.

[0206] The present technology (the method 2-1 described above) can be applied to the intra decoding processing having such a configuration. For example, in step S854, the up-converter 812 of the decoding device 850 can up-convert the current base mesh that is the base mesh of the current frame. In step S855, the displacement video decoding unit 863 can decode the encoded data to generate the current displacement vector that indicates the positional difference between the vertices of the current mesh and the vertices of the up-converted current base mesh. Therefore, the displacement video decoding unit 863 is also referred to as a current displacement vector decoding unit. In step S856, the current mesh reconstruction unit 872 can reconstruct the current mesh by applying the current displacement vector to the vertices of the up-converted current base mesh.

[0207] By performing the intra decoding processing in this way, the decoding device 850 can suppress the increase in the amount of information in the displacement vector to be transmitted, and suppress the reduction in the encoding efficiency.

[0208] Encoding device (inter coding) The encoding device that performs the inter coding is similar to the encoding device that performs the intra coding. Figure 30 is a block diagram that shows an example of the configuration of an encoding device that is an aspect of the information processing device to which the present technology (the method 2-2 described above) is applied. Figure 30 The encoding device 900 shown in FIG. 9 is a device that encodes a dynamic mesh. The encoding device 900 obtains a current mesh that is a dynamic mesh, inter-codes the obtained current mesh to generate a bitstream, and outputs the generated bitstream.

[0209] Note that, Figure 30The main processing units, data flows, and the like are shown, but all the processing units, data flows, and the like are not necessarily shown. That is, in the encoding apparatus 900, there can be processing units shown as blocks in Figure 30 , or there can be processing or data flows shown as arrows and the like in Figure 30 , which are not shown in .

[0210] As shown in Figure 30 , the encoding apparatus 900 includes a control unit 901, a transmission data generation unit 902, and a bitstream generation unit 903. The bitstream generation unit 903 includes a motion vector encoding unit 911, an up-converter 912, a current displacement vector derivation unit 913, a displacement video encoding unit 914, a control information encoding unit 915, and a multiplexer 916.

[0211] The motion vector encoding unit 911 derives a motion vector using a current base mesh and a reference base mesh, and encodes the motion vector. The motion vector encoding unit 911 provides encoded data of the motion vector to the multiplexer 916. The other processing units each perform a processing operation similar to that of a processing unit of the same name in the encoding apparatus 800 (intra-frame encoding). For example, the up-converter 912 is a processing unit similar to the up-converter 812 ( Figure 26 ), and performs similar processing. The current displacement vector derivation unit 913 is a processing unit similar to the current displacement vector derivation unit 813 ( Figure 26 ), and performs similar processing. The displacement video encoding unit 914 is a processing unit similar to the displacement video encoding unit 814 ( Figure 26 ), and performs similar processing. The control information encoding unit 915 is a processing unit similar to the control information encoding unit 815 ( Figure 26 ), and performs similar processing. The multiplexer 916 is a processing unit similar to the multiplexer 816 ( Figure 26 ), and performs similar processing.

[0212] The present technology (the method 2-2 described above) can be applied to the encoding apparatus 900 having such a configuration. For example, the up-converter 912 of the encoding apparatus 900 can up-convert a current base mesh that is a base mesh of a current frame. The current displacement vector derivation unit 913 of the encoding apparatus 900 can derive a current displacement vector that indicates a positional difference between a vertex of a current mesh and a vertex of the up-converted current base mesh. The displacement video encoding unit 914 can then encode the current displacement vector. The encoding apparatus 900 can further include the motion vector encoding unit 911 that encodes a motion vector that indicates a positional difference between a vertex of a reference base mesh that is a base mesh of a reference frame referred to by the current frame and a vertex of the current base mesh.

[0213] With such a configuration, the encoding device 900 can suppress an increase in the amount of information in the displacement vector to be transmitted, and suppress a reduction in encoding efficiency.

[0214] Flow of inter-frame encoding processing Figure 30 The encoding device 900 in the example of FIG. 10 performs inter-frame encoding on the current mesh. The flow of the inter-frame encoding processing will be described with reference to the flowchart in Figure 31 In step S902, the motion vector encoding unit 911 generates and encodes a motion vector. The present technology (Method 2-2 described above) can be applied to such inter-frame encoding processing.

[0215] By performing the inter-frame encoding processing in this way, the encoding device 900 can suppress an increase in the amount of information in the displacement vector to be transmitted, and suppress a reduction in encoding efficiency.

[0216] Decoding device (inter-frame decoding)

[0217] FIG. 11 is a block diagram showing an example of a configuration of a decoding device that is an aspect of the information processing device to which the present technology (Method 2-2 described above) is applied. Figure 32 The decoding device 950 shown in FIG. 11 is a device that performs inter-frame decoding on encoded data (bitstream) to generate (restore) a dynamic mesh. The decoding device 950 obtains a bitstream and decodes the bitstream to generate a dynamic mesh, and outputs the generated dynamic mesh. Figure 32 Note that

[0218] The main processing units, data flows, and the like are shown, but not necessarily all of the processing units, data flows, and the like are shown. That is, in the decoding device 950, there can be processing units that are not shown as blocks in Figure 32 Figure 32 There can be processing or data flows that are not shown as arrows and the like in Figure 32

[0219] As shown in FIG. 11, the decoding device 950 includes a demultiplexer 951, a decoding unit 952, a control unit 953, and a reconstruction unit 954. The decoding unit 952 includes a control information decoding unit 961, a motion vector decoding unit 962, and a displacement video decoding unit 963. The reconstruction unit 954 includes a current base mesh reconstruction unit 971, an up-converter 972, and a current mesh reconstruction unit 973. Figure 32

[0220] ​​​The demultiplexer 951, control unit 953, control information decoding unit 961, displacement video decoding unit 963, up-converter 972, and current mesh reconstruction unit 973 respectively perform processing operations similar to those of the demultiplexer 851, control unit 853, control information decoding unit 861, displacement video decoding unit 863, up-converter 871, and current mesh reconstruction unit 872.

[0221] Motion vector decoding unit 962 decodes the encoded data of motion vectors to generate motion vectors and provides the motion vectors to current base mesh reconstruction unit 971. Current base mesh reconstruction unit 971 applies the motion vectors to a reference base mesh to reconstruct the current base mesh. Current base mesh reconstruction unit 971 provides the reconstructed current base mesh to upconverter 972. Upconverter 972 upconverts the current base mesh and provides the upconverted current base mesh to current mesh reconstruction unit 973.

[0222] This technique (method 2-2 described above) can be applied to a decoding apparatus 950 with such a configuration. For example, the motion vector decoding unit 962 can decode the encoded data and generate motion vectors indicating the positional differences between the vertices of the reference base grid and the vertices of the current base grid. The current base grid reconstruction unit 971 can reconstruct the current base grid by applying the motion vectors to the vertices of the reference base grid. Note that the reference base grid is the base grid of the reference frame referenced by the current frame, and the upconverter 972 can upconvert the reconstructed current base grid.

[0223] With this configuration, the decoding device 950 can suppress the increase in the amount of information in the displacement vector to be transmitted and suppress the decrease in coding efficiency.

[0224] Inter-frame decoding process Reference Figure 33 The flowchart in the document describes the process by Figure 32 This is an example of the flow of inter-frame decoding processing performed by the decoding device 950. In this case, the processing of steps S951 and S952 is respectively compared with... Figure 29 The processes in steps S851 and S852 are performed in a similar manner.

[0225] In step S953, the motion vector decoding unit 962 decodes the encoded data of the motion vector to generate the motion vector. In step S954, the current base mesh reconstruction unit 971 reconstructs the current base mesh.

[0226] The processes in steps S955 to S957 are performed in a similar manner to those in steps S854 to S857. This technique (method 2-2 described above) can be applied to such inter-frame decoding processes.

[0227] By performing inter-frame decoding in this manner, the decoding device 950 can suppress the increase in the amount of information in the displacement vector to be transmitted and suppress the decrease in coding efficiency.

[0228] 5. Mode Selection Method 3 The methods described above can be combined with any number of other methods. The methods described above can also be combined with methods other than those described above. When multiple methods are available, the choice of which method to apply can be made. That is, as... Figure 5 As shown in the bottom row of the table, you can select a mode (Method 3).

[0229] For example, the encoding device can evaluate each candidate method (pattern) and select the pattern to be applied based on the evaluation results. In this way, a method with better encoding efficiency can be selected. That is, the reduction in encoding efficiency can be further suppressed.

[0230] Encoding device Figure 34 This is a block diagram illustrating an example configuration of an encoding device, which is an aspect of an information processing apparatus to which this technology (method 3 described above) is applied. Figure 34 The encoding device 1100 shown encodes a dynamic grid. That is, the encoding device 1100 obtains the current grid as a dynamic grid, encodes the obtained current grid to generate a bit stream, and outputs the generated bit stream.

[0231] Notice, Figure 34 The main processing units and data streams are shown, but not all processing units and data streams are necessarily shown. That is, within the encoding device 1100, there may be components not shown in the diagram. Figure 34 The processing unit shown in the diagram is a block, or there may be units not shown in the diagram. Figure 34 The processing or data flow is shown as arrows, etc.

[0232] like Figure 34 As shown, the encoding device 1100 includes a control unit 1101, a transmission data generation unit 1102, an encoding unit 1103, a decoding unit 1104, and a mode selection unit 1105.

[0233] The control unit 1101 controls all processing units of the encoding apparatus 1100. The control unit 1101 can generate control information related to, for example, reconstruction of the current mesh, and provide the control information to the encoding unit 1103. The transmission data generation unit 1102 obtains the mesh and performs processing related to generation of transmission data.

[0234] For example, the transmission data generation unit 1102 provides information such as the current mesh to the encoding unit 1103. The transmission data generation unit 1102 also provides the current mesh to the mode selection unit 1105.

[0235] The encoding unit 1103 encodes the provided current mesh, and provides the encoded data (bitstream) to the decoding unit 1104 and the mode selection unit 1105. At this time, the encoding unit 1103 encodes the provided current mesh using all candidate methods including the various methods described above. The encoding unit 1103 provides the encoded data generated by all the methods to the decoding unit 1104 and the mode selection unit 1105.

[0236] The decoding unit 1104 decodes the encoded data (bitstream) to reconstruct the current mesh. That is, the decoding unit 1104 decodes the encoded data (bitstream) using all candidate methods including the various methods described above. The decoding unit 1104 provides the current mesh and the like generated using all the methods to the mode selection unit 1105.

[0237] The mode selection unit 1105 obtains the current mesh before encoding, the encoded data of the current mesh, and the current mesh after decoding for all modes, and calculates and compares rate-distortion optimization (RDO) costs of all the modes using this information and selects the best mode. The mode selection unit 1105 then applies the mode selected as the best mode, and outputs the encoded data (bitstream) generated in this mode to the outside of the encoding apparatus 1100. This bitstream is transmitted to a decoder. The decoder performs a decoding process only in the applied mode to reconstruct the current mesh.

[0238] By applying the present technology (Method 3 described above) in this way, the encoding apparatus 1100 can suppress an increase in the amount of information in the displacement vector to be transmitted, and suppress a decrease in encoding efficiency.

[0239] Flow of encoding process The flow of the encoding process will be described with reference to Figure 35An example of the flow of the encoding process in this case will be described using the flowchart in FIG. 11. When the encoding process starts, in step S1101, the transmission data generating unit 1102 generates the current transmission data. In step S1102, the encoding unit 1103 encodes the current transmission data in all the candidate modes. In step S1103, the decoding unit 1104 decodes the encoded data generated by encoding in all the modes in these modes. In step S1104, the mode selecting unit 1105 derives and compares the RDO costs for all the modes, and selects a mode based on the comparison result.

[0240] When the process in step S1104 is completed, the encoding process ends. By performing the encoding process in this way, the encoding apparatus 1100 can select and apply the optimal mode from a greater variety of methods. Therefore, the encoding apparatus 1100 can suppress a decrease in encoding efficiency.

[0241] 6. Supplementary Note Refinement The decoded original mesh can have any refinement. For example, the refinement of the decoded original mesh can be substantially the same as the refinement of the original mesh before encoding. Here, "substantially the same refinement" means that the refinement is almost the same. The refinement can also consider the number of vertices, connections, polygons, and the like. That is, when "the refinement is substantially the same", the number of vertices, the number of connections, the number of polygons, and the like are almost the same. For example, when the base mesh is generated, by subdividing the base mesh so that the same number of vertices as the number of vertices removed from the original mesh is added, the number of vertices of the decoded original mesh can be substantially the same as the number of vertices of the original mesh before encoding. That is, the refinement of the decoded original mesh can be made substantially the same as the refinement of the original mesh before encoding.

[0242] The refinement of the decoded original mesh can be lower than the refinement of the original mesh before encoding. For example, by applying an encoding scheme that enables scalable decoding as a mesh encoding scheme, the refinement of the decoded original mesh can be variable. For example, by enabling the level of subdivision of the base mesh (the degree to which the base mesh is subdivided) to be changed, scalable decoding is realized. In this case, for example, as described above, the refinement of the decoded original mesh can be made lower than the refinement of the original mesh before encoding.

[0243] In other words, the refinement of the reference mesh or the current mesh can be any refinement, and for example, can be substantially the same as the refinement of the original mesh before encoding, or can be lower than the refinement of the original mesh before encoding.

[0244] Computer The series of processing steps described above can be executed by hardware or software. When the series of processing steps is executed by software, a program included in the software is installed on a computer. Here, the computer includes a computer built in a dedicated hardware, and a general-purpose personal computer that can perform various functions by installing various programs, for example.

[0245] Figure 36 is a block diagram showing an example of a hardware configuration of a computer that executes the series of processing steps described above by a program.

[0246] In Figure 36 In the computer 1900 shown in FIG. 19, a central processing unit (CPU) 1901, a read only memory (ROM) 1902, and a random access memory (RAM) 1903 are interconnected via a bus 1904.

[0247] An input / output interface 1910 is also connected to the bus 1904. An input section 1911, an output section 1912, a storage section 1913, a communication unit 1914, and a drive 1915 are connected to the input / output interface 1910.

[0248] The input section 1911 can be a keyboard, a mouse, a microphone, a touch panel, or an input terminal. The output section 1912 can be a display, a speaker, or an output terminal. The storage section 1913 can be a hard disk, a RAM disk, or a non-volatile memory. The communication unit 1914 can be a network interface. The drive 1915 drives a removable medium 1921 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0249] In the computer implemented as described above, the CPU 1901 loads a program stored in the storage section 1913 into the RAM 1903 via the input / output interface 1910 and the bus 1904, and executes the program, thereby executing the series of processing steps described above. The RAM 1903 also appropriately stores data required by the CPU 1901 to perform various processing operations.

[0250] For example, the program executed by the computer can be applied by being recorded on the removable medium 1921 used as a package medium. In this case, the program can be installed in the storage section 1913 by being inserted into the drive 1915 via the input / output interface 1910.

[0251] The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or a digital satellite broadcast. In this case, the program can be received by the communication unit 1914 and installed in the storage section 1913.

[0252] In addition, the program can be pre-installed in the ROM 1902 or the storage section 1913.

[0253] Applications of the present technology The present technology can be applied to any configuration. For example, the present technology can be applied to various electronic devices.

[0254] For example, the present technology can be implemented as a component of a device, such as a processor (e.g., a video processor) of a system large scale integration (LSI) or the like, a module (e.g., a video module) using a plurality of processors or the like, a unit (e.g., a video unit) using a plurality of modules or the like, or a set (e.g., a video set) in which further functions are added to the unit.

[0255] Further, for example, the present technology can also be applied to a network system including a plurality of devices. For example, the present technology can be implemented as cloud computing in which a plurality of devices share processing via a network. For example, the present technology can be implemented in a cloud service that provides a service related to an image (moving image) to any terminal such as a computer, an audio-visual (AV) device, a portable information processing terminal, or an Internet of Things (IoT) device.

[0256] Note that, in the present specification, a system means a set of a plurality of constituent elements (devices, modules (components), and the like), regardless of whether all the constituent elements are housed in the same housing. Thus, both a plurality of devices housed in separate housings and connected via a network and a single device having a plurality of modules housed in a single housing are systems.

[0257] Fields and applications to which the present technology can be applied Systems, devices, processing units, and the like to which the present technology is applied can be used in any field such as transportation, medical care, crime prevention, agriculture, animal husbandry, mining, beauty, factories, home appliances, weather and natural monitoring. They can be used for any purpose.

[0258] Others Note that, in the present specification, a "flag" refers to information for identifying a plurality of states, and includes not only information for identifying two states (true (1) or false (0)), but also information capable of identifying three or more states. Thus, the value that a "flag" can take can be two values such as 1 and 0, or three or more values. That is, the number of bits of a "flag" can be freely set, and can be one bit or more. In addition, it is assumed that identification information (including a flag) can be included in a bitstream not only as identification information itself but also as differential information of the identification information with respect to certain reference information, and thus in the present specification, a "flag" and "identification information" include not only the information itself but also the differential information with respect to the reference information.

[0259] Various types of information (metadata, etc.) related to encoded data (bitstream) can be transmitted or recorded in any form as long as the information is associated with the encoded data. Here, the term "associated" means, for example, that one piece of data can be used (linked) with another piece of data when processing the one piece of data. That is, data associated with each other can be combined into one piece of data, or can be separate pieces of data. For example, information associated with encoded data (image) can be transmitted via a transmission path different from that of the encoded data (image). For example, information associated with encoded data (image) can be recorded on a different recording medium from the encoded data (image) (or on a different recording area of the same recording medium). Note that this "association" does not have to be associated with the entire data, but can be associated with only part of the data. For example, an image and information corresponding to the image can be associated with each other in any unit (e.g., a plurality of frames, one frame, or a part of a frame).

[0260] Note that in the present specification, terms such as "synthesis", "multiplexing", "adding", "integrating", "including", "storing", "putting", "plugging in", and "inserting" mean combining a plurality of things into one, for example, combining encoded data and metadata into one piece of data, and mean one method of the above-described "association".

[0261] Embodiments of the present technology are not limited to the above-described embodiments, and various modifications can be made without departing from the gist of the present technology.

[0262] For example, a configuration described as one apparatus (or processing unit) can be divided and configured as a plurality of apparatuses (or processing units). Conversely, a configuration described above as a plurality of apparatuses (or processing units) can be integrated into one apparatus (or processing unit). Needless to say, a configuration other than the above-described configuration can be added to the configuration of each apparatus (or each processing unit). When the configuration and operation as a whole system are substantially the same, a part of the configuration of a certain apparatus (or processing unit) can be included in the configuration of another apparatus (or another processing unit).

[0263] For example, the above-described program can be executed in any apparatus. In the above-described case, the apparatus needs to have a necessary function (functional block, etc.) to obtain necessary information.

[0264] For example, each step in a single flowchart can be executed by a single apparatus, or can be shared and executed by a plurality of apparatuses. Furthermore, when a single step includes a plurality of processes, the plurality of processes can be executed by a single apparatus, or can be shared and executed by a plurality of apparatuses. In other words, the plurality of processes included in a single step can be executed as processes in a plurality of steps. Conversely, processes described as a plurality of steps can be collectively executed as a single step.

[0265] For example, a program to be executed by a computer can be configured so that the processing in the steps describing the program is executed in the time sequence described in this specification, or in parallel, or individually at necessary timing, for example, at the time of a call. That is, as long as no contradiction occurs, the processing in the steps can be executed in a different order from the above-described order. Furthermore, the processing in the steps describing the program can be executed in parallel with the processing in another program, or can be executed in combination with the processing in another program.

[0266] For example, as long as no contradiction occurs, a plurality of technologies related to the present technology can be implemented independently. Obviously, any two or more technologies in the present technology can be implemented in combination. For example, part or all of the present technology described in any of the embodiments can be implemented in combination with part or all of the present technology described in another embodiment. Furthermore, part or all of any of the technologies described above can be implemented in combination with another technology not described above.

[0267] The present technology can employ the configurations described below. (1) An information processing apparatus including: a motion vector decoding unit that decodes encoded data and generates a motion vector indicating a position difference between vertices of a reference base mesh and vertices of a current base mesh; a current base mesh reconstruction unit that reconstructs the current base mesh by applying the motion vector to the vertices of the reference base mesh; and a current mesh reconstruction unit that reconstructs a current mesh by applying a reference displacement vector to the vertices of the current base mesh, wherein the reference base mesh is a base mesh of a reference frame referred to by a current frame, the current base mesh is a base mesh of the current frame, a base mesh is a mesh having lower fineness than an original mesh to be encoded, the original mesh includes vertices and connections representing a three-dimensional structure of an object, the base mesh is generated by removing the vertices from the middle of the original mesh, and the reference displacement vector indicates a position difference between vertices obtained by subdividing the reference base mesh and vertices of a reference mesh. (2) The information processing apparatus according to (1), further including a displacement vector application unit that further applies a current displacement vector to the vertices of the current mesh, the current displacement vector indicating a position difference between vertices obtained by subdividing the current base mesh and the vertices of the current mesh. (3) The information processing apparatus according to (1) or (2), wherein the current mesh reconstruction unit applies the reference displacement vector to all of the vertices generated by subdividing the current base mesh. (4) The information processing apparatus according to (1) or (2), wherein the current mesh reconstruction unit applies the reference displacement vector to some of the vertices generated by subdividing the current base mesh, and applies a current displacement vector indicating a positional difference between the vertices obtained by subdividing the current base mesh and the vertices of the current mesh to the remaining ones of the vertices. (5) The information processing apparatus according to any one of (1) to (4), further comprising: a control information decoding unit that decodes the encoded data and generates control information related to application of the reference displacement vector to the vertices of the current base mesh; and a control unit that controls reconstruction of the current mesh based on the control information. (6) An information processing method comprising: decoding encoded data and generating a motion vector indicating a positional difference between vertices of a reference base mesh and vertices of a current base mesh; reconstructing the current base mesh by applying the motion vector to the vertices of the reference base mesh; and reconstructing a current mesh by applying a reference displacement vector to the vertices of the current base mesh, wherein the reference base mesh is a base mesh of a reference frame referred to by a current frame, the current base mesh is a base mesh of the current frame, a base mesh is a mesh having a lower degree of fineness than an original mesh to be encoded, the original mesh includes vertices and connections representing a three-dimensional structure of an object, the base mesh is generated by removing vertices from the middle of the original mesh, and the reference displacement vector indicates a positional difference between vertices obtained by subdividing the reference base mesh and vertices of a reference mesh.

[0268] (7) An information processing apparatus comprising: an encoding unit that encodes a motion vector indicating a positional difference between vertices of a reference base mesh and vertices of a current base mesh; and a control information encoding unit that encodes control information related to application of a reference displacement vector to the vertices of the current base mesh at the time of reconstruction of a current mesh, wherein the reference base mesh is a base mesh of a reference frame referred to by a current frame, the current base mesh is a base mesh of the current frame, a base mesh is a mesh having a lower degree of fineness than an original mesh to be encoded, the original mesh includes vertices and connections representing a three-dimensional structure of an object, the base mesh is generated by removing vertices from the middle of the original mesh, and the reference displacement vector indicates a positional difference between vertices obtained by subdividing the reference base mesh and vertices of a reference mesh. (8) The information processing apparatus according to (7), further comprising: a current displacement vector deriving unit that derives a current displacement vector indicating a positional difference between the vertices of the current mesh and the vertices of a mesh obtained by applying the reference displacement vector to the vertices of the current base mesh; and a current displacement vector encoding unit that encodes the current displacement vector. (9) The information processing apparatus according to (7), further comprising a current displacement vector encoding unit that encodes a current displacement vector indicating a positional difference between some vertices of a current mesh and vertices of a current base mesh. (10) An information processing method comprising: encoding a motion vector indicating a positional difference between vertices of a reference base mesh and vertices of a current base mesh; and encoding control information related to application of a reference displacement vector to the vertices of the current base mesh in reconstructing a current mesh, wherein the reference base mesh is a base mesh of a reference frame referred to by a current frame, the current base mesh is a base mesh of the current frame, a base mesh is a mesh having lower fineness than an original mesh to be encoded, the original mesh includes vertices and connections representing a three-dimensional structure of an object, the base mesh is generated by removing vertices from the original mesh, and the reference displacement vector indicates a positional difference between vertices obtained by subdividing the reference base mesh and vertices of a reference mesh.

[0269] (11) An information processing apparatus comprising: a current displacement vector decoding unit that decodes encoded data and generates a current displacement vector indicating a positional difference between vertices of a current mesh and vertices of a reference mesh; and a displacement vector application unit that reconstructs the current mesh by applying the current displacement vector to the vertices of the reference mesh, wherein the current mesh is a mesh in a current frame that includes vertices and connections representing a three-dimensional structure of an object, and the reference mesh is a mesh in a reference frame referred to by the current frame that includes vertices and connections representing a three-dimensional structure of an object.

[0270] (12) The information processing apparatus according to (11), further comprising: a control information decoding unit that decodes the encoded data and generates control information related to application of the current displacement vector to the vertices of the reference mesh; and a control unit that controls application of the current displacement vector to the vertices of the reference mesh based on the control information. (13) An information processing method comprising: decoding encoded data and generating a current displacement vector indicating a positional difference between vertices of a current mesh and vertices of a reference mesh; and reconstructing the current mesh by applying the current displacement vector to the vertices of the reference mesh, wherein the current mesh is a mesh in a current frame that includes vertices and connections representing a three-dimensional structure of an object, and the reference mesh is a mesh in a reference frame referred to by the current frame that includes vertices and connections representing a three-dimensional structure of an object.

[0271] (14) An information processing apparatus including: a current displacement vector deriving that derives a current displacement vector indicating a positional difference between a vertex of a current mesh and a vertex of a reference mesh; and a current displacement vector encoding unit that encodes the current displacement vector, wherein the current mesh is a mesh including vertices and connections that represent a three-dimensional structure of an object in a current frame, and the reference mesh is a mesh including vertices and connections that represent a three-dimensional structure of the object in a reference frame that is referred to by the current frame. (15) The information processing apparatus according to (14), further including a control information encoding unit that encodes control information related to application of the current displacement vector to the vertex of the reference mesh when reconstructing the current mesh. (16) An information processing method including: deriving a current displacement vector indicating a positional difference between a vertex of a current mesh and a vertex of a reference mesh; and encoding the current displacement vector, wherein the current mesh is a mesh including vertices and connections that represent a three-dimensional structure of an object in a current frame, and the reference mesh is a mesh including vertices and connections that represent a three-dimensional structure of the object in a reference frame that is referred to by the current frame.

[0272] (17) An information processing apparatus including: an up-conversion unit that up-converts a current base mesh that is a base mesh of a current frame; a current displacement vector decoding unit that decodes encoded data and generates a current displacement vector indicating a positional difference between a vertex of a current mesh and a vertex of the up-converted current base mesh; and a current mesh reconstructing unit that reconstructs the current mesh by applying the current displacement vector to the vertex of the up-converted current base mesh, wherein the base mesh is a mesh having lower fineness than an original mesh to be encoded, the original mesh including vertices and connections that represent a three-dimensional structure of an object, and the base mesh is generated by removing vertices from the middle of the original mesh. (18) The information processing apparatus according to (17), further including a base mesh decoding unit that decodes the encoded data and generates the current base mesh, wherein the up-conversion unit up-converts the generated current base mesh. (19) The information processing apparatus according to (17), further including: a motion vector decoding unit that decodes the encoded data and generates a motion vector indicating a positional difference between a vertex of a reference base mesh and a vertex of the current base mesh; and a current base mesh reconstructing unit that reconstructs the current base mesh by applying the motion vector to the vertex of the reference base mesh, wherein the reference base mesh is a base mesh of a reference frame that is referred to by the current frame, and the up-conversion unit up-converts the reconstructed current base mesh. (20) An information processing method including: up-converting a current base mesh that is a base mesh of a current frame; decoding encoded data and generating a current displacement vector that indicates a positional difference between vertices of a current mesh and vertices of the up-converted current base mesh; and reconstructing the current mesh by applying the current displacement vector to the vertices of the up-converted current base mesh, wherein the base mesh is a mesh having a lower degree of fineness than an original mesh to be encoded, the original mesh including vertices and connections that represent a three-dimensional structure of an object, and the base mesh is generated by removing vertices from the original mesh.

[0273] (21) An information processing apparatus including: an up-conversion unit that up-converts a current base mesh that is a base mesh of a current frame; a current displacement vector derivation unit that derives a current displacement vector that indicates a positional difference between vertices of a current mesh and vertices of the up-converted current base mesh; and a current displacement vector encoding unit that encodes the current displacement vector, wherein the base mesh is a mesh having a lower degree of fineness than an original mesh to be encoded, the original mesh including vertices and connections that represent a three-dimensional structure of an object, and the base mesh is generated by removing vertices from the original mesh. (22) The information processing apparatus according to (21), further including a base mesh encoding unit that encodes the current base mesh. (23) The information processing apparatus according to (21), further including a motion vector encoding unit that encodes a motion vector that indicates a positional difference between vertices of a reference base mesh and vertices of the current base mesh, the reference base mesh being a base mesh of a reference frame that is referenced by the current frame. (24) An information processing method including: up-converting a current base mesh that is a base mesh of a current frame; deriving a current displacement vector that indicates a positional difference between vertices of a current mesh and vertices of the up-converted current base mesh; and encoding the current displacement vector, wherein the base mesh is a mesh having a lower degree of fineness than an original mesh to be encoded, the original mesh including vertices and connections that represent a three-dimensional structure of an object, and the base mesh is generated by removing vertices from the original mesh. LIST OF REFERENCE NUMERALS

[0274] 200 encoding device, 201 control unit, 202 transmission data generating unit, 203 bit stream generating unit, 211 motion vector encoding unit, 212 displacement video encoding unit, 213 control information encoding unit, 214 multiplexer, 250 decoding device, 251 demultiplexer, 252 decoding unit, 253 control unit, 254 reconstruction unit, 261 control information decoding unit, 262 motion vector decoding unit, 263 displacement video decoding unit, 271 current base mesh reconstruction unit, 272 current mesh reconstruction unit, 1900 computer.

Claims

1. An information processing apparatus comprising: a motion vector decoding unit configured to decode encoded data and generate a motion vector indicating a positional difference between vertices of a reference base mesh and vertices of a current base mesh; a current base mesh reconstructing unit configured to reconstruct the current base mesh by applying the motion vector to the vertices of the reference base mesh; and a current mesh reconstructing unit configured to reconstruct a current mesh by applying a reference displacement vector to the vertices of the current base mesh, wherein the reference base mesh is a base mesh of a reference frame referred to by a current frame, the current base mesh is a base mesh of the current frame, the base mesh is a mesh having lower fineness than an original mesh to be encoded, the original mesh including vertices and connections representing a three-dimensional structure of an object, the base mesh being generated by removing the vertices from the middle of the original mesh, and the reference displacement vector indicates a positional difference between vertices obtained by subdividing the reference base mesh and vertices of a reference mesh. 2.The information processing apparatus according to claim 1, further comprising: a displacement vector applying unit configured to further apply a current displacement vector to the vertices of the current mesh, the current displacement vector indicating a positional difference between vertices obtained by subdividing the current base mesh and the vertices of the current mesh. 3.The information processing apparatus according to claim 1, wherein the current mesh reconstructing unit applies the reference displacement vector to all of the vertices generated by subdividing the current base mesh. 4.The information processing apparatus according to claim 1, wherein the current mesh reconstructing unit applies the reference displacement vector to some of the vertices generated by subdividing the current base mesh, and applies a current displacement vector indicating a positional difference between vertices obtained by subdividing the current base mesh and the vertices of the current mesh to the remaining ones of the vertices. 5.The information processing apparatus according to claim 1, further comprising: a control information decoding unit configured to decode encoded data and generate control information related to applying the reference displacement vector to the vertices of the current base mesh; and a control unit configured to control the reconstruction of the current mesh based on the control information. 6.An information processing method comprising: decoding encoded data and generating a motion vector indicating a positional difference between vertices of a reference base mesh and vertices of a current base mesh; reconstructing the current base mesh by applying the motion vector to the vertices of the reference base mesh; and reconstructing a current mesh by applying a reference displacement vector to the vertices of the current base mesh, wherein the reference base mesh is a base mesh of a reference frame referred to by a current frame, the current base mesh is a base mesh of the current frame, the base mesh is a mesh having lower fineness than an original mesh to be encoded, the original mesh including vertices and connections representing a three-dimensional structure of an object, the base mesh being generated by removing the vertices from the middle of the original mesh, and the reference displacement vector indicates a positional difference between vertices obtained by subdividing the reference base mesh and vertices of a reference mesh. ​ The base mesh is a mesh having lower fineness than an original mesh to be encoded, the original mesh including vertices and connections representing a three-dimensional structure of an object, the base mesh being generated by removing the vertices from the middle of the original mesh, and The reference displacement vector indicates a positional difference between vertices obtained by subdividing the reference base mesh and vertices of a reference mesh. 7.An information processing apparatus comprising: an encoding unit configured to encode a motion vector indicating a positional difference between vertices of a reference base mesh and vertices of a current base mesh; and a control information encoding unit configured to encode control information related to application of a reference displacement vector to the vertices of the current base mesh when reconstructing a current mesh, wherein the reference base mesh is a base mesh of a reference frame referred to by a current frame, the current base mesh is a base mesh of the current frame, the base mesh is a mesh having lower fineness than an original mesh to be encoded, the original mesh including vertices and connections representing a three-dimensional structure of an object, the base mesh being generated by removing the vertices from the middle of the original mesh, and the reference displacement vector indicates a positional difference between vertices obtained by subdividing the reference base mesh and vertices of a reference mesh. 8.The information processing apparatus according to claim 7, further comprising: a current displacement vector deriving unit configured to derive a current displacement vector indicating a positional difference between vertices of the current mesh and vertices of a mesh obtained by applying the reference displacement vector to the vertices of the current base mesh; and a current displacement vector encoding unit configured to encode the current displacement vector. 9.The information processing apparatus according to claim 7, further comprising: a current displacement vector encoding unit configured to encode a current displacement vector indicating a positional difference between some vertices of the current mesh and the vertices of the current base mesh. 10.An information processing method comprising: encoding a motion vector indicating a positional difference between vertices of a reference base mesh and vertices of a current base mesh; and encoding control information related to application of a reference displacement vector to the vertices of the current base mesh when reconstructing a current mesh, wherein the reference base mesh is a base mesh of a reference frame referred to by a current frame, the current base mesh is a base mesh of the current frame, the base mesh is a mesh having lower fineness than an original mesh to be encoded, the original mesh including vertices and connections representing a three-dimensional structure of an object, the base mesh being generated by removing the vertices from the middle of the original mesh, and the reference displacement vector indicates a positional difference between vertices obtained by subdividing the reference base mesh and vertices of a reference mesh. 11.An information processing apparatus comprising: a current displacement vector decoding unit configured to decode encoded data and generate a current displacement vector indicating a positional difference between vertices of a current mesh and vertices of a reference mesh; and a displacement vector application unit configured to reconstruct the current mesh by applying the current displacement vector to vertices of the reference mesh, wherein the current mesh is a mesh in a current frame including vertices and connections representing a three-dimensional structure of an object, and the reference mesh is a mesh in a reference frame referred to by the current frame including vertices and connections representing the three-dimensional structure of the object.

12. An information processing method comprising: decoding encoded data and generating a current displacement vector indicating a positional difference between vertices of a current mesh and vertices of a reference mesh; and reconstructing the current mesh by applying the current displacement vector to vertices of the reference mesh, wherein the current mesh is a mesh in a current frame including vertices and connections representing a three-dimensional structure of an object, and the reference mesh is a mesh in a reference frame referred to by the current frame including vertices and connections representing the three-dimensional structure of the object.

13. An information processing apparatus comprising: a current displacement vector derivation configured to derive a current displacement vector indicating a positional difference between vertices of a current mesh and vertices of a reference mesh; and a current displacement vector encoding unit configured to encode the current displacement vector, wherein the current mesh is a mesh in a current frame including vertices and connections representing a three-dimensional structure of an object, and the reference mesh is a mesh in a reference frame referred to by the current frame including vertices and connections representing the three-dimensional structure of the object.

14. An information processing method comprising: deriving a current displacement vector indicating a positional difference between vertices of a current mesh and vertices of a reference mesh; and encoding the current displacement vector, wherein the current mesh is a mesh in a current frame including vertices and connections representing a three-dimensional structure of an object, and the reference mesh is a mesh in a reference frame referred to by the current frame including vertices and connections representing the three-dimensional structure of the object.

15. An information processing apparatus comprising: an up-conversion unit configured to up-convert a current base mesh that is a base mesh of a current frame; a current displacement vector decoding unit configured to decode encoded data and generate a current displacement vector indicating a positional difference between vertices of a current mesh and vertices of the up-converted current base mesh; and a current mesh reconstruction unit configured to reconstruct the current mesh by applying the current displacement vector to vertices of the up-converted current base mesh, wherein the base mesh is a mesh having lower fineness than an original mesh to be encoded, the original mesh including vertices and connections representing a three-dimensional structure of an object, the base mesh being generated by intermediate division of the vertices from the original mesh.

16. The information processing apparatus according to claim 15, further comprising: a base mesh decoding unit configured to decode encoded data and generate the current base mesh, wherein the up-conversion unit up-converts the generated current base mesh.

17. The information processing apparatus according to claim 15, further comprising: a motion vector decoding unit configured to decode the encoded data and generate a motion vector indicating a position difference between vertices of a reference base mesh and vertices of the current base mesh; and a current base mesh reconstructing unit configured to reconstruct the current base mesh by applying the motion vector to the vertices of the reference base mesh, wherein the reference base mesh is the base mesh of a reference frame referred to by the current frame, and the up-conversion unit up-converts the reconstructed current base mesh.

18. An information processing method comprising: up-converting a current base mesh that is a base mesh of a current frame; decoding encoded data and generating a current displacement vector indicating a position difference between vertices of a current mesh and vertices of the up-converted current base mesh; and reconstructing the current mesh by applying the current displacement vector to the vertices of the up-converted current base mesh, wherein the base mesh is a mesh having a lower degree of fineness than an original mesh to be encoded, the original mesh including vertices and connections representing a three-dimensional structure of an object, the base mesh being generated by removing the vertices from the middle of the original mesh.

19. An information processing apparatus comprising: an up-conversion unit configured to up-convert a current base mesh that is a base mesh of a current frame; a current displacement vector deriving unit configured to derive a current displacement vector indicating a position difference between vertices of a current mesh and vertices of the up-converted current base mesh; and a current displacement vector encoding unit configured to encode the current displacement vector, wherein the base mesh is a mesh having a lower degree of fineness than an original mesh to be encoded, the original mesh including vertices and connections representing a three-dimensional structure of an object, the base mesh being generated by removing the vertices from the middle of the original mesh.

20. An information processing method comprising: up-converting a current base mesh that is a base mesh of a current frame; deriving a current displacement vector indicating a position difference between vertices of a current mesh and vertices of the up-converted current base mesh; and encoding the current displacement vector, wherein the base mesh is a mesh having a lower degree of fineness than an original mesh to be encoded, the original mesh including vertices and connections representing a three-dimensional structure of an object, the base mesh being generated by removing the vertices from the middle of the original mesh.

Images