Information processing device and method

Abstract

The present disclosure relates to an information processing device and method that make it possible to suppress deterioration in subjective quality and suppress an increase in encoding / decoding processing load. In the present invention, geometry patch coordinate systems derived for respective geometry patches of a base mesh are set on the basis of a canonical coordinate system, displacement vectors in the geometry patch coordinate systems are derived, and the base mesh and the displacement vectors therefor are encoded. In addition, the encoded data of the displacement vectors is decoded, the displacement vectors in the geometry patch coordinate systems are generated, the geometry patch coordinate systems are set for the respective geometry patches of the subdivided base mesh, and the displacement vectors are applied to the subdivided base mesh by using the geometry patch coordinate systems. The present disclosure can be applied to, for example, information processing devices, electronic apparatuses, information processing methods, and programs.

Description

Information processing device and method

[0001] The present disclosure relates to an information processing device and method, and more particularly to an information processing device and method that can suppress a decrease in subjective quality and suppress an increase in the load of encoding and decoding processes.

[0002] Conventionally, V-DMC (Video-based Dynamic Mesh Coding) has been used as a method for encoding meshes, which are 3D data that represent the three-dimensional structure of an object using vertices and connections (see, for example, Non-Patent Document 1). In V-DMC, a base mesh, displacement vectors for vertex correction, attributes, and atlas information are encoded and transmitted as a bitstream. During decoding, each piece of information in the bitstream is decoded, the base mesh is subdivided, and displacement vectors are applied to the subdivided vertices to reconstruct a decoded mesh. A canonical coordinate system has been used to represent displacement vectors. A canonical coordinate system is a single coordinate system used for all vertices. Another coordinate system that represents displacement vectors has been the local coordinate system (see, for example, Non-Patent Document 2). A local coordinate system is a coordinate system for each vertex.

[0003] Khaled Mammou, Jungsun Kim, Alexis Tourapis, Dimitri Podborski, Krasimir Kolarov, "[V-CG] Apple's Dynamic Mesh Coding CfP Response", ISO / IEC JTC 1 / SC 29 / WG 7 m59281, April 2022Satoru Kuma, Kao Hayashi, Danillo Graziosi, Ohji Nakagami, "[V-DMC][New Proposal] Improvement on local coordinate system for displacement coding", ISO / IEC JTC 1 / SC 29 / WG 7 m61020, October 2022

[0004] However, in the case of the canonical coordinate system, the error of the displacement vector increases depending on the displacement direction, which may reduce the subjective quality of the reconstructed mesh.In contrast, in the case of the local coordinate system, normal calculation is required for each vertex, which may increase the load of the encoding and decoding processes.

[0005] The present disclosure has been made in light of such circumstances, and makes it possible to suppress a decrease in subjective quality and suppress an increase in the load of encoding and decoding processes.

[0006] An information processing device according to one aspect of the present technology includes a geometry patch coordinate system setting unit that sets a geometry patch coordinate system derived for each geometry patch of a base mesh based on a canonical coordinate system, a displacement vector derivation unit that derives a displacement vector in the geometry patch coordinate system, a base mesh encoding unit that encodes the base mesh, and a displacement vector encoding unit that encodes the displacement vector, wherein the base mesh is a mesh with lower resolution than the original mesh to be encoded, which is composed of vertices and connections that represent the three-dimensional structure of an object, and is generated by thinning out vertices from the original mesh, and the displacement vector is vector information that indicates the displacement of the vertices of the subdivided base mesh.

[0007] An information processing method according to one aspect of the present technology includes setting a geometry patch coordinate system derived for each geometry patch of a base mesh based on a canonical coordinate system, deriving a displacement vector in the geometry patch coordinate system, encoding the base mesh, and encoding the displacement vector, wherein the base mesh is a mesh with lower resolution than an original mesh to be encoded, the original mesh being composed of vertices and connections that represent the three-dimensional structure of an object, and is generated by thinning out vertices from the original mesh, and the displacement vector is vector information indicating the displacement of the vertices of the subdivided base mesh.

[0008] an information processing device according to another aspect of the present technology, the information processing device including: a base mesh decoding unit that decodes encoded data of a base mesh and generates the base mesh; a subdivision unit that subdivides the base mesh; a displacement vector decoding unit that decodes encoded data of a displacement vector and generates the displacement vector in a geometry patch coordinate system derived for each geometry patch of the base mesh with a canonical coordinate system as a reference; a geometry patch coordinate system setting unit that sets the geometry patch coordinate system for each geometry patch of the subdivided base mesh; and a displacement vector application unit that applies the displacement vector to the subdivided base mesh using the geometry patch coordinate system, wherein the base mesh is a mesh with lower resolution than the original mesh to be encoded, the original mesh being composed of vertices and connections that represent a three-dimensional structure of an object, and the displacement vector is vector information indicating the displacement of the vertices of the subdivided base mesh.

[0009] Another aspect of the present technology provides an information processing method that includes decoding encoded data of a base mesh, generating the base mesh, subdividing the base mesh, decoding encoded data of a displacement vector, generating the displacement vector in a geometry patch coordinate system derived for each geometry patch of the base mesh based on a canonical coordinate system, setting the geometry patch coordinate system for each geometry patch of the subdivided base mesh, and applying the displacement vector to the subdivided base mesh using the geometry patch coordinate system, wherein the base mesh is a mesh with lower resolution than the original mesh that is generated by thinning vertices from an original mesh to be encoded that is composed of vertices and connections that represent a three-dimensional structure of an object, and the displacement vector is vector information that indicates displacements of vertices of the subdivided base mesh.

[0010] In an information processing device and method according to one aspect of the present technology, a geometry patch coordinate system is set for each geometry patch of a base mesh based on a canonical coordinate system, a displacement vector in the geometry patch coordinate system is derived, the base mesh is encoded, and the displacement vector is encoded. Note that the base mesh is a mesh with lower resolution than an original mesh to be encoded, which is composed of vertices and connections that represent a three-dimensional structure of an object, and is generated by thinning out vertices from the original mesh. The displacement vector is vector information indicating the displacement of vertices of the subdivided base mesh.

[0011] In another aspect of the present technology, an information processing device and method decode encoded data of a base mesh to generate the base mesh, subdivide the base mesh, decode encoded data of displacement vectors, generate displacement vectors in a geometry patch coordinate system derived for each geometry patch of the base mesh based on a canonical coordinate system, set a geometry patch coordinate system for each geometry patch of the subdivided base mesh, and apply the displacement vectors to the subdivided base mesh using the geometry patch coordinate system. Note that the base mesh is a mesh with lower resolution than the original mesh to be encoded, which is composed of vertices and connections that represent the three-dimensional structure of an object, and is generated by thinning out vertices from the original mesh. Furthermore, the displacement vectors are vector information indicating the displacements of the vertices of the subdivided base mesh.

[0012] FIG. 1 is a diagram illustrating a mesh. FIG. 1 is a diagram illustrating V-DMC. FIG. 1 is a diagram illustrating an example of packing of displacement vectors. FIG. 1 is a diagram illustrating a coordinate system of displacement vectors. FIG. 1 is a diagram illustrating an example of a method for deriving a displacement vector in a local coordinate system. FIG. 2 is a diagram illustrating an example of atlas information. FIG. 2 is a diagram illustrating an example of a texture development method. FIG. 3 is a diagram illustrating an example of a method for setting a coordinate system of a displacement vector. FIG. 3 is a diagram illustrating an example of an original mesh. FIG. 4 is a diagram illustrating an example of a base mesh. FIG. 4 is a diagram illustrating an example of a geometry patch. FIG. 5 is a diagram illustrating an example of a displacement vector deriving method. FIG. 6 is a diagram illustrating an example of geometry patch coordinate system information. FIG. 7 is a diagram illustrating an example of coordinate system identification information. FIG. 8 is a diagram illustrating an example of processing of geometry patch boundary vertices. FIG. 9 is a block diagram illustrating an example of the main configuration of an encoding device. FIG. 10 is a block diagram illustrating an example of the main configuration of a displacement vector generation unit. FIG. 11 is a block diagram illustrating an example of the main configuration of a V-DMC encoding unit. FIG. 12 is a flowchart illustrating an example of the flow of encoding processing. FIG. 13 is a flowchart illustrating an example of the flow of displacement vector generation processing. FIG. 14 is a flowchart illustrating an example of the flow of V-DMC encoding processing. FIG. 15 is a block diagram illustrating an example of the main configuration of a decoding device. FIG. 16 is a block diagram illustrating an example of the main configuration of a displacement vector application unit. FIG. 17 is a flowchart illustrating an example of the flow of decoding processing. FIG. 18 is a flowchart illustrating an example of the flow of displacement vector application processing. FIG. 19 is a block diagram illustrating an example of the main configuration of a computer.

[0013] Hereinafter, modes for carrying out the present disclosure (hereinafter referred to as embodiments) will be described. The description will be made in the following order: 1. Literature supporting technical content and technical terms 2. Coordinate system of displacement vector 3. Geometry patch coordinate system 4. First embodiment (encoding device) 5. Second embodiment (decoding device) 6. Supplementary notes

[0014] <1. Literature, etc. supporting technical content and technical terminology> The scope of disclosure of the present technology includes not only the content described in the embodiments, but also the content described in the following non-patent documents, etc. that were publicly known at the time of filing, and the content of other documents referenced in the following non-patent documents.

[0015] Non-patent document 1: (described above) Non-patent document 2: (described above)

[0016] In other words, the contents of the above-mentioned non-patent documents and the contents of other documents referenced in the above-mentioned non-patent documents are also used as the basis for determining the support requirements.

[0017] <2. Coordinate system of displacement vectors> <V-DMC> Conventionally, 3D data representing the three-dimensional structure of a three-dimensional structure (object with a three-dimensional shape) has been available as a mesh, which represents the three-dimensional shape of the object surface by forming polygons with vertices and connections (also called edges).

[0018] As shown in the upper left of Figure 1, in a mesh, vertices 11 and connections 12 connecting these vertices 11 form polygonal planes (polygons). These polygons (also called faces) represent the surface of a three-dimensional object, i.e., the three-dimensional shape of the object. A texture 13 can be applied to each face of this mesh.

[0019] Mesh data is composed of information such as that shown in the lower part of Figure 1. Vertex information 14, shown first from the left in the lower part of Figure 1, is information indicating the three-dimensional position (three-dimensional coordinates (X, Y, Z)) of each vertex 11 that constitutes the mesh. Connection information 15, shown second from the left in the lower part of Figure 1, is information indicating each connection (edge) 12 that constitutes the mesh. A texture image 16, shown third from the left in the lower part of Figure 1, is map information for the texture 13 that is applied to each face. A UV map 17, shown fourth from the left in the lower part of Figure 1, is information indicating the correspondence between the vertices 11 and the texture 13. The UV map 17 indicates the coordinates (UV coordinates) of each vertex 11 in the texture image 16.

[0020] As an example of a method for encoding such meshes, there is V-DMC (Video-based Dynamic Mesh Coding) as disclosed in Non-Patent Document 1.

[0021] In V-DMC, the mesh to be encoded (referred to in this specification as the original mesh) is represented as a base mesh that is less fine (i.e., coarser) than the original mesh, and displacement vectors of the division points obtained by subdividing the base mesh, and the base mesh and displacement vectors are then encoded.

[0022] For example, assume that there is an original mesh as shown in the top row of Figure 2. The original mesh is a mesh composed of vertices and connections that represent the three-dimensional structure of an object, and is the target of encoding. For example, the original mesh is generated from a captured image of an object in real space (by camera capture). In Figure 2, black dots represent vertices, and lines connecting the black dots represent connections (edges). As described above, a mesh essentially forms polygons using vertices and edges, but for convenience of explanation, it is described here as a group of vertices connected linearly (in series).

[0023] By simplifying the original mesh, a coarse (low-resolution) mesh like the one shown in the second row from the top of Figure 2 is formed. This is called the base mesh. One simplification method is to thin out some of the vertices (decimate). In other words, the base mesh is a mesh with lower resolution than the original mesh, generated by thinning out vertices from the original mesh (i.e., simplifying the original mesh).

[0024] By subdividing each polygon of this base mesh, vertices and edges are added, as shown in the third row from the top of Figure 2. The degree of subdivision is arbitrary. That is, the number of vertices and edges added is arbitrary. For example, this subdivision can add vertices equal to the number of vertices thinned out from the original mesh. That is, subdivision can be used to maintain the same number of vertices as the original mesh. In this specification, these added vertices are also referred to as division points. This subdivision can also be repeated recursively. For example, in a technique called midpoint, the process of adding vertices to the midpoints of edges (subdivision) is repeated recursively. In other words, recursive subdivision increases the number of vertices and improves the resolution of the mesh. In this way, it is possible to perform subdivision up to any desired level of resolution (i.e., control the resolution of the subdivided mesh). In other words, the subdivided mesh can be layered according to its level of resolution. In other words, this can be considered a layering of the subdivision process and the vertices (division points) and edges obtained by the subdivision process.

[0025] However, the connections of the base mesh are updated when the vertices of the original mesh are thinned out. Therefore, the division points obtained by subdivision are formed on these updated connections (edges). As a result, the shape of the subdivided base mesh differs from the shape of the original mesh. More specifically, as shown in the bottom part of Figure 2, the positions of the division points (on the dotted line) differ from those of the original mesh. In addition, the positions of the vertices of the base mesh may differ from those of the original mesh.

[0026] In other words, by moving the positions of the vertices of the subdivided base mesh (the vertices or division points of the base mesh) closer to the vertex positions of the original mesh, the difference in shape between the subdivided base mesh and the original mesh can be reduced. In this specification, such movement of the vertices of the subdivided base mesh (the vertices or division points of the base mesh) is also referred to as displacement. Furthermore, the amount and direction of this displacement, expressed as a vector, is also referred to as a displacement vector. Ideally, by displacing each vertex of the subdivided base mesh, the shape of the subdivided base mesh can be made to match the shape of the original mesh. In other words, the original mesh can be expressed as a base mesh and a displacement vector.

[0027] In V-DMC, such base meshes and displacement vectors are coded instead of the original mesh (geometry). By coding the base meshes and displacement vectors in this way, it is possible to code with a reduced number of polygons (i.e., the number of vertices and edges) compared to coding the original mesh, which generally reduces the amount of code for the same quality. In other words, it is possible to improve coding efficiency.

[0028] During decoding, as described above, a mesh is restored (generated) by subdividing a base mesh and applying a displacement vector to each vertex of the subdivided base mesh to displace it. In this specification, this mesh is also referred to as a decoded mesh. Attributes are then applied to the decoded mesh, and the decoded mesh with the applied attributes is rendered. While the shape of the polygon (face) may be any polygonal shape, the following description will be given assuming that the polygon is triangular. Therefore, the polygon (face) will also be referred to as a triangle in the following description.

[0029] <Encoding and Decoding of V-DMC Data> In the case of V-DMC, mesh data consists of a base mesh, displacement vectors, attributes, and atlas information. This data group is also referred to as V-DMC data. The base mesh consists of information indicating vertices and connections, and is coded using an existing mesh coding method such as Draco. Note that the base mesh can be coded not only intra-coded but also inter-coded. When inter-coding, the base mesh is coded as a motion vector between frames.

[0030] Displacement vectors are arithmetically coded or packed into a two-dimensional plane and encoded as video data using a coding method for 2D video. For example, when encoded as video data, displacement vectors are converted into displacement coefficients using a predetermined method. The displacement coefficients are arranged as pixel values ​​in a two-dimensional area (also called a displacement map). This arrangement (mapping) of displacement coefficients is also called packing. Video (also called displacement video) using the displacement map as frame images is encoded using a coding method for 2D video. In other words, displacement coefficients are scalar values ​​corresponding to the displacement vectors. A displacement map is map information (also called image data) that stores displacement coefficients as pixel values. Displacement video is video image data using the displacement map as frame images.

[0031] An attribute is non-geometry information applied to a mesh (geometry), which is 3D data. For example, an attribute may include a texture applied to a face of the mesh (geometry). The attribute (e.g., texture) is divided into multiple subregions, each of which is projected in a predetermined projection direction, and the projected images (patches) are arranged in a two-dimensional region (also called an attribute map). In other words, attribute patches are packed into the attribute map. A video (also called attribute video) using the attribute map as frame images is encoded using a 2D video encoding method. In other words, the attribute map is map information (also called image data) that stores the patches (projected textures) as pixel values. Attribute video is video data using the attribute map as frame images.

[0032] Atlas information is information used when reconstructing a mesh. For example, atlas information may include correspondence between the base mesh and a displacement map or attribute map (such as a UV map), quantized values ​​of displacement vectors, etc. This atlas information is encoded using a predetermined encoding method.

[0033] The coded data (bit stream) of each data is decoded by a decoding method corresponding to the coding method. In other words, by decoding the coded data (bit stream), various information such as base meshes, displacement vectors, attributes, and atlas information is restored (generated).

[0034] <Displacement Vector> V-DMC supports scalable decoding. Displacement vectors are layered for each level of resolution, as shown in FIG. 3, and packed as data for each layer. In the example of FIG. 3, LoD0 packed in the two-dimensional image 31 indicates the displacement vector data of the segmentation point in the highest layer (lowest resolution) of the segmentation points layered for each level of resolution. Similarly, LoD1 indicates the displacement vector data of the segmentation point in the layer immediately above LoD0. LoD2 indicates the displacement vector data of the segmentation point in the layer immediately above LoD1. In this way, the displacement vectors may be divided into layers (grouped into data for each layer) and packed.

[0035] Furthermore, the displacement vector may be one-dimensional information or multidimensional information. For example, when the displacement vector is one-dimensional information, the displacement vector is composed of one value. This value indicates the length of a predetermined directional component of the displacement vector. In other words, this predetermined direction becomes one of the coordinate axes of the coordinate system that expresses the displacement vector. In this specification, this coordinate axis (predetermined direction) in the coordinate system that expresses the displacement vector is also referred to as the "reference coordinate axis."

[0036] <Canonical Coordinate System> A coordinate system that represents such displacement vectors is the canonical coordinate system. A canonical coordinate system is a single coordinate system used by all vertices. For example, consider a mesh 40 as shown in FIG. 4. The mesh 40 has vertices 41 to 45 and the faces between them. In the canonical coordinate system, as shown in FIG. 4A, one coordinate system (i.e., the canonical coordinate system (x, y, z)) is applied to the displacement vector of each vertex. Therefore, if a displacement vector is considered to be one-dimensional information (i.e., the length of a specified directional component), the displacement vector 41A of vertex 41, the displacement vector 42A of vertex 42, the displacement vector 43A of vertex 43, the displacement vector 44A of vertex 44, and the displacement vector 45A of vertex 45 are all vectors in the same direction (the reference coordinate axis of the canonical coordinate system).

[0037] Therefore, depending on the displacement direction, the error in the displacement vector may increase, which may result in a decrease in restoration performance and a decrease in the subjective quality of the restored mesh. For example, the closer the vertex displacement direction is to a direction perpendicular to the reference coordinate axis of the canonical coordinate system, the greater the error in the displacement vector may be.

[0038] <Local Coordinate System> In contrast, local coordinate systems have also been used to represent displacement vectors. A local coordinate system is a coordinate system for each vertex. In other words, in this case, different local coordinate systems are applied to the displacement vectors of each vertex. Therefore, even if the displacement vector is one-dimensional information (i.e., the length of a specific directional component), the directions (reference coordinate axes) from displacement vector 41A to displacement vector 45A can be different from each other, as shown in FIG. 4B. For example, the reference coordinate axis may be the normal direction (normal) of each vertex, and the local coordinate system of each vertex may be a Cartesian coordinate system (normal, tangent, bitangent). Generally, the displacement of a vertex has the largest component in the normal direction of that vertex. Therefore, compared to when a canonical coordinate system is applied, the increase in error in the displacement vector can be suppressed.

[0039] However, in this case, during encoding, it was necessary to set a local coordinate system for each vertex to derive the displacement vector. Furthermore, during decoding, it was necessary to convert the displacement vector expressed in local coordinates into canonical coordinates in order to apply the displacement vector. In other words, it was necessary to set a local coordinate system during encoding and decoding. For example, as described above, if the local coordinate system of each vertex is a Cartesian coordinate system (normal, tangent, bitangent), it was necessary to calculate a normal vector for each vertex. The normal vector of a vertex (also referred to as a vertex normal vector) is derived by adding the normal vectors (also referred to as face normal vectors) of the faces surrounding the vertex. When deriving a vertex normal vector, for example, the following process is performed.

[0040] Step 1: The face normal vector of each face adjacent to the target vertex (the target vertex from which the vertex normal vector is derived) is calculated from the coordinates of each vertex of that face. Step 2: The face normal vectors of each face adjacent to the target vertex are added. Step 3: The result of Step 2 is normalized.

[0041] In the example of FIG. 5 , in step 1, a face normal vector 51A of a face 51 is derived using the coordinates of each vertex of the face 51. Similarly, a face normal vector 52A of a face 52 is derived using the coordinates of each vertex of the face 52. A face normal vector 53A of a face 53 is derived using the coordinates of each vertex of the face 53. A face normal vector 54A of a face 54 is derived using the coordinates of each vertex of the face 54. A face normal vector 55A of a face 55 is derived using the coordinates of each vertex of the face 55. A face normal vector 56A of a face 56 is derived using the coordinates of each vertex of the face 56.

[0042] In step 2, the face normal vector 56A is added to the face normal vector 51A. In step 3, the result of this addition is normalized to derive the vertex normal vector 57A of the target vertex 57.

[0043] As described above, in the encoding and decoding processes, it is necessary to perform such complicated calculations for each vertex to derive its normal vector, which may increase the load of the encoding and decoding processes.

[0044] <UV Atlas> As described above, textures are placed in an attribute map (two-dimensional region) and encoded. To reconstruct a mesh and apply the texture to each face of the mesh, it is necessary to establish a correspondence between the base mesh and the texture. In other words, the position (UV coordinates) in the attribute map (two-dimensional region) corresponding to each vertex of the base mesh is derived. In this specification, the process of generating UV coordinates for each vertex of the base mesh is also referred to as "unwrapping (UV unwrapping)."

[0045] V-DMC uses UVAtlas as a UV coordinate generation method (UV unwrapping method). When UVAtlas is used to unwrap a base mesh, atlas information 61 (see FIG. 6) is transmitted, for example. As shown in FIG. 6, the atlas information 61 includes information such as a vertex list, triangle list, UV list, and UV triangle list. The UV list is information indicating the UV coordinates (u, v) of each vertex of the base mesh. In other words, with UVAtlas, the UV coordinates of each vertex are explicitly transmitted.

[0046] <orthoAtlas> In recent years, orthoAtlas has been proposed as a UV coordinate generation method (UV unfolding method) different from UVAtlas. In orthoAtlas, UV coordinates are generated using orthogonal projection (six directions). In other words, UV coordinates can be derived from the projection direction, so they can be derived at the decoder. Therefore, there is no need to transmit UV coordinates.

[0047] That is, when orthoAtlas is applied, for example, the atlas information 62 in FIG. 6 is transmitted. As shown in FIG. 6, in the case of the atlas information 62, the UV list indicates the identification information (patch ID) of the patch to which each vertex belongs instead of the UV coordinates. This patch identification information is also referred to as a CC id (Connected Component ID). A projected image in which the texture is projected in a specified direction is mapped onto the attribute map. At this time, the texture is divided into small regions and projected to avoid overlapping or to improve resolution by projecting in a more optimal projection direction. These small regions are also referred to as patches. In particular, a texture patch is also referred to as a texture patch. Because CC ids can be expressed as integer values, the amount of code can be reduced compared to floating-point UV coordinates. In other words, the amount of code for this UV list can be reduced compared to the case of a UVAtlas.

[0048] However, since the projection direction cannot be determined by CC ID alone, patch information for each patch is transmitted as atlas information. Patch information is information about the mapping of the patch. For example, patch information (patch info) may include information indicating the projection direction of each patch (projId), information indicating the rotation angle (orientId), information indicating the 2D position (2dPosX, 2dPosY), information indicating the patch size (2dSizeX, 2dSizeY), information indicating the patch scale (magnification) (ScaleFlag(,Scale)), etc.

[0049] <Attribute Map> There are several texture packing methods available when using orthoAtlas, including default, small on top, Tetris, and projection. In other words, the texture placement in the attribute map can vary depending on the unwrapping and packing methods. FIG. 7 shows examples of attribute maps. In attribute maps 71, 72, and 73 shown in FIG. 7, the gray areas indicate placed texture patches. Attribute map 71 shows an example of an attribute map when UVAtlas is used. Attribute map 72 shows an example of an attribute map when orthoAtlas (default packing) is used. Attribute map 73 shows an example of an attribute map when orthoAtlas (projection packing) is used.

[0050] In this specification, the texture patches arranged in the attribute map are also referred to as UV connected regions. A UV connected region is a collection (block) of textures for each face on a two-dimensional plane where the textures are packed. In a UV connected region, the texture for each face is arranged so that it shares an edge with the texture of the adjacent face (connected to the adjacent face). In other words, a UV connected region is made up of a group of triangles that share an edge.

[0051] <3. Geometry Patch Coordinate System> <Reduction in Subjective Quality and Increased Load> As described above, when a canonical coordinate system is used to represent a displacement vector, the error of the displacement vector increases depending on the displacement direction, potentially reducing the subjective quality of the reconstructed mesh. In contrast, when a local coordinate system is used to represent a displacement vector, it is necessary to derive a normal vector for each vertex, potentially increasing the load of the encoding and decoding processes. In particular, the encoding process further includes a fitting process that establishes correspondence between the original mesh vertices and the vertices of the subdivided base mesh. More specifically, the encoding process involves dividing the original mesh to generate a kd-tree of vertices, performing a neighborhood search to establish correspondence with the vertices of the subdivided base mesh, deriving displacement vectors between the corresponding vertices, deriving normal vectors for the vertices, and packing the normal directions as the first components. In conventional methods, such fitting processing must be performed using three-dimensional information (vertex coordinates) (that is, by finding correspondence in three-dimensional space), which can increase the load.

[0052] <Method 1> Therefore, as shown in the top row of the table in Figure 8, the displacement vector is expressed in the geometry patch coordinate system (Method 1). The geometry patch coordinate system is a coordinate system derived for each geometry patch of the base mesh based on the canonical coordinate system. A geometry patch is a small area (patch) obtained by dividing the mesh (geometry). This small area is a collection of faces and vertices, and is composed of a face connected to at least one other face (i.e., a face that shares an edge with one or more adjacent faces) and the vertices belonging to that face.

[0053] For example, the base mesh 112 shown in Fig. 10 is obtained by decimating the original mesh 111 shown in Fig. 9. Small regions obtained by dividing the base mesh 112 (for example, a partial region surrounded by a rectangular frame 113 or a partial region surrounded by a rectangular frame 114) are also referred to as geometry patches. By setting a coordinate system (i.e., a geometry patch coordinate system) and setting reference coordinate axes for each such geometry patch of the base mesh 112 (or a subdivided base mesh 112), the direction of the displacement vector of each vertex can differ for each geometry patch.

[0054] For example, suppose there exists a mesh 130 as shown in FIG. 11 . The mesh 130 is composed of vertices 131 to 135 and the faces between them. Suppose this mesh 130 is divided into geometry patch A and geometry patch B. Geometry patch A is composed of vertices 131, 132, and 133, which are indicated by black circles, and faces indicated by solid lines. Geometry patch B is composed of vertices 134 and 135, which are indicated by white circles, and faces indicated by dotted lines. A geometry patch coordinate system and reference coordinate axes are set for each of geometry patch A and geometry patch B. Therefore, the displacement vector 131A of vertex 131, the displacement vector 132A of vertex 132, and the displacement vector 133A of vertex 133 are vectors that are in the same direction. Similarly, the displacement vector 134A of vertex 134 and the displacement vector 135A of vertex 135 are vectors that are in the same direction. However, the set of displacement vectors 131A, 132A, and 133A may have vectors in different directions from the set of displacement vectors 134A and 135A. In the example of Fig. 11, displacement vectors 131A, 132A, and 133A are vectors pointing upward in the figure, and displacement vectors 134A and 135A are vectors pointing rightward in the figure.

[0055] In this way, the directions of the reference coordinate axes (displacement vectors) can be diversified compared to the canonical coordinate system, thereby suppressing an increase in displacement vector errors and a decrease in the subjective quality of the restored mesh. Furthermore, unlike the local coordinate system, there is no need to derive normal vectors for each vertex and set a coordinate system. Furthermore, as described above, the geometry patch coordinate system can be derived based on the canonical coordinate system, so setting the geometry patch coordinate system is easier than in the case of the local coordinate system. Therefore, an increase in the load of the encoding and decoding processes can be suppressed.

[0056] Furthermore, as mentioned above, the encoding process (deriving the displacement vectors) requires a correspondence between the vertices of the original mesh and the vertices of the subdivided base mesh. In the case of the geometry patch coordinate system, this correspondence can be achieved using 2D information on the two-dimensional plane (also called the projection plane) onto which the mesh is projected.

[0057] For example, as shown in A of FIG. 12 , a depth map 141 of the original mesh on the projection plane is generated. The depth values ​​of this depth map 141 indicate the distance from the projection plane to the original mesh. Next, as shown in B of FIG. 12 , a depth map 142 on the projection plane is generated for the vertices of the subdivided base mesh. The depth values ​​of this depth map 142 indicate the distance from the projection plane to the vertices of the subdivided base mesh. In other words, this depth map 142 is composed only of depth values ​​at positions on the projection plane corresponding to the vertices (black dots in B of FIG. 12 ). Then, as shown in C of FIG. 12 , the positions of the depth map 141 and the depth map 142 are made to correspond to each other. By deriving the difference between the depth value of a vertex in the depth map 142 and the depth value of the same position in the depth map 141, the displacement vector of the vertex (the displacement from the subdivided base mesh to the original mesh) can be derived. In addition, when depth map 141 does not correspond to the vertex positions of depth map 142, as in the example C of Figure 12, the difference can be derived by matching nearby depth values ​​of depth map 141, as shown by the dotted arrow.

[0058] In this way, 2D information can be used to establish correspondence between the vertices of the original mesh and the vertices of the subdivided base mesh. Furthermore, displacement vectors can be derived simply by calculating the difference in depth values. Therefore, by applying the geometry patch coordinate system, the increase in the load of the fitting process can be suppressed compared to the case of the local coordinate system. In other words, the increase in the load of the encoding process can be suppressed.

[0059] Hereinafter, an information processing device (encoding device) that encodes 3D data including a base mesh and a displacement vector and generates a bitstream is also referred to as a first information processing device. For example, the first information processing device includes a geometry patch coordinate system setting unit that sets a geometry patch coordinate system derived for each geometry patch of the base mesh based on a canonical coordinate system, a displacement vector derivation unit that derives a displacement vector in the geometry patch coordinate system, a base mesh encoding unit that encodes the base mesh, and a displacement vector encoding unit that encodes the displacement vector.

[0060] In addition, in a first information processing method executed by a first information processing device, a geometry patch coordinate system is set that is derived for each geometry patch of a base mesh based on a canonical coordinate system, a displacement vector in the geometry patch coordinate system is derived, the base mesh is encoded, and the displacement vector is encoded.

[0061] The base mesh is a mesh with lower resolution than the original mesh to be coded, which is composed of vertices and connections that represent the three-dimensional structure of the object, and is generated by thinning out vertices from the original mesh. The displacement vector is vector information that indicates the displacement of the vertices of the subdivided base mesh.

[0062] By doing so, the first information processing device can suppress a decrease in subjective quality and suppress an increase in the load of encoding processing and decoding processing, as described above.

[0063] In the following description, an information processing device (decoding device) that decodes a bit stream of 3D data including a base mesh and a displacement vector and reconstructs the decoded mesh is also referred to as a second information processing device. For example, the second information processing device includes a base mesh decoding unit that decodes encoded data of the base mesh and generates a base mesh, a subdivision unit that subdivides the base mesh, a displacement vector decoding unit that decodes encoded data of the displacement vector and generates a displacement vector in a geometry patch coordinate system derived for each geometry patch of the base mesh based on a canonical coordinate system, a geometry patch coordinate system setting unit that sets a geometry patch coordinate system for each geometry patch of the subdivided base mesh, and a displacement vector application unit that applies a displacement vector to the subdivided base mesh using the geometry patch coordinate system.

[0064] In addition, in a second information processing method executed by a second information processing device, encoded data of a base mesh is decoded, a base mesh is generated, the base mesh is subdivided, encoded data of a displacement vector is decoded, a displacement vector in a geometry patch coordinate system derived for each geometry patch of the base mesh based on a canonical coordinate system is generated, a geometry patch coordinate system is set for each geometry patch of the subdivided base mesh, and the displacement vector is applied to the subdivided base mesh using the geometry patch coordinate system.

[0065] The base mesh is a mesh with lower resolution than the original mesh to be coded, which is composed of vertices and connections that represent the three-dimensional structure of the object, and is generated by thinning out vertices from the original mesh. The displacement vector is vector information that indicates the displacement of the vertices of the subdivided base mesh.

[0066] By doing so, the second information processing device can suppress a decrease in subjective quality and suppress an increase in the load of the decoding process, as described above.

[0067] <Geometry Patch Coordinate System> Note that any of the coordinate axes of the canonical coordinate system, or the direction of that coordinate axis rotated by a predetermined angle, may be used as the reference coordinate axis of the geometry patch coordinate system. In other words, the direction of the displacement vector in the geometry patch coordinate system may be any of the coordinate axes of the canonical coordinate system, or the direction of that coordinate axis rotated by a predetermined angle.

[0068] For example, in the first information processing device described above, the geometry patch coordinate system setting unit may set a geometry patch coordinate system whose reference coordinate axis is one of the coordinate axes of the canonical coordinate system or a coordinate axis obtained by rotating that coordinate axis by a predetermined angle. Also, in the second information processing device described above, the geometry patch coordinate system setting unit may set a geometry patch coordinate system whose reference coordinate axis is one of the coordinate axes of the canonical coordinate system or a coordinate axis obtained by rotating that coordinate axis by a predetermined angle. By doing so, the first information processing device and the second information processing device can more easily set the geometry patch coordinate system.

[0069] The "predetermined angle" which is the rotation angle of the coordinate axes may be, for example, 45 degrees, 90 degrees, or 135 degrees. In this way, the first information processing device and the second information processing device can more easily set the geometry patch coordinate system.

[0070] Note that the geometry patch may be the same as the texture patch. In this case, the patch information of the texture patch may be used as the patch information of the geometry patch. For example, in a first information processing device, a geometry patch coordinate system setting unit may set a geometry patch coordinate system in which the projection direction of the texture patch is the reference coordinate axis. Also, in a second information processing device, a geometry patch coordinate system setting unit may set a geometry patch coordinate system in which the projection direction of the texture patch is the reference coordinate axis. In this way, the first information processing device and the second information processing device can more easily set the geometry patch coordinate system.

[0071] For example, the texture patch may be expanded in orthoAtlas. For example, in a first information processing device, a geometry patch coordinate system setting unit may set a geometry patch coordinate system whose reference coordinate axes are the projection directions of the texture patch expanded in orthoAtlas. Also, in a second information processing device, a geometry patch coordinate system setting unit may set a geometry patch coordinate system whose reference coordinate axes are the projection directions of the texture patch expanded in orthoAtlas. In this way, the first information processing device and the second information processing device can more easily set the geometry patch coordinate systems.

[0072] <Displacement Vector> The displacement vector may be one-dimensional information consisting only of a component in the reference coordinate axis direction of the geometry patch coordinate system. For example, the displacement vector may be the amount of displacement in the normal direction of a vertex. Alternatively, the displacement vector may be three-dimensional information including a component in the reference coordinate axis direction of the geometry patch coordinate system. For example, the displacement vector may be composed of a normal component and its tangent component and bitangent component (normal, tangent, bitangent).

[0073] <Fitting Process> As described above, the fitting process may use 2D information to establish correspondence between the vertices of the original mesh and the vertices of the subdivided base mesh. For example, in the first information processing device, the displacement vector derivation unit may use a depth map of the original mesh on the projection plane onto which the original mesh is projected to derive displacement vectors for each vertex of the subdivided base mesh. This reduces the load of the fitting process compared to when 3D information is used to establish correspondence between vertices, as in the case of a local coordinate system. Furthermore, by using the depth map on the projection plane in this way, the displacement vector can be derived simply by calculating the difference between depth values. This reduces the load of the encoding process.

[0074] Note that, although the above description has been given of a two-dimensional plane from which a depth map is obtained as a projection surface, in reality, the original mesh and base mesh (i.e., geometry) are not projected onto the projection surface. In other words, the original mesh and base mesh are not packed onto the projection surface. Because the generation of a depth map for a geometry patch on a two-dimensional plane is similar to the projection of a texture patch, for convenience of explanation, in this specification, the two-dimensional plane is referred to as the projection surface. Note that the projection surface for this geometry patch may be set in any manner. For example, the projection surface for the geometry patch may be set in the same manner as the projection surface for the texture patch. For example, the projection surface for the geometry patch may be set in the same manner as the projection surface for the texture patch deployed in orthAtlas. For example, the projection surface for the geometry patch may be the same as the projection surface for the texture patch. In other words, the projection surface for the texture patch may be the projection surface for the geometry patch. For example, the projection surface for the texture patch deployed in orthAtlas may be the projection surface for the geometry patch.

[0075] <Geometry Patch Coordinate System Information> Information related to the geometry patch coordinate system may be transmitted. For example, the first information processing device may further include an atlas information encoding unit that encodes atlas information including geometry patch coordinate system information indicating the geometry patch coordinate system. This geometry patch coordinate system information may be generated in any manner. For example, the geometry patch coordinate system setting unit may generate this geometry patch coordinate system information, and the atlas information encoding unit may encode the geometry patch coordinate system information as atlas information.

[0076] The second information processing device may further include an atlas information decoding unit that decodes encoded data of the atlas information and generates atlas information including geometry patch coordinate system information indicating a geometry patch coordinate system. The geometry patch coordinate system information thus obtained may be used for any processing. For example, the geometry patch coordinate system setting unit may set a geometry patch coordinate system based on the geometry patch coordinate system information. Furthermore, the displacement vector application unit may refer to the geometry patch coordinate system information when applying a displacement vector.

[0077] By transmitting the geometry patch coordinate system information in this manner, it is possible to more easily set the geometry patch coordinate system and apply the displacement vector in the decoding process. In other words, it is possible to suppress an increase in the load of the decoding process. Note that if the geometry patch coordinate system can be set using information related to the projection of the texture patch, the transmission of the geometry patch coordinate system information may be omitted.

[0078] Note that the geometry patch coordinate system information may be information indicating a reference coordinate axis of the geometry patch coordinate system. For example, the geometry patch coordinate system information may be information indicating a reference coordinate axis of the geometry patch coordinate system using the coordinate axes of a canonical coordinate system. For example, the geometry patch coordinate system information may include information indicating a coordinate axis of the canonical coordinate system corresponding to the reference coordinate axis, information indicating a positive / negative direction, and information indicating a rotation direction. For example, the geometry patch coordinate system information may include an index corresponding to a combination of a coordinate axis of the canonical coordinate system corresponding to the reference coordinate axis, a positive / negative direction, and a rotation direction. This is the same in both the first information processing apparatus and the second information processing apparatus.

[0079] For example, as shown in Table 151 in FIG. 13 , an index (gp_coordinateID) is associated with a combination of a normal axis, a tangent axis, a bitangent axis, a normal sign, and a rotation axis, and the geometry patch coordinate system information may use this index (gp_coordinateID) to indicate (the reference coordinate axis of) the geometry patch coordinate system. The normal axis, the tangent axis, and the bitangent axis are information that define the three axes of the geometry patch coordinate system. The normal sign is information that defines whether the coordinate axis is positive or negative. The rotation axis is information that defines the rotation axis (rotation direction) of the coordinate axis.

[0080] In the example of Table 151, the normal axis, the tangent axis, and the bitangent axis are associated with different coordinate axes of the canonical coordinate system by indexes ("0," "1," or "2"). In other words, the coordinate axes of the canonical coordinate system are used to indicate (the reference coordinate axes of) the geometry patch coordinate system. Note that although each axis of the geometry patch coordinate system is shown, only the reference coordinate axis is shown, and the other coordinate axes may be derived based on the defined reference coordinate axis.

[0081] The normal sign (the direction of the coordinate axis) is defined as either a positive or negative direction by an index ("0" or "1"). For example, a normal sign of "0" indicates that the coordinate axis is in a positive direction (the same direction as in the canonical coordinate system). A normal sign of "1" indicates that the coordinate axis is in a negative direction (the opposite direction to that in the canonical coordinate system).

[0082] As shown in Table 152 in FIG. 13, the rotation axis (rotation direction) is defined by an index ("0," "1," "2," or "3") to indicate the direction of rotation. For example, a rotation axis of "0" indicates no rotation. A rotation axis of "1" indicates rotation around the y-axis (rotation axis). A rotation axis of "2" indicates rotation around the x-axis (rotation axis). A rotation axis of "3" indicates rotation around the z-axis (rotation axis).

[0083] Note that (the reference coordinate axis of) the geometry patch coordinate system may be defined using an existing index. For example, the projection direction of a texture patch may be defined as the reference coordinate axis of the geometry patch coordinate system, and (the reference coordinate axis of) the geometry patch coordinate system may be defined using an index (projectionID) indicating the projection direction of the texture patch. In other words, the geometry patch coordinate system information may include an index indicating the projection direction of the texture patch corresponding to a combination of a coordinate axis of a canonical coordinate system corresponding to the reference coordinate axis, a positive / negative direction, and a rotation direction. This is the same for both the first information processing device and the second information processing device. In this case, the patch information of the texture patch including the index may be reused, and transmission of the geometry patch coordinate system information may be omitted.

[0084] <Geometry Patch Correspondence Information> Information regarding the correspondence between faces and geometry patches may also be transmitted. For example, the first information processing device may further include an atlas information encoding unit that encodes atlas information including geometry patch correspondence information indicating the correspondence between faces of the base mesh and geometry patches. This geometry patch correspondence information may be generated in any manner. For example, the first information processing device may include a geometry patch generation unit that generates geometry patches, the geometry patch generation unit may generate this geometry patch correspondence information, and the atlas information encoding unit may encode the geometry patch coordinate system information as atlas information.

[0085] The second information processing device may further include an atlas information decoding unit that decodes encoded data of the atlas information and generates atlas information including geometry patch correspondence information indicating correspondence relationships between faces of the base mesh and geometry patches. The geometry patch correspondence information thus obtained may be used for any processing. For example, the second information processing device may include a geometry patch generating unit that generates geometry patches from the subdivided base mesh, and the geometry patch generating unit may generate the geometry patches based on the geometry patch correspondence information. Furthermore, the displacement vector application unit may refer to the geometry patch correspondence information when applying the displacement vector.

[0086] By transmitting the geometry patch correspondence information in this manner, it is possible to more easily generate geometry patches and apply displacement vectors in the decoding process, which in turn makes it possible to suppress an increase in the load of the decoding process.

[0087] The geometry patch correspondence information may include a UV list of the base mesh. For example, the UV list included in the atlas information 62 shown in FIG. 6 may be included in the geometry patch correspondence information. The geometry patch correspondence information may also include an attribute of the base mesh indicating identification information of a geometry patch corresponding to a face. In other words, information indicating the correspondence between a face and a geometry patch may be transmitted as an attribute of the base mesh. The geometry patch correspondence information may also include metadata indicating identification information of a geometry patch corresponding to a face. In other words, a list of identification information of a geometry patch corresponding to a face may be transmitted as metadata. This applies to both the first information processing apparatus and the second information processing apparatus.

[0088] <UV Coordinates of Each Vertex> Information regarding the UV coordinates of each vertex may also be transmitted. For example, the first information processing device may further include an atlas information encoding unit that encodes atlas information including the UV coordinates of the vertices of the base mesh.

[0089] The second information processing device may further include an atlas information decoding unit that decodes the encoded data of the atlas information and generates atlas information including UV coordinates of the vertices of the base mesh. The UV coordinates may be used for any processing.

[0090] By transmitting the UV coordinates of the vertices of the base mesh in this way, the vertex positions can be more easily determined during the decoding process, which means that the load of the decoding process can be reduced.

[0091] <Method 1-1> When the above-described method 1 is applied, for example, as shown in the second row from the top of the table in FIG. 8, the coordinate system of the displacement vector may be selectable (Method 1-1). That is, a coordinate system selected from among multiple candidates may be applied to the displacement vector. For example, a canonical coordinate system, a local coordinate system, and a geometry patch coordinate system may be prepared as candidates, and one of them may be selected and applied to the displacement vector. In this way, a more suitable coordinate system may be applied to the displacement vector, thereby suppressing a decrease in encoding efficiency.

[0092] In this case, for example, as shown in the table of Fig. 14, identification information (coordinate system ID) of the coordinate system may be prepared, and the identification information may be used to clearly indicate which coordinate system has been applied. In other words, the coordinate system ID applied to the displacement vector may be transmitted (for example, as atlas information).

[0093] For example, the first information processing device may further include an atlas information encoding unit that encodes atlas information including coordinate system identification information for identifying a coordinate system to be applied to the representation of the displacement vector. The second information processing device may further include an atlas information decoding unit that decodes the encoded data of the atlas information and generates atlas information including coordinate system identification information for identifying a coordinate system to be applied to the representation of the displacement vector. By transmitting such a coordinate system ID, the decoding side can more easily determine which coordinate system has been applied. Therefore, an increase in the load of the decoding process can be suppressed.

[0094] Note that the coordinate system to be applied to this displacement vector may be specified for any data unit, such as a sequence, a scene, an object, or a tile. In other words, the range in which the specified coordinate system is applied to the displacement vector may be the entire data or may be limited. In other words, the coordinate system ID indicating the coordinate system applied to the displacement vector may be transmitted for any data unit. For example, the coordinate system ID may be transmitted for each sequence, each scene, each object, or each tile. Furthermore, this coordinate system ID may be transmitted for multiple types of data units.

[0095] <Method 1-2> A vertex located on the boundary of a geometry patch is also called a geometry patch boundary vertex. When a mesh is decomposed into geometry patches, this geometry patch boundary vertex can belong to multiple geometry patches. For example, the mesh shown in FIG. 15A has vertex V1, vertex V2, vertex V3, vertex V4, vertex V5, face F12, face F23, face F34, and face F45. Suppose this mesh is divided into geometry patch A and geometry patch B. Geometry patch A includes vertex V1, vertex V2, face F12, and face F23. Geometry patch B includes vertex V4, vertex V5, face F34, and face F45.

[0096] In this case, vertex V3 is a geometry patch boundary vertex, and its position means that it can belong to either geometry patch A or geometry patch B. A coordinate system for such a geometry patch boundary vertex may be set using a method different from that for other vertices. That is, when the above-mentioned method 1 is applied, a coordinate system for the displacement vector of the geometry patch boundary vertex may be set (method 1-2), for example, as shown in the third row from the top of the table in FIG.

[0097] <Method 1-2-1> When this method 1-2 is applied, any of the geometry patch coordinate systems to which the geometry patch boundary vertex may belong may be applied to the displacement vector of the geometry patch boundary vertex, as shown in the fourth row from the top of the table in Figure 8 (Method 1-2-1).

[0098] For example, in the first information processing device, the displacement vector derivation unit may derive the displacement vector of a geometry patch boundary vertex by applying the geometry patch coordinate system of any geometry patch that may include the geometry patch boundary vertex. For example, as shown in B of FIG. 15 , vertex V3 may belong to geometry patch A, and the displacement vector of vertex V3 may be derived by applying the geometry patch coordinate system of geometry patch A. In this case, the displacement vector of vertex V3 will be a vector with the same direction as the displacement vectors of vertices V1 and V2. Of course, vertex V3 may also belong to geometry patch B. In that case, the displacement vector of vertex V3 will be a vector with the same direction as the displacement vectors of vertices V4 and V5.

[0099] In this way, by making a geometry patch boundary vertex belong to a geometry patch, it is possible to easily apply the geometry patch coordinate system to the geometry patch boundary vertex. When decoding, the geometry patch coordinate system of the geometry patch to which the geometry patch boundary vertex belongs is also applied, so that the geometry patch boundary vertex can be processed in the same way as other vertices.

[0100] For example, in the first information processing device, the displacement vector derivation unit may select a geometry patch to which a geometry patch boundary vertex belongs in accordance with the decoding order, and derive a displacement vector of the geometry patch boundary vertex in the geometry patch coordinate system of the selected geometry patch. For example, in B of Fig. 15, vertex V3 may be assigned to geometry patch A based on the fact that geometry patch A is decoded before geometry patch B. Of course, the geometry patch boundary vertex may also be assigned to a geometry patch that is decoded later.

[0101] In this way, the geometry patch to which the boundary vertex of the geometry patch belongs can be selected in accordance with the decoding order of the geometry patch, thereby making the selection easy.

[0102] For example, in a first information processing apparatus, a displacement vector derivation unit may select a geometry patch in which the normal direction of a face adjacent to a geometry patch boundary vertex is closest to the reference coordinate axis of the geometry patch coordinate system of the geometry patch to which the face belongs, and derive a displacement vector of the geometry patch boundary vertex in the geometry patch coordinate system of the selected geometry patch. For example, in FIG. 15B, the reference coordinate axis PA of the geometry patch coordinate system of geometry patch A and the normal direction N23 of the face F23 adjacent to vertex V3 of geometry patch A are identical. In contrast, the reference coordinate axis PB of the geometry patch coordinate system of geometry patch B and the normal direction N34 of the face F34 adjacent to vertex V3 of geometry patch B are not identical. In other words, the normal direction of the face adjacent to the geometry patch boundary vertex in geometry patch A is closer to the reference coordinate axis of the geometry patch coordinate system of the geometry patch to which the face belongs in geometry patch A than in geometry patch B. Therefore, based on this, vertex V3 may be assigned to geometry patch A, and its displacement vector may be derived by applying the geometry patch coordinate system of geometry patch A.

[0103] By selecting a geometry patch to which a geometry patch boundary vertex belongs in consideration of the normal direction of the adjacent face in this way, it is possible to apply a geometry patch coordinate system with a reference coordinate axis that is closer to the normal direction of the geometry patch boundary vertex. This makes it possible to suppress an increase in the error of the displacement vector. It also makes it possible to suppress an increase in the amount of coding.

[0104] <Method 1-2-2> When Method 1-2 is applied, a local coordinate system may be applied to the displacement vector of the boundary vertex of the geometry patch (Method 1-2-2), as shown in the fifth row from the top of the table in Fig. 8. In other words, a coordinate system may be set for the boundary vertex of the geometry patch without making the boundary vertex belong to the geometry patch.

[0105] For example, in a first information processing device, the geometry patch coordinate system setting unit may further set a local coordinate system, which is a coordinate system for each vertex, for the geometry patch boundary vertices.The displacement vector derivation unit may further derive a displacement vector for the geometry patch boundary vertex by applying the local coordinate system.Also, in a second information processing device, the displacement vector for the geometry patch boundary vertex of a subdivided base mesh may be expressed using a local coordinate system, which is a coordinate system for each vertex.The geometry patch coordinate system setting unit may further set a local coordinate system for the geometry patch boundary vertex.The displacement vector application unit may further apply a displacement vector to the geometry patch boundary vertex of the subdivided base mesh using the local coordinate system.

[0106] By doing so, the geometry patch coordinate system setting unit can set a more suitable coordinate system for the geometry patch boundary vertices. For example, the geometry patch coordinate system setting unit can set a local coordinate system whose reference coordinate axis is closer to the normal direction of the geometry patch boundary vertex. Therefore, it is possible to suppress an increase in the error of the displacement vector. It is also possible to suppress an increase in the amount of code.

[0107] Note that any method may be used to set the local coordinates for the geometry patch boundary vertices. For example, in the first information processing device or the second information processing device, the geometry patch coordinate system setting unit may set the local coordinate system using the normals of the faces adjacent to the geometry patch boundary vertices. For example, the geometry patch coordinate system setting unit may set the average of the normals of the faces adjacent to the geometry patch boundary vertices as the reference coordinate axes of the local coordinate system. Furthermore, the geometry patch coordinate system setting unit may set the weighted average of the normals of the faces adjacent to the geometry patch boundary vertices as the reference coordinate axes of the local coordinate system. For example, weighting may be performed based on the area of ​​the faces.

[0108] Furthermore, in the first information processing device or the second information processing device, the geometry patch coordinate system setting unit may set a local coordinate system using reference coordinate axes of the geometry patch coordinate systems of geometry patches that may include the geometry patch boundary vertices. For example, the geometry patch coordinate system setting unit may set the average of the reference coordinate axes of the geometry patch coordinate systems of geometry patches that may include the geometry patch boundary vertices as the reference coordinate axes of the local coordinate system. Furthermore, the geometry patch coordinate system setting unit may set the weighted average of the reference coordinate axes of the geometry patch coordinate systems of geometry patches that may include the geometry patch boundary vertices as the reference coordinate axes of the local coordinate system. For example, weighting may be performed based on the area of ​​the geometry patch.

[0109] <Method 1-2-3> When Method 1-2 is applied, the geometry patch boundary vertex may be duplicated (Method 1-2-3), as shown in the sixth row from the top of the table in Fig. 8. In other words, the geometry patch boundary vertex may be duplicated and assigned to each geometry patch that may contain the geometry patch boundary vertex.

[0110] In this case, in each geometry patch, the geometry patch coordinate system is applied to the geometry patch boundary vertices. For example, as shown in C of FIG. 15 , vertex V3, which is a geometry patch boundary vertex, is duplicated and assigned to both geometry patch A and geometry patch B. Therefore, a displacement vector is derived for vertex V3 in each of geometry patches A and B. If the directions of the (reference coordinate axes of) the geometry patch coordinate systems of geometry patch A and geometry patch B are different from each other, the derived displacement vectors for vertex V3 will be vectors with different directions from each other. Therefore, during decoding, these multiple displacement vectors are combined and then applied. Alternatively, each displacement vector may be applied to a geometry patch boundary vertex, and the multiple application results may be combined.

[0111] For example, in the first information processing device, the displacement vector derivation unit may derive a displacement vector of a geometry patch boundary vertex in a geometry patch coordinate system of each geometry patch that may include the geometry patch boundary vertex, and in the second information processing device, the displacement vector application unit may further combine and apply displacement vectors in multiple geometry patch coordinate systems to the geometry patch boundary vertex of the subdivided base mesh.

[0112] <Method 1-3> When the above-described method 1 is applied, the value of the displacement vector may be the difference from the projection surface (Method 1-3), for example, as shown in the bottom row of the table in FIG. 8 . The displacement vector indicates the displacement of the vertices of the subdivided base mesh, as shown in FIG. 16A. Therefore, as shown in this figure, the displacement vectors (DV1 to DV3) should essentially be the differences between the vertex positions of the original mesh and the vertex positions of the subdivided base mesh. However, in this case, it is necessary to associate the vertices between the original mesh and the subdivided base mesh. Therefore, as shown in FIG. 16B, the distance (depth value) between the original mesh and the projection surface of the geometry patch may be used as a displacement vector.

[0113] For example, in the first information processing device or the second information processing device, the displacement vector may be vector information whose length is the depth value from the projection plane onto which the original mesh is projected to a vertex of the original mesh.

[0114] This eliminates the need to associate vertices between the original mesh and the subdivided base mesh, thereby reducing the load of the encoding and decoding processes. When applying a displacement vector to the subdivided base mesh in the decoding process, the distance (depth value) between the subdivided base mesh and the projection surface may be subtracted from the displacement vector. To prevent a reduction in the accuracy of the displacement vector, other correction processes may be applied to the displacement vector.

[0115] In this case, information about the projection surface may be transmitted. For example, the first information processing device may further include an atlas information encoding unit that encodes atlas information including projection surface information about the position and orientation of the projection surface. Furthermore, the second information processing device may further include an atlas information decoding unit that decodes the encoded data of the atlas information and generates atlas information including projection surface information about the position and orientation of the projection surface.

[0116] This makes it easier to identify the position and orientation of the projection surface when applying the displacement vector to the subdivided base mesh in the decoding process, and therefore makes it easier to apply the displacement vector.

[0117] <Scope of application of explanation> In this specification, an explanation given for a higher-level method also applies to lower-level methods belonging to that method, unless a contradiction arises. For example, when it is explained that "Method 1 may be applied," it means that any of Method 1-1, Method 1-2, and Method 1-3 may be applied. Of course, lower-level methods (e.g., Method 1-2-1, Method 1-2-2, Method 1-2-3, etc.) may also be applied.

[0118] <Combination> Each of the above-described methods may be applied in combination with any other method as long as no contradiction occurs. Three or more methods may be applied in combination. Furthermore, techniques that can be combined may include not only those shown in the table of FIG. 8 as "methods," but also all elements described in this specification. Furthermore, each of the above-described methods may be applied in combination with methods other than those described above.

[0119] <4. First embodiment> <Encoding device> The present technology can be applied to any device. For example, the present technology can be applied to an encoding device that encodes a mesh and generates a bitstream. FIG. 17 is a block diagram showing an example of the configuration of an encoding device, which is one aspect of an information processing device to which the present technology is applied. The encoding device 300 (first information processing device) shown in FIG. 17 is a device that encodes a mesh and generates a bitstream thereof. Therefore, the encoding device 300 can also be said to be a bitstream generation device that generates a bitstream.

[0120] Fig. 17 shows the main processing units, data flows, etc., but is not limited to all that is shown in Fig. 17. In other words, in encoding device 300, there may be processing units that are not shown as blocks in Fig. 17, and there may be processing and data flows that are not shown as arrows, etc. in Fig. 17.

[0121] The encoding device 300 encodes a mesh using a method essentially similar to the V-DMC method described in the aforementioned non-patent document, except that the present technology is applied. For example, the encoding device 300 acquires an original mesh to be encoded and an attribute map including a texture corresponding to the original mesh. Note that the original mesh includes not only information about the mesh's geometry but also information indicating the correspondence with the attribute map (e.g., a UV list). The encoding device 300 encodes the original mesh and attribute map using the V-DMC method, generates a V-DMC bitstream, and outputs it.

[0122] As shown in Fig. 17 , the encoding device 300 (first information processing device) has a preprocessing unit 311 and a V-DMC encoding unit 312. The preprocessing unit 311 performs preprocessing related to encoding. As shown in Fig. 9 , the preprocessing unit 311 has a base mesh generation unit 321, an atlas information generation unit 322, and a displacement vector generation unit 323.

[0123] The base mesh generation unit 321 performs processing related to the generation of a base mesh. For example, the base mesh generation unit 321 may obtain an original mesh to be input to the encoding device 300. The base mesh generation unit 321 may also perform decimation processing (thinning out vertices) on the original mesh to generate a base mesh. The base mesh generation unit 321 may supply the generated base mesh together with the original mesh to the atlas information generation unit 322.

[0124] The atlas information generation unit 322 performs processing related to the generation of atlas information corresponding to the base mesh. For example, the atlas information generation unit 322 may acquire the base mesh or original mesh supplied from the base mesh generation unit 321. The atlas information generation unit 322 may generate the atlas information by UV unwrapping the base mesh, etc. The atlas information generation unit 322 may supply the generated atlas information to the displacement vector generation unit 323 together with the base mesh, etc.

[0125] The displacement vector generation unit 323 performs processing related to the generation of displacement vectors. For example, the displacement vector generation unit 323 may acquire a base mesh, atlas information, etc. supplied from the atlas information generation unit 322. The displacement vector generation unit 323 may also acquire an original mesh input to the encoding device 300. The displacement vector generation unit 323 may also use this information to generate displacement vectors that displace vertices of the subdivided base mesh. The displacement vector generation unit 323 may supply the generated displacement vectors to the V-DMC encoding unit 312 together with the base mesh, atlas information, etc.

[0126] The V-DMC encoder 312 performs processing related to encoding of V-DMC data. For example, the V-DMC encoder 312 may acquire an original mesh input to the encoding device 300. The V-DMC encoder 312 may also acquire a base mesh, displacement vectors, atlas information, and the like, supplied from the displacement vector generator 323. The V-DMC encoder 312 may also acquire an attribute map input to the encoding device 300. The V-DMC encoder 312 may use this information to encode the atlas information, base mesh, displacement vectors, and attribute map, respectively, and generate the respective encoded data. Therefore, the V-DMC encoder 312 can also be referred to as an encoder. The V-DMC encoder 312 may also multiplex these encoded data as substreams to generate a single bitstream. This bitstream is also referred to as a V-DMC bitstream. Therefore, the V-DMC encoder 312 can also be referred to as a bitstream generator (or V-DMC bitstream generator). The V-DMC encoding section 312 may output the generated V-DMC bitstream to the outside of the encoding device 300 .

[0127] <Displacement Vector Generator> Fig. 18 is a block diagram showing an example of the main configuration of the displacement vector generator 323. Note that Fig. 18 shows the main processing units, data flows, etc., and is not limited to what is shown in Fig. 18. In other words, the displacement vector generator 323 may include processing units that are not shown as blocks in Fig. 18, or processes or data flows that are not shown as arrows, etc. in Fig. 18.

[0128] As shown in FIG. 18, the displacement vector generation unit 323 includes a subdivision unit 331 , a geometry patch generation unit 332 , a geometry patch coordinate system setting unit 333 , and a displacement vector derivation unit 334 .

[0129] The subdivision unit 331 executes processing related to subdivision of the faces of a base mesh. For example, the subdivision unit 331 may acquire a base mesh or the like supplied from the atlas information generation unit 322. The subdivision unit 331 may subdivide (the faces of) the base mesh to generate division points. In other words, the subdivision unit 331 may subdivide the base mesh generated by the base mesh generation unit 321. The subdivision unit 331 may supply the "subdivided base mesh" to the geometry patch generation unit 332 together with the base mesh or the like.

[0130] The geometry patch generation unit 332 executes processing related to the generation of geometry patches. For example, the geometry patch generation unit 332 may acquire a "subdivided base mesh" or a base mesh supplied from the subdivision unit 331. The geometry patch generation unit 332 may acquire atlas information supplied from the atlas information generation unit 322. The geometry patch generation unit 332 may divide the subdivided base mesh into multiple connected regions to generate geometry patches. Note that any method may be used to divide the subdivided base mesh (a method for generating geometry patches). The geometry patch generation unit 332 may also generate geometry patches using atlas information. The geometry patch generation unit 332 may generate information about the generated geometry patches (e.g., geometry patch correspondence information). The geometry patch generation unit 332 may supply the information about the generated geometry patches to the geometry patch coordinate system setting unit 333 together with the subdivided base mesh or the base mesh.

[0131] The geometry patch coordinate system setting unit 333 executes processing related to setting of the geometry patch coordinate system. For example, the geometry patch coordinate system setting unit 333 may acquire information about the geometry patch, a subdivided base mesh, a base mesh, etc., supplied from the geometry patch generation unit 332. The geometry patch coordinate system setting unit 333 may acquire atlas information supplied from the atlas information generation unit 322. The geometry patch coordinate system setting unit 333 may set a geometry patch coordinate system for each geometry patch. Note that any method may be used to set this geometry patch coordinate system. Alternatively, the geometry patch coordinate system setting unit 333 may set the geometry patch coordinate system using atlas information. Alternatively, the geometry patch coordinate system setting unit 333 may set the geometry patch coordinate system using information about the geometry patch. The geometry patch coordinate system setting unit 333 may generate information about the set geometry patch coordinate system (e.g., geometry patch coordinate system information, coordinate system identification information, etc.). The geometry patch coordinate system setting unit 333 may supply information about the generated geometry patch coordinate system to the displacement vector derivation unit 334 together with information about the geometry patch, the subdivided base mesh, the base mesh, and the like.

[0132] The displacement vector derivation unit 334 performs processing related to the derivation of a displacement vector. For example, the displacement vector derivation unit 334 may acquire information about the geometry patch coordinate system, information about the geometry patch, a subdivided base mesh, a base mesh, etc., supplied from the geometry patch coordinate system setting unit 333. The displacement vector derivation unit 334 may also acquire an original mesh input to the encoding device 300. The displacement vector derivation unit 334 may also acquire atlas information supplied from the atlas information generation unit 322. The displacement vector derivation unit 334 may use the original mesh to derive a displacement vector of a subdivided base mesh. Note that any method may be used to derive this displacement vector. The displacement vector derivation unit 334 may also use atlas information to derive a displacement vector of a subdivided base mesh. The displacement vector derivation unit 334 may also use information about the geometry patch coordinate system (i.e., the set geometry patch coordinate system) to derive a displacement vector of a subdivided base mesh. The displacement vector derivation unit 334 may also use information about the geometry patch (i.e., the generated geometry patch) to derive the displacement vector of the subdivided base mesh.

[0133] The displacement vector derivation unit 334 may supply the derived displacement vector to the V-DMC encoding unit 312. The displacement vector derivation unit 334 may supply a base mesh or the like to the V-DMC encoding unit 312. The displacement vector derivation unit 334 may supply atlas information to the V-DMC encoding unit 312. The atlas information may include information about the geometry patches, information about the geometry patch coordinate system, and the like.

[0134] <V-DMC Encoder> Fig. 19 is a block diagram showing an example of the main configuration of the V-DMC encoder 312. Note that Fig. 19 shows the main processing units, data flows, etc., and does not necessarily show everything. In other words, the V-DMC encoder 312 may include processing units that are not shown as blocks in Fig. 19, and may include processing and data flows that are not shown as arrows, etc. in Fig. 19.

[0135] As shown in FIG. 19, the V-DMC encoding unit 312 has an atlas information encoding unit 351, a base mesh encoding unit 352, a displacement vector correction unit 353, a displacement vector encoding unit 354, a mesh reconstruction unit 355, an attribute map conversion unit 356, an attribute encoding unit 357, and a multiplexing unit 358.

[0136] The atlas information encoding unit 351 performs processing related to encoding of atlas information. For example, the atlas information encoding unit 351 may acquire atlas information supplied from the displacement vector generation unit 323. The atlas information may include information related to geometry patches, information related to the geometry patch coordinate system, and the like. The atlas information encoding unit 351 may also encode the acquired atlas information using a predetermined encoding method to generate encoded data of the atlas information. The atlas information encoding unit 351 may also supply the generated encoded data of the atlas information to the multiplexing unit 358.

[0137] The base mesh encoding unit 352 performs processing related to encoding of the base mesh. For example, the base mesh encoding unit 352 may acquire a base mesh supplied from the displacement vector generation unit 323. The base mesh encoding unit 352 may acquire atlas information supplied from the displacement vector generation unit 323. The base mesh encoding unit 352 may quantize the acquired base mesh and encode it using a predetermined encoding method (e.g., Draco) to generate encoded data of the base mesh. The base mesh encoding unit 352 may supply the generated encoded data of the base mesh to the displacement vector correction unit 353. The base mesh encoding unit 352 may also supply the generated encoded data of the base mesh to the multiplexing unit 358.

[0138] The displacement vector correction unit 353 performs processing related to the correction of the displacement vector. For example, the displacement vector correction unit 353 may acquire a base mesh and a displacement vector supplied from the displacement vector generation unit 323. Alternatively, the displacement vector correction unit 353 may acquire encoded data of the base mesh supplied from the base mesh encoding unit 352. The displacement vector correction unit 353 may correct the displacement vector based on this information. For example, the displacement vector correction unit 353 may decode the acquired encoded data of the base mesh, compare the base mesh before and after encoding to determine encoding distortion of the base mesh, and correct the displacement vector in accordance with the encoding distortion. The displacement vector correction unit 353 may supply the corrected displacement vector to the displacement vector encoding unit 354. Alternatively, the displacement vector correction unit 353 may dequantize the decoded base mesh and supply it to the mesh reconstruction unit 355.

[0139] The displacement vector encoding unit 354 performs processing related to encoding of displacement vectors. For example, the displacement vector encoding unit 354 may acquire displacement vectors supplied from the displacement vector correction unit 353. The displacement vector encoding unit 354 may also generate a displacement map by wavelet transforming the displacement vectors, quantizing them, and packing them into a two-dimensional region. The displacement vector encoding unit 354 may also generate a displacement video in which the displacement map is used as frame images. In other words, the displacement video is a moving image in which the frame images are a displacement map, which is a two-dimensional region in which displacement vectors are packed. The displacement vector encoding unit 354 may also encode the generated displacement video using a predetermined encoding method for 2D moving images to generate encoded data of displacement vectors (displacement video). The displacement vector encoding unit 354 may also supply the encoded data of the displacement vectors generated in this manner to the multiplexing unit 358. The displacement vector encoding unit 354 may also decode the generated encoded data, unpack the displacement vectors from the displacement map, and dequantize the displacement vectors. The displacement vector encoding unit 354 may supply the dequantized displacement vector to the mesh reconstruction unit 355 .

[0140] The displacement vector encoding unit 354 may arithmetically encode the displacement vector to generate encoded data of the displacement vector. In this case, the displacement vector encoding unit 354 may arithmetically decode the encoded data to generate a displacement vector and supply the generated displacement vector to the mesh reconstruction unit 355.

[0141] The mesh reconstruction unit 355 performs processing related to mesh reconstruction. For example, the mesh reconstruction unit 355 may acquire a base mesh supplied from the displacement vector correction unit 353. The mesh reconstruction unit 355 may also acquire a displacement vector supplied from the displacement vector encoding unit 354. The mesh reconstruction unit 355 may use these to reconstruct a mesh. The mesh reconstruction unit 355 may supply the reconstructed mesh to the attribute map conversion unit 356.

[0142] The attribute map conversion unit 356 performs processing related to attribute map conversion. For example, the attribute map conversion unit 356 may acquire a reconstructed mesh supplied from the mesh reconstruction unit 355. The attribute map conversion unit 356 may also acquire atlas information supplied from the displacement vector generation unit 323. The attribute map conversion unit 356 may also acquire an original mesh and attribute map input to the encoding device 300. The attribute map conversion unit 356 may convert the acquired attribute map based on other acquired information. For example, the attribute map conversion unit 356 may convert the attribute map based on the atlas information, the original mesh, etc., so that it corresponds to the reconstructed mesh. In other words, the attribute map conversion unit 356 can be said to generate a converted attribute map. Therefore, the attribute map conversion unit 356 can also be said to be an attribute map generation unit. The attribute map conversion unit 356 may supply the converted attribute map to the attribute encoding unit 357 .

[0143] The attribute encoding unit 357 performs processing related to encoding of attributes. For example, the attribute encoding unit 357 may acquire an attribute map supplied from the attribute map conversion unit 356. The attribute encoding unit 357 may also generate attribute video using the acquired attribute map as frame images. The attribute encoding unit 357 may also encode the generated attribute video using a predetermined encoding method for 2D video to generate encoded data of attributes. The attribute encoding unit 357 may also supply the generated encoded data of attributes to the multiplexing unit 358.

[0144] The multiplexing unit 358 performs processing related to multiplexing of encoded data (substreams). For example, the multiplexing unit 358 may acquire encoded data of atlas information supplied from the atlas information encoding unit 351. Alternatively, the multiplexing unit 358 may acquire encoded data of base meshes supplied from the base mesh encoding unit 352. Alternatively, the multiplexing unit 358 may acquire encoded data of displacement vectors supplied from the displacement vector encoding unit 354. Alternatively, the multiplexing unit 358 may acquire encoded data of attributes supplied from the attribute encoding unit 357. The multiplexing unit 358 may multiplex these pieces of encoded data as substreams to generate a V-DMC bitstream. Therefore, the multiplexing unit 358 can also be referred to as a bitstream generation unit (or V-DMC bitstream generation unit). The multiplexing unit 358 may output the generated V-DMC bitstream to the outside of the encoding device 300. For example, the multiplexing unit 358 may supply the V-DMC bitstream to a decoding device 400 (described later). Therefore, the multiplexing unit 358 can also be said to be a supply unit (providing unit) for the V-DMC bitstream.

[0145] The present technology described above may be applied to the encoding device 300 configured as described above. For example, the encoding device 300 may apply the above-described method 1. For example, in the encoding device 300, the geometry patch coordinate system setting unit 333 may set a geometry patch coordinate system derived for each geometry patch of a base mesh based on a canonical coordinate system. Then, the displacement vector derivation unit 334 may derive a displacement vector in the geometry patch coordinate system. Then, the base mesh encoding unit 352 may encode the base mesh. Then, the displacement vector encoding unit 354 may encode the displacement vector.

[0146] With this configuration, the encoding device 300 can suppress a decrease in subjective quality and suppress an increase in the load of encoding and decoding processes.

[0147] <Flow of Encoding Process> An example of the flow of the encoding process executed by the encoding device 300 will be described with reference to the flowchart of FIG.

[0148] When the encoding process starts, in step S301, the base mesh generation unit 321 of the encoding device 300 decimates the original mesh to be encoded and generates a base mesh.

[0149] In step S302, the atlas information generating unit 322 generates atlas information for the base mesh.

[0150] In step S303, the displacement vector generation unit 323 executes a displacement vector generation process and generates a displacement vector by applying the above-described method 1. That is, the displacement vector generation unit 323 generates a displacement vector using the geometry patch coordinate system.

[0151] In step S304, the V-DMC encoding unit 312 performs a V-DMC encoding process, applying the above-described method 1 to encode the V-DMC data and generate a V-DMC bitstream.

[0152] The encoding process ends when the process of step S304 is completed. The encoding device 300 executes such encoding process for each frame of the original mesh.

[0153] <Flow of Displacement Vector Generation Processing> Next, an example of the flow of the displacement vector generation processing executed in step S303 of FIG. 20 will be described with reference to the flowchart of FIG.

[0154] When the displacement vector generation process starts, the subdivision unit 331 subdivides the base mesh in step S321.

[0155] In step S322, the geometry patch generator 332 divides the subdivided base mesh to generate geometry patches.

[0156] In step S323, the geometry patch coordinate system setting unit 333 applies the above-described method 1 to set a geometry patch coordinate system for each of the generated geometry patches, with the canonical coordinate system as the reference.

[0157] In step S324, the displacement vector derivation unit 334 applies the above-mentioned method 1 to derive a displacement vector in the geometry patch coordinate system.

[0158] When the process of step S324 ends, the displacement vector generation process ends, and the process returns to FIG.

[0159] <Flow of V-DMC Encoding Process> Next, an example of the flow of the V-DMC encoding process executed in step S304 of FIG. 20 will be described with reference to the flowchart of FIG.

[0160] When the V-DMC encoding process starts, the atlas information encoding unit 351 encodes atlas information including priority control information in step S341. This atlas information may include information about geometry patches and information about the geometry patch coordinate system.

[0161] In step S342, the base mesh encoding unit 352 applies the above-described method 1 to encode the base mesh.

[0162] In step S343, the displacement vector correction unit 353 corrects the displacement vector.

[0163] In step S344, the displacement vector encoding unit 354 applies the above-described method 1 to encode the corrected displacement vector. This displacement vector is a displacement vector in the geometry patch coordinate system. For example, the displacement vector encoding unit 354 may pack the displacement vector into a displacement video and encode it using a 2D encoding method. Alternatively, the displacement vector encoding unit 354 may arithmetically encode the displacement vector.

[0164] In step S345, the mesh reconstructing unit 355 reconstructs the mesh.

[0165] In step S346, the attribute map conversion unit 356 converts the attribute map.

[0166] In step S347, the attribute encoding unit 357 encodes the attribute video using the attribute map as a frame image.

[0167] In step S348, the multiplexing unit 358 multiplexes the coded data of the atlas information, the coded data of the base mesh, the coded data of the displacement vector, and the coded data of the attribute to generate a V-DMC bitstream.

[0168] When the process of step S348 ends, the V-DMC encoding process ends, and the process returns to FIG. 20 .

[0169] By performing each process as described above, the encoding device 300 can suppress a decrease in subjective quality and suppress an increase in the load of the encoding process and the decoding process.

[0170] 5. Second embodiment Decoding device The present technology can be applied to a decoding device that decodes encoded data of a mesh. Fig. 23 is a block diagram showing an example of the configuration of a decoding device that is one aspect of an information processing device to which the present technology is applied. The decoding device 400 (second information processing device) shown in Fig. 23 is a device that decodes, for example, encoded data of a mesh generated in the encoding device 300 (Fig. 17) (a V-DMC bitstream generated by the multiplexing unit 358 (Fig. 19)) and reconstructs a decoded mesh.

[0171] Fig. 23 shows the main processing units, data flows, etc., but does not necessarily include all of them. In other words, in the decoding device 400, there may be processing units that are not shown as blocks in Fig. 23, and there may be processing and data flows that are not shown as arrows, etc. in Fig. 23.

[0172] The decoding device 400 decodes coded data of a mesh that has been coded using a method essentially similar to the V-DMC method described in the aforementioned non-patent document, except that the present technology is applied, and reconstructs the decoded mesh. For example, the decoding device 400 obtains a V-DMC bitstream. This V-DMC bitstream may be generated by, for example, the coding device 300. As part of the reconstruction process, the decoding device 400 decodes the V-DMC bitstream and reconstructs a mesh (also referred to as a decoded mesh). The decoding device 400 also applies a texture to the decoded mesh, generates a display image for displaying the decoded mesh, and outputs the display image to an external device. For example, the decoding device 400 supplies the display image to an external display device for display.

[0173] As shown in Figure 23, the decoding device 400 (second information processing device) has a demultiplexing unit 411, an atlas information decoding unit 412, a base mesh decoding unit 413, a subdivision unit 414, a displacement vector decoding unit 415, a displacement vector application unit 416, an attribute decoding unit 417, an attribute application unit 418, and a display processing unit 419.

[0174] The demultiplexing unit 411 performs demultiplexing processing. For example, the demultiplexing unit 411 may acquire a V-DMC bitstream to be decoded and supplied to the decoding device 400. The demultiplexing unit 411 may also demultiplex the acquired V-DMC bitstream to extract coded data of atlas information, coded data of base meshes, coded data of displacement vectors, and coded data of attributes. Therefore, the demultiplexing unit 411 can also be considered an acquirer of a V-DMC bitstream or various information contained in the V-DMC bitstream. The demultiplexing unit 411 may supply the coded data of the extracted atlas information to the atlas information decoding unit 412. The demultiplexing unit 411 may also supply the coded data of the extracted base meshes to the base mesh decoding unit 413. The demultiplexing unit 411 may also supply the coded data of the extracted displacement vectors to the displacement vector decoding unit 415. Furthermore, the demultiplexing unit 411 may supply the coded data of the extracted attributes to the attribute decoding unit 417 .

[0175] The atlas information decoding unit 412 performs processing related to decoding of the atlas information. For example, the atlas information decoding unit 412 may acquire encoded data of the atlas information supplied from the demultiplexing unit 411. The atlas information decoding unit 412 may also decode the acquired encoded data of the atlas information to generate (restore) atlas information. This atlas information may include information about the geometry patches and information about the geometry patch coordinate system. The atlas information decoding unit 412 may supply the generated atlas information to the base mesh decoding unit 413. The atlas information decoding unit 412 may supply the generated atlas information to the subdivision unit 414. The atlas information decoding unit 412 may supply the generated atlas information to the displacement vector decoding unit 415. The atlas information decoding unit 412 may supply the generated atlas information to the displacement vector application unit 416. The atlas information decoding unit 412 may supply the generated atlas information to the attribute decoding unit 417. The atlas information decoder 412 may provide the generated atlas information to the attribute application unit 418 .

[0176] The base mesh decoding unit 413 performs processing related to decoding of the base mesh. For example, the base mesh decoding unit 413 may acquire coded data of the base mesh supplied from the demultiplexing unit 411. The base mesh decoding unit 413 may also decode the acquired coded data (bit stream) of the base mesh using a predetermined decoding method (e.g., Draco) to generate (restore) a base mesh (e.g., a vertex list, a triangle list, etc.). In this case, the base mesh decoding unit 413 may acquire atlas information supplied from the atlas information decoding unit 412 and decode the coded data of the base mesh based on the atlas information. The base mesh decoding unit 413 may also supply the generated base mesh to the subdivision unit 414.

[0177] The subdivision unit 414 performs processing related to subdivision of the faces of the base mesh. For example, the subdivision unit 414 may obtain a base mesh supplied from the base mesh decoding unit 413. The subdivision unit 414 may subdivide (the faces of) the base mesh to generate division points. In this case, the subdivision unit 414 may obtain atlas information supplied from the atlas information decoding unit 412 and perform subdivision of the base mesh based on the atlas information. The subdivision unit 414 may supply the subdivided base mesh to the displacement vector application unit 416.

[0178] The displacement vector decoding unit 415 performs processing related to the decoding of displacement vectors. For example, the displacement vector decoding unit 415 may acquire encoded data of displacement vectors (i.e., a displacement bitstream) supplied from the demultiplexing unit 411. The displacement vector decoding unit 415 may decode the encoded data (bitstream) of displacement vectors to generate (restore) displacement vectors. For example, if the displacement vectors are encoded as displacement video, the displacement vector decoding unit 415 may decode the encoded data of the displacement vectors using a predetermined decoding method for 2D video, generate (restore) the displacement video, and unpack the displacement vectors from a displacement map, which is a frame image of the displacement video. Furthermore, if the displacement video is arithmetically coded, the displacement vector decoding unit 415 may arithmetically decode the encoded data of the displacement vectors to generate displacement vectors. In this case, the displacement vector decoding unit 415 may acquire atlas information supplied from the atlas information decoding unit 412 and decode the encoded data of the displacement vectors based on the atlas information. The displacement vector decoding unit 415 may supply the displacement vector obtained in this manner to the displacement vector application unit 416 .

[0179] The displacement vector application unit 416 performs processing related to application of a displacement vector to a subdivided base mesh. For example, the displacement vector application unit 416 may acquire a subdivided base mesh supplied from the subdivision unit 414. The displacement vector application unit 416 may acquire a displacement vector supplied from the displacement vector decoding unit 415. The displacement vector application unit 416 may apply a displacement vector to the vertices of the subdivided base mesh. In other words, the displacement vector application unit 416 may generate (reconstruct) a decoded mesh. In this case, the displacement vector application unit 416 may acquire atlas information supplied from the atlas information decoding unit 412 and apply a displacement vector based on the atlas information. The displacement vector application unit 416 may supply the decoded mesh generated in this manner to the attribute application unit 418.

[0180] The attribute decoding unit 417 executes processing related to attribute decoding. For example, the attribute decoding unit 417 may acquire coded attribute data supplied from the demultiplexing unit 411. The attribute decoding unit 417 may also decode the acquired coded attribute data using a predetermined decoding method for 2D video to generate (restore) attribute video. In this case, the attribute decoding unit 417 may acquire atlas information supplied from the atlas information decoding unit 412 and decode attributes based on the atlas information. The attribute decoding unit 417 may also supply an attribute map, which is a frame image of the generated attribute video, to the attribute application unit 418.

[0181] The attribute application unit 418 performs processing related to the application of attributes to the decoded mesh. For example, the attribute application unit 418 may acquire the decoded mesh supplied from the displacement vector application unit 416. The attribute application unit 418 may acquire an attribute map supplied from the attribute decoding unit 417. The attribute application unit 418 may apply attributes of the attribute map to the decoded mesh. In this case, the attribute application unit 418 may acquire atlas information supplied from the atlas information decoding unit 412 and apply attributes to the decoded mesh based on the atlas information. The attribute application unit 418 may supply the decoded mesh to which the attributes have been applied in this manner to the display processing unit 419.

[0182] The display processing unit 419 performs processing related to mesh display. For example, the display processing unit 419 may acquire a decoded mesh to which attributes have been applied, supplied from the attribute application unit 418. The display processing unit 419 may render the acquired decoded mesh and generate a display image for displaying the decoded mesh. The display processing unit 419 may then supply the generated display image to an external device outside the decoding device 400, and display the display image on another device or the like.

[0183] <Displacement Vector Application Unit> Fig. 24 is a block diagram showing an example of the main configuration of the displacement vector application unit 416. Note that Fig. 24 shows the main processing units, data flows, etc., and does not necessarily show everything. In other words, the displacement vector application unit 416 may include processing units that are not shown as blocks in Fig. 24, or processes or data flows that are not shown as arrows, etc. in Fig. 24.

[0184] As shown in FIG. 24, the displacement vector application unit 416 includes a geometry patch generation unit 431 , a geometry patch coordinate system setting unit 432 , and a displacement vector application unit 433 .

[0185] The geometry patch generation unit 431 executes processing related to the generation of geometry patches. For example, the geometry patch generation unit 431 may acquire a "subdivided base mesh" supplied from the subdivision unit 414. The geometry patch generation unit 431 may divide the subdivided base mesh into multiple connected regions to generate geometry patches. Note that any method may be used to divide the subdivided base mesh (method of generating geometry patches). For example, the geometry patch generation unit 431 may acquire atlas information supplied from the atlas information decoding unit 412 and generate geometry patches based on information about the geometry patches included in the atlas information. The geometry patch generation unit 432 may supply the generated geometry patches to the geometry patch coordinate system setting unit 432 together with the subdivided base mesh, etc.

[0186] The geometry patch coordinate system setting unit 432 executes processing related to setting of a geometry patch coordinate system. For example, the geometry patch coordinate system setting unit 432 may acquire a geometry patch, a subdivided base mesh, etc. supplied from the geometry patch generation unit 431. The geometry patch coordinate system setting unit 432 may set a geometry patch coordinate system for each geometry patch. Note that any method for setting this geometry patch coordinate system may be used. For example, the geometry patch coordinate system setting unit 432 may acquire atlas information supplied from the atlas information decoding unit 412 and set the geometry patch coordinate system based on information related to the geometry patch coordinate system, coordinate system identification information, etc. included in the atlas information. The geometry patch coordinate system setting unit 333 may supply the set geometry patch coordinate system to the displacement vector application unit 433 together with the geometry patch, the subdivided base mesh, etc.

[0187] The displacement vector application unit 433 executes processing related to the derivation of a displacement vector. For example, the displacement vector application unit 433 may acquire a geometry patch coordinate system, a geometry patch, a subdivided base mesh, or the like, supplied from the geometry patch coordinate system setting unit 432. The displacement vector application unit 433 may also acquire a displacement vector supplied from the displacement vector decoding unit 415. This displacement vector is expressed in the geometry patch coordinate system. The displacement vector application unit 433 may apply the displacement vector to the subdivided base mesh using the geometry patch coordinate system. For example, the displacement vector application unit 433 may acquire atlas information supplied from the atlas information decoding unit 412 and apply the displacement vector to the subdivided base mesh based on the atlas information. In other words, the displacement vector application unit 433 may generate a decoded mesh. The displacement vector application unit 433 may supply the decoded mesh generated by applying the displacement vector in this manner to the attribute application unit 418.

[0188] The above-described present technology may be applied to a decoding device 400 configured as described above. For example, the decoding device 400 may apply the above-described method 1. For example, in the decoding device 400, the base mesh decoding unit 413 may decode encoded data of a base mesh to generate the base mesh. Then, the subdivision unit 414 may subdivide the base mesh. Then, the displacement vector decoding unit 415 may decode encoded data of a displacement vector to generate a displacement vector in a geometry patch coordinate system derived for each geometry patch of the base mesh based on the canonical coordinate system. Then, the geometry patch coordinate system setting unit 432 may set a geometry patch coordinate system for each geometry patch of the subdivided base mesh. Then, the displacement vector application unit 433 may apply a displacement vector to the subdivided base mesh using the geometry patch coordinate system.

[0189] With this configuration, the decoding device 400 can suppress a decrease in subjective quality and suppress an increase in the load of the encoding process and the decoding process.

[0190] <Flow of Decoding Process> An example of the flow of the decoding process executed by the decoding device 400 will be described with reference to the flowchart of FIG.

[0191] When the decoding process starts, the demultiplexing unit 411 of the decoding device 400 demultiplexes the V-DMC bitstream in step S401.

[0192] In step S402, the atlas information decoding unit 412 decodes the encoded data of the atlas information to generate (restore) the atlas information. This atlas information may include information about the geometry patches and information about the geometry patch coordinate system.

[0193] In step S403, the base mesh decoding unit 413 applies the above-described method 1 to decode the coded data of the base mesh and generate (restore) the base mesh.

[0194] In step S404, the subdivision unit 414 applies the above-mentioned method 1 to subdivide the base mesh and generate division points.

[0195] In step S405, the displacement vector decoding unit 415 applies the above-mentioned method 1 to decode the encoded data (bit stream) of the displacement vector and generate (restore) a displacement vector in the geometry patch coordinate system derived for each geometry patch of the base mesh based on the canonical coordinate system.

[0196] In step S406, the displacement vector application unit 416 applies the above-described method 1 to perform a displacement vector application process, applying displacement vectors to the vertices of the subdivided base mesh, and generating a decoded mesh.

[0197] In step S407, the attribute decoding unit 417 decodes the coded data of the attribute and generates (restores) an attribute map.

[0198] In step S408, the attribute application unit 418 applies the attributes contained in the attribute map to the decoded mesh.

[0199] In step S409, the display processing unit 419 renders the decoded mesh to generate a display image.

[0200] When the process of step S409 is completed, the decoding process is completed. The decoding device 400 executes such a decoding process for each frame of the original mesh.

[0201] <Flow of Displacement Vector Application Processing> Next, an example of the flow of the displacement vector application processing executed in step S406 of FIG. 25 will be described with reference to the flowchart of FIG.

[0202] When the displacement vector application process is started, the geometry patch generation unit 431 divides the subdivided base mesh in step S421 to generate geometry patches.

[0203] In step S422, the geometry patch coordinate system setting unit 432 applies the above-described method 1 to set a geometry patch coordinate system for each of the generated geometry patches, with the canonical coordinate system as the reference.

[0204] In step S423, the displacement vector application unit 433 applies the above-described method 1 and uses the geometry patch coordinate system to apply the displacement vector in the geometry patch coordinate system to the subdivided base mesh.

[0205] When the process of step S423 ends, the displacement vector application process ends, and the process returns to FIG.

[0206] By performing each process as described above, the decoding device 400 can suppress a decrease in subjective quality and suppress an increase in the load of the encoding process and the decoding process.

[0207] 6. Supplementary Notes Polygon Shape In the above description, the polygon shape is described as being triangular, but this shape is merely an example. The polygon shape may be any polygonal shape.

[0208] <Encoding Method> In the above, V-DMC has been used as an example of an encoding method to which the present technology can be applied, but the present technology is not limited to this example, and can be applied to any encoding method that encodes a base mesh, a displacement vector, an attribute map including texture, atlas information, or information equivalent thereto.

[0209] <Computer> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the programs that make up the software are installed on a computer. Here, the term "computer" includes computers built into dedicated hardware, and general-purpose personal computers, etc., that can execute various functions by installing various programs.

[0210] FIG. 27 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0211] In a computer 900 shown in FIG. 27, a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, and a RAM (Random Access Memory) 903 are interconnected via a bus 904.

[0212] An input / output interface 910 is also connected to the bus 904. To the input / output interface 910, an input unit 911, an output unit 912, a storage unit 913, a communication unit 914, and a drive 915 are connected.

[0213] The input unit 911 includes, for example, a keyboard, a mouse, a microphone, a touch panel, and an input terminal. The output unit 912 includes, for example, a display, a speaker, and an output terminal. The storage unit 913 includes, for example, a hard disk, a RAM disk, and a non-volatile memory. The communication unit 914 includes, for example, a network interface. The drive 915 drives removable media 921 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0214] In a computer configured as described above, the CPU 901 performs the above-described series of processes by, for example, loading a program stored in the storage unit 913 into the RAM 903 via the input / output interface 910 and the bus 904 and executing the program. The RAM 903 also stores data necessary for the CPU 901 to execute various processes as appropriate.

[0215] The program executed by the computer can be applied by recording it on, for example, a removable medium 921 such as a package medium. In this case, the program can be installed in the storage unit 913 via the input / output interface 910 by inserting the removable medium 921 into the drive 915.

[0216] This program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, digital satellite broadcasting, etc. In this case, the program can be received by the communication unit 914 and installed in the storage unit 913.

[0217] Alternatively, this program can be installed in advance in the ROM 902 or the storage unit 913 .

[0218] <Application of the Present Technology> The present technology can be applied to any configuration. For example, the present technology can be applied to various electronic devices.

[0219] Furthermore, for example, the present technology can also be implemented as part of an apparatus, such as a processor (e.g., a video processor) as a system LSI (Large Scale Integration), a module using multiple processors (e.g., a video module), a unit using multiple modules (e.g., a video unit), or a set in which other functions are added to a unit (e.g., a video set).

[0220] Furthermore, for example, the present technology can also be applied to a network system configured with multiple devices. For example, the present technology may be implemented as cloud computing in which multiple devices share and collaborate on processing via a network. For example, the present technology may be implemented in a cloud service that provides image (video)-related services to any terminal, such as a computer, an AV (Audio Visual) device, a portable information processing terminal, or an IoT (Internet of Things) device.

[0221] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are housed in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0222] <Fields and uses to which this technology can be applied> Systems, devices, processing units, etc. to which this technology is applied can be used in any field, for example, transportation, medical care, crime prevention, agriculture, livestock farming, mining, beauty, factories, home appliances, weather, nature monitoring, etc. In addition, the uses thereof are also arbitrary.

[0223] <Others> In this specification, a "flag" refers to information for identifying multiple states, and includes not only information used to identify two states, true (1) or false (0), but also information capable of identifying three or more states. Therefore, the value that this "flag" can take may be, for example, two values, 1 / 0, or three or more values. That is, the number of bits constituting this "flag" is arbitrary, and may be one bit or multiple bits. Furthermore, identification information (including flags) can be included not only in a bitstream, but also in a bitstream that includes differential information of the identification information relative to certain reference information. Therefore, in this specification, "flag" and "identification information" encompass not only the information itself, but also differential information relative to the reference information.

[0224] Furthermore, various types of information (e.g., metadata) related to the coded data (bitstream) may be transmitted or recorded in any form as long as they are associated with the coded data. Here, the term "associate" means, for example, that one piece of data can be used (linked) when processing the other piece of data. That is, the associated pieces of data may be combined into one piece of data or may be separate pieces of data. For example, information associated with coded data (image) may be transmitted over a transmission path separate from that of the coded data (image). Furthermore, for example, information associated with coded data (image) may be recorded on a recording medium separate from that of the coded data (image) (or on a different recording area of ​​the same recording medium). Note that this "association" may refer not to the entire data, but to only part of the data. For example, an image and information corresponding to that image may be associated with each other in any unit, such as multiple frames, one frame, or a portion of a frame.

[0225] In this specification, terms such as "composite," "multiplex," "add," "integrate," "include," "store," "embed," "insert," and the like refer to combining multiple items into one, such as combining encoded data and metadata into one piece of data, and refer to one method of "associating" as described above.

[0226] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present technology.

[0227] For example, a configuration described as one device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, configurations described above as multiple devices (or processing units) may be combined and configured as one device (or processing unit). Of course, configurations other than those described above may be added to the configuration of each device (or each processing unit). Furthermore, as long as the configuration and operation of the entire system are substantially the same, part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0228] Furthermore, for example, the above-described program may be executed in any device, as long as the device has the necessary functions (functional blocks, etc.) and is able to obtain the necessary information.

[0229] Also, for example, each step of a single flowchart may be executed by a single device, or may be shared and executed by multiple devices. Furthermore, when a single step includes multiple processes, the multiple processes may be executed by a single device, or may be shared and executed by multiple devices. In other words, multiple processes included in a single step can be executed as multiple step processes. Conversely, processes described as multiple steps can be executed collectively as a single step.

[0230] For example, the steps of a program executed by a computer may be executed in chronological order in the order described herein, or may be executed in parallel or individually at the required timing, such as when a call is made. In other words, as long as no contradiction occurs, the steps may be executed in an order different from the order described above. Furthermore, the steps of this program may be executed in parallel with the processing of another program, or may be executed in combination with the processing of another program.

[0231] Furthermore, for example, multiple technologies related to the present technology can be implemented independently and independently, as long as no contradiction occurs. Of course, any multiple technologies can also be implemented in combination. For example, part or all of the present technology described in any embodiment can be implemented in combination with part or all of the present technology described in another embodiment. Furthermore, part or all of any of the above-described present technologies can be implemented in combination with other technologies not described above.

[0232] The present technology can also be configured as follows. (1) An information processing device including: a geometry patch coordinate system setting unit that sets a geometry patch coordinate system derived for each geometry patch of a base mesh based on a canonical coordinate system; a displacement vector derivation unit that derives a displacement vector in the geometry patch coordinate system; a base mesh encoding unit that encodes the base mesh; and a displacement vector encoding unit that encodes the displacement vector, wherein the base mesh is a mesh with lower resolution than an original mesh to be encoded, the original mesh being composed of vertices and connections that represent a three-dimensional structure of an object, and the displacement vector is vector information indicating the displacement of vertices of the subdivided base mesh. (2) The information processing device described in (1), wherein the geometry patch coordinate system setting unit sets the geometry patch coordinate system using any of the coordinate axes of the canonical coordinate system or coordinate axes obtained by rotating the coordinate axes at a predetermined angle as reference coordinate axes. (3) The information processing device described in (2), wherein the predetermined angle is any of 45 degrees, 90 degrees, and 135 degrees. (4) The information processing device according to (3), wherein the geometry patch coordinate system setting unit sets the geometry patch coordinate system with the projection direction of the texture patch as the reference coordinate axis. (5) The information processing device according to (4), wherein the geometry patch coordinate system setting unit sets the geometry patch coordinate system with the projection direction of the texture patch expanded in an orthoAtlas as the reference coordinate axis. (6) The information processing device according to any of (1) to (5), wherein the displacement vector is one-dimensional information consisting only of a component in the reference coordinate axis direction of the geometry patch coordinate system. (7) The information processing device according to any of (1) to (5), wherein the displacement vector is three-dimensional information including a component in the reference coordinate axis direction of the geometry patch coordinate system.(8) The information processing device according to any one of (1) to (7), wherein the displacement vector derivation unit derives the displacement vector of each vertex of the subdivided base mesh using a depth map of the original mesh in a projection plane onto which the original mesh is projected. (9) The information processing device according to any one of (1) to (8), further comprising an atlas information encoding unit that encodes atlas information including geometry patch coordinate system information indicating the geometry patch coordinate system. (10) The information processing device according to (9), wherein the geometry patch coordinate system information indicates reference coordinate axes of the geometry patch coordinate system. (11) The information processing device according to (10), wherein the geometry patch coordinate system information indicates the reference coordinate axes using coordinate axes of the canonical coordinate system. (12) The information processing device according to (11), wherein the geometry patch coordinate system information includes information indicating coordinate axes of the canonical coordinate system corresponding to the reference coordinate axes, information indicating a positive / negative direction, and information indicating a rotation direction. (13) The information processing device according to (12), wherein the geometry patch coordinate system information includes an index corresponding to a combination of a coordinate axis of the canonical coordinate system corresponding to the reference coordinate axis, a positive / negative direction, and a rotation direction. (14) The information processing device according to (13), wherein the index indicates a projection direction of a texture patch. (15) The information processing device according to any of (1) to (14), further comprising an atlas information encoding unit that encodes atlas information including geometry patch correspondence information indicating a correspondence between faces of the base mesh and the geometry patches. (16) The information processing device according to (15), wherein the geometry patch correspondence information includes a UV list of the base mesh. (17) The information processing device according to (15), wherein the geometry patch correspondence information includes an attribute of the base mesh indicating identification information of the geometry patch corresponding to the face. (18) The information processing device according to (15), wherein the geometry patch correspondence information includes metadata indicating identification information of the geometry patch corresponding to the face.(19) The information processing device according to any one of (1) to (18), further comprising an atlas information encoding unit that encodes atlas information including UV coordinates of vertices of the base mesh. (20) The information processing device according to any one of (1) to (19), further comprising an atlas information encoding unit that encodes atlas information including coordinate system identification information for identifying a coordinate system to be applied to express the displacement vector. (21) The information processing device according to any one of (1) to (20), wherein the displacement vector derivation unit derives the displacement vector of a geometry patch boundary vertex by applying the geometry patch coordinate system of any of the geometry patches that may include the geometry patch boundary vertex. (22) The information processing device according to (21), wherein the displacement vector derivation unit selects the geometry patch to which the geometry patch boundary vertex belongs in accordance with a decoding order, and derives the displacement vector of the geometry patch boundary vertex in the geometry patch coordinate system of the selected geometry patch. (23) The information processing device according to (21), wherein the displacement vector derivation unit selects the geometry patch in which a normal direction of a face adjacent to the geometry patch boundary vertex is closest to a reference coordinate axis of the geometry patch coordinate system of the geometry patch to which the face belongs, and derives the displacement vector of the geometry patch boundary vertex in the geometry patch coordinate system of the selected geometry patch. (24) The information processing device according to any of (1) to (23), wherein the geometry patch coordinate system setting unit further sets a local coordinate system that is a coordinate system for each vertex for the geometry patch boundary vertex, and the displacement vector derivation unit further derives the displacement vector of the geometry patch boundary vertex by applying the local coordinate system. (25) The information processing device according to (24), wherein the geometry patch coordinate system setting unit sets the local coordinate system using a normal of a face adjacent to the geometry patch boundary vertex. (26) The information processing device according to (24), wherein the geometry patch coordinate system setting unit sets the local coordinate system using a reference coordinate axis of the geometry patch coordinate system of the geometry patch that may include the geometry patch boundary vertex.(27) The information processing device according to any of (1) to (26), wherein the displacement vector derivation unit derives the displacement vector of the geometry patch boundary vertex in the geometry patch coordinate system of each of the geometry patches that may include the geometry patch boundary vertex. (28) The information processing device according to any of (1) to (27), wherein the displacement vector is vector information whose length is a depth value from a projection plane onto which the original mesh is projected to a vertex of the original mesh. (29) The information processing device according to (28), further comprising an atlas information encoding unit that encodes atlas information including projection plane information related to the position and orientation of the projection plane. (30) An information processing method, comprising: setting a geometry patch coordinate system derived for each geometry patch of a base mesh based on a canonical coordinate system; deriving a displacement vector in the geometry patch coordinate system; encoding the base mesh; encoding the displacement vector; the base mesh is a mesh with lower resolution than an original mesh to be encoded, which is composed of vertices and connections that represent a three-dimensional structure of an object, and is generated by thinning out vertices from the original mesh; and the displacement vector is vector information indicating the displacement of the vertices of the subdivided base mesh.

[0233] (41) An information processing device comprising: a base mesh decoding unit that decodes encoded data of a base mesh and generates the base mesh; a subdivision unit that subdivides the base mesh; a displacement vector decoding unit that decodes encoded data of a displacement vector and generates the displacement vector in a geometry patch coordinate system derived for each geometry patch of the base mesh with a canonical coordinate system as a reference; a geometry patch coordinate system setting unit that sets the geometry patch coordinate system for each geometry patch of the subdivided base mesh; and a displacement vector application unit that applies the displacement vector to the subdivided base mesh using the geometry patch coordinate system, wherein the base mesh is a mesh with lower resolution than the original mesh to be encoded, which is composed of vertices and connections that represent a three-dimensional structure of an object, and is generated by thinning out vertices from the original mesh, (42) The information processing device according to (41), wherein the geometry patch coordinate system setting unit sets the geometry patch coordinate system using any of the coordinate axes of the canonical coordinate system or coordinate axes obtained by rotating the coordinate axes at a predetermined angle as reference coordinate axes. (43) The information processing device according to (42), wherein the predetermined angle is any of 45 degrees, 90 degrees, and 135 degrees. (44) The information processing device according to (43), wherein the geometry patch coordinate system setting unit sets the geometry patch coordinate system using the projection direction of a texture patch as the reference coordinate axis. (45) The information processing device according to (44), wherein the geometry patch coordinate system setting unit sets the geometry patch coordinate system using the projection direction of the texture patch expanded in an orthoAtlas as the reference coordinate axis. (46) The information processing device according to any of (41) to (45), wherein the displacement vector is one-dimensional information consisting only of a component in the reference coordinate axis direction of the geometry patch coordinate system. (47) The information processing device according to any one of (41) to (45), wherein the displacement vector is three-dimensional information including a component in the direction of a reference coordinate axis of the geometry patch coordinate system.(48) The information processing device according to any one of (41) to (47), further comprising an atlas information decoding unit that decodes encoded data of atlas information and generates the atlas information including geometry patch coordinate system information that indicates the geometry patch coordinate system. (49) The information processing device according to (48), in which the geometry patch coordinate system information indicates reference coordinate axes of the geometry patch coordinate system. (50) The information processing device according to (49), in which the geometry patch coordinate system information indicates the reference coordinate axes using coordinate axes of the canonical coordinate system. (51) The information processing device according to (50), in which the geometry patch coordinate system information includes information indicating coordinate axes of the canonical coordinate system that correspond to the reference coordinate axes, information indicating positive / negative directions, and information indicating a rotation direction. (52) The information processing device according to (51), in which the geometry patch coordinate system information includes an index that corresponds to a combination of the coordinate axis of the canonical coordinate system that corresponds to the reference coordinate axis, the positive / negative direction, and the rotation direction. (53) The information processing device according to (52), in which the index indicates a projection direction of a texture patch. (54) The information processing device according to any one of (41) to (53), further comprising an atlas information decoding unit that decodes encoded data of atlas information and generates the atlas information including geometry patch correspondence information that indicates correspondence between faces of the base mesh and the geometry patches. (55) The information processing device according to (54), wherein the geometry patch correspondence information includes a UV list of the base mesh. (56) The information processing device according to (54), wherein the geometry patch correspondence information includes an attribute of the base mesh that indicates identification information of the geometry patch that corresponds to the face. (57) The information processing device according to (54), wherein the geometry patch correspondence information includes metadata that indicates identification information of the geometry patch that corresponds to the face. (58) The information processing device according to any one of (41) to (57), further comprising an atlas information decoding unit that decodes encoded data of atlas information and generates the atlas information including UV coordinates of vertices of the base mesh.(59) The information processing device according to any of (41) to (58), further comprising an atlas information decoding unit that decodes encoded data of atlas information and generates the atlas information including coordinate system identification information for identifying a coordinate system to be applied to the expression of the displacement vector. (60) The information processing device according to any of (41) to (59), wherein the displacement vectors of geometry patch boundary vertices of the subdivided base mesh are expressed using a local coordinate system that is a coordinate system for each vertex, the geometry patch coordinate system setting unit is further configured to set the local coordinate system of the geometry patch boundary vertex, and the displacement vector application unit is further configured to apply the displacement vector to the geometry patch boundary vertex of the subdivided base mesh using the local coordinate system. (61) The information processing device according to (60), wherein the geometry patch coordinate system setting unit sets the local coordinate system using a normal of a face adjacent to the geometry patch boundary vertex. (62) The information processing device according to (60), wherein the geometry patch coordinate system setting unit sets the local coordinate system using a reference coordinate axis of the geometry patch coordinate system of the geometry patch that may include the geometry patch boundary vertex. (63) The information processing device according to any of (41) to (62), wherein the displacement vector application unit further combines and applies the displacement vectors in a plurality of the geometry patch coordinate systems to geometry patch boundary vertices of the subdivided base mesh. (64) The information processing device according to any of (41) to (63), wherein the displacement vector is vector information whose length is a depth value from a projection plane onto which the original mesh is projected to a vertex of the original mesh. (65) The information processing device according to (64), further comprising an atlas information decoding unit that decodes encoded data of atlas information and generates the atlas information including projection plane information related to the position and orientation of the projection plane.(66) An information processing method comprising: decoding encoded data of a base mesh to generate the base mesh; subdividing the base mesh; decoding encoded data of a displacement vector to generate the displacement vector in a geometry patch coordinate system derived for each geometry patch of the base mesh based on a canonical coordinate system; setting the geometry patch coordinate system for each geometry patch of the subdivided base mesh; applying the displacement vector to the subdivided base mesh using the geometry patch coordinate system; the base mesh is a mesh with lower resolution than the original mesh to be encoded, which is composed of vertices and connections that represent a three-dimensional structure of an object, and is generated by thinning out vertices from the original mesh; and the displacement vector is vector information indicating the displacement of the vertices of the subdivided base mesh.

[0234] 300 Encoding device, 311 Preprocessing unit, 312 V-DMC encoding unit, 321 Base mesh generation unit, 322 Atlas information generation unit, 323 Displacement vector generation unit, 331 Subdivision unit, 332 Geometry patch generation unit, 333 Geometry patch coordinate system setting unit, 334 Displacement vector derivation unit, 351 Atlas information encoding unit, 352 Base mesh encoding unit, 353 Displacement vector correction unit, 354 Displacement vector encoding unit, 355 Mesh reconstruction unit, 356 Attribute map conversion unit, 357 Attribute encoding unit, 358 Multiplexing unit, 400 Decoding device, 411 Demultiplexing unit, 412 Atlas information decoding unit, 413 Base mesh decoding unit, 414 Subdivision unit, 415 Displacement vector decoding unit, 416 Displacement vector application unit 417 Attribute decoding unit 418 Attribute application unit 419 Display processing unit 431 Geometry patch generation unit 432 Geometry patch coordinate system setting unit 433 Displacement vector application unit 900 Computer

Claims

1. An information processing device comprising: a geometry patch coordinate system setting unit that sets a geometry patch coordinate system derived for each geometry patch of a base mesh based on a canonical coordinate system; a displacement vector derivation unit that derives a displacement vector in the geometry patch coordinate system; a base mesh encoding unit that encodes the base mesh; and a displacement vector encoding unit that encodes the displacement vector, wherein the base mesh is a mesh with lower resolution than an original mesh to be encoded, which is composed of vertices and connections that represent the three-dimensional structure of an object, and is generated by thinning out vertices from the original mesh, and the displacement vector is vector information that indicates the displacement of the vertices of the subdivided base mesh.

2. The information processing device according to claim 1, wherein the geometry patch coordinate system setting unit sets the geometry patch coordinate system using one of the coordinate axes of the canonical coordinate system or a coordinate axis obtained by rotating the coordinate axis at a predetermined angle as the reference coordinate axis.

3. The information processing device according to claim 2, wherein the geometry patch coordinate system setting unit sets the geometry patch coordinate system with the projection direction of the texture patch as the reference coordinate axis.

4. The information processing device according to claim 1, wherein the displacement vector derivation unit derives the displacement vector of each vertex of the subdivided base mesh using a depth map of the original mesh on a projection plane onto which the original mesh is projected.

5. The information processing device according to claim 1, further comprising an atlas information encoding unit that encodes atlas information including geometry patch coordinate system information indicating a reference coordinate axis of the geometry patch coordinate system.

6. The information processing device according to claim 5, wherein the geometry patch coordinate system information includes an index corresponding to a combination of a coordinate axis of the canonical coordinate system corresponding to the reference coordinate axis, a positive / negative direction, and a rotation direction.

7. The information processing device according to claim 1, further comprising an atlas information encoding unit that encodes atlas information including geometry patch correspondence information that indicates the correspondence between the faces of the base mesh and the geometry patches.

8. The information processing device according to claim 1, further comprising an atlas information encoding unit that encodes atlas information including UV coordinates of the vertices of the base mesh.

9. The information processing device according to claim 1, further comprising an atlas information encoding unit that encodes atlas information including coordinate system identification information for identifying a coordinate system applied to the representation of the displacement vector.

10. An information processing method comprising: setting a geometry patch coordinate system derived for each geometry patch of a base mesh based on a canonical coordinate system; deriving a displacement vector in the geometry patch coordinate system; encoding the base mesh; encoding the displacement vector; the base mesh is a mesh with lower resolution than an original mesh to be encoded, which is composed of vertices and connections that represent the three-dimensional structure of an object, and is generated by thinning out vertices from the original mesh; and the displacement vector is vector information indicating the displacement of the vertices of the subdivided base mesh.

11. An information processing device comprising: a base mesh decoding unit that decodes encoded data of a base mesh and generates the base mesh; a subdivision unit that subdivides the base mesh; a displacement vector decoding unit that decodes encoded data of a displacement vector and generates the displacement vector in a geometry patch coordinate system derived for each geometry patch of the base mesh with a canonical coordinate system as a reference; a geometry patch coordinate system setting unit that sets the geometry patch coordinate system for each geometry patch of the subdivided base mesh; and a displacement vector application unit that applies the displacement vector to the subdivided base mesh using the geometry patch coordinate system, wherein the base mesh is a mesh with lower resolution than the original mesh to be encoded, which is composed of vertices and connections that represent the three-dimensional structure of an object, and is generated by thinning out vertices from the original mesh, 12. The information processing device according to claim 11, wherein the geometry patch coordinate system setting unit sets the geometry patch coordinate system using one of the coordinate axes of the canonical coordinate system or a coordinate axis obtained by rotating the coordinate axis at a predetermined angle as the reference coordinate axis.

13. The information processing device according to claim 12, wherein the geometry patch coordinate system setting unit sets the geometry patch coordinate system with the projection direction of the texture patch as the reference coordinate axis.

14. The information processing device according to claim 11, further comprising an atlas information decoding unit that decodes encoded data of the atlas information and generates the atlas information including geometry patch coordinate system information indicating the geometry patch coordinate system.

15. The information processing device according to claim 14, wherein the geometry patch coordinate system information indicates a reference coordinate axis of the geometry patch coordinate system.

16. The information processing device according to claim 15, wherein the geometry patch coordinate system information includes an index corresponding to a combination of a coordinate axis of the canonical coordinate system corresponding to the reference coordinate axis, a positive / negative direction, and a rotation direction.

17. The information processing device according to claim 11, further comprising an atlas information decoding unit that decodes encoded data of the atlas information and generates the atlas information including geometry patch correspondence information that indicates the correspondence between the faces of the base mesh and the geometry patches.

18. The information processing device according to claim 11, further comprising an atlas information decoding unit that decodes encoded data of the atlas information and generates the atlas information including UV coordinates of the vertices of the base mesh.

19. The information processing device according to claim 11, further comprising an atlas information decoding unit that decodes encoded data of the atlas information and generates the atlas information including coordinate system identification information for identifying the coordinate system applied to the representation of the displacement vector.

20. An information processing method comprising: decoding encoded data of a base mesh to generate the base mesh; subdividing the base mesh; decoding encoded data of a displacement vector to generate the displacement vector in a geometry patch coordinate system derived for each geometry patch of the base mesh based on a canonical coordinate system; setting the geometry patch coordinate system for each geometry patch of the subdivided base mesh; applying the displacement vector to the subdivided base mesh using the geometry patch coordinate system; the base mesh is a mesh with lower resolution than an original mesh to be encoded, which is composed of vertices and connections that represent a three-dimensional structure of an object, and is generated by thinning out vertices from the original mesh; and the displacement vector is vector information indicating the displacement of the vertices of the subdivided base mesh.

Images