Information processing device and method

Abstract

The present disclosure pertains to an information processing device and method which are capable of suppressing reduction in encoding efficiency of 3D data including a base mesh and a displacement vector when applying a deployment method involving transmission of patch information. The present invention involves: deploying a base mesh in a two-dimensional area by means of a first deployment method involving transmission of patch information; deploying a local area of the base mesh in a two-dimensional area by means of a second deployment method different from the first deployment method; generating, as atlas information, patch information on the base mesh deployed by means of the first deployment method and meta information on the local area generated by means of the second deployment method; and mapping the texture by using the patch information and the meta information to generate an attribute map. The present disclosure can be applied to, for example, information processing devices, electronic apparatuses, information processing methods, and programs.

Description

Information processing device and method

[0001] The present disclosure relates to an information processing device and method, and more particularly to an information processing device and method that can suppress a decrease in encoding efficiency of 3D data including a base mesh and a displacement vector when applying an unfolding method that involves transmission of patch information.

[0002] Conventionally, V-DMC (Video-based Dynamic Mesh Coding) has been used as a method for encoding meshes, which are 3D data that represent the three-dimensional structure of an object using vertices and connections (see, for example, Non-Patent Document 1). In addition to UVAtlas, orthoAtlas has been proposed as a UV coordinate generation method (UV unfolding method) for V-DMC (see, for example, Non-Patent Documents 2 to 4). orthoAtlas is a unfolding method that involves the transmission of patch information.

[0003] Khaled Mammou, Jungsun Kim, Alexis Tourapis, Dimitri Podborski, Krasimir Kolarov, “[V-CG] Apple's Dynamic Mesh Coding CfP Response”, ISO / IEC JTC 1 / SC 29 / WG 7 m59281, April 2022Danillo B Graziosi, Alexandre Zaghetto, Ali Tabatabai, “[VDMC-NEW] orthoAtlas: a new texture mapping proposal for V-DMC”, ISO / IEC JTC 1 / SC 29 / WG 7 m60982, Mainz, DE - October 2022Danillo Bracco Graziosi, Alexandre Zaghetto, Ali Tabatabai, “[V-DMC EE4.8 related] Optimization of orthoAtlas parameters”, SO / IEC JTC 1 / SC 29 / WG 7 m62560, Antalya - April 2023Kao Hayashi, Satoru Kuma, Danillo Bracco Graziosi, Ohji Nakagami. “[V-DMC][EE4.8-related] Improvement of inter coding performance for orthoAtlas”, ISO / IEC JTC 1 / SC 29 / WG 7 m64125, Geneva - July 2023

[0004] However, in the case of this orthoAtlas, there is a risk that the coding efficiency may be reduced compared to the case of UVAtlas.

[0005] The present disclosure has been made in consideration of such circumstances, and makes it possible to suppress a decrease in the coding efficiency of 3D data including base meshes and displacement vectors when applying an unfolding method involving the transmission of patch information.

[0006] An information processing device according to one aspect of the present technology includes an atlas information generation unit that expands a base mesh into a two-dimensional region by a first expansion method involving transmission of patch information, expands a local region of the base mesh into the two-dimensional region by a second expansion method different from the first expansion method, and generates, as atlas information, the patch information for the base mesh expanded by the first expansion method and meta information for the local region generated by the second expansion method; an atlas information encoding unit that encodes the atlas information including the generated patch information and the meta information; an attribute map generation unit that uses the generated patch information to map a projected image of a texture corresponding to the base mesh onto the two-dimensional region in a manner compatible with the first expansion method, and uses the generated meta information to map the texture corresponding to the local region onto the two-dimensional region in a manner compatible with the second expansion method, thereby generating an attribute map; and an attribute map generation unit that uses the generated attribute map as a frame image. and a bitstream generation unit that generates a bitstream including encoded data of the base mesh, encoded data of the displacement video, encoded data of the attribute video, and encoded data of the atlas information, wherein the base mesh is a mesh with lower resolution than the original mesh to be encoded, which is composed of vertices and connections that represent the three-dimensional structure of an object, and is generated by thinning out vertices from the original mesh; the displacement video is a moving image whose frame images are a displacement map, which is a two-dimensional area into which displacement vectors indicating the displacement of division points, which are vertices generated by subdividing the base mesh, are packed; the atlas information is information used to reconstruct the mesh, including information indicating the correspondence between the base mesh, the displacement map, and the attribute map; and the patch information is information regarding the mapping of patches of the projection image of the texture.

[0007] An information processing method according to one aspect of the present technology includes: expanding a base mesh into a two-dimensional region by a first expansion method involving transmission of patch information; expanding a local region of the base mesh into the two-dimensional region by a second expansion method different from the first expansion method; generating, as atlas information, the patch information for the base mesh expanded by the first expansion method and meta information for the local region generated by the second expansion method; encoding the atlas information including the generated patch information and the meta information; using the generated patch information to map a projection image of a texture corresponding to the base mesh onto the two-dimensional region in a manner compatible with the first expansion method; and using the generated meta information to map the texture corresponding to the local region onto the two-dimensional region in a manner compatible with the second expansion method, thereby generating an attribute map; An information processing method for encoding a video to generate a bitstream including coded data of the base mesh, coded data of the displacement video, coded data of the attribute video, and coded data of the atlas information, wherein the base mesh is a mesh with lower resolution than an original mesh to be encoded, which is composed of vertices and connections that represent the three-dimensional structure of an object, and is generated by thinning out vertices from the original mesh; the displacement video is a moving image whose frame images are a displacement map, which is a two-dimensional area packed with displacement vectors that indicate the displacement of division points, which are vertices generated by subdividing the base mesh; the atlas information is information used for mesh reconstruction, including information indicating the correspondence between the base mesh, the displacement map, and the attribute map; and the patch information is information regarding the mapping of patches of the projection image of the texture.

[0008] According to another aspect of the present technology, an information processing device includes a demultiplexing unit that demultiplexes a bitstream and extracts coded data of a base mesh, coded data of a displacement video, coded data of an attribute video, and coded data of atlas information; an atlas information decoding unit that decodes the extracted coded data of the atlas information to generate the atlas information that corresponds to a first unfolding method involving transmission of patch information and a second unfolding method different from the first unfolding method, and that includes the patch information about the base mesh unfolded by the first unfolding method and meta information about a local region of the base mesh unfolded by the second unfolding method; a base mesh decoding unit that decodes the extracted coded data of the base mesh to generate information about the base mesh; a base mesh reconstruction unit that reconstructs the base mesh from information about the base mesh using the atlas information; and an attribute video reconstruction unit that decodes the coded data of the attribute video to generate attribute video information configured as frame images of the attribute video. and a display processing unit that uses the patch information included in the atlas information to extract a projection image of a texture corresponding to the base mesh mapped to the attribute map so as to correspond to the first unfolding method, uses the meta information included in the atlas information to extract the texture corresponding to the local region mapped to the attribute map so as to correspond to the second unfolding method, and applies the extracted projection image and the texture to a decoded mesh for rendering to generate an image for display, wherein the base mesh is a mesh with lower resolution than the original mesh to be coded, which is composed of vertices and connections that represent a three-dimensional structure of an object, and is generated by thinning out vertices from the original mesh, and the displacement video is a moving image in which frame images are displacement maps that are two-dimensional regions in which displacement vectors indicating displacements of division points that are vertices generated by subdividing the base mesh are packed, and the atlas information includes the base mesh andThe information used for reconstructing a mesh includes information indicating a correspondence between the displacement map and the attribute map, the patch information is information regarding the mapping of patches of the projection image of the texture, and the decoded mesh is generated by applying the displacement vector derived from the encoded data of the displacement video to the reconstructed base mesh.

[0009] An information processing method according to another aspect of the present technology includes: demultiplexing a bitstream to extract coded data of a base mesh, coded data of a displacement video, coded data of an attribute video, and coded data of atlas information; decoding the extracted coded data of the atlas information to generate atlas information that corresponds to a first unfolding method involving transmission of patch information and a second unfolding method different from the first unfolding method, the atlas information including the patch information about the base mesh unfolded by the first unfolding method and meta information about a local region of the base mesh unfolded by the second unfolding method; decoding the extracted coded data of the base mesh to generate information about the base mesh; reconstructing the base mesh from information about the base mesh using the atlas information; decoding the coded data of the attribute video to generate an attribute map configured as frame images of the attribute video; and reconstructing the attribute map so as to correspond to the first unfolding method using the patch information included in the atlas information. extracting a projection image of a texture corresponding to the base mesh mapped to the attribute map using the meta information included in the atlas information, extracting the texture corresponding to the local region mapped to the attribute map so as to correspond to the second unfolding method, and applying the extracted projection image and the texture to a decoded mesh and rendering the decoded mesh to generate an image for display, wherein the base mesh is a mesh with lower resolution than the original mesh to be coded, the original mesh being composed of vertices and connections that represent a three-dimensional structure of an object, and is generated by thinning out vertices from the original mesh, the original mesh being composed of vertices and connections that represent a three-dimensional structure of the object, and the displacement video is a moving image whose frame images are displacement maps, which are two-dimensional regions into which displacement vectors indicating displacements of division points, which are vertices generated by subdividing the base mesh, are packed, the atlas information is information used for reconstructing a mesh, including information indicating a correspondence between the base mesh, the displacement map, and the attribute map, and the patch information is information regarding mapping of patches of the projection image of the texture,The decoded mesh is generated by applying the displacement vector derived from the encoded data of the displacement video to the reconstructed base mesh.

[0010] In an information processing device and method according to one aspect of the present technology, a base mesh is expanded into a two-dimensional region by a first expansion method involving transmission of patch information, a local region of the base mesh is expanded into the two-dimensional region by a second expansion method different from the first expansion method, patch information for the base mesh expanded by the first expansion method and meta information for the local region generated by the second expansion method are generated as atlas information, the atlas information including the generated patch information and meta information is encoded, a projected image of a texture corresponding to the base mesh is mapped onto the two-dimensional region using the generated patch information in accordance with the first expansion method, and the generated meta information is used to map the texture corresponding to the local region onto the two-dimensional region in accordance with the second expansion method, thereby generating an attribute map, an attribute video having the generated attribute map as frame images is encoded, and a bitstream including encoded data for the base mesh, encoded data for the displacement video, encoded data for the attribute video, and encoded data for the atlas information is generated.

[0011] In an information processing device and method according to another aspect of the present technology, a bitstream is demultiplexed to extract coded data of a base mesh, coded data of a displacement video, coded data of an attribute video, and coded data of atlas information, and the extracted coded data of the atlas information is decoded to generate atlas information corresponding to a first expansion method involving transmission of patch information and a second expansion method different from the first expansion method, the atlas information including patch information about the base mesh expanded by the first expansion method and meta information about a local region of the base mesh expanded by the second expansion method, and the extracted coded data of the base mesh is decoded to generate information about the base mesh, and The truss information is used to reconstruct a base mesh from information about the base mesh, the encoded data of the attribute video is decoded to generate an attribute map configured as a frame image of the attribute video, the patch information included in the atlas information is used to extract a projection image of a texture corresponding to the base mesh mapped to the attribute map so as to correspond to a first expansion method, and the meta information included in the atlas information is used to extract a texture corresponding to the local region mapped to the attribute map so as to correspond to a second expansion method, and the extracted projection image and texture are applied to the decoded mesh and rendered to generate an image for display.

[0012] FIG. 1 is a diagram illustrating a mesh. FIG. 1 is a diagram illustrating V-DMC. FIG. 1 is a diagram illustrating an example of information to be transmitted. FIG. 1 is a diagram illustrating an example of an attribute map. FIG. 1 is a diagram illustrating an example of texture resolution evaluation. FIG. 1 is a diagram illustrating an example of texture resolution evaluation. FIG. 1 is a diagram illustrating an example of PSNR comparison. FIG. 1 is a diagram illustrating an example of bit rate comparison. FIG. 1 is a diagram illustrating an example of a deployment method. FIG. 1 is a diagram illustrating an example of a large patch. FIG. 1 is a diagram illustrating example of patch information. FIG. 1 is a diagram illustrating an example of an attribute map. FIG. 1 is a diagram illustrating an example of texture mapping. A block diagram illustrating an example of the main configuration of an encoding device. A block diagram illustrating an example of the main configuration of an encoding unit. A flowchart illustrating an example of the flow of encoding processing. A flowchart illustrating an example of the flow of encoding processing. A flowchart illustrating an example of the flow of encoding processing. A flowchart illustrating an example of the flow of encoding processing. A flowchart illustrating an example of the flow of encoding processing. A block diagram illustrating an example of the main configuration of a decoding device. A flowchart illustrating an example of the flow of decoding processing. A flowchart illustrating an example of the flow of decoding processing. A flowchart illustrating an example of the flow of decoding processing. A block diagram illustrating an example of the main configuration of a computer.

[0013] Below, modes for carrying out the present disclosure (hereinafter referred to as embodiments) will be described. The description will be made in the following order: 1. Literature etc. supporting technical content and technical terminology 2. UV coordinate generation method 3. Improvement of UV coordinate generation method 4. First embodiment (encoding device) 5. Second embodiment (decoding device) 6. Supplementary notes

[0014] <1. Literature, etc. supporting technical content and technical terminology> The scope of disclosure of the present technology includes not only the content described in the embodiments, but also the content described in the following non-patent documents, etc. that were publicly known at the time of filing, and the content of other documents referenced in the following non-patent documents.

[0015] Non-patent document 1: (described above) Non-patent document 2: (described above) Non-patent document 3: (described above) Non-patent document 4: (described above)

[0016] In other words, the contents of the above-mentioned non-patent documents and the contents of other documents referenced in the above-mentioned non-patent documents are also used as the basis for determining the support requirements.

[0017] <2. UV Coordinate Generation Method> <V-DMC> Conventionally, 3D data representing the three-dimensional structure of a three-dimensional structure (object with a three-dimensional shape) has been available as a mesh, which represents the three-dimensional shape of the object surface by forming polygons with vertices and connections (also called edges).

[0018] As shown in the upper left of Figure 1, in a mesh, vertices 11 and connections 12 connecting these vertices 11 form polygonal planes (polygons). These polygons (also called faces) represent the surface of a three-dimensional object, i.e., the three-dimensional shape of the object. A texture 13 can be applied to each face of this mesh.

[0019] Mesh data is composed of information such as that shown in the lower part of Figure 1. Vertex information 14, shown first from the left in the lower part of Figure 1, is information indicating the three-dimensional position (three-dimensional coordinates (X, Y, Z)) of each vertex 11 that constitutes the mesh. Connection information 15, shown second from the left in the lower part of Figure 1, is information indicating each connection (edge) 12 that constitutes the mesh. A texture image 16, shown third from the left in the lower part of Figure 1, is map information for the texture 13 that is applied to each face. A UV map 17, shown fourth from the left in the lower part of Figure 1, is information indicating the correspondence between the vertices 11 and the texture 13. The UV map 17 indicates the coordinates (UV coordinates) of each vertex 11 in the texture image 16.

[0020] As an example of such a mesh coding method, there is V-DMC (Video-based Dynamic Mesh Coding) as disclosed in Non-Patent Document 1.

[0021] In V-DMC, the mesh to be encoded (referred to in this specification as the original mesh) is represented as a base mesh that is less fine (i.e., coarser) than the original mesh, and displacement vectors of the division points obtained by subdividing the base mesh, and the base mesh and displacement vectors are then encoded.

[0022] For example, assume that an original mesh exists as shown in the top row of Figure 2. The original mesh is a mesh composed of vertices and connections that represent the three-dimensional structure of an object, and is the target for encoding. For example, the original mesh is generated from a captured image of an object in real space (by camera capture). The original mesh is non-registered mesh data (also called a dynamic mesh) whose structure changes every frame. In Figure 2, black dots represent vertices, and lines connecting the black dots represent connections (edges). As described above, a mesh essentially forms polygons using vertices and edges, but for convenience of explanation, it is described here as a group of vertices connected linearly (in series).

[0023] By decimating some of the vertices of the original mesh, a coarse (low-resolution) mesh is formed, as shown in the second row from the top of Figure 2. This is called the base mesh. In other words, the base mesh is a mesh with lower resolution than the original mesh, generated by decimating vertices from the original mesh.

[0024] By subdividing each polygon of this base mesh, vertices and edges are added, as shown in the third row from the top of Figure 2. The degree of subdivision is arbitrary. That is, the number of vertices and edges to be added is arbitrary. For example, the process of adding vertices to the midpoints of edges may be repeated recursively. This subdivision can add vertices by the number thinned out from the original mesh, for example. In other words, subdivision can make the number of vertices the same as the original mesh. In this specification, these added vertices are also referred to as division points.

[0025] However, because the connections are updated when the vertices of the original mesh are thinned out, and the division points are formed on these updated connections (edges), the shape of this subdivided base mesh differs from that of the original mesh. More specifically, as shown in the bottom part of Figure 2, the positions of the division points (on the dotted lines) differ from those of the original mesh.

[0026] Therefore, for example, by moving the positions of the division points so that they approach the vertex positions of the original mesh, the difference in shape from the original mesh can be reduced. In this specification, such movement of the division points is also referred to as displacement. Furthermore, the amount and direction of this displacement, expressed as a vector, is also referred to as a displacement vector. For example, ideally, by displacing the division points, the shape of the subdivided base mesh can be made to match the shape of the original mesh. In other words, the original mesh can be expressed as a base mesh and a displacement vector.

[0027] In V-DMC, the base mesh and displacement vectors are encoded instead of the original mesh. More specifically, the base mesh is encoded using an existing mesh encoding method such as Draco. The displacement vectors are packed into a two-dimensional region (also called a displacement map) and encoded as a displacement video using a 2D video encoding method with the displacement map as a frame image. The texture's projected image is mapped to a two-dimensional region (also called an attribute map) and encoded as an attribute video using the attribute map as a frame image with a 2D video encoding method. Atlas information used to reconstruct the mesh is encoded using a predetermined encoding method. The atlas information may include, for example, a correspondence between the base mesh and the displacement map or attribute map (such as a UV map), a quantized value of the displacement vector, etc.

[0028] By encoding the base mesh and displacement vectors in this way, it is possible to encode with a reduced number of polygons (i.e., the number of vertices and edges) compared to encoding the original mesh, which generally reduces the amount of code for the same quality, i.e., improves encoding efficiency.

[0029] During decoding, each bitstream is decoded using a decoding method corresponding to the encoding method, and various information such as a base mesh, displacement vectors, texture, and atlas information is restored (generated). The base mesh is then subdivided, and displacement vectors are applied to each division point to restore (generate) the original mesh. In reality, the restored mesh may contain coding artifacts, etc., and may not completely match the original mesh before encoding. However, for the sake of convenience, the following description will be given assuming that the original mesh is restored by decoding, without taking into account coding artifacts, etc. Furthermore, in the following description, polygons are assumed to be triangular. Therefore, in the following description, polygons (faces) will also be referred to as triangles.

[0030] <UV Atlas> As described above, textures are placed in an attribute map (two-dimensional region) and encoded. To reconstruct a mesh and apply the texture to each face of the mesh, it is necessary to establish a correspondence between the base mesh and the texture. In other words, the position (UV coordinates) in the attribute map (two-dimensional region) corresponding to each vertex of the base mesh is derived. In this specification, the process of generating UV coordinates for each vertex of the base mesh is also referred to as "unwrapping (UV unwrapping)."

[0031] In V-DMC, UVAtlas is one such UV coordinate generation method (UV unfolding method). When UVAtlas is applied to unfold a base mesh, transmission information such as a vertex list, triangle list, UV list, and UV triangle list is encoded, as shown in a square 31 in Fig. 3. The UV list is information indicating the UV coordinates (u, v) of each vertex of the base mesh, and is transmitted to the decoding side as atlas information.

[0032] <orthoAtlas> In recent years, orthoAtlas has been proposed as a UV coordinate generation method (UV unfolding method) different from UVAtlas, as shown in, for example, Non-Patent Documents 2 to 4. In the case of orthoAtlas, UV coordinates are generated using orthogonal projection (six directions). In other words, UV coordinates can be derived from the projection direction, so that they can be derived at the decoder. Therefore, there is no need to transmit UV coordinates.

[0033] In other words, when an orthoAtlas is applied, instead of UV coordinates, the UV list indicates the identification information (patch ID) of the patch to which each vertex belongs, as shown in the box 32 in Figure 3. This patch identification information is also called a CC id (Connected Component ID). A projected image in which the texture is projected in a specified direction is mapped onto the attribute map. At this time, the texture is divided into small regions and projected to avoid overlapping or to improve resolution by projecting in a more optimal projection direction. These small regions are also called patches. Because CC ids can be expressed as integer values, the amount of code can be reduced compared to floating-point UV coordinates. In other words, the amount of code for this UV list can be reduced compared to when using a UVAtlas.

[0034] However, because the projection direction cannot be determined from the CC ID alone, patch information for each patch is transmitted as atlas information. Patch information is information related to the mapping of the patch. For example, as shown in the box 33 in Figure 3, patch information (patch info) may include information such as the projection direction of each patch (projId), the rotation angle (orientId), the 2D position (2dPosX, 2dPosY), the patch size (2dSizeX, 2dSizeY), and the patch scale (scaleFlag(,Scale)). Therefore, the patch information of the corresponding CC indicates where each vertex is projected on the attribute map and at what projection direction and angle. Thus, orthoAtlas is an expansion method that involves transmitting patch information. Even when including this patch information, applying orthoAtlas generally reduces the amount of coding compared to applying UVAtlas.

[0035] There are multiple texture packing methods when orthoAtlas is applied, such as default, small on top, Tetris, Projection, etc. Non-Patent Document 4 proposes a method for improving the correlation in the time direction of Projection packing.

[0036] <Image Quality Comparison> However, when orthoAtlas is applied, there is a risk that the proportion of texture in the attribute map (also referred to as the effective area ratio) may be reduced compared to when UVAtlas is applied. FIG. 4 shows examples of textures (patches) placed in attribute maps. Attribute map 41 shows an example of an attribute map when UVAtlas is applied. Attribute map 42 shows an example of an attribute map when orthoAtlas (default packing) is applied. Attribute map 43 shows an example of an attribute map when orthoAtlas (projection packing) is applied. In each of attribute maps 41 to 43, the gray areas indicate the mapped textures (patches). As shown in FIG. 4, for attribute map 41 (UVAtlas), the effective area ratio is 55%. For attribute map 42 (orthoAtlas (default packing)), the effective area ratio is 46%. In the case of the attribute map 43 (orthoAtlas (projection packing)), the effective area ratio is 28.5%.

[0037] Therefore, when UVAtlas is applied, the texture resolution is as shown in textures 51 and 52 in FIG. 5 . When orthoAtlas (default packing) is applied, the texture resolution is as shown in textures 61 and 62 in FIG. 6 . When orthoAtlas (projection packing) is applied, the texture resolution is as shown in textures 71 and 72 in FIG. 7 . The resolution of each texture shown in FIGS. 5 to 7 is indicated by density. Lighter density (i.e., closer to white) indicates higher resolution, and darker density (i.e., closer to black) indicates lower resolution. The density is lighter and the black area is smaller in the order of FIGS. 5 , 6 , and 7 . Therefore, among the examples in FIGS. 5 to 7 , textures 51 and 52 have the highest resolution. In other words, textures 71 and 72 have the lowest resolution.

[0038] As described above, applying orthoAtlas reduces texture resolution (i.e., image quality) compared to applying UVAtlas. Figure 8 shows an example comparing the PSNR (Peak Signal-to-Noise Ratio) of textures when applying orthoAtlas and UVAtlas. Graph 81 (top) and graph 82 (bottom) show the PSNR comparison results for different samples. In graphs 81 and 82, the solid line indicates the PSNR when applying UVAtlas, and the dotted line indicates the PSNR when applying orthoAtlas. In both cases, the PSNR is higher when applying UVAtlas than when applying orthoAtlas. In other words, assuming the same coding amount (bit rate), applying orthoAtlas may result in lower texture image quality than when applying UVAtlas. In other words, applying orthoAtlas may result in lower coding efficiency than when applying UVAtlas.

[0039] In this specification, the encoding efficiency of a mesh refers to the quality of the mesh relative to the amount of code (including the image quality of textures and display images). In other words, for example, when the amount of code is the same, the higher the quality of the mesh reconstructed after decoding, the higher the encoding efficiency. In other words, when the quality of the mesh reconstructed after decoding is the same, the smaller the amount of code, the higher the encoding efficiency.

[0040] <Amount of meta information> In addition, with orthoAtlas, the amount of code required for patch information increases in proportion to the number of patches. Therefore, if patches are divided into smaller pieces to improve texture resolution, the amount of code required for meta information may increase due to the increase in patch information transmitted instead of UV coordinates.

[0041] The table shown in FIG. 9 compares the geometry bit rate, texture bit rate, meta information bit rate, and overall bit rate when UV Atlas is applied and when ortho Atlas is applied. This comparison is performed for each of the meshes with five bit rates, Sequence 1 to Sequence 3. As shown in FIG. 9, when UV Atlas is applied, the meta information (Meta) bit rate is approximately 0.004 Mbps. In contrast, when ortho Atlas is applied, the meta information (Meta) bit rate is approximately 0.026 Mbps to 0.170 Mbps. In other words, when ortho Atlas is applied, the proportion of meta information is higher than when UV Atlas is applied.

[0042] Therefore, when orthoAtlas is applied, the increase in the number of patches causes an increase in the amount of meta information to be coded, which may result in lower coding efficiency than when UVAtlas is applied.

[0043] <Method for Improving Texture Quality> One possible method for improving texture quality is to increase the number of texture projection directions. A large angular difference between the vertical direction of a face and the projection direction can easily reduce texture quality. Therefore, the more diverse the projection directions, the easier it is to suppress this increase in angular difference. For example, in the case of orthoAtlas, allowing projection in diagonal directions in addition to the six vertical directions can easily reduce texture quality. However, with this method, the increased number of projection directions makes it easier for patches to be divided into smaller pieces. This increases the amount of code required for meta-information, potentially reducing coding efficiency.

[0044] Another possible method is to increase the resolution by enlarging the entire texture placed in the attribute map. However, this method enlarges areas other than those requiring increased resolution, which unnecessarily increases the amount of coding required for the texture and can reduce coding efficiency.

[0045] Another possible method is to divide CCs into smaller parts and select a more appropriate projection direction (projection ID) for each triangle. However, this method increases the number of patches, which increases the amount of meta-information coding required and may reduce coding efficiency.

[0046] As described above, when orthoAtlas is applied, there is a risk that coding efficiency may be reduced compared to when UVAtlas is applied.

[0047] <3. Improvements to UV Coordinate Generation Method> <Method 1> Therefore, as shown in the top row of the table in Fig. 10, common patch information common to multiple frames may be transmitted (Method 1). In the case of conventional orthoAtlas, patch information for all intra-patches was encoded and transmitted as atlas information. Instead of this patch information, common patch information common to multiple frames (applicable to multiple frames) is generated, encoded, and transmitted.

[0048] Hereinafter, an information processing device that encodes 3D data including a base mesh and a displacement vector is also referred to as a “first information processing device.” For example, this first information processing device may include an atlas information generation unit that expands the base mesh into a two-dimensional region using an expansion method involving transmission of patch information and generates, as atlas information, common patch information that is patch information common to multiple frames, an atlas information encoding unit that encodes the atlas information including the generated common patch information, an attribute map generation unit that uses the generated common patch information to map a projected image of a texture of the original mesh into a two-dimensional region in accordance with the expansion method to generate an attribute map, an attribute video encoding unit that encodes an attribute video using the generated attribute map as a frame image, and a bitstream generation unit that generates a bitstream including encoded data of the base mesh, encoded data of the displacement video, encoded data of the attribute video, and encoded data of the atlas information.

[0049] In addition, in the first information processing device, a base mesh is expanded into a two-dimensional area using an expansion method involving transmission of patch information, common patch information that is patch information common to a plurality of frames is generated as atlas information, the atlas information including the generated common patch information is encoded, a projected image of the texture of the original mesh is mapped onto the two-dimensional area using the generated common patch information in accordance with the expansion method to generate an attribute map, an attribute video having the generated attribute map as a frame image is encoded, and a bitstream including encoded data of the base mesh, encoded data of the displacement video, encoded data of the attribute video, and encoded data of the atlas information is generated.

[0050] The base mesh is a mesh with lower resolution than the original mesh to be coded, which is composed of vertices and connections that represent the three-dimensional structure of the object, and is generated by thinning out vertices from the original mesh. The displacement video is a moving image whose frame images are displacement maps, which are two-dimensional regions packed with displacement vectors that indicate the displacement of division points, which are vertices generated by subdividing the base mesh. The atlas information is information used to reconstruct the mesh, including information indicating the correspondence between the base mesh, the displacement map, and the attribute map. The patch information is information related to the mapping of patches of the texture projection image.

[0051] As described above, since the common patch information can be applied to multiple frames, the first information processing device does not need to transmit patch information for each frame, thereby reducing redundancy. This allows the first information processing device to reduce an increase in the amount of code for the patch information. In other words, it is possible to reduce a decrease in the coding efficiency of 3D data including base meshes and displacement vectors when applying an unfolding method that involves transmitting patch information.

[0052] In addition, hereinafter, the information processing device that decodes the bitstream and generates 3D data including the base mesh and the displacement vector is also referred to as a second information processing device. This second information processing device includes a demultiplexing unit that demultiplexes the bitstream and extracts coded data of the base mesh, coded data of the displacement video, coded data of the attribute video, and coded data of the atlas information; an atlas information decoding unit that decodes the extracted coded data of the atlas information to generate atlas information that corresponds to an expansion method involving transmission of patch information and includes common patch information that is patch information common to multiple frames; a base mesh decoding unit that decodes the extracted coded data of the base mesh to generate information about the base mesh; a base mesh reconstruction unit that reconstructs the base mesh from information about the base mesh using the atlas information; an attribute video decoding unit that decodes the coded data of the attribute video to generate an attribute map configured as frame images of the attribute video; and a display processing unit that uses the common patch information included in the atlas information to extract a projected image of the texture of the original mesh mapped to the attribute map so as to correspond to the expansion method, and applies it to the decoded mesh for rendering to generate an image for display.

[0053] In addition, in a second information processing device, the bit stream is demultiplexed to extract coded data of a base mesh, coded data of a displacement video, coded data of an attribute video, and coded data of atlas information, and the extracted coded data of the atlas information is decoded to generate atlas information that corresponds to an unfolding method involving transmission of patch information and that includes common patch information that is patch information common to a plurality of frames, the extracted coded data of the base mesh is decoded to generate information about the base mesh, the atlas information is used to reconstruct the base mesh from the information about the base mesh, and the coded data of the attribute video is decoded to generate an attribute map configured as a frame image of the attribute video, and the common patch information included in the atlas information is used to extract a projected image of the texture of the original mesh mapped to the attribute map so as to correspond to the unfolding method, and the projected image is applied to the decoded mesh and rendered to generate an image for display.

[0054] The base mesh is a mesh with lower resolution than the original mesh to be coded, which is composed of vertices and connections that represent the three-dimensional structure of an object, and is generated by thinning out vertices from the original mesh. The displacement video is a moving image whose frame images are displacement maps, which are two-dimensional regions packed with displacement vectors that indicate the displacement of division points, which are vertices generated by subdividing the base mesh. The atlas information is information used to reconstruct the mesh, including information indicating the correspondence between the base mesh, the displacement map, and the attribute map. The patch information is information related to the mapping of patches of the texture projection image. The decoded mesh is generated by applying displacement vectors derived from the encoded data of the displacement video to the reconstructed base mesh.

[0055] As described above, since the common patch information can be applied to multiple frames, the second information processing device does not need to acquire patch information for each frame, thereby reducing redundancy. This allows the second information processing device to reduce an increase in the amount of code for the patch information. That is, when applying an unfolding method involving transmission of patch information, it is possible to reduce a decrease in the coding efficiency of 3D data including base meshes and displacement vectors.

[0056] The common patch information may be information about a large patch that is common to multiple frames and contains multiple small patches that are projected in the same direction.

[0057] For example, in the case of orthoAtlas (projection packing) described in Cited Document 4, patches (also referred to as small patches in this specification) are grouped by projection direction. Therefore, it is possible to align the positions, orientations (and sizes) of these small patches to a certain extent between frames. Therefore, if a large patch (a patch for each projection direction) containing a group of small patches with the same projection direction is set, the position, orientation, size, etc. of the large patch can be made common between frames. In other words, the patch information of the large patch can be common patch information that can be applied to multiple frames.

[0058] For example, FIG. 11 shows an example of an attribute map when orthoAtlas (projection packing) described in Cited Document 4 is applied. In the attribute map 110 shown in FIG. 11, the gray areas represent small patches. In the case of projection packing, these small patches are arranged so as to be grouped together to some extent for each projection direction, as described above. Therefore, small patches can be grouped together to form large patches for each projection direction, such as large patches 111 to 116. Furthermore, as described above, the position, orientation, size, etc. of each large patch can be aligned to some extent between frames, so the patch information for each large patch can be used as common patch information. By using common patch information corresponding to multiple frames, the patch information for a large patch only needs to be transmitted once for those multiple frames.

[0059] By using common patch information in this way, it is not necessary to transmit the patch information of the large patches 111 to 116 every frame, which reduces redundancy and prevents an increase in the amount of code for the patch information. In other words, when applying an unfolding method that involves transmitting patch information, it is possible to prevent a decrease in the coding efficiency of 3D data including base meshes and displacement vectors.

[0060] The common patch information may correspond to any continuous frame interval. For example, the common patch information may correspond to a group of frames (GOF) (i.e., the same common patch information may be applied to each frame of a GOF). Alternatively, the common patch information may correspond to an entire sequence (i.e., the same common patch information may be applied to each frame in a sequence). In other words, one common patch information may be applied to any number of frames.

[0061] Note that the common patch information may include any information. For example, the common patch information may include at least one of information indicating the projection direction of the large patch, information indicating the rotation angle of the large patch, information indicating the position of the large patch in a two-dimensional area, information indicating the size of the large patch, and information indicating the magnification of the large patch. Information other than these pieces of information may also be included in the common patch information.

[0062] <Method 1-1> When the above-mentioned Method 1 is applied, unique patch information for each frame may also be transmitted (Method 1-1), as shown in the second row from the top of the table in Fig. 10. In addition to the above-mentioned common patch information, information that cannot be covered by the common patch information may also be transmitted as unique patch information for each frame.

[0063] For example, the atlas information generating unit of the first information processing device may further generate, as the atlas information, specific patch information that is patch information specific to a frame. That is, the atlas information may further include specific patch information that is patch information specific to a frame. Furthermore, the display processing unit of the second information processing device may extract a projection image (texture) using the common patch information and the specific patch information included in the atlas information, and apply the projection image (texture) to the decoded mesh for rendering to generate a display image.

[0064] In this way, by transmitting common patch information and unique patch information and applying the common patch information and the unique patch information of each frame to each frame, the patch information of each frame can be transmitted as more accurate information.

[0065] Note that the unique patch information may include any information. For example, the unique patch information may include information (position shift information) indicating the displacement between frames of a large patch corresponding to the common patch information. For example, in FIG. 12 , frames 121 and 122 are assumed to be two consecutive frames of attribute video. Furthermore, assume that a large patch 133 containing small patches 131 and 132 is set in these frames. This large patch 133 is displaced (moves) between frames 121 and 122. Information indicating such a position shift (arrow 141) between frames may be transmitted as the unique patch information. In this way, it is sufficient to transmit information indicating the arrow 141 as the unique patch information, thereby suppressing an increase in the amount of code compared to transmitting patch information for a large patch in each of frames 121 and 122.

[0066] The unique patch information may also include information about small patches contained in the large patch corresponding to the common patch information. In the example of FIG. 12 , the small patches 131 and 132 are displaced and deformed between frames 121 and 122. In other words, the patch information for the small patches 131 and 132 is frame-specific information. Therefore, this patch information (the patch information for the small patches 131 and 132) may be transmitted as unique patch information, separate from the common patch information. In this case, the unique patch information may be transmitted as a difference from the common patch information.

[0067] Furthermore, the unique patch information may include information about a small patch located outside of a large patch corresponding to the common patch information. In the example of Fig. 12, a small patch 134 exists in the frame 122. This small patch 134 is located outside of the large patch 133. The patch information about this small patch 134 may be included in the unique patch information of the frame 122.

[0068] <Method 2> As shown in the third row of the table in FIG. 10 , orthoAtlas and another method may be used together as a UV coordinate generation method (UV unfolding method) (Method 2). As described above, applying orthoAtlas as a UV unfolding method is likely to reduce texture resolution. This is particularly true in areas with complex shapes (areas where the angle of the face relative to the projection direction is likely to be large), such as the black areas in textures 71 and 72 in FIG. 7 , or at patch boundaries. Therefore, local areas where the resolution (encoding efficiency) is reduced by orthoAtlas are UV unfolded using another UV coordinate generation method (UV unfolding method). This improves texture resolution, as in textures 51 and 52 in FIG. 5 . This reduces texture quality degradation.

[0069] In this case, applying other expansion methods to all regions of the texture may result in a larger amount of code than when orthoAtlas is applied. By applying orthoAtlas to areas where sufficient resolution can be obtained even with orthoAtlas, and applying other expansion methods only to local regions where the resolution is reduced by orthoAtlas (i.e., using orthoAtlas in combination with other methods), it is possible to suppress this unnecessary increase in the amount of code while also suppressing a decrease in texture quality. Therefore, it is possible to suppress a decrease in coding efficiency.

[0070] For example, a first information processing device may include an atlas information generating unit that expands a base mesh into a two-dimensional region by a first expansion method involving transmission of patch information, expands a local region of the base mesh into a two-dimensional region by a second expansion method different from the first expansion method, and generates, as atlas information, patch information for the base mesh expanded by the first expansion method and meta information for the local region generated by the second expansion method; an atlas information encoding unit that encodes the atlas information including the generated patch information and meta information; and an atlas information encoding unit that encodes, using the generated patch information, a projection image of a texture corresponding to the base mesh into a two-dimensional region. The system further comprises an attribute map generation unit that generates an attribute map by mapping a local region onto a two-dimensional region so as to correspond to a first expansion method and then using the generated meta information to map a texture corresponding to the local region onto a two-dimensional region so as to correspond to the second expansion method; an attribute video encoding unit that encodes an attribute video in which the generated attribute map is used as a frame image; and a bitstream generation unit that generates a bitstream including encoded data of the base mesh, encoded data of the displacement video, encoded data of the attribute video, and encoded data of the atlas information.

[0071] Furthermore, in a first information processing device, a base mesh is expanded into a two-dimensional region by a first expansion method involving transmission of patch information, a local region of the base mesh is expanded into the two-dimensional region by a second expansion method different from the first expansion method, patch information for the base mesh expanded by the first expansion method and meta information for the local region generated by the second expansion method are generated as atlas information, the atlas information including the generated patch information and meta information is encoded, a projection image of a texture corresponding to the base mesh is mapped onto the two-dimensional region in accordance with the first expansion method using the generated patch information, and a texture corresponding to the local region is mapped onto the two-dimensional region in accordance with the second expansion method using the generated meta information to generate an attribute map, an attribute video having the generated attribute map as a frame image is encoded, and a bitstream including encoded data of the base mesh, encoded data of the displacement video, encoded data of the attribute video, and encoded data of the atlas information is generated.

[0072] The base mesh is a mesh with lower resolution than the original mesh to be coded, which is composed of vertices and connections that represent the three-dimensional structure of an object. The displacement video is a moving image whose frame images are displacement maps, which are two-dimensional regions packed with displacement vectors indicating the displacement of division points, which are vertices generated by subdividing the base mesh. The atlas information is information used for mesh reconstruction, including information indicating the correspondence between the base mesh, the displacement map, and the attribute map. The patch information is information regarding the mapping of patches of a texture projection image. The local region may be, for example, a portion of the base mesh whose resolution (encoding efficiency) is reduced by the first unfolding method. In other words, the second unfolding method may be applied to a portion of the base mesh whose texture would have lower image quality (lower resolution) than a predetermined standard if the first unfolding method were applied.

[0073] In this way, the first information processing device can appropriately apply the first decompression method and the second decompression method, and can suppress a reduction in texture quality while suppressing an unnecessary increase in the amount of code, thereby suppressing a reduction in coding efficiency.

[0074] For example, the second information processing device may include a demultiplexing unit that demultiplexes the bitstream and extracts coded data of the base mesh, coded data of the displacement video, coded data of the attribute video, and coded data of the atlas information; an atlas information decoding unit that decodes the coded data of the extracted atlas information to generate atlas information that corresponds to a first unfolding method involving transmission of patch information and a second unfolding method different from the first unfolding method, and that includes patch information about the base mesh unfolded by the first unfolding method and meta information about a local region of the base mesh unfolded by the second unfolding method; a base mesh decoding unit that decodes the coded data of the extracted base mesh to generate information about the base mesh; and a decoder that uses the atlas information. The video decoder includes a base mesh reconstruction unit that reconstructs a base mesh from information about the base mesh, an attribute video decoding unit that generates an attribute map configured as frame images of the attribute video by decoding encoded data of the attribute video, and a display processing unit that uses patch information included in the atlas information to extract a projection image of a texture corresponding to the base mesh mapped to the attribute map so as to correspond to the first expansion method, uses meta information included in the atlas information to extract a texture corresponding to the local region mapped to the attribute map so as to correspond to the second expansion method, and applies the extracted projection image and texture to the decoded mesh for rendering to generate an image to be displayed.

[0075] Furthermore, in the second information processing device, the bit stream is demultiplexed to extract coded data of the base mesh, coded data of the displacement video, coded data of the attribute video, and coded data of the atlas information, and the coded data of the extracted atlas information is decoded to generate atlas information corresponding to a first unfolding method involving transmission of patch information and a second unfolding method different from the first unfolding method, the atlas information including patch information about the base mesh unfolded by the first unfolding method and meta information about local regions of the base mesh unfolded by the second unfolding method, and the coded data of the extracted base mesh is decoded to generate information about the atlas information. The atlas information is used to reconstruct a base mesh from information about the base mesh, and the encoded data of the attribute video is decoded to generate an attribute map configured as frame images of the attribute video. Patch information included in the atlas information is used to extract a projection image of a texture corresponding to the base mesh mapped to the attribute map so as to correspond to the first expansion method. Meta information included in the atlas information is used to extract a texture corresponding to the local region mapped to the attribute map so as to correspond to the second expansion method. The extracted projection image and texture are applied to the decoded mesh and rendered to generate an image for display.

[0076] The base mesh is a mesh with lower resolution than the original mesh to be coded, which is composed of vertices and connections that represent the three-dimensional structure of an object. The displacement video is a moving image whose frame images are displacement maps, which are two-dimensional regions packed with displacement vectors indicating the displacement of division points, which are vertices generated by subdividing the base mesh. The atlas information is information used for mesh reconstruction, including information indicating the correspondence between the base mesh, the displacement map, and the attribute map. The patch information is information regarding the mapping of patches of a texture projection image. The decoded mesh is generated by applying displacement vectors derived from the encoded data of the displacement video to the reconstructed base mesh. The local region may be, for example, a portion of the base mesh whose resolution (encoding efficiency) is reduced by the first unfolding method. In other words, the second unfolding method may be applied to a portion of the base mesh whose texture would have lower image quality (lower resolution) than a predetermined standard if the first unfolding method were applied.

[0077] In this way, the second information processing device can appropriately apply the first decompression method and the second decompression method, and can suppress a reduction in texture quality while suppressing an unnecessary increase in the amount of code, thereby suppressing a reduction in coding efficiency.

[0078] <Method 2-1> When the above-described method 2 is applied, as shown in the fourth row from the top of the table in FIG. 10 , orthoAtlas (first unfolding method) may be applied to the entire texture (i.e., the entire base mesh), and another unfolding method (second unfolding method) may be applied to some patches (i.e., local regions of the base mesh). For example, first, the entire base mesh is UV-unfolded using orthoAtlas to generate atlas information such as patch information. Then, the second unfolding method is again applied to UV-unfold a local region of the base mesh. When reconstructing a mesh, orthoAtlas is applied to derive UV coordinates for the entire base mesh, and a texture (texture corresponding to orthoAtlas) is applied to the reconstructed mesh (decoded mesh) according to the UV coordinates. Then, the second unfolding method is again applied to the local region to which the second unfolding method was applied, and a texture (texture corresponding to the second unfolding method) is applied to the decoded mesh. In other words, the texture of the local region is updated. This local region may be, for example, a part of the base mesh where the texture resolution (image quality) is reduced by applying the orthoAtlas.

[0079] For example, the atlas information generating unit of the first information processing device may expand the entire base mesh into a two-dimensional region using a first expansion method, and expand its local region into a two-dimensional region using a second expansion method.

[0080] Furthermore, an atlas information decoding unit of the second information processing device may decode encoded data of the extracted atlas information to generate atlas information corresponding to a first unfolding method and a second unfolding method, the atlas information decoding unit generating atlas information including patch information for the entire base mesh unfolded by the first unfolding method and meta-information for a local region unfolded by the second unfolding method. Furthermore, a display processing unit may use the patch information included in the atlas information to extract a texture corresponding to the entire base mesh mapped to the attribute map so as to correspond to the first unfolding method, use the meta-information included in the atlas information to extract a texture corresponding to the local region mapped to the attribute map so as to correspond to the second unfolding method, and apply the extracted projection image and texture to the decoded mesh for rendering to generate an image for display.

[0081] For example, the second decompression method is applied to portions of the image with lower image quality than a predetermined standard. This makes it easier to appropriately apply the first and second decompression methods, thereby suppressing unnecessary increases in the amount of code while also suppressing degradation of texture quality. Therefore, it is possible to suppress degradation of coding efficiency.

[0082] <Method 2-2> When the above-mentioned Method 2 is applied, orthoAtlas may be applied to some regions (some patches) of the base mesh, and another unfolding method (second unfolding method) may be applied to other regions (remaining patches) of the base mesh, as shown in the fifth row from the top of the table in Fig. 10. In other words, the regions (patches) to which each unfolding method is applied may not overlap each other.

[0083] For example, the atlas information generation unit of the first information processing device may use the first expansion method to expand the base mesh except for the above-mentioned local region into a two-dimensional region, and then use the second expansion method to expand the above-mentioned local region into a two-dimensional region.

[0084] Furthermore, an atlas information decoding unit of the second information processing device may decode the encoded data of the extracted atlas information to generate atlas information corresponding to a first unfolding method and a second unfolding method, the atlas information decoding unit generating atlas information including patch information for a region other than the local region of the base mesh unfolded by the first unfolding method and meta-information for the local region unfolded by the second unfolding method. Furthermore, a display processing unit may use the patch information included in the atlas information to extract a texture corresponding to the region other than the local region of the base mesh that is mapped to an attribute map corresponding to the first unfolding method, and use the meta-information included in the atlas information to extract a texture corresponding to the region other than the local region of the base mesh that is mapped to an attribute map corresponding to the second unfolding method, and apply the extracted projection image and texture to the decoded mesh for rendering to generate an image for display.

[0085] For example, the second decompression method is applied to parts of the image that have lower image quality than a predetermined standard. By doing so, it is possible to prevent duplication of processing, and it is possible to prevent an unnecessary increase in the amount of processing and a decrease in coding efficiency.

[0086] <Method 2-3> When applying the above-described Method 2, orthoAtlas and UVAtlas may be used together, as shown in the sixth row from the top of the table in FIG. 10 . That is, UVAtlas may be applied as the second unwrapping method described above. As described with reference to FIGS. 5 to 7 , applying UVAtlas generally improves texture resolution compared to applying orthoAtlas. Therefore, orthoAtlas is applied to areas where sufficient resolution can be achieved even with orthoAtlas (areas other than the local areas described below), and UVAtlas is applied to local areas where applying orthoAtlas reduces texture resolution. In other words, orthoAtlas and UVAtlas are used together. This prevents unnecessary increases in the amount of code while also suppressing degradation of texture quality. Therefore, it is possible to suppress degradation of coding efficiency. Note that the processing in this case may be performed as in Method 2-1 or Method 2-2 described above.

[0087] For example, the second unfolding method may involve transmitting coordinate information indicating the coordinates of the vertices of the base mesh in a two-dimensional region, and the atlas information generating unit of the first information processing device may generate, as meta-information, coordinate information for a local region of the base mesh to which the second unfolding method is applied, and generate atlas information including the coordinate information. Furthermore, the attribute map generating unit may use the generated meta-information (including the coordinate information) to map a projection image of a texture corresponding to the local region onto an attribute map in accordance with the second unfolding method. Furthermore, the atlas information decoding unit of the second information processing device may decode the encoded data of the extracted atlas information to generate atlas information including the coordinate information as meta-information. This local region may be, for example, a portion of the base mesh where texture resolution (image quality) is reduced by applying the first unfolding method.

[0088] <Transmission of UV Coordinates> For example, as shown in Fig. 13 , a patch 151 to which a UV Atlas is applied may be placed in an area of ​​an attribute map 150 in which a group of patches to which an ortho Atlas is applied is placed, but no patches are placed. The UV coordinates of the vertices of this patch 151 are transmitted. Any method may be used to transmit these UV coordinates. For example, the UV coordinates of each vertex of a patch to which a UV Atlas is applied may be transmitted using a UV list.

[0089] For example, in the UV list, for vertices that belong to a patch to which an orthoAtlas is applied, a combination of CCid and a predetermined value (e.g., (CCid, 0)) may be transmitted, and for vertices that belong to a patch to which a UVAtlas is applied, the UV coordinates (u, v) may be transmitted. Note that the predetermined value to be combined with CCid may be any value that is not used as a UV coordinate (the above-mentioned "0" is an example, and the value is not limited to this example).

[0090] For example, an atlas information generation unit of the first information processing device may generate, as meta-information, coordinate information for each vertex present in the local region. Furthermore, an atlas information decoding unit of the second information processing device may decode the encoded data of the extracted atlas information to generate atlas information including, as meta-information, coordinate information for each vertex present in the local region. This local region is a region to which the second unfolding method is applied. For example, this local region may be a portion of the base mesh where texture resolution (image quality) is reduced by applying the first unfolding method.

[0091] In other words, in this case, a vertex that transmits (CCid, 0) in the UV list is considered by the decoder to be a vertex of a patch in the area to which the orthoAtlas is applied, and the UV coordinates are derived from the patch information. In other words, this (CCid, 0) is converted into the UV coordinates of that vertex and used. This process is performed, for example, when reconstructing a base mesh. Also, a vertex that transmits (u, v) (v>0) in the UV list is considered by the decoder to be a vertex of a patch in the local area to which the UVAtlas is applied, and these values ​​are used as the UV coordinates.

[0092] By doing so, it is possible to easily distinguish between a patch to which the orthoAtlas is applied and a patch to which the UVAtlas is applied.

[0093] Furthermore, for example, as shown below, for the vertices of a patch to which a UV Atlas is applied, a list of face identification information (face ID list), a list of UV coordinates, and UV connectivity may be transmitted as meta information separate from the UV list. In this case, the decoder can derive the UV coordinates based on the meta information. This makes it easy to distinguish between a patch to which an orthoAtlas is applied and a patch to which a UV Atlas is applied.

[0094] face ID list = (5, 6, 10, 11,15,…) UV coordinate list = {uv0(u0, v0), uv1(u1, v1), uv2(u2, v2), …} UV connectivity = {(0, 1, 2), …}

[0095] For example, an atlas information generation unit of the first information processing device may generate, as meta-information, identification information of each face, coordinate information of each vertex, and information regarding each connection present in the local region. Furthermore, an atlas information decoding unit of the second information processing device may decode the encoded data of the extracted atlas information to generate atlas information including, as meta-information, identification information of each face, coordinate information of each vertex, and information regarding each connection present in the local region. This local region is a region to which the second unfolding method is applied. For example, this local region may be a portion of the base mesh where texture resolution (image quality) is reduced by applying the first unfolding method.

[0096] Of course, the UV coordinates of the vertices of the patch to which the UV Atlas is applied may be transmitted by methods other than these.

[0097] Note that the control of whether to apply the orthoAtlas or the UVAtlas can be performed on any data unit. For example, the selection may be made on a triangle-by-triangle basis or on a patch-by-patch basis. Furthermore, any criteria may be used to determine which unfolding method to apply. For example, the texture image quality (PSNR, etc.) may be actually derived, and the unfolding method may be selected based on that value. Furthermore, the unfolding method may be selected based on information that can estimate whether the texture resolution level is high or low, such as the face direction relative to the projection direction.

[0098] As described above, by applying UV Atlas to a local region of the base mesh and ortho Atlas to other regions, the texture can be projected in a more appropriate direction, thereby suppressing a reduction in texture resolution. Furthermore, an increase in patch division (an increase in the number of patches) can be suppressed, and an increase in patch information (an increase in the amount of meta-information) can be suppressed. Therefore, a reduction in coding efficiency can be suppressed. For example, this local region may be a portion of the base mesh where the texture resolution (image quality) is reduced by applying ortho Atlas.

[0099] <Method 2-4> When the above-mentioned method 2 is applied, the texture of some triangles may be arranged according to a rule, as shown in the seventh row from the top of the table in Fig. 10. In other words, as the above-mentioned second expansion method, an expansion method in which textures are arranged in the attribute map according to a predetermined rule may be applied.

[0100] Note that such regularly arranged textures may be arranged in a state in which they are applied to the face. That is, they do not have to be projected images projected in the projection direction (e.g., six directions). In other words, this texture can be considered a projected image projected perpendicularly to the face. Therefore, this texture can suppress reduction in resolution due to the projection direction. Furthermore, by ensuring an area of ​​sufficient size as the area in which the texture is to be arranged, the texture can be arranged with sufficient resolution.

[0101] In other words, the resolution of a texture arranged in this order can generally be increased more easily than the resolution of a texture arranged by applying orthoAtlas. Therefore, orthoAtlas is applied to areas where sufficient resolution can be achieved even with orthoAtlas (areas other than the local area described below), and this expansion method (an expansion method in which some triangular textures are arranged according to a rule) is applied to local areas where the texture resolution is reduced by applying orthoAtlas. In other words, orthoAtlas and this expansion method are used in combination. This makes it possible to suppress a reduction in texture quality while suppressing an unnecessary increase in the amount of code. Therefore, it is possible to suppress a reduction in coding efficiency. Note that the processing in this case may be performed as in Method 2-1 or Method 2-2 described above.

[0102] In this expansion method, the texture of some triangles is arranged according to a rule, and information about the arranged texture must be transmitted. Therefore, this expansion method can be said to involve the transmission of information about regularly mapped textures.

[0103] For example, the second unfolding method may involve transmitting information about regularly mapped textures, in which an atlas information generating unit of the first information processing device generates information about the textures as meta information, and an attribute map generating unit uses the generated information about the textures to regularly map textures corresponding to faces of the base mesh onto the attribute map. Furthermore, an atlas information decoding unit of the second information processing device may generate atlas information including meta information about the textures by decoding encoded data of the atlas information extracted from the bitstream. Furthermore, a display processing unit of the second information processing device may use the meta information included in the atlas information to extract textures corresponding to faces of the base mesh that are regularly mapped onto the attribute map in accordance with the second unfolding method. For example, the second unfolding method may be applied to a local region of the base mesh. For example, this local region may be a portion of the base mesh where texture resolution (image quality) is reduced by applying the first unfolding method.

[0104] <Method 2-4-1> When the above-mentioned Method 2-4 is applied, the texture may be arranged according to a predetermined rule, as shown in the eighth row from the top of the table in FIG. 10 (Method 2-4-1).

[0105] For example, as shown in Fig. 14, textures 161 corresponding to faces (triangles) of a base map may be regularly laid out in areas where no patches are placed in an attribute map 160 in which a group of patches to which orthoAtlas is applied are placed. In Fig. 14, only one gray triangle is assigned the reference number "161", but all gray squares in the attribute map 160 are regularly laid out textures 161.

[0106] That is, in this case, textures corresponding to some faces are arranged at predetermined positions according to a predetermined rule. The textures to be arranged in this manner may be any texture. For example, the textures may be textures corresponding to a group of triangles (group of faces) belonging to a patch that has become smaller than a predetermined standard due to patch division (for example, a patch having fewer than a predetermined number of triangles), or a group of triangles of a patch for which a separate high-resolution texture is to be transmitted in order to improve the texture resolution.

[0107] As described above, this expansion method requires the transmission of information about regularly arranged textures. Because the textures are arranged according to a rule, the decoder can determine the area in which the textures are arranged from the arrangement order based on the rule. In other words, for example, if a list of faces is transmitted in which the arrangement order is indicated by the position (the number of faces), the decoder can determine the area in which the texture corresponding to each face is arranged based on its position in the list.

[0108] Therefore, instead of patch information or UV coordinates, the texture information may be transmitted as a list of identification information (face IDs) of triangles (faces) corresponding to each regularly arranged texture. In other words, the texture information may include a list of identification information of faces corresponding to regularly mapped textures. For example, as shown in FIG. 14 , assume that texture 161 is arranged at the top of attribute map 160, starting from the left. If the first face ID in the face ID list is "ID5," it can be seen that the texture corresponding to this face with "ID5" is arranged in the upper left corner of attribute map 160, as shown in FIG. 14 . Therefore, the transmission of patch information or UV information can be omitted, thereby suppressing an increase in the amount of code.

[0109] Note that instead of the identification of a face, the identification of the patch to which the face belongs (e.g. CC id) may be transmitted, i.e. the information about textures may include a list of identifications of patches to which faces corresponding to regularly mapped textures belong.

[0110] Furthermore, since the shape of a mesh face (triangle) is not constant, textures that are arranged in an attribute map according to a rule may be appropriately shaped (deformed, enlarged, reduced, rotated, etc.) to fit the shape of the region when arranged in the attribute map. The texture arrangement pattern shown in FIG. 14 is merely an example. The texture arrangement may be regular in any way and is not limited to the examples shown in FIGS. 14 and 15. For example, although the textures are arranged horizontally in the figures in FIGS. 14 and 15, the textures may also be arranged vertically in the figures. Of course, other arrangements are also possible.

[0111] Furthermore, the area (shape, size, position, etc.) in which the texture is arranged according to this predetermined rule is arbitrary. For example, the shape of this area may be any shape and is not limited to the right triangle in the example of Figure 14. Furthermore, the size and position of this area are also not limited to the example of Figure 14.

[0112] <Method 2-4-2> Furthermore, when the above-mentioned Method 2-4 is applied, the rules may be set as shown in the bottom row of the table in Fig. 10. In other words, textures may be arranged according to the set rules (Method 2-4-2).

[0113] For example, the shape, size, position, etc. of the area where the texture is placed according to this predetermined rule may be variable. In other words, it may be possible to set this area (its shape, size, position, etc.). In this case, information indicating the settings that define this area (its shape, size, position, etc.) may be transmitted as meta information (atlas information).

[0114] For example, the information about the texture may include a list of identities of faces that correspond to the regularly mapped texture, and parameters about the area to which the texture is regularly mapped.

[0115] This parameter may be any parameter. For example, this parameter may include information indicating the length of the sides of the region. For example, this parameter may be information indicating the length of one side, or information indicating the lengths of each side. For example, if the shape of the region is fixed, the size of the region can be set by specifying the length of one side. The shape of the region can also be set by specifying the length of each side. Furthermore, instead of the length of the sides, the coordinates of the vertices or sides of the region may be specified. Furthermore, this parameter may include information specifying the position, scale (magnification), rotation, etc. of the region.

[0116] In this way, by making the area where the texture is placed (its shape, size, position, etc.) variable, the texture can be placed in an area that is more suitable for that texture, which makes it possible to suppress degradation of the texture's image quality, for example.

[0117] Furthermore, there may be multiple types of shapes, sizes, positions, etc. of regions in which textures are arranged. For example, multiple sizes of regions may be prepared, and each texture may be arranged in a region of a size corresponding to the resolution required for that texture. For example, in the attribute map 171 shown in FIG. 15 , textures 181, 182, and 183 are arranged at different resolutions (sizes). Among these, texture 181 is the largest and texture 183 is the smallest. In other words, the resolution required for texture 181 is the highest and the resolution required for texture 183 is the lowest. In this way, textures can be arranged at sizes corresponding to the resolution required for the texture, thereby suppressing an unnecessary increase in the amount of code for the attribute map while suppressing a reduction in the resolution of the texture.

[0118] In such a case, for example, the above-mentioned parameters may include information indicating the length of the sides of the region for each resolution, that is, information indicating the length of the sides of the region for each size of the region may be transmitted as a parameter. In this way, regions of each size can be set individually.

[0119] When arranging textures at multiple sizes (resolutions) in this way, the textures may be arranged together (arranged contiguously) for each size (resolution), as in the example shown in FIG. 15 . In this case, the number of textures (regions) of each size (resolution) to be arranged may be variable. That is, the number of textures (regions) may be set. For example, the above-mentioned parameters may include information indicating the number of regions for each resolution. For example, the maximum number of textures (regions) that can be arranged may be specified. In this case, information indicating this maximum value may be included in the above-mentioned parameters. Furthermore, the number of textures (regions) to be arranged in a row (or the maximum value thereof) may be specified. In this case, information indicating this number (or the maximum value) may be included in the above-mentioned parameters.

[0120] Furthermore, textures may be arranged with a gap between them. For example, in the attribute map 172 shown in FIG. 15, textures 184 are arranged with a predetermined gap between them. By forming gaps between the textures in this way, it is possible to perform a dilation process or the like on the attribute map. This makes it possible to suppress a decrease in the encoding efficiency of the attribute map. In such a case, the length of the gap between the textures (i.e., the size of the gap) may be specified. For example, information indicating the length of the gap between the regularly mapped textures may be included in the above-mentioned parameters.

[0121] The parameters may include parameters relating to rules other than areas, such as information specifying the order in which textures are arranged.

[0122] Furthermore, this parameter may be applied to only one frame, or to multiple frames (a predetermined frame interval). That is, the parameter may be specified for each frame, or may be specified for each predetermined frame interval. Furthermore, difference information from the previous time may be specified as the parameter.

[0123] <Combination> Each of the above-described methods can be applied in combination with other methods as appropriate. For example, the above-described methods 1 and 2 can be applied in combination.

[0124] For example, Method 2 may be applied to a first information processing apparatus to which Method 1 is applied. In this case, in the first information processing apparatus to which Method 1 is applied, an atlas information generation unit may expand a base mesh into a two-dimensional region using a first expansion method involving transmission of patch information, expand a local region of the base mesh into a two-dimensional region using a second expansion method different from the first expansion method, and generate, as atlas information, common patch information for the base mesh expanded by the first expansion method and meta information for the local region generated by the second expansion method. Furthermore, an attribute map generation unit may generate an attribute map by using the generated common patch information to map a projection image corresponding to the base mesh in accordance with the first expansion method, and then using the generated meta information to map a texture corresponding to the local region in accordance with the second expansion method.

[0125] Method 2 may also be applied to a second information processing device to which Method 1 is applied. In this case, in the second information processing device to which Method 1 is applied, an atlas information decoding unit may decode the encoded data of the extracted atlas information to generate atlas information that corresponds to a first unfolding method involving transmission of patch information and a second unfolding method different from the first unfolding method, and that includes common patch information for a base mesh unfolded by the first unfolding method and meta-information for a local region of the base mesh unfolded by the second unfolding method. Furthermore, a display processing unit may use the common patch information included in the atlas information to extract a texture corresponding to the base mesh mapped to an attribute map so as to correspond to the first unfolding method, use the meta-information included in the atlas information to extract a texture corresponding to the local region mapped to the attribute map so as to correspond to the second unfolding method, and apply the extracted projected image and texture to the decoded mesh for rendering to generate an image for display.

[0126] This local region may be, for example, a portion of the base mesh where the texture resolution (image quality) is reduced by applying the first unfolding method. By doing so, it is possible to obtain the same effect as that described above in <Method 2>. Of course, in this case, other methods that are applicable when Method 2 is applied (e.g., Methods 2-1 to 2-4) can also be applied.

[0127] For example, Method 1 may be applied to a first information processing device to which Method 2 is applied. In this case, in the first information processing device to which Method 2 is applied, an atlas information generation unit may generate, as atlas information, common patch information that is patch information common to multiple frames for a base mesh unfolded by the first unfolding method. Furthermore, an attribute map generation unit may use the generated common patch information to map a projection image corresponding to the base mesh so as to correspond to the first unfolding method.

[0128] Furthermore, Method 1 may be applied to a second information processing device to which Method 2 is applied. In this case, in the second information processing device to which Method 2 is applied, an atlas information decoding unit may decode encoded data of the extracted atlas information to generate atlas information including common patch information that is patch information common to multiple frames for a base mesh unfolded by the first unfolding method. Furthermore, a display processing unit may use the common patch information included in the atlas information to extract a texture corresponding to the base mesh mapped to the attribute map so as to correspond to the first unfolding method.

[0129] By doing so, it is possible to obtain the same effect as that described above in <Method 1>. Of course, in this case, other methods that can be applied when Method 1 is applied (for example, Method 1-1, etc.) can also be applied.

[0130] 4. First embodiment Encoding device The present technology can be applied to an encoding device that encodes a mesh. Fig. 16 is a block diagram showing an example of the configuration of an encoding device that is one aspect of an information processing device to which the present technology is applied. The encoding device 300 shown in Fig. 16 is a device that encodes a mesh.

[0131] Fig. 16 shows the main processing units, data flows, etc., but does not necessarily include everything. In other words, in encoding device 300, there may be processing units that are not shown as blocks in Fig. 16, and there may be processing and data flows that are not shown as arrows, etc. in Fig. 16.

[0132] The encoding device 300 encodes a mesh using a method essentially similar to the V-DMC method described in the aforementioned non-patent document, except that the present technology is applied. For example, the encoding device 300 acquires an original mesh to be encoded and an attribute map including a texture corresponding to the original mesh. Note that the original mesh includes not only information about the mesh's geometry but also information indicating the correspondence with the attribute map (e.g., a UV list). The encoding device 300 encodes the original mesh and attribute map using the V-DMC method, generates a V-DMC bitstream, and outputs it.

[0133] As shown in Fig. 16 , the encoding device 300 includes a pre-processing unit 311 and an encoding unit 312. The pre-processing unit 311 performs pre-processing before encoding. As shown in Fig. 19 , the pre-processing unit 311 includes a mesh decimation unit 321, an atlas information generation unit 322, and a displacement vector generation unit 323.

[0134] The mesh decimation unit 321 performs processing related to mesh decimation. For example, the mesh decimation unit 321 may obtain an original mesh to be input to the encoding device 300. The mesh decimation unit 321 may also perform decimation processing (thinning out vertices) on the original mesh to generate a base mesh. The mesh decimation unit 321 may supply the generated base mesh together with the original mesh to the atlas information generation unit 322.

[0135] The atlas information generation unit 322 performs processing related to the generation of atlas information corresponding to the base mesh. For example, the atlas information generation unit 322 may acquire the base mesh or the original mesh supplied from the mesh decimation unit 321. The atlas information generation unit 322 may also generate atlas information by, for example, UV unwrapping the base mesh. In this case, the atlas information generation unit 322 may generate the atlas information by applying orthoAtlas as a UV unwrapping method. In other words, the atlas information generation unit 322 may generate atlas information corresponding to at least orthoAtlas. The atlas information generation unit 322 may supply the generated atlas information to the displacement vector generation unit 323 together with the base mesh, etc.

[0136] The displacement vector generation unit 323 performs processing related to the generation of displacement vectors. For example, the displacement vector generation unit 323 may acquire a base mesh, atlas information, etc. supplied from the atlas information generation unit 322. Alternatively, the displacement vector generation unit 323 may acquire an original mesh input to the encoding device 300. Alternatively, the displacement vector generation unit 323 may use the information to generate displacement vectors that displace division points generated by subdividing the base mesh. The displacement vector generation unit 323 may supply the generated displacement vectors to the encoding unit 312 together with the base mesh, atlas information, etc.

[0137] The encoding unit 312 performs processing related to encoding of V-DMC data. For example, the encoding unit 312 may acquire an original mesh input to the encoding device 300. The encoding unit 312 may also acquire a base mesh, a displacement vector, atlas information, and the like, supplied from the displacement vector generation unit 323. The encoding unit 312 may also acquire an attribute map input to the encoding device 300. The encoding unit 312 may use this information to encode the atlas information, the base mesh, the displacement vector, and the attribute map, respectively, and generate the respective encoded data. The encoding unit 312 may also multiplex these encoded data as substreams to generate a single bitstream. This bitstream is also referred to as a V-DMC bitstream. The encoding unit 312 may output the generated V-DMC bitstream to the outside of the encoding device 300.

[0138] <Encoding Unit> Fig. 17 is a block diagram showing an example of the main configuration of the encoding unit 312. Note that Fig. 17 shows the main processing units, data flows, etc., and is not limited to what is shown in Fig. 17. In other words, the encoding unit 312 may include processing units that are not shown as blocks in Fig. 17, and may include processing and data flows that are not shown as arrows, etc. in Fig. 17.

[0139] As shown in FIG. 17, the encoding unit 312 has an atlas information encoding unit 351, a base mesh encoding unit 352, a displacement vector correction unit 353, a displacement video encoding unit 354, a mesh reconstruction unit 355, an attribute map conversion unit 356, an attribute video encoding unit 357, and a multiplexing unit 358.

[0140] The atlas information encoding unit 351 performs processing related to encoding of the atlas information. For example, the atlas information encoding unit 351 may acquire atlas information supplied from the displacement vector generation unit 323. The atlas information encoding unit 351 may also encode the acquired atlas information using a predetermined encoding method to generate encoded data of the atlas information. The atlas information encoding unit 351 may also supply the generated encoded data of the atlas information to the multiplexing unit 358.

[0141] The base mesh encoding unit 352 performs processing related to encoding of the base mesh. For example, the base mesh encoding unit 352 may acquire the base mesh supplied from the displacement vector generation unit 323. The base mesh encoding unit 352 may also quantize the acquired base mesh and encode it using a predetermined encoding method (e.g., Draco) to generate encoded data of the base mesh. The base mesh encoding unit 352 may also supply the generated encoded data of the base mesh to the displacement vector correction unit 353. The base mesh encoding unit 352 may also supply the generated encoded data of the base mesh to the multiplexing unit 358.

[0142] The displacement vector correction unit 353 performs processing related to the correction of the displacement vector. For example, the displacement vector correction unit 353 may acquire a base mesh and a displacement vector supplied from the displacement vector generation unit 323. Alternatively, the displacement vector correction unit 353 may acquire encoded data of the base mesh supplied from the base mesh encoding unit 352. The displacement vector correction unit 353 may correct the displacement vector based on this information. For example, the displacement vector correction unit 353 may decode the acquired encoded data of the base mesh, compare the base mesh before and after encoding to determine encoding distortion of the base mesh, and correct the displacement vector in accordance with the encoding distortion. The displacement vector correction unit 353 may supply the corrected displacement vector to the displacement video encoding unit 354. Alternatively, the displacement vector correction unit 353 may dequantize the decoded base mesh and supply it to the mesh reconstruction unit 355.

[0143] The displacement video encoding unit 354 performs processing related to encoding of the displacement video. The displacement video is a moving image having a displacement map, which is a two-dimensional area in which displacement vectors are packed, as frame images. For example, the displacement video encoding unit 354 may acquire displacement vectors supplied from the displacement vector correction unit 353. The displacement video encoding unit 354 may also generate a displacement map by wavelet transforming the displacement vectors, quantizing them, and packing them into a two-dimensional area. The displacement video encoding unit 354 may also generate a displacement video having the displacement map as frame images. The displacement video encoding unit 354 may also encode the generated displacement video using a predetermined encoding method for 2D moving images to generate encoded data of the displacement video. The displacement video encoding unit 354 may also supply the generated encoded data of the displacement video to the multiplexing unit 358. The displacement video encoding unit 354 may also decode the generated encoded data of the displacement video, unpack the displacement vectors from the displacement map, and dequantize the displacement vectors. The displacement video encoder 354 may provide the dequantized displacement vectors to the mesh reconstructor 355 .

[0144] The mesh reconstruction unit 355 performs processing related to mesh reconstruction. For example, the mesh reconstruction unit 355 may acquire a base mesh supplied from the displacement vector correction unit 353. The mesh reconstruction unit 355 may also acquire a displacement vector supplied from the displacement video encoding unit 354. The mesh reconstruction unit 355 may reconstruct a mesh using the acquired displacement vectors. The mesh reconstruction unit 355 may supply the reconstructed mesh to the attribute map conversion unit 356.

[0145] The attribute map conversion unit 356 performs processing related to attribute map conversion. For example, the attribute map conversion unit 356 may acquire a reconstructed mesh supplied from the mesh reconstruction unit 355. The attribute map conversion unit 356 may also acquire atlas information supplied from the displacement vector generation unit 323. The attribute map conversion unit 356 may also acquire an original mesh and attribute map input to the encoding device 300. The attribute map conversion unit 356 may convert the acquired attribute map based on other acquired information. For example, the attribute map conversion unit 356 may convert the attribute map based on the atlas information, the original mesh, etc., so that it corresponds to the reconstructed mesh. In this case, the attribute map conversion unit 356 may convert the attribute map so that it corresponds to at least the orthoAtlas. In other words, the attribute map conversion unit 356 can be said to generate a converted attribute map. Therefore, the attribute map conversion unit 356 can also be called an attribute map generation unit. The attribute map conversion unit 356 may supply the converted attribute map to the attribute video encoding unit 357.

[0146] The attribute video encoding unit 357 performs processing related to encoding of the attribute video. For example, the attribute video encoding unit 357 may acquire an attribute map supplied from the attribute map conversion unit 356. The attribute video encoding unit 357 may also generate attribute video using the acquired attribute map as frame images. The attribute video encoding unit 357 may also encode the generated attribute video using a predetermined encoding method for 2D moving images to generate encoded data of the attribute video. The attribute video encoding unit 357 may also supply the generated encoded data of the attribute video to the multiplexing unit 358.

[0147] The multiplexing unit 358 performs processing related to multiplexing of encoded data (substreams). For example, the multiplexing unit 358 may acquire encoded data of atlas information supplied from the atlas information encoding unit 351. Alternatively, the multiplexing unit 358 may acquire encoded data of base meshes supplied from the base mesh encoding unit 352. Alternatively, the multiplexing unit 358 may acquire encoded data of displacement video supplied from the displacement video encoding unit 354. Alternatively, the multiplexing unit 358 may acquire encoded data of attribute video supplied from the attribute video encoding unit 357. The multiplexing unit 358 may multiplex these encoded data as substreams to generate a V-DMC bitstream. Therefore, the multiplexing unit 358 can also be referred to as a bitstream generation unit. The multiplexing unit 358 may output the generated V-DMC bitstream to an external device from the encoding device 300. For example, the multiplexing unit 358 may supply the V-DMC bitstream to a decoding device 400, which will be described later. Therefore, the multiplexing unit 358 can also be said to be a supply unit (providing unit) of the V-DMC bitstream.

[0148] In the encoding device 300 configured as above, the various methods (the present technology) described above in <3. Improvement of UV coordinate generation method> may be applied.

[0149] For example, the encoding device 300 may be the first information processing device described above, and the method 1 of the present technology may be applied thereto. In this case, in the encoding device 300, the atlas information generation unit 322 may unfold a base mesh into a two-dimensional region using an unfolding method involving transmission of patch information, and generate, as atlas information, common patch information that is patch information common to multiple frames. The atlas information encoding unit 351 may encode atlas information including the generated common patch information. The attribute map conversion unit 356 may use the generated common patch information to map a projected image of the texture of the original mesh into a two-dimensional region in accordance with the unfolding method, thereby generating an attribute map. The attribute video encoding unit 357 may encode an attribute video using the generated attribute map as a frame image. The multiplexing unit 358 may generate a bitstream including encoded data of the base mesh, encoded data of the displacement video, encoded data of the attribute video, and encoded data of the atlas information.

[0150] Alternatively, the encoding device 300 may be the first information processing device described above, and the method 2 of the present technology may be applied. In this case, in the encoding device 300, the atlas information generation unit 322 may expand a base mesh into a two-dimensional region using a first expansion method involving transmission of patch information, expand a local region of the base mesh into a two-dimensional region using a second expansion method different from the first expansion method, and generate, as atlas information, patch information for the base mesh expanded using the first expansion method and meta-information for the local region generated using the second expansion method. The atlas information encoding unit 351 may encode atlas information including the generated patch information and meta-information. The attribute map conversion unit 356 may use the generated patch information to map a projection image of a texture corresponding to the base mesh onto a two-dimensional region in accordance with the first expansion method, and then use the generated meta-information to map a texture corresponding to the local region onto a two-dimensional region in accordance with the second expansion method, thereby generating an attribute map. The attribute video encoding unit 357 may encode an attribute video using the generated attribute map as a frame image. The multiplexing unit 358 may generate a bitstream including encoded data of the base mesh, encoded data of the displacement video, encoded data of the attribute video, and encoded data of the atlas information. This local region may be, for example, a portion of the base mesh where texture resolution (image quality) is reduced by applying the first unfolding method.

[0151] Of course, the encoding device 300 may further apply other methods that are applicable when Method 1 or Method 2 is applied.

[0152] By doing so, the encoding device 300 can obtain the same effect as that described above in <3. Improvement of UV coordinate generation method>. In other words, the encoding device 300 can suppress a decrease in the encoding efficiency of 3D data including a base mesh and a displacement vector when applying an unfolding method involving the transmission of patch information.

[0153] <Flow of Encoding Process> An example of the flow of the encoding process executed by the encoding device 300 will be described with reference to the flowchart in Fig. 18. Note that the flowchart in Fig. 18 shows an example of the flow of the encoding process when method 1 of the present technology is applied.

[0154] When the encoding process starts, in step S301, the mesh decimation unit 321 of the encoding device 300 decimates the original mesh to be encoded to generate a base mesh.

[0155] In step S302, the atlas information generation unit 322 applies orthoAtlas to the base mesh to generate atlas information.

[0156] In step S303, the atlas information generator 322 generates common patch information as necessary. That is, if common patch information to be applied to the frame being processed has not been generated, the atlas information generator 322 generates common patch information. On the other hand, if common patch information to be applied to the frame being processed has already been generated, the atlas information generator 322 skips (omits) this process.

[0157] In step S304, the atlas information generator 322 generates unique patch information as necessary. That is, if unique patch information exists for the frame to be processed, the atlas information generator 322 generates unique patch information corresponding to the frame to be processed. On the other hand, if unique patch information does not exist for the frame to be processed, the atlas information generator 322 skips (omits) this process.

[0158] In step S305, the displacement vector generating unit 323 generates a displacement vector.

[0159] In step S306, the encoding unit 312 performs encoding processing on the V-DMC data (atlas information, base mesh, displacement vector, and attribute map) generated as described above, and generates a V-DMC bitstream.

[0160] In step S307, the encoding unit 312 determines whether all frames have been processed. If unprocessed frames remain, the process returns to step S301, and the subsequent processes are executed. That is, the processes of steps S301 to S307 are executed for each frame. Then, if it is determined in step S307 that all frames have been processed (that is, if there are no more frames to be processed), the encoding process ends.

[0161] <Flow of Encoding Process> Next, an example of the flow of the encoding process executed in step S306 in FIG. 18 will be described with reference to the flowchart in FIG.

[0162] When the encoding process starts, the atlas information encoding unit 351 encodes the atlas information including the common patch information in step S311.

[0163] In step S312, the base mesh encoding unit 352 encodes the base mesh.

[0164] In step S313, the displacement vector correction unit 353 corrects the displacement vector.

[0165] In step S314, the displaced video encoding unit 354 encodes a displaced video in which the displaced map in which the corrected displaced vectors are packed is used as a frame image.

[0166] In step S315, the mesh reconstructing unit 355 reconstructs the mesh.

[0167] In step S316, the attribute map conversion unit 356 generates an attribute map corresponding to the orthoAtlas using the atlas information including the common patch information.

[0168] In step S317, the attribute video encoding unit 357 encodes the attribute video using the attribute map as a frame image.

[0169] In step S318, the multiplexing unit 358 multiplexes the coded data of the atlas information, the coded data of the bitstream, the coded data of the displacement video, and the coded data of the attribute video to generate a V-DMC bitstream.

[0170] When the process of step S318 ends, the encoding process of FIG. 19 ends, and the process returns to the process of FIG.

[0171] By performing each process as described above, the encoding device 300 can apply Method 1. Therefore, the encoding device 300 can suppress a decrease in the encoding efficiency of 3D data including base meshes and displacement vectors when applying an unfolding method involving the transmission of patch information.

[0172] <Flow of Encoding Process> Another example of the flow of the encoding process executed by the encoding device 300 will be described with reference to the flowchart in Fig. 20. Note that the flowchart in Fig. 20 shows an example of the flow of the encoding process when method 2-3 of the present technology is applied.

[0173] When the encoding process starts, in step S331, the mesh decimation unit 321 of the encoding device 300 decimates the original mesh to be encoded to generate a base mesh.

[0174] In step S332, the atlas information generation unit 322 applies orthoAtlas to the entire base mesh to generate atlas information.

[0175] In step S333, the atlas information generation unit 322 applies UV Atlas to some patches of the base mesh (such as patches where the texture quality is reduced) to generate atlas information.

[0176] In step S334, the displacement vector generating unit 323 generates a displacement vector.

[0177] In step S335, the encoding unit 312 performs encoding processing on the V-DMC data (atlas information, base mesh, displacement vector, and attribute map) generated as described above, and generates a V-DMC bitstream.

[0178] In step S336, the encoding unit 312 determines whether all frames have been processed. If unprocessed frames remain, the process returns to step S331, and the subsequent processes are executed. That is, the processes of steps S331 to S336 are executed for each frame. Then, if it is determined in step S336 that all frames have been processed (that is, if there are no more frames to be processed), the encoding process ends.

[0179] <Flow of Encoding Process> Next, an example of the flow of the encoding process executed in step S335 of FIG. 20 will be described with reference to the flowchart of FIG.

[0180] When the encoding process starts, in step S341, the atlas information encoding unit 351 encodes the atlas information corresponding to the orthoAtlas and the UVAtlas.

[0181] In step S342, the base mesh encoding unit 352 encodes the base mesh.

[0182] In step S343, the displacement vector correction unit 353 corrects the displacement vector.

[0183] In step S344, the displaced video encoding unit 354 encodes a displaced video in which the displaced map in which the corrected displaced vectors are packed is used as a frame image.

[0184] In step S345, the mesh reconstructing unit 355 reconstructs the mesh.

[0185] In step S346, the attribute map conversion unit 356 uses the atlas information corresponding to the orthoAtlas and the UVAtlas to generate an attribute map including patches corresponding to the orthoAtlas and patches corresponding to the UVAtlas.

[0186] In step S347, the attribute video encoding unit 357 encodes the attribute video using the attribute map as a frame image.

[0187] In step S348, the multiplexing unit 358 multiplexes the coded data of the atlas information, the coded data of the bitstream, the coded data of the displacement video, and the coded data of the attribute video to generate a V-DMC bitstream.

[0188] When the process of step S348 ends, the encoding process of FIG. 21 ends, and the process returns to the process of FIG.

[0189] By performing the above-described processes, the encoding device 300 can apply Method 2-3. Therefore, the encoding device 300 can suppress a decrease in the encoding efficiency of 3D data including base meshes and displacement vectors when applying an unfolding method involving the transmission of patch information.

[0190] <Flow of Encoding Process> Another example of the flow of the encoding process executed by the encoding device 300 will be described with reference to the flowchart in Fig. 22. Note that the flowchart in Fig. 22 shows an example of the flow of the encoding process when method 2-4 of the present technology is applied.

[0191] When the encoding process starts, in step S361, the mesh decimation unit 321 of the encoding device 300 decimates the original mesh to be encoded to generate a base mesh.

[0192] In step S362, the atlas information generation unit 322 applies orthoAtlas to the entire base mesh to generate atlas information.

[0193] In step S363, the atlas information generating unit 322 generates atlas information for arranging textures according to rules for some patches of the base mesh (such as patches where the texture quality is reduced).

[0194] In step S364, the displacement vector generating unit 323 generates a displacement vector.

[0195] In step S365, the encoding unit 312 performs encoding processing on the V-DMC data (atlas information, base mesh, displacement vector, and attribute map) generated as described above, and generates a V-DMC bitstream.

[0196] In step S366, the encoding unit 312 determines whether all frames have been processed. If unprocessed frames remain, the process returns to step S361, and the subsequent processes are executed. That is, the processes of steps S361 to S366 are executed for each frame. Then, if it is determined in step S366 that all frames have been processed (that is, if there are no more frames to be processed), the encoding process ends.

[0197] <Flow of Encoding Process> Next, an example of the flow of the encoding process executed in step S335 of FIG. 22 will be described with reference to the flowchart of FIG.

[0198] When the encoding process starts, in step S371, the atlas information encoding unit 351 encodes atlas information corresponding to the regular arrangement of orthoAtlas and texture.

[0199] In step S372, the base mesh encoding unit 352 encodes the base mesh.

[0200] In step S373, the displacement vector correction unit 353 corrects the displacement vector.

[0201] In step S374, the displaced video encoding unit 354 encodes a displaced video in which the displaced map in which the corrected displaced vectors are packed is used as a frame image.

[0202] In step S375, the mesh reconstructing unit 355 reconstructs the mesh.

[0203] In step S376, the attribute map conversion unit 356 uses the orthoAtlas and atlas information corresponding to the regular arrangement of textures to generate an attribute map including patches corresponding to the orthoAtlas and triangles arranged according to the rules.

[0204] In step S377, the attribute video encoding unit 357 encodes the attribute video using the attribute map as a frame image.

[0205] In step S378, the multiplexing unit 358 multiplexes the coded data of the atlas information, the coded data of the bitstream, the coded data of the displacement video, and the coded data of the attribute video to generate a V-DMC bitstream.

[0206] When the process of step S378 ends, the encoding process of FIG. 23 ends, and the process returns to the process of FIG.

[0207] By performing the above-described processes, the encoding device 300 can apply methods 2 to 4. Therefore, the encoding device 300 can suppress a decrease in the encoding efficiency of 3D data including base meshes and displacement vectors when applying an unfolding method involving the transmission of patch information.

[0208] 24 is a block diagram showing an example of the configuration of a decoding device, which is one aspect of an information processing device to which the present technology is applied. The decoding device 400 shown in Fig. 24 is a device that decodes, for example, encoded data of a mesh generated in the encoding device 300 (a V-DMC bitstream generated by the multiplexing unit 358).

[0209] Fig. 24 shows the main processing units, data flows, etc., but does not necessarily include all of them. In other words, in the decoding device 400, there may be processing units that are not shown as blocks in Fig. 24, and there may be processing and data flows that are not shown as arrows, etc. in Fig. 24.

[0210] The decoding device 400 decodes coded data of meshes that have been coded using a method essentially similar to the V-DMC method described in the aforementioned non-patent document, except that the present technology is applied. For example, the decoding device 400 obtains a V-DMC bitstream. This V-DMC bitstream may be generated by, for example, the coding device 300. The decoding device 400 decodes the V-DMC bitstream, reconstructs a mesh (also referred to as a decoded mesh), applies a texture to the decoded mesh, generates a display image for displaying the decoded mesh, and outputs the display image to an external device.

[0211] As shown in FIG. 24, the decoding device 400 has a demultiplexing unit 411, an atlas information decoding unit 412, a base mesh decoding unit 413, a base mesh reconstruction unit 414, a subdivision unit 415, a displacement video decoding unit 416, an unpacking unit 417, a displacement vector application unit 418, an attribute video decoding unit 419, and a display processing unit 420.

[0212] The demultiplexing unit 411 performs demultiplexing processing. For example, the demultiplexing unit 411 may acquire a V-DMC bitstream to be decoded and supplied to the decoding device 400. The demultiplexing unit 411 may also demultiplex the acquired V-DMC bitstream to extract coded data of atlas information, coded data of base meshes, coded data of displacement video, and coded data of attribute video. Therefore, the demultiplexing unit 411 can also be considered an acquisition unit for the V-DMC bitstream or various information contained in the V-DMC bitstream. The demultiplexing unit 411 may supply the extracted coded data of atlas information to the atlas information decoding unit 412. The demultiplexing unit 411 may also supply the extracted coded data of base meshes to the base mesh decoding unit 413. The demultiplexing unit 411 may also supply the extracted coded data of displacement video to the displacement video decoding unit 416. Furthermore, the demultiplexing unit 411 may supply the extracted coded data of the attribute video to the attribute video decoding unit 419 .

[0213] The atlas information decoding unit 412 performs processing related to decoding of the coded data of the atlas information. For example, the atlas information decoding unit 412 may acquire the coded data of the atlas information supplied from the demultiplexing unit 411. The atlas information decoding unit 412 may also decode the acquired coded data of the atlas information using a predetermined decoding method to generate (restore) the atlas information. The atlas information decoding unit 412 may supply the generated atlas information to the base mesh reconstruction unit 414 and the display processing unit 420.

[0214] The base mesh decoding unit 413 performs processing related to decoding of the coded data of the base mesh. For example, the base mesh decoding unit 413 may acquire the coded data of the base mesh supplied from the demultiplexing unit 411. The base mesh decoding unit 413 may also decode the acquired coded data of the base mesh using a predetermined decoding method (e.g., Draco) to generate (restore) information about the base mesh (e.g., a vertex list, a triangle list, etc.). The base mesh decoding unit 413 may supply the generated information about the base mesh to the base mesh reconstruction unit 414.

[0215] The base mesh reconstruction unit 414 executes processing related to the reconstruction of the base mesh. For example, the base mesh reconstruction unit 414 may acquire information about the base mesh supplied from the base mesh decoding unit 413. Alternatively, the base mesh reconstruction unit 414 may acquire atlas information supplied from the atlas information decoding unit 412. The base mesh reconstruction unit 414 may reconstruct the base mesh using the acquired atlas information and information about the base mesh. The base mesh reconstruction unit 414 may supply the generated base mesh to the subdivision unit 415.

[0216] The subdivision unit 415 performs processing related to subdivision of triangles of a base mesh. For example, the subdivision unit 415 may obtain a base mesh provided from the base mesh reconstruction unit 414. The subdivision unit 415 may subdivide triangles of the base mesh to generate division points. The subdivision unit 415 may provide the subdivided base mesh to the position vector application unit 418.

[0217] The displacement video decoding unit 416 performs processing related to decoding of encoded data of the displacement video. For example, the displacement video decoding unit 416 may acquire encoded data of the displacement video supplied from the demultiplexing unit 411. Furthermore, the displacement video decoding unit 416 may decode the acquired encoded data of the displacement video using a predetermined decoding method for 2D moving images to generate (restore) the displacement video. The displacement video decoding unit 416 may supply the generated displacement video to the unpacking unit 417.

[0218] The unpacking unit 417 performs processing related to unpacking of the displacement vector. For example, the unpacking unit 417 may acquire the displacement video supplied from the displacement video decoding unit 416. The unpacking unit 417 may also unpack the displacement vector from a displacement map, which is a frame image of the displacement video. The unpacking unit 417 may supply the displacement vector obtained in this way to the displacement vector application unit 418.

[0219] The displacement vector application unit 418 performs processing related to application of a displacement vector to a subdivided base mesh. For example, the displacement vector application unit 418 may obtain the subdivided base mesh supplied from the subdivision unit 415. Alternatively, the displacement vector application unit 418 may obtain a displacement vector supplied from the unpacking unit 417. Alternatively, the displacement vector application unit 418 may apply a displacement vector to a division point of the subdivided base mesh. In other words, the displacement vector application unit 418 may generate a decoded mesh. The displacement vector application unit 418 may supply the generated decoded mesh to the display processing unit 420.

[0220] The attribute video decoding unit 419 performs processing related to decoding of the attribute video. For example, the attribute video decoding unit 419 may acquire coded data of the attribute video supplied from the demultiplexing unit 411. The attribute video decoding unit 419 may also decode the acquired coded data of the attribute video using a predetermined decoding method for 2D moving images to generate (restore) the attribute video. The attribute video decoding unit 419 may supply an attribute map, which is a frame image of the generated attribute video, to the display processing unit 420.

[0221] The display processing unit 420 performs processing related to mesh display. For example, the display processing unit 420 may acquire atlas information supplied from the atlas information decoding unit 412. The display processing unit 420 may also acquire a decoded mesh supplied from the displacement vector application unit 418. The display processing unit 420 may also acquire an attribute map supplied from the attribute video decoding unit 419. The display processing unit 420 may use the acquired atlas information to extract a texture from the attribute map and apply it to a face of the decoded mesh corresponding to the texture. In other words, the display processing unit 420 may attach the texture to the face. The display processing unit 420 may render the decoded mesh to which the texture has been applied and generate a display image for displaying the decoded mesh to which the texture has been applied. The display processing unit 420 may then supply the generated display image to an external device outside the decoding device 400, causing the display image to be displayed by another device or the like.

[0222] In the decoding device 400 configured as above, the various methods (the present technology) described above in <3. Improvement of UV coordinate generation method> may be applied.

[0223] For example, the decoding device 400 may be the second information processing device described above, and the method 1 of the present technology may be applied thereto. In this case, in the decoding device 400, the demultiplexing unit 411 may demultiplex the bitstream and extract coded data of the base mesh, coded data of the displacement video, coded data of the attribute video, and coded data of the atlas information. The atlas information decoding unit 412 may decode the coded data of the extracted atlas information to generate atlas information that corresponds to an unfolding method involving transmission of patch information and includes common patch information that is patch information common to multiple frames. The base mesh decoding unit 413 may decode the coded data of the extracted base mesh to generate information related to the base mesh. The base mesh reconstruction unit 414 may reconstruct a base mesh from information related to the base mesh using the atlas information. The attribute video decoding unit 419 may decode the coded data of the attribute video to generate an attribute map configured as a frame image of the attribute video. In addition, the display processing unit 420 may use common patch information included in the atlas information to extract a projected image of the texture of the original mesh mapped to the attribute map in accordance with the expansion method, and apply it to the decoded mesh and render it to generate an image for display.

[0224] Alternatively, the decoding device 400 may be the second information processing device described above, and the method 2 of the present technology may be applied. In this case, in the decoding device 400, the demultiplexing unit 411 may demultiplex the bitstream and extract coded data of the base mesh, coded data of the displacement video, coded data of the attribute video, and coded data of the atlas information. The atlas information decoding unit 412 may decode the coded data of the extracted atlas information to generate atlas information corresponding to a first unfolding method involving transmission of patch information and a second unfolding method different from the first unfolding method, the atlas information including patch information about the base mesh unfolded by the first unfolding method and meta-information about local regions of the base mesh unfolded by the second unfolding method. The base mesh decoding unit 413 may generate information about the base mesh by decoding the coded data of the extracted base mesh. The base mesh reconstruction unit 414 may use the atlas information to reconstruct the base mesh from the information about the base mesh. The attribute video decoding unit 419 may decode the encoded data of the attribute video to generate an attribute map configured as frame images of the attribute video. The display processing unit 420 may use patch information included in the atlas information to extract a projection image of a texture corresponding to a base mesh mapped to the attribute map so as to correspond to the first unfolding method, use meta information included in the atlas information to extract a texture corresponding to a local region mapped to the attribute map so as to correspond to the second unfolding method, and apply the extracted projection image and texture to the decoded mesh for rendering to generate an image for display. This local region may be, for example, a portion of the base mesh where texture resolution (image quality) is reduced by applying the first unfolding method.

[0225] Of course, the decoding device 400 may further apply other methods that are applicable when Method 1 or Method 2 is applied.

[0226] By doing so, the decoding device 400 can obtain the same effect as that described above in <3. Improvement of UV coordinate generation method>. In other words, the decoding device 400 can suppress a decrease in the coding efficiency of 3D data including base meshes and displacement vectors when applying an unfolding method involving the transmission of patch information.

[0227] <Flow of Decoding Process> An example of the flow of the decoding process executed by the decoding device 400 will be described with reference to the flowchart in Fig. 25. Note that the flowchart in Fig. 25 shows an example of the flow of the encoding process when method 1 of the present technology is applied.

[0228] When the decoding process starts, the demultiplexing unit 411 of the decoding device 400 demultiplexes the V-DMC bitstream in step S401.

[0229] In step S402, the atlas information decoding unit 412 decodes the coded data of the atlas information to generate (restore) atlas information including common patch information.

[0230] In step S403, the base mesh decoding unit 413 decodes the coded data of the base mesh and generates (restores) information about the base mesh.

[0231] In step S404, the base mesh reconstructing unit 414 reconstructs the base mesh using the atlas information including the common patch information and information about the base mesh.

[0232] In step S405, the subdivision unit 415 subdivides the reconstructed base mesh to generate division points.

[0233] In step S406, the displaced video decoding unit 416 decodes the encoded data of the displaced video to generate (restore) the displaced video.

[0234] In step S407, the unpacking unit 417 unpacks the displacement map, which is the frame images of the displacement video, and extracts the displacement vectors.

[0235] In step S408, the displacement vector application unit 418 applies the unpacked displacement vectors to the division points of the subdivided base mesh to generate a decoded mesh.

[0236] In step S409, the attribute video decoding unit 419 decodes the coded data of the attribute video to generate (restore) the attribute video.

[0237] In step S410, the display processing unit 420 uses atlas information including common patch information to apply textures extracted from an attribute map, which is a frame image of the attribute video, to the faces of the generated decoded mesh, and renders the decoded mesh to generate a display image.

[0238] When the process of step S410 is completed, the decoding process ends.

[0239] By performing each process as described above, the decoding device 400 can apply Method 1. Therefore, the decoding device 400 can suppress a decrease in the coding efficiency of 3D data including base meshes and displacement vectors when applying an unfolding method involving transmission of patch information.

[0240] <Flow of Decoding Process> Another example of the flow of the decoding process executed by the decoding device 400 will be described with reference to the flowchart of Fig. 26. Note that the flowchart of Fig. 26 illustrates an example of the flow of the encoding process when method 2-3 of the present technology is applied.

[0241] When the decoding process starts, the demultiplexing unit 411 of the decoding device 400 demultiplexes the V-DMC bitstream in step S431.

[0242] In step S432, the atlas information decoding unit 412 decodes the coded data of the atlas information and generates (restores) atlas information corresponding to the orthoAtlas and UVAtlas.

[0243] In step S433, the base mesh decoding unit 413 decodes the coded data of the base mesh and generates (restores) information about the base mesh.

[0244] In step S434, the base mesh reconstruction unit 414 reconstructs the base mesh using atlas information corresponding to the orthoAtlas and UVAtlas, and information about the base mesh. At this time, for example, for vertices UV-expanded by the orthoAtlas, UV coordinates are derived based on the CCid and patch information.

[0245] In step S435, the subdivision unit 415 subdivides the reconstructed base mesh to generate division points.

[0246] In step S436, the displaced video decoding unit 416 decodes the encoded data of the displaced video to generate (restore) the displaced video.

[0247] In step S437, the unpacking unit 417 unpacks the displacement map, which is the frame images of the displacement video, and extracts the displacement vectors.

[0248] In step S438, the displacement vector application unit 418 applies the unpacked displacement vectors to the division points of the subdivided base mesh to generate a decoded mesh.

[0249] In step S439, the attribute video decoding unit 419 decodes the coded data of the attribute video to generate (restore) the attribute video.

[0250] In step S440, the display processing unit 420 uses atlas information corresponding to the orthoAtlas and UVAtlas to apply textures extracted from the attribute map, which is a frame image of the attribute video, to the faces of the generated decoded mesh, and renders the decoded mesh to generate a display image.

[0251] When the process of step S440 is completed, the decoding process ends.

[0252] By performing the processes described above, the decoding device 400 can apply Method 2-3. Therefore, the decoding device 400 can suppress a decrease in the coding efficiency of 3D data including base meshes and displacement vectors when applying an unfolding method involving the transmission of patch information.

[0253] <Flow of Decoding Process> Another example of the flow of the decoding process executed by the decoding device 400 will be described with reference to the flowchart of Fig. 27. Note that the flowchart of Fig. 27 shows an example of the flow of the encoding process when method 2-4 of the present technology is applied.

[0254] When the decoding process starts, the demultiplexing unit 411 of the decoding device 400 demultiplexes the V-DMC bitstream in step S461.

[0255] In step S462, the atlas information decoding unit 412 decodes the coded data of the atlas information, and generates (restores) atlas information corresponding to the regular arrangement of the orthoAtlas and texture.

[0256] In step S463, the base mesh decoding unit 413 decodes the coded data of the base mesh and generates (restores) information about the base mesh.

[0257] In step S464, the base mesh reconstruction unit 414 reconstructs the base mesh using the atlas information corresponding to the regular arrangement of the orthoAtlas and textures, and information about the base mesh. At this time, for example, for vertices UV-expanded by the orthoAtlas, UV coordinates are derived based on the CCid and patch information.

[0258] In step S465, the subdivision unit 415 subdivides the reconstructed base mesh to generate division points.

[0259] In step S466, the displaced video decoding unit 416 decodes the encoded data of the displaced video to generate (restore) the displaced video.

[0260] In step S467, the unpacking unit 417 unpacks the displacement map, which is the frame images of the displacement video, and extracts the displacement vectors.

[0261] In step S468, the displacement vector application unit 418 applies the unpacked displacement vectors to the division points of the subdivided base mesh to generate a decoded mesh.

[0262] In step S469, the attribute video decoding unit 419 decodes the coded data of the attribute video to generate (restore) the attribute video.

[0263] In step S470, the display processing unit 420 uses the orthoAtlas and atlas information corresponding to the regular arrangement of textures to apply textures extracted from the attribute map, which is a frame image of the attribute video, to the faces of the generated decoded mesh, and renders the decoded mesh to generate a display image.

[0264] When the process of step S470 is completed, the decoding process ends.

[0265] By performing each process as described above, the decoding device 400 can apply methods 2 to 4. Therefore, the decoding device 400 can suppress a decrease in the coding efficiency of 3D data including base meshes and displacement vectors when applying an unfolding method involving the transmission of patch information.

[0266] 6. Supplementary Notes Polygon Shape In the above description, the polygon shape is a triangle, but this shape is just an example. The polygon shape may be any polygonal shape.

[0267] <Scope of Application of the Present Technology> In the above, V-DMC has been used as an example of an encoding method to which the present technology can be applied, but the present technology is not limited to this example, and can be applied to any encoding method that encodes a base mesh, a displacement vector, an attribute map including texture, and atlas information, or information equivalent thereto.

[0268] Furthermore, in the above, orthoAtlas has been used as an example of the UV unwrapping method (first unwrapping method) to which the present technology is applied, but the present technology is not limited to this example, and the first unwrapping method may be any unwrapping method that involves the transmission of patch information.

[0269] In the above description, UVAtlas has been used as an example of a UV unfolding method (second unfolding method) to be used in conjunction with the first unfolding method, but this second unfolding method may be any unfolding method that is different from the first unfolding method. For example, it may be a unfolding method that involves the transmission of coordinate information indicating the coordinates of the vertices of the base mesh in a two-dimensional area, a unfolding method that involves the transmission of information related to a regularly mapped texture, or any other unfolding method.

[0270] <Computer> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the programs that make up the software are installed on a computer. Here, the term "computer" includes computers built into dedicated hardware, and general-purpose personal computers, etc., that can execute various functions by installing various programs.

[0271] FIG. 28 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0272] In a computer 900 shown in FIG. 28, a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, and a RAM (Random Access Memory) 903 are interconnected via a bus 904.

[0273] An input / output interface 910 is also connected to the bus 904. To the input / output interface 910, an input unit 911, an output unit 912, a storage unit 913, a communication unit 914, and a drive 915 are connected.

[0274] The input unit 911 includes, for example, a keyboard, a mouse, a microphone, a touch panel, and an input terminal. The output unit 912 includes, for example, a display, a speaker, and an output terminal. The storage unit 913 includes, for example, a hard disk, a RAM disk, and a non-volatile memory. The communication unit 914 includes, for example, a network interface. The drive 915 drives removable media 921 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0275] In a computer configured as described above, the CPU 901 performs the above-described series of processes by, for example, loading a program stored in the storage unit 913 into the RAM 903 via the input / output interface 910 and the bus 904 and executing the program. The RAM 903 also stores data necessary for the CPU 901 to execute various processes as appropriate.

[0276] The program executed by the computer can be applied by recording it on, for example, a removable medium 921 such as a package medium. In this case, the program can be installed in the storage unit 913 via the input / output interface 910 by inserting the removable medium 921 into the drive 915.

[0277] This program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, digital satellite broadcasting, etc. In this case, the program can be received by the communication unit 914 and installed in the storage unit 913.

[0278] Alternatively, this program can be installed in advance in the ROM 902 or the storage unit 913 .

[0279] <Application of the Present Technology> The present technology can be applied to any configuration. For example, the present technology can be applied to various electronic devices.

[0280] Furthermore, for example, the present technology can also be implemented as part of an apparatus, such as a processor (e.g., a video processor) as a system LSI (Large Scale Integration), a module using multiple processors (e.g., a video module), a unit using multiple modules (e.g., a video unit), or a set in which other functions are added to a unit (e.g., a video set).

[0281] Furthermore, for example, the present technology can also be applied to a network system configured with multiple devices. For example, the present technology may be implemented as cloud computing in which multiple devices share and collaborate on processing via a network. For example, the present technology may be implemented in a cloud service that provides image (video)-related services to any terminal, such as a computer, an AV (Audio Visual) device, a portable information processing terminal, or an IoT (Internet of Things) device.

[0282] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are housed in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0283] <Fields and uses to which this technology can be applied> Systems, devices, processing units, etc. to which this technology is applied can be used in any field, for example, transportation, medical care, crime prevention, agriculture, livestock farming, mining, beauty, factories, home appliances, weather, nature monitoring, etc. In addition, the uses thereof are also arbitrary.

[0284] <Others> In this specification, a "flag" refers to information for identifying multiple states, and includes not only information used to identify two states, true (1) or false (0), but also information capable of identifying three or more states. Therefore, the value that this "flag" can take may be, for example, two values, 1 / 0, or three or more values. That is, the number of bits constituting this "flag" is arbitrary, and may be one bit or multiple bits. Furthermore, identification information (including flags) can be included not only in a bitstream, but also in a bitstream that includes differential information of the identification information relative to certain reference information. Therefore, in this specification, "flag" and "identification information" encompass not only the information itself, but also differential information relative to the reference information.

[0285] Furthermore, various types of information (e.g., metadata) related to the coded data (bitstream) may be transmitted or recorded in any form as long as they are associated with the coded data. Here, the term "associate" means, for example, that one piece of data can be used (linked) when processing the other piece of data. That is, the associated pieces of data may be combined into one piece of data or may be separate pieces of data. For example, information associated with coded data (image) may be transmitted over a transmission path separate from that of the coded data (image). Furthermore, for example, information associated with coded data (image) may be recorded on a recording medium separate from that of the coded data (image) (or on a different recording area of ​​the same recording medium). Note that this "association" may refer not to the entire data, but to only part of the data. For example, an image and information corresponding to that image may be associated with each other in any unit, such as multiple frames, one frame, or a portion of a frame.

[0286] In this specification, terms such as "composite," "multiplex," "add," "integrate," "include," "store," "embed," "insert," and the like refer to combining multiple items into one, such as combining encoded data and metadata into one piece of data, and refer to one method of "associating" as described above.

[0287] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present technology.

[0288] For example, a configuration described as one device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, configurations described above as multiple devices (or processing units) may be combined and configured as one device (or processing unit). Of course, configurations other than those described above may be added to the configuration of each device (or each processing unit). Furthermore, as long as the configuration and operation of the entire system are substantially the same, part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0289] Furthermore, for example, the above-described program may be executed in any device, as long as the device has the necessary functions (functional blocks, etc.) and is able to obtain the necessary information.

[0290] Also, for example, each step of a single flowchart may be executed by a single device, or may be shared and executed by multiple devices. Furthermore, when a single step includes multiple processes, the multiple processes may be executed by a single device, or may be shared and executed by multiple devices. In other words, multiple processes included in a single step can be executed as multiple step processes. Conversely, processes described as multiple steps can be executed collectively as a single step.

[0291] For example, the steps of a program executed by a computer may be executed in chronological order in the order described herein, or may be executed in parallel or individually at the required timing, such as when a call is made. In other words, as long as no contradiction occurs, the steps may be executed in an order different from the order described above. Furthermore, the steps of this program may be executed in parallel with the processing of another program, or may be executed in combination with the processing of another program.

[0292] Furthermore, for example, multiple technologies related to the present technology can be implemented independently and independently, as long as no contradiction occurs. Of course, any multiple technologies can also be implemented in combination. For example, part or all of the present technology described in any embodiment can be implemented in combination with part or all of the present technology described in another embodiment. Furthermore, part or all of any of the above-described present technologies can be implemented in combination with other technologies not described above.

[0293] The present technology can also be configured as follows: (1) An atlas information generation unit that unfolds a base mesh into a two-dimensional region by an unfolding method involving transmission of patch information, and generates, as atlas information, common patch information that is the patch information common to a plurality of frames, an atlas information encoding unit that encodes the atlas information including the generated common patch information, an attribute map generation unit that uses the generated common patch information to map a projection image of a texture of an original mesh into a two-dimensional region in accordance with the unfolding method, and generates an attribute map, an attribute video encoding unit that encodes an attribute video having the generated attribute map as a frame image, and a bitstream generation unit that generates a bitstream including encoded data of the base mesh, encoded data of a displacement video, encoded data of the attribute video, and encoded data of the atlas information, wherein the base mesh is a mesh with lower resolution than the original mesh to be encoded, which is composed of vertices and connections that represent the three-dimensional structure of an object, and The information processing device according to (1), wherein the displacement video is a moving image having frame images of a displacement map, which is a two-dimensional region packed with displacement vectors indicating displacements of division points, which are vertices generated by subdividing the base mesh, the atlas information is information used for reconstructing a mesh, including information indicating a correspondence between the base mesh, the displacement map, and the attribute map, and the patch information is information regarding mapping of patches of the projection image of the texture. (2) The information processing device according to (1), wherein the common patch information is information regarding a large patch that is common to the plurality of frames and contains a plurality of small patches projected in the same direction.(3) The information processing device according to (2), wherein the common patch information includes at least one of information indicating a projection direction of the large patch, information indicating a rotation angle of the large patch, information indicating a position of the large patch in the two-dimensional region, information indicating a size of the large patch, and information indicating a magnification of the large patch. (4) The information processing device according to (2) or (3), wherein the atlas information generation unit further generates, as the atlas information, specific patch information that is the patch information specific to a frame. (5) The information processing device according to (4), wherein the specific patch information includes information indicating a displacement between frames of the large patch corresponding to the common patch information. (6) The information processing device according to (4) or (5), wherein the specific patch information includes information about the small patches contained in the large patch corresponding to the common patch information. (7) The information processing device according to any of (4) to (6), wherein the specific patch information includes information about the small patches located outside the large patch corresponding to the common patch information. (8) The information processing device according to any one of (1) to (7), wherein the atlas information generation unit expands the base mesh into a two-dimensional region by a first expansion method involving transmission of the patch information, expands local regions of the base mesh into a two-dimensional region by a second expansion method different from the first expansion method, and generates, as the atlas information, the common patch information for the base mesh expanded by the first expansion method and meta information for the local region generated by the second expansion method, and the attribute map generation unit generates the attribute map by mapping the projection image corresponding to the base mesh using the generated common patch information in a manner compatible with the first expansion method and mapping the texture corresponding to the local region using the generated meta information in a manner compatible with the second expansion method. (9) The information processing device according to (8), wherein the atlas information generation unit expands the entire base mesh into a two-dimensional region by the first expansion method, and expands the local region into a two-dimensional region by the second expansion method.(10) The information processing device according to (8), wherein the atlas information generation unit expands the base mesh other than the local region into a two-dimensional region using the first expansion method, and expands the local region into a two-dimensional region using the second expansion method. (11) The information processing device according to any of (8) to (10), wherein the second expansion method involves transmitting coordinate information indicating coordinates of vertices of the base mesh in a two-dimensional region, and the attribute map generation unit uses the generated meta information to map the projected image of the texture corresponding to the local region onto the attribute map in accordance with the second expansion method. (12) The information processing device according to (11), wherein the atlas information generation unit generates, as the meta information, the coordinate information of each vertex present in the local region. (13) The information processing device according to (11) or (12), wherein the atlas information generation unit generates, as the meta information, identification information of each face present in the local region, the coordinate information of each vertex, and information regarding each connection. (14) The information processing device according to any one of (8) to (13), wherein the second expansion method is an expansion method involving transmission of information related to the regularly mapped texture, wherein the atlas information generation unit generates information related to the texture as the meta information, and wherein the attribute map generation unit regularly maps the texture corresponding to a face of the base mesh to the attribute map using the generated information related to the texture. (15) The information processing device according to (14), wherein the information related to the texture includes a list of identification information of the faces corresponding to the regularly mapped texture. (16) The information processing device according to (14), wherein the information related to the texture includes a list of identification information of the patches to which the faces corresponding to the regularly mapped texture belong. (17) The information processing device according to any one of (14) to (16), wherein the information related to the texture includes a list of identification information of the faces corresponding to the regularly mapped texture and parameters related to an area to which the texture is regularly mapped.(18) The information processing device according to (17), wherein the parameters include information indicating the length of a side of the region. (19) The information processing device according to (17) or (18), wherein the parameters include information indicating the length of a side of the region for each resolution. (20) The information processing device according to any of (17) to (19), wherein the parameters include information indicating the number of the regions for each resolution. (21) The information processing device according to any of (17) to (20), wherein the parameters include information indicating the length of an interval between the regularly mapped textures. (22) Expanding a base mesh into a two-dimensional region by an expansion method involving transmission of patch information, and generating common patch information as atlas information, which is the patch information common to a plurality of frames; encoding the atlas information including the generated common patch information; using the generated common patch information, mapping a projection image of a texture of the original mesh into a two-dimensional region in accordance with the expansion method to generate an attribute map; encoding an attribute video having the generated attribute map as a frame image; generating a bitstream including encoded data of the base mesh, encoded data of a displacement video, encoded data of the attribute video, and encoded data of the atlas information; the base mesh is a mesh with lower resolution than the original mesh, which is generated by thinning out vertices from the original mesh to be encoded, which is composed of vertices and connections that represent the three-dimensional structure of an object; and the displacement video is a moving image having frame images of a displacement map, which is a two-dimensional region into which displacement vectors indicating displacements of division points, which are vertices generated by subdividing the base mesh, are packed. An information processing method, wherein the atlas information is information used to reconstruct a mesh, including information indicating the correspondence between the base mesh and the displacement map and the attribute map, and the patch information is information regarding the mapping of patches of the projection image of the texture.

[0294] (31) A video processing system comprising: a demultiplexing unit that demultiplexes a bitstream and extracts coded data of a base mesh, coded data of a displacement video, coded data of an attribute video, and coded data of atlas information; an atlas information decoding unit that decodes the extracted coded data of the atlas information to generate the atlas information that corresponds to an unfolding method involving transmission of patch information and includes common patch information that is the patch information common to a plurality of frames; a base mesh decoding unit that decodes the extracted coded data of the base mesh to generate information about the base mesh; a base mesh reconstruction unit that reconstructs the base mesh from information about the base mesh using the atlas information; an attribute video decoding unit that decodes the coded data of the attribute video to generate an attribute map configured as a frame image of the attribute video; and a display processing unit that uses the common patch information included in the atlas information to extract a projected image of a texture of an original mesh mapped to the attribute map so as to correspond to the unfolding method, and applies the projected image to a decoded mesh for rendering to generate an image for display, An information processing device wherein the base mesh is a mesh with lower resolution than the original mesh to be encoded, which is composed of vertices and connections that represent the three-dimensional structure of an object, and is generated by thinning out vertices from the original mesh; the displacement video is a moving image whose frame images are displacement maps, which are two-dimensional areas packed with displacement vectors that indicate the displacement of division points, which are vertices generated by subdividing the base mesh; the atlas information is information used to reconstruct a mesh, including information indicating the correspondence between the base mesh, the displacement map, and the attribute map; the patch information is information regarding the mapping of patches of the projection image of the texture; and the decoded mesh is generated by applying the displacement vectors derived from the encoded data of the displacement video to the reconstructed base mesh.(32) The information processing device according to (31), wherein the common patch information is information about a large patch that is common to the plurality of frames and that includes a plurality of small patches projected in the same direction. (33) The information processing device according to (32), wherein the common patch information includes at least one of information indicating the projection direction of the large patch, information indicating the rotation angle of the large patch, information indicating the position of the large patch in the two-dimensional region, information indicating the size of the large patch, and information indicating the magnification of the large patch. (34) The information processing device according to (32) or (33), wherein the atlas information further includes specific patch information that is the patch information specific to a frame, and the display processing unit generates the display image by extracting the projected image using the common patch information and the specific patch information included in the atlas information and applying the projected image to the decoded mesh for rendering. (35) The information processing device according to (34), wherein the specific patch information includes information indicating displacement between frames of the large patch that corresponds to the common patch information. (36) The information processing device according to (34) or (35), wherein the unique patch information includes information about the small patches contained in the large patch corresponding to the common patch information. (37) The information processing device according to any one of (34) to (36), wherein the unique patch information includes information about the small patches located outside the large patch corresponding to the common patch information.(38) The information processing device described in any of (31) to (37), wherein the atlas information decoding unit generates the atlas information corresponding to a first expansion method involving transmission of the patch information and a second expansion method different from the first expansion method, and including the common patch information for the base mesh expanded by the first expansion method and meta-information for a local region of the base mesh expanded by the second expansion method, by decoding the encoded data of the extracted atlas information; and the display processing unit uses the common patch information included in the atlas information to extract the texture corresponding to the base mesh mapped to the attribute map so as to correspond to the first expansion method, uses the meta-information included in the atlas information to extract the texture corresponding to the local region mapped to the attribute map so as to correspond to the second expansion method, and applies the extracted projection image and the texture to the decoded mesh and renders it to generate the display image. (39) The information processing device described in (38), wherein the atlas information decoding unit generates the atlas information corresponding to the first expansion method and the second expansion method, and including the common patch information for the entire base mesh expanded by the first expansion method and the meta-information for the local region expanded by the second expansion method, by decoding the encoded data of the extracted atlas information; and the display processing unit uses the common patch information included in the atlas information to extract the texture corresponding to the entire base mesh mapped to the attribute map so as to correspond to the first expansion method, uses the meta-information included in the atlas information to extract the texture corresponding to the local region mapped to the attribute map so as to correspond to the second expansion method, and applies the extracted projection image and the texture to the decoded mesh and renders it, thereby generating the display image.(40) The information processing device described in (38), wherein the atlas information decoding unit generates the atlas information corresponding to the first expansion method and the second expansion method, and including the common patch information for the base mesh other than the local region expanded by the first expansion method and the meta-information for the local region expanded by the second expansion method, by decoding the encoded data of the extracted atlas information; and the display processing unit uses the common patch information included in the atlas information to extract the texture corresponding to the base mesh other than the local region mapped to the attribute map so as to correspond to the first expansion method, by using the meta-information included in the atlas information, to extract the texture corresponding to the local region mapped to the attribute map so as to correspond to the second expansion method, and generates the display image by applying the extracted projection image and the texture to the decoded mesh and rendering it. (41) The information processing device according to any of (38) to (40), wherein the second unfolding method is an unfolding method involving transmission of coordinate information indicating coordinates of vertices of the base mesh in a two-dimensional region, and the atlas information decoding unit generates the atlas information including the coordinate information as the meta-information by decoding encoded data of the extracted atlas information. (42) The information processing device according to (41), wherein the atlas information decoding unit generates the atlas information including the coordinate information of each vertex present in the local region as the meta-information by decoding encoded data of the extracted atlas information. (43) The information processing device according to (41) or (42), wherein the atlas information decoding unit generates the atlas information including identification information of each face present in the local region, the coordinate information of each vertex, and information on each connection as the meta-information by decoding encoded data of the extracted atlas information.(44) The information processing device according to any one of (38) to (43), wherein the second expansion method is a expansion method involving transmission of information related to the regularly mapped texture, the atlas information decoding unit generates the atlas information including information related to the texture as the meta information by decoding encoded data of the extracted atlas information, and the display processing unit extracts the texture corresponding to faces of the base mesh regularly mapped to the attribute map so as to correspond to the second expansion method, using the meta information included in the atlas information. (45) The information processing device according to (44), wherein the information related to the texture includes a list of identification information of the faces corresponding to the regularly mapped texture. (46) The information processing device according to (44), wherein the information related to the texture includes a list of identification information of the patches to which the faces corresponding to the regularly mapped texture belong. (47) The information processing device according to any one of (44) to (46), wherein the information related to the texture includes a list of identification information of the faces corresponding to the regularly mapped texture and parameters related to an area to which the texture is regularly mapped. (48) The information processing device according to (47), wherein the parameters include information indicating the length of a side of the region. (49) The information processing device according to (47) or (48), wherein the parameters include information indicating the length of a side of the region for each resolution. (50) The information processing device according to any of (47) to (49), wherein the parameters include information indicating the number of the regions for each resolution. (51) The information processing device according to any of (47) to (50), wherein the parameters include information indicating the length of an interval between the regularly mapped textures.(52) Demultiplexing a bitstream to extract coded data of a base mesh, coded data of a displacement video, coded data of an attribute video, and coded data of atlas information; decoding the extracted coded data of the atlas information to generate the atlas information corresponding to an unfolding method involving transmission of patch information and including common patch information which is the patch information common to a plurality of frames; decoding the extracted coded data of the base mesh to generate information about the base mesh; reconstructing the base mesh from information about the base mesh using the atlas information; decoding the coded data of the attribute video to generate an attribute map configured as a frame image of the attribute video; extracting a projected image of the texture of the original mesh mapped to the attribute map corresponding to the unfolding method using the common patch information included in the atlas information, and applying the projected image to the decoded mesh for rendering to generate an image for display; the base mesh is a mesh with lower resolution than the original mesh to be encoded, which is configured by vertices and connections that represent the three-dimensional structure of an object, and is generated by thinning out vertices from the original mesh, The displacement video is a moving image whose frame images are displacement maps, which are two-dimensional regions packed with displacement vectors indicating the displacement of division points, which are vertices generated by subdividing the base mesh; the atlas information is information used to reconstruct a mesh, including information indicating the correspondence between the base mesh, the displacement map, and the attribute map; the patch information is information regarding the mapping of patches of the projection image of the texture; and the decoded mesh is generated by applying the displacement vectors derived from the encoded data of the displacement video to the reconstructed base mesh.

[0295] (61) An atlas information generation unit that expands a base mesh into a two-dimensional region by a first expansion method involving transmission of patch information, expands a local region of the base mesh into a two-dimensional region by a second expansion method different from the first expansion method, and generates, as atlas information, the patch information for the base mesh expanded by the first expansion method and meta information for the local region generated by the second expansion method; an atlas information encoding unit that encodes the atlas information including the generated patch information and the meta information; an attribute map generation unit that generates an attribute map by using the generated patch information to map a projection image of a texture corresponding to the base mesh into a two-dimensional region in accordance with the first expansion method, and using the generated meta information to map the texture corresponding to the local region into a two-dimensional region in accordance with the second expansion method; and an attribute video encoding unit that encodes an attribute video using the generated attribute map as a frame image. an information processing apparatus comprising: a bitstream generation unit that generates a bitstream including encoded data of the base mesh, encoded data of a displacement video, encoded data of the attribute video, and encoded data of the atlas information; wherein the base mesh is a mesh with lower resolution than an original mesh to be encoded, which is composed of vertices and connections that represent a three-dimensional structure of an object, and is generated by thinning out vertices from the original mesh; the displacement video is a moving image whose frame images are displacement maps that are two-dimensional regions into which displacement vectors that indicate the displacement of division points, which are vertices generated by subdividing the base mesh, are packed; the atlas information is information used for mesh reconstruction, including information indicating the correspondence between the base mesh, the displacement map, and the attribute map; and the patch information is information regarding mapping of patches of the projection image of the texture.(62) The information processing device according to (61), wherein the atlas information generation unit unfolds the entire base mesh into a two-dimensional region using the first unfolding method, and unfolds the local region into a two-dimensional region using the second unfolding method. (63) The information processing device according to (61), wherein the atlas information generation unit unfolds the base mesh other than the local region into a two-dimensional region using the first unfolding method, and unfolds the local region into a two-dimensional region using the second unfolding method. (64) The information processing device according to any of (61) to (63), wherein the second unfolding method is a unfolding method involving transmission of coordinate information indicating coordinates of vertices of the base mesh in a two-dimensional region, and the attribute map generation unit uses the generated meta information to map the projection image of the texture corresponding to the local region onto the attribute map so as to correspond to the second unfolding method. (65) The information processing device according to (64), wherein the atlas information generation unit generates, as the meta information, the coordinate information of each vertex present in the local region. (66) The information processing device according to (64) or (65), wherein the atlas information generation unit generates, as the meta-information, identification information of each face present in the local region, the coordinate information of each vertex, and information on each connection. (67) The information processing device according to any of (61) to (66), wherein the second unfolding method is an unfolding method involving transmission of information on the regularly mapped texture, the atlas information generation unit generates information on the texture as the meta-information, and the attribute map generation unit regularly maps the texture corresponding to a face of the base mesh to the attribute map using the generated information on the texture. (68) The information processing device according to (67), wherein the information on the texture includes a list of identification information of the faces corresponding to the regularly mapped texture. (69) The information processing device according to (67), wherein the information on the texture includes a list of identification information of the patches to which the faces corresponding to the regularly mapped texture belong.(70) The information processing device according to any one of (67) to (69), wherein the information related to the texture includes a list of identification information of the faces corresponding to the regularly mapped texture and parameters related to the area to which the texture is regularly mapped. (71) The information processing device according to (70), wherein the parameters include information indicating the length of a side of the area. (72) The information processing device according to (70) or (71), wherein the parameters include information indicating the length of a side of the area for each resolution. (73) The information processing device according to any one of (70) to (72), wherein the parameters include information indicating the number of the areas for each resolution. (74) The information processing device according to any one of (70) to (73), wherein the parameters include information indicating the length of an interval between the regularly mapped textures. (75) The information processing device according to any one of (61) to (74), wherein the atlas information generation unit generates, as the atlas information, common patch information that is the patch information common to a plurality of frames for the base mesh unfolded by the first unfolding method, and the attribute map generation unit uses the generated common patch information to map the projection image corresponding to the base mesh so as to be compatible with the first unfolding method. (76) The information processing device according to (75), wherein the common patch information is information about a large patch that is common to the plurality of frames and contains a plurality of small patches projected in the same direction. (77) The information processing device according to (76), wherein the common patch information includes at least one of information indicating the projection direction of the large patch, information indicating the rotation angle of the large patch, information indicating the position of the large patch in the two-dimensional region, information indicating the size of the large patch, and information indicating the magnification of the large patch. (78) The information processing device according to (76) or (77), wherein the atlas information generating unit further generates, as the atlas information, specific patch information that is the patch information specific to a frame. (79) The information processing device according to (78), wherein the specific patch information includes information indicating displacement between frames of the large patch corresponding to the common patch information.(80) The information processing device according to (78) or (79), wherein the unique patch information includes information about the small patches contained in the large patch corresponding to the common patch information. (81) The information processing device according to any one of (78) to (80), wherein the unique patch information includes information about the small patches located outside the large patch corresponding to the common patch information.(82) Expanding a base mesh into a two-dimensional region by a first expansion method involving transmission of patch information, expanding a local region of the base mesh into a two-dimensional region by a second expansion method different from the first expansion method, and generating, as atlas information, the patch information for the base mesh expanded by the first expansion method and meta information for the local region generated by the second expansion method; encoding the atlas information including the generated patch information and meta information; using the generated patch information, mapping a projection image of a texture corresponding to the base mesh onto a two-dimensional region in accordance with the first expansion method, and using the generated meta information, mapping the texture corresponding to the local region onto a two-dimensional region in accordance with the second expansion method, thereby generating an attribute map; encoding an attribute video having the generated attribute map as a frame image; and generating a bitstream including encoded data of the base mesh, encoded data of a displacement video, encoded data of the attribute video, and encoded data of the atlas information. An information processing method in which the base mesh is a mesh with lower resolution than an original mesh to be encoded, which is composed of vertices and connections that represent the three-dimensional structure of an object, and is generated by thinning out vertices from the original mesh; the displacement video is a moving image whose frame images are displacement maps, which are two-dimensional areas packed with displacement vectors that indicate the displacement of division points, which are vertices generated by subdividing the base mesh; the atlas information is information used to reconstruct a mesh, including information indicating the correspondence between the base mesh, the displacement map, and the attribute map; and the patch information is information regarding the mapping of patches of the projection image of the texture.

[0296] (91) A demultiplexing unit that demultiplexes a bitstream and extracts coded data of a base mesh, coded data of a displacement video, coded data of an attribute video, and coded data of atlas information; an atlas information decoding unit that decodes the extracted coded data of the atlas information to generate the atlas information corresponding to a first unfolding method involving transmission of patch information and a second unfolding method different from the first unfolding method, the atlas information including the patch information for the base mesh unfolded by the first unfolding method and meta information for a local region of the base mesh unfolded by the second unfolding method; a base mesh decoding unit that decodes the extracted coded data of the base mesh to generate information about the base mesh; a base mesh reconstruction unit that reconstructs the base mesh from information about the base mesh using the atlas information; and an attribute video decoding unit that decodes the coded data of the attribute video to generate an attribute map configured as a frame image of the attribute video. a display processing unit that uses the patch information included in the atlas information to extract a projection image of a texture corresponding to the base mesh mapped to the attribute map so as to correspond to the first unfolding method, uses the meta information included in the atlas information to extract the texture corresponding to the local region mapped to the attribute map so as to correspond to the second unfolding method, and applies the extracted projection image and the texture to a decoded mesh for rendering to generate an image for display, wherein the base mesh is a mesh with lower resolution than an original mesh to be coded, the original mesh being composed of vertices and connections that represent a three-dimensional structure of an object, and is generated by thinning out vertices from the original mesh, the original mesh being composed of vertices and connections that represent a three-dimensional structure of the object, and the displacement video is a moving image in which frame images are displacement maps that are two-dimensional regions into which displacement vectors indicating displacements of division points that are vertices generated by subdividing the base mesh are packed,An information processing device, wherein the atlas information is information used to reconstruct a mesh, including information indicating the correspondence between the base mesh and the displacement map and the attribute map; the patch information is information regarding mapping of patches of the projection image of the texture; and the decoded mesh is generated by applying the displacement vector derived from the encoded data of the displacement video to the reconstructed base mesh. (92) The information processing device described in (91), wherein the atlas information decoding unit generates the atlas information corresponding to the first expansion method and the second expansion method, and including the patch information for the entire base mesh expanded by the first expansion method and the meta-information for the local region expanded by the second expansion method, by decoding the encoded data of the extracted atlas information; and the display processing unit uses the patch information included in the atlas information to extract the texture corresponding to the entire base mesh mapped to the attribute map so as to correspond to the first expansion method, uses the meta-information included in the atlas information to extract the texture corresponding to the local region mapped to the attribute map so as to correspond to the second expansion method, and applies the extracted projection image and the texture to the decoded mesh and renders it, thereby generating the display image. (93) The atlas information decoding unit generates the atlas information corresponding to the first expansion method and the second expansion method by decoding the encoded data of the extracted atlas information, and including the patch information for the region other than the local region of the base mesh expanded by the first expansion method and the meta-information for the region expanded by the second expansion method, and the display processing unit uses the patch information included in the atlas information to extract the texture corresponding to the region other than the local region of the base mesh mapped to the attribute map so as to correspond to the first expansion method,The information processing device according to (91), which uses the meta information included in the atlas information to extract the texture corresponding to the local region mapped to the attribute map so as to correspond to the second unfolding method, and generates the display image by applying the extracted projected image and the texture to the decoded mesh and rendering it. (94) The information processing device according to any of (91) to (93), wherein the second unfolding method is an unfolding method involving transmission of coordinate information indicating coordinates of vertices of the base mesh in a two-dimensional region, and the atlas information decoding unit generates the atlas information including the coordinate information as the meta information by decoding encoded data of the extracted atlas information. (95) The information processing device according to (94), wherein the atlas information decoding unit generates the atlas information including the coordinate information of each vertex existing in the local region as the meta information by decoding encoded data of the extracted atlas information. (96) The information processing device according to (94), wherein the atlas information decoding unit generates the atlas information including, as the meta-information, identification information of each face, the coordinate information of each vertex, and information on each connection present in the local region, by decoding encoded data of the extracted atlas information. (97) The information processing device according to any of (91) to (96), wherein the second unfolding method is an unfolding method involving transmission of information on the regularly mapped texture, the atlas information decoding unit generates the atlas information including, as the meta-information, information on the texture, by decoding encoded data of the extracted atlas information, and the display processing unit uses the meta-information included in the atlas information to extract the texture corresponding to faces of the base mesh regularly mapped to the attribute map so as to correspond to the second unfolding method. (98) The information processing device according to (97), wherein the information on texture includes a list of identification information of the faces corresponding to the regularly mapped texture.(99) The information processing device according to (97), wherein the information about the texture includes a list of identification information of the patches to which the faces corresponding to the regularly mapped texture belong. (100) The information processing device according to any of (97) to (99), wherein the information about the texture includes a list of identification information of the faces corresponding to the regularly mapped texture and parameters related to areas to which the texture is regularly mapped. (101) The information processing device according to (100), wherein the parameters include information indicating the lengths of the sides of the areas. (102) The information processing device according to (100) or (101), wherein the parameters include information indicating the lengths of the sides of the areas for each resolution. (103) The information processing device according to any of (100) to (102), wherein the parameters include information indicating the number of the areas for each resolution. (104) The information processing device according to any of (100) to (103), wherein the parameters include information indicating the length of the intervals between the regularly mapped textures. (105) The information processing device according to any one of (91) to (104), wherein the atlas information decoding unit generates the atlas information including common patch information, which is the patch information common to a plurality of frames, for the base mesh unfolded by the first unfolding method, by decoding encoded data of the extracted atlas information, and the display processing unit uses the common patch information included in the atlas information to extract the texture corresponding to the base mesh mapped to the attribute map so as to correspond to the first unfolding method. (106) The information processing device according to (105), wherein the common patch information is information about a large patch that is common to the plurality of frames and includes a plurality of small patches projected in the same direction. (107) The information processing device according to (106), wherein the common patch information includes at least one of information indicating a projection direction of the large patch, information indicating a rotation angle of the large patch, information indicating a position of the large patch in the two-dimensional region, information indicating a size of the large patch, and information indicating a magnification of the large patch.(108) The information processing device according to (106) or (107), wherein the atlas information further includes specific patch information which is the patch information specific to a frame, and the display processing unit generates the display image by extracting the projection image using the common patch information and the specific patch information included in the atlas information, applying the projection image to the decoded mesh, and rendering it. (109) The information processing device according to (108), wherein the specific patch information includes information indicating displacement between frames of the large patch corresponding to the common patch information. (110) The information processing device according to (108) or (109), wherein the specific patch information includes information about the small patches contained in the large patch corresponding to the common patch information. (111) The information processing device according to any of (108) to (110), wherein the specific patch information includes information about the small patches located outside the large patch corresponding to the common patch information. (112) Demultiplexing the bitstream to extract coded data of a base mesh, coded data of a displacement video, coded data of an attribute video, and coded data of atlas information; decoding the extracted coded data of the atlas information to generate the atlas information corresponding to a first unfolding method involving transmission of patch information and a second unfolding method different from the first unfolding method, the atlas information including the patch information about the base mesh unfolded by the first unfolding method and meta-information about a local region of the base mesh unfolded by the second unfolding method; decoding the extracted coded data of the base mesh to generate information about the base mesh; reconstructing the base mesh from information about the base mesh using the atlas information; decoding the coded data of the attribute video to generate an attribute map configured as a frame image of the attribute video;using the patch information included in the atlas information to extract a projection image of a texture corresponding to the base mesh mapped to the attribute map so as to correspond to the first unfolding method, using the meta information included in the atlas information to extract the texture corresponding to the local region mapped to the attribute map so as to correspond to the second unfolding method, and applying the extracted projection image and the texture to a decoded mesh for rendering to generate an image for display; the base mesh is a mesh with lower resolution than the original mesh to be coded, which is composed of vertices and connections that represent a three-dimensional structure of an object, and is generated by thinning out vertices from the original mesh; the displacement video is a moving image in which frame images are displacement maps, which are two-dimensional regions in which displacement vectors indicating displacements of division points, which are vertices generated by subdividing the base mesh, are packed; the atlas information is information used for reconstructing a mesh, including information indicating a correspondence between the base mesh, the displacement map, and the attribute map; and the patch information is information regarding mapping of patches of the projection image of the texture. The decoded mesh is generated by applying the displacement vector derived from the encoded data of the displacement video to the reconstructed base mesh.

[0297] 300 Encoding device, 311 Preprocessing unit, 312 Encoding unit, 321 Mesh decimation unit, 322 Atlas information generation unit, 323 Displacement vector generation unit, 351 Atlas information encoding unit, 352 Base mesh encoding unit, 353 Displacement vector correction unit, 354 Displacement video encoding unit, 355 Mesh reconstruction unit, 356 Attribute map conversion unit, 357 Attribute video encoding unit, 358 Multiplexing unit, 400 Decoding device, 411 Demultiplexing unit, 412 Atlas information decoding unit, 413 Base mesh decoding unit, 414 Base mesh reconstruction unit, 415 Subdivision unit, 416 Displacement video decoding unit, 417 Unpacking unit, 418 Displacement vector application unit, 419 Attribute video decoding unit, 420 Display processing unit 900 Computer

Claims

1. An atlas information generating unit that expands a base mesh into a two-dimensional region by a first expansion method involving the transmission of patch information, expands a local region of the base mesh into a two-dimensional region by a second expansion method different from the first expansion method, and generates, as atlas information, the patch information for the base mesh expanded by the first expansion method and meta information for the local region generated by the second expansion method; an atlas information encoding unit that encodes the atlas information including the generated patch information and the meta information; an attribute map generating unit that generates an attribute map by using the generated patch information to map a projection image of a texture corresponding to the base mesh into a two-dimensional region in a manner corresponding to the first expansion method, and using the generated meta information to map the texture corresponding to the local region into a two-dimensional region in a manner corresponding to the second expansion method; and an attribute video encoding unit that encodes an attribute video having the generated attribute map as a frame image. an information processing apparatus comprising: a bitstream generation unit that generates a bitstream including encoded data of the base mesh, encoded data of a displacement video, encoded data of the attribute video, and encoded data of the atlas information; wherein the base mesh is a mesh with lower resolution than an original mesh to be encoded, which is composed of vertices and connections that represent a three-dimensional structure of an object, and is generated by thinning out vertices from the original mesh; the displacement video is a moving image whose frame images are a displacement map, which is a two-dimensional region into which displacement vectors indicating the displacement of division points, which are vertices generated by subdividing the base mesh, are packed; the atlas information is information used for reconstructing a mesh, which includes information indicating the correspondence between the base mesh, the displacement map, and the attribute map; and the patch information is information regarding mapping of patches of the projection image of the texture.

2. The information processing device according to claim 1, wherein the atlas information generation unit expands the entire base mesh into a two-dimensional region using the first expansion method, and expands the local region into a two-dimensional region using the second expansion method.

3. The information processing device according to claim 1, wherein the atlas information generation unit expands the base mesh other than the local region into a two-dimensional region using the first expansion method, and expands the local region into a two-dimensional region using the second expansion method.

4. The information processing device of claim 1, wherein the second expansion method involves the transmission of coordinate information indicating coordinates of the vertices of the base mesh in a two-dimensional region, and the attribute map generation unit uses the generated meta-information to map the projected image of the texture corresponding to the local region onto the attribute map in a manner corresponding to the second expansion method.

5. The information processing device according to claim 4, wherein the atlas information generating unit generates, as the meta information, the coordinate information of each vertex present in the local region.

6. The information processing device according to claim 4, wherein the atlas information generating unit generates, as the meta-information, identification information of each face present in the local region, the coordinate information of each vertex, and information regarding each connection.

7. The information processing device according to claim 1, wherein the second expansion method is an expansion method involving the transmission of information relating to the regularly mapped texture, the atlas information generation unit generates information relating to the texture as the meta information, and the attribute map generation unit uses the generated information relating to the texture to regularly map the texture corresponding to a face of the base mesh onto the attribute map.

8. The information processing device according to claim 7, wherein the information relating to the texture includes a list of identification information of the faces corresponding to the regularly mapped texture.

9. The information processing device according to claim 7, wherein the information relating to the texture includes a list of identification information of the patch to which the face corresponding to the regularly mapped texture belongs.

10. An information processing device according to claim 7, wherein the information relating to the texture includes a list of identification information of the face corresponding to the regularly mapped texture, and parameters relating to the area onto which the texture is regularly mapped.

11. The information processing device according to claim 10, wherein the parameters include information indicating the length of a side of the region.

12. The information processing device according to claim 10, wherein the parameters include information indicating the length of a side of the region for each resolution.

13. The information processing device according to claim 10, wherein the parameters include information indicating the number of the regions for each resolution.

14. The information processing device according to claim 10, wherein the parameters include information indicating the length of an interval of the regularly mapped texture.

15. The information processing device of claim 1, wherein the atlas information generation unit generates common patch information as the atlas information for the base mesh expanded by the first expansion method, which is patch information common to a plurality of frames, and the attribute map generation unit uses the generated common patch information to map the projection image corresponding to the base mesh so as to correspond to the first expansion method.

16. The information processing device according to claim 15, wherein the common patch information is information about a large patch that is common to the multiple frames and contains multiple small patches that are projected in the same direction.

17. The information processing device according to claim 16, wherein the atlas information generating section further generates, as the atlas information, unique patch information that is the patch information unique to a frame.

18. Expanding a base mesh into a two-dimensional region by a first expansion method involving transmission of patch information, expanding a local region of the base mesh into a two-dimensional region by a second expansion method different from the first expansion method, and generating, as atlas information, the patch information for the base mesh expanded by the first expansion method and meta information for the local region generated by the second expansion method; encoding the atlas information including the generated patch information and meta information; using the generated patch information, mapping a projection image of a texture corresponding to the base mesh into a two-dimensional region in a manner corresponding to the first expansion method, and using the generated meta information, mapping the texture corresponding to the local region into a two-dimensional region in a manner corresponding to the second expansion method, thereby generating an attribute map; encoding an attribute video having the generated attribute map as a frame image; and generating a bit stream including encoded data of the base mesh, encoded data of a displacement video, encoded data of the attribute video, and encoded data of the atlas information; An information processing method in which the base mesh is a mesh with lower resolution than an original mesh to be encoded, which is composed of vertices and connections that represent the three-dimensional structure of an object, and is generated by thinning out vertices from the original mesh; the displacement video is a moving image whose frame images are a displacement map, which is a two-dimensional region packed with displacement vectors that indicate the displacement of division points, which are vertices generated by subdividing the base mesh; the atlas information is information used for mesh reconstruction, including information indicating the correspondence between the base mesh, the displacement map, and the attribute map; and the patch information is information regarding the mapping of patches of the projection image of the texture.

19. A demultiplexing unit that demultiplexes a bit stream and extracts coded data of a base mesh, coded data of a displacement video, coded data of an attribute video, and coded data of atlas information; an atlas information decoding unit that generates the atlas information by decoding the extracted coded data of the atlas information, the atlas information corresponding to a first unfolding method involving transmission of patch information and a second unfolding method different from the first unfolding method, and including the patch information on the base mesh unfolded by the first unfolding method and meta information on a local region of the base mesh unfolded by the second unfolding method; a base mesh decoding unit that generates information on the base mesh by decoding the extracted coded data of the base mesh; a base mesh reconstruction unit that reconstructs the base mesh from information on the base mesh using the atlas information; an attribute video decoding unit that generates an attribute map configured as a frame image of the attribute video by decoding the coded data of the attribute video; a display processing unit that uses the patch information included in the atlas information to extract a projection image of a texture corresponding to the base mesh mapped to the attribute map so as to correspond to the first expansion method, uses the meta information included in the atlas information to extract the texture corresponding to the local region mapped to the attribute map so as to correspond to the second expansion method, and applies the extracted projection image and the texture to a decoded mesh for rendering to generate an image for display, wherein the base mesh is a mesh with lower resolution than the original mesh that is generated by thinning out vertices from an original mesh to be encoded that is composed of vertices and connections that express a three-dimensional structure of an object, and the displacement video is a moving image in which a displacement map that is a two-dimensional region in which displacement vectors indicating the displacement of division points that are vertices generated by subdividing the base mesh are packed as frame images,An information processing device, wherein the atlas information is information used for reconstructing a mesh, including information indicating a correspondence between the base mesh and the displacement map and the attribute map; the patch information is information regarding mapping of a patch of the projection image of the texture; and the decoded mesh is generated by applying the displacement vector derived from the encoded data of the displacement video to the reconstructed base mesh.

20. Demultiplexing the bit stream to extract coded data of the base mesh, coded data of the displacement video, coded data of the attribute video, and coded data of the atlas information; decoding the extracted coded data of the atlas information to generate the atlas information corresponding to a first unfolding method involving transmission of patch information and a second unfolding method different from the first unfolding method, the atlas information including the patch information about the base mesh unfolded by the first unfolding method and meta-information about a local region of the base mesh unfolded by the second unfolding method; decoding the extracted coded data of the base mesh to generate information about the base mesh; reconstructing the base mesh from information about the base mesh using the atlas information; decoding the coded data of the attribute video to generate an attribute map configured as a frame image of the attribute video; using the patch information included in the atlas information to extract a projection image of a texture corresponding to the base mesh mapped to the attribute map so as to correspond to the first expansion method, using the meta information included in the atlas information to extract the texture corresponding to the local region mapped to the attribute map so as to correspond to the second expansion method, and applying the extracted projection image and the texture to a decoded mesh and rendering to generate an image for display; the base mesh is a mesh with lower resolution than the original mesh generated by thinning out vertices from an original mesh to be encoded, the original mesh being composed of vertices and connections that express a three-dimensional structure of an object; the displacement video is a moving image in which frame images are displacement maps, which are two-dimensional regions packed with displacement vectors indicating the displacement of division points, which are vertices generated by subdividing the base mesh; the atlas information is information used for reconstructing a mesh, including information indicating a correspondence between the base mesh, the displacement map, and the attribute map;An information processing method, wherein the patch information is information regarding mapping of a patch of the projected image of the texture, and the decoded mesh is generated by applying the displacement vector derived from the encoded data of the displacement video to the reconstructed base mesh.

Images