Information processing device and method

Abstract

The present disclosure relates to an information processing device and method that make it possible to suppress a reduction in the quality of an attribute while suppressing a reduction in the encoding efficiency of a base mesh. The method executes processing which includes: encoding, as a base mesh, a single piece of geometry information corresponding to at least one face of the base mesh and a plurality of UV sets corresponding to the face and associated with the single piece of geometry information; and encoding, as attributes corresponding to the base mesh, a plurality of attribute maps corresponding to each of the plurality of UV sets. The present disclosure is applicable, for example, to information processing devices, electronic instruments, information processing methods and programs.

Description

Information Processing Apparatus and Method

[0001] The present disclosure relates to an information processing apparatus and method, and more particularly to an information processing apparatus and method capable of suppressing a reduction in the quality of attributes while suppressing a reduction in the coding efficiency of a base mesh.

[0002] Conventionally, as a method for coding a mesh, which is 3D data representing the three-dimensional structure of an object by connection of vertices, there has been V-DMC (Video-based Dynamic Mesh Coding) (see, for example, Non-Patent Document 1). In V-DMC, a mesh to be coded (original mesh) is represented by a coarse (i.e., low in fineness) base mesh and displacement vectors of division points obtained by subdividing the base mesh, and the base mesh, the displacement vectors, and attributes are coded.

[0003] In a mesh, there has been a method of improving the quality of a texture in a rendered image by assigning a plurality of textures to the same portion and selecting a more suitable texture according to the viewpoint position during rendering (see, for example, Non-Patent Document 2).

[0004] When applying such a technique in V-DMC, a method can be considered in which a plurality of attributes are assigned to the same portion of the base mesh and a more suitable attribute can be selected according to the viewpoint position during rendering. In the case of V-DMC, an attribute is arranged and coded on a UV map obtained by UV-unfolding the base mesh. Therefore, when linking a plurality of attributes (textures) to the base mesh as described above, it is considered desirable to prepare a UV map by UV-unfolding the base mesh for each attribute in order to suppress a reduction in the quality of the attribute.

[0005] Khaled Mammou, Jungsun Kim, Alexis Tourapis, Dimitri Podborski, Krasimir Kolarov, "[V-CG] Apple's Dynamic Mesh Coding CfP Response", ISO / IEC JTC 1 / SC 29 / WG 7 m59281, April 2022https: / / reflectorange.net / archives / 231.html

[0006] However, in that case, there was a risk that the geometric information (geometric vertex coordinates and geometric connectivity) contained in the base mesh would be duplicated and encoded. Therefore, there was a risk of reducing the encoding efficiency of the base mesh.

[0007] This disclosure is made in view of these circumstances and aims to suppress a reduction in attribute quality while suppressing a reduction in the coding efficiency of the base mesh.

[0008] An information processing device, representing one aspect of this technology, comprises a base mesh encoding unit that encodes single geometric information corresponding to at least one face of a base mesh, and a plurality of UV sets corresponding to the face and linked to the single geometric information, as the base mesh, and an attribute encoding unit that encodes a plurality of attribute maps corresponding to the plurality of UV sets as attributes corresponding to the base mesh, wherein the base mesh is a mesh obtained by thinning out the vertices from the original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of an object.

[0009] One aspect of this technology is an information processing method which includes encoding a single geometric information corresponding to at least one face of a base mesh and a plurality of UV sets corresponding to the face and linked to the single geometric information as the base mesh, and encoding a plurality of attribute maps corresponding to each of the plurality of UV sets as attributes corresponding to the base mesh, wherein the base mesh is a mesh obtained by thinning out the vertices from the original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of an object.

[0010] The information processing device for another aspect of this technology comprises a base mesh decoding unit that decodes a single geometry information bitstream and a plurality of UV set bitstreams associated with the single geometry information bitstream as a base mesh, and generates a single geometry information corresponding to at least one face of the base mesh and a plurality of UV sets corresponding to the face; and an attribute decoding unit that decodes a plurality of attribute bitstreams corresponding to the plurality of UV set bitstreams as attributes corresponding to the base mesh, and generates a plurality of attribute maps corresponding to the plurality of UV sets, wherein the base mesh is an information processing device in which vertices have been thinned out from the original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of an object.

[0011] Another aspect of this technology involves an information processing method which includes decoding a single geometry information bitstream and a plurality of UV set bitstreams associated with the single geometry information bitstream as a base mesh, generating a single geometry information corresponding to at least one face of the base mesh and a plurality of UV sets corresponding to the face, and decoding a plurality of attribute bitstreams corresponding to the plurality of UV set bitstreams as attributes corresponding to the base mesh, and generating a plurality of attribute maps corresponding to the plurality of UV sets, wherein the base mesh is a mesh obtained by thinning out the vertices from the original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of an object.

[0012] In one aspect of this technology, the information processing device and method include the following steps: encoding a single geometric information corresponding to at least one face of a base mesh, and a plurality of UV sets corresponding to that face and linked to the single geometric information, as the base mesh; and encoding a plurality of attribute maps corresponding to each of the plurality of UV sets as attributes corresponding to the base mesh.

[0013] In other aspects of the information processing device and method of this technology, the following processes are performed: decoding a single geometry information bitstream and a plurality of UV set bitstreams associated with the single geometry information bitstream as a base mesh; generating a single geometry information corresponding to at least one face of the base mesh and a plurality of UV sets corresponding to that face; and decoding a plurality of attribute bitstreams corresponding to the plurality of UV set bitstreams as attributes corresponding to the base mesh, and generating a plurality of attribute maps corresponding to the plurality of UV sets.

[0014] This is a diagram to explain mesh. This is a diagram to explain V-DMC. This is a diagram to explain the case of a single attribute. This is a diagram to explain the case of multiple attributes. This is a diagram to explain UV maps in the case of multiple attributes. This is a diagram showing an example of a base mesh to be encoded. This is a diagram showing an example of the data structure of the base mesh during encoding. This is a diagram explaining the encoding process. This is a diagram showing an example of the bitstream structure. This is a diagram explaining the decoding process. This is a diagram showing an example of the encoding and decoding method in the case of multiple attributes. This is a diagram showing an example of the data structure of the base mesh during encoding. This is a diagram showing an example of the data structure of the base mesh during encoding. This is a diagram showing an example of the bitstream structure. This is a block diagram showing an example of the main configuration of the encoding device. This is a block diagram showing an example of the main configuration of the base mesh encoding unit. This is a flowchart explaining an example of the encoding process flow. This is a flowchart explaining an example of the base mesh encoding process flow. This is a block diagram showing an example of the main configuration of the decoding device. This is a block diagram showing an example of the main configuration of the base mesh decoding unit. This is a flowchart explaining an example of the decoding process flow. This is a flowchart explaining an example of the base mesh decoding process flow. This is a diagram showing an example of the bitstream structure. This is a diagram showing an example of the bitstream structure. This is a diagram showing an example of the data structure of the base mesh during encoding in the case of an interframe. This is a diagram showing an example of the main configuration of the distribution system. This is a block diagram showing the main configuration examples for a server. This is a block diagram showing the main configuration examples for a terminal device. This is a flowchart illustrating an example of the distribution process flow. This is a flowchart illustrating an example of the V-DMC bitstream generation process flow. This is a block diagram showing the main configuration examples for a computer.

[0015] The following describes the forms for implementing this disclosure (hereinafter referred to as embodiments). The explanation will be given in the following order: 1. Supporting literature, etc., for technical content and technical terminology 2. View-dependent texture 3. Linking geometry information and UV set 4. Appendix

[0016] <1. Supporting Documents for Technical Content and Terminology> The scope disclosed in this technology includes not only the contents described in the embodiments, but also the contents described in the following non-patent documents that were publicly known at the time of filing, as well as the contents of other documents referenced in the following non-patent documents.

[0017] Non-patent document 1: (mentioned above) Non-patent document 2: (mentioned above) Non-patent document 3: Danillo Bracco Graziosi, Alexandre Zaghetto, Ali Tabatabai, "[V-DMC][Requirements] Meshes with multiple attributes", ISO / IEC JTC 1 / SC 29 / WG 7 m64028, July 2023 Non-patent document 4: Danillo B Graziosi, Alexandre Zaghetto, Ali Tabatabai, "[V-DMC][EE4.17] Report of EE 4.17 on meshes with multiple texture maps", ISO / IEC JTC 1 / SC 29 / WG 7 m67494, April 2024

[0018] In other words, the content described in the aforementioned non-patent literature, as well as the content of other documents referenced in the aforementioned non-patent literature, can also serve as a basis for determining the support requirements.

[0019] <2. View-dependent texture> <V-DMC> Traditionally, meshes have been used as 3D data to represent the three-dimensional structure of three-dimensional structures (objects with three-dimensional shapes). These meshes represent the three-dimensional shape of the object's surface by forming polygons with vertices and connections (also called edges).

[0020] As shown in the upper left of Figure 1, in the mesh, polygonal planes are formed by vertices 11 and connections 12 that connect these vertices 11. These polygons (also called faces) represent the surface of a three-dimensional object, that is, the three-dimensional shape of the object. A texture 13 can be applied to each face of this mesh.

[0021] The mesh data is composed of information such as that shown in the lower part of Figure 1. The vertex information 14, shown first from the left in the lower part of Figure 1, is information indicating the three-dimensional position (three-dimensional coordinates (X, Y, Z)) of each vertex 11 that makes up the mesh. The connection information 15, shown second from the left in the lower part of Figure 1, is information indicating each connection (edge) 12 that makes up the mesh. The texture image 16, shown third from the left in the lower part of Figure 1, is map information of the texture 13 that is applied to each face. The UV vertex coordinate list 17, shown fourth from the left in the lower part of Figure 1, is information indicating the correspondence between the vertices 11 and the texture 13. In other words, the UV vertex coordinate list 17 shows the coordinates (UV coordinates) of each vertex 11 in the texture image 16.

[0022] One method for encoding such meshes is V-DMC (Video-based Dynamic Mesh Coding), as disclosed in Non-Patent Document 1.

[0023] In V-DMC, the mesh to be encoded (referred to herein as the original mesh) is represented by a base mesh with lower resolution (i.e., coarser) than the original mesh, and displacement vectors of the division points obtained by subdividing the base mesh, and the base mesh and displacement vectors are encoded.

[0024] For example, suppose we have an original mesh as shown in the top row of Figure 2. In Figure 2, the black dots represent vertices, and the lines connecting the black dots represent connections (edges). As mentioned above, a mesh is originally formed by polygons consisting of vertices and edges, but here, for the sake of explanation, it is described as a group of vertices connected linearly (in series).

[0025] By decimating some of the vertices in the original mesh, a coarse (low-resolution) mesh was formed, as shown in the second row from the top in Figure 2. This will be used as the base mesh.

[0026] By subdividing each polygon of this base mesh, vertices and edges were added, as shown in the third row from the top in Figure 2. Here, it is assumed that this subdivision adds the same number of vertices as the original mesh. As a result, a mesh with the same number of vertices as the original mesh was obtained. In this specification, these added vertices are also referred to as subdivision points.

[0027] However, when vertices of the original mesh are thinned, the connections are updated, and the division points are formed on these updated connections (edges). Therefore, the shape of this subdivided base mesh differs from that of the original mesh. More specifically, as shown in the bottom row of Figure 2, the positions of the division points (on the dotted lines) differ from those of the original mesh. Thus, a displacement vector representing the difference between the position of this division point and the position of the vertices in the original mesh is generated.

[0028] By representing the original mesh as a base mesh and displacement vector in this way, the number of polygons (i.e., the number of vertices and edges) is reduced. Therefore, by encoding the base mesh and displacement vector instead of the original mesh, the reduction in encoding efficiency (increase in the amount of code) is suppressed.

[0029] During decoding, the bitstream was decoded using a decoding method corresponding to the encoding method, and the base mesh and displacement vectors were restored (generated). Then, the base mesh was subdivided, and the displacement vectors were applied to each subdivision point, thereby restoring (generating) the original mesh. In reality, the restored mesh may contain encoding distortions, etc., so it may not perfectly match the original mesh before encoding. However, for the sake of explanation, encoding distortions, etc., will not be considered below, and the explanation will assume that the original mesh is restored by decoding. Also, in the following, polygons will be assumed to be triangular in shape. Therefore, in the following, polygons (faces) will also be referred to as triangles.

[0030] Furthermore, a mesh can be assigned information other than vertices and connections, such as the textures mentioned above. This type of information is also called an attribute. In contrast to these attributes, the vertices and connections of the mesh are called geometry. Attributes are not limited to textures; they can be any type of information. For example, information about normals or reflectivity may be included in attributes. Also, an attribute may be composed of multiple pieces of information, such as texture and normals, or texture and reflectivity. V-DMC can encode and decode these attributes.

[0031] In the case of V-DMC, as shown in texture image 16 in Figure 1, the attributes of three-dimensional space are unfolded onto a two-dimensional plane, encoded using an image encoding scheme, and then decoded (also referred to as 2D encoding and 2D decoding, respectively). More specifically, the geometry of three-dimensional space is unfolded onto a two-dimensional plane, and the attributes of three-dimensional space are unfolded onto that two-dimensional plane according to the pattern (distribution) of the unfolding of that geometry. Here, "unfolding" refers to arranging the mesh (geometry and attributes) of three-dimensional space onto a two-dimensional plane.

[0032] In this specification, the two-dimensional plane on which the mesh (geometry and attributes) of three-dimensional space is unfolded is also referred to as the "UV plane." The unfolding onto this UV plane is also referred to as "UV unfolding." The coordinates (two-dimensional coordinates) on this UV plane are also referred to as "UV coordinates." The vertices of the mesh unfolded on the UV plane are also referred to as "UV vertices," and the UV coordinates of these UV vertices are also referred to as "UV vertex coordinates." The connections between UV vertices that correspond to connections in the mesh of three-dimensional space are also referred to as "UV connectivity." In other words, UV connectivity can be said to be a UV-unfolded connection. The faces on the UV plane that correspond to faces in the mesh of three-dimensional space (i.e., UV-unfolded faces) are also referred to as "UV faces."

[0033] Furthermore, in this specification, to distinguish them from the UV vertices mentioned above, vertices in a mesh in three-dimensional space are also referred to as "geometry vertices." The three-dimensional coordinates of these geometry vertices are also referred to as "geometry vertex coordinates." The connections between geometry vertices (i.e., connections in a mesh in three-dimensional space) are also referred to as "geometry connectivity."

[0034] In V-DMC, the base mesh is UV unwrapped as described above. Then, the geometric vertex coordinates, geometric connectivity, UV vertex coordinates, and UV connectivity are encoded and decoded as information of the base mesh. In this specification, geometric vertex coordinates and geometric connectivity are also referred to as "geometry information." UV vertex coordinates and UV connectivity are also referred to as "UV sets." The bitstream in which the geometry information is encoded is also referred to as the "geometry information bitstream." The bitstream in which the UV sets are encoded is also referred to as the "UV set bitstream."

[0035] Furthermore, as mentioned above, in V-DMC, attributes in three-dimensional space are UV unwrapped, 2D encoded, and 2D decoded. In this specification, the UV-unwrapped attributes are also referred to as "attribute maps," and the UV-unwrapped textures are also referred to as "texture maps."

[0036] <Assigning Textures to Meshes> Generally, in a mesh, one texture is assigned to the same part (for example, one face). For example, in Figure 3, the captured image 33A taken at viewpoint 32A is assigned as a texture to the mesh object 31. In V-DMC, the texture (captured image 33A) assigned to the mesh object 31 is UV unwrapped and encoded as a texture map 34A.

[0037] In this case, since only one texture is assigned to the mesh object 31, the same texture will be rendered even when rendering from viewpoint 32B or viewpoint 32C. In other words, the color, etc., of the mesh object 31 will not change even if the viewing position or angle is changed. Consequently, there was a risk that the texture (color, etc.) of the mesh object in the rendered image would appear unnatural. In other words, there was a risk that the quality of the attributes would be reduced.

[0038] In contrast, there is a technique called view-dependent texture, which assigns multiple textures to the same part of a mesh and selects the most suitable texture during rendering, for example, depending on the viewpoint position. In this technique, as shown in Figure 4, for example, the image 33A taken at viewpoint 32A, the image 33B taken at viewpoint 32B, and the image 33C taken at viewpoint 32C are each assigned as textures to the mesh object 31. In other words, three textures (the texture obtained from image 33A, the texture obtained from image 33B, and the texture obtained from image 33C) are assigned to the same part of the mesh object 31 (for example, a single face). Then, for example, when rendering is performed with viewpoint 32A applied, the texture of image 33A is applied. When rendering is performed with viewpoint 32B applied, the texture of image 33B is applied. When rendering is performed with viewpoint 32C applied, the texture of image 33C is applied. In this way, during rendering, the texture applied is switched to the most suitable one depending on the viewpoint, resulting in a more natural-looking rendered image. In other words, applying view-dependent textures improves the quality of attributes compared to assigning a single texture to the mesh.

[0039] As mentioned above, attributes (textures) are UV unwrapped using the same pattern as geometry. That is, the attributes of geometry vertices are placed in the attribute map at the UV vertex coordinates corresponding to those geometry vertices. In the case where multiple textures are assigned to the same part of a mesh, as in the example in Figure 4, if the UV unwrapping pattern of each texture is common, only one type of UV connectivity and UV vertex coordinates are needed to constitute the base mesh, but that UV unwrapping pattern will be applied to all textures. However, that UV unwrapping pattern is not necessarily the best for all textures, and there is a risk that the quality of the attributes will be reduced due to unwanted distortion in the textures. For example, in the case of Figure 4, the textures of captured image 33B and captured image 33C will also be UV unwrapped using the pattern of the texture map 34A corresponding to captured image 33A, which could reduce the quality.

[0040] To suppress such attribute quality degradation, it is desirable that each attribute (texture) be UV unwrapped using the pattern best suited to that attribute, as shown in the example in Figure 5 (texture map 34A, texture map 34B, texture map 34C).

[0041] For example, Non-Patent Document 2 describes a method for associating multiple UV maps with a single face of a mesh. Non-Patent Documents 3 and 4 describe a method for separating attribute maps for each part of the base mesh in V-DMC (identifying maps by applying texture IDs). In this method, one attribute is assigned to each face, but by combining this with the method described in Non-Patent Document 2, it seems possible to associate multiple attribute maps with different UV unwrappings with a single face in the V-DMC base mesh.

[0042] However, since the geometry information (geometry vertex coordinates and geometry connectivity) and UV set (UV vertex coordinates and UV connectivity) that make up the base mesh are encoded as a set, in the case of a base mesh with multiple texture maps assigned, as in the example in Figure 6, the faces of the base mesh are duplicated and encoded, as shown in Figure 7. The faces shown in Figure 6 are assigned two textures that are UV unwrapped independently of each other, and for each geometry vertex (v1, v2, v3, v4), two UV vertices (vt1 and vt5, vt2 and vt6, vt3 and vt7, vt4 and vt8) are set. In other words, two UV vertex coordinates are set for one geometry vertex coordinate. When encoding such a base mesh, the faces in Figure 6 are converted into two faces (Face1 and Face2) as shown in Figure 7, and each is encoded.

[0043] In other words, as shown in Figure 8, in the encoding device 50, the metadata generation unit 51 duplicates the geometry vertex coordinates and geometry connectivity, and generates N (number of cameras) pairs of geometry information and UV sets. That is, N base meshes (base meshes corresponding to the captured images of each camera) are generated. Then, the geometry connectivity encoding unit 52, UV connectivity encoding unit 53, geometry vertex coordinate encoding unit 54, and UV vertex coordinate encoding unit 55 encode the respective information of each pair. That is, the geometry connectivity encoding unit 52, UV connectivity encoding unit 53, geometry vertex coordinate encoding unit 54, and UV vertex coordinate encoding unit 55 encode the respective information N (number of cameras), and generate a base mesh bitstream, which is a bitstream of the encoded base meshes.

[0044] Therefore, as shown in FIG. 9, a plurality of base mesh bit streams store the same geometric vertex coordinates and geometric connectivity. As shown in FIG. 10, in the decoding device 60, a geometric connectivity decoding unit 61, a UV connectivity decoding unit 62, a geometric vertex coordinate decoding unit 63, and a UV vertex coordinate decoding unit 64 decode each piece of information (bit stream) N (the number of cameras) times and generate N decoded meshes (decoded meshes corresponding to the captured images of each camera). This decoded mesh is a mesh obtained by decoding processing and corresponds to the base mesh before encoding. If the encoding / decoding is lossless, the decoded mesh matches the base mesh before encoding.

[0045] Thus, in the above method, the base mesh bit stream contains redundant information, and there is a possibility that its encoding efficiency will be reduced.

[0046] <3. Association between Geometric Information and UV Set> <Method 1> Therefore, as shown in the uppermost row of the table shown in FIG. 11, as the base mesh, a single geometric information and a plurality of UV sets associated with the geometric information are transmitted (Method 1).

[0047] For example, the first information processing device may include a base mesh encoding unit that encodes, as the base mesh, a single geometric information corresponding to at least one face of the base mesh and a plurality of UV sets corresponding to the face and associated with the single geometric information, and an attribute encoding unit that encodes a plurality of attribute maps corresponding to the plurality of UV sets as attributes corresponding to the base mesh.

[0048] For example, the first information processing method may include encoding, as the base mesh, single geometry information corresponding to at least one face of the base mesh and a plurality of UV sets corresponding to the face and linked to the single geometry information, and encoding, as an attribute corresponding to the base mesh, a plurality of attribute maps respectively corresponding to the plurality of UV sets.

[0049] For example, the first program may be a program for causing a computer to execute a process including encoding, as the base mesh, single geometry information corresponding to at least one face of the base mesh and a plurality of UV sets corresponding to the face and linked to the single geometry information, and encoding, as an attribute corresponding to the base mesh, a plurality of attribute maps respectively corresponding to the plurality of UV sets.

[0050] Further, the second information processing apparatus may include a base mesh decoding unit that decodes, as a base mesh, a single geometry information bitstream and a plurality of UV set bitstreams linked to the single geometry information bitstream, and generates single geometry information corresponding to at least one face of the base mesh and a plurality of UV sets corresponding to the face, and an attribute decoding unit that decodes, as an attribute corresponding to the base mesh, a plurality of attribute bitstreams respectively corresponding to the plurality of UV set bitstreams, and generates a plurality of attribute maps respectively corresponding to the plurality of UV sets.

[0051] For example, the second information processing method may include decoding a single geometry information bitstream and a plurality of UV set bitstreams associated with that single geometry information bitstream as a base mesh, generating a single geometry information corresponding to at least one face of the base mesh and a plurality of UV sets corresponding to that face, and decoding a plurality of attribute bitstreams corresponding to each of the plurality of UV set bitstreams as attributes corresponding to the base mesh, and generating a plurality of attribute maps corresponding to each of the plurality of UV sets.

[0052] For example, the second program may be a program that causes a computer to perform the following processes: decode a single geometry information bitstream and multiple UV set bitstreams associated with that single geometry information bitstream as a base mesh, generate a single geometry information corresponding to at least one face of the base mesh and multiple UV sets corresponding to that face; and decode multiple attribute bitstreams corresponding to each of the multiple UV set bitstreams as attributes corresponding to the base mesh, and generate multiple attribute maps corresponding to each of the multiple UV sets.

[0053] For example, the base mesh in Figure 6 is encoded as shown in Figure 12. In other words, multiple UV sets (UV vertex coordinates and UV connectivity) are linked to the same geometry information (geometry vertex coordinates and geometry connectivity). This linking of geometry information and UV sets can be done in any way.

[0054] For example, as shown in Figure 13, this association may be performed using the submesh ID, which is the identification information of the geometry information bitstream. In other words, each UV set bitstream stores a reference submesh ID that indicates the geometry information of the linked (referenced) source. In the example in Figure 13, the submesh ID of the geometry information bitstream 111 (= 0) is stored as the reference submesh ID in both UV set bitstream 112 and UV set bitstream 113. This indicates that UV set bitstream 112 and UV set bitstream 113 are linked to the geometry information bitstream 111.

[0055] Any number of UV set bitstreams can be associated with a single geometry information bitstream. If there are three UV sets with common geometry information, as in the example in Figure 9, then three UV set bitstreams may be associated with a single geometry information bitstream, as shown in Figure 14. In the example in Figure 14, UV set bitstreams 122, 123, and 124 are associated with geometry information bitstream 121. That is, each of UV set bitstreams 122, 123, and 124 stores the submesh ID (= i) of geometry information bitstream 121 as a reference submesh ID.

[0056] By doing so, the redundancy of geometric connectivity and geometric vertex coordinates can be suppressed, and the reduction in the encoding efficiency of the base mesh can be mitigated.

[0057] <Attributes> Attributes can be any type of information. For example, an attribute may include textures. An attribute may also include information indicating normals. An attribute may also include information indicating reflectivity. An attribute may also include other types of information.

[0058] <UV Sets> UV sets may be generated for each projection direction (camera) of geometry and attributes, for example. In other words, multiple UV sets may correspond to each of multiple captured images applied as attributes of the base mesh. These multiple captured images may correspond to each of multiple different viewpoints. In this way, view-dependent textures can be realized while suppressing the reduction in encoding efficiency of the base mesh, for example.

[0059] Alternatively, all UV sets may be transmitted, and the UV set to be applied during rendering may be selected. For example, in the first information processing device, the base mesh encoding unit may encode all of its multiple UV sets. The attribute encoding unit may encode all of its multiple attribute maps. In addition, in the second information processing device, the base mesh decoding unit may decode the single geometry information bitstream and all of the multiple UV set bitstreams associated with that single geometry information bitstream. By doing so, the decoding side can select the desired UV set from all the UV sets associated with the same geometry information, for example, during rendering, thereby suppressing a reduction in attribute quality while suppressing a reduction in the encoding efficiency of the base mesh.

[0060] <Encoding and Decoding> This base mesh is encoded and transmitted as a bitstream on the encoding side. On the decoding side, that bitstream is decoded and the base mesh is generated.

[0061] The base mesh may be encoded in any way or procedure. For example, as described above, the geometry information and UV sets may be encoded separately to create different bitstreams. Alternatively, geometry information and UV sets corresponding to the same faces may be linked, the geometry information may be encoded to generate a geometry information bitstream, and then the UV sets linked to that geometry information may be encoded to generate a UV set bitstream. For example, in the first information processing device, the base mesh encoding unit may link multiple UV sets to a single geometry information, encode that single geometry information to generate a single geometry information bitstream, and then encode each of the multiple UV sets to generate multiple UV set bitstreams linked to that single geometry information bitstream.

[0062] Furthermore, in generating the geometry information bitstream, the geometry connectivity may be encoded before the geometry vertex coordinates. Similarly, in generating the UV set bitstream, the UV connectivity may be encoded before the UV vertex coordinates. For example, in the first information processing device, multiple UV sets may each contain UV connectivity and UV vertex coordinates. The base mesh encoding unit may encode the geometry connectivity and geometry vertex coordinates contained in the single geometry information to generate the single geometry information bitstream, and then encode the UV connectivity and UV vertex coordinates for each of the multiple UV sets to generate the multiple UV set bitstreams.

[0063] The same applies to the decoding of the base mesh; it can be performed in any way or procedure. For example, as described above, the geometry information bitstream and the UV set bitstream may be decoded separately. Alternatively, the geometry information bitstream may be decoded first, and then the UV set bitstream associated with that geometry information bitstream may be decoded. For example, in the second information processing device, the base mesh decoding unit may decode the single geometry information bitstream and then decode the multiple UV set bitstreams associated with that single geometry information bitstream.

[0064] Furthermore, in decoding the geometry information bitstream, the encoded data for geometry connectivity may be decoded first, followed by the encoded data for geometry vertex coordinates. Similarly, in decoding the UV set bitstream, the encoded data for UV connectivity may be decoded first, followed by the encoded data for UV vertex coordinates. For example, in the second information processing device, each of the multiple UV set bitstreams may contain encoded data for UV connectivity and encoded data for UV vertex coordinates. The base mesh decoding unit may decode the encoded data for geometry connectivity and the encoded data for geometry vertex coordinates contained in the single geometry information bitstream, and then decode the encoded data for UV connectivity and the encoded data for UV vertex coordinates for each of the multiple UV set bitstreams.

[0065] The encoding and decoding method for the base mesh may be any method, for example, MPEG edge breaker or draco. The encoding and decoding method for UV vertex coordinates may be any method, for example, absolute values ​​may be encoded and decoded, or predictive encoding and decoding using geometry information may be applied. Furthermore, the encoding and decoding method for UV connectivity may be any method, for example, whether or not a vertex is a UV boundary edge may be encoded and decoded for each vertex of each face, except for vertices at the geometry edge boundary. In this specification, the encoding and decoding method for UV connectivity that encodes information indicating whether or not a vertex is a UV boundary edge is also referred to as "UVSeams". The encoding and decoding may also be performed in the traverse order of the geometry connectivity.

[0066] The method for encoding and decoding the attributes can be anything; for example, each attribute map may be 2D encoded and each attribute bitstream may be 2D decoded.

[0067] <Method of linking> The method of linking geometry information and UV sets can be anything. For example, as explained with reference to Figures 13 and 14, the ID of the referenced geometry information (reference submesh ID) may be stored in the UV set bitstream. For example, in the first information processing device, the base mesh encoding unit may generate linking information that links the single geometry information with multiple UV sets, and store that linking information as metadata in each of the generated UV set bitstreams. Alternatively, in the second information processing device, the base mesh decoding unit may identify and decode the multiple UV set bitstreams linked to the decoded single geometry information bitstream based on the linking information that links the single geometry information with multiple UV sets, which is stored as metadata in each of the multiple UV set bitstreams. By configuring the bitstream in this way, the linking between the geometry information bitstream and the UV set bitstream can be easily understood without requiring any other information.

[0068] Furthermore, for example, table information showing the correspondence between the ID of the referenced geometry information (reference submesh ID) and the ID of the UV set bitstream (UV bitstream ID (camera ID)) may be transmitted (separately). For example, in the first information processing device, the base mesh encoding unit may generate table information showing the correspondence between the single geometry information and multiple UV sets as linking information that links the single geometry information with multiple UV sets. Alternatively, in the second information processing device, the base mesh decoding unit may identify and decode the multiple UV set bitstreams associated with the decoded single geometry information bitstream based on the table information showing the correspondence between the single geometry information and multiple UV sets. By doing so, it becomes unnecessary to store identification information of the geometry information (geometry information bitstream) within the UV set bitstream, and thus the reduction in the encoding efficiency of the UV set bitstream can be suppressed.

[0069] Furthermore, this technology can be applied, for example, to an intraframe (static mesh) of a mesh (base mesh). For example, in the first information processing device, the base mesh encoding unit may encode the single geometry information and the multiple UV sets associated with that single geometry information as the base mesh in the intraframe. Alternatively, in the second information processing device, the base mesh decoding unit may decode the single geometry information bitstream and the multiple UV set bitstreams associated with that single geometry information bitstream as the base mesh in the intraframe. By doing so, it is possible to suppress a reduction in attribute quality while suppressing a reduction in the encoding efficiency of the base mesh in the intraframe (static mesh).

[0070] <Encoding Device> This technology can be applied to any device. For example, this technology can be applied to an encoding device that encodes a mesh and generates a bitstream. Figure 15 is a block diagram showing an example of the configuration of an encoding device, which is one aspect of a first information processing device to which this technology is applied. The encoding device 300 shown in Figure 15 is a device that encodes a mesh and generates its bitstream. Therefore, the encoding device 300 can also be said to be a bitstream generator that generates a bitstream. Note that Figure 15 shows the main configuration example when processing an intraframe.

[0071] Figure 15 shows the main components such as the processing unit and data flow, but it does not necessarily represent everything. In other words, the encoding device 300 may have processing units that are not shown as blocks in Figure 15, or processes and data flows that are not shown as arrows or other symbols in Figure 15.

[0072] The encoding device 300 encodes the mesh in essentially the same way as the V-DMC described in the above-mentioned non-patent literature, except that it applies this technology. For example, the encoding device 300 obtains the original mesh to be encoded and an attribute map containing the texture corresponding to that original mesh. The encoding device 300 encodes the original mesh and attribute map using the V-DMC method, generates a V-DMC bitstream, and outputs it.

[0073] As shown in Figure 15, the encoding device 300 includes a pre-processing unit 311, an atlas information encoding unit 312, a base mesh encoding unit 313, a displacement vector encoding unit 314, an attribute encoding unit 315, and a multiplexing unit 316. The pre-processing unit 311 performs pre-processing before encoding. As shown in Figure 15, the pre-processing unit 311 includes a base mesh generation unit 321, a UV set generation unit 322, a displacement vector generation unit 323, and an attribute conversion unit 324.

[0074] The base mesh generation unit 321 performs processing related to the generation of a base mesh. For example, the base mesh generation unit 321 may acquire the original mesh that is supplied to the encoding device 300. Alternatively, the base mesh generation unit 321 may perform decimation processing (vertex thinning) on ​​the original mesh to generate a base mesh. Here, the original mesh is composed of information indicating vertices and connections in three-dimensional space (i.e., geometry information). Therefore, the base mesh generated by the decimation processing is also composed of geometry information (geometry vertex coordinates and geometry connectivity). In other words, the base mesh generation unit 321 may generate the geometry vertex coordinates and geometry connectivity of the base mesh by thinning out vertex information from the geometry vertex coordinates and geometry connectivity of the original mesh. The base mesh generation unit 321 may supply the base mesh thus generated together with the original mesh to the UV set generation unit 322.

[0075] The UV set generation unit 322 performs processing related to the generation of UV sets corresponding to geometry information. For example, the UV set generation unit 322 may obtain a base mesh or original mesh supplied from the base mesh generation unit 321. The UV set generation unit 322 may UV unwrap the base mesh (geometry information) to generate a UV set (UV vertex coordinates and UV connectivity). In this case, if multiple attributes are assigned to the same part of the base mesh (for example, a single face), the UV set generation unit 322 may generate a UV set corresponding to each attribute. In other words, the UV set generation unit 322 may generate multiple UV sets for a single piece of geometry information. In this case, the UV unwrapping patterns of the geometry information for generating each UV set may be independent of each other. For example, the UV set generation unit 322 may generate each UV set by UV unwrapping the geometry information in the optimal pattern for each attribute.

[0076] The UV set generation unit 322 may supply the geometry information and the generated UV set as a base mesh to the base mesh encoding unit 313. Alternatively, the UV set generation unit 322 may generate atlas information using the base mesh and supply the generated atlas information to the atlas information encoding unit 312. Furthermore, the UV set generation unit 322 may supply the original mesh and the base mesh (geometry information and the generated UV set) to the displacement vector generation unit 323 and the attribute conversion unit 324.

[0077] The displacement vector generation unit 323 performs processing related to the generation of displacement vectors. For example, the displacement vector generation unit 323 may acquire the base mesh and original mesh supplied from the UV set generation unit 322. Using this information, the displacement vector generation unit 323 may generate displacement vectors that displace the vertices of the subdivided base mesh. The displacement vector generation unit 323 may supply the generated displacement vectors to the displacement vector encoding unit 314.

[0078] The attribute conversion unit 324 performs processing related to the conversion of attribute maps. For example, the attribute conversion unit 324 may acquire an attribute map supplied to the encoding device 300. This attribute map corresponds to the original mesh. The attribute conversion unit 324 may acquire a base mesh and an original mesh supplied from the UV set generation unit 322. The attribute conversion unit 324 may use the acquired base mesh and original mesh to convert the acquired attribute map and generate an attribute map corresponding to the base mesh. The attribute conversion unit 324 may supply the converted attribute map (the attribute map corresponding to the base mesh) to the attribute encoding unit 315.

[0079] Furthermore, if multiple attributes are assigned to the same part of the base mesh (for example, a single face), the encoding device 300 is supplied with these multiple attribute maps. In that case, the attribute conversion unit 324 may convert each of these multiple attribute maps and supply each converted attribute map to the attribute encoding unit 315.

[0080] The Atlas information encoding unit 312 performs processing related to encoding Atlas information. For example, the Atlas information encoding unit 312 may acquire Atlas information supplied from the UV set generation unit 322. The Atlas information encoding unit 312 may encode the acquired Atlas information using a predetermined encoding scheme and generate encoded Atlas information data (also referred to as Atlas information bitstream). The Atlas information encoding unit 312 may supply the generated encoded Atlas information data (Atlas information bitstream) to the multiplexing unit 316.

[0081] The base mesh encoding unit 313 performs processing related to the encoding of the base mesh. For example, the base mesh encoding unit 313 may acquire the base mesh (geometry information and UV set) supplied from the UV set generation unit 322. The base mesh encoding unit 313 may encode the acquired base mesh and generate encoded base mesh data. For example, the base mesh encoding unit 313 may quantize the acquired base mesh and encode it using a predetermined encoding method (e.g., Draco). The base mesh encoding unit 313 may supply the generated encoded base mesh data to the multiplexing unit 316.

[0082] The displacement vector coding unit 314 performs processing related to the coding of the displacement vector. For example, the displacement vector coding unit 314 may acquire the displacement vector supplied from the displacement vector generation unit 323. The displacement vector coding unit 314 may code the acquired displacement vector and generate coded data of the displacement vector (also referred to as a displacement vector bitstream). For example, the displacement vector coding unit 314 may generate a displacement map by performing a wavelet transform on the displacement vector, quantizing it, and packing it into a two-dimensional region. Alternatively, the displacement vector coding unit 314 may generate a displacement video using the displacement map as the frame image. In other words, a displacement video is a moving image in which the displacement map, which is a two-dimensional region in which the displacement vector is packed, is used as the frame image. The displacement vector coding unit 314 may code the generated displacement video using a predetermined coding scheme for 2D moving images and generate coded data of the displacement vector (displacement vector bitstream). The displacement vector coding unit 314 may also generate coded data of the displacement vector (displacement vector bitstream) by arithmetic coding the displacement vector. The displacement vector encoding unit 314 may supply the encoded data of the generated displacement vector (displacement vector bitstream) to the multiplexing unit 316.

[0083] The attribute encoding unit 315 performs processing related to the encoding of attributes. For example, the attribute encoding unit 315 may acquire an attribute map (an attribute map corresponding to the base mesh) supplied from the attribute conversion unit 324. The attribute encoding unit 315 may encode the acquired attribute map and generate encoded attribute data (also referred to as an attribute bitstream). In this case, the attribute encoding unit 315 may generate an attribute video using the attribute map as a frame image, encode the generated attribute video using a predetermined encoding method for 2D moving images, and generate encoded attribute data (attribute bitstream). The attribute encoding unit 315 may supply the generated encoded attribute data (attribute bitstream) to the multiplexing unit 316.

[0084] Furthermore, if multiple attributes are assigned to the same part of the base mesh (for example, a single face), the attribute encoding unit 315 is supplied with these multiple converted attribute maps. In this case, the attribute encoding unit 315 may encode each of these multiple attribute maps individually. In other words, the attribute encoding unit 315 may generate encoded data (attribute bitstreams) for these multiple attributes and supply them to the multiplexing unit 316.

[0085] The multiplexing unit 316 performs processing related to the multiplexing of encoded data (substreams). For example, the multiplexing unit 316 may acquire encoded data of atlas information (atlas information bitstream) supplied from the atlas information encoding unit 312. The multiplexing unit 316 may acquire encoded data of base mesh supplied from the base mesh encoding unit 313. The multiplexing unit 316 may acquire encoded data of displacement vectors (displacement vector bitstream) supplied from the displacement vector encoding unit 314. The multiplexing unit 316 may acquire encoded data of attributes (attribute bitstream) supplied from the attribute encoding unit 315. The multiplexing unit 316 may multiplex these encoded data as substreams to generate a V-DMC bitstream. Therefore, the multiplexing unit 316 can also be called a bitstream generation unit (or V-DMC bitstream generation unit). The multiplexing unit 316 may output the generated V-DMC bitstream to the outside of the encoding device 300. For example, the multiplexing unit 316 may supply its V-DMC bitstream to the decoding device 400, which will be described later. Therefore, the multiplexing unit 316 can also be called a bitstream (V-DMC bitstream) supply unit (provider).

[0086] <Base Mesh Encoding Unit> Figure 16 is a block diagram showing an example of the main configuration of the base mesh encoding unit 313 (Figure 15). Figure 16 shows the main processing units and data flows, but it does not necessarily show everything. In other words, the base mesh encoding unit 313 may have processing units that are not shown as blocks in Figure 16, or processing and data flows that are not shown as arrows, etc. in Figure 16.

[0087] As shown in Figure 16, the base mesh coding unit 313 includes a metadata generation unit 351, a geometry connectivity coding unit 352, a geometry vertex coordinate coding unit 353, a UV connectivity coding unit 354-1, a UV vertex coordinate coding unit 355-1, a UV connectivity coding unit 354-2, a UV vertex coordinate coding unit 355-2, ..., a UV connectivity coding unit 354-N, and a UV vertex coordinate coding unit 355-N. N is an arbitrary natural number indicating the maximum number of UV sets corresponding to one geometry information in the supplied base mesh. When it is not necessary to distinguish between the UV connectivity coding units 354-1, 354-2, ..., and 354-N, they are also referred to as the UV connectivity coding unit 354. When there is no need to distinguish between UV vertex coordinate coding units 355-1, UV vertex coordinate coding units 355-2, ..., UV vertex coordinate coding units 355-N, they are also referred to as UV vertex coordinate coding unit 355. In other words, the base mesh coding unit 313 has one metadata generation unit 351, one geometry connectivity coding unit 352, and one geometry vertex coordinate coding unit 353, and further has N UV connectivity coding units 354 and N UV vertex coordinate coding units 355.

[0088] The base mesh encoding unit 313 is supplied with geometry vertex coordinates, geometry connectivity, UV vertex coordinates, and UV connectivity as the base mesh.

[0089] The metadata generation unit 351 performs processing related to metadata generation. For example, the metadata generation unit 351 may acquire a base mesh to be supplied to the base mesh encoding unit 313. The metadata generation unit 351 may generate metadata using the acquired base mesh (geometry vertex coordinates, geometry connectivity, UV vertex coordinates, and UV connectivity). For example, the metadata generation unit 351 may generate linking information as metadata that links geometry information and UV sets corresponding to the same faces. This linking information may consist of any information. For example, the metadata generation unit 351 may generate this linking information using identification information of the geometry information. The metadata generation unit 351 may supply the generated metadata, along with the base mesh (geometry vertex coordinates, geometry connectivity, UV vertex coordinates, and UV connectivity), to the geometry connectivity encoding unit 352.

[0090] The geometry connectivity encoding unit 352 performs processing related to the encoding of geometry connectivity. For example, the geometry connectivity encoding unit 352 may acquire a base mesh (geometry vertex coordinates, geometry connectivity, UV vertex coordinates, and UV connectivity) and metadata supplied from the metadata generation unit 351. The geometry connectivity encoding unit 352 may encode the geometry connectivity included in the acquired base mesh and generate encoded geometry connectivity data. The method for encoding geometry connectivity may be any method. The geometry connectivity encoding unit 352 may supply the encoded geometry connectivity data it has generated, along with the base mesh (including at least geometry vertex coordinates, UV connectivity, and UV vertex coordinates) and metadata, to the geometry vertex coordinate encoding unit 353.

[0091] The geometry vertex coordinate encoding unit 353 performs processing related to encoding geometry vertex coordinates. For example, the geometry vertex coordinate encoding unit 353 may obtain a base mesh (including at least geometry vertex coordinates, UV connectivity, and UV vertex coordinates) supplied from the geometry connectivity encoding unit 352, metadata, and encoded geometry connectivity data. The geometry vertex coordinate encoding unit 353 may encode the geometry vertex coordinates included in the obtained base mesh and generate encoded geometry vertex coordinate data. Any method may be used to encode the geometry vertex coordinates.

[0092] The geometry vertex coordinate encoding unit 353 may generate a single geometry information bitstream and store encoded data of geometry vertex coordinates and encoded data of geometry connectivity in that geometry information bitstream. In other words, the geometry information bitstream is a bitstream in which geometry information has been encoded. The geometry vertex coordinate encoding unit 353 may also store acquired metadata in the generated geometry information bitstream. For example, the geometry vertex coordinate encoding unit 353 may store identification information of geometry information in the geometry information bitstream.

[0093] The geometry vertex coordinate encoding unit 353 may supply the geometry information bitstream (including metadata) generated in this manner to the multiplexing unit 316 as encoded data for the base mesh. The geometry vertex coordinate encoding unit 353 may also supply the base mesh (including at least UV connectivity and UV vertex coordinates) and metadata to the UV connectivity encoding unit 354.

[0094] The UV connectivity encoding unit 354 performs processing related to the encoding of UV connectivity. For example, the UV connectivity encoding unit 354 may acquire a base mesh (including at least UV connectivity and UV vertex coordinates) and metadata supplied from the geometry vertex coordinate encoding unit 353. The UV connectivity encoding unit 354 may encode the UV connectivity included in the acquired base mesh and generate encoded UV connectivity data. Any method may be used to encode this UV connectivity. For example, as described above, UVSeams may be applied. The UV connectivity encoding unit 354 may supply the generated encoded UV connectivity data, along with the base mesh (including at least UV vertex coordinates) and metadata, to the UV vertex coordinate encoding unit 355.

[0095] The UV vertex coordinate encoding unit 355 performs processing related to encoding UV vertex coordinates. For example, the UV vertex coordinate encoding unit 355 may obtain a base mesh (including at least UV vertex coordinates), metadata, and encoded UV connectivity data supplied from the UV connectivity encoding unit 354. The UV vertex coordinate encoding unit 355 may encode the UV vertex coordinates included in the obtained base mesh and generate encoded UV vertex coordinate data. Any method can be used to encode the UV vertex coordinates. For example, in this encoding, as described above, the absolute value of the UV vertex coordinates may be encoded, or predictive encoding using geometry information may be applied.

[0096] The UV vertex coordinate encoding unit 355 may generate a single UV set bitstream and store the encoded data of UV vertex coordinates and the encoded data of UV connectivity in that UV set bitstream. In other words, the UV set bitstream is a bitstream that encodes the UV set. The UV vertex coordinate encoding unit 355 may also store acquired metadata in the generated UV set bitstream. For example, the UV vertex coordinate encoding unit 355 may store identification information (reference submesh ID) of the referenced geometry information in the UV set bitstream. In this way, the UV set bitstream is linked to the geometry information bitstream.

[0097] The UV vertex coordinate coding unit 355 may supply the UV set bitstream (including metadata) generated in this manner to the multiplexing unit 316 as coded data for the base mesh.

[0098] As described above, the base mesh coding unit 313 has N UV connectivity coding units 354 and N UV vertex coordinate coding units 355. Each pair of UV connectivity coding units 354 and UV vertex coordinate coding units 355 performs the above-described processing on the UV set corresponding to itself within a group of UV sets associated with the same geometry information. For example, the UV connectivity coding unit 354-1 and the UV vertex coordinate coding unit 355-1 perform coding on the first UV set (ID=1), generate the first UV set bitstream (ID=1), and supply it to the multiplexing unit 316. Similarly, the UV connectivity coding unit 354-2 and the UV vertex coordinate coding unit 355-2 perform coding on the second UV set (ID=2), generate the second UV set bitstream (ID=2), and supply it to the multiplexing unit 316. Similarly, the UV connectivity coding unit 354-N and the UV vertex coordinate coding unit 355-N perform coding for the Nth UV set (ID=N), generate the Nth UV set bitstream (ID=N), and supply it to the multiplexing unit 316. In other words, up to N UV set bitstreams, each associated with a single geometry information bitstream, are supplied to the multiplexing unit 316 as coded data for the base mesh.

[0099] The encoding device 300 having the above configuration may be used as the first information processing device, and this technology may be applied to it.

[0100] For example, the encoding device 300 may include a base mesh encoding unit 313 (Figure 15) that encodes a single geometric information corresponding to at least one face of the base mesh, and a plurality of UV sets corresponding to that face and linked to that single geometric information, as its base mesh, and an attribute encoding unit 315 that encodes a plurality of attribute maps corresponding to each of the plurality of UV sets as attributes corresponding to the base mesh.

[0101] Furthermore, as described above, the encoding device 300 may further include a UV set generation unit 322 that generates a plurality of UV sets corresponding to its face. In that case, the base mesh encoding unit 313 may encode the single geometry information and the plurality of UV sets that have been generated and linked to the single geometry information as the base mesh.

[0102] Furthermore, as described above, the encoding device 300 may further include a base mesh generation unit 321 that generates a base mesh by thinning out the vertices of the original mesh. In that case, the UV set generation unit 322 may generate a plurality of UV sets corresponding to the faces of the generated base mesh.

[0103] Furthermore, as described above, the encoding device 300 may further include an attribute conversion unit that converts the attribute map corresponding to the original mesh and generates multiple attribute maps corresponding to the base mesh. In that case, the attribute encoding unit may encode the multiple attribute maps that have been generated.

[0104] Furthermore, as described above, the encoding device 300 may further include a displacement vector generation unit 323 that generates a displacement vector which is the difference in vertex positions between the original mesh and the subdivided base mesh, and a displacement vector encoding unit 314 that encodes the generated displacement vector.

[0105] Furthermore, as described above, the encoding device 300 may further include an atlas information encoding unit 312 that encodes atlas information used when reconstructing the mesh.

[0106] Furthermore, as described above, the encoding device 300 may further include a multiplexing unit 316 that multiplexes a single geometry information bitstream generated by encoding the single geometry information, a plurality of UV set bitstreams generated by encoding the plurality of UV sets, a displacement vector bitstream generated by encoding the displacement vector, and an atlas information bitstream generated by encoding the atlas information.

[0107] With this configuration, the encoding device 300 (first information processing device) can suppress the redundancy of geometry connectivity and geometry vertex coordinates, thereby suppressing a reduction in the encoding efficiency of the base mesh.

[0108] The attributes supplied to this encoding device 300 can be any kind of information, and may include, for example, textures.

[0109] Furthermore, the multiple UV sets encoded by the base mesh encoding unit 313 may correspond to each of the multiple captured images applied as attributes of the base mesh. These multiple captured images may correspond to each of the multiple different viewpoints. In this way, for example, a view-dependent texture can be realized while suppressing a reduction in the encoding efficiency of the base mesh.

[0110] Furthermore, the base mesh encoding unit 313 may encode all of its multiple UV sets. Similarly, the attribute encoding unit 315 may encode all of its multiple attribute maps. By doing so, the decoding side can select a desired UV set from all UV sets associated with the same geometry information, for example, during rendering, thereby suppressing a reduction in attribute quality while suppressing a reduction in the encoding efficiency of the base mesh.

[0111] The base mesh encoding unit 313 may associate multiple UV sets with a single geometry information, encode the single geometry information to generate a single geometry information bitstream, encode each of the multiple UV sets to generate multiple UV set bitstreams associated with the single geometry information bitstream. Alternatively, each of the multiple UV sets may include UV connectivity and UV vertex coordinates. The base mesh encoding unit 313 may then encode the geometry connectivity and geometry vertex coordinates contained in the single geometry information to generate a single geometry information bitstream, and encode the UV connectivity and UV vertex coordinates for each of the multiple UV sets to generate multiple UV set bitstreams.

[0112] Furthermore, the base mesh encoding unit 313 may generate linking information that links the single geometry information with multiple UV sets, and store this linking information as metadata in each of the generated UV set bitstreams. By configuring the bitstream in this way, the linking between the geometry information bitstream and the UV set bitstream can be easily understood without requiring any other information.

[0113] Furthermore, the base mesh encoding unit 313 may generate table information showing the correspondence between the single geometry information and the multiple UV sets as linking information that links the single geometry information with the multiple UV sets. By doing so, it becomes unnecessary to store identification information of the geometry information (geometry information bitstream) within the UV set bitstream, and thus the reduction in encoding efficiency of the UV set bitstream can be suppressed.

[0114] Furthermore, the base mesh encoding unit 313 may encode a single geometry information and multiple UV sets associated with that single geometry information as the base mesh in the intra frame. By doing so, it is possible to suppress a reduction in attribute quality while suppressing a reduction in the encoding efficiency of the base mesh in the intra frame (static mesh).

[0115] <Encoding Process Flow> An example of the encoding process flow performed by this encoding device 300 will be explained with reference to the flowchart shown in Figure 17.

[0116] When the encoding process is started, the base mesh generation unit 321 of the encoding device 300 decimates the original mesh to be encoded in step S301 and generates a base mesh (geometry information).

[0117] In step S302, the UV set generation unit 322 UV unwraps the base mesh (geometry information) and generates a UV set. If there are multiple attributes that can be assigned to the same part of the base mesh (for example, if multiple attributes are assigned to a single face of the base mesh), the UV set generation unit 322 UV unwraps the geometry information for each attribute and generates a UV set corresponding to that geometry information. In other words, multiple UV sets are generated, each UV unwrapped in an independent pattern.

[0118] In step S303, the displacement vector generation unit 323 subdivides the base mesh and generates a displacement vector using the subdivided base mesh and the original mesh.

[0119] In step S304, the attribute conversion unit 324 converts the attribute map corresponding to the original mesh based on the base mesh and the original mesh, and generates an attribute map corresponding to the base mesh.

[0120] In step S305, the Atlas information encoding unit 312 encodes the Atlas information to generate an Atlas information bitstream.

[0121] In step S306, the base mesh encoding unit 313 performs base mesh encoding processing to encode a base mesh consisting of geometry information and UV sets corresponding to that geometry information. For example, the base mesh encoding unit 313 encodes a single piece of geometry information corresponding to at least one face of the base mesh, and a plurality of UV sets corresponding to that face and associated with that single piece of geometry information, as its base mesh. Through this base mesh encoding processing, the base mesh encoding unit 313 generates a geometry information bitstream and a UV set bitstream as encoded data for the base mesh.

[0122] In step S307, the displacement vector encoding unit 314 encodes the displacement vector to generate a displacement vector bitstream.

[0123] In step S308, the attribute encoding unit 315 encodes attributes (attribute maps corresponding to the base mesh) to generate an attribute bitstream. For example, the attribute encoding unit 315 encodes multiple attribute maps corresponding to multiple UV sets of the base mesh as attributes corresponding to the base mesh.

[0124] In step S309, the multiplexing unit 316 multiplexes the atlas information bitstream, the encoded data of the base mesh (geometry information bitstream and UV set bitstream), the displacement vector bitstream, and the attribute bitstream as substreams to generate a V-DMC bitstream.

[0125] The encoding process ends when the processing in step S309 is completed. The encoding device 300 performs this encoding process for each frame of the original mesh.

[0126] <Flow of Base Mesh Coding Process> Next, with reference to the flowchart in Figure 18, an example of the flow of the base mesh coding process performed in step S306 of Figure 17 will be explained.

[0127] When the base mesh coding process is started, the metadata generation unit 351 of the base mesh coding unit 313 generates metadata in step S351 using the base mesh (geometry vertex coordinates, geometry connectivity, UV vertex coordinates, and UV connectivity). For example, the metadata generation unit 351 generates linking information as metadata that links geometry information and UV sets corresponding to the same faces. For example, the metadata generation unit 351 generates this linking information using identification information of the geometry information. For example, the metadata generation unit 351 generates information indicating the identification information of the geometry information referenced by the UV set as linking information.

[0128] In step S352, the geometry connectivity encoding unit 352 encodes the geometry connectivity and generates encoded geometry connectivity data.

[0129] In step S353, the geometry vertex coordinate encoding unit 353 encodes the geometry vertex coordinates and generates encoded geometry vertex coordinate data. The geometry vertex coordinate encoding unit 353 then generates a single geometry information bitstream and stores the encoded geometry vertex coordinate data and the encoded geometry connectivity data in that geometry information bitstream. Furthermore, the geometry vertex coordinate encoding unit 353 stores the geometry information identification information as metadata in the generated geometry information bitstream.

[0130] In step S354, the base mesh coding unit 313 selects the UV set to be processed.

[0131] In step S355, among the UV connectivity encoding units 354-1 to 354-N, the UV connectivity encoding unit 354 that processes the selected UV set encodes the UV connectivity of the UV set to be processed and generates encoded UV connectivity data.

[0132] In step S356, among the UV vertex coordinate encoding units 355-1 to 355-N, the UV vertex coordinate encoding unit 355 that processes the selected UV set encodes the UV vertex coordinates of the UV set to be processed and generates encoded UV vertex coordinate data. The UV vertex coordinate encoding unit 355 generates a single UV set bitstream and stores the encoded UV vertex coordinate data and the encoded UV connectivity data in that UV set bitstream. Furthermore, the UV vertex coordinate encoding unit 355 stores the identification information of the referenced geometry information (reference submesh ID) as metadata in the generated UV set bitstream.

[0133] In step S357, the base mesh coding unit 313 determines whether all UV sets have been processed. If it is determined that there are unprocessed UV sets, the process returns to step S354, and the subsequent processing is performed for those unprocessed UV sets. In other words, each process from step S354 to step S357 is performed for each UV set. If it is determined in step S357 that all UV sets have been processed, the base mesh coding process ends, and the process returns to Figure 17.

[0134] By performing each process in this manner, the encoding device 300 (first information processing device) can suppress the redundancy of geometry connectivity and geometry vertex coordinates, thereby suppressing the reduction in the encoding efficiency of the base mesh.

[0135] <Decoding Device> This technology can be applied to a decoding device that decodes mesh encoded data. Figure 19 is a block diagram showing an example of the configuration of a decoding device, which is one embodiment of an information processing device to which this technology is applied. The decoding device 400 shown in Figure 19 is a device that decodes the mesh encoded data (V-DMC bitstream generated by the multiplexing unit 316) generated in the encoding device 300 (Figure 15) and reconstructs the decoded mesh. Note that Figure 19 shows the main configuration example when processing an intraframe.

[0136] Figure 19 shows the main components such as the processing unit and data flow, but it does not necessarily represent everything. In other words, the decoding device 400 may have processing units that are not shown as blocks in Figure 19, or processes and data flows that are not shown as arrows or other symbols in Figure 19.

[0137] The decoding device 400 decodes the encoded mesh data, which has been encoded in essentially the same way as the V-DMC described in the above-mentioned non-patent document, except for applying this technology, and reconstructs the decoded mesh. For example, the decoding device 400 acquires a V-DMC bitstream. This V-DMC bitstream may be generated, for example, by the encoding device 300. As part of the reconstruction process, the decoding device 400 decodes the V-DMC bitstream and reconstructs the mesh (also referred to as the decoded mesh). The decoding device 400 also applies a texture to the decoded mesh, generates a display image for displaying the decoded mesh, and outputs it to the outside of the decoding device 400. For example, the decoding device 400 supplies the display image to an external display device for display.

[0138] As shown in Figure 19, the decoding device 400 includes a demultiplexing unit 411, an atlas information decoding unit 412, a base mesh decoding unit 413, a subdivision unit 414, a displacement vector decoding unit 415, a displacement vector application unit 416, an attribute decoding unit 417, an attribute application unit 418, and a display processing unit 419.

[0139] The demultiplexing unit 411 performs processing related to demultiplexing. For example, the demultiplexing unit 411 may acquire the V-DMC bitstream to be decoded, which is supplied to the decoding device 400. The demultiplexing unit 411 may also demultiplex the acquired V-DMC bitstream and extract encoded data of atlas information (atlas information bitstream), encoded data of the base mesh (geometry information bitstream and UV set bitstream), encoded data of displacement vectors (displacement vector bitstream), and encoded data of attributes (attribute bitstream). Therefore, the demultiplexing unit 411 can also be said to be an acquisition unit for the V-DMC bitstream or the various information contained in the V-DMC bitstream. The demultiplexing unit 411 may supply the extracted encoded data of atlas information to the atlas information decoding unit 412. The demultiplexing unit 411 may also supply the extracted encoded data of the base mesh to the base mesh decoding unit 413. Furthermore, the demultiplexing unit 411 may supply the encoded data of the extracted displacement vector to the displacement vector decoding unit 415. Also, the demultiplexing unit 411 may supply the encoded data of the extracted attributes to the attribute decoding unit 417.

[0140] The atlas information decoding unit 412 performs processing related to decoding the atlas information. For example, the atlas information decoding unit 412 may acquire encoded data (atlas information bitstream) of the atlas information supplied from the demultiplexing unit 411. Alternatively, the atlas information decoding unit 412 may decode the acquired encoded data of the atlas information and generate (restore) the atlas information. The atlas information decoding unit 412 may supply the generated atlas information to one or more of the base mesh decoding unit 413, subdivision unit 414, displacement vector decoding unit 415, displacement vector application unit 416, attribute decoding unit 417, attribute application unit 418, and display processing unit 419.

[0141] The base mesh decoding unit 413 performs processing related to the decoding of the base mesh. For example, the base mesh decoding unit 413 may acquire encoded data of the base mesh (geometry information bitstream and UV set bitstream) supplied from the demultiplexing unit 411. Alternatively, the base mesh decoding unit 413 may decode the acquired encoded data of the base mesh using a predetermined decoding method (e.g., Draco) to generate (restore) the base mesh (geometry information and UV set). In this case, the base mesh decoding unit 413 may acquire atlas information supplied from the atlas information decoding unit 412 and decode the encoded data of the base mesh based on that atlas information. The base mesh decoding unit 413 may supply the generated base mesh to the subdivision unit 414.

[0142] The subdivision unit 414 performs processing related to the subdivision of the triangles of the base mesh. For example, the subdivision unit 414 may obtain the base mesh supplied from the base mesh decoding unit 413. The subdivision unit 414 may subdivide the base mesh (its triangles) and generate subdivision points. In this case, the subdivision unit 414 updates the geometry vertex coordinates, geometry connectivity, UV vertex coordinates, and UV connectivity, adding subdivision point information to each. At this time, the subdivision unit 414 may obtain atlas information supplied from the atlas information decoding unit 412 and subdivide the base mesh based on that atlas information. The subdivision unit 414 may supply the subdivided base mesh (geometry vertex coordinates, geometry connectivity, UV vertex coordinates, and UV connectivity with added subdivision point information) to the displacement vector application unit 416.

[0143] Furthermore, if multiple (e.g., N) attributes are assigned to the same part of the base mesh (e.g., a single face), the base mesh includes a UV set corresponding to each of those multiple (e.g., N) attributes. In other words, the base mesh contains a single piece of geometric information and multiple (e.g., N) UV sets for the same part. Therefore, in that case, the subdivision unit 414 may add subdivision point information to both the single piece of geometric information and the multiple (e.g., N) UV sets. The subdivision unit 414 may then supply the "subdivision base mesh," which includes the single piece of geometric information with the added subdivision point information and the multiple (e.g., N) UV sets, to the displacement vector application unit 416.

[0144] The displacement vector decoding unit 415 performs processing related to the decoding of the displacement vector. For example, the displacement vector decoding unit 415 may acquire encoded data of the displacement vector (displacement vector bitstream) supplied from the demultiplexing unit 411. The displacement vector decoding unit 415 may decode the encoded data of the displacement vector and generate (restore) the displacement vector. For example, if the displacement vector is encoded as a displacement video, the displacement vector decoding unit 415 may decode the encoded data of the displacement vector using a predetermined decoding method for 2D moving images to generate (restore) the displacement video, and then unpack the displacement vector from the displacement map, which is a frame image of the displacement video. Alternatively, if the displacement video is arithmetic encoded, the displacement vector decoding unit 415 may arithmetic decode the encoded data of the displacement vector and generate the displacement vector. In this case, the displacement vector decoding unit 415 may acquire atlas information supplied from the atlas information decoding unit 412 and decode the displacement vector based on that atlas information. The displacement vector decoding unit 415 may supply the displacement vector obtained in this way to the displacement vector application unit 416.

[0145] The displacement vector application unit 416 performs processing related to the application of displacement vectors to the subdivided base mesh. For example, the displacement vector application unit 416 may acquire the subdivided base mesh supplied from the subdivision unit 414. The displacement vector application unit 416 may acquire the displacement vector supplied from the displacement vector decoding unit 415. The displacement vector application unit 416 may generate a decoded mesh by applying the displacement vector to the vertices of the subdivided base mesh. In this case, the displacement vector application unit 416 applies the displacement vector to the geometry vertex coordinates. At that time, the displacement vector application unit 416 may acquire atlas information supplied from the atlas information decoding unit 412 and apply the displacement vector to the vertices of the subdivided base mesh based on that atlas information. The displacement vector application unit 416 may supply the decoded mesh thus generated to the attribute application unit 418.

[0146] The attribute decoding unit 417 performs processing related to attribute decoding. For example, the attribute decoding unit 417 may acquire the encoded attribute data (attribute bitstream) supplied from the demultiplexing unit 411. Alternatively, the attribute decoding unit 417 may decode the acquired encoded attribute data using a predetermined decoding method for 2D moving images to generate (restore) an attribute video. In this case, the attribute decoding unit 417 may acquire atlas information supplied from the atlas information decoding unit 412 and decode the encoded attribute data based on that atlas information. The attribute decoding unit 417 may supply the attribute map, which is a frame image of the generated attribute video, to the attribute application unit 418.

[0147] Furthermore, if multiple (e.g., N) attributes are assigned to the same part of the base mesh (e.g., a single face), the attribute decoding unit 417 is supplied with encoded data (attribute bitstreams) of those multiple (e.g., N) attributes. In this case, the attribute decoding unit 417 may decode the encoded data of each of the supplied multiple (e.g., N) attributes and generate (restore) an attribute video. In other words, the attribute decoding unit 417 may supply multiple (e.g., N) attribute maps to the attribute application unit 418.

[0148] The attribute application unit 418 performs processing related to the application of attributes to the decoded mesh. For example, the attribute application unit 418 may acquire the decoded mesh (geometry information and UV set) supplied from the displacement vector application unit 416. The attribute application unit 418 may acquire the attribute map supplied from the attribute decoding unit 417. The attribute application unit 418 may apply the attributes of the attribute map to the decoded mesh. In other words, the attribute application unit 418 may apply attributes to the faces of the decoded mesh in the three-dimensional space indicated by the supplied geometry information. In this case, the attribute application unit 418 may acquire atlas information supplied from the atlas information decoding unit 412 and apply attributes to the decoded mesh based on that atlas information. The attribute application unit 418 may supply the decoded mesh with the attributes applied in this way to the display processing unit 419.

[0149] Furthermore, if multiple (e.g., N) attributes are assigned to the same part of the base mesh (e.g., a single face), the attribute application unit 418 is supplied with an attribute map for each of those multiple (e.g., N) attributes. The attribute application unit 418 may apply those multiple (e.g., N) attribute maps to the decoded mesh. In that case, the decoded mesh supplied to the attribute application unit 418 has a UV set corresponding to each attribute (i.e., multiple (e.g., N) UV sets). The attribute application unit 418 may use those multiple (e.g., N) UV sets to apply the corresponding attributes to the faces of the decoded mesh in the three-dimensional space indicated by the geometry information. In other words, the attribute application unit 418 may supply the display processing unit 419 with a decoded mesh to which multiple attributes have been applied to a single face.

[0150] The display processing unit 419 performs processing related to the display of the mesh. For example, the display processing unit 419 may acquire a decoded mesh to which attributes have been applied, supplied from the attribute application unit 418. The display processing unit 419 may render the acquired decoded mesh from a desired viewpoint and generate a rendered image. The display processing unit may then generate a display image for displaying the rendered image (that is, a display image for displaying the decoded mesh). The display processing unit 419 may then supply the generated display image to an external device, such as another device, to display the display image.

[0151] Furthermore, if multiple attributes are applied to a single face in the decoded mesh, the display processing unit 419 may select which of those multiple attributes to apply to the rendered image (display image) during rendering. For example, the display processing unit 419 may select attributes according to the viewpoint during rendering.

[0152] <Base Mesh Decoding Unit> Figure 20 is a block diagram showing an example of the main configuration of the base mesh decoding unit 413 (Figure 19). Figure 20 shows the main processing units and data flows, but not all of them are shown in Figure 20. In other words, there may be processing units in the base mesh decoding unit 413 that are not shown as blocks in Figure 20, and there may be processes and data flows that are not shown as arrows, etc., in Figure 20.

[0153] As shown in Figure 20, the base mesh decoding unit 413 includes a geometry connectivity decoding unit 451, a geometry vertex coordinate decoding unit 452, a UV connectivity decoding unit 453-1, a UV vertex coordinate decoding unit 454-1, a UV connectivity decoding unit 453-2, a UV vertex coordinate decoding unit 454-2, ..., a UV connectivity decoding unit 453-N, and a UV vertex coordinate decoding unit 454-N. N is an arbitrary natural number indicating the maximum number of UV sets corresponding to one piece of geometry information in the base mesh. When it is not necessary to distinguish between the UV connectivity decoding units 453-1, 453-2, ..., and 453-N, they are also referred to as the UV connectivity decoding unit 453. When there is no need to distinguish between UV vertex coordinate decoding units 454-1, UV vertex coordinate decoding units 454-2, ..., UV vertex coordinate decoding units 454-N, they are also referred to as UV vertex coordinate decoding units 454. In other words, the base mesh decoding unit 413 has one geometry connectivity decoding unit 451 and one geometry vertex coordinate decoding unit 452, and further has N UV connectivity decoding units 453 and N UV vertex coordinate decoding units 454.

[0154] The geometry connectivity decoding unit 451 performs processing related to the decoding of geometry connectivity. For example, the geometry connectivity decoding unit 451 may acquire a geometry information bitstream supplied from the demultiplexing unit 411 (Figure 19) as encoded data for the base mesh. The geometry connectivity decoding unit 451 may decode the encoded data of geometry connectivity contained in the acquired geometry information bitstream and generate (restore) geometry connectivity. This decoding method can be any method that corresponds to the encoding method adopted in the encoded data of the geometry connectivity. For example, the geometry connectivity decoding unit 451 may decode the encoded data of geometry connectivity generated by the geometry connectivity encoding unit 352 (Figure 16) using a decoding method that corresponds to the encoding method used by the geometry connectivity encoding unit 352. The geometry connectivity decoding unit 451 may supply the generated geometry connectivity, along with the geometry information bitstream, to the geometry vertex coordinate decoding unit 452.

[0155] The geometry vertex coordinate decoding unit 452 performs processing related to the decoding of geometry vertex coordinates. For example, the geometry vertex coordinate decoding unit 452 may acquire the geometry information bitstream and geometry connectivity supplied from the geometry connectivity decoding unit 451. The geometry vertex coordinate decoding unit 452 may decode the encoded data of geometry vertex coordinates contained in the acquired geometry information bitstream and generate (restore) geometry connectivity. This decoding method can be any method that corresponds to the encoding method adopted in the encoded data of the geometry vertex coordinates. For example, the geometry vertex coordinate decoding unit 452 may decode the encoded data of geometry vertex coordinates generated by the geometry vertex coordinate encoding unit 353 (Figure 16) using a decoding method that corresponds to the encoding method used by the geometry vertex coordinate encoding unit 353. The geometry vertex coordinate decoding unit 452 may supply the generated geometry vertex coordinates together with the geometry connectivity to the UV connectivity decoding unit 453.

[0156] The UV connectivity decoding unit 453 performs processing related to the decoding of UV connectivity. For example, the UV connectivity decoding unit 453 may acquire geometry information (geometry connectivity and geometry vertex coordinates) supplied from the geometry vertex coordinate decoding unit 452. Alternatively, the UV connectivity decoding unit 453 may acquire a UV set bitstream supplied from the demultiplexing unit 411 (Figure 19) as encoded data for the base mesh. The UV connectivity decoding unit 453 may decode the encoded data of UV connectivity contained in the acquired UV set bitstream and generate (restore) UV connectivity. This decoding method can be any method that corresponds to the encoding method adopted in the encoded data of the UV connectivity. For example, as described above, UVSeams may be applied. In this case, the UV connectivity decoding unit 453 may use the acquired geometry connectivity to decode the encoded data of UV connectivity. The UV connectivity decoding unit 453 may supply the generated UV connectivity, along with geometry information (geometry connectivity and geometry vertex coordinates) and a UV set bitstream, to the UV vertex coordinate decoding unit 454.

[0157] The UV vertex coordinate decoding unit 454 performs processing related to the decoding of UV vertex coordinates. For example, the UV vertex coordinate decoding unit 454 may acquire geometry information (geometry connectivity and geometry vertex coordinates), UV connectivity, and a UV set bitstream supplied from the UV connectivity decoding unit 453. The UV vertex coordinate decoding unit 454 may decode the encoded data of UV vertex coordinates contained in the acquired UV set bitstream and generate (restore) UV vertex coordinates. This decoding method can be any method that corresponds to the encoding method adopted in the encoded data of the UV vertex coordinates. For example, the UV vertex coordinate decoding unit 454 may decode the encoded data of UV vertex coordinates using the acquired geometry information. The generated UV vertex coordinates, together with UV connectivity, form a UV set. The UV vertex coordinate decoding unit 454 may supply the UV set (UV connectivity and UV vertex coordinates) and geometry information (geometry connectivity and geometry vertex coordinates) as a base mesh to the subdivision unit 414.

[0158] As described above, the base mesh decoding unit 413 has N UV connectivity decoding units 453 and N UV vertex coordinate decoding units 454. Each pair of UV connectivity decoding units 453 and UV vertex coordinate decoding units 454 performs the above-described processing on the UV set corresponding to itself within a group of UV sets associated with the same geometry information. For example, the UV connectivity decoding unit 453-1 and the UV vertex coordinate decoding unit 454-1 perform decoding on the first UV set bitstream (ID=1) and generate the first UV set (ID=1). In other words, by combining that UV set with the geometry information, a base mesh (base mesh for Cam1) corresponding to the first attribute is generated. Similarly, the UV connectivity decoding unit 453-2 and the UV vertex coordinate decoding unit 454-2 perform decoding on the second UV set bitstream (ID=2) and generate the second UV set (ID=2). In other words, the UV set and geometry information are combined to generate a base mesh (base mesh for Cam2) corresponding to the second attribute. Similarly, the UV connectivity decoding unit 453-N and the UV vertex coordinate decoding unit 454-N perform decoding on the Nth UV set bitstream (ID=N) to generate the Nth UV set (ID=N). In other words, the UV set and geometry information are combined to generate a base mesh (base mesh for CamN) corresponding to the Nth attribute. Note that in Figure 20, the base meshes for Cam1, CamN, and base meshes corresponding to each attribute are shown individually, but the geometry information is shared among these base meshes. In other words, the base mesh decoding unit 413 generates a base mesh consisting of one piece of geometry information and up to N UV sets associated with that geometry information, and supplies it to the subdivision unit 414.

[0159] The decoding device 400 having the above configuration may be used as the second information processing device, and this technology may be applied to it.

[0160] For example, the decoding device 400 may include a base mesh decoding unit 413 that decodes a single geometry information bitstream and a plurality of UV set bitstreams associated with the single geometry information bitstream as a base mesh, and generates a single geometry information corresponding to at least one face of the base mesh and a plurality of UV sets corresponding to that face; and an attribute decoding unit 417 that decodes a plurality of attribute bitstreams corresponding to the plurality of UV set bitstreams as attributes corresponding to the base mesh, and generates a plurality of attribute maps corresponding to the plurality of UV sets.

[0161] Furthermore, as described above, the decoding device 400 may further include a subdivision unit 414 that subdivides the base mesh generated by decoding the single geometry information bitstream and the multiple UV set bitstreams.

[0162] Furthermore, as described above, the decoding device 400 may further include a displacement vector decoding unit 415 that decodes the displacement vector bitstream and generates a displacement vector which is the difference in vertex positions between the original mesh and the subdivided base mesh, and a displacement vector application unit 416 that applies the generated displacement vector to the vertices of the subdivided base mesh.

[0163] Furthermore, as described above, the attribute decoding unit 417 may decode the multiple attribute bitstreams to generate multiple attribute maps, and then unpack the multiple attribute maps to generate an attribute. The decoding device 400 may also further include an attribute application unit 418 that applies the generated attribute to a base mesh that has been subdivided and to which displacement vectors have been applied.

[0164] Furthermore, as described above, the decoding device 400 may further include a display processing unit 419 that renders a base mesh that has been subdivided and to which displacement vectors and attributes have been applied, and generates a display image.

[0165] Furthermore, as described above, the decoding device 400 may further include a demultiplexing unit 411 that demultiplexes the bitstream to extract a single geometry information bitstream, multiple UV set bitstreams, and multiple attribute bitstreams. The base mesh decoding unit 413 may then decode the extracted single geometry information bitstream and multiple UV set bitstreams as the base mesh. The attribute decoding unit 417 may also decode the extracted multiple attribute bitstreams.

[0166] With this configuration, the decoding device 400 (second information processing device) can suppress the redundancy of geometry connectivity and geometry vertex coordinates, thereby suppressing the reduction in the encoding efficiency of the base mesh.

[0167] The attributes generated (restored) by this decoding device 400 can be any kind of information, and may include, for example, textures.

[0168] Furthermore, the multiple UV sets generated (restored) by the base mesh decoding unit 413 may correspond to each of the multiple captured images applied as attributes of the base mesh. These multiple captured images may correspond to each of several different viewpoints. In this way, for example, a view-dependent texture can be realized while suppressing a reduction in the encoding efficiency of the base mesh.

[0169] Furthermore, the base mesh decoding unit 413 may decode the single geometry information bitstream and all of the multiple UV set bitstreams associated with that single geometry information bitstream. By doing so, for example, during rendering, it is possible to select a desired UV set from all the UV sets associated with the same geometry information, thereby suppressing a reduction in attribute quality while suppressing a reduction in the encoding efficiency of the base mesh.

[0170] The base mesh decoding unit 413 may decode the single geometry information bitstream and decode each of the multiple UV set bitstreams associated with that single geometry information bitstream. Each of the multiple UV set bitstreams may also contain encoded data for UV connectivity and encoded data for UV vertex coordinates. The base mesh decoding unit 413 may decode the encoded data for geometry connectivity and geometry vertex coordinates contained in the single geometry information bitstream and decode the encoded data for UV connectivity and encoded data for UV vertex coordinates for each of the multiple UV set bitstreams.

[0171] Furthermore, the base mesh decoding unit 413 may identify and decode the multiple UV set bitstreams associated with the decoded single geometry information bitstream based on the association information that links the single geometry information to the multiple UV sets, which is stored as metadata in each of the multiple UV set bitstreams. In this way, the association between the geometry information bitstream and the UV set bitstream can be easily understood without requiring any other information.

[0172] Furthermore, the base mesh decoding unit 413 may identify and decode multiple UV set bitstreams associated with the decoded single geometry information bitstream based on table information showing the correspondence between the single geometry information and multiple UV sets. By doing so, it becomes unnecessary to store identification information of the geometry information (geometry information bitstream) within the UV set bitstream, thereby suppressing the reduction in encoding efficiency of the UV set bitstream.

[0173] Furthermore, the base mesh decoding unit 413 may decode the single geometry information bitstream and the multiple UV set bitstreams associated with that single geometry information bitstream as the base mesh in the intra frame. By doing so, it is possible to suppress a reduction in attribute quality while suppressing a reduction in the encoding efficiency of the base mesh in the intra frame (static mesh).

[0174] <Decryption Process Flow> An example of the decoding process flow performed by this decoding device 400 will be explained with reference to the flowchart shown in Figure 21.

[0175] When the decoding process is started, the demultiplexing unit 411 of the decoding device 400 demultiplexes the V-DMC bitstream in step S401 and extracts the atlas information bitstream, the encoded data of the base mesh (geometry information bitstream and UV set bitstream), the displacement vector bitstream, and the attribute bitstream.

[0176] In step S402, the Atlas information decoding unit 412 decodes the encoded data of the Atlas information (Atlas information bitstream) and generates (restores) the Atlas information.

[0177] In step S403, the base mesh decoding unit 413 performs base mesh decoding processing, decodes the encoded data of the base mesh, and generates (restores) the base mesh. In other words, the base mesh decoding unit 413 decodes the geometry information bitstream and the UV set bitstream, respectively, and generates (restores) the geometry information and UV set. For example, the base mesh decoding unit 413 decodes a single geometry information bitstream and multiple UV set bitstreams associated with that single geometry information bitstream as a base mesh, and generates single geometry information corresponding to at least one face of the base mesh and multiple UV sets corresponding to that face.

[0178] In step S404, the subdivision unit 414 subdivides the base mesh. That is, the subdivision unit 414 adds subdivision point information to the geometry information and UV set.

[0179] In step S405, the displacement vector decoding unit 415 decodes the encoded data of the displacement vector (displacement vector bitstream) and generates (restores) the displacement vector.

[0180] In step S406, the displacement vector application unit 416 applies the displacement vector to the vertices of the subdivided base mesh to generate the decoded mesh. In other words, the displacement vector application unit 416 generates the decoded mesh by updating the geometric vertex coordinates of the subdivided base mesh based on the displacement vector.

[0181] In step S407, the attribute decoding unit 417 decodes the encoded attribute data (attribute bitstream) and generates (restores) an attribute map. If there are multiple attributes that can be assigned to the same part of the base mesh, and multiple attribute bitstreams are extracted from the V-DMC bitstream, the attribute decoding unit 417 decodes each attribute bitstream and generates (restores) multiple attribute maps. For example, the attribute decoding unit 417 decodes multiple attribute bitstreams corresponding to multiple UV set bitstreams as attributes corresponding to the base mesh, and generates multiple attribute maps corresponding to each of those multiple UV sets.

[0182] In step S408, the attribute application unit 418 applies the generated attributes to the decoded mesh. If multiple attribute maps are generated, the attribute application unit 418 applies the attributes of each attribute map to the decoded mesh.

[0183] In step S409, the display processing unit 419 renders the decoded mesh to which the attributes have been applied to generate a display image.

[0184] When the process in step S409 is completed, the decoding process is finished. The decoding device 400 performs this decoding process for each frame of the original mesh.

[0185] <Base Mesh Decoding Process Flow> Next, with reference to the flowchart in Figure 22, an example of the base mesh decoding process flow performed in step S403 of Figure 21 will be explained.

[0186] When the base mesh decoding process is started, the geometry connectivity decoding unit 451 of the base mesh decoding unit 413 decodes the encoded data of geometry connectivity included in the geometry information bitstream in step S451 and generates (restores) geometry connectivity.

[0187] In step S452, the geometry vertex coordinate decoding unit 452 decodes the encoded data of geometry vertex coordinates contained in the geometry information bitstream and generates (restores) geometry vertex coordinates.

[0188] In step S453, the base mesh decoding unit 413 selects the UV set to be processed. If there are multiple attributes that can be assigned to the same part of the base mesh, multiple UV set bitstreams are associated with a single geometry information bitstream. The base mesh decoding unit 413 selects the UV set bitstream to be decoded from among these multiple UV set bitstreams. For example, the base mesh decoding unit 413 selects the UV set bitstream that stores the identification information of the decoded geometry information and makes it the target of processing.

[0189] In step S454, among the UV connectivity decoding units 453-1 to 453-N, the UV connectivity decoding unit 453 that processes the selected UV set decodes the encoded data of the UV connectivity of the UV set to be processed and generates (restores) the UV connectivity.

[0190] In step S455, among the UV vertex coordinate decoding units 454-1 to 454-N, the UV vertex coordinate decoding unit 454 that processes the selected UV set decodes the encoded data of the UV vertex coordinates of the UV set to be processed and generates (restores) the UV vertex coordinates.

[0191] In step S456, the base mesh decoding unit 413 determines whether all UV sets have been processed. If it is determined that there are unprocessed UV sets, the process returns to step S453, and the subsequent processing is performed for those unprocessed UV sets. In other words, each process from step S453 to step S456 is performed for each UV set. If it is determined in step S456 that all UV sets have been processed, the base mesh decoding process ends, and the process returns to Figure 21.

[0192] By performing each process in this manner, the decoding device 400 (second information processing device) can suppress the redundancy of geometry connectivity and geometry vertex coordinates, thereby suppressing the reduction in the encoding efficiency of the base mesh.

[0193] <Method 1-1> Alternatively, as shown in the second row from the top of the table in Figure 11, UV face skip information may be transmitted (Method 1-1). For example, in the first information processing device, the base mesh coding unit may generate skip information relating to faces that skip the coding of at least one of the plurality of UV sets. For example, in the coding device 300, the base mesh coding unit 313 may generate skip information relating to skipping the coding of at least one of the plurality of UV sets. Alternatively, in the second information processing device, the base mesh decoding unit may decode the plurality of UV set bitstreams based on the skip information indicating whether to skip the coding of at least one of the plurality of UV sets. For example, in the decoding device 400, the base mesh decoding unit 413 may decode at least one of the plurality of UV set bitstreams based on the skip information relating to faces that skip the coding of at least one of the plurality of UV sets. For example, the base mesh encoding unit 313 may generate skip information for a face that skips the encoding of at least one of the corresponding UV sets, and the base mesh decoding unit 413 may skip decoding the encoding of at least one of the UV sets corresponding to the face indicated by the skip information.

[0194] By transmitting this skip information, it is possible to easily control the skipping of UV set encoding and decoding on a face-by-face basis. Furthermore, this control allows for the skipping of unnecessary UV set encoding and decoding, thereby suppressing an increase in the load on encoding and decoding processes.

[0195] This skip information may include a list of UV faces corresponding to faces whose encoding will be skipped for at least one of the multiple UV sets. In other words, in this case, the encoding and decoding of the UV sets of the faces presented in this list will be skipped. Such a list may be generated for each UV set as information corresponding to that UV set. That is, for a single UV set, a list of UV faces whose encoding will be skipped for that UV set may be generated. Such skip information for each UV set may be generated for all UV sets, or for some UV sets. By using such a list as skip information, it is easy to determine which UV faces (UV-unwrapped faces) have their encoding and decoding skipped.

[0196] Furthermore, this skip information may be stored in the UV set bitstream. For example, in the first information processing device, the base mesh encoding unit may store the generated skip information in a plurality of UV set bitstreams generated by encoding the plurality of UV sets. For example, in the encoding device 300, the base mesh encoding unit 313 may store the generated skip information in the UV set bitstream. Also, for example, in the second information processing device, the base mesh decoding unit may decode the encoded data of each UV face included in the UV set bitstream based on the skip information stored in the UV set bitstream. For example, in the decoding device 400, the base mesh decoding unit 413 may decode the encoded data of each UV face included in the UV set bitstream based on the skip information stored in the UV set bitstream. By storing and transmitting the skip information in the UV set bitstream in this way, the decoding side can easily select whether or not to skip decoding the UV set of each face in the UV set bitstream without needing to refer to information other than the UV bitstream.

[0197] For example, as shown in Figure 23, a list of faces whose encoding or decoding of UV sets is skipped (UV face skip list) may be stored as skip information in the UV set bitstream and transmitted. In the example in Figure 23, the geometry information bitstream 511 is associated with the UV set bitstream 512, UV set bitstream 513, and UV set bitstream 514 (see submesh ID = i). The UV face skip list is stored as metadata in the UV set bitstream 512, UV set bitstream 513, and UV set bitstream 514, respectively. This UV face skip list is a list of faces (UV faces) whose encoding or decoding of UV sets is skipped. In other words, the encoding or decoding of the UV sets of faces included in this UV face skip list is skipped. Note that this UV face skip list may be information specific to each UV set. That is, this UV face skip list may be a list of UV faces whose encoding was skipped in the UV set bitstream that stores the UV face skip list. Note that such a UV face skip list may be generated for all UV sets, or for some UV sets.

[0198] Alternatively, instead of this list, a list of faces that will not be skipped during UV set encoding or decoding may be stored as skip information in the UV set bitstream and transmitted. Furthermore, for each face, a list indicating whether or not to skip UV set encoding or decoding may be stored as skip information in the UV set bitstream and transmitted.

[0199] Furthermore, generally speaking, if encoding of a certain UV set is skipped for a given face, the transmission of the encoded data for that UV set is also skipped. In other words, the UV set bitstream for a given UV set does not contain the encoded data for the faces whose encoding was skipped. Therefore, the list described above can also be considered a list of faces whose encoded data is not stored in the UV set bitstream. Alternatively, instead of this list, a list of faces whose encoded data is stored in the UV set bitstream may be stored in the UV set bitstream as skip information and transmitted. Alternatively, for each face, a list indicating whether or not (or whether or not) the encoded data is stored in the UV set bitstream may be stored in the UV set bitstream as skip information and transmitted.

[0200] Furthermore, as shown in the example in Figure 23, if there are multiple UV set bitstreams, skip information for that UV set bitstream may be stored in each UV set bitstream. In other words, the skip information may be configured as information for each UV set, and whether or not to skip the encoding or decoding of each face may be controlled independently for each UV set. Note that the skip information (UV face skip list) may be stored in some of the UV set bitstreams among the multiple UV set bitstreams. In that case, the absence of the skip information (UV face skip list) in a UV set bitstream may indicate that the encoded data for all faces is stored in that UV set bitstream.

[0201] <Method 1-2> Furthermore, if the UV connectivity perfectly matches the geometry connectivity, the geometry connectivity can be reused as the UV connectivity. In that case, the encoding and decoding of the UV connectivity may be skipped. To realize such control, skip information for UV connectivity transmission may be transmitted, as shown in the third row from the top of the table in Figure 11 (Method 1-2). For example, in the first information processing device, the base mesh encoding unit may generate skip information regarding the skipping of UV connectivity encoding. For example, in the encoding device 300, the base mesh encoding unit 313 may generate skip information regarding the skipping of UV connectivity encoding. Also, in the second information processing device, the base mesh decoding unit may decode the encoded data of the UV connectivity based on the skip information regarding the skipping of UV connectivity encoding. For example, in the decoding device 400, the base mesh decoding unit 413 may decode the encoded data of the UV connectivity based on skip information regarding the skipping of encoding of the UV connectivity.

[0202] For example, the base mesh encoding unit 313 may skip encoding UV connectivity that perfectly matches geometry connectivity, generate skip information associated with the UV set indicating that UV connectivity encoding has been skipped, and when the base mesh decoding unit 413 decodes the UV set, it may skip decoding the encoded data of the UV connectivity indicated by the skip information.

[0203] By transmitting such skip information, it is possible to easily control the skipping of encoding and decoding of geometry connectivity. Furthermore, by realizing such control, it is possible to skip the encoding and decoding of unnecessary UV connectivity, thereby suppressing the increase in the load on encoding and decoding processes.

[0204] Furthermore, this skip information may include flag information indicating whether or not to skip the encoding of UV connectivity. That is, if the value of this flag information is true, the encoding or decoding of the UV connectivity for the UV set corresponding to that flag information will be skipped. Conversely, if the value of this flag information is false, the encoding or decoding of the UV connectivity for the UV set corresponding to that flag information will be performed. By using such flag information as skip information, it is easy to determine whether or not to skip the encoding or decoding of UV connectivity.

[0205] Furthermore, this skip information may be stored in the UV set bitstream of the UV set to which the skip information corresponds. For example, in the first information processing device, the base mesh encoding unit may store the generated skip information in a plurality of UV set bitstreams generated by encoding the plurality of UV sets. For example, in the encoding device 300, the base mesh encoding unit 313 may store the generated skip information in a plurality of UV set bitstreams generated by encoding the plurality of UV sets to which the skip information corresponds. Also, for example, in the second information processing device, the base mesh decoding unit may decode the encoded UV connectivity data stored in each of the plurality of UV set bitstreams based on the skip information stored in each of the plurality of UV set bitstreams. For example, in the decoding device 400, the base mesh decoding unit 413 may decode the encoded UV connectivity data stored in the UV set bitstream based on the skip information stored in the UV set bitstream. By storing and transmitting the skip information in the UV set bitstream in this way, the decoding side can easily select whether or not to skip decoding the UV connectivity of that UV set bitstream without needing to refer to information other than the UV bitstream.

[0206] Furthermore, if the decoding of UV connectivity is skipped on the decoding side, geometry connectivity may be reused. For example, in the second information processing device, the base mesh decoding unit may reuse the geometry connectivity generated by decoding the single geometry information bitstream as the UV connectivity whose decoding was skipped. For example, in the decoding device 400, the base mesh decoding unit 413 may reuse the geometry connectivity generated by decoding the geometry information bitstream corresponding to the UV set bitstream as the UV connectivity whose decoding was skipped.

[0207] For example, as shown in Figure 24, flag information (skip_flag) indicating whether or not to skip the encoding or decoding of UV connectivity may be stored as skip information in the UV set bitstream and transmitted. In the example in Figure 24, UV set bitstreams 522, 523, and 524 are associated with geometry information bitstream 521 (see submesh ID = i). Each of the UV set bitstreams 522, 523, and 524 stores skip_flag as metadata. This skip_flag is flag information indicating whether or not to skip the encoding or decoding of UV connectivity. If its value is true, it indicates that the encoding or decoding of the UV connectivity for that UV set will be skipped, and if its value is false, it indicates that the encoding or decoding of the UV connectivity for that UV set will be performed. Therefore, if this skip_flag is true, geometry connectivity may be reused as UV connectivity.

[0208] Generally, if UV connectivity encoding is skipped, the transmission of the encoded data for that UV connectivity is also skipped. In other words, the encoded data for that UV connectivity is not stored in the UV set bitstream. Therefore, the flag information mentioned above can be said to be flag information indicating whether (or whether) the encoded data for UV connectivity is stored in the UV set bitstream.

[0209] Furthermore, as shown in the example in Figure 24 (skip_flag), if there are multiple UV set bitstreams, skip information for that UV set bitstream may be stored in each UV set bitstream. In other words, the skip information may be configured as information for each UV set, and whether or not to skip the encoding or decoding of UV connectivity may be controlled for each UV set. Note that the skip information (skip_flag) may be stored in some of the UV set bitstreams among the multiple UV set bitstreams. In that case, the absence of the skip information (skip_flag) in a UV set bitstream may indicate that the skip information (skip_flag) is false. In other words, the absence of the skip information (skip_flag) in a UV set bitstream may indicate that the encoding and decoding of UV connectivity for that UV set will not be skipped. To put it another way, the absence of the skip information (skip_flag) in a UV set bitstream may indicate that the encoded data for UV connectivity will be stored in that UV set bitstream.

[0210] <Method 1-3> Alternatively, as shown in the fourth row from the top of the table in Figure 11, UV sets for each attribute hierarchy may be transmitted (Method 1-3). For example, multiple UV sets corresponding to the same face as a single piece of geometry information may correspond to each hierarchy of the hierarchical attributes.

[0211] <Method 1-4> Alternatively, as shown in the fifth row from the top of the table in Figure 11, an encoding method may be set for each UV set (Method 1-4). For example, in the first information processing device, the base mesh encoding unit may encode multiple UV sets by applying the parameters set for each set. For example, in the encoding device 300, the base mesh encoding unit 313 may encode multiple UV sets by applying the parameters set for each set. Alternatively, in the second information processing device, the base mesh decoding unit may decode multiple UV set bitstreams by applying the parameters related to the encoding of the UV set. For example, in the decoding device 400, the base mesh decoding unit 413 may decode multiple UV set bitstreams by applying the parameters related to the encoding of the UV set.

[0212] This parameter can be anything. For example, it may include quantized values ​​of UV vertex coordinates, or it may include prediction modes for multiple sets of UVs.

[0213] Furthermore, setting information related to the encoding of UV sets, such as the value of this parameter, may also be transmitted. For example, in the first information processing device, the base mesh encoding unit may store this parameter in a plurality of UV set bitstreams generated by encoding the plurality of UV sets. For example, in the encoding device 300, the base mesh encoding unit 313 may store this parameter in a plurality of UV set bitstreams generated by encoding the plurality of UV sets. Also, in the second information processing device, the base mesh decoding unit may decode the plurality of UV set bitstreams by applying the parameters related to the encoding of the plurality of UV sets stored in the plurality of UV set bitstreams. For example, in the decoding device 400, the base mesh decoding unit 413 may decode the plurality of UV set bitstreams by applying the parameters related to the encoding of UV sets stored in the UV set bitstreams.

[0214] <Method 1-5> Alternatively, as shown in the sixth row from the top of the table in Figure 11, the method may be applied to vertex attributes (Method 1-5). That is, the attribute may include vertex attributes that relate to the vertices of the base mesh. For example, the vertex attribute may include the UV coordinates of the vertex. The vertex attribute may also include the color information of the vertex. The vertex attribute may also include the normal vector of the vertex.

[0215] <Method 1-6> Alternatively, as shown in the seventh row from the top of the table in Figure 11, it may be applied to an interframe (Method 1-6). For example, in the first information processing device, the base mesh encoding unit may encode the single geometry information and the multiple UV sets associated with that single geometry information as the base mesh in the interframe. For example, in the encoding device 300, the base mesh encoding unit 313 may encode the single geometry information and the multiple UV sets associated with that single geometry information as the base mesh in the interframe. Alternatively, in the second information processing device, the base mesh decoding unit may decode the single geometry information bitstream and the multiple UV set bitstreams associated with that single geometry information bitstream as the base mesh in the interframe. For example, in the decoding device 400, the base mesh decoding unit 413 may decode the geometry information bitstream and the multiple UV set bitstreams associated with that geometry information bitstream as the base mesh in the interframe.

[0216] In the case of interframes, where encoding and decoding are performed by referencing information from other frames (also called keyframes), the geometry information includes reference information indicating the keyframe being referenced, as shown in the example in Figure 25. For example, in Figure 25, the geometry information bitstream 531 is the information of an intraframe (frame ID = 0), and the geometry information bitstream 532, UV set bitstream 533, and UV set bitstream 534 are the information of an interframe (frame ID = 1). Furthermore, the encoding and decoding of this interframe (frame ID = 1) are performed by referencing the information of an intraframe (frame ID = 0). In other words, the geometry information bitstream 532 is a bitstream in which the geometry information of the interframe (frame ID = 1) is encoded by referencing the geometry information of the intraframe (frame ID = 0). Therefore, this geometry information bitstream 532 stores reference information (ref frame ID) indicating that the intraframe (frame ID = 0) is the reference target (i.e., the keyframe). Furthermore, this geometry information bitstream 532 stores geometry vertex motion information (mv1, mv2, mv3) instead of geometry vertex coordinates. This geometry vertex motion information indicates the difference in geometry vertex coordinates between the current frame (i.e., intraframe (frame ID = 0)) and its keyframe (i.e., interframe (frame ID = 1)). Note that geometry connectivity is the same as keyframes and can therefore be omitted.

[0217] Note that the UV set is the same in both interframe and intraframe cases. Therefore, the UV set bitstream 533 and UV set bitstream 534 store association information (reference submesh ID) indicating that they are associated with the geometry information bitstream 532.

[0218] <Method 1-7> Alternatively, as shown in the bottom row of the table in Figure 11, the UV set required for rendering may be selected and transmitted (Method 1-7). For example, Method 1-7 may be applied in a distribution system that encodes 3D data and distributes it from a server to a terminal device.

[0219] <Distribution System> Figure 26 shows an example of the main configuration of a distribution system to which this technology is applied. The distribution system 600 shown in Figure 26 has a server 611 and a terminal device 612, and the server 611 distributes 3D data (original mesh and attribute map) as a V-DMC bitstream to the terminal device 612.

[0220] Server 611 performs processing related to the distribution of 3D data (original mesh and attribute map). For example, Server 611 may receive a request from Terminal Device 612. In response to the request, Server 611 may encode the 3D data (original mesh and attribute map) to generate a V-DMC bitstream. Server 611 may then supply the V-DMC bitstream to Terminal Device 612. Terminal Device 612 performs processing related to the acquisition and playback of the distributed 3D data. For example, Terminal Device 612 may request Server 611 to distribute 3D data. In response, Terminal Device 612 may receive the V-DMC bitstream supplied by Server 611. Terminal Device 612 may then decode the V-DMC bitstream to generate (restore) 3D data and render it to generate a display image.

[0221] Server 611 and terminal device 612 are connected via network 610 to enable communication. In other words, communication (requests, responses, etc.) between server 611 and terminal device 612 takes place via this network 610. For example, the V-DMC bitstream distributed from server 611 is transmitted to terminal device 612 via this network 610.

[0222] Network 610 is a communication network composed of any communication medium. Communication conducted through Network 610 may be wired communication, wireless communication, or both. In other words, Network 610 may be a communication network for wired communication, a communication network for wireless communication, or a communication network composed of both. Furthermore, Network 610 may be composed of a single communication network or of multiple communication networks.

[0223] For example, the internet may be included in this network 610. Public telephone networks may also be included in this network 610. Furthermore, wide-area communication networks for wireless mobile devices, such as so-called 3G and 4G networks, may also be included in this network 610. For example, LPWA (Low Power Wide Area) communication networks such as LTE-M, which enable long-distance data communication and have low power consumption, may also be included in this network 610. Furthermore, WANs (Wide Area Networks) and LANs (Local Area Networks) may also be included in this network 610. Furthermore, wireless communication networks that perform communication compliant with the Bluetooth® standard may also be included in this network 610. Near-field communication (NFC) communication channels may also be included in this network 610. Furthermore, infrared communication channels may also be included in this network 610. Finally, wired communication networks compliant with standards such as HDMI (High-Definition Multimedia Interface)® and USB (Universal Serial Bus)® may also be included in this network 610. Thus, the network 610 may include communication networks and communication channels of any communication standard. Furthermore, these communication networks and communication channels may include not only communication media such as cables, but also devices and circuits necessary for communication, such as communication equipment and relay equipment.

[0224] In Figure 26, one server 611 and one terminal device 612 are shown as an example configuration of the distribution system 600, but the number of servers 611 and terminal devices 612 can be any number. For example, the distribution system 600 may have multiple servers 611 or multiple terminal devices 612. The number of servers 611 and terminal devices 612 may be the same or different. Also, if the distribution system 600 has multiple servers 611, the specifications (functions and performance) of each server 611 may be the same or different. Also, if the distribution system 600 has multiple terminal devices 612, the specifications (functions and performance) of each terminal device 612 may be the same or different. Furthermore, the distribution system 600 may have devices other than servers 611 and terminal devices 612, and these devices may be of any kind. Also, the distribution system 600 may have multiple networks 610.

[0225] In such a distribution system 600, the above-described technology (Method 1-7) may be applied. For example, terminal device 612 supplies viewpoint information indicating the rendering viewpoint to server 611 via network 610. The 3D data distributed by server 611 consists of an original mesh and attribute maps. In this 3D data, multiple attribute maps are assigned to the same part of the original mesh (for example, a single face). Server 611 selects UV sets and attribute maps according to the rendering viewpoint indicated by the viewpoint information supplied from terminal device 612. In other words, server 611 selects the information necessary for rendering from the 3D data. Server 611 then encodes the original mesh and its selected attribute maps to generate a V-DMC bitstream. Server 611 then supplies the generated V-DMC bitstream to terminal device 612. In other words, the server 611 selects the attributes appropriate for the viewpoint requested by the terminal device 612 (i.e., the attributes necessary for rendering that viewpoint) and delivers 3D data (V-DMC bitstream) containing those attributes.

[0226] By doing so, the distribution system 600 can achieve distribution with view-dependent textures applied. Furthermore, since only the attributes (and UV sets) necessary for rendering are selected and distributed, a reduction in the encoding efficiency of the V-DMC bitstream can be suppressed. For example, an increase in the amount of data in the V-DMC bitstream distributed by this distribution system 600 can be suppressed. Therefore, an increase in the communication bandwidth required for distribution can be suppressed. In addition, an increase in the decoding load of the V-DMC bitstream distributed by this distribution system 600 can be suppressed. Moreover, if there is an upper limit on the amount of data in the V-DMC bitstream distributed by this distribution system 600, a reduction in the quality of the original mesh and attribute map can be suppressed by suppressing the increase in the amount of encoding due to unnecessary information. Therefore, a reduction in the quality of the rendered image can be suppressed.

[0227] <Server> Figure 27 is a block diagram showing an example of the main configuration of server 611. Figure 27 shows the main components such as processing units and data flows, and does not necessarily represent everything. In other words, server 611 may have processing units that are not shown as blocks in Figure 27, or processes and data flows that are not shown as arrows or other symbols in Figure 27.

[0228] As shown in Figure 27, the server 611 has, in addition to the configuration of the encoding device 300 (Figure 15), a communication unit 631 and a UV set selection unit 632.

[0229] The communication unit 631 performs communication-related processing. The communication unit 631 includes a viewpoint information acquisition unit 641 and a V-DMC bitstream supply unit 642.

[0230] The viewpoint information acquisition unit 641 performs processing related to acquiring viewpoint information for rendering. For example, the viewpoint information acquisition unit 641 may acquire viewpoint information supplied from the terminal device 612 via the network 610. The viewpoint information acquisition unit 641 may also supply the acquired viewpoint information to the UV set selection unit 632.

[0231] In the case of server 611, the multiplexing unit 316 may supply the generated V-DMC bitstream to the V-DMC bitstream supply unit 642 of the communication unit 631. Other processing of the multiplexing unit 316 is the same as in the case of encoding device 300. The V-DMC bitstream supply unit 642 performs processing related to the supply of V-DMC bitstreams. For example, the V-DMC bitstream supply unit 642 may acquire the V-DMC bitstream supplied from the multiplexing unit 316. The V-DMC bitstream supply unit 642 may supply the acquired V-DMC bitstream to terminal device 612 via the network 610 at a predetermined timing or triggered by a predetermined event. Therefore, the V-DMC bitstream supply unit 642 can also be said to be a supply unit that supplies V-DMC bitstreams.

[0232] In the case of server 611, the preprocessing unit 311 further includes a UV set selection unit 632 in addition to the configuration in the case of encoding device 300 (Figure 15). The UV set generation unit 322 may supply the base mesh (geometry information and UV set), original mesh, and atlas information to the UV set selection unit 632. Other processing of the UV set generation unit 322 is the same as in the case of encoding device 300.

[0233] The UV set selection unit 632 performs processing related to the selection of UV sets. For example, the UV set selection unit 632 may acquire viewpoint information supplied from the viewpoint information acquisition unit 641. The UV set selection unit 632 may acquire the base mesh (geometry information and UV set), original mesh, and atlas information supplied from the UV set generation unit 322. As shown in the example in Figure 5, the geometry information of this base mesh can be UV unwrapped in multiple patterns for the same part (for example, a single face). In other words, there may be multiple UV sets corresponding to a single piece of geometry information. In this case, where there are multiple UV sets corresponding to a single piece of geometry information, the UV set selection unit 632 may select the UV set necessary for rendering the viewpoint indicated by the viewpoint information from the group of UV sets (i.e., multiple UV sets associated with a single piece of geometry information). If there is only one UV set, it is sufficient to select that UV set. The UV set selection unit 632 may supply the atlas information to the atlas information encoding unit 312. The UV set selection unit 632 may supply the geometry information and the selected UV set as the base mesh encoding unit 313. The UV set selection unit 632 may supply the base mesh (geometry information and the selected UV set) along with the original mesh to the displacement vector generation unit 323 and the attribute conversion unit 324.

[0234] In the case of server 611, the atlas information encoding unit 312 may acquire atlas information supplied from UV set selection unit 632. Other processing of the atlas information encoding unit 312 is the same as in the case of encoding device 300.

[0235] Furthermore, the base mesh encoding unit 313 may acquire the base mesh (geometry information and the UV set selected by the UV set selection unit 632) supplied from the UV set selection unit 632. Other processing of the base mesh encoding unit 313 is the same as in the case of the encoding device 300. However, the UV set supplied to the attribute conversion unit 324 is only the UV set selected by the UV set selection unit 632 (the UV set required for rendering). Therefore, as described above, even if the UV set generation unit 322 generates multiple UV sets corresponding to a single piece of geometry information, the base mesh encoding unit 313 may encode that single piece of geometry information and the UV set selected by the UV set selection unit 632 as the base mesh.

[0236] The displacement vector generation unit 323 may also acquire the base mesh (geometry information and the UV set selected by the UV set selection unit 632) supplied from the UV set selection unit 632 and the original mesh. Other processing of the displacement vector generation unit 323 is basically the same as in the case of the encoding device 300.

[0237] The attribute conversion unit 324 may acquire the base mesh (geometry information and the UV set selected by the UV set selection unit 632) supplied from the UV set selection unit 632 and the original mesh. Other processing of the attribute conversion unit 324 is basically the same as in the case of the encoding device 300. However, the UV set supplied to the attribute conversion unit 324 is only the UV set selected by the UV set selection unit 632 (the UV set necessary for rendering). Therefore, the attribute conversion unit 324 may generate only the attribute map corresponding to this selected UV set. In other words, the attribute conversion unit 324 can be said to select the attribute map necessary for rendering. The attribute conversion unit 324 may supply the attribute map generated (selected) in this way (the attribute map corresponding to the base mesh) to the attribute encoding unit 315.

[0238] <Terminal Device> Figure 28 is a block diagram showing an example of the main configuration of the terminal device 612. Figure 28 shows the main components such as the processing unit and data flow, but it does not necessarily show everything. In other words, the terminal device 612 may have processing units that are not shown as blocks in Figure 28, or processes and data flows that are not shown as arrows etc. in Figure 28.

[0239] As shown in Figure 28, the terminal device 612 has a communication unit 661 in addition to the configuration of the decoding device 400 (Figure 19). The communication unit 661 performs communication-related processing. The communication unit 661 has a viewpoint information supply unit 671 and a V-DMC bitstream acquisition unit 672.

[0240] In the case of terminal device 612, the display processing unit 419 supplies viewpoint information related to the rendering viewpoint to the viewpoint information supply unit 671. The viewpoint information supply unit 671 performs processing related to the supply of viewpoint information. For example, the viewpoint information supply unit 671 may acquire viewpoint information supplied from the display processing unit 419. The viewpoint information supply unit 671 may supply the acquired viewpoint information to the server 611 via the network 610 at a predetermined timing or triggered by a predetermined event.

[0241] The V-DMC bitstream acquisition unit 672 may acquire a V-DMC bitstream supplied from the server 611 via the network 610. In this V-DMC bitstream, the base mesh is composed of UV sets selected based on geometry information and viewpoint information, as described above. Similarly, attributes are selected based on viewpoint information. In other words, this V-DMC bitstream is composed only of information necessary for rendering, with information unnecessary for rendering removed. To put it another way, the V-DMC bitstream acquisition unit 672 can be said to be an acquisition unit that acquires a single geometry information bitstream, a UV set bitstream linked to that single geometry information bitstream and selected from among multiple UV set bitstreams based on viewpoint information supplied by the viewpoint information supply unit 671, and an attribute bitstream corresponding to the selected UV set bitstream. The V-DMC bitstream acquisition unit 672 may supply the acquired V-DMC bitstream to the demultiplexing unit 411.

[0242] The demultiplexing unit 411 may acquire the V-DMC bitstream supplied from the V-DMC bitstream acquisition unit 672. Other processing of the demultiplexing unit 411 is the same as in the case of the decoding device 400.

[0243] In other words, the base mesh decoding unit 413 can be said to decode the single geometry information bitstream and UV set bitstream acquired by the V-DMC bitstream acquisition unit 672. Similarly, the attribute decoding unit 417 can be said to decode the attribute bitstream acquired by the V-DMC bitstream acquisition unit 672.

[0244] By configuring the system as described above, the distribution system 600 can achieve distribution with view-dependent textures applied. Furthermore, since only the attributes (and UV sets) necessary for rendering are selected and distributed, a reduction in the encoding efficiency of the V-DMC bitstream can be suppressed. For example, an increase in the amount of data in the V-DMC bitstream distributed by this distribution system 600 can be suppressed. Therefore, an increase in the communication bandwidth required for distribution can be suppressed. In addition, an increase in the decoding load of the V-DMC bitstream distributed by this distribution system 600 can be suppressed. Moreover, if there is an upper limit on the amount of data in the V-DMC bitstream distributed by this distribution system 600, the reduction in the quality of the original mesh and attribute map can be suppressed by suppressing the increase in the amount of encoding due to unnecessary information. Therefore, a reduction in the quality of the rendered image can be suppressed.

[0245] Furthermore, by applying this technology to such distribution, it is possible to suppress the reduction in attribute quality while suppressing the reduction in the coding efficiency of the base mesh, similar to the case of the encoding device 300 and the decoding device 400.

[0246] <Flow of Distribution Processing> An example of the flow of distribution processing performed by this distribution system 600 will be explained with reference to the flowchart in Figure 29.

[0247] When the distribution process begins, the viewpoint information supply unit 671 of the terminal device 612 supplies the viewpoint information supplied from the display processing unit 419 to the server 611 via the network 610 in step S601. In step S611, the viewpoint information acquisition unit 641 of the server 611 acquires the viewpoint information.

[0248] In step S612, the server 611 executes the V-DMC bitstream generation process and generates a V-DMC bitstream to which attributes corresponding to the rendering viewpoint indicated by the viewpoint information are applied.

[0249] In step S613, the V-DMC bitstream supply unit 642 of the server 611 supplies the generated V-DMC bitstream to the terminal device 612 via the network 610. In step S602, the V-DMC bitstream acquisition unit 672 of the terminal device 612 acquires the V-DMC bitstream.

[0250] In step S603, the terminal device 612 performs a decoding process (Figure 21) to decode the acquired V-DMC bitstream and generate a display image. This decoding process is performed in the same manner as described for the processing performed by the decoding device 400. When the processing in step S603 is completed, the distribution process is completed.

[0251] <Flow of V-DMC Bitstream Generation Process> An example of the flow of the V-DMC bitstream generation process performed in step S612 of Figure 29 will be explained with reference to the flowchart in Figure 30.

[0252] When the V-DMC bitstream generation process is started, the base mesh generation unit 321 of the server 611 decimates the original mesh to be encoded in step S651 and generates a base mesh (geometry information).

[0253] In step S652, the UV set generation unit 322 UV unwraps the base mesh (geometry information) and generates a UV set. If there are multiple attributes that can be assigned to the same part of the base mesh (for example, if multiple attributes are assigned to a single face of the base mesh), the UV set generation unit 322 UV unwraps the geometry information for each attribute and generates a UV set corresponding to that geometry information. In other words, multiple UV sets are generated, each UV unwrapped in an independent pattern.

[0254] In step S653, the UV set selection unit 632 selects a UV set based on the viewpoint information. That is, the UV set selection unit 632 selects the UV set necessary for rendering the viewpoint indicated by the viewpoint information from among the group of UV sets (i.e., multiple UV sets associated with a single geometry information). If there is only one UV set, the UV set selection unit 632 selects that UV set.

[0255] In step S654, the displacement vector generation unit 323 subdivides the base mesh and generates a displacement vector using the subdivided base mesh and the original mesh.

[0256] In step S655, the attribute conversion unit 324 converts the attribute map corresponding to the original mesh based on the base mesh and the original mesh, and generates an attribute map corresponding to the base mesh. The UV set of this base mesh is the UV set selected in step S653. In other words, through this process, the attribute conversion unit 324 generates an attribute map corresponding to the viewpoint information (i.e., an attribute map necessary for rendering).

[0257] In step S656, the Atlas information encoding unit 312 encodes the Atlas information to generate an Atlas information bitstream.

[0258] In step S657, the base mesh encoding unit 313 performs base mesh encoding processing to encode a base mesh consisting of geometry information and a UV set corresponding to that geometry information. Through this base mesh encoding processing, the base mesh encoding unit 313 generates a geometry information bitstream and a UV set bitstream as encoded data for the base mesh.

[0259] In step S658, the displacement vector encoding unit 314 encodes the displacement vector to generate a displacement vector bitstream.

[0260] In step S659, the attribute encoding unit 315 encodes the attribute map corresponding to the viewpoint information generated in step S655 (i.e., the attribute map required for rendering) and generates an attribute bitstream.

[0261] In step S660, the multiplexing unit 316 multiplexes the atlas information bitstream, the encoded data of the base mesh (geometry information bitstream and UV set bitstream), the displacement vector bitstream, and the attribute bitstream as substreams to generate a V-DMC bitstream.

[0262] Once the processing in step S660 is complete, the V-DMC bitstream generation process ends, and the process returns to Figure 29.

[0263] By performing each process as described above, the distribution system 600 can achieve distribution with view-dependent textures applied. Furthermore, since only the attributes (and UV sets) necessary for rendering are selected and distributed, a reduction in the encoding efficiency of the V-DMC bitstream can be suppressed. For example, an increase in the amount of data in the V-DMC bitstream distributed by this distribution system 600 can be suppressed. Therefore, an increase in the communication bandwidth required for distribution can be suppressed. In addition, an increase in the decoding load of the V-DMC bitstream distributed by this distribution system 600 can be suppressed. Moreover, if there is an upper limit on the amount of data in the V-DMC bitstream distributed by this distribution system 600, the reduction in the quality of the original mesh and attribute map can be suppressed by suppressing the increase in the amount of encoding due to unnecessary information. Therefore, a reduction in the quality of the rendered image can be suppressed.

[0264] Furthermore, by applying this technology to such distribution, it is possible to suppress the reduction in attribute quality while suppressing the reduction in the coding efficiency of the base mesh, similar to the case of the encoding device 300 and the decoding device 400.

[0265] <4. Notes> <Polygon Shape> In the above explanation, the polygon shape was described as a triangle, but this shape is just one example. The polygon shape can be any polygon.

[0266] <Encoding Method> In the above explanation, V-DMC was used as an example of an encoding method to which this technology can be applied. However, this technology is not limited to this example and can be applied to any encoding method that encodes a mesh.

[0267] <Computer> The series of processes described above can be executed by hardware or by software. When the series of processes are executed by software, the programs that make up that software are installed on the computer. Here, "computer" includes computers built into dedicated hardware, as well as general-purpose personal computers that can perform various functions by installing various programs.

[0268] Figure 31 is a block diagram showing an example of the hardware configuration of a computer that executes the series of processes described above using a program.

[0269] In the computer 900 shown in Figure 31, the CPU (Central Processing Unit) 901, ROM (Read-Only Memory) 902, and RAM (Random Access Memory) 903 are interconnected via a bus 904.

[0270] An input / output interface 910 is also connected to the bus 904. An input / output interface 910 is connected to an input unit 911, an output unit 912, a storage unit 913, a communication unit 914, and a drive 915.

[0271] The input unit 911 consists of, for example, a keyboard, mouse, microphone, touch panel, and input terminals. The output unit 912 consists of, for example, a display, speaker, and output terminals. The storage unit 913 consists of, for example, a hard disk, RAM disk, and non-volatile memory. The communication unit 914 consists of, for example, a network interface. The drive 915 drives removable media 921 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0272] In a computer configured as described above, the CPU 901 loads, for example, a program stored in the memory unit 913 into the RAM 903 via the input / output interface 910 and the bus 904, and executes it, thereby performing the series of processes described above. The RAM 903 also appropriately stores data necessary for the CPU 901 to perform various processes.

[0273] The program executed by the computer can be recorded and applied, for example, on removable media 921 such as a package medium. In this case, the program can be installed in the storage unit 913 via the input / output interface 910 by inserting the removable media 921 into the drive 915.

[0274] Furthermore, this program can also be provided via wired or wireless transmission media such as a local area network, the internet, or digital satellite broadcasting. In that case, the program can be received by the communication unit 914 and installed in the storage unit 913.

[0275] In addition, this program can be pre-installed in ROM 902 or memory unit 913.

[0276] <Applications of this technology> This technology can be applied to any configuration. For example, this technology can be applied to various electronic devices.

[0277] Furthermore, this technology can also be implemented as part of a device, such as a processor as a system LSI (Large Scale Integration) (e.g., a video processor), a module using multiple processors (e.g., a video module), a unit using multiple modules (e.g., a video unit), or a set with additional functions added to a unit (e.g., a video set).

[0278] Furthermore, this technology can also be applied to network systems composed of multiple devices. For example, this technology may be implemented as cloud computing, where multiple devices share and collaborate on processing via a network. For example, this technology may be implemented in a cloud service that provides image (video) related services to any terminal such as computers, AV (Audio Visual) equipment, portable information processing terminals, and IoT (Internet of Things) devices.

[0279] In this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all components are located in the same enclosure. Therefore, multiple devices housed in separate enclosures and connected via a network, and a single device containing multiple modules within a single enclosure, are both considered systems.

[0280] <Applicable Fields and Applications of This Technology> Systems, devices, and processing units incorporating this technology can be used in any field, such as transportation, medical care, security, agriculture, livestock farming, mining, beauty, factories, home appliances, weather, and nature monitoring. Furthermore, the applications are entirely arbitrary.

[0281] <Other> In this specification, "flag" refers to information used to identify multiple states, and includes not only information used to identify two states, true (1) or false (0), but also information capable of identifying three or more states. Therefore, the values ​​that this "flag" can take are, for example, two values, 1 / 0, or three or more values. In other words, the number of bits that constitute this "flag" is arbitrary, and can be 1 bit or multiple bits. Furthermore, identification information (including flags) is envisioned not only in the form of including the identification information itself in the bitstream, but also in the form of including difference information of the identification information relative to a certain reference information in the bitstream. Therefore, in this specification, "flag" and "identification information" include not only the information itself, but also difference information relative to the reference information.

[0282] Furthermore, various types of information (metadata, etc.) related to encoded data (bitstream) may be transmitted or recorded in any form as long as they are associated with the encoded data. Here, the term "associate" means, for example, making it possible to use (link) one data when processing the other. In other words, associated data may be combined into a single data, or they may be individual data. For example, information associated with encoded data (image) may be transmitted on a different transmission path than the encoded data (image). Also, for example, information associated with encoded data (image) may be recorded on a different recording medium (or a different recording area on the same recording medium) than the encoded data (image). Note that this "association" may not apply to the entire data, but only to a part of it. For example, an image and the information corresponding to that image may be associated with each other in any unit, such as multiple frames, one frame, or a part within a frame.

[0283] In this specification, terms such as "combine," "multiplex," "add," "integrate," "include," "store," "insert," "insert," and "place" mean combining multiple things into one, such as combining encoded data and metadata into a single data, and represent one method of "associating" as described above.

[0284] Furthermore, the embodiments of this technology are not limited to those described above, and various modifications are possible without departing from the spirit of this technology.

[0285] For example, the configuration described as a single device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, the configurations described above as multiple devices (or processing units) may be combined and configured as a single device (or processing unit). Furthermore, it is also possible to add configurations other than those described above to the configuration of each device (or each processing unit). In addition, if the overall system configuration and operation are substantially the same, a part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0286] Furthermore, for example, the program described above may be executed on any device. In that case, the device should have the necessary functions (such as functional blocks) and be able to obtain the necessary information.

[0287] Furthermore, for example, each step of a flowchart may be executed by one device, or it may be divided among multiple devices. Additionally, if a single step includes multiple processes, these processes may be executed by one device, or they may be divided among multiple devices. In other words, multiple processes included in a single step can be executed as multiple steps. Conversely, processes described as multiple steps can be combined and executed as a single step.

[0288] Furthermore, for example, a program executed by a computer may be structured so that the steps of the program are executed chronologically in the order described herein, or they may be executed in parallel or individually at necessary times, such as when a call is made. In other words, the steps may be executed in an order different from the order described above, as long as no inconsistencies arise. Moreover, the steps of this program may be executed in parallel with the processing of other programs, or in combination with the processing of other programs.

[0289] Furthermore, for example, multiple technologies relating to this technology can be implemented independently, as long as they do not create a contradiction. Of course, any multiple technologies can also be implemented in combination. For example, some or all of the technologies described in one embodiment can be implemented in combination with some or all of the technologies described in another embodiment. Also, some or all of the above-mentioned technologies can be implemented in combination with other technologies not mentioned above.

[0290] Furthermore, this technology can also take the following configurations: (1) An information processing device comprising: a base mesh encoding unit that encodes single geometry information corresponding to at least one face of a base mesh, and a plurality of UV sets corresponding to the face and linked to the single geometry information, as the base mesh; and an attribute encoding unit that encodes a plurality of attribute maps corresponding to the plurality of UV sets as attributes corresponding to the base mesh, wherein the base mesh is a mesh obtained by thinning out the vertices from an original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of an object. (2) The information processing device according to (1), wherein the base mesh encoding unit links the plurality of UV sets to the single geometry information, encodes the single geometry information to generate a single geometry information bitstream, and encodes each of the plurality of UV sets to generate a plurality of UV set bitstreams linked to the single geometry information bitstream. (3) The information processing device according to (2), wherein each of the plurality of UV sets includes UV connectivity and UV vertex coordinates, the base mesh encoding unit encodes the geometry connectivity and geometry vertex coordinates included in the single geometry information to generate the single geometry information bitstream, and encodes the UV connectivity and UV vertex coordinates for each of the plurality of UV sets to generate the plurality of UV set bitstreams. (4) The information processing device according to (2) or (3), wherein the base mesh encoding unit generates linking information that links the single geometry information and the plurality of UV sets, and stores the linking information as metadata in each of the generated plurality of UV set bitstreams. (5) The information processing device according to (2) or (3), wherein the base mesh encoding unit generates table information showing the correspondence between the single geometry information and the plurality of UV sets as linking information that links the single geometry information and the plurality of UV sets.(6) The information processing device according to any one of (1) to (5), wherein the attribute includes a texture. (7) The information processing device according to any one of (1) to (6), wherein the plurality of UV sets correspond to each of a plurality of captured images applied as the attribute of the base mesh, and the plurality of captured images correspond to each of a plurality of different viewpoints. (8) The information processing device according to any one of (1) to (7), wherein the base mesh encoding unit encodes all of the plurality of UV sets, and the attribute encoding unit encodes all of the plurality of attribute maps. (9) The information processing device according to any one of (1) to (8), wherein the base mesh encoding unit generates skip information relating to the face which the encoding of at least one of the plurality of UV sets is skipped. (10) The information processing device according to (9), wherein the skip information includes a list of UV faces corresponding to the face which the encoding of at least one of the plurality of UV sets is skipped. (11) The information processing device according to (9) or (10), wherein the base mesh encoding unit stores the generated skip information in a plurality of UV set bitstreams generated by encoding the plurality of UV sets. (12) The information processing device according to any one of (1) to (11), wherein the base mesh encoding unit generates skip information relating to skipping the encoding of UV connectivity. (13) The information processing device according to (12), wherein the skip information includes flag information indicating whether or not to skip the encoding of UV connectivity. (14) The information processing device according to (12) or (13), wherein the base mesh encoding unit stores the generated skip information in a plurality of UV set bitstreams generated by encoding the plurality of UV sets. (15) The information processing device according to any one of (1) to (14), wherein the plurality of UV sets correspond to each layer of the layered attribute. (16) The information processing device according to any one of (1) to (15), wherein the base mesh encoding unit encodes the plurality of UV sets by applying the parameters set for each of them.(17) The information processing device according to (16), wherein the parameter includes the quantized value of the UV vertex coordinates. (18) The information processing device according to (16) or (17), wherein the parameter includes the prediction mode of the plurality of UV sets. (19) The information processing device according to any one of (16) to (18), wherein the base mesh encoding unit stores the parameter in a plurality of UV set bitstreams generated by encoding the plurality of UV sets. (20) The information processing device according to any one of (1) to (19), wherein the attribute includes a vertex attribute which is the attribute relating to the vertex of the base mesh. (21) The information processing device according to (20), wherein the vertex attribute includes the UV coordinates of the vertex. (22) The information processing device according to (20) or (21), wherein the vertex attribute includes the color information of the vertex. (23) The information processing device according to any one of (20) to (22), wherein the vertex attribute includes the normal of the vertex. (24) The information processing device according to any one of (1) to (23), wherein the base mesh encoding unit encodes the single geometry information and the plurality of UV sets associated with the single geometry information as the base mesh in an intra frame. (25) The information processing device according to any one of (1) to (24), wherein the base mesh encoding unit encodes the single geometry information and the plurality of UV sets associated with the single geometry information as the base mesh in an interframe. (26) The information processing device according to any one of (1) to (25), further comprising: a viewpoint information acquisition unit that acquires viewpoint information relating to the viewpoint of rendering; and a UV set selection unit that selects the UV set necessary for rendering from the plurality of UV sets associated with the single geometry information based on the acquired viewpoint information, wherein the base mesh encoding unit encodes the single geometry information and the selected UV set as the base mesh.(27) The information processing device according to any one of (1) to (26), further comprising a UV set generation unit that generates the plurality of UV sets corresponding to the faces, wherein the base mesh encoding unit encodes the single geometry information and the plurality of UV sets generated and associated with the single geometry information as the base mesh. (28) The information processing device according to (27), further comprising a base mesh generation unit that generates the base mesh by thinning out the vertices of the original mesh, wherein the UV set generation unit generates the plurality of UV sets corresponding to the faces of the generated base mesh. (29) The information processing device according to any one of (1) to (28), further comprising an attribute conversion unit that converts the attribute map corresponding to the original mesh and generates the plurality of attribute maps corresponding to the base mesh, wherein the attribute encoding unit encodes the plurality of attribute maps generated. (30) An information processing device according to any one of (1) to (29), further comprising: a displacement vector generation unit that generates a displacement vector which is the difference in the positions of the vertices between the original mesh and the subdivided base mesh; and a displacement vector encoding unit that encodes the generated displacement vector. (31) An information processing device according to (30), further comprising: an atlas information encoding unit that encodes atlas information used when reconstructing a mesh. (32) An information processing device according to (31), further comprising: a multiplexing unit that multiplexes a single geometry information bitstream generated by encoding the single geometry information, a plurality of UV set bitstreams generated by encoding the plurality of UV sets, a displacement vector bitstream generated by encoding the displacement vector, and an atlas information bitstream generated by encoding the atlas information.(33) An information processing method comprising: encoding single geometric information corresponding to at least one face of a base mesh, and a plurality of UV sets corresponding to the face and linked to the single geometric information, as the base mesh; and encoding a plurality of attribute maps corresponding to each of the plurality of UV sets as attributes corresponding to the base mesh, wherein the base mesh is a mesh obtained by thinning out the vertices from an original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of an object. (34) A program for causing a computer to perform the following processing: encoding single geometric information corresponding to at least one face of a base mesh, and a plurality of UV sets corresponding to the face and linked to the single geometric information, as the base mesh; and encoding a plurality of attribute maps corresponding to each of the plurality of UV sets as attributes corresponding to the base mesh, wherein the base mesh is a mesh obtained by thinning out the vertices from an original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of an object.

[0291] (41) An information processing device comprising: a base mesh decoding unit that decodes a single geometry information bitstream and a plurality of UV set bitstreams associated with the single geometry information bitstream as a base mesh, and generates a single geometry information corresponding to at least one face of the base mesh and a plurality of UV sets corresponding to the face; and an attribute decoding unit that decodes a plurality of attribute bitstreams corresponding to the plurality of UV set bitstreams as attributes corresponding to the base mesh, and generates a plurality of attribute maps corresponding to the plurality of UV sets, wherein the base mesh is a mesh obtained by thinning out the vertices from an original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of an object. (42) The information processing device according to (41), wherein the base mesh decoding unit decodes the single geometry information bitstream and decodes the plurality of UV set bitstreams associated with the single geometry information bitstream. (43) The information processing device according to (42), wherein each of the plurality of UV set bitstreams includes encoded data for UV connectivity and encoded data for UV vertex coordinates, and the base mesh decoding unit decodes the encoded data for geometry connectivity and encoded data for geometry vertex coordinates contained in the single geometry information bitstream, and decodes the encoded data for UV connectivity and encoded data for UV vertex coordinates for each of the plurality of UV set bitstreams. (44) The information processing device according to (42) or (43), wherein the base mesh decoding unit identifies and decodes the plurality of UV set bitstreams associated with the decoded single geometry information bitstream based on the association information that links the single geometry information and the plurality of UV sets, which is stored as metadata in each of the plurality of UV set bitstreams.(45) The information processing device according to (42) or (43), wherein the base mesh decoding unit identifies and decodes the plurality of UV set bitstreams associated with the decoded single geometry information bitstream based on table information showing the correspondence between the single geometry information and the plurality of UV sets. (46) The information processing device according to any one of (41) to (45), wherein the attribute includes a texture. (47) The information processing device according to any one of (41) to (46), wherein the plurality of UV sets correspond to each of a plurality of captured images applied as the attribute of the base mesh, and the plurality of captured images correspond to each of a plurality of different viewpoints. (48) The information processing device according to any one of (41) to (47), wherein the base mesh decoding unit decodes the single geometry information bitstream and all of the plurality of UV set bitstreams associated with the single geometry information bitstream. (49) The information processing device according to any one of (41) to (48), wherein the base mesh decoding unit decodes at least one of the plurality of UV set bitstreams based on skip information relating to the face which the encoding of at least one of the plurality of UV sets is skipped. (50) The information processing device according to (49), wherein the skip information includes a list of UV faces corresponding to the face which the encoding of at least one of the plurality of UV sets is skipped. (51) The information processing device according to (49) or (50), wherein the skip information is stored in the plurality of UV set bitstreams. (52) The information processing device according to any one of (41) to (51), wherein the base mesh decoding unit decodes the encoded data of the UV connectivity based on skip information relating to the skipping of encoding of the UV connectivity. (53) The information processing device according to (52), wherein the skip information includes flag information indicating whether or not to skip the encoding of the UV connectivity.(54) The information processing device according to (52) or (53), wherein the base mesh decoding unit reuses the geometry connectivity generated by decoding the single geometry information bitstream as the UV connectivity that has been skipped. (55) The information processing device according to any one of (52) to (54), wherein the skipped information is stored in the plurality of UV set bitstreams. (56) The information processing device according to any one of (41) to (55), wherein the plurality of UV sets correspond to each layer of the layered attribute. (57) The information processing device according to any one of (41) to (56), wherein the base mesh decoding unit decodes the plurality of UV set bitstreams by applying parameters relating to the encoding of the plurality of UV sets, which are stored in the plurality of UV set bitstreams. (58) The information processing device according to (57), wherein the parameters include quantized values ​​of UV vertex coordinates. (59) The information processing device according to (57) or (58), wherein the parameters include prediction modes of the plurality of UV sets. (60) The information processing device according to any one of (41) to (59), wherein the attribute includes a vertex attribute which is the attribute relating to the vertex of the base mesh. (61) The information processing device according to (60), wherein the vertex attribute includes the UV coordinates of the vertex. (62) The information processing device according to (60) or (61), wherein the vertex attribute includes color information of the vertex. (63) The information processing device according to any one of (60) to (62), wherein the vertex attribute includes the normal of the vertex. (64) The information processing device according to any one of (41) to (63), wherein the base mesh decoding unit decodes the single geometry information bitstream and the plurality of UV set bitstreams associated with the single geometry information bitstream as the base mesh in an intraframe.(65) The information processing device according to any one of (41) to (64), wherein the base mesh decoding unit decodes the single geometry information bitstream and the plurality of UV set bitstreams associated with the single geometry information bitstream as the base mesh in an interframe. (66) The information processing device according to any one of (41) to (65), further comprising: a viewpoint information supply unit that supplies viewpoint information relating to the viewpoint of rendering; and an acquisition unit that acquires the single geometry information bitstream, a UV set bitstream associated with the single geometry information bitstream and selected from the plurality of UV set bitstreams based on the supplied viewpoint information, and an attribute bitstream corresponding to the selected UV set bitstream, wherein the base mesh decoding unit decodes the acquired single geometry information bitstream and the UV set bitstream, and the attribute decoding unit decodes the acquired attribute bitstream. (67) An information processing device according to any one of (41) to (66), further comprising a subdivision unit for subdividing the base mesh generated by decoding the single geometry information bitstream and the plurality of UV set bitstreams. (68) An information processing device according to (67), further comprising a displacement vector decoding unit for decoding a displacement vector bitstream to generate a displacement vector which is the difference in the positions of the vertices between the original mesh and the subdivided base mesh, and a displacement vector application unit for applying the generated displacement vector to the vertices of the subdivided base mesh. (69) An information processing device according to (68), further comprising an attribute application unit for the attribute decoding unit for decoding the plurality of attribute bitstreams to generate the plurality of attribute maps, unpacking the plurality of attribute maps to generate the attribute, and applying the generated attribute to the subdivided base mesh to which the displacement vector has been applied.(70) The information processing apparatus according to (69), further comprising a display processing unit that renders the base mesh, to which the displacement vectors and attributes have been subdivided and applied, to generate a display image. (71) The information processing apparatus according to any one of (41) to (70), further comprising a demultiplexing unit that demultiplexes a bitstream to extract the single geometry information bitstream, the plurality of UV set bitstreams, and the base mesh decoding unit that decodes the extracted single geometry information bitstream and the plurality of UV set bitstreams as the base mesh, and the attribute decoding unit that decodes the extracted plurality of attribute bitstreams. (72) An information processing method comprising: decoding a single geometry information bitstream and a plurality of UV set bitstreams associated with the single geometry information bitstream as a base mesh; generating a single geometry information corresponding to at least one face of the base mesh and a plurality of UV sets corresponding to the face; and decoding a plurality of attribute bitstreams corresponding to the plurality of UV set bitstreams as attributes corresponding to the base mesh; and generating a plurality of attribute maps corresponding to the plurality of UV sets, wherein the base mesh is a mesh obtained by thinning out the vertices from an original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of an object.(73) A program for causing a computer to perform a process which includes decoding a single geometry information bitstream and a plurality of UV set bitstreams associated with the single geometry information bitstream as a base mesh, generating a single geometry information corresponding to at least one face of the base mesh and a plurality of UV sets corresponding to the face, and decoding a plurality of attribute bitstreams corresponding to each of the plurality of UV set bitstreams as attributes corresponding to the base mesh, wherein the base mesh is a mesh obtained by thinning out the vertices from an original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of an object.

[0292] 300 Encoding device, 311 Preprocessing unit, 312 Atlas information encoding unit, 313 Base mesh encoding unit, 314 Displacement vector encoding unit, 315 Attribute encoding unit, 316 Multiplexing unit, 321 Base mesh generation unit, 322 UV set generation unit, 323 Displacement vector generation unit, 324 Attribute conversion unit, 351 Metadata generation unit, 352 Geometry connectivity encoding unit, 353 Geometry vertex coordinate encoding unit, 354 UV connectivity encoding unit, 355 UV vertex coordinate encoding unit, 400 Decoding device, 411 Demultiplexing unit, 412 Atlas information decoding unit, 413 Base mesh decoding unit, 414 Subdivision unit, 415 Displacement vector decoding unit, 416 Displacement vector application unit, 417 Attribute decoding unit, 418 Attribute application unit, 419 Display processing unit, 451 Geometry connectivity decoding unit, 452 Geometry vertex coordinate decoding unit, 453 UV connectivity decoding unit, 454 UV vertex coordinate decoding unit, 600 Distribution system, 610 Network, 611 Server, 612 Terminal device, 631 Communication unit, 632 UV set selection unit, 641 Viewpoint information acquisition unit, 642 V-DMC bitstream supply unit, 661 Communication unit, 671 Viewpoint information supply unit, 672 V-DMC bitstream acquisition unit, 900 Computer

Claims

1. An information processing device comprising: a base mesh encoding unit that encodes single geometric information corresponding to at least one face of a base mesh and a plurality of UV sets corresponding to the face and linked to the single geometric information as the base mesh; and an attribute encoding unit that encodes a plurality of attribute maps corresponding to each of the plurality of UV sets as attributes corresponding to the base mesh, wherein the base mesh is a mesh obtained by thinning out the vertices from an original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of an object.

2. The information processing apparatus according to claim 1, wherein the base mesh encoding unit associates the plurality of UV sets with the single geometry information, encodes the single geometry information to generate a single geometry information bitstream, and encodes each of the plurality of UV sets to generate a plurality of UV set bitstreams associated with the single geometry information bitstream.

3. The information processing apparatus according to claim 2, wherein each of the plurality of UV sets includes UV connectivity and UV vertex coordinates, the base mesh encoding unit encodes the geometry connectivity and geometry vertex coordinates included in the single geometry information to generate the single geometry information bitstream, and encodes the UV connectivity and UV vertex coordinates for each of the plurality of UV sets to generate the plurality of UV set bitstreams.

4. The information processing apparatus according to claim 2, wherein the base mesh encoding unit generates linking information that links the single geometry information with the plurality of UV sets, and stores the linking information as metadata in each of the generated plurality of UV set bitstreams.

5. The information processing apparatus according to claim 1, wherein the plurality of UV sets correspond to each of the plurality of captured images applied as attributes of the base mesh, and the plurality of captured images correspond to each of the plurality of different viewpoints.

6. The information processing apparatus according to claim 1, wherein the base mesh coding unit generates skip information relating to the face, which skips the coding of at least one of the plurality of UV sets.

7. The information processing device according to claim 1, wherein the base mesh coding unit generates skip information relating to the skipping of coding of UV connectivity.

8. The information processing device according to claim 1, wherein the base mesh encoding unit encodes the single geometry information and the plurality of UV sets associated with the single geometry information as the base mesh in an intra frame.

9. The information processing device according to claim 1, wherein the base mesh encoding unit encodes the single geometry information and the plurality of UV sets associated with the single geometry information as the base mesh in an interframe.

10. An information processing method comprising: encoding single geometric information corresponding to at least one face of a base mesh, and a plurality of UV sets corresponding to the face and linked to the single geometric information, as the base mesh; and encoding a plurality of attribute maps corresponding to each of the plurality of UV sets as attributes corresponding to the base mesh, wherein the base mesh is a mesh obtained by thinning out the vertices from an original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of an object.

11. An information processing device comprising: a base mesh decoding unit that decodes a single geometry information bitstream and a plurality of UV set bitstreams associated with the single geometry information bitstream as a base mesh, and generates a single geometry information corresponding to at least one face of the base mesh and a plurality of UV sets corresponding to the face; and an attribute decoding unit that decodes a plurality of attribute bitstreams corresponding to the plurality of UV set bitstreams as attributes corresponding to the base mesh, and generates a plurality of attribute maps corresponding to the plurality of UV sets, wherein the base mesh is a mesh obtained by thinning out the vertices from an original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of an object.

12. The information processing apparatus according to claim 11, wherein the base mesh decoding unit decodes the single geometry information bitstream and decodes each of the plurality of UV set bitstreams associated with the single geometry information bitstream.

13. The information processing apparatus according to claim 12, wherein each of the plurality of UV set bitstreams includes encoded data for UV connectivity and encoded data for UV vertex coordinates, the base mesh decoding unit decodes the encoded data for geometry connectivity and encoded data for geometry vertex coordinates included in the single geometry information bitstream, and decodes the encoded data for UV connectivity and encoded data for UV vertex coordinates for each of the plurality of UV set bitstreams.

14. The information processing apparatus according to claim 12, wherein the base mesh decoding unit identifies and decodes the plurality of UV set bitstreams associated with the decoded single geometry information bitstream based on the association information that links the single geometry information to the plurality of UV sets, which is stored as metadata in each of the plurality of UV set bitstreams.

15. The information processing apparatus according to claim 11, wherein the plurality of UV sets correspond to each of the plurality of captured images applied as attributes of the base mesh, and the plurality of captured images correspond to each of the plurality of different viewpoints.

16. The information processing apparatus according to claim 11, wherein the base mesh decoding unit decodes at least one of the plurality of UV set bitstreams based on skip information relating to the face which the encoding of at least one of the plurality of UV sets is skipped.

17. The information processing apparatus according to claim 11, wherein the base mesh decoding unit decodes the encoded data of UV connectivity based on skip information relating to the skipping of encoding of UV connectivity.

18. The information processing device according to claim 11, wherein the base mesh decoding unit decodes the single geometry information bitstream and the plurality of UV set bitstreams associated with the single geometry information bitstream as the base mesh in an intra frame.

19. The information processing apparatus according to claim 11, wherein the base mesh decoding unit decodes the single geometry information bitstream and the plurality of UV set bitstreams associated with the single geometry information bitstream as the base mesh in an interframe.

20. An information processing method comprising: decoding a single geometry information bitstream and a plurality of UV set bitstreams associated with the single geometry information bitstream as a base mesh; generating a single geometry information corresponding to at least one face of the base mesh and a plurality of UV sets corresponding to the face; and decoding a plurality of attribute bitstreams corresponding to the plurality of UV set bitstreams as attributes corresponding to the base mesh; and generating a plurality of attribute maps corresponding to the plurality of UV sets, wherein the base mesh is a mesh obtained by thinning out the vertices from an original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of an object.

Images