Encoding device and method, and decoding device and method

Abstract

The present disclosure relates to an information processing device and method that make it possible to suppress a reduction in the quality of 3D data due to encoding and decoding. A mesh representing an object having a three-dimensional structure is divided into patches, thereby generating pair information indicating pairs configured from a single vertex of the mesh or a plurality of vertexes of a patch generated from a plurality of vertexes in the same position, and the generated pair information is encoded. For example, the present disclosure can be applied to information processing devices, electronic devices, information processing methods, programs, and the like.

Description

[Technical Field]

[0001] The present disclosure provides: Encoding device and method, and decoding device and method In particular, it is possible to suppress the degradation of the quality of 3D data due to encoding and decoding. Encoding device and method, and decoding device and method Regarding. [Background technology]

[0002] Conventionally, meshes have been used as 3D data representing three-dimensional objects. As a method for compressing meshes, a method has been proposed in which VPCC (Video-based Point Cloud Compression) is extended (see, for example, Non-Patent Document 1).

[0003] However, this method can cause distortions in the vertices of the patch boundaries due to encoding and decoding, which can lead to cracks in the object (3D data), degrading the appearance.To address this issue, an algorithm called Zippering has been proposed, which searches for nearby boundary vertices in 3D and corrects cracks by moving the vertex coordinates (see, for example, Non-Patent Document 2). [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Danillo Graziosi, Alexandre Zaghetto, Ali Tabatabai, "[VPCC] [EE2.6-related] Mesh Patch Data", ISO / IEC JTC 1 / SC 29 / WG 7 m 55368, October 2020 [Non-patent document 2] Danillo Graziosi, Alexandre Zaghetto, Ali Tabatabai, "[VPCC] [EE2.6-related] Mesh Geometry Smoothing Filter", ISO / IEC JTC 1 / SC 29 / WG 7 m 55374, October 2020 Summary of the Invention [Problem to be solved by the invention]

[0005] However, with zippering, there was a risk that cracks would not be properly corrected, reducing the quality of the 3D data.

[0006] The present disclosure has been made in light of such circumstances, and makes it possible to suppress degradation of the quality of 3D data due to encoding and decoding. [Means for solving the problem]

[0007] One aspect of this technology encoding device The encoding method includes a generating unit that divides a mesh representing an object with a three-dimensional structure into at least a first patch and a second patch, arranges the divided patches in a single image, and generates pair information indicating that at least one vertex of the first patch is paired with at least one vertex of the second patch that is located at the same position as the vertex of the first patch in the object before encoding; and an encoding unit that encodes the pair information. the pair information includes, for a first edge of the first patch and a second edge of the second patch, which correspond to the boundary between the first patch and the second patch in the object before the encoding, information indicating a correspondence relationship between a scan direction for the first edge at the time of decoding and a scan direction for the second edge at the time of decoding, and the pair information is information that is applied when correcting cracks in the reconstructed mesh. is. Another aspect of the present technology is an encoding device that includes: a generation unit that divides a mesh representing an object with a three-dimensional structure into at least a first patch and a second patch, arranges the patches within a single image, and generates pair information indicating that at least one vertex of the first patch is paired with at least one vertex of the second patch that is located at the same position as the vertex of the first patch in the object before encoding; and an encoding unit that encodes the pair information, wherein the pair information includes information indicating a correspondence relationship between a combination of the vertex located at both ends of the first edge and the vertex located midway between the first edge, and a combination of the vertex located at both ends of the second edge and the vertex located midway between the second edge, with respect to a first edge of the first patch and a second edge of the second patch that respectively correspond to the boundary between the first patch and the second patch in the object before encoding, and the pair information is information that is applied when correcting cracks in the reconstructed mesh. According to yet another aspect of the present technology, there is provided an encoding device that divides a mesh representing an object with a three-dimensional structure into at least a first patch and a second patch, arranges the divided patches within a single image, and generates pair information indicating that at least one vertex of the first patch is paired with at least one vertex of the second patch that is located at the same position as the vertex of the first patch in the object before encoding; and an encoding unit that encodes the pair information, wherein the pair information indicates that multiple vertices of the first patch and corresponding multiple vertices of the second patch are paired, and is information that is applied when correcting cracks in the reconstructed mesh. According to yet another aspect of the present technology, there is provided an encoding device that includes: a generation unit that divides a mesh representing an object with a three-dimensional structure into at least a first patch and a second patch, arranges the divided patches within a single image, and generates pair information indicating that at least one vertex of the first patch is paired with at least one vertex of the second patch that is located at the same position as the vertex of the first patch in the object before encoding; and an encoding unit that encodes the pair information, wherein the pair information indicates the pair using identification information corresponding to the state of each of the vertices in the object before encoding, and is information that is applied when correcting cracks in the reconstructed mesh.

[0008] One aspect of this technology Encoding methoddivides a mesh representing an object having a three-dimensional structure into at least a first patch and a second patch, arranges the divided patches in a single image, generates pair information indicating that at least one vertex of the first patch is paired with at least one vertex of the second patch that is located at the same position as the vertex of the first patch in the object before encoding, and encodes the generated pair information. wherein the pair information includes, for a first edge of the first patch and a second edge of the second patch, which correspond to a boundary between the first patch and the second patch in the object before encoding, information indicating a correspondence relationship between a scan direction for the first edge at the time of decoding and a scan direction for the second edge at the time of decoding, and the pair information is information that is applied when correcting cracks in the reconstructed mesh. is.

[0009] This technology moreover Other aspects Decryption device teeth, a 2D decoding unit that acquires coded data of a geometry image and generates a geometry image; Meshes that represent 3D structured objects Regarding the first and second patches representing and decoding encoded data of pair information indicating that at least one vertex of the first patch and at least one vertex of the second patch that is located at the same position as the vertex of the first patch in the object before encoding are paired, thereby obtaining the pair information. Pair Information a decoding unit; a reconstruction unit that reconstructs a mesh from the geometry image; a correction unit that corrects the reconstructed mesh based on the pair information. the correction unit deletes one of the vertices of the first patch and the vertices of the second patch that are indicated as being a pair by the pair information in the reconstructed mesh, and converts the connection of the deleted vertex into the connection of the other vertex. is. A decoding device according to yet another aspect of the present technology includes a 2D decoding unit that acquires encoded data of a geometry image and generates a geometry image; a pair information decoding unit that decodes encoded data of pair information indicating that, with respect to a first patch and a second patch that represent a mesh that represents an object with a three-dimensional structure arranged in the geometry image, at least one vertex of the first patch and at least one vertex of the second patch that is located at the same position as a vertex of the first patch in the object before encoding are paired, and obtains the pair information; a reconstruction unit that reconstructs a mesh from the geometry image; and a correction unit that corrects the reconstructed mesh based on the pair information, wherein the correction unit moves one of the vertices of the first patch and the vertex of the second patch that are indicated to be paired by the pair information to the position of the other vertex in the reconstructed mesh. A decoding device according to yet another aspect of the present technology includes a 2D decoding unit that acquires encoded data of a geometry image and generates a geometry image; a pair information decoding unit that decodes encoded data of pair information indicating that, regarding a first patch and a second patch that represent a mesh that represents an object with a three-dimensional structure placed in the geometry image, at least one vertex of the first patch and at least one vertex of the second patch that is located at the same position as a vertex of the first patch in the object before encoding are paired, and obtains the pair information; a reconstruction unit that reconstructs a mesh from the geometry image; and a correction unit that corrects the reconstructed mesh based on the pair information, wherein the correction unit forms a polygon in the reconstructed mesh that includes the vertex of the first patch and the vertex of the second patch that are indicated to be paired by the pair information. A decoding device according to yet another aspect of the present technology includes a 2D decoding unit that acquires encoded data of a geometry image and generates a geometry image; a pair information decoding unit that decodes encoded data of pair information indicating that, regarding a first patch and a second patch that represent a mesh that represents an object with a three-dimensional structure placed in the geometry image, at least one vertex of the first patch and at least one vertex of the second patch that is located at the same position as a vertex of the first patch in the object before encoding are paired, and obtains the pair information; a reconstruction unit that reconstructs a mesh from the geometry image; and a correction unit that corrects the reconstructed mesh based on the pair information, wherein the correction unit interpolates the vertices that constitute the pair but are not included in the pair information.

[0010] This technology moreover Other aspects Decryption method teeth, The coded data of the geometry image is acquired to generate a geometry image, and the coded data arranged in the geometry image is Meshes that represent 3D structured objects Regarding the first and second patches representing decoding encoded data of pair information indicating that at least one vertex of the first patch is paired with at least one vertex of the second patch that is located at the same position as the vertex of the first patch in the object before encoding; reconstructing a mesh from the geometry image; The reconstructed mesh In The pair information obtained by decoding the encoded data is a decoding method for forming a polygon including the vertices of the first patch and the vertices of the second patch that are indicated to be the pair by is.

[0011] One aspect of this technology encoding deviceIn the method, a mesh representing an object with a three-dimensional structure is divided into at least a first patch and a second patch, and the patches are arranged in a single image, and pair information indicating a pair of at least one vertex of the first patch and at least one vertex of the second patch that is located at the same position as the vertex of the first patch in the object before encoding is generated, and the generated pair information is encoded. The pair information includes information indicating the correspondence between the scanning direction for the first edge at the time of decoding and the scanning direction for the second edge at the time of decoding, for the first edge of the first patch and the second edge of the second patch, which correspond to the boundary between the first patch and the second patch in the object before encoding, and is information that is applied when correcting the reconstructed mesh.

[0012] Other aspects of the technology Decryption device and in the method, The coded data of the geometry image is obtained to generate a geometry image, and the Meshes that represent 3D structured objects Regarding the first and second patches representing At least one vertex of the first patch and the the In objects the It is located at the same position as the vertex of the first patch. the decoding encoded data of pair information indicating that at least one vertex of the second patch is a pair; A mesh is reconstructed from the geometry image, and Reconstructed Mesh In , and the pair information obtained by decoding the encoded data is A polygon including the vertices of the first patch and the vertices of the second patch that are shown to be a pair is formed. will be done. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram illustrating a video-based approach. [Figure 2] FIG. 1 is a diagram illustrating a Mesh. [Figure 3] FIG. 10 is a diagram illustrating the difference in data structure between VPCC and Mesh. [Figure 4] FIG. 10 is a diagram illustrating a zipper ring. [Figure 5] FIG. 10 is a diagram illustrating transmission of pair information. [Figure 6] FIG. 10 is a diagram illustrating transmission of pair information. [Figure 7] FIG. 10 is a diagram illustrating pair information. [Figure 8] FIG. 10 is a diagram illustrating generation of pair information. [Figure 9] FIG. 10 is a diagram illustrating an example of a transmission order. [Figure 10] FIG. 10 is a diagram illustrating an example of the contents of pair information. [Figure 11] FIG. 10 is a diagram illustrating an example of the contents of pair information. [Figure 12] FIG. 10 is a diagram illustrating an example of the contents of pair information. [Figure 13] FIG. 10 is a diagram illustrating an example of the configuration of pair information. [Figure 14] FIG. 10 is a diagram illustrating an example of the configuration of pair information. [Figure 15] FIG. 10 is a diagram illustrating an example of the configuration of pair information. [Figure 16] FIG. 10 is a diagram illustrating an example of vertex interpolation. [Figure 17] FIG. 10 is a diagram illustrating an example of vertex interpolation. [Figure 18] FIG. 10 is a diagram illustrating an example of mesh correction. [Figure 19] FIG. 10 is a diagram illustrating an example of mesh correction. [Figure 20] FIG. 10 is a diagram illustrating an example of mesh correction. [Figure 21] FIG. 1 is a block diagram illustrating an example of the main configuration of an encoding device. [Figure 22] 10 is a flowchart illustrating an example of the flow of an encoding process. [Figure 23] FIG. 2 is a block diagram illustrating an example of the main configuration of a decoding device. [Figure 24] 10 is a flowchart illustrating an example of the flow of a decoding process. [Figure 25] FIG. 10 is a diagram illustrating an example of deriving correct coordinates. [Figure 26] FIG. 10 is a diagram illustrating an example of deriving correct coordinates. [Figure 27] FIG. 10 is a diagram illustrating an example of a pairing process. [Figure 28] FIG. 10 is a diagram illustrating an example of a pairing process. [Figure 29] FIG. 1 is a block diagram illustrating an example of the main configuration of an encoding device. [Figure 30]10 is a flowchart illustrating an example of the flow of an encoding process. [Figure 31] FIG. 2 is a block diagram illustrating an example of the main configuration of a decoding device. [Figure 32] 10 is a flowchart illustrating an example of the flow of a decoding process. [Figure 33] FIG. 1 is a block diagram illustrating an example of the main configuration of a computer. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, modes for carrying out the present disclosure (hereinafter referred to as embodiments) will be described in the following order. 1. Mesh compression with VPCC extension 2.Transmission of pair information 3. First embodiment (encoding device) 4. Second embodiment (decoding device) 5. Deriving the correct coordinates 6. Two-dimensional pairing process 7. Third embodiment (encoding device) 8. Fourth embodiment (decoding device) 9. Notes

[0015] <1. Mesh compression with VPCC expansion> <References supporting technical content and technical terminology> The scope of disclosure of the present technology includes not only the contents described in the embodiments but also the contents described in the following non-patent documents that were publicly known at the time of filing, as well as the contents of other documents referenced in the following non-patent documents.

[0016] Non-patent document 1: (mentioned above) Non-patent document 2: (mentioned above) [Non-patent document 3] "Information technology - Coded Representation of Immersive Media - Part 5: Visual Volumetric Video-based Coding (V3C) and Video-based Point Cloud Compression (V-PCC)", ISO / IEC 23090-5:2020(E),ISO / IEC JTC 1 / SC 29 / WG 11 w19579

[0017] In other words, the contents of the above-mentioned non-patent documents and the contents of other documents referenced in the above-mentioned non-patent documents are also used as the basis for determining the support requirements.

[0018] <Point Cloud> Previously, there was 3D data such as point clouds that represent three-dimensional structures using point position information and attribute information.

[0019] For example, in the case of a point cloud, a three-dimensional structure (a three-dimensional object) is represented as a collection of many points. A point cloud is composed of position information (also called geometry) and attribute information (also called attributes) for each point. Attributes can include any information. For example, attributes may include color information, reflectance information, normal information, etc. for each point. In this way, point clouds have a relatively simple data structure, and by using a sufficient number of points, they can represent any three-dimensional structure with sufficient accuracy.

[0020] <vpcc> VPCC (Video-based Point Cloud Compression) is one of such point cloud encoding technologies, which encodes point cloud data, which is 3D data representing a three-dimensional structure, using a codec for two-dimensional images.

[0021] In VPCC, the geometry and attributes of a point cloud are decomposed into small regions (also called patches), and each patch is projected onto a projection surface, which is a two-dimensional plane. For example, the geometry and attributes are projected onto one of the six faces of a bounding box that contains an object. The geometry and attributes projected onto this projection surface are also called a projection image. The patch projected onto the projection surface is also called a patch image.

[0022] For example, the geometry of point cloud 1, which indicates a three-dimensional object shown in Figure 1A, is decomposed into patches 2 as shown in Figure 1B, and each patch is projected onto a projection plane. In other words, a patch image of the geometry (projected image of each patch) is generated. Each pixel value of the patch image of the geometry indicates the distance (depth value) from the projection plane to the point.

[0023] The attributes of point cloud 1 are also decomposed into patches 2 in the same way as the geometry, and each patch is projected onto the same projection plane as the geometry. In other words, an attribute patch image of the same size and shape as the geometry patch image is generated. Each pixel value of the attribute patch image indicates the attribute (color, normal vector, reflectance, etc.) of the point at the same position in the corresponding geometry patch image.

[0024] Each patch image thus generated is then arranged within a frame image (also called a video frame) of a video sequence, i.e., each patch image on the projection surface is arranged on a predetermined two-dimensional plane.

[0025] For example, a frame image in which a geometry patch image is arranged is also called a geometry video frame. This geometry video frame is also called a geometry image, a geometry map, etc. The geometry image 11 shown in C of FIG. 1 is a frame image (geometry video frame) in which a geometry patch image 3 is arranged. This patch image 3 corresponds to patch 2 in B of FIG. 1 (the geometry patch 2 is projected onto the projection surface).

[0026] A frame image in which an attribute patch image is arranged is also called an attribute video frame. This attribute video frame is also called an attribute image or an attribute map. The attribute image 12 shown in D of FIG. 1 is a frame image (attribute video frame) in which an attribute patch image 4 is arranged. This patch image 4 corresponds to patch 2 in B of FIG. 1 (the attribute patch 2 is projected onto the projection surface).

[0027] These video frames are then encoded using a coding method for 2D images, such as AVC (Advanced Video Coding) or HEVC (High Efficiency Video Coding). In other words, point cloud data, which is 3D data representing a 3D structure, can be encoded using a codec for 2D images. Generally, 2D data encoders are more widely used than 3D data encoders and can be implemented more inexpensively. In other words, by applying the video-based approach described above, it is possible to suppress increases in costs.

[0028] In such a video-based approach, an occupancy image (also called an occupancy map) can also be used. The occupancy image is map information that indicates the presence or absence of a projected image (patch image) for each NxN pixel of a geometry video frame or an attribute video frame. For example, the occupancy image indicates areas (NxN pixels) in the geometry image or attribute image where a patch image exists with a value of "1" and areas (NxN pixels) in which a patch image does not exist with a value of "0."

[0029] This occupancy image is coded as separate data from the geometry image and attribute image and transmitted to the decoding side. By referencing this occupancy map, the decoder can determine whether a patch exists in the area, thereby suppressing the effects of noise caused by encoding and decoding, and more accurately reconstructing the point cloud. For example, even if the depth value changes due to encoding and decoding, the decoder can ignore the depth value of the area where the patch image does not exist by referring to the occupancy map (so that it is not processed as position information of 3D data).

[0030] For example, an occupancy image 13 as shown in Fig. 1E may be generated for the geometry image 11 in Fig. 1C or the attribute image 12 in Fig. 1D. In the occupancy image 13, white areas indicate a value of "1" and black areas indicate a value of "0."

[0031] The occupancy image can also be transmitted as a video frame, similar to the geometry video frame, attribute video frame, etc. In other words, like the geometry and attributes, it is coded using a coding method for 2D images, such as AVC or HEVC.

[0032] In other words, in the case of VPCC, the geometry and attributes of the point cloud are projected onto the same projection plane and placed at the same position in the frame image. In other words, the geometry and attributes of each point correspond to each other by their position on the frame image.

[0033] <mesh> By the way, as 3D data representing an object with a three-dimensional structure, in addition to point clouds, for example, there is a mesh. As shown in FIG. 2, the mesh represents the surface of an object in three-dimensional space by polygons, which are planes (polygons) surrounded by edges 22 connecting vertices 21. As 3D data representing the object, as shown in FIG. 2, this mesh and texture 23 attached to each polygon are included.

[0034] 3D data using a mesh, for example, as shown in the lower part of FIG. 2, consists of vertex information 31 composed of the position information (three-dimensional coordinates (X, Y, Z)) of each vertex 21, connectivity 32 indicating vertices 21 and edges 22 that make up each polygon, texture image 33 which is map information of texture 23 attached to each polygon, and a UV map 34 indicating the position in the texture image 33 of the texture corresponding to each vertex 21 (that is, the position of each vertex 21 in the texture image 33). The UV map 34 indicates the position of each vertex by UV coordinates, which are coordinates on the texture image 33.

[0035] In the case of 3D data using a mesh, different from the case of the above-mentioned VPCC, the UV map 34 shows the correspondence between each vertex 21 and the texture 23. Therefore, as in the example of FIG. 2, the texture image 33 is configured as map information independent of the vertex information 31 composed of the three-dimensional coordinates of each vertex. Therefore, in the texture image 33, the texture 23 of each polygon can be arbitrarily set in terms of its projection direction and resolution.

[0036] <Compression of Mesh Using VPCC> As a method for compressing 3D data using such a mesh, for example, in Non-Patent Document 1 and the like, a method has been proposed to extend the above-mentioned VPCC to compress (encode) 3D data using a mesh.

[0037] However, this method had the risk of causing distortions in the vertices of the patch boundaries due to encoding and decoding, which could lead to cracks in the object (3D data) and a deterioration in appearance.

[0038] In a mesh, as shown in A of FIG. 3, the geometry of the vertices (the positions of the vertices) is represented as non-image vertex information. Therefore, for example, a vertex 43 shared by polygons 41 and 42 is represented as a single point. In contrast, textures are arranged in texture image 44. That is, texture 45 attached to polygon 41 and texture 46 attached to polygon 42 are each independently arranged in texture image 44. Therefore, for example, as shown in A of FIG. 3, textures 45 and 46 can be arranged at positions separated from each other. That is, in this case, vertex 43 is represented as two points, vertex 47 and vertex 48, in texture image 44. However, since the geometry is represented as a single point as described above, when a mesh is reconstructed, the textures of vertices 47 and 48 are attached to vertex 43 (at the same positions). Even if distortion occurs at vertex 43 due to encoding, the position of vertex 43 is simply shifted, and the textures at vertices 47 and 48 are attached to the same positions.

[0039] In contrast, in VPCC, as shown in B of FIG. 3, geometry is also represented as geometry image 51, just as in the case of texture. In other words, when VPCC is extended to compress (encode) 3D data using meshes, the mesh representing an object with a three-dimensional structure is divided into at least a first patch and a second patch and arranged in a single image. Therefore, polygon 41 is arranged as polygon 52, and polygon 42 is arranged as polygon 54, each independently, in geometry image 51. More precisely, the vertices constituting each polygon are arranged in geometry image 51 so as to form the respective polygons. Therefore, for example, as shown in B of FIG. 3, polygon 52 and polygon 54 can be arranged at positions separated from each other.

[0040] In other words, a vertex shared by multiple polygons in a mesh is arranged as a vertex constituting each polygon in the geometry image, and therefore can be arranged as multiple points. For example, in the example of B in Figure 3, in geometry image 51, vertex 53 of polygon 52 and vertex 55 of polygon 54 both correspond to vertex 43 (one vertex) of the mesh.

[0041] In this way, a single vertex in the mesh may be represented as multiple vertices in the geometry image. Also, multiple vertices (also called overlapping points) that existed at the same position in the mesh may be located at different positions in the geometry image.

[0042] If coding distortion (misalignment) occurs in the geometry of such vertices, when the mesh is reconstructed, each vertex of the geometry that was originally one may become a vertex in a different position, which may cause cracks in the object (mesh). In other words, there is a risk that the quality of the 3D data using the mesh may be reduced.

[0043] <zippering> Therefore, an algorithm called Zippering has been proposed, as described in Non-Patent Document 2, for example. Zippering searches for nearby boundary vertices in 3D, and the vertex to be processed is moved to match the searched point (also called matching). By correcting the coordinates of each vertex in the part where a crack has occurred in this way, the crack is eliminated.

[0044] However, with this method, if no matching points are found within the set search range, the points are not matched, and therefore, cracks may not be removed with this method.

[0045] For example, suppose a crack occurs between patch 61 and patch 62 shown in A of FIG. 4. Vertex 63, indicated by a white circle in the figure, is a vertex on the boundary of patch 61 or patch 62. In FIG. 4, only one vertex is marked with a symbol. Assume that the boundary where vertex 63 of patch 61 is located and the boundary where vertex 63 of patch 62 is located are in contact with each other in the pre-encoding state. Also, assume that vertex A1 of patch 61 and vertex B1 of patch 62 were a single vertex in the pre-encoding state. Similarly, assume that vertices A2 and B2, vertices A3 and B3, vertices A4 and B4, and vertices A5 and B5 were each a single vertex in the pre-encoding state.

[0046] In this specification, the boundary of a patch refers to the boundary between an area included in the patch and an area not included in the patch. For example, the edge (connection) of a polygon that forms the outline of the patch is called the boundary.

[0047] For example, if the zippering search does not find a vertex 63 in patch 61 that corresponds to vertex B2, vertex B2 will not be matched with other vertices. In other words, the geometry of vertex B2 will remain unchanged without being corrected. Therefore, as shown in B of Figure 4, there is a risk that cracks will occur (the cracks will not be removed).

[0048] In addition, this method could potentially match inappropriate points. Zippering searches for corresponding points in one direction (the normal direction of the patch), so there is a risk of matching points other than the original points.

[0049] For example, in the case of A in Fig. 4, vertex B3 is matched to vertex A4, and vertex B4 is matched to vertex A5. Therefore, as shown in B in Fig. 4, there is a risk that the texture of patch 61 and the texture of patch 62 will become discontinuous (i.e., the continuity of the texture that existed in the state before encoding will be lost).

[0050] As described above, when zippering is applied, there is a risk that cracks will not be corrected properly, resulting in a decrease in the quality of the 3D data.

[0051] Additionally, since zippering involves searching for matching points, there is a risk that the processing load will increase.

[0052] <2. Transmission of pair information> Therefore, when reconstructing a mesh, pair information indicating pairs of patch vertices that are to be at the same position is transmitted from the encoder to the decoder. That is, as shown in the top row of the table in Figure 5, pair information is transmitted (for example, the encoder generates and encodes the pair information, and the decoder decodes and uses the encoded data).

[0053] For example, in an information processing method, a generation unit of an information processing device divides a mesh representing an object with a three-dimensional structure into at least a first patch and a second patch and arranges them within a single image, generates pair information indicating that at least one vertex of the first patch is paired with at least one vertex of the second patch that is located at the same position as the vertex of the first patch in the object before encoding, and an encoding unit of the information processing device encodes the pair information.

[0054] For example, an information processing device may include a generation unit that divides a mesh representing an object with a three-dimensional structure into at least a first patch and a second patch and arranges them within a single image, generates pair information indicating that at least one vertex of the first patch is paired with at least one vertex of the second patch that is located at the same position as the vertex of the first patch in the object before encoding, and an encoding unit that encodes the pair information.

[0055] For example, in an information processing method, a decoding unit of an information processing device divides a mesh representing an object with a three-dimensional structure into at least a first patch and a second patch and arranges them within a single image, decodes encoded data of pair information indicating that at least one vertex of the first patch is paired with at least one vertex of the second patch that is located at the same position as the vertex of the first patch in the object before encoding, and a correction unit of the information processing device corrects the reconstructed mesh based on the pair information obtained by decoding the encoded data.

[0056] For example, an information processing device may include a decoding unit that divides a mesh representing an object with a three-dimensional structure into at least a first patch and a second patch and arranges them within a single image, decodes encoded data of pair information indicating that at least one vertex of the first patch is paired with at least one vertex of the second patch that is located at the same position as the vertex of the first patch in the object before encoding, and obtains the pair information, and a correction unit that corrects the reconstructed mesh based on the pair information.

[0057] For example, suppose that polygon 101 and polygon 102 shown in Fig. 7A are composed of vertices 103 indicated by white circles. Note that in Fig. 7A and Fig. 7B, only one vertex is assigned a symbol. The numbers shown near each vertex 103 are identification information for each vertex 103.

[0058] Assume that polygon 110 and polygon 102 are in contact with each other before encoding. In other words, assume that vertex 103 with identification information "3" and vertex 103 with identification information "6" are one vertex 103 before encoding, and vertex 103 with identification information "2" and vertex 103 with identification information "4" are one vertex. By extending the VPCC and encoding the mesh, polygon 101 and polygon 102 are placed independently of each other in the geometry image, and therefore, different identification information is assigned to each vertex 103 of each polygon.

[0059] In such a case, the encoder generates pair information that represents the vertex 103 with identification information "3" and the vertex 103 with identification information "6," and the vertex 103 with identification information "2" and the vertex 103 with identification information "4" as pairs. The decoder matches the vertices 103 according to the pair information. That is, the decoder corrects the position of each vertex 103 in the reconstructed mesh so that the vertices 103 that were one vertex 103 before encoding become one vertex in the pair information.

[0060] For example, as shown in B of Fig. 7, the decoder matches vertex 103 of identification information "3" with vertex 103 of identification information "6", and matches vertex 103 of identification information "2" with vertex 103 of identification information "4". This causes polygon 101 and polygon 102 to be in contact with each other (they are in the same state as before encoding). This makes it possible to suppress the occurrence of cracks.

[0061] Since the pair information (transmitted from the encoder) indicates which vertices are to be matched, the decoder can match vertices more reliably and accurately than in the case of zippering. In other words, cracks can be removed more reliably, and more appropriate vertices can be matched (the occurrence of texture discontinuities can be suppressed). Therefore, it is possible to suppress the degradation of 3D data quality due to encoding and decoding. Furthermore, since search processing is not required, it is possible to suppress an increase in processing load.

[0062] In this specification, the term "pair" refers to a "set" (which can also be expressed as a "contour" or "group") consisting of multiple vertices of a patch generated from a single vertex of a mesh. In other words, the vertices that make up the pair correspond to the same vertex in the mesh. In other words, the "pair information" that indicates this pair is information that indicates the correspondence between the vertices of multiple patches (which vertices correspond to which vertices). Note that a pair may be made up of three or more vertices. In other words, one vertex of a mesh may be divided into three or more patches.

[0063] The process of matching vertices based on such pair information is also called pairing processing.

[0064] Although the above describes an example in which one vertex in a mesh is divided into multiple patches, each vertex constituting a pair may correspond to an "overlapping point" in the mesh. In other words, multiple vertices at the same position in the mesh may be divided into different patches to form a pair.

[0065] Furthermore, although the above description has been given assuming that pairs are configured by vertices, they may be configured by "edges" instead of vertices. Adjacent polygons share not only vertices but also "edges" ("connections" connecting vertices). A "set" of consecutive edges or vertices shared by a pair of patches is also called an "edge." In other words, a pair may indicate a "set" configured by multiple edges. In other words, pair information may be information indicating the correspondence between edges of multiple patches.

[0066] <Generate pair information> Next, we will explain how pair information is generated. As shown in the third row from the top of the table in Figure 5, each vertex (or edge) separated when dividing a mesh into patches is paired. This process of dividing a mesh into patches is also called patch segmentation. In patch segmentation, patches are generated by connecting adjacent polygons (triangles) of the mesh. Information about adjacent polygons is obtained from connectivity. When adjacent polygons are in different patches, the edges (vertices or sides) shared by those polygons are divided into multiple parts (each vertex has multiple UV coordinates). The vertices located at each of these multiple UV coordinates are paired.

[0067] For example, in A of FIG. 8, white circles represent vertices of a mesh 111. The vertices represented by these white circles are referred to as vertices 113. Although only one white circle is assigned a reference number in A of FIG. 8, all of the white circles are vertices 113. Furthermore, the numbers near each vertex 113 schematically represent the identification information assigned to that vertex 113. That is, each of the five vertices 113 is assigned identification information "1" to "5." Furthermore, the connections (edges of the polygon) connecting the vertices 113 are referred to as edges 112. Although only one connection is assigned a reference number in A of FIG. 8, all of the connections connecting the vertices 113 are edges 112. The other vertices are not shown.

[0068] Assume that patch segmentation divides this mesh 111 at edge 112 as shown in B of FIG. 8, dividing it into patch 111A and patch 111B. Through this process, edge 112 and vertex 113 are each divided into two. The vertex 113 divided on the patch 111A side in this way is referred to as vertex 113A. Also, the vertex 113 divided on the patch 111B side is referred to as vertex 113B. Note that in B of FIG. 8, only one of the white circles in patch 111A is assigned a symbol, but all of the white circles in patch 111A are vertices 113A. Similarly, only one of the white circles in patch 111B is assigned a symbol, but all of the white circles in patch 111B are vertices 113B.

[0069] That is, the vertex 113 of identification information "1" is divided into a vertex 113A of identification information "A1" and a vertex 113B of identification information "B1". Similarly, the vertex 113 of identification information "2" is divided into a vertex 113A of identification information "A2" and a vertex 113B of identification information "B2". The vertex 113 of identification information "3" is divided into a vertex 113A of identification information "A3" and a vertex 113B of identification information "B3". The vertex 113 of identification information "4" is divided into a vertex 113A of identification information "A4" and a vertex 113B of identification information "B4". The vertex 113 of identification information "5" is divided into a vertex 113A of identification information "A5" and a vertex 113B of identification information "B5".

[0070] Similarly, the edge 112 separated on the patch 111A side is referred to as edge 112A. Also, the edge 112 separated on the patch 111B side is referred to as edge 112B. In FIG. 8B, only one of the connections connecting the vertices 113A of the patch 111A is labeled, but all of the connections connecting the vertices 113A are edges 112A. Similarly, only one of the connections connecting the vertices 113B of the patch 111B is labeled, but all of the connections connecting the vertices 113B are edges 112B.

[0071] That is, the edge formed by vertex 113A and side 112A and the edge formed by vertex 113B and side 112B indicate edges shared by patch 111A and patch 111B.

[0072] The patches are projected onto a two-dimensional plane, and the projected images (also referred to as patch images) are arranged in the geometry image. Fig. 8C shows an example of such an arrangement. In the geometry image 120 shown in Fig. 8C, patch image 121A shows a projected image of patch 111A. Furthermore, patch image 121B shows a projected image of patch 111B.

[0073] Vertex 123A indicated by a white circle indicates a vertex of patch image 121A. Similarly, vertex 123B indicated by a white circle indicates a vertex of patch image 121B. Vertex 123A is the vertex corresponding to vertex 113A (the projection of vertex 113A), and vertex 123B is the vertex corresponding to vertex 113B (the projection of vertex 113B). Note that in FIG. 8C, only one of the white circles in patch image 121A is assigned a symbol, but all of the white circles in patch image 121A are vertex 123A. Similarly, only one of the white circles in patch image 121B is assigned a symbol, but all of the white circles in patch image 121B are vertex 123B.

[0074] That is, the vertex 123A of the identification information "A1" and the vertex 123B of the identification information "B1" form one pair. Similarly, the vertex 123A of the identification information "A2" and the vertex 123B of the identification information "B2" form one pair. The vertex 123A of the identification information "A3" and the vertex 123B of the identification information "B3" form one pair. The vertex 123A of the identification information "A4" and the vertex 123B of the identification information "B4" form one pair. The vertex 123A of the identification information "A5" and the vertex 123B of the identification information "B5" form one pair.

[0075] Furthermore, a connection connecting vertices 123A together is referred to as an edge 122A. This edge 122A is the edge corresponding to edge 112A (a projection of edge 112A) and constitutes the boundary of patch image 121A. Similarly, a connection connecting vertices 123B together is referred to as an edge 122B. This edge 122B is the edge corresponding to edge 112B (a projection of edge 112A) and constitutes the boundary of patch image 121B. In FIG. 8C, only one of the connections connecting vertices 123A together in patch image 121A is labeled, but all of the connections connecting vertices 123A are edges 122A. Similarly, only one of the connections connecting vertices 123B together in patch image 121B is labeled, but all of the connections connecting vertices 123B are edges 122B.

[0076] That is, the edge formed by vertex 123A and side 122A and the edge formed by vertex 123B and side 122B indicate edges shared by patch image 121A and patch image 121B. That is, the edge formed by vertex 123A and side 122A and the edge formed by vertex 123B and side 122B form one pair.

[0077] The encoder generates pair information indicating such pairs produced by patch segmentation.

[0078] <Pair information unit> Next, the pair information generated as described above will be described. The pair information may indicate only pairs formed by vertices included in the pair information, as shown in the fifth row from the top of the table in Fig. 5. In other words, the pair information may be information indicating all pairs obtained by performing patch segmentation on a mesh.

[0079] Furthermore, the pair information may also be information indicating pairs formed by vertices not indicated by the pair information, as shown in the sixth row from the top of the table in Fig. 5. In other words, the pair information may be information formed for each group of vertices, and by indicating pairs formed by some of the vertices in that group, the information may indicate all pairs formed by the vertices in that group.

[0080] <Pair information for each pair (all pairs)> First, a case where the pair information indicates only pairs formed by the vertices included in the pair information will be described.

[0081] <Pair transmission order> The transmission order of each pair in the pair information will be explained, that is, the order in which pairs are indicated in the pair information will be explained.

[0082] For example, as shown in A of Fig. 9, assume that patch image 131, patch image 132, and patch image 133 are arranged in geometry image 130. Patch image 131, patch image 132, and patch image 133 each have a vertex 134 indicated by a white circle on their boundary. In A of Fig. 9, only one vertex is assigned a symbol, but all of the vertices indicated by white circles are vertices 134. Numbers indicated near each vertex indicate identification information for that vertex.

[0083] B of FIG. 9 shows pairs formed by vertices 134 using the identification information assigned to the vertices 134. In B of FIG. 9, square frames 141 to 144 show pairs formed by vertices 134. That is, a vertex 134 assigned with identification information "1" and a vertex 134 assigned with identification information "4" form a pair. A vertex 134 assigned with identification information "2" and a vertex 134 assigned with identification information "15" form a pair. A vertex 134 assigned with identification information "5" and a vertex 134 assigned with identification information "8" form a pair. A vertex 134 assigned with identification information "20", a vertex 134 assigned with identification information "21", and a vertex 134 assigned with identification information "24" form a pair.

[0084] The pair information for such pairs may indicate the vertices constituting the pair one by one (for each pair), as shown in the eleventh row from the top of the table in FIG. 5, for example. In other words, the pair information may indicate that multiple vertices in a first patch and corresponding multiple vertices in a second patch are each paired. For example, if the number of vertices constituting a pair is known to the decoder in advance, such as if the number of vertices constituting a pair is limited to two, the pair information may indicate an array of identification information for the vertices arranged for each pair. A decoder that acquires the pair information can easily identify each vertex constituting each pair by dividing the array of vertex identification information by the number of vertices constituting the pair.

[0085] Note that, as shown in the twelfth row from the top of the table in FIG. 5, the pair information may include a parameter indicating the number of vertices constituting each pair. If the number of vertices constituting each pair is indicated, the decoder can easily identify each vertex constituting each pair by dividing the array of vertex identification information included in the pair information by the number of vertices indicated by the parameter. In other words, in this case, the decoder does not need to know in advance the number of vertices constituting each pair. Note that the number of this parameter transmitted as pair information is arbitrary. For example, the value of this parameter may be indicated for each pair, for a predetermined number of pairs equal to or greater than two, or one for all pairs in the pair information. In other words, the number of vertices constituting a pair may be variable. In other words, the number of vertices constituting a pair does not need to be the same for all pairs.

[0086] Furthermore, instead of including a parameter indicating the number of vertices that make up a pair in the pair information, a flag (for example, a 1-bit flag "0") indicating the separation of the pair may be inserted into the array of vertex identification information included in the pair information. This flag is information that is known to the decoder in advance. A decoder that has acquired the pair information can easily determine which vertices make up a pair by detecting the flag from the array of vertex identification information. In other words, in this case too, the decoder does not need to know in advance the number of vertices that make up each pair. Furthermore, the number of vertices that make up a pair may be variable.

[0087] The pair information may also include information indicating a group of vertices in a first patch and a group of vertices in a second patch. For example, as shown in the thirteenth row from the top of the table in Fig. 5, a list of vertices in pairs may be generated for each patch, and the pair information may indicate the pairs in that list. In the example of B in Fig. 9, the vertices 134 in patch image 131 indicated by a rectangular frame 145 and the vertices 134 in patch image 132 indicated by a rectangular frame 146 may be listed, and the pair information may indicate the correspondence between these lists.

[0088] For example, in the pair information, the identification information of vertices may be arranged by group, and an array in which the groups are arranged by pair may be displayed. In this case, the dividing positions of the groups in the array of the identification information of the vertices may be indicated by a parameter indicating the number of vertices constituting the group. The value of this parameter may be indicated for each group, for each predetermined number of groups equal to or greater than two, or one for all groups. In other words, the number of vertices constituting a group may be variable. In other words, the number of vertices constituting a group does not have to be the same for all pairs.

[0089] Furthermore, the position of a group separator in the array of vertex identification information may be indicated by inserting a flag (for example, a 1-bit flag "0") indicating the group separator. This flag is information known to the decoder in advance. A decoder that has acquired pair information can easily determine which vertices constitute a group by detecting the flag from the array of vertex identification information. In other words, in this case too, the decoder does not need to know in advance the number of vertices that constitute each group. Furthermore, the number of vertices that constitute a group may be variable.

[0090] In this way, by enclosing the identification information of the vertices that make up a pair in parentheses or braces, it is possible to clearly indicate the vertices that make up each pair for each group, and therefore the decoder can more easily identify the vertices that make up each pair based on this pair information.

[0091] <Pair information content> In the pair information, information indicating the vertices (or edges) that make up the pair is arbitrary.

[0092] <id> For example, as shown in the 15th row from the top of the table in Fig. 5, the pair information may indicate pairs using identification information corresponding to the state of each vertex in the object before encoding. This identification information may be any information.

[0093] For example, as shown in the 16th row from the top of the table in Fig. 5, this identification information may be identification information (all vertex IDs) for identifying each vertex among the vertices included in the mesh. In other words, this identification information may be information assigned to all vertices included in the mesh.

[0094] For example, as shown in the 17th row from the top of the table in Fig. 5, this identification information may be identification information (intra-patch vertex ID) for identifying each vertex among the vertices included in the patch. In other words, this identification information may be information that is independently assigned to each vertex for each patch.

[0095] In this case, the pair information may further include patch identification information (patch ID), as shown in the 18th row from the top of the table in Fig. 5. The decoder can identify each vertex among the vertices included in the mesh by combining the patch ID and the intra-patch vertex ID.

[0096] In this case, as shown in the 19th row from the top of the table in Fig. 5, the patch ID may be transmitted for each pair. That is, the pair information may include patch identification information for each pair. In other words, in this case, a patch ID is transmitted for each vertex that constitutes each pair. Therefore, it is indicated for each vertex that constitutes each pair which patch the vertex belongs to.

[0097] Furthermore, as shown in the 20th row from the top of the table in FIG. 5, the patch ID may be transmitted for each of a plurality of pairs. That is, the pair information may include patch identification information for each of a plurality of pairs. In this case, the patch ID is transmitted for each vertex constituting some pairs. For example, the patch ID transmitted most recently (the latest patch ID) may be applied to the vertices of the patch for which the patch ID is not transmitted. For example, the patch ID may be transmitted only when the patch ID to be applied changes, and the transmission of the patch ID may be omitted for pairs to which the same patch ID as that of the previous pair is applied. In this way, an increase in the amount of code can be suppressed, and an increase in coding efficiency can be suppressed.

[0098] 5, the identification information may be identification information for identifying each vertex among the vertices located on the boundary of the patch (ID in the boundary vertex list). In other words, this identification information may be information assigned to each vertex located on the boundary of the patch.

[0099] In this case, the encoding unit may further encode a list of vertices located on the boundary of the patch (a boundary vertex list), as shown in the 22nd row from the top of the table in Fig. 5. By referring to this boundary vertex list, the decoder can easily identify vertices among the vertices located on the boundary of the patch.

[0100] Note that the pair information may indicate the pair using the absolute value of the identification information, as shown in the 23rd row from the top of the table in Fig. 5. By expressing the pair in this way, the decoder can more easily grasp the identification information of each vertex that makes up the pair.

[0101] Also, as shown in the bottom row of the table shown in FIG. 5, the pair information may indicate a pair using the relative value of this identification information from the identification information of other vertices. By expressing the pair in this way, the amount of information (code amount) of the pair information can be reduced compared to the case of indicating the pair using the identification information of each vertex constituting the pair. That is, it is possible to suppress a reduction in the coding efficiency of the pair information.

[0102] <UV coordinates> Also, for example, as shown in the top row of the table shown in FIG. 6, the pair information may indicate a pair based on the coordinates of a plurality of vertices in the two-dimensional plane of the image instead of the above-described identification information.

[0103] In that case, as shown in the second row from the top of the table shown in FIG. 6, the pair information may indicate the pair using the UV coordinates of each vertex constituting the pair.

[0104] For example, assume that patches A to D each have vertices #0 to #8 with UV coordinates as shown in the table of FIG. 10. A to D indicate patch IDs, and #0 to #8 indicate in-patch IDs. For example, the UV coordinates of vertex #0 of patch A are (u00, v00), and the UV coordinates of vertex #1 of patch A are (u10, v10). Also, the UV coordinates of vertex #0 of patch B are (u01, v01), and the UV coordinates of vertex #1 of patch B are (u11, v11). Assume that vertex #0 of patch A and vertex #0 of patch B form a pair. Also, assume that vertex #1 of patch A and vertex #1 of patch B form a pair.

[0105] In the pair coordinates, the pairs formed by these vertices may be shown as (u00, v00), (u01, v01) using their respective UV coordinates. By expressing the pair in this way, the decoder can more easily grasp the UV coordinates of each vertex constituting the pair.

[0106] Furthermore, as shown in the third row from the top of the table in FIG. 6, the pair information may indicate a pair using the difference in UV coordinates (relative coordinates (du, dv)). For example, the pair information may indicate a pair using the UV coordinates (u, v) of a reference vertex, which is one of the vertices constituting the pair, and the difference in UV coordinates (relative coordinates (du, dv)), such as (u, v), (du, dv). By expressing the pair in this way, the amount of information (amount of code) in the pair information can be reduced compared to when the pair is indicated using the UV coordinates of each vertex constituting the pair. In other words, it is possible to suppress a decrease in the coding efficiency of the pair information.

[0107] For example, as shown in the fourth row from the top of the table in FIG. 6, the pair information may indicate the relative coordinates of a vertex that is different from the reference vertex, among multiple vertices, using the coordinates of the reference vertex as a reference.

[0108] For example, suppose the patches and vertices shown in Fig. 10 are arranged in the geometry image as shown in Fig. 11. That is, suppose patch image 151 of patch A and patch image 153 of patch B are arranged in the geometry image as shown in Fig. 11. Then, suppose vertex #0 of patch A is arranged at the position of white circle 152, and vertex #0 of patch B is arranged at the position of white circle 154. Furthermore, the relative position of patch B with respect to patch A (the relative vector between the patches) is represented by vector AB.

[0109] In this case, vertex #0 of patch A may be set as the reference vertex, and (u, v) may be set as the coordinates (u00, v00) of the white circle 152. In other words, (u, v) = (u00, v00). Also, the relative coordinates of vertex #0 of patch B seen from vertex #0 of patch A, i.e., vector 155, may be set as (du, dv). In other words, (du, dv) = (u01-u00, v01-v00).

[0110] For example, as shown in the fifth row from the top of the table in Figure 6, the pair information may indicate the relative coordinates of a vertex that is different from the reference vertex based on the position of the relative vector between the first patch and the second patch based on the coordinates of the reference vertex among multiple vertices.

[0111] For example, in FIG. 11, vertex #0 of patch A may be set as the reference vertex, and (u, v) may be set as the coordinates (u00, v00) of the white circle 152. That is, (u, v) = (u00, v00). Also, vector AB may be used to move vertex #0 of patch A to the position it would be if it were a vertex of patch B. That is, vertex #0 of patch A is moved from (u00, v00) as shown by vector 156. Then, the relative coordinates of vertex #0 of patch B as viewed from the position after this movement (the tip of vector 156), i.e., vector 157, may be set as (du, dv). That is, instead of using vector 155, (du, dv) may be expressed as the result of adding vector AB (i.e., vector 156) and vector 157.

[0112] Incidentally, when placing a patch on a geometry image, the orientation of the patch may be corrected. The method for correcting the orientation of the patch is arbitrary. For example, the patch may be flipped upside down, flipped left to right, or rotated by a predetermined angle.

[0113] This type of posture correction is performed so that the absolute values ​​of du and dv become smaller. For example, the posture of patch B may be corrected so that the position of vertex #0 of patch B approaches the position of vertex A of patch A. In the example of FIG. 12, patch B is positioned with its left and right sides flipped. Therefore, the UV coordinates of vertex #0 of patch B are (widthB-u01, v01). WidthB indicates the width of patch B (the horizontal length in the figure).

[0114] For example, if vertex #0 of patch A is the reference vertex, and the relative coordinates of vertex #0 of patch B seen from vertex #0 of patch A are (du, dv), then (du, dv) becomes vector 163 when the posture of patch B is corrected. That is, (du, dv) = (widthB-u01-u00, v01-v00). The length of vector 163 is shorter than vector 155, which further reduces the reduction in encoding efficiency of pair information.

[0115] For example, as shown in the sixth row from the top of the table in Figure 6, the pair information may indicate the relative coordinates of a vertex other than the reference vertex in the second patch in a posture-corrected state, based on the position of the relative vector between the first patch and the second patch, which is based on the coordinates of a reference vertex among the multiple vertices.

[0116] 12, for example, vector AB is used to move vertex #0 of patch A from (u00, v00) as shown by vector 161. The relative coordinates of vertex #0 of patch B as seen from the position after this movement (the tip of vector 161), i.e., vector 162, may be expressed as (du, dv). In other words, instead of using vector 163, (du, dv) may be expressed as the result of adding vector AB (i.e., vector 161) and vector 162.

[0117] For example, as shown in the seventh row from the top of the table in Figure 6, the pair information may indicate the relative coordinates of a vertex of the pair being processed that is different from the reference vertex, based on the relative coordinates of the vertex that is different from the reference vertex of the multiple vertices of another pair, based on the coordinates of the reference vertex.

[0118] For example, suppose that vertex #1 of patch A and vertex #1 of patch B that make up a pair are arranged as shown in Fig. 12. This pair is different from the pair made up of vertex #0 of patch A and vertex #0 of patch B described above.

[0119] When expressing the pair formed by vertex #1 of patch A and vertex #1 of patch B, information on the pair formed by vertex #0 of patch A and vertex #0 of patch B may be used. For example, vertex #1 of patch A is set as the reference vertex, and (u, v) = (u10, v10). Then, vertex #0 of patch A is moved using vector 163 connecting vertex #0 of patch A and vertex #0 of patch B. That is, the position of vertex #1 of patch A is moved according to the positional relationship between vertex #0 of patch A and vertex #0 of patch B. As a result, vertex #1 of patch A moves as shown by vector 164. Then, the position of vertex #1 of patch A moved in this way (the tip of vector 164), that is, the relative coordinates of vertex #0 of patch B as seen from the position of vertex #1 of patch A reflecting the positional relationship between vertex #0 of patch A and vertex #0 of patch B, that is, vector 165, may be set as (du, dv). That is, (du, dv) may be expressed as the result of adding vector 164 and vector 165 .

[0120] In the example of FIG. 12, the orientation of patch B is corrected, but this method can be applied even if the orientation of patch B is not corrected as in the example of FIG.

[0121] <By pair group (some pairs)> Up to this point, we have explained the case where the pair information indicates only pairs formed by vertices included in the pair information. From here on, we will explain the case where the pair information also indicates pairs formed by vertices not indicated by the pair information.

[0122] In this case, the pair information includes information about only some of the pairs. In other words, information about the remaining pairs is omitted. Therefore, the correction unit of the decoder restores the omitted information based on the transmitted pair information.

[0123] In this case, the transmission order of the pairs is the same as that in the case where the pair information indicates only pairs formed by the vertices included in the pair information, as described above in <Transmission Order of Pairs>. In addition, the contents of the pair information in this case are as follows: <Contents of Pair Information> <id>and is the same as the case where only the pairs constituted by the vertices containing the pair information described above in <UV coordinates> are shown.

[0124] <Format of pair information> The information included in the pair information in this case (in other words, the way of information omission) is arbitrary.

[0125] <Three-point pair> For example, as shown in the seventh row from the top of the table shown in FIG. 5, the pair information may include information indicating the correspondence relationship between the combination of the vertices located at both ends of the first edge and the vertices located in the middle of the first edge of the first patch and the combination of the vertices located at both ends of the second edge and the vertices located in the middle of the second edge of the second patch, with respect to the first edge and the second edge respectively corresponding to the boundaries between the first patch and the second patch in the object before encoding. That is, the pair information may have information indicating the pairs constituted by the edge endpoints and the pairs constituted by the midpoints, and the information regarding the pairs constituted by other vertices may be omitted.

[0126] For example, in A of FIG. 13, the patch 170A has vertices 171A, 172A, and 173A on its boundary. Also, the patch 170B has vertices 171B, 172B, and 173B on its boundary. And, vertex 171A and vertex 171B constitute a pair, vertex 172A and vertex 172B constitute a pair, and vertex 173A and vertex 173B constitute a pair. That is, in the mesh, patch 170A and patch 170B are adjacent to each other at the boundary between vertex 171A and vertex 173A (the boundary between vertex 171B and vertex 173B).

[0127] In this case, the boundary between vertex 171A and vertex 173A (the boundary between vertex 171B and vertex 173B) constitutes the edge shared by patch 170A and patch 170B. In other words, vertex 171A and vertex 173A (or vertex 171B and vertex 173B) are the endpoints of the edge. Also, vertex 172A (or vertex 172B) is the midpoint of the edge.

[0128] In patch 170A, a vertex other than vertex 172A may exist at the boundary (edge) between vertex 171A and vertex 173A, as in the example shown in B of Fig. 13. Similarly, in patch 170B, a vertex other than vertex 172B may exist at the boundary (edge) between vertex 171B and vertex 173B, as in the example shown in B of Fig. 13.

[0129] However, the pair information stores information about pairs formed by the six vertices described above. For example, in the case of B in Fig. 13, information about the pair formed by vertices 171A and 171B, the pair formed by vertices 172A and 172B, and the pair formed by vertices 173A and 173A is stored in the pair information. Information about pairs formed by other vertices indicated by dotted circles in B in Fig. 13 is omitted.

[0130] In this way, by including only information about pairs formed by both end points and the midpoint of an edge, the amount of code for the pair information can be reduced compared to when information about pairs formed by all vertices of the edge is included, which means that a decrease in the coding efficiency of the pair information can be suppressed.

[0131] Any vertex between both end points (also called both endpoints) of the edge may be set as the midpoint.

[0132] The decoder's correction unit not only obtains information about these three pairs from the pair information, but also restores (interpolates) the other vertices that exist on the edge from the end points and midpoint of the edge, and obtains information about the pairs formed by those vertices.

[0133] <2-piece pair> Furthermore, for example, as shown in the eighth row from the top of the table in FIG. 5, the pair information may include information indicating the correspondence between the scanning direction for the first edge at the time of decoding and the scanning direction for the second edge at the time of decoding, for the first edge of the first patch and the second edge of the second patch, which correspond respectively to the boundary between the first patch and the second patch in the object before encoding.

[0134] For example, the scan direction of the vertices located on the boundary of a patch is fixed, the first vertex of the edge in the scan order is the start point, and the last vertex of the edge is the end point. Then, in the pair information, the edge is expressed as (start point, end point), and a pair is indicated for each edge.

[0135] For example, in Fig. 14, the scan order is the direction of the arrow (counterclockwise). In this case, the boundary (edge) on the left side of the figure between a vertex assigned with identification information "15" and a vertex assigned with identification information "8" in patch 181 is expressed as (15, 8). In other words, the vertex assigned with identification information "15" is the start point of the edge, and the vertex assigned with identification information "8" is the end point of the edge.

[0136] The pair information includes information about the pair that this edge (15, 8) forms, and information about the pairs that other vertices 182 that exist on that edge form is omitted.

[0137] Similarly, the boundary (edge) on the right side of the figure between the vertex assigned the identification information "15" and the vertex assigned the identification information "8" in patch 183 is expressed as (8, 15). In other words, the vertex assigned the identification information "8" is the start point of the edge, and the vertex assigned the identification information "15" is the end point of the edge. The pair information includes information about the pair formed by this (8, 15). Information about the pairs formed by the other vertices 184 on that edge is omitted.

[0138] For example, the pair information expresses a pair consisting of (start point 1, end point 1) and (start point 2, end point 2) as (start point 1, end point 1), (start point 2, end point 2). In this case, it is necessary to indicate whether the start points (end points) form a pair, or whether the start point and end point form a pair.

[0139] Therefore, the pair information includes flag information indicating the combination of the start point and end point in the pair. For example, if this flag is false (e.g., value "0"), it indicates that the pair correspondence is the same as the scanning direction, i.e., the viewpoints form a pair and the end points form a pair. On the other hand, if this flag is true (e.g., value "1"), it indicates that the pair correspondence is opposite to the scanning direction, i.e., the viewpoint and end point form a pair. For example, the pair information may be expressed as (start point 1, end point 1), (start point 2, end point 2), "flag value".

[0140] 15, the boundary on the right side of patch 190A between a vertex assigned with identification information "2" and a vertex assigned with identification information "5" and the boundary on the left side of patch 190B between a vertex assigned with identification information "15" and a vertex assigned with identification information "8" form an edge shared by patch 190A and patch 190B. In other words, as indicated by dotted double-headed arrow 193, the vertex assigned with identification information "2" and the vertex assigned with identification information "15" form a pair, and as indicated by dotted double-headed arrow 194, the vertex assigned with identification information "5" and the vertex assigned with identification information "8" form a pair.

[0141] In this case, if the scanning direction is counterclockwise, the edge of patch 190A has a vertex assigned with identification information "5" as its start point and a vertex assigned with identification information "2" as its end point, and is expressed as (5,2). Similarly, the edge of patch 190B has a vertex assigned with identification information "15" as its start point and a vertex assigned with identification information "8" as its end point, and is expressed as (15,8). In other words, the pair indicated by dotted double-headed arrow 193 is composed of the start point and the end point. Similarly, the pair indicated by dotted double-headed arrow 194 is also composed of the start point and the end point. In other words, the flag for this edge pair is true (value "1"). Therefore, in the pair information, this edge pair may be expressed as (5,2), (15,8), 1.

[0142] 15, the boundary on the right side of the figure between a vertex assigned with identification information "5" and a vertex assigned with identification information "21" in patch 190A, and the boundary on the right side of the figure between a vertex assigned with identification information "24" and a vertex assigned with identification information "20" in patch 190C constitute an edge shared by patch 190A and patch 190C. In other words, as indicated by solid double-headed arrow 195, the vertex assigned with identification information "5" and the vertex assigned with identification information "24" form a pair, and as indicated by solid double-headed arrow 196, the vertex assigned with identification information "21" and the vertex assigned with identification information "20" form a pair.

[0143] In this case, if the scanning direction is counterclockwise, the edge of patch 190A has a vertex assigned with identification information "21" as its start point and a vertex assigned with identification information "5" as its end point, and is expressed as (21,5). Similarly, the edge of patch 190C has a vertex assigned with identification information "20" as its start point and a vertex assigned with identification information "24" as its end point, and is expressed as (20,24). In other words, the pair indicated by solid double-headed arrow 195 is made up of the end points of both edges. Similarly, the pair indicated by solid double-headed arrow 196 is made up of the start points of both edges. In other words, the flag for this edge pair is false (value "0"). Therefore, in the pair information, this edge pair may be expressed as (21,5), (20,24), 0.

[0144] Therefore, in this case, information regarding the pair formed by the other vertex 191A included in the above-mentioned edge of patch 190A, the other vertex 191B included in the above-mentioned edge of patch 190B, and the other vertex 191C included in the above-mentioned edge of patch 190C is omitted from the pair information.

[0145] In this way, by including only information about pairs formed by the start and end points of edges and flags indicating the correspondence between the pairs, the amount of code for the pair information can be reduced compared to when information about pairs formed by all vertices of edges is included, which means that a decrease in the coding efficiency of the pair information can be suppressed.

[0146] The decoder's correction unit not only obtains information about these two pairs for each edge from the pair information, but also restores (interpolates) the other vertices that exist on that edge from the start and end points of the edge, and obtains information about the pairs formed by those vertices.

[0147] <Vertex Interpolation> Next, we will explain the interpolation of vertices by the decoder described in <3-point pair> and <2-point pair>. When pair information in the format described in <3-point pair> and <2-point pair> is acquired, the correction unit of the decoder interpolates vertices not shown in the pair information and performs pairing processing on those points as well. In other words, the correction unit of the decoder interpolates vertices that make up a pair but are not included in the pair information.

[0148] The vertex interpolation method is arbitrary. For example, as shown in the ninth row from the top of the table in FIG. 6, the decoder correction unit may perform vertex interpolation based on the occupancy image. In this case, the decoder correction unit identifies vertices on the boundary of the patch using the occupancy image obtained by decoding and obtains their UV coordinates. Then, the decoder correction unit identifies the boundary connecting each of the identified vertices based on the connectivity.

[0149] For example, if the pair information includes information only about pairs of endpoints and a single midpoint, the decoder correction unit searches for vertices along the identified boundary from the midpoint toward the endpoints. For example, in FIG. 16, assume that the polygon shown in gray is a patch, vertex 201 is the midpoint, and vertices 202 and 203 are endpoints. The decoder correction unit searches for vertices along the identified boundary from vertex 201, which is the midpoint, toward vertices 202 and 203, which are endpoints, as indicated by the arrows. This search detects vertices indicated by white circles in the figure. The decoder correction unit performs pairing processing on the detected vertices.

[0150] In this way, the decoder correction unit can perform pairing processing for all vertices present on an edge based on pair information that includes information about only pairs of both end points and a single intermediate point, thereby making it possible to suppress a decrease in coding efficiency compared to when the pair information includes information about pairs formed by all vertices of an edge.

[0151] For example, if the pair information includes information related only to pairs of start and end points and flag information indicating the combination of the start and end points in the pair, the decoder correction unit searches for a vertex along the identified boundary in the specified scanning direction from the start point to the end point. For example, in FIG. 17, the gray polygon is a patch, with vertex 211 being the start point and vertex 212 being the end point. The decoder correction unit searches for a vertex along the identified boundary in the specified scanning direction from vertex 211, the start point, to vertex 212, the end point, as indicated by the arrow. The determination of whether the specified scanning direction is met is based on the connectivity information of the boundary polygon. That is, the decoder correction unit searches for the side of edges 213 and 214 connected to vertex 211, the start point, that matches the scanning direction of the boundary polygon (edge ​​213 in this case). This search detects the vertex indicated by the white circle in the figure. The decoder correction unit then performs pairing processing on the detected vertices.

[0152] In this way, the decoder correction unit can perform pairing processing for all vertices present in an edge based on pair information including information about only pairs of start and end vertices and flag information indicating the combination of the start and end vertices in the pair. Therefore, it is possible to suppress a decrease in coding efficiency compared to when the pair information includes information about pairs formed by all vertices of an edge.

[0153] The decoder correction unit may interpolate the vertices based on connectivity information instead of the occupancy image, as shown in the tenth row from the top of the table in Figure 6. In this case, the decoder correction unit identifies polygons that have no adjacent polygons (triangles) within the patch as boundary polygons based on the connectivity. The decoder correction unit also identifies the boundary of the patch from the identified boundary polygon. The decoder correction unit searches for vertices along the identified boundary of the patch from the midpoint toward both end points as described above, and performs pairing processing on the detected vertices.

[0154] The method for searching for the vertex is the same as the example described above with reference to FIG. 16 or FIG.

[0155] <Pairing process> Next, the pairing process will be described. The correction unit of the decoder performs the pairing process based on the pair information. In this pairing process, the vertex matching method can be any method.

[0156] For example, as shown in the 12th row from the top of the table in Figure 6, the correction unit of the decoder may delete one of the vertices of the first patch and the second patch that are indicated to be a pair by the pair information in the reconstructed mesh, and convert the connection of the deleted vertex to the connection of the other vertex.

[0157] For example, in A of Fig. 18, it is assumed that identification information "1," "2," or "3" is assigned to three vertices of polygon 221. It is also assumed that identification information "4," "5," or "6" is assigned to three vertices of polygon 222. It is also assumed that, based on the pair information, the vertex assigned with identification information "3" and the vertex assigned with identification information "6" form a pair, and the vertex assigned with identification information "2" and the vertex assigned with identification information "4" form a pair.

[0158] In such a case, the decoder correction unit first sets the position of a single vertex after transformation based on the pair information. Then, the decoder correction unit updates the vertex information, leaving one of the multiple vertices that make up the pair and erasing the other vertices. For example, as shown in A of Figure 18, the decoder correction unit erases the vertex assigned with identification information "6" and the vertex assigned with identification information "4". The decoder correction unit updates the coordinates of the remaining vertices as necessary.

[0159] Then, the correction unit of the decoder updates the connectivity and converts connections (edges) with the deleted vertices into connections (edges) with the remaining vertices. For example, as shown in B of Fig. 18, the correction unit of the decoder converts a connection connecting a vertex assigned with identification information "6" and a vertex assigned with identification information "5" into a connection connecting a vertex assigned with identification information "3" and a vertex assigned with identification information "5". Similarly, the correction unit of the decoder converts a connection connecting a vertex assigned with identification information "4" and a vertex assigned with identification information "5" into a connection connecting a vertex assigned with identification information "2" and a vertex assigned with identification information "5".

[0160] By the correction unit of the decoder performing the pairing process in this way, the polygon 221 and the polygon 222 become adjacent to each other, as shown in B of Fig. 18. Therefore, the crack between the polygon 221 and the polygon 222 is eliminated.

[0161] By having the decoder's correction unit perform such pairing processing based on the pair information, the decoder can more reliably eliminate cracks. In other words, it can more reliably suppress the occurrence of cracks. Therefore, it is possible to suppress the degradation of 3D data quality due to encoding and decoding.

[0162] Also, for example, as shown in the 13th row from the top of the table in Figure 6, the correction unit of the decoder may move one of the vertices of the first patch and the second patch, which are indicated to be a pair by the pair information, to the position of the other vertex in the reconstructed mesh.

[0163] In such a case, the decoder correction unit first sets the position of the overlapping point based on the pair information. Then, the decoder correction unit updates the vertex information and updates the coordinates of the multiple vertices that make up the pair to the coordinates of the overlapping point. For example, as shown in A of FIG. 19, the decoder correction unit converts the coordinates of the vertex assigned with identification information "3" and the coordinates of the vertex assigned with identification information "6" into the same coordinates. As a result, as shown in B of FIG. 19, both vertices become overlapping points. Similarly, the decoder correction unit converts the coordinates of the vertex assigned with identification information "2" and the coordinates of the vertex assigned with identification information "4" into the same coordinates. As a result, as shown in B of FIG. 19, both vertices become overlapping points.

[0164] By the correction unit of the decoder performing the pairing process in this way, the polygon 221 and the polygon 222 become adjacent to each other, as shown in B of Fig. 19. Therefore, the crack between the polygon 221 and the polygon 222 is eliminated.

[0165] By having the decoder's correction unit perform such pairing processing based on the pair information, the decoder can more reliably eliminate cracks. In other words, it can more reliably suppress the occurrence of cracks. Therefore, it is possible to suppress the degradation of 3D data quality due to encoding and decoding.

[0166] Furthermore, for example, as shown in the 14th row from the top of the table in Figure 6, the correction unit of the decoder may form a polygon in the reconstructed mesh that includes vertices of the first patch and vertices of the second patch that are indicated as a pair by the pair information.

[0167] For example, as shown in A of Figure 20, if a crack occurs between polygon 221 and polygon 222, the correction unit of the decoder forms a new polygon using vertex 231, vertex 233, vertex 232, and vertex 234 that form a pair based on the pair information.

[0168] The correction unit of the decoder updates the connectivity to connect vertex 231, vertex 233, vertex 232, and vertex 234. That is, as shown in B of Fig. 20, the correction unit of the decoder generates polygon 241 having vertices 231, vertex 232, and vertex 233 as vertices, and polygon 242 having vertices 232, vertex 234, and vertex 233 as vertices. In doing so, the correction unit of the decoder sets the order of the vertices of the connectivity so that the boundary polygon and the surface face the same direction.

[0169] The decoder correction unit then applies a texture to the added polygon. Any texture may be used, but by using a texture that matches the textures of the surrounding polygons, degradation of quality can be suppressed. For example, the decoder correction unit may copy the color of the surrounding boundary polygons and apply the resulting texture to the added polygon. Alternatively, the decoder correction unit may calculate a (weighted) average of the colors of multiple surrounding boundary polygons and apply the derived texture to the added polygon. Furthermore, a dedicated texture to be applied to the added polygon may be transmitted from the encoder to the decoder. The decoder correction unit may then apply the transmitted texture to the added polygon.

[0170] As described above, by adding polygon 241 and polygon 242, the gap between polygon 221 and polygon 222 is filled and the crack is removed. In other words, by having the correction unit of the decoder perform pairing processing based on the pair information in this way, the decoder can more reliably remove the crack between polygon 221 and polygon 222. In other words, the occurrence of cracks can be more reliably suppressed. Therefore, it is possible to suppress a decrease in the quality of 3D data due to encoding and decoding.

[0171] When a large number of vertices form pairs, the correction unit of the decoder may generate polygons using three vertices each, as shown in C of FIG.

[0172] Furthermore, in the pairing process, the correction unit of the decoder may be configured to select and apply any one of the above-described methods. The selection method is arbitrary.

[0173] For example, when pairing a pair of vertices that was a single vertex before encoding, or when it is not necessary to maintain overlapping points, the decoder correction unit may convert the pair of vertices into a single vertex. Furthermore, when there is an overlapping point before encoding and it is desired to maintain the overlapping point, the decoder correction unit may convert the pair of vertices into overlapping points. Furthermore, when subjective evaluation shows that applying a mesh is better than moving points, the decoder correction unit may add a polygon using the pair of vertices. Furthermore, information specifying the pairing method may be transmitted from the encoder to the decoder. For example, this information may be transmitted in any unit, such as a sequence unit, a list unit, or a pair unit.

[0174] <Coordinate determination method> As described above, in the pairing process, when vertices forming a pair are converted into one vertex, the method of determining the coordinates of the converted vertex is arbitrary.

[0175] For example, as shown in the 16th row from the top of the table in Fig. 6, the correction unit of the decoder may set the position of a vertex after conversion to a single vertex to the position of any one of the multiple vertices before conversion (i.e., the multiple vertices that make up a pair). In other words, another vertex may be moved to the position of any one of the multiple vertices that make up a pair.

[0176] In this case, the method of selecting a vertex to which another vertex is to be moved (the method of determining the position of one of the vertices forming a pair to which the other vertex is to be moved) is arbitrary. For example, as shown in the 17th row from the top of the table in Figure 6, information indicating the vertex to be selected as the destination may be transmitted from the encoder to the decoder. In other words, this information indicates one of the multiple vertices before transformation.

[0177] The decoder correction unit may then set the position of a single vertex after the transformation based on information indicating one of the multiple vertices before the transformation. In other words, the decoder performs pairing processing to move another vertex to the position of the vertex specified by the information.

[0178] In this way, the decoder's correction unit selects the vertex to be moved to in accordance with the encoder's instructions, so that the decoder can prevent the coordinates used from changing depending on the order of reconstruction.

[0179] Also, for example, as shown in the bottom row of the table in Figure 6, the correction unit of the decoder may set the position of the vertex after converting it into a single vertex to the (weighted) average position of the positions of the multiple vertices before conversion (the coordinates of each vertex that makes up a pair).

[0180] Similarly, in the pairing process, when the vertices that make up a pair are made into overlapping points, the method of determining the coordinates of the overlapping points is also arbitrary.

[0181] For example, as shown in the 16th row from the top of the table in FIG. 6, the correction unit of the decoder may set the position of the overlapping point to the position of one of the multiple vertices that make up the pair.

[0182] In this case, the vertex can be selected in any manner. For example, as shown in the 17th row from the top of the table in Fig. 6, information indicating the vertex to be selected as the position of the overlapping point may be transmitted from the encoder to the decoder. In other words, this information indicates one of the multiple vertices before the transformation.

[0183] The decoder correction unit may then set the position of the overlapping point based on information indicating one of the multiple vertices before the transformation. That is, the decoder correction unit performs pairing processing so as to form the overlapping point at the position of the vertex specified by the information.

[0184] By using this method, for example, the encoder can generate this information based on the mesh before encoding and transmit it to the decoder, and the decoder can then set overlap points based on that information, which makes it easier for the decoder to reconstruct the state before the crack occurred.

[0185] Also, for example, as shown in the bottom row of the table in Figure 6, the correction unit of the decoder may set the position of the overlapping point to the (weighted) average position of the positions (coordinates) of the multiple vertices that make up the pair.

[0186] <Application example> The pair information may be transmitted for all vertices (or edges) constituting a pair, or may be transmitted for some vertices (or edges). By transmitting the pair information for only some vertices, it is possible to suppress an increase in the amount of code for the pair information. In other words, it is possible to suppress a decrease in the coding efficiency of the pair information.

[0187] For example, pair information may be transmitted only for pairs that could not be corrected by other correction methods such as zippering, etc. In this case, the encoder may have a function of decoding the coded data of the 3D data, identifying portions that cannot be corrected by performing zippering or the like, and generating pair information for the pairs in the identified portions.

[0188] 3. First Embodiment <Encoding device> The present technology described above may be applied to any device. For example, the present technology may be applied to an encoding device 300 as shown in FIG. 21. FIG. 21 is a block diagram showing an example of the configuration of an encoding device that is an embodiment of an information processing device to which the present technology is applied. The encoding device 300 shown in FIG. 21 is a device that encodes 3D data using a mesh by extending a VPCC as a video frame using an encoding method for 2D images. In this case, the encoding device 300 performs encoding by applying a single method or a combination of multiple methods from among the various methods of the present technology described above.

[0189] Note that Fig. 21 shows the main processing units, data flows, etc., and does not necessarily show everything. In other words, in encoding device 300, there may be processing units that are not shown as blocks in Fig. 21, and there may be processing and data flows that are not shown as arrows, etc. in Fig. 21.

[0190] 21 , the encoding device 300 includes a mesh voxelization unit 311, a patch generation unit 312, a paired information generation unit 313, a geometry image generation unit 314, an occupancy image generation unit 315, a paired information encoding unit 316, a meta information encoding unit 317, a 2D encoding unit 318, a 2D encoding unit 319, a 2D encoding unit 320, and a multiplexing unit 321. In this disclosure, the geometry image generation unit 314 and the occupancy image generation unit 315 may be regarded as an image generation unit 331. In this disclosure, the paired information encoding unit 316, the meta information encoding unit 317, the 2D encoding unit 318, the 2D encoding unit 319, and the 2D encoding unit 320 may be regarded as an encoding unit 332. In this disclosure, the paired information generation unit 313 may also be referred to simply as a generation unit. In this disclosure, the paired information encoding unit 316 may also be referred to simply as an encoding unit.

[0191] The encoding device 300 is supplied with connectivity 351, vertex information 352, UV map 353, and texture 354 as 3D data using a mesh.

[0192] Connectivity 351 is information similar to connectivity 32 (Fig. 2) and indicates each vertex that forms a polygon (each vertex that connects to another vertex) for each polygon. Vertex information 352 is information similar to vertex information 31 (Fig. 2) and indicates the coordinates of each vertex that forms a mesh. UV map 353 is information similar to UV map 34 (Fig. 2) and indicates the position of each vertex on the texture image. Texture 354 is information similar to texture image 33 (Fig. 2) and indicates the texture to be applied to the polygon. In other words, texture 354 is information that includes a texture image.

[0193] The mesh voxelization unit 311 acquires vertex information 352 supplied to the encoding device 300. The mesh voxelization unit 311 converts the coordinates of each vertex included in the acquired vertex information 352 into a voxel grid. The mesh voxelization unit 311 supplies the vertex information 352 of the converted voxel grid to the patch generation unit 312.

[0194] The patch generation unit 312 acquires the connectivity 351 and UV map 353 supplied to the encoding device 300. The patch generation unit 312 also acquires voxel grid vertex information 352 supplied from the mesh voxelization unit 311. The patch generation unit 312 generates a geometry patch based on this information. The patch generation unit 312 also projects the generated geometry patch onto a projection plane to generate a patch image.

[0195] The patch generation unit 312 supplies information such as the generated patch image, connectivity 351, vertex information 352, and UV map 353 to the pair information generation unit 313. The patch generation unit 312 also supplies information such as the connectivity 351 and UV map 353 as meta information to the meta information encoding unit 317. The patch generation unit 312 also supplies the generated patch image to the geometry image generation unit 314. The patch generation unit 312 also supplies the generated patch image to the occupancy image generation unit 315.

[0196] The pair information generation unit 313 acquires information such as the patch image, connectivity 351, vertex information 352, and UV map 353 supplied from the patch generation unit 312. Based on this information, the pair information generation unit 313 generates pair information indicating pairs each consisting of one vertex or multiple vertices that were overlapping points in the mesh. This pair information may include information such as that described in <2. Transmission of Pair Information> (including the sections <Generation of Pair Information> to <Application Examples>). The pair information generation unit 313 generates this pair information as described in the section <Generation of Pair Information> and the like.

[0197] In other words, the pair information generation unit 313 (generation unit) divides a mesh representing an object with a three-dimensional structure into at least a first patch and a second patch, arranges them within a single image, and generates pair information indicating that at least one vertex of the first patch is paired with at least one vertex of the second patch that is located at the same position as the vertex of the first patch in the object before encoding.

[0198] The pair information may be information indicating only the pairs formed by the vertices included in the pair information.

[0199] This pair information may include information indicating the correspondence between the combination of vertices located at both ends of the first edge and the vertex located midway between the first edge, and the combination of vertices located at both ends of the second edge and the vertex located midway between the second edge, for the first edge of the first patch and the second edge of the second patch, which correspond respectively to the boundary between the first patch and the second patch in the object before encoding.

[0200] This pair information may include information indicating the correspondence between the scanning direction for the first edge at the time of decoding and the scanning direction for the second edge at the time of decoding, for the first edge of the first patch and the second edge of the second patch, which correspond respectively to the boundary between the first patch and the second patch in the object before encoding.

[0201] The pair information may indicate that a plurality of vertices in the first patch and a corresponding plurality of vertices in the second patch are each paired.

[0202] This pair information may include a parameter indicating the number of vertices that make up the pair.

[0203] This pair information may include information indicating a group of vertices in the first patch and a group of vertices in the second patch.

[0204] This pair information may indicate the pair using identification information corresponding to the state of each vertex in the object before encoding.

[0205] This pair information may indicate pairs based on the coordinates of a plurality of vertices on a two-dimensional plane of the image.

[0206] This pair information may indicate the relative coordinates of a vertex that is different from the reference vertex, using the coordinates of the reference vertex as a reference among the plurality of vertices.

[0207] This pair information may indicate the relative coordinates of a vertex that is different from the reference vertex, based on the position of a relative vector between the first patch and the second patch, which is based on the coordinates of the reference vertex among the multiple vertices.

[0208] This pair information may indicate the relative coordinates of a vertex different from the reference vertex in the second patch in a posture-corrected state, based on the position of a relative vector between the first patch and the second patch, which is based on the coordinates of a reference vertex among the multiple vertices.

[0209] This pair information may indicate the relative coordinates of a vertex of the pair being processed that is different from the reference vertex, based on the relative coordinates of the vertex that is different from the reference vertex of the other pair, based on the coordinates of the reference vertex among the multiple vertices of the other pair.

[0210] The pair information generating unit 313 supplies the generated pair information to the pair information encoding unit 316 .

[0211] The image generation unit 331 performs processing related to the generation of images (frame images). The geometry image generation unit 314 acquires patch images supplied from the patch generation unit 312. The geometry image generation unit 314 arranges the patch images on a two-dimensional plane to generate a geometry image. The geometry image generation unit 314 supplies the geometry image to the 2D encoding unit 318 as a geometry video frame.

[0212] The occupancy image generation unit 315 acquires the patch image supplied from the patch generation unit 312. The occupancy image generation unit 315 generates an occupancy image using the patch image. The occupancy image generation unit 315 supplies the generated occupancy image to the 2D encoding unit 319.

[0213] The encoding unit 332 performs processing related to encoding. The pair information encoding unit 316 (encoding unit) acquires pair information supplied from the pair information generation unit 313. The pair information encoding unit 316 encodes the acquired pair information using a predetermined encoding method to generate encoded data of the pair information. This encoding method is arbitrary. The pair information encoding unit 316 supplies the generated encoded data of the pair information to the multiplexing unit 321.

[0214] The meta information encoder 317 acquires the meta information (including the connectivity 351 and the UV map 353) supplied from the patch generator 312. The meta information encoder 317 encodes the acquired meta information to generate encoded data of the meta information. The meta information encoder 317 supplies the generated encoded data of the meta information to the multiplexer 321.

[0215] The 2D encoding unit 318 acquires the geometry image supplied from the geometry image generation unit 314. The 2D encoding unit 318 encodes the acquired geometry image using a 2D image encoding method to generate encoded data for the geometry image. The 2D encoding unit 318 supplies the generated encoded data of the geometry image to the multiplexing unit 321.

[0216] The 2D encoding unit 319 acquires the occupancy image supplied from the occupancy image generation unit 315. The 2D encoding unit 319 encodes the acquired occupancy image using an encoding method for 2D images to generate encoded data for the occupancy image. The 2D encoding unit 319 supplies the generated encoded data of the occupancy image to the multiplexing unit 321.

[0217] The 2D encoding unit 320 acquires the texture 354 supplied to the encoding device 300. The 2D encoding unit 320 encodes the acquired texture 354 (i.e., texture image) using a 2D image encoding method to generate encoded data for the texture image. The 2D encoding unit 320 supplies the generated encoded data for the texture image to the multiplexing unit 321.

[0218] The multiplexing unit 321 acquires the coded data of the paired information supplied from the paired information encoding unit 316. The multiplexing unit 321 also acquires the coded data of the meta information supplied from the meta information encoding unit 317. The multiplexing unit 321 also acquires the coded data of the geometry image supplied from the 2D encoding unit 318. The multiplexing unit 321 also acquires the coded data of the occupancy image supplied from the 2D encoding unit 319. The multiplexing unit 321 also acquires the coded data of the texture image supplied from the 2D encoding unit 320. The multiplexing unit 321 multiplexes the acquired information to generate a single bit stream. The multiplexing unit 321 outputs the generated bit stream to the outside of the encoding device 300.

[0219] These processing units (mesh voxelization unit 311 to multiplexing unit 321) may have any configuration. For example, each processing unit may be configured with a logic circuit that realizes the above-described processing. Furthermore, each processing unit may have, for example, a central processing unit (CPU), read-only memory (ROM), random access memory (RAM), etc., and may execute a program using these to realize the above-described processing. Of course, each processing unit may have both of these configurations, and may implement part of the above-described processing using a logic circuit and other parts by executing a program. The configurations of each processing unit may be independent of each other. For example, some processing units may implement part of the above-described processing using a logic circuit, other processing units may implement the above-described processing by executing a program, and still other processing units may implement the above-described processing using both a logic circuit and by executing a program.

[0220] <Encoding process flow> An example of the flow of the encoding process executed by this encoding device 300 will be described with reference to the flowchart of FIG.

[0221] When the encoding process starts, in step S301, the mesh voxelization unit 311 converts the coordinates of each vertex included in the vertex information 352 into a voxel grid, thereby converting the mesh into a voxel grid.

[0222] In step S302, the patch generation unit 312 generates a patch using the vertex information 352 and the like that was converted into a voxel grid in step S301, and projects the generated patch onto the projection surface to generate a patch image.

[0223] In step S303, the pair information generation unit 313 generates pair information indicating pairs each consisting of one vertex or multiple vertices that were overlapping points in the mesh, by appropriately using information such as connectivity 351, vertex information 352 converted into a voxel grid in step S301, and UV map 353. This pair information may include information such as that described in <2. Transmission of Pair Information> (including sections <Generation of Pair Information> to <Application Examples>). The pair information generation unit 313 generates this pair information as described in section <Generation of Pair Information> and the like.

[0224] In other words, the pair information generation unit 313 (generation unit) divides a mesh representing an object with a three-dimensional structure into at least a first patch and a second patch, arranges them within a single image, and generates pair information indicating that at least one vertex of the first patch is paired with at least one vertex of the second patch that is located at the same position as the vertex of the first patch in the object before encoding.

[0225] The pair information may be information indicating only the pairs formed by the vertices included in the pair information.

[0226] This pair information may include information indicating the correspondence between the combination of vertices located at both ends of the first edge and the vertex located midway between the first edge, and the combination of vertices located at both ends of the second edge and the vertex located midway between the second edge, for the first edge of the first patch and the second edge of the second patch, which correspond respectively to the boundary between the first patch and the second patch in the object before encoding.

[0227] This pair information may include information indicating the correspondence between the scanning direction for the first edge at the time of decoding and the scanning direction for the second edge at the time of decoding, for the first edge of the first patch and the second edge of the second patch, which correspond respectively to the boundary between the first patch and the second patch in the object before encoding.

[0228] The pair information may indicate that a plurality of vertices in the first patch and a corresponding plurality of vertices in the second patch are each paired.

[0229] This pair information may include a parameter indicating the number of vertices that make up the pair.

[0230] This pair information may include information indicating a group of vertices in the first patch and a group of vertices in the second patch.

[0231] This pair information may indicate the pair using identification information corresponding to the state of each vertex in the object before encoding.

[0232] This pair information may indicate pairs based on the coordinates of a plurality of vertices on a two-dimensional plane of the image.

[0233] This pair information may indicate the relative coordinates of a vertex that is different from the reference vertex, using the coordinates of the reference vertex as a reference among the plurality of vertices.

[0234] This pair information may indicate the relative coordinates of a vertex that is different from the reference vertex, based on the position of a relative vector between the first patch and the second patch, which is based on the coordinates of the reference vertex among the multiple vertices.

[0235] This pair information may indicate the relative coordinates of a vertex different from the reference vertex in the second patch in a posture-corrected state, based on the position of a relative vector between the first patch and the second patch, which is based on the coordinates of a reference vertex among the multiple vertices.

[0236] This pair information may indicate the relative coordinates of a vertex of the pair being processed that is different from the reference vertex, based on the relative coordinates of the vertex that is different from the reference vertex of the other pair, based on the coordinates of the reference vertex among the multiple vertices of the other pair.

[0237] In step S304, the geometry image generation unit 314 generates a geometry image based on the patch image generated in step S302 and the UV map 353.

[0238] In step S305, the occupancy image generation unit 315 generates an occupancy image corresponding to the geometry image generated in step S304.

[0239] In step S306, the pair information encoding unit 316 (encoding unit) encodes the pair information generated in step S303 using a predetermined encoding method to generate encoded data of the pair information. This encoding method is arbitrary.

[0240] In step S307, the meta information encoding unit 317 encodes the meta information (the connectivity 351, the UV map 353, etc.) to generate encoded data of the meta information.

[0241] In step S308, the 2D encoding unit 318 encodes the geometry image generated in step S304 to generate encoded data of the geometry image.

[0242] In step S309, the 2D encoding unit 319 encodes the occupancy image generated by the occupancy image generation unit 315 in step S305, and generates encoded data of the occupancy image.

[0243] In step S310, the 2D encoding unit 320 encodes the texture 354 (that is, the texture image) to generate encoded data of the texture image.

[0244] In step S311, the multiplexing unit 321 multiplexes the coded data of the pair information generated in step S306, the coded data of the meta information generated in step S307, the coded data of the geometry image generated in step S308, the coded data of the occupancy image generated in step S309, and the coded data of the texture image generated in step S310 to generate one bit stream. The multiplexing unit 321 outputs the generated bit stream to the outside of the encoding device 300.

[0245] When the process of step S311 ends, the encoding process ends.

[0246] The encoding device 300 has the above configuration, performs the above-described various processes, and transmits pair information to the decoder, so that the decoder that decodes this bitstream can more reliably suppress the occurrence of cracks and perform pairing processing between more appropriate vertices. Therefore, the encoding device 300 can suppress degradation of 3D data quality due to encoding and decoding.

[0247] 4. Second Embodiment <Decryption device> The present technology can also be applied to, for example, a decoding device 400 as shown in Fig. 23. Fig. 23 is a block diagram showing an example of the configuration of a decoding device, which is one aspect of an image processing device to which the present technology is applied. The decoding device 400 shown in Fig. 23 is a device that decodes, by a decoding method for 2D images, coded data that has been coded using a 2D image coding method to generate 3D data using meshes by extending VPCCs as video frames, and generates (reconstructs) 3D data using meshes. In this case, the decoding device 400 applies a single method or a combination of multiple methods from among the various methods of the present technology described above to decode the coded data and reconstruct the 3D data.

[0248] Note that Fig. 23 shows the main processing units, data flows, etc., and does not necessarily show everything. That is, in the decoding device 400, there may be processing units that are not shown as blocks in Fig. 23, and there may be processing and data flows that are not shown as arrows, etc. in Fig. 23.

[0249] 23 , the decoding device 400 includes a demultiplexing unit 411, a pairing information decoding unit 412, a meta information decoding unit 413, a 2D decoding unit 414, a 2D decoding unit 415, a 2D decoding unit 416, a patch reconstruction unit 417, a vertex information reconstruction unit 418, and a pairing processing unit 419. In this disclosure, the pairing information decoding unit 412, the meta information decoding unit 413, the 2D decoding unit 414, the 2D decoding unit 415, and the 2D decoding unit 416 may be regarded as a decoding unit 431. In this disclosure, the pairing information decoding unit 412 may also be simply referred to as a decoding unit. In addition, in this disclosure, the pairing processing unit 419 may also be referred to as a correction unit.

[0250] The demultiplexing unit 411 obtains a bitstream to be input to the decoding device 400. As described above in the first embodiment, this bitstream is, for example, a bitstream generated by the encoding device 300, and is generated by encoding 3D data using meshes by extending VPCCs.

[0251] The demultiplexing unit 411 demultiplexes this bitstream to generate each piece of coded data included in the bitstream. That is, the demultiplexing unit 411 extracts each piece of coded data from the bitstream through the demultiplexing. For example, the demultiplexing unit 411 extracts coded data of pair information from the bitstream. The demultiplexing unit 411 also extracts coded data of meta information from the bitstream. The demultiplexing unit 411 also extracts coded data of geometry images from the bitstream. The demultiplexing unit 411 also extracts coded data of occupancy images from the bitstream. The demultiplexing unit 411 also extracts coded data of texture images from the bitstream.

[0252] The demultiplexing unit 411 supplies the extracted coded data to the decoding unit 431. For example, the demultiplexing unit 411 supplies the extracted coded data of pair information to the pair information decoding unit 412. The demultiplexing unit 411 also supplies the extracted coded data of meta information to the meta information decoding unit 413. The demultiplexing unit 411 also supplies the extracted coded data of the geometry image to the 2D decoding unit 414. The demultiplexing unit 411 also supplies the extracted coded data of the occupancy image to the 2D decoding unit 415. The demultiplexing unit 411 also supplies the extracted coded data of the texture image to the 2D decoding unit 416.

[0253] Decoding unit 431 executes processing related to decoding. Pairing information decoding unit 412 acquires coded data of pairing information supplied from demultiplexing unit 411. Pairing information decoding unit 412 decodes the coded data using a predetermined decoding method to generate pairing information. This decoding method may be any method as long as it corresponds to the coding method applied by pairing information coding unit 316 (FIG. 21) of coding device 300 to code the pairing information.

[0254] In other words, the pair information decoding unit 412 (decoding unit) divides a mesh representing an object with a three-dimensional structure into at least a first patch and a second patch, arranges them within a single image, and decodes the encoded data of the pair information indicating that at least one vertex of the first patch is paired with at least one vertex of the second patch that is located at the same position as the vertex of the first patch in the object before encoding, thereby obtaining the pair information.

[0255] The pair information decoding unit 412 supplies the generated pair information to the pairing processing unit 419.

[0256] The meta information decoding unit 413 acquires the encoded data of the meta information supplied from the demultiplexing unit 411. The meta information decoding unit 413 decodes the acquired encoded data of the meta information to generate meta information. This meta information includes connectivity 451 and a UV map 452. The meta information decoding unit 413 outputs the generated connectivity 451 and UV map 452 to the outside of the decoding device 400 as (data constituting) 3D data using the reconstructed mesh. The meta information decoding unit 413 also supplies the generated connectivity 451 and UV map 452 to the patch reconstruction unit 417.

[0257] The 2D decoding unit 414 acquires the coded data of the geometry image supplied from the demultiplexing unit 411. The 2D decoding unit 414 decodes the acquired coded data of the geometry image using a decoding method for 2D images to generate a geometry image. This decoding method corresponds to the coding method applied by the 2D coding unit 318 ( FIG. 21 ) of the coding device 300 to encode the geometry image. The 2D decoding unit 414 supplies the generated geometry image to the patch reconstruction unit 417.

[0258] The 2D decoding unit 415 acquires the coded data of the occupancy image supplied from the demultiplexing unit 411. The 2D decoding unit 415 decodes the acquired coded data of the occupancy image using a decoding method for 2D images to generate an occupancy image. This decoding method corresponds to the coding method applied by the 2D coding unit 319 ( FIG. 21 ) of the coding device 300 to code the occupancy image. The 2D decoding unit 415 supplies the generated occupancy image to the patch reconstruction unit 417.

[0259] The 2D decoding unit 416 acquires the coded data of the texture image supplied from the demultiplexing unit 411. The 2D decoding unit 416 decodes the acquired coded data of the texture image using a decoding method for 2D images to generate a texture image (texture 454). This decoding method corresponds to the coding method applied by the 2D encoding unit 320 ( FIG. 21 ) of the encoding device 300 when encoding the texture image. The 2D decoding unit 416 outputs the generated texture image (texture 454) to the outside of the decoding device 400 as (data constituting) 3D data using the restored mesh.

[0260] The patch reconstruction unit 417 acquires meta-information (connectivity 451, UV map 452, etc.) supplied from the meta-information decoding unit 413. The patch reconstruction unit 417 also acquires a geometry image supplied from the 2D decoding unit 414. The patch reconstruction unit 417 also acquires an occupancy image supplied from the 2D decoding unit 415. The patch reconstruction unit 417 extracts a patch image from the geometry image using the occupancy image and meta-information, and reconstructs a patch corresponding to the extracted patch image. The patch reconstruction unit 417 supplies the reconstructed patch and the used meta-information (connectivity 451, UV map 452, etc.) to the vertex information reconstruction unit 418.

[0261] The vertex information reconstruction unit 418 acquires the patch and meta information supplied from the patch reconstruction unit 417. The vertex information reconstruction unit 418 reconstructs the vertices included in the area of ​​the acquired patch, and generates vertex information 453. The vertex information reconstruction unit 418 outputs the generated vertex information 453 to the outside of the decoding device 400 as 3D data (data constituting) the restored mesh. In addition, the vertex information reconstruction unit 418 supplies the generated vertex information 453 and meta information (connectivity 451, UV map 452, etc.) to the pairing processing unit 419.

[0262] The pairing processing unit 419 acquires the pair information supplied from the pair information decoding unit 412. This pair information is information transmitted from the encoding device 300, and may include information such as that described in <2. Transmission of Pair Information> (including the sections <Generation of Pair Information> to <Application Examples>) and the like.

[0263] For example, this pair information may be information indicating only pairs formed by the vertices included in the pair information.

[0264] This pair information may include information indicating the correspondence between the combination of vertices located at both ends of the first edge and the vertex located midway between the first edge, and the combination of vertices located at both ends of the second edge and the vertex located midway between the second edge, for the first edge of the first patch and the second edge of the second patch, which correspond respectively to the boundary between the first patch and the second patch in the object before encoding.

[0265] This pair information may include information indicating the correspondence between the scanning direction for the first edge at the time of decoding and the scanning direction for the second edge at the time of decoding, for the first edge of the first patch and the second edge of the second patch, which correspond respectively to the boundary between the first patch and the second patch in the object before encoding.

[0266] The pair information may indicate that a plurality of vertices in the first patch and a corresponding plurality of vertices in the second patch are each paired.

[0267] This pair information may include a parameter indicating the number of vertices that make up the pair.

[0268] This pair information may include information indicating a group of vertices in the first patch and a group of vertices in the second patch.

[0269] This pair information may indicate the pair using identification information corresponding to the state of each vertex in the object before encoding.

[0270] This pair information may indicate pairs based on the coordinates of a plurality of vertices on a two-dimensional plane of the image.

[0271] This pair information may indicate the relative coordinates of a vertex that is different from the reference vertex, using the coordinates of the reference vertex as a reference among the plurality of vertices.

[0272] This pair information may indicate the relative coordinates of a vertex that is different from the reference vertex, based on the position of a relative vector between the first patch and the second patch, which is based on the coordinates of the reference vertex among the multiple vertices.

[0273] This pair information may indicate the relative coordinates of a vertex different from the reference vertex in the second patch in a posture-corrected state, based on the position of a relative vector between the first patch and the second patch, which is based on the coordinates of a reference vertex among the multiple vertices.

[0274] This pair information may indicate the relative coordinates of a vertex of the pair being processed that is different from the reference vertex, based on the relative coordinates of the vertex that is different from the reference vertex of the other pair, based on the coordinates of the reference vertex among the multiple vertices of the other pair.

[0275] Furthermore, the pairing processing unit 419 acquires the vertex information 453 and meta information supplied from the vertex information reconstruction unit 418. Based on the pair information, the pairing processing unit 419 performs pairing processing of the vertices included in the vertex information 453. At this time, the pairing processing unit 419 performs pairing processing as described in the sections <Vertex interpolation>, <Pairing processing>, <Coordinate determination method>, and <Application examples>.

[0276] That is, the pairing processing unit 419 (correction unit) corrects the reconstructed mesh based on the pair information.

[0277] In this case, the pairing processing unit 419 (correction unit) may delete one of the vertices of the first patch and the second patch that are indicated to be paired by the pair information in the reconstructed mesh, and convert the connection of the deleted vertex to the connection of the other vertex.

[0278] In addition, the pairing processing unit 419 (correction unit) may move one of the vertices of the first patch and the second patch, which are indicated as a pair by the pair information, to the position of the other vertex in the reconstructed mesh.

[0279] Furthermore, the pairing processing unit 419 (correction unit) may form, in the reconstructed mesh, a polygon including the vertices of the first patch and the vertices of the second patch that are indicated as a pair by the pair information.

[0280] Furthermore, the pairing processing unit 419 (correction unit) may interpolate vertices that make up a pair but are not included in the pair information.

[0281] This pairing process appropriately updates the vertex information 453. The pairing processing unit 419 outputs the updated vertex information 453 to the outside of the decoding device 400 as (data constituting) 3D data using the restored mesh.

[0282] These processing units (the demultiplexing unit 411 to the pairing processing unit 419) may have any configuration. For example, each processing unit may be configured with a logic circuit that realizes the above-described processing. Furthermore, each processing unit may have, for example, a CPU, a ROM, a RAM, etc., and may execute a program using these to realize the above-described processing. Of course, each processing unit may have both of these configurations, and may realize part of the above-described processing using a logic circuit and other parts by executing a program. The configurations of the processing units may be independent of each other. For example, some processing units may realize part of the above-described processing using a logic circuit, other processing units may execute a program to realize the above-described processing, and still other processing units may realize the above-described processing using both a logic circuit and by executing a program.

[0283] <Decryption process flow> An example of the flow of the decoding process executed by this decoding device 400 will be described with reference to the flowchart of FIG.

[0284] When the decoding process starts, in step S401, the demultiplexing unit 411 demultiplexes the bitstream input to the decoding device 400. Through this demultiplexing, the demultiplexing unit 411 extracts coded data of pair information from the bitstream. The demultiplexing unit 411 also extracts coded data of meta information from the bitstream. The demultiplexing unit 411 also extracts coded data of geometry images from the bitstream. The demultiplexing unit 411 also extracts coded data of occupancy images from the bitstream. The demultiplexing unit 411 also extracts coded data of texture images from the bitstream.

[0285] In step S402, the pairing information decoding unit 412 decodes the coded data of the pairing information extracted from the bitstream in step S401 using a predetermined decoding method to generate (restore) the pairing information. This pairing information is information transmitted from the encoding device 300, and may include information such as that described in <2. Transmission of Pairing Information> (including the sections <Generation of Pairing Information> to <Application Examples>) and the like.

[0286] In other words, the pair information decoding unit 412 (decoding unit) decodes the encoded data of pair information indicating a pair formed by a single vertex of a mesh or multiple vertices of a patch generated from multiple vertices at the same position by dividing a mesh representing an object with a three-dimensional structure into patches, and obtains the pair information.

[0287] For example, this pair information may be information indicating only pairs formed by the vertices included in the pair information.

[0288] This pair information may include information indicating the correspondence between the combination of vertices located at both ends of the first edge and the vertex located midway between the first edge, and the combination of vertices located at both ends of the second edge and the vertex located midway between the second edge, for the first edge of the first patch and the second edge of the second patch, which correspond respectively to the boundary between the first patch and the second patch in the object before encoding.

[0289] This pair information may include information indicating the correspondence between the scanning direction for the first edge at the time of decoding and the scanning direction for the second edge at the time of decoding, for the first edge of the first patch and the second edge of the second patch, which correspond respectively to the boundary between the first patch and the second patch in the object before encoding.

[0290] The pair information may indicate that a plurality of vertices in the first patch and a corresponding plurality of vertices in the second patch are each paired.

[0291] This pair information may include a parameter indicating the number of vertices that make up the pair.

[0292] This pair information may include information indicating a group of vertices in the first patch and a group of vertices in the second patch.

[0293] This pair information may indicate the pair using identification information corresponding to the state of each vertex in the object before encoding.

[0294] This pair information may indicate pairs based on the coordinates of a plurality of vertices on a two-dimensional plane of the image.

[0295] This pair information may indicate the relative coordinates of a vertex that is different from the reference vertex, using the coordinates of the reference vertex as a reference among the plurality of vertices.

[0296] This pair information may indicate the relative coordinates of a vertex that is different from the reference vertex, based on the position of a relative vector between the first patch and the second patch, which is based on the coordinates of the reference vertex among the multiple vertices.

[0297] This pair information may indicate the relative coordinates of a vertex different from the reference vertex in the second patch in a posture-corrected state, based on the position of a relative vector between the first patch and the second patch, which is based on the coordinates of a reference vertex among the multiple vertices.

[0298] This pair information may indicate the relative coordinates of a vertex of the pair being processed that is different from the reference vertex, based on the relative coordinates of the vertex that is different from the reference vertex of the other pair, based on the coordinates of the reference vertex among the multiple vertices of the other pair.

[0299] The decoding method applied in this process may be any method as long as it corresponds to the encoding method applied in step S306 (FIG. 22) of the encoding process.

[0300] In step S403, the meta information decoder 413 decodes the encoded data of the meta information extracted from the bitstream in step S401, and generates (restores) the meta information. This meta information includes connectivity 451 and a UV map 452.

[0301] In step S404, the 2D decoding unit 414 decodes the coded data of the geometry image extracted from the bitstream in step S401, and generates (restores) the geometry image.

[0302] In step S405, the 2D decoding unit 415 decodes the coded data of the occupancy image extracted from the bitstream in step S401, and generates (restores) the occupancy image.

[0303] In step S406, the 2D decoding unit 416 decodes the coded data of the texture image extracted from the bitstream in step S401, and generates (restores) the texture image (texture 454).

[0304] In step S407, the patch reconstruction unit 417 uses the meta information (connectivity 451 and UV map 452, etc.) generated in step S402 to extract a patch image from the geometry image generated in step S404 and reconstructs a patch corresponding to the patch image.

[0305] In step S408, the vertex information reconstructing unit 418 reconstructs the vertices included in the area of ​​the patch reconstructed in step S407, and generates vertex information 453.

[0306] In step S409, based on the pair information generated (restored) in step S402, the pairing processing unit 419 performs pairing processing of the vertices included in the vertex information 453. At this time, the pairing processing unit 419 performs pairing processing as described in the sections <Vertex interpolation>, <Pairing processing>, <Coordinate determination method>, and <Application examples>.

[0307] That is, the pairing processing unit 419 (correction unit) corrects the reconstructed mesh based on the pair information.

[0308] In this case, the pairing processing unit 419 (correction unit) may delete one of the vertices of the first patch and the second patch that are indicated to be paired by the pair information in the reconstructed mesh, and convert the connection of the deleted vertex to the connection of the other vertex.

[0309] In addition, the pairing processing unit 419 (correction unit) may move one of the vertices of the first patch and the second patch, which are indicated as a pair by the pair information, to the position of the other vertex in the reconstructed mesh.

[0310] Furthermore, the pairing processing unit 419 (correction unit) may form, in the reconstructed mesh, a polygon including the vertices of the first patch and the vertices of the second patch that are indicated as a pair by the pair information.

[0311] Furthermore, the pairing processing unit 419 (correction unit) may interpolate vertices that make up a pair but are not included in the pair information.

[0312] By such pairing processing, the vertex information 453 is updated appropriately.

[0313] When the process of step S409 is completed, the decoding process ends.

[0314] The decoding device 400 has the above configuration and performs the above various processes to perform pairing processing based on the pair information transmitted from the encoding device 300. This allows the decoding device 400 to more reliably suppress the occurrence of cracks and perform pairing processing between more appropriate vertices. Therefore, the decoding device 400 can suppress degradation of 3D data quality due to encoding and decoding.

[0315] <5. Deriving the correct coordinates> In the pairing process, when converting vertices that make up a pair into a single vertex, the coordinates of the converted vertex may be determined, for example, by using the coordinates of the vertex before encoding (also called correct coordinates) as the coordinates of the converted vertex. In other words, in the pairing process, the vertex that has moved due to encoding and decoding may be returned to its original position.

[0316] For example, when the projection directions of the patches of each vertex forming a pair in a three-dimensional space are different from each other, the intersection of the straight lines of the projection directions passing through each vertex is the coordinate (correct coordinate) of the vertex before encoding. For example, as shown in FIG. 25, a vertex 512 of a patch 511 and a vertex 514 of a patch 513 form a pair, and the projection direction of the patch 511 (i.e., the normal direction of the projection surface of the patch 511) and the projection direction of the patch 513 (i.e., the normal direction of the projection surface of the patch 513) are different from each other. In this case, the intersection 517 of the normal 515 of the projection surface of the patch 511 passing through the vertex 512 and the normal 516 of the projection surface of the patch 513 passing through the vertex 514 is the correct coordinate. Therefore, the decoding device can obtain the correct coordinate from the patches of the geometry.

[0317] In other words, if the normal direction of the first patch and the normal direction of the second patch do not match during the pairing process, the correction unit of the decoding device may move the vertices of the first patch and the second patch that make up the pair to the intersection of the normal of the first patch and the normal of the second patch.

[0318] On the other hand, when the projection directions of the patches of each vertex constituting a pair in the three-dimensional space are the same (forward or backward), the intersection of the straight lines of the projection directions passing through each vertex cannot be obtained, and therefore the decoding device cannot obtain the correct coordinates from the patches of the geometry. For example, as shown in Fig. 26, a vertex 512 of a patch 511 and a vertex 514 of a patch 513 form a pair, and the projection direction of the patch 511 (i.e., the normal direction of the projection surface of the patch 511) and the projection direction of the patch 513 (i.e., the normal direction of the projection surface of the patch 513) are the same (forward or backward). In this case, the normal of the projection surface of the patch 511 passing through the vertex 512 and the normal of the projection surface of the patch 513 passing through the vertex 514 are both normal 516, and therefore the intersection of the normals (correct coordinates) cannot be obtained from the coordinates of each vertex.

[0319] In this way, when the projection directions of the patches of each vertex constituting a pair in three-dimensional space are the same, the pairing process may be performed by using, for example, the average of the coordinates of each vertex as the ground truth coordinate. In other words, when the normal direction of the first patch and the normal direction of the second patch are the same, the correction unit of the decoding device may move the vertices of the first patch and the second patch constituting the pair to positions that are the weighted average of their respective positions. For example, in the case of Figure 26, the components of the projection plane direction of the ground truth coordinates are the same as those of vertex 512 and vertex 514, and the component of the direction of the normal 516 of the ground truth coordinates is the average of the coordinates of vertex 512 and vertex 514.

[0320] Furthermore, when the projection directions of the patches of each vertex forming a pair in three-dimensional space are the same, for example, the pre-encoding coordinates (correct coordinates) for each pair may be transmitted from the encoding device to the decoding device as pair information. That is, the pair information may include information indicating the destination positions of the vertices of the pair when they are moved to the same positions in the pairing process. In this case, the encoding device stores the correct coordinates for each pair in the pair information. Then, the decoding device moves each vertex of the pair to the correct coordinates included in the pair information in the pairing process. In this way, when the normal direction of the first patch and the normal direction of the second patch are the same, the correction unit of the decoding device may move the vertices of the first patch and the second patch forming the pair to the positions indicated by the pair information.

[0321] Even in these cases, the decoder can more reliably suppress the occurrence of cracks and perform pairing processing with more appropriate vertices, which means that degradation of 3D data quality due to encoding and decoding can be suppressed.

[0322] <6. Two-dimensional pairing processing> In the second embodiment, it has been described that the pairing process is performed in a three-dimensional space (i.e., the three-dimensional coordinates of each vertex forming a pair are corrected). That is, in this case, for example, as shown in FIG. 27, 2D decoding 531 of the geometry is performed, 3D data reconstruction 532 is performed, and then pairing process 533 is performed on the 3D data in the three-dimensional space. For example, as shown in FIG. 27, it is assumed that a vertex 542 of a polygon 541 and a vertex 545 of a polygon 544 form a pair, and a vertex 543 of a polygon 541 and a vertex 546 of a polygon 544 form a pair. Then, it is assumed that the pairing process moves the vertex 545 to the position of the vertex 542, and the vertex 546 to the position of the vertex 543. In this case, the pairing process converts the three-dimensional coordinates (x4, y4, z4) of the vertex 545 into the three-dimensional coordinates (x2, y2, z2) of the vertex 542. Furthermore, the three-dimensional coordinates (x6, y6, z6) of the vertex 546 are converted into the three-dimensional coordinates (x3, y3, z3) of the vertex 543.

[0323] The pairing process is not limited to this example and may be performed, for example, on a two-dimensional plane. For example, as shown in FIG. 28, after 2D decoding 551 of geometry is performed, pairing process 552 is performed on the two-dimensional plane, and then 3D data reconstruction 553 is performed. This two-dimensional plane may be, for example, a geometry image (depth image) or a two-dimensional plane (xyz image) in three-dimensional space. In the case of pairing process performed on a two-dimensional plane, pixel values ​​p corresponding to each vertex constituting a pair on the two-dimensional plane are updated. For example, as shown in FIG. 28, it is assumed that a vertex corresponding to pixel 561 and a vertex corresponding to pixel 563 on a two-dimensional plane 560 form a pair, and a vertex corresponding to pixel 562 and a vertex corresponding to pixel 564 form a pair. It is also assumed that, through the pairing process, the vertex corresponding to pixel 563 is moved to the three-dimensional position of the vertex corresponding to pixel 561, and the vertex corresponding to pixel 564 is moved to the three-dimensional position of the vertex corresponding to pixel 562. In this case, the pairing process on the two-dimensional plane updates the pixel value (p4) of pixel 563 to the same pixel value (p2) as pixel 561. Similarly, the pixel value (p6) of pixel 564 is updated to the same pixel value (p3) as pixel 562. For example, if this two-dimensional plane is a geometry image (depth image), the pixel value indicates a depth value (coordinate in the normal direction of the patch in three-dimensional space). Also, if this two-dimensional plane is an xyz image, the pixel value indicates an (x, y, z) coordinate.

[0324] In other words, the correction unit of the decoding device may move the vertices of the first patch and the vertices of the second patch that make up a pair to the same positions in the geometry image generated by decoding the encoded data, the patch construction unit may reconstruct the patch using the geometry image corrected by the correction unit, and the vertex reconstruction unit may reconstruct the vertices using the reconstructed patch.

[0325] The method for determining the corrected vertex positions (correct coordinates) when pairing processing is performed on a two-dimensional plane in this way is the same as in pairing processing in a three-dimensional space. For example, if the projection directions of each vertex constituting a pair are the same, the decoding device may derive correct coordinates (average, etc.) using the pixel values ​​of each vertex and move each vertex to those correct coordinates. Alternatively, as described above, the decoding device may move the vertex using the correct coordinates stored in the pair information. On the other hand, if the projection directions of each vertex constituting a pair are different from each other, the decoding device may derive the three-dimensional coordinates of each vertex, use the three-dimensional coordinates to find the correct coordinates (three-dimensional coordinates), and update the pixel values ​​corresponding to each vertex in the two-dimensional plane so that the vertex moves to those three-dimensional coordinates.

[0326] Even in these cases, the decoder can more reliably suppress the occurrence of cracks and perform pairing processing with more appropriate vertices, which means that degradation of 3D data quality due to encoding and decoding can be suppressed.

[0327] 7. Third Embodiment <Encoding device> Fig. 29 is a block diagram showing another example of the configuration of an encoding device that is an embodiment of an information processing device to which the present technology is applied. Similar to the encoding device 300 shown in Fig. 21, the encoding device 600 shown in Fig. 29 is a device that encodes 3D data using a mesh by extending VPCCs as video frames using an encoding method for 2D images. This encoding device 600 has a local decoding function, decodes encoded geometry, and updates a texture image using the decoding result. In this case, the encoding device 600 applies a single method or a combination of multiple methods from among the various methods of the present technology described above to generate pair information and perform pairing processing.

[0328] Note that Fig. 29 shows the main processing units, data flows, etc., and is not necessarily all that is shown in Fig. 29. In other words, in encoding device 600, there may be processing units that are not shown as blocks in Fig. 29, and there may be processing and data flows that are not shown as arrows, etc. in Fig. 29.

[0329] As shown in FIG. 29, the encoding device 600 includes a mesh voxelization unit 611, a patch generation unit 612, a meta information encoding unit 613, an image generation unit 614, a 2D encoding unit 615, a 2D decoding unit 616, a pair information generation unit 617, a pair information encoding unit 618, a pairing processing unit 619, an image generation unit 620, a 2D encoding unit 621, and a multiplexing unit 622.

[0330] As in the case of the encoding device 300, the encoding device 600 is supplied with connectivity 351, vertex information 352, UV map 353, and texture 354 as 3D data using a mesh.

[0331] The mesh voxelization unit 611 is a processing unit similar to the mesh voxelization unit 311, and executes the same processing as the mesh voxelization unit 311. For example, the mesh voxelization unit 611 acquires the vertex information 352 supplied to the encoding device 600. The mesh voxelization unit 611 converts the coordinates of each vertex included in the acquired vertex information 352 into a voxel grid. The mesh voxelization unit 611 supplies the vertex information 352 of the converted voxel grid to the patch generation unit 612.

[0332] The patch generation unit 612 is a processing unit similar to the patch generation unit 312, and performs the same processing as the patch generation unit 312. For example, the patch generation unit 612 acquires the connectivity 351 supplied to the encoding device 600. The patch generation unit 612 also acquires the vertex information 352 of the voxel grid supplied from the mesh voxelization unit 611. The patch generation unit 612 generates a geometry patch based on the acquired information. The patch generation unit 612 also projects the generated geometry patch onto a projection plane to generate a patch image. The patch generation unit 612 supplies information such as the connectivity 351 and the vertex information 352 as meta information to the meta information encoding unit 613. The patch generation unit 612 also supplies the generated patch image to the image generation unit 614 and the image generation unit 620.

[0333] The meta information encoder 613 is a processing unit similar to the meta information encoder 317, and executes the same processing as the meta information encoder 317. The meta information encoder 613 acquires the meta information (including the connectivity 351, the vertex information 352, etc.) supplied from the patch generator 612. The meta information encoder 613 encodes the acquired meta information to generate encoded data of the meta information. The meta information encoder 613 supplies the generated encoded data of the meta information to the multiplexer 622.

[0334] The image generation unit 614 acquires the patch image supplied from the patch generation unit 612. The image generation unit 614 arranges the patch image on a two-dimensional plane to generate a geometry image. The image generation unit 614 supplies the generated geometry image to the 2D encoding unit 615 as a geometry video frame. The image generation unit 614 also generates an occupancy image corresponding to the geometry image. The image generation unit 614 supplies the generated occupancy image to the 2D encoding unit 615.

[0335] The 2D encoding unit 615 acquires the geometry image and the occupancy image supplied from the image generation unit 614. The 2D encoding unit 615 encodes the acquired geometry image using an encoding method for 2D images to generate encoded data for the geometry image. In other words, the 2D encoding unit 615 can also be considered a geometry encoding unit that encodes the geometry image and generates encoded data for the geometry image. The 2D encoding unit 615 supplies the encoded data of the generated geometry image to the 2D decoding unit 616 and the multiplexing unit 622. The 2D encoding unit 615 also encodes the acquired occupancy image using an encoding method for 2D images to generate encoded data for the occupancy image. The 2D encoding unit 615 supplies the encoded data of the generated occupancy image to the multiplexing unit 622.

[0336] The 2D decoding unit 616 acquires the coded data of the geometry image supplied from the 2D encoding unit 615. The 2D decoding unit 616 decodes the acquired coded data of the geometry image using a decoding method for 2D images to generate (restore) the geometry image. In other words, the 2D decoding unit 616 can also be considered a geometry decoding unit that decodes the coded data of the geometry image and generates the geometry image. The 2D decoding unit 616 supplies the generated geometry image to the pair information generation unit 617.

[0337] The pair information generation unit 617 acquires the geometry image supplied from the 2D decoding unit 616. The pair information generation unit 617 may also acquire information such as the connectivity 351, the vertex information 352, and the UV map 353, as necessary. Based on this information, the pair information generation unit 617 generates pair information indicating pairs each consisting of a single vertex or multiple vertices that were overlapping points in the mesh. That is, the pair information generation unit 617 divides a mesh representing an object with a three-dimensional structure into at least a first patch and a second patch, arranges the patches in a single image, and generates pair information indicating that at least one vertex of the first patch is paired with at least one vertex of the second patch that is located at the same position as the vertex of the first patch in the object before encoding. The pair information generation unit 617 then generates the pair information based on the geometry image generated by the geometry decoding unit. This pair information may include information such as that described in Section <2. Transmission of Pair Information>, Section <5. Deriving Solution Coordinates>, and Section <6. Two-Dimensional Pairing Processing>. The pair information generation unit 617 supplies the generated pair information to the pair information encoding unit 618. The pair information generation unit 617 also supplies the generated pair information and acquired information (geometry image, meta information, etc.) to the pairing processing unit 619.

[0338] The pair information encoder 618 acquires the pair information supplied from the pair information generator 617. The pair information encoder 618 encodes the acquired pair information using a predetermined encoding method to generate encoded data of the pair information. This encoding method is arbitrary. The pair information encoder 618 supplies the generated encoded data of the pair information to the multiplexer 622.

[0339] The pairing processing unit 619 acquires pair information, geometry images, meta information, and the like supplied from the pair information generation unit 617. The pairing processing unit 619 performs pairing processing on patches included in the geometry image based on the acquired pair information. At this time, the pairing processing unit 619 performs pairing processing as described in sections <2. Transmission of Pair Information>, <5. Deriving Solution Coordinates>, and <6. Pairing Processing in Two Dimensions>. The pairing processing unit 619 may perform this pairing processing on a two-dimensional plane or in three-dimensional space. When performing pairing processing in three-dimensional space, the pairing processing unit 619 reconstructs 3D data using the geometry images, then performs pairing processing, and generates a geometry image using the 3D data after the pairing processing. The pairing processing unit 619 supplies the geometry image after the pairing processing to the image generation unit 620.

[0340] The image generation unit 620 acquires the UV map 353 and texture 354 to be input to the encoding device 600. The image generation unit 620 also acquires a patch image supplied from the patch generation unit 612. The image generation unit 620 also acquires a geometry image after the pairing process supplied from the pairing processing unit 619. The image generation unit 620 updates the texture 354 using the acquired patch image, geometry image, UV map 353, etc., and generates a texture image corresponding to the geometry image after the pairing process. The image generation unit 620 may also update the UV map 353 as necessary. The image generation unit 620 supplies the generated texture image to the 2D encoding unit 621.

[0341] The 2D encoding unit 621 acquires the texture image supplied from the image generation unit 620. The 2D encoding unit 621 encodes the acquired texture image using a 2D image encoding method to generate encoded data for the texture image. The 2D encoding unit 621 supplies the generated encoded data of the texture image to the multiplexing unit 622.

[0342] The multiplexing unit 321 acquires the coded data of the meta information supplied from the meta information encoding unit 613. Furthermore, the multiplexing unit 622 acquires the coded data of the geometry image and the coded data of the occupancy image supplied from the 2D encoding unit 615. Furthermore, the multiplexing unit 622 acquires the coded data of the pair information supplied from the pair information encoding unit 618. Furthermore, the multiplexing unit 622 acquires the coded data of the texture image supplied from the 2D encoding unit 621. The multiplexing unit 622 multiplexes the acquired information to generate a single bit stream. The multiplexing unit 622 outputs the generated bit stream to the outside of the encoding device 600.

[0343] With this configuration, the encoding device 600 can more reliably suppress the occurrence of cracks in the decoder and perform pairing processing between more appropriate vertices. In other words, the encoding device 600 can suppress degradation of the quality of 3D data due to encoding and decoding.

[0344] These processing units (mesh voxelization unit 611 to multiplexing unit 622) may have any configuration. For example, each processing unit may be configured with a logic circuit that realizes the above-described processing. Furthermore, each processing unit may have, for example, a central processing unit (CPU), read-only memory (ROM), random access memory (RAM), etc., and may execute a program using these to realize the above-described processing. Of course, each processing unit may have both of these configurations, and may implement part of the above-described processing using a logic circuit and other parts by executing a program. The configurations of each processing unit may be independent of each other. For example, some processing units may implement part of the above-described processing using a logic circuit, other processing units may implement the above-described processing by executing a program, and still other processing units may implement the above-described processing using both a logic circuit and by executing a program.

[0345] <Encoding process flow> An example of the flow of the encoding process executed by this encoding device 600 will be described with reference to the flowchart of FIG.

[0346] When the encoding process starts, in step S601, the mesh voxelization unit 611 converts the coordinates of each vertex included in the vertex information 352 into a voxel grid, thereby converting the mesh into a voxel grid.

[0347] In step S602, the patch generation unit 612 generates a patch using the vertex information 352 and the like that was converted into a voxel grid in step S601, and projects the generated patch onto the projection surface to generate a patch image.

[0348] In step S603, the meta information encoding unit 613 encodes the meta information (the connectivity 351, the UV map 353, etc.) to generate encoded data of the meta information.

[0349] In step S604, the image generation unit 614 generates a geometry image and an occupancy image based on the patch image and the UV map 353 generated in step S602.

[0350] In step S605, the 2D encoding unit 615 encodes the geometry image generated in step S604 to generate encoded data of the geometry image. The 2D encoding unit 615 also encodes the occupancy image generated in step S604 to generate encoded data of the occupancy image.

[0351] In step S606, the 2D decoding unit 616 decodes the coded data of the geometry image generated in step S605 to generate a geometry image.

[0352] In step S607, the pair information generation unit 617 generates pair information indicating pairs each consisting of one vertex or multiple vertices that were overlapping points in the mesh, using the geometry image, meta information, etc. generated in step S606. This pair information may include information such as those described in sections <2. Transmission of pair information>, <5. Deriving ground truth coordinates>, and <6. Two-dimensional pairing process>.

[0353] In step S608, the pair information encoding unit 618 encodes the pair information generated in step S607 using a predetermined encoding method to generate encoded data of the pair information. This encoding method is arbitrary.

[0354] In step S609, the pairing processing unit 619 performs pairing processing on the geometry image generated in step S606 based on the pair information generated in step S607, etc. At this time, the pairing processing unit 619 performs pairing processing as described in sections <2. Transmission of pair information>, <5. Deriving solution coordinates>, and <6. Two-dimensional pairing processing>, etc.

[0355] In step S610, the image generation unit 620 updates the texture 354 using the patch image generated in step S602 and the geometry image after pairing processing generated in step S609, and generates a texture image.

[0356] In step S611, the 2D encoding unit 621 encodes the texture image generated in step S610 to generate encoded data of the texture image.

[0357] In step S612, the multiplexing unit 622 multiplexes the encoded data of the meta information generated in step S603, the encoded data of the geometry image and the encoded data of the occupancy image generated in step S605, the encoded data of the pair information generated in step S608, and the encoded data of the texture image generated in step S611, to generate a bitstream.

[0358] By performing each process in this manner, the encoding device 600 can enable the decoder to more reliably suppress the occurrence of cracks and perform pairing processing between more appropriate vertices. In other words, the encoding device 600 can suppress degradation of 3D data quality due to encoding and decoding.

[0359] 8. Fourth Embodiment <Decryption device> Fig. 31 is a block diagram showing an example of the configuration of a decoding device, which is one aspect of an image processing device to which the present technology is applied. Similar to the decoding device 400 shown in Fig. 23, the decoding device 700 shown in Fig. 31 is a device that decodes, using a decoding method for 2D images, coded data that has been encoded using a 2D image coding method to generate (reconstruct) 3D data using meshes by extending VPCCs and using a 2D image coding method, and generates (reconstructs) 3D data using meshes. In this case, the decoding device 700 applies a single method or a combination of multiple methods from among the various methods of the present technology described above to decode the coded data and reconstruct the 3D data. However, the decoding device 700 performs pairing processing on a 2D plane.

[0360] Note that Fig. 31 shows the main processing units, data flows, etc., and does not necessarily show everything. That is, in decoding device 700, there may be processing units that are not shown as blocks in Fig. 31, and there may be processing or data flows that are not shown as arrows, etc. in Fig. 31.

[0361] As shown in FIG. 31 , the decoding device 700 includes a demultiplexing unit 711, a pair information decoding unit 712, a meta information decoding unit 713, a 2D decoding unit 714, a pairing processing unit 715, a patch reconstruction unit 716, a vertex information reconstruction unit 717, and a 2D decoding unit 718.

[0362] The demultiplexing unit 711 obtains a bitstream to be input to the decoding device 700. As described above in the first embodiment, this bitstream is a bitstream generated by, for example, the encoding device 300 or the encoding device 600, and is generated by encoding 3D data using meshes by extending the VPCC.

[0363] The demultiplexing unit 711 demultiplexes this bitstream to generate each piece of coded data included in the bitstream. That is, the demultiplexing unit 711 extracts each piece of coded data from the bitstream through demultiplexing. For example, the demultiplexing unit 711 extracts coded data of pair information from the bitstream. The demultiplexing unit 711 also extracts coded data of meta information from the bitstream. The demultiplexing unit 711 also extracts coded data of geometry images from the bitstream. The demultiplexing unit 711 also extracts coded data of occupancy images from the bitstream. The demultiplexing unit 711 also extracts coded data of texture images from the bitstream. That is, the demultiplexing unit 711 can also be said to be an acquisition unit that acquires these pieces of coded data from the bitstream.

[0364] The demultiplexing unit 711 supplies the coded data of the extracted pair information to a pair information decoding unit 712. The demultiplexing unit 711 also supplies the coded data of the extracted meta information to a meta information decoding unit 713. The demultiplexing unit 711 also supplies the coded data of the extracted geometry image and coded data of the occupancy image to a 2D decoding unit 714. The demultiplexing unit 711 also supplies the coded data of the extracted texture image to a 2D decoding unit 718.

[0365] The pairing information decoding unit 712 acquires the coded data of pairing information supplied from the demultiplexing unit 711. The pairing information decoding unit 712 decodes the coded data using a predetermined decoding method to generate pairing information. This decoding method may be any method as long as it corresponds to the coding method used by the encoding device 300 or the encoding device 600 when encoding the pairing information.

[0366] That is, the pair information decoding unit 712 divides a mesh representing an object with a three-dimensional structure into at least a first patch and a second patch, arranges them within a single image, and decodes the encoded data of pair information indicating that at least one vertex of the first patch is paired with at least one vertex of the second patch that is located at the same position as the vertex of the first patch in the object before encoding, thereby obtaining pair information. The pair information decoding unit 712 supplies the generated pair information to the pairing processing unit 715.

[0367] The meta information decoding unit 713 acquires the encoded data of the meta information supplied from the demultiplexing unit 711. The meta information decoding unit 713 decodes the acquired encoded data of the meta information to generate meta information. This meta information includes connectivity 451 and a UV map 452. The meta information decoding unit 713 outputs the generated connectivity 451 and UV map 452 to the outside of the decoding device 700 as (data constituting) 3D data using the reconstructed mesh. The meta information decoding unit 713 also supplies the generated connectivity 451 and UV map 452 to the patch reconstruction unit 716.

[0368] The 2D decoding unit 714 acquires the coded data of the geometry image and the coded data of the occupancy image supplied from the demultiplexing unit 711. The 2D decoding unit 714 decodes the coded data of the acquired geometry image using a decoding method for 2D images to generate a geometry image. The 2D decoding unit 714 also decodes the coded data of the acquired occupancy image using a decoding method for 2D images to generate an occupancy image. These decoding methods correspond to the coding methods applied by the encoding device 300 and the encoding device 600 when encoding the geometry image. The 2D decoding unit 714 supplies the generated geometry image and occupancy image to the pairing processing unit 715.

[0369] The pairing processing unit 715 acquires pair information supplied from the pair information decoding unit 712. This pair information is, for example, information transmitted from the encoding device 300 or the encoding device 600, and may include information such as that described in sections <2. Transmission of Pair Information>, <5. Deriving Solution Coordinates>, and <6. Pairing Processing in Two Dimensions>. The pairing processing unit 715 also acquires a geometry image and an occupancy image supplied from the 2D decoding unit 714. The pairing processing unit 715 performs pairing processing on patches included in the geometry image based on the acquired pair information. In other words, the pairing processing unit 715 can also be considered a correction unit that corrects the geometry of vertices constituting pairs based on the pair information. In this case, the pairing processing unit 715 performs pairing processing on a two-dimensional plane (e.g., a geometry image) as described in section <6. Pairing Processing in Two Dimensions>. The pairing processing unit 715 supplies the geometry image, occupancy image, etc. after the pairing processing to the patch reconstruction unit 716.

[0370] The patch reconstructor 716 acquires meta-information (such as the connectivity 451 and UV map 452) supplied from the meta-information decoding unit 713. The patch reconstructor 716 also acquires the geometry image and occupancy image after pairing processing supplied from the pairing processing unit 715. The patch reconstructor 716 extracts patch images from the geometry image using the occupancy image and meta-information, and reconstructs patches corresponding to the extracted patch images. The patch reconstructor 716 supplies the reconstructed patches and the used meta-information (such as the connectivity 451 and UV map 452) to the vertex information reconstructor 717.

[0371] The vertex information reconstruction unit 717 acquires patches and meta information supplied from the patch reconstruction unit 716. The vertex information reconstruction unit 717 reconstructs the vertices included in the area of ​​the acquired patch, and generates the vertex information 453. In other words, the vertex information reconstruction unit 717 can also be said to be a vertex reconstruction unit that reconstructs vertices using the reconstructed patch. The vertex information reconstruction unit 717 outputs the generated vertex information 453 to the outside of the decoding device 700 as (data constituting) 3D data using the restored mesh.

[0372] The 2D decoding unit 718 acquires the coded data of the texture image supplied from the demultiplexing unit 711. The 2D decoding unit 718 decodes the acquired coded data of the texture image using a decoding method for 2D images to generate a texture image (texture 454). This decoding method corresponds to the coding method applied by the coding device 300 or the coding device 600 when encoding the texture image. The 2D decoding unit 718 outputs the generated texture image (texture 454) to the outside of the decoding device 700 as (data constituting) 3D data using the restored mesh.

[0373] With this configuration, the decoding device 700 can more reliably suppress the occurrence of cracks and perform pairing processing between more appropriate vertices. In other words, the decoding device 700 can suppress degradation of 3D data quality due to encoding and decoding.

[0374] These processing units (the demultiplexing unit 711 to the 2D decoding unit 718) may have any configuration. For example, each processing unit may be configured with a logic circuit that realizes the above-described processing. Furthermore, each processing unit may have, for example, a CPU, a ROM, a RAM, etc., and may execute a program using these to realize the above-described processing. Of course, each processing unit may have both of these configurations, and may realize part of the above-described processing using a logic circuit and other parts by executing a program. The configurations of the processing units may be independent of each other. For example, some processing units may realize part of the above-described processing using a logic circuit, other processing units may execute a program to realize the above-described processing, and still other processing units may realize the above-described processing using both a logic circuit and by executing a program.

[0375] <Decryption process flow> An example of the flow of the decoding process executed by this decoding device 700 will be described with reference to the flowchart of FIG.

[0376] When the decoding process starts, in step S701, the demultiplexing unit 711 demultiplexes the bitstream input to the decoding device 700. Through this demultiplexing, the demultiplexing unit 711 extracts (obtains) coded data of pair information from the bitstream. The demultiplexing unit 711 also extracts (obtains) coded data of meta information from the bitstream. The demultiplexing unit 711 also extracts (obtains) coded data of geometry images from the bitstream. The demultiplexing unit 711 also extracts (obtains) coded data of occupancy images from the bitstream. The demultiplexing unit 711 also extracts (obtains) coded data of texture images from the bitstream.

[0377] In step S702, the pairing information decoding unit 712 decodes the coded data of the pairing information extracted from the bitstream in step S701 using a predetermined decoding method to generate (restore) pairing information. This pairing information is information transmitted from the encoding device 300, the encoding device 600, etc., and may include information such as those described in sections <2. Transmission of pairing information>, <5. Deriving solution coordinates>, <6. Two-dimensional pairing processing>, etc.

[0378] In other words, the pair information decoding unit 712 decodes the encoded data of pair information indicating a pair formed by a single vertex of a mesh or multiple vertices of a patch generated from multiple vertices at the same position by dividing a mesh representing an object with a three-dimensional structure into patches, and obtains the pair information.

[0379] In step S703, the meta information decoder 713 decodes the encoded data of the meta information extracted from the bitstream in step S701, and generates (restores) the meta information. This meta information includes the connectivity 451 and the UV map 452.

[0380] In step S704, the 2D decoding unit 714 decodes the coded data of the geometry image extracted from the bitstream in step S701 to generate (restore) a geometry image. Also, the 2D decoding unit 714 decodes the coded data of the occupancy image extracted from the bitstream in step S701 to generate (restore) an occupancy image.

[0381] In step S705, based on the pairing information generated (restored) in step S702, the pairing processing unit 715 performs pairing processing of the vertices included in the vertex information 453. At this time, the pairing processing unit 715 performs pairing processing on a two-dimensional plane (for example, a geometry image) as described in section <6. Two-dimensional pairing processing>.

[0382] In step S706, the patch reconstruction unit 716 uses the meta information (connectivity 451 and UV map 452, etc.) generated in step S702 to extract a patch image from the geometry image generated in step S704, and reconstructs a patch corresponding to the patch image.

[0383] In step S707, the vertex information reconstructing unit 717 reconstructs the vertices included in the area of ​​the patch reconstructed in step S706, and generates the vertex information 453.

[0384] In step S708, the 2D decoding unit 718 decodes the coded data of the texture image extracted from the bitstream in step S701, and generates (restores) the texture image (texture 454).

[0385] By performing each process in this manner, the decoding device 700 can more reliably suppress the occurrence of cracks and perform pairing processing between more appropriate vertices. In other words, the decoding device 700 can suppress degradation of 3D data quality due to encoding and decoding.

[0386] <9. Notes> The above describes a case where 3D data using meshes is encoded by extending the VPCC standard, but V3C (Visual Volumetric Video-based Coding) or MIV (metadata immersive video) may be applied instead of VPCC. V3C and MIV are standards that use encoding techniques similar to those of VPCC, and can be extended in the same way as VPCC to encode 3D data using meshes. Therefore, the above-described technology can also be applied when V3C or MIV is applied to encoding 3D data using meshes.

[0387] <3D data> While the above describes the application of this technology to encoding and decoding of meshes, this technology is not limited to these examples and can be applied to encoding and decoding of 3D data of any standard. In other words, as long as it does not conflict with the above-mentioned technology, various processes such as encoding and decoding methods and specifications of various data such as 3D data and metadata are arbitrary. Furthermore, as long as it does not conflict with the above-mentioned technology, some of the above-mentioned processes and specifications may be omitted.

[0388] <Computer> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the programs constituting the software are installed on a computer. Here, the term "computer" includes computers built into dedicated hardware, and general-purpose personal computers, etc., that can execute various functions by installing various programs.

[0389] FIG. 33 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0390] In a computer 900 shown in FIG. 33, a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, and a RAM (Random Access Memory) 903 are interconnected via a bus 904.

[0391] An input / output interface 910 is also connected to the bus 904. To the input / output interface 910, an input unit 911, an output unit 912, a storage unit 913, a communication unit 914, and a drive 915 are connected.

[0392] The input unit 911 includes, for example, a keyboard, a mouse, a microphone, a touch panel, an input terminal, etc. The output unit 912 includes, for example, a display, a speaker, an output terminal, etc. The storage unit 913 includes, for example, a hard disk, a RAM disk, a non-volatile memory, etc. The communication unit 914 includes, for example, a network interface. The drive 915 drives removable media 921 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0393] In a computer configured as above, the CPU 901 performs the above-described series of processes by, for example, loading a program stored in the storage unit 913 into the RAM 903 via the input / output interface 910 and the bus 904 and executing the program. The RAM 903 also stores data necessary for the CPU 901 to execute various processes as appropriate.

[0394] The program executed by the computer can be applied by recording it on removable media 921 such as package media, for example. In this case, the program can be installed in storage unit 913 via input / output interface 910 by inserting removable media 921 into drive 915.

[0395] This program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, digital satellite broadcasting, etc. In this case, the program can be received by the communication unit 914 and installed in the storage unit 913.

[0396] Alternatively, this program can be installed in advance in the ROM 902 or the storage unit 913 .

[0397] <Applicable targets of this technology> The present technology can be applied to any configuration, for example, various electronic devices.

[0398] Furthermore, for example, the present technology can also be implemented as a part of an apparatus, such as a processor (e.g., a video processor) as a system LSI (Large Scale Integration), a module (e.g., a video module) using multiple processors, a unit (e.g., a video unit) using multiple modules, or a set in which other functions are added to a unit (e.g., a video set).

[0399] Furthermore, for example, the present technology can also be applied to a network system configured with multiple devices. For example, the present technology may be implemented as cloud computing in which multiple devices share and collaborate on processing via a network. For example, the present technology may be implemented in a cloud service that provides image (video)-related services to any terminal, such as a computer, AV (Audio Visual) equipment, a portable information processing terminal, or an IoT (Internet of Things) device.

[0400] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0401] <Fields and applications where this technology can be applied> Systems, devices, processing units, etc. to which the present technology is applied can be used in any field, such as transportation, medical care, crime prevention, agriculture, livestock farming, mining, beauty, factories, home appliances, weather, and nature monitoring. In addition, the applications thereof are also arbitrary.

[0402] <Other> In this specification, a "flag" refers to information for identifying multiple states, and includes not only information used to identify two states, true (1) or false (0), but also information capable of identifying three or more states. Therefore, the value that this "flag" can take may be, for example, two values, 1 / 0, or three or more values. In other words, the number of bits constituting this "flag" is arbitrary, and may be one bit or multiple bits. Furthermore, identification information (including flags) can be assumed not only to include the identification information in the bit stream, but also to include difference information of the identification information relative to certain reference information in the bit stream. Therefore, in this specification, "flag" and "identification information" include not only the information itself, but also difference information relative to the reference information.

[0403] Furthermore, various types of information (metadata, etc.) related to the coded data (bitstream) may be transmitted or recorded in any form as long as they are associated with the coded data. Here, the term "associate" means, for example, making one piece of data available (linked) when processing the other piece of data. In other words, mutually associated data may be combined into one piece of data or may be individual pieces of data. For example, information associated with coded data (image) may be transmitted over a transmission path separate from that of the coded data (image). Also, for example, information associated with coded data (image) may be recorded on a recording medium separate from that of the coded data (image) (or on a different recording area of ​​the same recording medium). Note that this "association" may refer to only a portion of the data, rather than the entire data. For example, an image and information corresponding to that image may be associated with each other in any unit, such as multiple frames, one frame, or a portion of a frame.

[0404] In this specification, terms such as "composite," "multiplex," "add," "integrate," "include," "store," "embed," "insert," and the like refer to combining multiple items into one, such as combining encoded data and metadata into one piece of data, and refer to one method of "associating" as described above.

[0405] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present technology.

[0406] For example, a configuration described as one device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, configurations described above as multiple devices (or processing units) may be combined and configured as one device (or processing unit). Of course, configurations other than those described above may be added to the configuration of each device (or each processing unit). Furthermore, as long as the configuration and operation of the entire system are substantially the same, part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0407] Furthermore, for example, the above-described program may be executed in any device, as long as the device has the necessary functions (functional blocks, etc.) and can obtain the necessary information.

[0408] Also, for example, each step of a single flowchart may be executed by one device, or may be shared and executed by multiple devices. Furthermore, when one step includes multiple processes, the multiple processes may be executed by one device, or may be shared and executed by multiple devices. In other words, multiple processes included in one step can be executed as multiple step processes. Conversely, processes described as multiple steps can be executed collectively as one step.

[0409] For example, the steps of a program executed by a computer may be executed in chronological order in the order described herein, or may be executed in parallel or individually at the required timing, such as when a call is made. In other words, as long as no contradiction occurs, the steps may be executed in an order different from the order described above. Furthermore, the steps of this program may be executed in parallel with the processing of another program, or may be executed in combination with the processing of another program.

[0410] Furthermore, for example, multiple technologies related to the present technology can be implemented independently and independently, as long as no contradiction occurs. Of course, any multiple technologies can also be implemented in combination. For example, part or all of the present technology described in any embodiment can be implemented in combination with part or all of the present technology described in another embodiment. Furthermore, part or all of any of the above-described present technologies can be implemented in combination with other technologies not described above.

[0411] The present technology can also be configured as follows. (1) a generation unit that divides a mesh representing an object with a three-dimensional structure into at least a first patch and a second patch, arranges the first patch and the second patch in a single image, and generates pair information indicating that at least one vertex of the first patch is paired with at least one vertex of the second patch that is located at the same position as the vertex of the first patch in the object before encoding; an encoding unit that encodes the pair information; An information processing device comprising: (2) The pair information is information indicating only the pair formed by the vertices included in the pair information. An information processing device according to (1). (3) The pair information is information that indicates the pair formed by the vertices not indicated by the pair information. An information processing device according to (1). (4) The pair information includes, with respect to a first edge of the first patch and a second edge of the second patch corresponding to the boundary between the first patch and the second patch in the object before encoding, information indicating a correspondence relationship between a combination of the vertices located at both ends of the first edge and the vertex located at an intermediate position of the first edge, and a combination of the vertices located at both ends of the second edge and the vertex located at an intermediate position of the second edge. (3) An information processing device according to the present invention. (5) The pair information includes, with respect to a first edge of the first patch and a second edge of the second patch corresponding to a boundary between the first patch and the second patch in the object before encoding, information indicating a correspondence relationship between a scan direction for the first edge at the time of decoding and a scan direction for the second edge at the time of decoding. An information processing device according to (3) or (4). (6) The pair information indicates that the plurality of vertices of the first patch and the corresponding plurality of vertices of the second patch are pairs. An information processing device according to any one of (1) to (5). (7) The pair information includes a parameter indicating the number of the vertices constituting the pair. (6) An information processing device according to (6). (8) The pair information includes information indicating a group of the vertices of the first patch and a group of the vertices of the second patch. An information processing device according to any one of (1) to (5). (9) The pair information indicates the pair using identification information corresponding to the state of each of the vertices in the object before encoding. An information processing device according to any one of (1) to (8). (10) The identification information is identification information for identifying each vertex among the vertices included in the mesh. (9) An information processing device according to (9). (11) The identification information is identification information for identifying each vertex among the vertices included in the patch. The information processing device according to (9) or (10). (12) The pair information further includes identification information of the patch. (11) An information processing device according to (11). (13) The pair information includes identification information of the patch for each of the pairs. (12) An information processing device according to (12). (14) The pair information includes identification information of the patch for each of the plurality of pairs. (12) An information processing device according to (12). (15) The identification information is identification information for identifying each vertex among the vertices located on the boundary of the patch. An information processing device according to any one of (9) to (14). (16) The encoding unit further encodes the list of vertices located on the boundary of the patch. (15) An information processing device according to (15). (17) The pair information indicates the pair using an absolute value of the identification information. An information processing device according to any one of (9) to (16). (18) The pair information indicates the pair using a relative value of the identification information from the identification information of the other vertex. An information processing device according to any one of (9) to (17). (19) The pair information indicates the pair based on the coordinates of each of the plurality of vertices in a two-dimensional plane of the image. An information processing device according to any one of (1) to (18). (20) The pair information indicates the pair using the coordinates of each vertex constituting the pair. (19) An information processing device according to (19). (21) The pair information indicates relative coordinates of the vertices different from the reference vertex, with the coordinates of the reference vertex being used as a reference. The information processing device according to (19) or (20). (22) The pair information indicates relative coordinates of the vertices different from the reference vertex, based on the position of a relative vector between the first patch and the second patch, based on the coordinates of a reference vertex among the plurality of vertices. An information processing device according to any one of (19) to (21). (23) The pair information indicates relative coordinates of the vertices, which are different from the reference vertex, in the second patch in a state where the orientation is corrected, based on the position of a relative vector between the first patch and the second patch, based on the coordinates of a reference vertex among the plurality of vertices. An information processing device according to any one of (19) to (22). (24) The pair information indicates relative coordinates of the vertex of the processing target pair that is different from the reference vertex, based on the relative coordinates of the vertex of the other pair that is different from the reference vertex, based on the coordinates of the reference vertex of the plurality of vertices of the other pair. An information processing device according to any one of (19) to (23). (25) Dividing a mesh representing an object having a three-dimensional structure into at least a first patch and a second patch, and arranging the divided patches in a single image, and generating pair information indicating that at least one vertex of the first patch is paired with at least one vertex of the second patch that is located at the same position as the vertex of the first patch in the object before encoding; The generated pair information is encoded. Information processing methods.

[0412] (31) A decoding unit that divides a mesh representing an object with a three-dimensional structure into at least a first patch and a second patch, arranges the divided patches in a single image, decodes encoded data of pair information indicating that at least one vertex of the first patch is paired with at least one vertex of the second patch that is located at the same position as the vertex of the first patch in the object before encoding, and obtains the pair information; a correction unit that corrects the reconstructed mesh based on the pair information; An information processing device comprising: (32) The correction unit deletes one of the vertices of the first patch and the vertices of the second patch that are indicated as being a pair by the pair information in the reconstructed mesh, and converts the connection of the deleted vertex to the connection of the other vertex. (31) An information processing device according to (31). (33) The correction unit sets the position of the single vertex to the position of any one of the plurality of vertices. (32) An information processing device according to (32). (34) The correction unit sets the position of the single vertex based on information indicating one of the plurality of vertices. (33) An information processing device according to (33). (35) The correction unit sets the position of the single vertex to the average of the positions of the plurality of vertices. An information processing device according to any one of (32) to (34). (36) The correction unit moves one of the vertices of the first patch and the vertex of the second patch, which are indicated as being a pair by the pair information, to a position of the other vertex in the reconstructed mesh. An information processing device according to any one of (31) to (35). (37) The correction unit sets the position of the overlapping point to the position of one of the plurality of vertices. (36) An information processing device according to (36). (38) The correction unit sets the position of the overlapping point based on information indicating one of the plurality of vertices. (37) An information processing device according to (37). (39) The correction unit sets the position of the overlapping point to the average of the positions of the plurality of vertices. An information processing device according to any one of (36) to (38). (40) The correction unit forms a polygon including the vertices of the first patch and the vertices of the second patch that are indicated as being the pair by the pair information in the reconstructed mesh. An information processing device according to any one of (31) to (39). (41) The correction unit interpolates the vertices that make up the pair and that are not included in the pair information. An information processing device according to any one of (31) to (40). (42) The correction unit performs interpolation of the vertices based on an occupancy image. (41) An information processing device according to (41). (43) The correction unit performs interpolation of the vertices based on connectivity information. The information processing device according to (41) or (42). (44) Dividing a mesh representing an object having a three-dimensional structure into at least a first patch and a second patch and arranging the divided patches in a single image, and decoding encoded data of pair information indicating that at least one vertex of the first patch is paired with at least one vertex of the second patch that is located at the same position as the vertex of the first patch in the object before encoding; The reconstructed mesh is corrected based on the pair information obtained by decoding the encoded data. Information processing methods.

[0413] (51) A pair information generating unit that divides a mesh representing an object with a three-dimensional structure into at least a first patch and a second patch, arranges the first patch and the second patch in a single image, and generates pair information indicating that at least one vertex of the first patch is paired with at least one vertex of the second patch that is located at the same position as the vertex of the first patch in the object before encoding; a pair information encoding unit that encodes the pair information; An information processing device comprising: (52) The pair information includes information indicating a destination position when each vertex constituting the pair is moved to the same position. (51) An information processing device according to (51). (53) a geometry encoding unit that encodes a geometry image in which the mesh geometry patches are arranged and generates encoded data of the geometry image; a geometry decoding unit that decodes the encoded data of the geometry image and generates the geometry image; Equipped with The pair information generating unit generates the pair information based on the geometry image generated by the geometry decoding unit. The information processing device according to (51) or (52). (54) Dividing a mesh representing an object having a three-dimensional structure into at least a first patch and a second patch, and arranging the divided patches in a single image, and generating pair information indicating that at least one vertex of the first patch is paired with at least one vertex of the second patch that is located at the same position as the vertex of the first patch in the object before encoding; Encoding the pair information Information processing methods.

[0414] (61) A pair information decoding unit that divides a mesh representing an object of a three-dimensional structure into at least a first patch and a second patch, arranges the divided patches in a single image, decodes encoded data of pair information indicating that at least one vertex of the first patch is paired with at least one vertex of the second patch that is located at the same position as the vertex of the first patch in the object before encoding, and obtains the pair information. a correction unit that corrects the geometry of the vertices that make up the pair based on the pair information; An information processing device comprising: (62) When the normal direction of the first patch and the normal direction of the second patch do not match, the correction unit moves the vertex of the first patch and the vertex of the second patch that make up the pair to an intersection of the normal of the first patch and the normal of the second patch. (61) An information processing device according to (61). (63) When a normal direction of the first patch and a normal direction of the second patch coincide with each other, the correction unit moves the vertices of the first patch and the vertices of the second patch that make up the pair to positions that are weighted averages of their respective positions. The information processing device according to (61) or (62). (64) The correction unit moves the first patch to a position indicated by the pair information when a normal direction of the first patch and a normal direction of the second patch coincide with each other. An information processing device according to any one of (61) to (63). (65) The correction unit moves vertices of the first patch and vertices of the second patch that make up the pair to the same positions in a geometry image generated by decoding the encoded data, a patch reconstruction unit that reconstructs a patch using the geometry image corrected by the correction unit; a vertex reconstruction unit that reconstructs vertices using the reconstructed patches; The information processing device according to any one of (61) to (64), further comprising: (66) Dividing a mesh representing an object having a three-dimensional structure into at least a first patch and a second patch and arranging the divided patches in a single image, decoding encoded data of pair information indicating that at least one vertex of the first patch and at least one vertex of the second patch that is located at the same position as the vertex of the first patch in the object before encoding are paired, and obtaining the pair information; The geometry of the vertices constituting the pair is corrected based on the pair information. Information processing methods. [Explanation of symbols]

[0415] 300 encoding device, 311 mesh voxelization unit, 312 patch generation unit, 313 pair information generation unit, 314 geometry image generation unit, 315 occupancy image generation unit, 316 pair information encoding unit, 317 meta information encoding unit, 318 to 320 2D encoding unit, 321 multiplexing unit, 331 image generation unit, 332 encoding unit, 400 decoding device, 411 demultiplexing unit, 412 pair information decoding unit, 413 meta information decoding unit, 414 to 416 2D decoding unit, 417 patch reconstruction unit, 418 vertex information reconstruction unit, 419 pairing processing unit, 600 encoding device, 611 mesh voxelization unit, 612 patch generation unit, 613 meta information encoding unit, 614 Image generation unit, 615 2D encoding unit, 616 2D decoding unit, 617 pair information generation unit, 618 pair information encoding unit, 619 pairing processing unit, 620 image generation unit, 621 2D encoding unit, 622 multiplexing unit, 700 decoding device, 711 demultiplexing unit, 712 pair information decoding unit, 713 meta information decoding unit, 714 2D decoding unit, 715 pairing processing unit, 716 patch reconstruction unit, 717 vertex information reconstruction unit, 718 2D decoding unit, 900 computer< / id> < / id> < / zippering> < / mesh> < / vpcc>

Claims

1. a generation unit that divides a mesh representing an object with a three-dimensional structure into at least a first patch and a second patch, arranges the divided patches within a single image, and generates pair information that indicates that at least one vertex of the first patch is paired with at least one vertex of the second patch that is located at the same position as the vertex of the first patch in the object before encoding; an encoding unit that encodes the pair information; Equipped with The pair information includes information indicating a correspondence relationship between a scan direction for the first edge at the time of decoding and a scan direction for the second edge at the time of decoding, the scan direction being related to a first edge of the first patch and a second edge of the second patch, the first edge and the second edge corresponding to the boundary between the first patch and the second patch in the object before encoding, and is information that is applied when correcting cracks in the reconstructed mesh. Encoding device.

2. The pair information is information that indicates only the pair that is formed by the vertices included in the pair information. The encoding device according to claim 1 .

3. The pair information includes information indicating a group of the vertices of the first patch and a group of the vertices of the second patch. The encoding device according to claim 1 .

4. The pair information indicates the pair based on the coordinates of each of the plurality of vertices in the two-dimensional plane of the image. The encoding device according to claim 1 .

5. The pair information indicates relative coordinates of the vertices that are different from the reference vertex, with the coordinates of the reference vertex being used as a reference.

5. The encoding device according to claim 4.

6. The pair information indicates relative coordinates of the vertices that are different from the reference vertex, based on the position of a relative vector between the first patch and the second patch, based on the coordinates of a reference vertex among the plurality of vertices. The encoding device according to claim 5 .

7. The pair information indicates relative coordinates of the vertices, which are different from the reference vertex, in the second patch in a state in which the orientation is corrected, based on the position of a relative vector between the first patch and the second patch, which is based on the coordinates of a reference vertex among the plurality of vertices. The encoding device according to claim 6.

8. The pair information indicates relative coordinates of the vertex of the processing target pair that is different from the reference vertex, based on the relative coordinates of the vertex of the other pair that is different from the reference vertex, based on the coordinates of the reference vertex of the plurality of vertices of the other pair.

5. The encoding device according to claim 4.

9. A generation unit that divides a mesh representing an object with a three-dimensional structure into at least a first patch and a second patch, arranges the patches within a single image, and generates pair information indicating that at least one vertex of the first patch is paired with at least one vertex of the second patch that is located at the same position as the vertex of the first patch in the object before encoding; an encoding unit that encodes the pair information; Equipped with The pair information includes information indicating a correspondence relationship between a combination of the vertices located at both ends of the first edge and the vertex located at the middle of the first edge, and a combination of the vertices located at both ends of the second edge and the vertex located at the middle of the second edge, with respect to a first edge of the first patch and a second edge of the second patch, which correspond respectively to the boundary between the first patch and the second patch in the object before encoding, and is information that is applied when correcting cracks in the reconstructed mesh. Encoding device.

10. A generation unit that divides a mesh representing an object of a three-dimensional structure into at least a first patch and a second patch, arranges the patches within a single image, and generates pair information indicating that at least one vertex of the first patch is paired with at least one vertex of the second patch that is located at the same position as the vertex of the first patch in the object before encoding; an encoding unit that encodes the pair information; Equipped with The pair information indicates that the plurality of vertices of the first patch and the corresponding plurality of vertices of the second patch are pairs, and is information to be applied when correcting cracks in the reconstructed mesh. Encoding device.

11. The pair information includes a parameter indicating the number of vertices that make up the pair. The encoding device according to claim 10.

12. A generation unit that divides a mesh representing an object of a three-dimensional structure into at least a first patch and a second patch, arranges the patches within a single image, and generates pair information indicating that at least one vertex of the first patch is paired with at least one vertex of the second patch that is located at the same position as the vertex of the first patch in the object before encoding; an encoding unit that encodes the pair information; Equipped with The pair information indicates the pair using identification information corresponding to the state of each of the vertices in the object before encoding, and is information that is applied when correcting cracks in the reconstructed mesh. Encoding device.

13. Dividing a mesh representing an object having a three-dimensional structure into at least a first patch and a second patch and arranging the divided patches in a single image, and generating pair information indicating that at least one vertex of the first patch is paired with at least one vertex of the second patch that is located at the same position as the vertex of the first patch in the object before encoding; Encoding the generated pair information; The pair information includes information indicating a correspondence relationship between a scan direction for the first edge at the time of decoding and a scan direction for the second edge at the time of decoding, the scan direction being related to a first edge of the first patch and a second edge of the second patch, the first edge and the second edge corresponding to the boundary between the first patch and the second patch in the object before encoding, and is information that is applied when correcting cracks in the reconstructed mesh. Encoding method.

14. A 2D decoding unit that obtains coded data of a geometry image and generates a geometry image; a pair information decoding unit that decodes encoded data of pair information indicating that at least one vertex of a first patch and at least one vertex of the second patch that is located at the same position as a vertex of the first patch in the object before encoding is a pair, with respect to a first patch and a second patch that represent a mesh that represents an object with a three-dimensional structure arranged in the geometry image, and obtains the pair information; a reconstruction unit that reconstructs a mesh from the geometry image; a correction unit that corrects the reconstructed mesh based on the pair information; Equipped with The correction unit deletes one of the vertices of the first patch and the vertices of the second patch that are indicated to be the pair by the pair information in the reconstructed mesh, and converts the connection of the deleted vertex to the connection of the other vertex. Decryption device.

15. A 2D decoding unit that obtains coded data of a geometry image and generates a geometry image; a pair information decoding unit that decodes encoded data of pair information indicating that at least one vertex of a first patch and at least one vertex of the second patch that is located at the same position as a vertex of the first patch in the object before encoding is a pair, with respect to a first patch and a second patch that represent a mesh that represents an object with a three-dimensional structure arranged in the geometry image, and obtains the pair information; a reconstruction unit that reconstructs a mesh from the geometry image; a correction unit that corrects the reconstructed mesh based on the pair information; Equipped with The correction unit moves one of the vertices of the first patch and the vertices of the second patch, which are indicated as being a pair by the pair information, to the position of the other vertex in the reconstructed mesh. Decryption device.

16. A 2D decoding unit that obtains encoded data of a geometry image and generates a geometry image; a pair information decoding unit that decodes encoded data of pair information indicating that at least one vertex of a first patch and at least one vertex of the second patch that is located at the same position as a vertex of the first patch in the object before encoding is a pair, with respect to a first patch and a second patch that represent a mesh that represents an object with a three-dimensional structure arranged in the geometry image, and obtains the pair information; a reconstruction unit that reconstructs a mesh from the geometry image; a correction unit that corrects the reconstructed mesh based on the pair information; Equipped with The correction unit forms a polygon including the vertices of the first patch and the vertices of the second patch that are indicated as being the pair by the pair information in the reconstructed mesh. Decryption device.

17. A 2D decoding unit that obtains encoded data of a geometry image and generates a geometry image; a pair information decoding unit that decodes encoded data of pair information indicating that at least one vertex of a first patch and at least one vertex of the second patch that is located at the same position as a vertex of the first patch in the object before encoding is a pair, with respect to a first patch and a second patch that represent a mesh that represents an object with a three-dimensional structure arranged in the geometry image, and obtains the pair information; a reconstruction unit that reconstructs a mesh from the geometry image; a correction unit that corrects the reconstructed mesh based on the pair information; Equipped with The correction unit interpolates the vertices that make up the pair and that are not included in the pair information. Decryption device.

18. Obtaining coded data of a geometry image and generating a geometry image; decoding encoded data of pair information indicating that, for a first patch and a second patch representing a mesh that represents an object of a three-dimensional structure arranged in the geometry image, at least one vertex of the first patch is paired with at least one vertex of the second patch that is located at the same position as the vertex of the first patch in the object before encoding; reconstructing a mesh from the geometry image; In the reconstructed mesh, a polygon is formed that includes the vertices of the first patch and the vertices of the second patch that are indicated as being the pair by the pair information obtained by decoding the encoded data. Decryption method.

Images