Information processing device and method

Abstract

The present disclosure relates to an information processing device and method that mitigate a decrease in encoding efficiency. At least some of internal vertices, which are the vertices of a mesh representing a three-dimensionally structured object that are located at a position other than the boundary of patches of a geometry, are deleted; vertex connection information indicating the vertices of the mesh and connections between the vertices is generated; and the vertex connection information is encoded. Further, encoded data of vertex connection information indicating boundary vertices, which are the vertices of a mesh representing the three-dimensionally structured object that are located at least at a position on the boundary of the patches of the geometry, and connections between the boundary vertices is decoded, and the vertex connection information obtained by decoding the encoded data and a geometry image in which the patches are arranged in a two-dimensional plane are used to reconfigure the vertices located in the patches and the connections between the vertices. The present disclosure is applicable, for example, to information processing devices, electronic instruments, information processing methods and programs.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus and method, and particularly to an information processing apparatus and method capable of suppressing a reduction in coding efficiency.

Background Art

[0002] Conventionally, there has been a mesh as 3D data representing an object with a three-dimensional shape. As a method for compressing this mesh, a method of extending VPCC (Video-based Point Cloud Compression) to compress the mesh has been proposed (see, for example, Non-Patent Document 1). In the case of this method, the geometry (position information) of the object is patched and encoded as a geometry image. Therefore, information indicating the position of each vertex in the geometry image is generated and encoded as meta-information.

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the case of this method, since the meta-information is generated for all vertices included in the geometry image, there is a risk of reducing the coding efficiency.

[0005] The present disclosure has been made in view of such a situation, and aims to suppress a reduction in coding efficiency. [Means for solving the problem]

[0006] One aspect of this technology is an information processing device comprising: a vertex connection information generation unit that deletes at least a portion of the internal vertices, which are vertices of a mesh representing a three-dimensional object, that are located outside the boundaries of a geometric patch, and generates vertex connection information indicating the vertices of the mesh and the connections between those vertices; and an encoding unit that encodes the vertex connection information.

[0007] One aspect of this technology is an information processing method that involves deleting at least some of the internal vertices, which are vertices of a mesh representing a three-dimensional object, that are located outside the boundaries of a geometric patch; generating vertex connection information that indicates the vertices of the mesh and the connections between those vertices; and encoding the vertex connection information.

[0008] The information processing device for another aspect of this technology includes a decoding unit that decodes encoded data of vertex connection information indicating connections between boundary vertices, which are vertices of a mesh representing a three-dimensional object located at least at the boundary of a geometric patch, and a geometric image in which the patch is arranged in a two-dimensional plane, using the vertex connection information obtained by decoding the encoded data by the decoding unit and the geometric image in which the patch is arranged in a two-dimensional plane. By linearly interpolating the boundary vertices, determining the patch region which is the area included in the patch, generating polygons, merging unnecessary polygons outside the patch region, and merging the polygons within the patch region based on the geometry, The information processing device includes a vertex connection reconstruction unit that reconstructs the vertices located in the aforementioned patch and the connections between those vertices. Another aspect of this technology involves an information processing method which involves decoding encoded data of vertex connection information indicating the connections between boundary vertices, which are vertices of a mesh representing a three-dimensional object located at least at the boundary of a geometric patch, and the connections between the boundary vertices; linearly interpolating the boundary vertices using the vertex connection information obtained from decoding the encoded data and a geometric image in which the patch is placed on a two-dimensional plane; determining the patch region, which is the area included in the patch; generating polygons; merging unnecessary polygons outside the patch region; and merging the polygons within the patch region based on the geometry, thereby reconstructing the vertices located in the patch and the connections between them.

[0009] An information processing device in yet another aspect of this technology comprises a decoding unit that decodes encoded data of vertex connection information indicating the connections between boundary vertices, which are vertices of a mesh representing a three-dimensional object located at least at the boundary of a geometric patch, and the connections between the boundary vertices; and a vertex connection reconstruction unit that uses the vertex connection information obtained by decoding the encoded data by the decoding unit and a geometric image of the patch arranged in a two-dimensional plane to divide the patch into a triangular region and a rectangular region, place the vertices in the rectangular region, determine the connections between the vertices in the rectangular region, and determine the connections between the vertices in the triangular region, thereby reconstructing the vertices located in the patch and the connections between the vertices. This technology moreover The other aspect of information processing method involves decoding encoded data of vertex connection information indicating the connections between boundary vertices, which are vertices of a mesh representing a three-dimensional structure object located at least at the boundary of the geometric patch, and using the vertex connection information obtained by decoding the encoded data and the geometric image in which the patch is arranged in a two-dimensional plane, By dividing the patch into a triangular region and a rectangular region, placing the vertices in the rectangular region, determining the connections between the vertices in the rectangular region, and determining the connections between the vertices in the triangular region,This is an information processing method for reconstructing the vertices located in the aforementioned patch and the connections between those vertices.

[0010] In one aspect of this technology, an information processing device and method, at least some of the internal vertices, which are vertices of a mesh representing a three-dimensional object and are located outside the boundaries of a geometric patch, are deleted. Vertex connection information indicating the vertices of the mesh and the connections between those vertices is generated, and this vertex connection information is encoded.

[0011] In other aspects of the information processing device and method of this technology, encoded data of vertex connection information indicating the connections between boundary vertices, which are vertices of a mesh representing a three-dimensional object located at the boundary of a geometric patch, is decoded, and the vertex connection information obtained by decoding the encoded data and a geometric image in which the patch is arranged in a two-dimensional plane are used. By linearly interpolating the boundary vertices, determining the patch region (the area included in the patch), generating polygons, merging unnecessary polygons outside the patch region, and merging the polygons within the patch region based on the geometry, The vertices located in that patch and the connections between those vertices are reconstructed. In other aspects of the information processing device and method of this technology, encoded data of vertex connection information indicating the connections between boundary vertices, which are vertices of a mesh representing a three-dimensional object located at least at the boundary of a geometric patch, and the connections between these boundary vertices is decoded. Using the vertex connection information obtained from the decoded encoded data and a geometric image in which the patch is placed on a two-dimensional plane, the patch is divided into triangular and rectangular regions, vertices are placed in the rectangular region, the connections between the vertices in the rectangular region are determined, and the connections between the vertices in the triangular region are determined, thereby reconstructing the vertices located in the patch and the connections between those vertices. [Brief explanation of the drawing]

[0012] [Figure 1] This is a diagram illustrating the video-based approach. [Figure 2] This is a diagram explaining mesh. [Figure 3] This is a diagram illustrating vertex connection information. [Figure 4] This diagram illustrates how vertices and connections are reconfigured. [Figure 5] This is a diagram illustrating boundary vertices. [Figure 6] This diagram illustrates an example of vertex connection information. [Figure 7] This diagram illustrates an example of vertex connection information. [Figure 8] This diagram illustrates an example of vertex connection information. [Figure 9] This is a diagram explaining dendritic cells. [Figure 10] This is a diagram for explaining the refinement of vertex positions. [Figure 11] This is a diagram for explaining the method of reconstructing connections between vertices. [Figure 12] This is a diagram for explaining the interpolation of boundary vertices. [Figure 13] This is a diagram for explaining the formation of polygons. [Figure 14] This is a diagram for explaining the merging of polygons outside the patch area. [Figure 15] This is a diagram for explaining the merging of polygons within the patch area. [Figure 16] This is a diagram for explaining the addition of internal vertices. [Figure 17] This is a diagram for explaining the method of reconstructing connections between vertices. [Figure 18] This is a diagram for explaining the method of setting boundaries. [Figure 19] This is a diagram for explaining the setting of triangular and rectangular areas. [Figure 20] This is a diagram for explaining the correction of boundaries. [Figure 21] This is a diagram for explaining the correction of boundaries. [Figure 22] This is a diagram for explaining the correction of boundaries. [Figure 23] This is a diagram for explaining the correction of boundaries. [Figure 24] This is a diagram for explaining the arrangement of internal vertices in a rectangular area. [Figure 25] This is a diagram for explaining the arrangement of internal vertices in a rectangular area. [Figure 26] This is a diagram for explaining the formation of connections in a rectangular area. [Figure 27] This is a diagram for explaining the formation of connections in a rectangular area. [Figure 28] This is a diagram for explaining the formation of connections in a triangular area. [Figure 29] This is a diagram for explaining the formation of connections in a triangular area. [Figure 30]This diagram illustrates the formation of connections within a triangular region. [Figure 31] This diagram illustrates the formation of connections within a triangular region. [Figure 32] This diagram illustrates the formation of connections within a triangular region. [Figure 33] This is a block diagram showing the main configurations of coding devices. [Figure 34] This is a flowchart illustrating an example of the encoding process. [Figure 35] This block diagram shows examples of the main configurations of a decoding device. [Figure 36] This is a flowchart illustrating an example of the decryption process. [Figure 37] This is a flowchart illustrating an example of the vertex connection reconstruction process. [Figure 38] This is a flowchart illustrating an example of the vertex connection reconstruction process. [Figure 39] This diagram illustrates the addition of internal vertices. [Figure 40] This diagram illustrates an example of vertex connection information. [Figure 41] This figure illustrates an example of occupancy image generation based on vertex connection information. [Figure 42] This figure illustrates an example of occupancy image generation based on vertex connection information. [Figure 43] This figure illustrates an example of occupancy image generation based on vertex connection information. [Figure 44] This diagram explains the front and back sides of a polygon. [Figure 45] This diagram illustrates an example of Omap correction information. [Figure 46] This is a block diagram showing the main configurations of coding devices. [Figure 47] This is a flowchart illustrating an example of the encoding process. [Figure 48] This flowchart illustrates an example of the Omap correction information generation process. [Figure 49]This block diagram shows examples of the main configurations of a decoding device. [Figure 50] This is a flowchart illustrating an example of the decryption process. [Figure 51] This is a flowchart illustrating an example of the occupancy image generation process. [Figure 52] This diagram illustrates an example of patch reconstruction. [Figure 53] This diagram illustrates an example of patch reconstruction. [Figure 54] This diagram illustrates an example of patch reconstruction. [Figure 55] This diagram illustrates an example of how polygons are added. [Figure 56] This diagram illustrates an example of how polygons are added. [Figure 57] This diagram illustrates an example of how polygons are added. [Figure 58] This is a block diagram showing common computer configurations. [Modes for carrying out the invention]

[0013] The following describes the forms for implementing this disclosure (hereinafter referred to as embodiments). The explanation will be given in the following order. 1. Mesh compression with VPCC extension 2. Transmission of vertex connection information 3. First Embodiment (Encoding Device) 4. Second Embodiment (Decoding Device) 5. Application Examples 6. Addendum

[0014] <1. Mesh compression with VPCC extension> <Supporting literature and other resources for technical content and terminology> The scope disclosed in this technology includes not only the contents described in the embodiments, but also the contents described in the following non-patent documents that were publicly known at the time of filing, as well as the contents of other documents referenced in the following non-patent documents.

[0015] Non-patent document 1: (mentioned above) Non-patent document 2: "Information technology - Coded Representation of Immersive Media - Part 5: Visual Volumetric Video-based Coding (V3C) and Video-based Point Cloud Compression (V-PCC)", ISO / IEC 23090-5:2020(E), ISO / IEC JTC 1 / SC 29 / WG 11 w19579

[0016] In other words, the content described in the aforementioned non-patent literature, as well as the content of other documents referenced in the aforementioned non-patent literature, can also serve as a basis for determining the support requirements.

[0017] <Point Cloud> Traditionally, 3D data such as point clouds have existed that represent three-dimensional structures based on point location information and attribute information.

[0018] For example, in the case of a point cloud, a three-dimensional structure (a three-dimensional object) is represented as a collection of many points. A point cloud consists of positional information (also called geometry) and attribute information (also called attributes) for each point. Attributes can include any information. For example, the attributes may include color information, reflectivity information, normal information, etc. for each point. In this way, point clouds have a relatively simple data structure and can represent any three-dimensional structure with sufficient accuracy by using a sufficiently large number of points.

[0019] <vpcc> VPCC (Video-based Point Cloud Compression) is one such point cloud encoding technique that encodes point cloud data, which is 3D data representing a three-dimensional structure, using a codec designed for two-dimensional images.

[0020] In VPCC, the geometry and attributes of a point cloud are broken down into smaller regions (also called patches), and each patch is projected onto a two-dimensional plane called a projection plane. For example, the geometry and attributes are projected onto one of the six faces of the bounding box that encloses the object. The geometry and attributes projected onto this projection plane are also called the projected image. Similarly, the patches projected onto the projection plane are also called the patch images.

[0021] For example, the geometry of point cloud 1, which represents a three-dimensional object as shown in Figure 1A, is decomposed into patch 2 as shown in Figure 1B, and each patch is projected onto the projection plane. In other words, a patch image of the geometry (a projected image for each patch) is generated. Each pixel value in the patch image of the geometry represents the distance from the projection plane to the point (depth value).

[0022] The attributes of point cloud 1 are also broken down into patches 2, similar to the geometry, and each patch is projected onto the same projection plane as the geometry. In other words, an attribute patch image of the same size and shape as the geometry patch image is generated. Each pixel value in the attribute patch image represents the attribute (color, normal vector, reflectivity, etc.) of the point at the same position in the corresponding geometry patch image.

[0023] Then, each patch image generated in this way is placed within a video sequence frame (also called a video frame). In other words, each patch image on the projection plane is placed on a predetermined two-dimensional plane.

[0024] For example, a frame image on which a geometry patch image is placed is also called a geometry video frame. This geometry video frame is also referred to as a geometry image or geometry map. The geometry image 11 shown in Figure 1C is a frame image (geometry video frame) on which a geometry patch image 3 is placed. This patch image 3 corresponds to patch 2 in Figure 1B (it is a projection of geometry patch 2 onto the projection plane).

[0025] Furthermore, a frame image on which attribute patch images are placed is also called an attribute video frame. This attribute video frame is also referred to as an attribute image or attribute map. Attribute image 12 shown in Figure 1D is a frame image (attribute video frame) on which attribute patch image 4 is placed. This patch image 4 corresponds to patch 2 in Figure 1B (it is attribute patch 2 projected onto the projection surface).

[0026] These video frames are then encoded using 2D image encoding methods such as AVC (Advanced Video Coding) or HEVC (High Efficiency Video Coding). In other words, point cloud data, which is 3D data representing a three-dimensional structure, can be encoded using a 2D image codec. Generally, 2D data encoders are more widely used and less expensive to implement than 3D data encoders. That is, by applying the video-based approach as described above, the increase in costs can be suppressed.

[0027] In addition, in such video-based approaches, occupancy images (also called occupancy maps) can be used. An occupancy image is map information that indicates the presence or absence of a projected image (patch image) for every NxN pixels of a geometry video frame or attribute video frame. For example, an occupancy image shows the region (NxN pixels) in the geometry image or attribute image where a patch image exists with a value of "1", and the region (NxN pixels) where a patch image does not exist with a value of "0".

[0028] Such occupancy images are encoded as separate data from geometry and attribute images and transmitted to the decoding side. By referring to this occupancy map, the decoder can determine whether or not a patch exists in that area, thereby suppressing the effects of noise and other factors caused by encoding and decoding, and reconstructing the point cloud more accurately. For example, even if the depth value changes due to encoding and decoding, the decoder can refer to the occupancy map and ignore the depth value of areas where no patch image exists (preventing it from being processed as positional information in 3D data).

[0029] For example, an occupancy image 13, as shown in Figure 1E, may be generated for the geometry image 11 in Figure 1C and the attribute image 12 in Figure 1D. In the occupancy image 13, the white areas represent a value of "1" and the black areas represent a value of "0".

[0030] Furthermore, this occupancy image can also be transmitted as a video frame, just like geometry video frames and attribute video frames. That is, it is encoded using a 2D image encoding method such as AVC or HEVC, similar to geometry and attributes.

[0031] In other words, in the case of VPCC, the geometry and attributes of the point cloud are projected onto the same projection plane and placed at the same position in the frame image. That is, the geometry and attributes of each point are associated with each other by their position in the frame image.

[0032] <mesh> By the way, in addition to point clouds, for example, meshes exist as 3D data representing three-dimensional objects. As shown in FIG. 2, a mesh represents the surface of an object in three-dimensional space by polygons, which are planes (polygons) surrounded by edges 22 connecting vertices 21. As 3D data representing the object, as shown in FIG. 2, this mesh and texture 23 attached to each polygon are included.

[0033] 3D data using a mesh, for example, as shown in the lower part of FIG. 2, consists of vertex information 31 composed of position information (three-dimensional coordinates (X, Y, Z)) of each vertex 21, connectivity 32 indicating vertices 21 and edges 22 constituting each polygon, texture image 33 which is map information of texture 23 attached to each polygon, and a UV map 34 indicating the position in the texture image 33 of the texture corresponding to each vertex 21 (that is, the position of each vertex 21 in the texture image 33). The UV map 34 indicates the position of each vertex by UV coordinates, which are coordinates on the texture image 33.

[0034] In the case of 3D data using a mesh, different from the case of the above-described VPCC, the UV map 34 shows the correspondence between each vertex 21 and the texture 23. Therefore, as in the example of FIG. 2, the texture image 33 is configured as map information independent of the vertex information 31 composed of the three-dimensional coordinates of each vertex. Therefore, in the texture image 33, the texture 23 of each polygon can be arbitrarily set in terms of its projection direction and resolution.

[0035] <Compression of Mesh Using VPCC> As a method for compressing 3D data using such meshes, for example, Non-Patent Document 1 has proposed a method for compressing (encoding) 3D data using meshes by extending the VPCC described above. In this method, the geometry (position information) of objects is patched and encoded as a geometry image, as shown in example C in Figure 1. As a result, information indicating the position of each vertex in the geometry image is generated and encoded as metadata.

[0036] However, with this method, metadata is generated for every vertex in the geometry image, which could potentially reduce encoding efficiency.

[0037] <2. Transmission of vertex connection information> Therefore, the encoder reduces the number of vertices that transmit positional information in the geometry image as metadata. In other words, the encoder generates vertex connection information indicating the positional information of some of the vertices of the mesh included in the geometry image, as well as the connections between those vertices, encodes this vertex connection information as metadata, and transmits it to the decoder.

[0038] In this process, the vertex connection information should include at least information about vertices located at the patch boundaries (also referred to as boundary vertices). In other words, as shown in the top row of the table in Figure 3, the encoder removes at least some of the vertices of the mesh included in the geometry image that are located outside the patch boundaries (inside the patch) (also referred to as internal vertices), and generates vertex connection information for the remaining vertices.

[0039] For example, in an information processing method, at least some of the internal vertices, which are vertices of a mesh representing a 3D structure object and are located outside the boundaries of a geometric patch, are deleted. Vertex connection information is then generated that shows the vertices of the mesh and the connections between those vertices, and this vertex connection information is encoded.

[0040] For example, an information processing device may include a vertex connection information generation unit that deletes at least some of the internal vertices, which are vertices of a mesh representing a three-dimensional object, that are located outside the boundaries of a geometric patch, and generates vertex connection information indicating the vertices of that mesh and the connections between those vertices, and an encoding unit that encodes that vertex connection information.

[0041] By doing so, the encoder can reduce the number of vertices indicated by the vertex connection information, thereby suppressing the increase in the amount of information in the vertex connection information. Therefore, the encoder can suppress the reduction in encoding efficiency caused by encoding the vertex connection information.

[0042] Furthermore, the decoder that receives this vertex connection information will reconstruct the vertices of the mesh contained in the geometry image and the connections between those vertices.

[0043] For example, in an information processing method, encoded data of vertex connection information indicating the connections between boundary vertices, which are the vertices of a mesh representing a 3D structure object located at least at the boundary of a geometric patch, is decoded, and the vertex connection information obtained by decoding the encoded data, along with a geometric image in which the patch is placed on a 2D plane, is used to reconstruct the vertices located in that patch and the connections between those vertices.

[0044] For example, an information processing device may include a decoding unit that decodes encoded data of vertex connection information indicating the connections between boundary vertices, which are vertices of a mesh representing a three-dimensional object located at least at the boundary of a geometric patch, and the connections between those boundary vertices; and a vertex connection reconstruction unit that uses the vertex connection information obtained by decoding the encoded data from the decoding unit and a geometric image in which the patch is arranged in a two-dimensional plane to reconstruct the vertices located in the patch and the connections between those vertices.

[0045] In this way, the decoder can reconstruct vertices and connections in the decoded geometry image. In other words, the decoder can suppress the reduction in the number of vertices in the mesh due to encoding and decoding. Therefore, the decoder can suppress the reduction in mesh quality caused by the encoder reducing the number of vertices in the vertex connection information. Thus, the reduction in the number of vertices in the vertex connection information becomes practically possible. In other words, the decoder can suppress the reduction in encoding efficiency caused by encoding the vertex connection information.

[0046] In this specification, the transmission of information from an encoder to a decoder includes not only transmission via any communication medium (i.e., transmission of information by communication) but also transmission via any storage medium (i.e., writing information to a storage medium and reading information from that storage medium).

[0047] <Vertex connection information> As explained with reference to Figure 2, in 3D data using a mesh, the geometry is represented only by vertices and consists of non-image data such as vertex information and connectivity. When encoding this 3D data using VPCC, a geometric image is generated from this information. In other words, the 3D data is divided into patches consisting of adjacent polygons, and each patch is projected onto a two-dimensional plane (projection plane), generating a patch image consisting of depth information from that projection plane. Then, this patch image is placed on the geometric image, which is a two-dimensional plane, and encoded as a frame image of a video sequence.

[0048] In this geometry image, the vertices of the mesh are not represented. Therefore, vertex connection information is generated, indicating where the vertices are in the geometry image and the connections between vertices, and this information is encoded as metadata.

[0049] The encoder reduces the amount of information in this vertex connection information by reducing the number of internal vertices. In other words, the encoder generates vertex connection information that includes information about boundary vertices, which are at least vertices located at the boundaries of a patch.

[0050] In other words, as shown in the third row from the top of the table in Figure 3, the vertex connection information generated by the encoder may include a "boundary vertex list," which is a list of boundary vertices that are located at the boundaries of a patch (patch image). In other words, the encoder may generate such a boundary vertex list and use it as vertex connection information. For example, the encoder may delete all internal vertices.

[0051] For example, Figure 5 shows patch image A, patch image B, patch image C, and patch image D. In the figure, the gray areas represent the areas within the patch (also called the patch regions). The white circles shown at the boundaries of each patch image (the boundary between the patch region and the non-patch region) indicate boundary vertices. The "#" followed by a number shown near each boundary vertex indicates identification information for that nearby boundary vertex (information for identifying vertices within the patch).

[0052] In the vertex connection information, a boundary vertex list may be shown for each boundary of the patch. For example, patch image A has a boundary where boundary vertices #0 to #4 are located, and a boundary where boundary vertices #5 to #8 are located. Patch image B has a boundary where boundary vertices #0 to #3 are located. Patch image C has a boundary where boundary vertices #2 to #4 are located. Patch image D has a boundary where boundary vertices #5 to #8 are located. A boundary vertex list may be generated for each such boundary.

[0053] As shown in Figure 5, the boundaries of a patch (patch image) can either have the patch region inside the boundary or the patch region outside the boundary. For example, the boundaries where boundary vertices #0 to #4 of patch image A are located, the boundaries where boundary vertices #0 to #3 of patch image B are located, the boundaries where boundary vertices #2 to #4 of patch image C are located, and the boundaries where boundary vertices #5 to #8 of patch image D are located are boundaries where the patch region is inside. On the other hand, the boundary where boundary vertices #5 to #8 of patch image A are located is a boundary where the patch region is outside.

[0054] A list of boundary vertices corresponding to a boundary whose interior is a patch region (a list consisting of boundary vertices located at that boundary and the connections between those boundary vertices) is called an inclusive list. Conversely, a list of boundary vertices corresponding to a boundary whose exterior is a patch region is called an exclusive list.

[0055] As shown in Figure 5, each patch (patch image) always has one boundary that forms the patch region on its inside. In contrast, a boundary that forms the patch region on its outside does not necessarily exist in a patch (patch image), as seen in patch images B through D. Patch image A has one boundary that forms the patch region on its outside, but a single patch (patch image) may have multiple boundaries that form the patch region on its outside.

[0056] In other words, as shown in the 10th row from the top of the table in Figure 3, the boundary vertex list may include an inclusive list in which the region enclosed by the boundary vertices that make up the list and the connections between those boundary vertices is included in the patch region. For example, one inclusive list may be formed for each patch.

[0057] Furthermore, as shown in the 11th row from the top of the table in Figure 3, the boundary vertex list may further include an exclusive list in which the region enclosed by the boundary vertices that make up the list and the connections between those boundary vertices is not included in the patch region. For example, for a single patch, one or more exclusive lists may be formed in addition to one inclusive list.

[0058] The boundary vertex list may contain arbitrary information. For example, as shown in the seventh row from the top of the table in Figure 3, the boundary vertex list may include identification information for each boundary vertex and positional information (UV coordinates) for each boundary vertex in the geometry image. Alternatively, as shown in the eighth row from the top of the table in Figure 3, the boundary vertex list may indicate the connections between boundary vertices in the order of the boundary vertices (list order). For example, in the boundary vertex list, each boundary vertex may be described in the order of the arrows shown in Figure 5 to indicate that adjacent boundary vertices are connected. In other words, the inclusive list and exclusive list may contain this information.

[0059] Figure 6 shows an example of vertex connection information. The vertex connection information 111 shown in Figure 6 includes a list of boundary vertices for each patch (patch image). In the figure, the "Inclusive" row shows the inclusive list, and the "Exclusive" row shows the exclusive list. In this example, the boundary vertex list consists of the identification information of each boundary vertex, indicated by "#number", and the UV coordinates of each boundary vertex.

[0060] For example, in the case of vertex connection information 111, the inclusive list of Patch A consists of boundary vertex #0 (UV coordinates (u00, v00)), boundary vertex #1 (UV coordinates (u10, v10)), boundary vertex #2 (UV coordinates (u20, v20)), and boundary vertex #4 (UV coordinates (u40, v40)). Boundary vertex #0 is connected to boundary vertex #1, boundary vertex #1 to boundary vertex #2, boundary vertex #2 to boundary vertex #4, and boundary vertex #4 to boundary vertex #0. The exclusive list of Patch A consists of boundary vertex #5 (UV coordinates (u50, v50)), boundary vertex #6 (UV coordinates (u60, v60)), boundary vertex #7 (UV coordinates (u70, v70)), and boundary vertex #8 (UV coordinates (u80, v80)). Then, boundary vertex #5 is connected to boundary vertex #6, boundary vertex #6 to boundary vertex #7, boundary vertex #7 to boundary vertex #8, and boundary vertex #8 to boundary vertex #5.

[0061] Similarly, the inclusive list for Patch B consists of boundary vertex #3 (UV coordinates (u31, v31)), boundary vertex #2 (UV coordinates (u21, v21)), boundary vertex #1 (UV coordinates (u11, v11)), and boundary vertex #0 (UV coordinates (u01, v01)). Boundary vertex #3 is connected to boundary vertex #2, boundary vertex #2 to boundary vertex #1, boundary vertex #1 to boundary vertex #0, and boundary vertex #0 to boundary vertex #3. The exclusive list for Patch B does not exist (none).

[0062] Similarly, the inclusive list of Patch C consists of boundary vertex #4 (UV coordinates (u42, v42)), boundary vertex #2 (UV coordinates (u22, v22)), and boundary vertex #3 (UV coordinates (u32, v32)). Boundary vertex #4 is connected to boundary vertex #2, boundary vertex #2 to boundary vertex #3, and boundary vertex #3 to boundary vertex #4. The exclusive list of Patch C does not exist (none).

[0063] Similarly, the inclusive list for Patch D consists of boundary vertex #5 (UV coordinates (u53, v53)), boundary vertex #8 (UV coordinates (u83, v83)), boundary vertex #7 (UV coordinates (u73, v73)), and boundary vertex #6 (UV coordinates (u63, v63)). Boundary vertex #5 and boundary vertex #8, boundary vertex #8 and boundary vertex #7, boundary vertex #7 and boundary vertex #6, and boundary vertex #6 and boundary vertex #5 are connected to each other. Also, there is no exclusive list for Patch D.

[0064] Furthermore, as shown in the fourth row from the top of the table in Figure 3, the vertex connection information may also include an internal vertex list, which is a list of internal vertices located within the patch region (non-boundary areas). As mentioned above, all internal vertices may be deleted, or only some of them may be deleted. In other words, the vertex connection information may include information about internal vertices. That is, for example, if only some of the internal vertices are deleted, the encoder may generate an internal vertex list showing information about such internal vertices, in addition to the boundary vertex list mentioned above, as part of the vertex connection information.

[0065] The internal vertex list may contain arbitrary information. For example, the internal vertex list may include identification information for the internal vertices and positional information for the internal vertices in the geometry image.

[0066] For example, in the case of vertex connection information 111 shown in Figure 6, the internal vertex list is shown in the "Non-boundary Vertex" row of each patch. For example, the internal vertex list of patch image A consists of internal vertex #9 (UV coordinates (u90, v90)) and internal vertex #10 (UV coordinates (u100, v100)). Note that there is no internal vertex list (none) in patch images B through D.

[0067] Alternatively, the boundary vertex list may contain only the identification information for each boundary vertex. For example, the encoder may generate a common boundary vertex list 120 as vertex connection information, as shown in Figure 7, and this common boundary vertex list 120 may indicate the UV coordinates of each boundary vertex.

[0068] An example of the boundary vertex list included in the vertex connection information in this case is shown in Figure 8A. In the boundary vertex list 121 shown in Figure 8A, the inclusive list and exclusive list are composed of boundary vertex identification information. In this case as well, similar to the example in Figure 6, the connections between boundary vertices are indicated by the order in which each boundary vertex is described.

[0069] Alternatively, the boundary vertex list may be composed of edge identification information. For example, in the common boundary vertex list 120 in Figure 7, edge identification information (E#1, E#2, E#3, E#4, E#5, E#6, E#7, E#8) is shown for the boundaries of the patch (boundary vertices and the connections between those boundary vertices), and the boundary list may be formed using this edge identification information. An example of the boundary vertex list in this case is shown in Figure 8B. In the boundary vertex list 122 shown in Figure 8B, the inclusive list and exclusive list are composed of edge identification information, not boundary vertex identification information.

[0070] For example, in the case of boundary vertex list 122, the inclusive list of patch image A consists of edges E#1, E#2, and E#4. The exclusive list of patch image A consists of edges E#5, E#6, E#7, and E#8.

[0071] Similarly, the inclusive list for Patch B consists of edges E#3, E#2, and E#1. The exclusive list for Patch B is none.

[0072] Similarly, the inclusive list for patch image C consists of edges E#4, E#2, and E#3. The exclusive list for patch image C is none.

[0073] Similarly, the inclusive list for patch image D consists of edges E#5, E#8, E#7, and E#6. The exclusive list for patch image D is none.

[0074] As shown in the fifth row from the top of the table in Figure 3, the vertex connection information may further include pair information indicating the correspondence between boundary vertices between patches. For example, the patch image shown in Figure 5 is obtained by dividing the 3D data of the mesh into patches and projecting them onto a 2D plane, as described above. Therefore, what was a single vertex or overlapping points (multiple vertices with the same position) in the 3D data may be represented as multiple vertices.

[0075] For example, Figure 5 shows that boundary vertices assigned the same identification information were a single vertex (or overlapping point) in the 3D data. That is, boundary vertex #0 of patch image A and boundary vertex #0 of patch image B are a single vertex (or overlapping point) in the 3D data. Similarly, boundary vertex #1 of patch image A and boundary vertex #1 of patch image B are a single vertex (or overlapping point) in the 3D data. Boundary vertex #2 of patch image A, boundary vertex #2 of patch image B, and boundary vertex #2 of patch image C are a single vertex (or overlapping point) in the 3D data. Boundary vertex #3 of patch image B and boundary vertex #3 of patch image C are a single vertex (or overlapping point) in the 3D data. Boundary vertex #4 of patch image A and boundary vertex #4 of patch image C are a single vertex (or overlapping point) in the 3D data. Boundary vertex #5 of patch image A and boundary vertex #5 of patch image D are a single vertex (or overlapping point) in the 3D data. Boundary vertex #6 of patch image A and boundary vertex #6 of patch image D are a single vertex (or overlapping point) in the 3D data. Boundary vertex #7 of patch image A and boundary vertex #7 of patch image D are a single vertex (or overlapping point) in the 3D data. Boundary vertex #8 of patch image A and boundary vertex #8 of patch image D are a single vertex (or overlapping point) in the 3D data. In other words, when the 3D data is reconstructed, these vertices are treated as a single vertex (or overlapping point).

[0076] In this specification, a "pair" (which can also be expressed as a "combination" or "group") consisting of multiple vertices of patches generated from a single vertex (or overlapping point) of such a mesh is referred to as a "pair." In other words, each vertex constituting a pair corresponds to an identical vertex in the mesh. To put it another way, the "pair information" indicating this pair is information that shows the correspondence between vertices of multiple patches (which vertices correspond to which vertices). Note that a pair may consist of three or more vertices. That is, one vertex of the mesh may be divided into three or more patches.

[0077] Furthermore, the process of matching vertices based on such pair information is also called the pairing process.

[0078] The encoder may include pair information indicating such pairs in the vertex connection information. For example, as shown in the example in Figure 5, the boundary vertices constituting a pair may be assigned the same identification information to each other, and the UV coordinates for each of these identification information may be shown in the common boundary vertex list 120 shown in Figure 7 to indicate that they constitute a pair. Furthermore, the edges formed by each boundary vertex constituting a pair are integrated (edges are joined) when the 3D data is reconstructed. Therefore, as shown in Figure 7, the same identification information may be assigned to each of these edges.

[0079] By including such pair information in the vertex connection information, the decoder can perform a pairing process based on that pair information when reconstructing the 3D data. Therefore, it is possible to suppress the reduction in the quality of the 3D data, such as suppressing the occurrence of cracks.

[0080] <Parameters for vertex connection reconstruction> As shown in the 12th row from the top of the table in Figure 3, the encoder may further generate parameters (also called vertex connection reconstruction parameters) used when reconstructing vertices and the connections between them, in addition to the vertex connection information described above. In other words, these vertex connection reconstruction parameters may be encoded as metadata together with the vertex connection information and transmitted to the decoder. The decoder may then reconstruct the vertices and connections based on these transmitted vertex connection reconstruction parameters.

[0081] These vertex connection reconstruction parameters can be anything, as long as they are used to reconstruct vertices and connections.

[0082] For example, as shown in the 13th row from the top of the table in Figure 3, the encoder may generate a flag (boundary_vertex_only_flag) as a parameter for vertex connection reconstruction, indicating whether or not at least some of the internal vertices of the mesh have been deleted. In other words, this flag (boundary_vertex_only_flag) may be transmitted from the encoder to the decoder. The decoder may then decide whether or not to perform vertex reconstruction based on this flag (boundary_vertex_only_flag).

[0083] For example, if boundary_vertex_only_flag is true (e.g., "1"), it indicates that at least some of the internal vertices have been removed and vertex connection information has been generated, as described above. Conversely, if boundary_vertex_only_flag is false (e.g., "0"), it indicates that vertex connection information showing all vertices included in the geometry image and the connections between those vertices has been generated, as in the conventional method.

[0084] In other words, if boundary_vertex_only_flag is true, the decoder reconstructs vertices and connections not included in the transmitted vertex connection information and generates connectivity, UV maps, vertex information, etc., that include them. If boundary_vertex_only_flag is false, the decoder generates connectivity, UV maps, connection information, etc., that include only the vertices and connections included in the transmitted vertex connection information.

[0085] By transmitting such flag information (boundary_vertex_only_flag) from the encoder to the decoder, the decoder can more easily and appropriately decide whether or not to reconstruct vertices or connections that are not included in the transmitted vertex connection information, based on this flag information (boundary_vertex_only_flag).

[0086] Furthermore, as shown in the 14th row from the top of the table in Figure 3, for example, the encoder may generate information about the density of vertices during reconstruction as a parameter for this vertex connection reconstruction. In other words, this information (density) may be transmitted from the encoder to the decoder. The decoder may then perform vertex reconstruction according to this information (density).

[0087] Density is a parameter that indicates the density (LoD) (sampling interval) of the internal vertices to be reconstructed. For example, Figure 9 shows an example of the reconstructed vertices. In Figure 9, the white circles indicate the reconstructed vertices. Figure 9A shows an example of reconstructed vertices when density=1. Figure 9B shows an example of reconstructed vertices when density=2. Figure 9C shows an example of reconstructed vertices when density=3. As these examples show, density=1 represents the highest density (in pixels), and as the density value increases, the spacing between vertices widens (density decreases).

[0088] The unit for setting this density is arbitrary. For example, density can be set per patch, or per frame (per geometry image).

[0089] This density is transmitted from the encoder to the decoder, allowing the decoder to reconstruct the vertices according to this density. In other words, the encoder can control the amount (quality) of information in the reconstructed 3D data.

[0090] Furthermore, for example, the decoder may be configured to correct the position of the reconstructed vertices based on the geometry image (depth values) when reconstructing them. Generally, the vertices of a mesh often occur at the extreme values ​​(maximum or minimum values) of the depth values ​​in the surrounding area. Therefore, if the decoder generates a vertex at a position other than this extreme value of the depth values ​​during reconstruction, it may correct the position of that vertex to the extreme value of the depth values.

[0091] In Figure 10A, the curve represents the pixel values ​​(i.e., depth values) of the geometry image. When correction is performed as described above, the decoder moves, for example, the position of vertex 141 generated by reconstruction to the position of the local minimum depth value in its vicinity (vertex 142). Similarly, the decoder moves the position of vertex 143 generated by reconstruction to the position of the local maximum depth value in its vicinity (vertex 144).

[0092] If the decoder is capable of correcting such vertex positions, the encoder may generate a flag (refinement_enable_flag) as a vertex connection reconstruction parameter, for example, as shown in the 15th row from the top of the table in Figure 3, indicating whether or not the decoder corrects the positions of the vertices it reconstructs. In other words, this flag (refinement_enable_flag) may be transmitted from the encoder to the decoder. The decoder may then decide whether or not to correct the vertices based on this flag (refinement_enable_flag).

[0093] For example, if refinement_enable_flag is true (e.g., "1"), it indicates that vertex position correction will be performed. Conversely, if refinement_enable_flag is false (e.g., "0"), it indicates that vertex position correction will not be performed. In other words, the decoder will perform vertex position correction if the transmitted refinement_enable_flag is true, and will not perform vertex position correction if this refinement_enable_flag is false.

[0094] In this way, by transmitting the refinement_enable_flag from the encoder to the decoder, the encoder can control whether or not the decoder performs vertex position correction.

[0095] Furthermore, for example, as shown in the 16th row from the top of the table in Figure 3, the encoder may generate information (refinement_type) regarding the geometric features that the decoder aims to correct for the reconstructed vertices, as a parameter for this vertex connection reconstruction. In other words, this information (refinement_type) may be transmitted from the encoder to the decoder. The decoder may then perform vertex correction according to this information (refinement_type).

[0096] The `refinement_type` parameter specifies the type of extreme value to target for correction (for example, only local minimals, only local maximals, or both local minimals and maximals). For example, if the value of `refinement_type` indicates only local minimals, the destination of the vertex due to correction is limited to the location where the depth value is at its minimum. Similarly, if the value of `refinement_type` indicates only local maximals, the destination of the vertex due to correction is limited to the location where the depth value is at its maximum. Furthermore, if the value of `refinement_type` indicates both local minimals and maximals, the destination of the vertex due to correction may be either the location where the depth value is at its minimum or the location where the depth value is at its maximum.

[0097] This refinement_type is transmitted from the encoder to the decoder, allowing the encoder to control how the decoder corrects the vertex positions.

[0098] Furthermore, as shown in the 17th row from the top of the table in Figure 3, for example, the encoder may generate information (refinement_range) regarding the search range of the target correction for the vertices to be reconstructed, as a parameter for this vertex connection reconstruction. In other words, this information (refinement_range) may be transmitted from the encoder to the decoder. The decoder may then perform vertex correction according to this information (refinement_range).

[0099] `refinement_range(r)` is a parameter that indicates the search range (the range for searching for extreme values ​​(which is also the range in which vertices can be moved)). For example, as shown in Figure 10B, the search for the extreme value of the depth value that is the target for correction of vertex 145 is performed within the distance r specified by `refinement_range`. By limiting the search range for the extreme value of the depth value in this way, it is possible to suppress an increase in the load of the correction process. Furthermore, by transmitting this `refinement_range` from the encoder to the decoder, the encoder can control the search range for the extreme value of the depth value in the vertex position correction performed by the decoder.

[0100] In other words, this information (refinement_range) indicates the movable range (correctable range) of the reconstructed vertices. Therefore, by transmitting this refinement_range from the encoder to the decoder, the encoder can control the movable range (correctable range) of the vertex positions by the decoder.

[0101] Furthermore, for example, as shown in the 18th row from the top of the table in Figure 3, the encoder may generate information regarding the corrected minimum interval between vertices to be reconstructed (refinement_minimum_interval) as a parameter for vertex connection reconstruction. In other words, this information (refinement_minimum_interval) may be transmitted from the encoder to the decoder. The decoder may then perform the correction according to this information (refinement_minimum_interval).

[0102] `refinement_minimum_interval` is a parameter that indicates the minimum acceptable distance (minimum interval) between vertices after correction. The decoder corrects the vertex positions according to this information. In other words, the decoder corrects the vertex interval so that it does not become smaller than the distance specified by `refinement_minimum_interval`.

[0103] Therefore, by transmitting this refinement_minimum_interval from the encoder to the decoder, the encoder can control how the decoder corrects the vertex positions. In other words, the encoder can control the quality of the corrected 3D data.

[0104] Furthermore, for example, as shown in the 19th row from the top of the table in Figure 3, the encoder may generate information (connectivity_type) regarding the method of connection between the vertices to be reconstructed as a parameter for vertex connection reconstruction. In other words, this information (connectivity_type) may be transmitted from the encoder to the decoder. The decoder may then reconstruct the connections between vertices according to this information (connectivity_type).

[0105] The `connectivity_type` parameter indicates how vertices are connected. For example, if `connectivity_type` is `triangle`, the decoder reconfigures the connections between vertices to form a polygon (triangle). If `connectivity_type` is `non-triangle`, the decoder reconfigures the connections between vertices to form a shape other than a polygon (triangle).

[0106] Note that the unit for setting `connectivity_type` is arbitrary. For example, `connectivity_type` could be set for each patch, or for each boundary vertex.

[0107] By transmitting this `connectivity_type` from the encoder to the decoder, the encoder can control how the decoder reconfigures the connection.

[0108] Furthermore, for example, as shown in the 20th row from the top of the table in Figure 3, the encoder may generate information regarding the merging method of the polygon composed of the vertices to be reconstructed, as parameters for this vertex connection reconstruction. In other words, this information may be transmitted from the encoder to the decoder. The decoder may then perform the polygon merging according to this information.

[0109] For example, information specifying which algorithm to apply as a polygon merging method (mesh reduction method) may be transmitted from the encoder to the decoder, and the decoder may perform polygon merging according to this information.

[0110] Note that the unit for setting this information is arbitrary. For example, it could be set to be configured for each patch.

[0111] By transmitting this information from the encoder to the decoder, the encoder can control how the decoder merges the polygons.

[0112] <Boundary vertex mode application determination> Furthermore, when the encoder generates vertex connection information as described above, it may also determine whether or not to apply a boundary vertex mode, which deletes some of the internal vertices included in the geometry image, as shown in the 21st row from the top of the table in Figure 3. In other words, the encoder may determine whether or not to delete the internal vertices of the mesh based on predetermined conditions, and if it determines that to delete them, it may apply the boundary vertex mode and delete at least some of the internal vertices of the mesh to generate vertex connection information.

[0113] The conditions for this determination are arbitrary. For example, as shown in the 22nd row from the top of the table in Figure 3, the encoder may compare the number of internal vertices with the number of boundary vertices and, based on the comparison result, determine whether or not to apply boundary vertex mode (i.e., whether or not to delete the internal vertices of the mesh).

[0114] For example, the encoder may decide to apply boundary vertex mode if the number of internal vertices is sufficiently large compared to the number of boundary vertices (i.e., sufficient reduction in the number of vertices is possible). In other words, the encoder may decide not to apply boundary vertex mode if the number of internal vertices is not sufficiently large compared to the number of boundary vertices (i.e., sufficient reduction in the number of vertices is not expected).

[0115] For example, if the number of boundary vertices is extremely large compared to the number of internal vertices, reducing the number of internal vertices may not sufficiently reduce the amount of information in the vertex connection data. In other words, it may not be possible to sufficiently reduce the encoding efficiency. In such cases, by not applying boundary vertex mode, the encoder can suppress the increase in the encoding processing load.

[0116] In contrast, if, for example, the number of internal vertices is sufficiently greater than the number of boundary vertices, a significant reduction in the amount of information in vertex connection data can be expected by reducing the number of internal vertices. Therefore, in such cases, by applying boundary vertex mode, the encoder can sufficiently suppress the reduction in encoding efficiency.

[0117] Alternatively, as shown in the bottom row of the table in Figure 3, the encoder may determine whether or not to apply boundary vertex mode (i.e., whether or not to delete the internal vertices of the mesh) based on the magnitude of the distortion caused by deleting at least some of the internal vertices and reconstructing the internal vertices.

[0118] For example, the encoder actually deletes internal vertices, then reconstructs them, and determines a threshold for the difference (magnitude of distortion) between the internal vertices before and after the process. If the distortion is below the threshold, the encoder applies boundary vertex mode. In other words, if the distortion is greater than the threshold, the encoder does not apply boundary vertex mode. That is, the encoder applies boundary vertex mode only if the amount of change in the internal vertices caused by applying boundary vertex mode is within an acceptable range.

[0119] By doing so, the encoder can suppress the reduction in 3D data quality caused by applying boundary vertex mode.

[0120] <Vertex connection reconstruction method 1> Next, we will explain how the decoder reconstructs vertices and the connections between them. The method of reconstructing vertices and connections is arbitrary.

[0121] For example, as shown in the second row from the top of the table in Figure 4, the decoder may reconstruct the vertices located in the patch and the connections between those vertices by linearly interpolating the boundary vertices, determining the patch region which is the area included in the patch, generating polygons, merging unnecessary polygons outside the patch region, and merging the polygons of the patch region based on the geometry (depth value).

[0122] For example, suppose that the vertex connection information transmitted from the encoder indicates the boundary vertices for a certain patch, as shown by the white circles in Figure 11. The decoder linearly interpolates these boundary vertices, as shown by the gray circles in Figure 12. In doing so, the decoder interpolates the boundary vertices at intervals based on parameters such as density.

[0123] The decoder then determines the patch region based on these boundary vertices. For example, if the boundary vertices shown in Figure 11 constitute an inclusive list, the inner region enclosed by the boundary vertices shown in Figure 12 is determined to be the patch region. Also, for example, if the boundary vertices shown in Figure 11 constitute an exclusive list, the outer region enclosed by the boundary vertices shown in Figure 12 is determined to be the patch region.

[0124] Next, the decoder generates polygons for the processing area, as shown in Figure 13, for example. In Figure 13, only the connections that form the polygons are shown, and the vertices are omitted. In this case, the decoder generates polygons such that the vertices are formed at intervals based on parameters supplied from an encoder, such as density. For example, the decoder places vertices in voxel units and connects them according to a predetermined rule (generating connectivity).

[0125] Next, the decoder merges any unnecessary polygons outside the patch region, as shown in Figure 14. In other words, the decoder removes polygons (vertices and connections) outside the patch region.

[0126] Furthermore, as shown in the third row from the top of the table in Figure 4, when applying pair information as described above, the decoder may also delete the linearly interpolated boundary vertices (gray circles). This is because linearly interpolated boundary vertices do not form pairs with boundary vertices of other patches, and the presence of such vertices may prevent the pairing process from being executed correctly. In other words, by deleting the linearly interpolated boundary vertices, the decoder can perform the pairing process correctly.

[0127] Next, the decoder merges the polygons in the patch region based on the geometry (depth value), as shown in Figure 15. In other words, the decoder performs mesh reduction using the depth value. This merging method is arbitrary. For example, the decoder may apply a method based on parameters supplied from the encoder (such as polygon merging method specification information).

[0128] For example, the decoder may determine whether each polygon can be merged based on the change in the depth value of neighboring vertex positions. Also, if an internal vertex list (non-boundary vertices) exists, the decoder merges the polygons in the patch region in a way that does not merge (delete) the internal vertices (black circles in Figure 16) included in that internal vertex list, as shown in Figure 16.

[0129] Furthermore, as described above, the decoder may correct the positions of the reconstructed vertices based on, for example, the geometry (depth value). In that case, the decoder may correct the vertices based, for example, on parameters for vertex connection reconstruction transmitted from the encoder.

[0130] <Vertex Connection Reconstruction Method 2> Alternatively, as shown in the bottom row of the table in Figure 4, for example, the decoder may reconstruct the vertices located in the patch and the connections between them by correcting the patch boundaries as needed, dividing the patch into triangular and rectangular regions, placing vertices in the rectangular regions, determining the connections between vertices in the rectangular regions, and determining the connections between vertices in the triangular regions.

[0131] For example, suppose the patch boundaries are shown in Figure 17 based on vertex connection information transmitted from the encoder. In Figure 17, the area enclosed by lines (the inner area) is the patch region.

[0132] The decoder divides this patch region into a triangular region and a rectangular region. For example, the decoder sets two candidate points (x1, y2) and (x2, y1) for two boundary vertices (x1, y1) and (x2, y2). The decoder then selects the points that lie inside the polygon from these candidate points and uses them as points on the boundary of the rectangular region. The decoder sets points in this way and sets a rectangular region within the patch region based on these points, as shown in the example in Figure 19. The decoder then sets a triangular region (a region composed of triangles) outside of that rectangular region.

[0133] In the example in Figure 18, if neither candidate point is inside the polygon, the decoder may perform a similar determination using the next vertex.

[0134] In selecting candidate points in this way, if it is not possible to set points within the polygon, the decoder may correct the shape of the patch boundary.

[0135] For example, the decoder may correct the shape of the patch boundary by moving the boundary vertices. For instance, the decoder may change the shape of the patch boundary, shown by the thick line, as shown in Figure 21, by moving the vertices shown by the white circles in Figure 20 as indicated by the arrows.

[0136] Furthermore, the decoder may correct the shape of the patch boundary by adding vertices, for example. For instance, the decoder may change the shape of the patch boundary of a patch having a boundary shape as shown by the thick line in Figure 22 by adding vertices as shown by the white circles in Figure 23.

[0137] Once a rectangular region is defined, the decoder places the internal vertices within that region, as shown in Figure 24. In Figure 24, the internal vertices indicated by black circles represent the internal vertices reconfigured within the rectangular region. The decoder places them uniformly at intervals according to, for example, density (LoD). However, necessary vertices may be placed separately. For example, lines may be drawn in the u and v directions for each of the vertices of the rectangular corners and boundary vertices, and the intersection points of these lines (within the rectangular region) and the vertices of the rectangular corners may be placed.

[0138] Figure 24 shows an example where vertices are arranged to be dense. Figure 25 shows an example where vertices are arranged to be sparse.

[0139] Once the internal vertices are placed, the decoder connects the internal vertices of the rectangular region in a predetermined manner (pattern). This connection pattern is arbitrary. For example, as shown in the table in Figure 26, the connections may be made in a pattern corresponding to the position relative to the center of the patch image (the point at the UV coordinates (center of the U axis, center of the V axis)). In this example, polygon (connection) patterns are provided for each of the four regions: upper left, upper right, lower left, and lower right of the center of the patch image. For example, in pattern 1, connectivity is formed to surround the center of the patch image (gray dot in the figure), as shown in the example in Figure 27. In pattern 2, connectivity is formed to extend radially from the center of the patch image.

[0140] Next, the decoder sets the connectivity for the triangular region. The decoder may interpolate vertices as needed. The method of setting this connectivity is arbitrary.

[0141] For example, suppose there is a triangular region as shown in Figure 28A. The boundary where vertices 152, 153, and 154 of this triangular region are located (the base of the triangular region in the figure) is the boundary adjacent to a rectangular region (also called the rectangular region boundary surface). Furthermore, the vertex furthest from the rectangular region boundary surface of the triangular region in the perpendicular direction (i.e., vertex 151) is also called the peak of the mountain.

[0142] The decoder may connect the vertices of such a triangular region, as shown in Figure 28B, connecting the peak of the mountain (vertex 151) to each vertex of the rectangular region boundary (vertices 152 to 154). In other words, the decoder may form connections 155 to 157.

[0143] Furthermore, as shown in Figure 29A, if there is a step (a right-angle bend) at the boundary of the rectangular region, the decoder may add connections to form a slope at that step. For example, in the case of example A in Figure 29, steps are formed at vertices 161, 162, and 163. The decoder forms a connection 164 connecting vertices 161 and 163 at this step.

[0144] Furthermore, as shown in Figure 29B, the decoder may add vertices (black circles in the figure) within the triangular region and form connections between the added vertices and the vertices of the rectangular region boundary. In the example of Figure 29B, the decoder adds vertex 171 to form connections 172 and 173, and adds vertex 174 to form connections 175 and 176.

[0145] Furthermore, the decoder may add vertices only to the internal polygons of the corner regions (polygons that do not contain the peaks of the mountain). For example, the decoder may add vertices and connections only to the areas shown in gray in Figure 30. In the example shown in Figure 30, vertices 181 to 188 are formed at the boundary of the triangular region, with vertices 181 and 182 forming the peaks of the mountain, and vertices 183 to 188 located on the boundary surface of the rectangular region. These points then form polygons 191 to 194. In such a case, the decoder may add vertices only to polygons 193 and 194 that do not contain the peaks of the mountain.

[0146] Furthermore, the decoder may form connections between vertices other than the peaks of the mountain, as shown in the examples in Figure 31A and Figure 32A. For example, it may form connections between the added vertices and the vertices of the rectangular region boundary. In addition, connections between the peaks of the mountain and other vertices may be added, as shown in the examples in Figure 31B and Figure 32B.

[0147] <3. First Embodiment> <Encoding device> The technology described above can be applied to any device. For example, the technology can be applied to an encoding device 300 as shown in Figure 33. Figure 33 is a block diagram showing an example of the configuration of an encoding device, which is one embodiment of an information processing device to which the technology is applied. The encoding device 300 shown in Figure 33 is a device that extends VPCC to encode 3D data using a Mesh as video frames using an encoding method for 2D images. In this case, the encoding device 300 encodes by applying one of the various methods of the technology described above, or by applying a combination of multiple methods.

[0148] Note that Figure 33 shows the main components such as the processing unit and data flow, and does not necessarily represent everything. In other words, the encoding device 300 may have processing units that are not shown as blocks in Figure 33, or processes and data flows that are not shown as arrows or other symbols in Figure 33.

[0149] As shown in Figure 33, the encoding device 300 includes a mesh voxelization unit 311, a patch generation unit 312, a vertex connection update unit 313, a geometry image generation unit 314, an occupancy image generation unit 315, a texture image generation unit 316, a metadata encoding unit 317, a 2D encoding unit 318, a 2D encoding unit 319, a 2D encoding unit 320, a multiplexing unit 321, and a vertex connection reconstruction unit 322. The geometry image generation unit 314, the occupancy image generation unit 315, and the texture image generation unit 316 may be considered as the image generation unit 331 in this disclosure. Also, the metadata encoding unit 317, the 2D encoding unit 318, the 2D encoding unit 319, and the 2D encoding unit 320 may be considered as the encoding unit 332 in this disclosure.

[0150] The encoding device 300 is supplied with 3D data using a mesh, including connectivity 351, vertex information 352, UV map 353, and texture 354.

[0151] Connectivity 351 contains the same information as Connectivity 32 (Figure 2), indicating each vertex that forms a polygon (each vertex that connects to each other) for each polygon. Vertex Information 352 contains the same information as Vertex Information 31 (Figure 2), indicating the coordinates of each vertex that forms the mesh. UV Map 353 contains the same information as UV Map 34 (Figure 2), indicating the position of each vertex on the texture image. Texture 354 contains the same information as Texture Image 33 (Figure 2), indicating the texture that is applied to the polygon. In other words, Texture 354 contains information including the texture image.

[0152] The mesh voxelization unit 311 acquires vertex information 352 that is supplied to the encoding device 300. The mesh voxelization unit 311 converts the coordinates of each vertex included in the acquired vertex information 352 into a voxel grid. The mesh voxelization unit 311 supplies the vertex information 352 of the converted voxel grid to the patch generation unit 312.

[0153] The patch generation unit 312 acquires connectivity information 351 supplied to the encoding device 300. The patch generation unit 312 also acquires vertex information 352 of the voxel grid supplied from the mesh voxelization unit 311. Based on this information, the patch generation unit 312 generates patches of geometry. The patch generation unit 312 also projects the generated geometric patches onto a projection plane to generate a patch image.

[0154] The patch generation unit 312 supplies the generated patch image, connectivity information 351, and vertex information 352 to the vertex connection update unit 313. The patch generation unit 312 also supplies the generated patch image to the geometry image generation unit 314. The patch generation unit 312 also supplies the generated patch image to the occupancy image generation unit 315. Furthermore, the patch generation unit 312 supplies the generated patch image to the texture image generation unit 316.

[0155] The vertex connection update unit 313 acquires information such as the patch image, connectivity 351, and vertex information 352 supplied from the patch generation unit 312.

[0156] The vertex connection update unit 313 generates vertex connection information based on that information. In doing so, the vertex connection update unit 313 generates vertex connection information by applying the technology described in the section <2. Transmission of Vertex Connection Information> (including the sections <Vertex Connection Information> to <Boundary Vertex Mode Application Determination>), etc. In other words, the vertex connection update unit 313 deletes at least some of the internal vertices of the mesh vertices included in the geometry image and generates vertex connection information for the remaining vertices (including at least boundary vertices). As a result, vertex connection information including the information described above in the section <Vertex Connection Information> is generated.

[0157] Furthermore, the vertex connection update unit 313 may generate vertex connection reconstruction parameters used when reconstructing vertices and the connections between vertices, as described above in the <Vertex Connection Reconstruction Parameters> section.

[0158] Furthermore, as described above in the section on <Boundary Vertex Mode Application Determination>, the vertex connection update unit 313 may determine whether or not to apply the boundary vertex mode, and only if it does apply, it may generate vertex connection information as described above. If the boundary vertex mode is not applied, the vertex connection update unit 313 generates vertex connection information for all mesh vertices included in the geometry image.

[0159] The vertex connection update unit 313 supplies the generated vertex connection information and vertex connection reconstruction parameters as metadata to the metadata encoding unit 317. The vertex connection update unit 313 also supplies the generated vertex connection information to the vertex connection reconstruction unit 322. The vertex connection update unit 313 may also supply the generated vertex connection reconstruction parameters to the vertex connection reconstruction unit 322.

[0160] The image generation unit 331 performs processing related to the generation of images (frame images). The geometry image generation unit 314 acquires patch images supplied from the patch generation unit 312. The geometry image generation unit 314 arranges the patch images on a two-dimensional plane and generates a geometry image. The geometry image generation unit 314 supplies the geometry video frame to the 2D encoding unit 318. The geometry image generation unit 314 also supplies the generated geometry image to the vertex connection reconstruction unit 322.

[0161] The occupancy image generation unit 315 acquires the patch image supplied from the patch generation unit 312. The occupancy image generation unit 315 generates an occupancy image using the patch image. The occupancy image generation unit 315 supplies the generated occupancy image to the 2D encoding unit 319.

[0162] The texture image generation unit 316 acquires the patch image supplied from the patch generation unit 312. The texture image generation unit 316 also acquires information about the reconstructed vertices and connections supplied from the vertex connection reconstruction unit 322. Furthermore, the texture image generation unit 316 acquires the UV map 353 and texture 354. Since the geometry image was generated independently of the texture 354, there is a possibility that the texture image and the geometry image do not match (the shape, size, arrangement, etc. of the patch images are different from each other). Therefore, the texture image generation unit 316 updates the texture image (texture 354) to match the geometry image. In other words, the texture image generation unit 316 updates the texture image so that the shape, size, arrangement, etc. of the patch images are identical to those of the geometry image. The texture image generation unit 316 performs this update using the information about the reconstructed vertices and connections and the UV map 353 supplied from the vertex connection reconstruction unit 322. The texture image generation unit 316 supplies the updated texture image to the 2D encoding unit 320.

[0163] The encoding unit 332 performs encoding-related processing. The metadata encoding unit 317 acquires metadata (including vertex connection information and parameters for vertex connection reconstruction) supplied from the vertex connection update unit 313. The metadata encoding unit 317 encodes the acquired metadata and generates encoded metadata data. The encoding scheme applied to this encoding is arbitrary. The metadata encoding unit 317 supplies the generated encoded metadata data to the multiplexing unit 321.

[0164] The 2D encoding unit 318 acquires the geometry image supplied from the geometry image generation unit 314. The 2D encoding unit 318 encodes the acquired geometry image using an encoding scheme for 2D images and generates encoded geometry image data. The 2D encoding unit 318 supplies the generated encoded geometry image data to the multiplexing unit 321.

[0165] The 2D encoding unit 319 acquires the occupancy image supplied from the occupancy image generation unit 315. The 2D encoding unit 319 encodes the acquired occupancy image using an encoding scheme for 2D images and generates encoded data for the occupancy image. The 2D encoding unit 319 supplies the generated encoded data for the occupancy image to the multiplexing unit 321.

[0166] The 2D encoding unit 320 acquires the texture image supplied to the encoding device 300. The 2D encoding unit 320 encodes the acquired texture image using an encoding method for 2D images and generates encoded texture image data. The 2D encoding unit 320 supplies the generated encoded texture image data to the multiplexing unit 321.

[0167] The multiplexing unit 321 acquires encoded metadata data supplied from the metadata encoding unit 317. The multiplexing unit 321 also acquires encoded geometry image data supplied from the 2D encoding unit 318. Furthermore, the multiplexing unit 321 acquires encoded occupancy image data supplied from the 2D encoding unit 319. Finally, the multiplexing unit 321 acquires encoded texture image data supplied from the 2D encoding unit 320. The multiplexing unit 321 then multiplexes this acquired information to generate a single bitstream. The multiplexing unit 321 outputs the generated bitstream to the outside of the encoding device 300.

[0168] The vertex connection reconstruction unit 322 acquires vertex connection information supplied from the vertex connection update unit 313. If vertex connection reconstruction parameters are supplied from the vertex connection update unit 313, the vertex connection reconstruction unit 322 also acquires those vertex connection reconstruction parameters. In addition, the vertex connection reconstruction unit 322 acquires the geometry image supplied from the geometry image generation unit 314.

[0169] The vertex connection reconstruction unit 322 uses the geometry image to reconstruct vertices and connections not included in the vertex connection information. In other words, the vertex connection reconstruction unit 322 adds vertices and connections to the geometry image. In doing so, the vertex connection reconstruction unit 322 may apply the methods described above in the sections on <Vertex Connection Reconstruction Method 1> and <Vertex Connection Reconstruction Method 2>. That is, the vertex connection reconstruction unit 322 reconstructs vertices and connections in the same way as the decoder (for example, the vertex connection reconstruction unit 416 of the decoding device 400 described later). The vertex connection reconstruction unit 322 may also reconstruct vertices and connections according to the vertex connection reconstruction parameters described in <Vertex Connection Reconstruction Parameters>. The vertex connection reconstruction unit 322 supplies vertex connection information and information about the reconstructed vertices and connections to the texture image generation unit 316.

[0170] With this configuration, the encoding device 300 can reduce the number of vertices indicated by the vertex connection information and suppress an increase in the amount of information in the vertex connection information. Therefore, the encoding device 300 can suppress a reduction in encoding efficiency caused by encoding the vertex connection information.

[0171] These processing units (mesh voxelization unit 311 to vertex connection reconstruction unit 322) can have any configuration. For example, each processing unit may be composed of logic circuits that implement the above-mentioned processing. Alternatively, each processing unit may have, for example, a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), etc., and implement the above-mentioned processing by executing a program using these. Of course, each processing unit may have both configurations, with some of the above-mentioned processing implemented by logic circuits and others by executing a program. The configurations of each processing unit may be independent of each other. For example, some processing units may implement some of the above-mentioned processing by logic circuits, some processing units may implement the above-mentioned processing by executing a program, and still other processing units may implement the above-mentioned processing by both logic circuits and program execution.

[0172] <Encoding process flow> An example of the coding process performed by this coding device 300 will be explained with reference to the flowchart in Figure 34.

[0173] When the encoding process is started, the mesh voxelization unit 311 converts the mesh into a voxel grid in step S301 by converting the coordinates of each vertex included in the vertex information 352 into a voxel grid.

[0174] In step S302, the patch generation unit 312 generates patches using the vertex information 352, etc., which was converted into a voxel grid in step S301, and projects the generated patches onto a projection plane to generate a patch image.

[0175] In step S303, the vertex connection update unit 313 performs a boundary vertex mode application determination process by applying the technology described in the section <Boundary Vertex Mode Application Determination>, etc. If it is determined that the boundary vertex mode should be applied, the process proceeds to step S304. If it is determined that the boundary vertex mode should not be applied, the processes in steps S304 and S305 are omitted, and the process proceeds to step S306.

[0176] In step S304, the vertex connection update unit 313 generates vertex connection information by applying the technology described in the section <2. Transmission of Vertex Connection Information> (including the sections <Vertex Connection Information> to <Boundary Vertex Mode Application Determination>), etc. That is, the vertex connection update unit 313 deletes at least some of the internal vertices of the mesh vertices included in the geometry image and generates vertex connection information for the remaining vertices (including at least boundary vertices). As a result, vertex connection information including the information described above in the section <Vertex Connection Information> is generated.

[0177] In step S305, the vertex connection update unit 313 applies the technology described in the section on <Vertex Connection Reconstruction Parameters>, etc., to generate vertex connection reconstruction parameters used when reconstructing vertices and the connections between them.

[0178] In step S306, the geometry image generation unit 314 arranges the patch images generated in step S302 onto a two-dimensional plane to generate a geometry image.

[0179] In step S307, the occupancy image generation unit 315 generates an occupancy image corresponding to the geometry image generated in step S306.

[0180] In step S308, the vertex connection reconstruction unit 322 applies the technology described above in the sections <Vertex Connection Reconstruction Method 1> and <Vertex Connection Reconstruction Method 2>, etc., to reconstruct vertices and connections that are not included in the vertex connection information generated in step S304. This process is the same as the process performed in the decoder, so the details of this process will be described later.

[0181] In step S309, the texture image generation unit 316 updates (corrects) the texture image so that the shape, size, arrangement, etc., of the patch image are the same as those of the geometry image.

[0182] In step S310, the metadata encoding unit 317 encodes metadata including vertex connection information generated in step S304 and vertex connection reconstruction parameters generated in step S305, and generates encoded metadata data.

[0183] In step S311, the 2D encoding unit 318 encodes the geometry image generated in step S306 and generates encoded data for the geometry image.

[0184] In step S312, the 2D encoding unit 319 encodes the occupancy image generated in step S307 and generates encoded data of the occupancy image.

[0185] In step S313, the 2D encoding unit 320 encodes the texture image corrected in step S309 and generates encoded data for the texture image.

[0186] In step S314, the multiplexing unit 321 multiplexes the encoded metadata data generated in step S310, the encoded geometry image data generated in step S311, the encoded occupancy image data generated in step S312, and the encoded texture image data generated in step S313 to generate a single bitstream. The multiplexing unit 321 outputs the generated bitstream to the outside of the encoding device 300.

[0187] The encoding process ends when the processing in step S314 is completed.

[0188] By performing the encoding process in this manner, the encoding device 300 can reduce the number of vertices indicated by the vertex connection information and suppress the increase in the amount of information in the vertex connection information. Therefore, the encoding device 300 can suppress the reduction in encoding efficiency caused by encoding the vertex connection information.

[0189] <4. Second Embodiment> <Decryption device> This technology can also be applied to a decoding device 400, for example, as shown in Figure 35. Figure 35 is a block diagram showing an example of the configuration of a decoding device, which is one embodiment of an image processing device to which this technology is applied. The decoding device 400 shown in Figure 35 is a device that extends VPCC to encode 3D data using a mesh as video frames using an encoding method for 2D images, decodes the encoded data using a decoding method for 2D images, and generates (reconstructs) 3D data using the mesh. In this case, the decoding device 400 decodes the encoded data and reconstructs the 3D data by applying one or a combination of the various methods of this technology described above.

[0190] Note that Figure 35 shows the main components such as the processing unit and data flow, and does not necessarily represent everything. In other words, the decoding device 400 may have processing units that are not shown as blocks in Figure 35, or processes and data flows that are not shown as arrows or other symbols in Figure 35.

[0191] As shown in Figure 35, the decoding device 400 includes a demultiplexing unit 411, a metadata decoding unit 412, a 2D decoding unit 413, a 2D decoding unit 414, a 2D decoding unit 415, a vertex connection reconstruction unit 416, a patch reconstruction unit 417, and a vertex information reconstruction unit 418. The metadata decoding unit 412, the 2D decoding unit 413, the 2D decoding unit 414, and the 2D decoding unit 415 may be considered as a decoding unit 431 in this disclosure.

[0192] The demultiplexing unit 411 acquires the bitstream input to the decoding device 400. This bitstream is, as described above in the first embodiment, a bitstream generated by, for example, the encoding device 300, and is a 3D data using a mesh that has been encoded by extending VPCC.

[0193] The demultiplexing unit 411 demultiplexes the bitstream and generates each encoded data contained in the bitstream. In other words, the demultiplexing unit 411 extracts each encoded data from the bitstream through demultiplexing. For example, the demultiplexing unit 411 extracts encoded metadata data from the bitstream. The demultiplexing unit 411 also extracts encoded geometry image data from the bitstream. Furthermore, the demultiplexing unit 411 extracts encoded occupancy image data from the bitstream. The demultiplexing unit 411 also extracts encoded texture image data from the bitstream.

[0194] The demultiplexing unit 411 supplies the extracted encoded data to the decoding unit 431. For example, the demultiplexing unit 411 supplies the extracted encoded metadata to the metadata decoding unit 412. The demultiplexing unit 411 also supplies the extracted encoded geometry image to the 2D decoding unit 413. Furthermore, the demultiplexing unit 411 supplies the extracted encoded occupancy image to the 2D decoding unit 414. Finally, the demultiplexing unit 411 supplies the extracted encoded texture image to the 2D decoding unit 415.

[0195] The decoding unit 431 performs decoding-related processing. The metadata decoding unit 412 acquires encoded metadata data supplied from the demultiplexing unit 411. The metadata decoding unit 412 decodes the acquired encoded metadata data and generates metadata. This metadata may include the vertex connection information and vertex connection reconstruction parameters mentioned above in the section <2. Transmission of Vertex Connection Information> (including the sections on <Vertex Connection Information> and <Parameters for Vertex Connection Reconstruction>). The metadata decoding unit 412 performs this decoding by applying a decoding method corresponding to the encoding method applied by the metadata encoding unit 317 (Figure 33) of the encoding device 300 when encoding the metadata. The metadata decoding unit 412 supplies the generated metadata to the vertex connection reconstruction unit 416.

[0196] The 2D decoding unit 413 acquires encoded data of the geometry image supplied from the demultiplexing unit 411. The 2D decoding unit 413 decodes the acquired encoded data of the geometry image using a decoding method for 2D images and generates a geometry image. This decoding method corresponds to the encoding method applied by the 2D encoding unit 318 (Figure 33) of the encoding device 300 when encoding the geometry image. The 2D decoding unit 413 supplies the generated geometry image to the vertex connection reconstruction unit 416 and the patch reconstruction unit 417.

[0197] The 2D decoding unit 414 acquires encoded data of the occupancy image supplied from the demultiplexing unit 411. The 2D decoding unit 414 decodes the acquired encoded data of the occupancy image using a decoding method for 2D images and generates an occupancy image. This decoding method corresponds to the encoding method applied by the 2D encoding unit 319 (Figure 33) of the encoding device 300 when encoding the occupancy image. The 2D decoding unit 414 supplies the generated occupancy image to the patch reconstruction unit 417.

[0198] The 2D decoding unit 415 acquires encoded texture image data supplied from the demultiplexing unit 411. The 2D decoding unit 415 decodes the acquired encoded texture image data using a decoding method for 2D images and generates a texture image (texture 454). This decoding method corresponds to the encoding method applied by the 2D encoding unit 320 (Figure 33) of the encoding device 300 when encoding the texture image. The 2D decoding unit 415 outputs the generated texture image (texture 454) as 3D data (or data constituting 3D data) using the restored mesh to the outside of the decoding device 400.

[0199] The vertex connection reconstruction unit 416 acquires metadata (including vertex connection information and parameters for vertex connection reconstruction) supplied from the metadata decoding unit 412. The vertex connection reconstruction unit 416 also acquires a geometry image supplied from the 2D decoding unit 413.

[0200] The vertex connection reconstruction unit 416 applies the technology described above in the sections <Vertex Connection Reconstruction Method 1> and <Vertex Connection Reconstruction Method 2>, and uses the geometry image to reconstruct vertices and connections not included in the vertex connection information into the geometry image. In other words, the vertex connection reconstruction unit 416 adds vertices and connections to the geometry image and sets the UV coordinates of each vertex. At that time, the vertex connection reconstruction unit 416 reconstructs vertices and connections according to the vertex connection reconstruction parameters transmitted from the encoder, as explained in <Parameters for Vertex Connection Reconstruction>.

[0201] The vertex connection reconstruction unit 416 supplies the vertex connection information provided by the encoder, as well as the information regarding the reconstructed vertices and connections, to the patch reconstruction unit 417 as information regarding vertices and connections. The vertex connection reconstruction unit 416 also generates connectivity 451 and UV map 452, which include the vertices and connections indicated in the vertex connection information provided by the encoder, as well as the reconstructed vertices and connections, and outputs these to the outside of the decoding device 400 as 3D data (or data that constitutes 3D data) using the restored mesh.

[0202] The patch reconstruction unit 417 acquires information about vertices and connections supplied from the vertex connection reconstruction unit 416. This information includes not only vertex connection information supplied from the encoder, but also information about vertices and connections reconstructed by the vertex connection reconstruction unit 416. The patch reconstruction unit 417 also acquires a geometry image supplied from the 2D decoding unit 413. Furthermore, the patch reconstruction unit 417 acquires an occupancy image supplied from the 2D decoding unit 414. Using the occupancy image and the information about vertices and connections supplied from the vertex connection reconstruction unit 416, the patch reconstruction unit 417 extracts patch images from the geometry image and reconstructs patches corresponding to the extracted patch images. The patch reconstruction unit 417 supplies the reconstructed patches and information about vertices and connections to the vertex information reconstruction unit 418.

[0203] The vertex information reconstruction unit 418 acquires information about patches, vertices, and connections supplied from the patch reconstruction unit 417. The vertex information reconstruction unit 418 reconstructs the vertices contained within the region of the acquired patch in three-dimensional space (determining the three-dimensional coordinates of each vertex) and generates vertex information 453. The vertex information reconstruction unit 418 outputs the generated vertex information 453 to the outside of the decoding device 400 as 3D data (or data that constitutes 3D data) using the reconstructed mesh.

[0204] With this configuration, the decoding device 400 can reconstruct vertices and connections in the decoded geometry image. In other words, the decoding device 400 can suppress the reduction in the number of vertices in the mesh due to encoding and decoding. Therefore, the decoding device 400 can suppress the reduction in mesh quality caused by the encoder reducing the number of vertices in the vertex connection information. Thus, a reduction in the number of vertices in the vertex connection information becomes practically possible. In other words, the decoding device 400 can suppress the reduction in encoding efficiency caused by encoding the vertex connection information.

[0205] These processing units (demultiplexing unit 411 to vertex information reconstruction unit 418) can have any configuration. For example, each processing unit may be composed of logic circuits that implement the above-mentioned processing. Alternatively, each processing unit may have, for example, a CPU, ROM, RAM, etc., and implement the above-mentioned processing by executing a program using these. Of course, each processing unit may have both configurations, with some of the above-mentioned processing implemented by logic circuits and others by executing a program. The configurations of each processing unit may be independent of each other. For example, some processing units may implement some of the above-mentioned processing by logic circuits, some other processing units may implement the above-mentioned processing by executing a program, and yet other processing units may implement the above-mentioned processing by both logic circuits and program execution.

[0206] <Decryption process flow> An example of the decoding process performed by this decoding device 400 will be explained with reference to the flowchart in Figure 36.

[0207] When the decoding process begins, the demultiplexing unit 411 demultiplexes the bitstream input to the decoding device 400 in step S401. Through this demultiplexing, the demultiplexing unit 411 extracts encoded metadata from the bitstream. The demultiplexing unit 411 also extracts encoded geometry image data from the bitstream. Furthermore, the demultiplexing unit 411 extracts encoded occupancy image data from the bitstream. Finally, the demultiplexing unit 411 extracts encoded texture image data from the bitstream.

[0208] In step S402, the metadata decoding unit 412 decodes the encoded metadata data extracted from the bitstream in step S401 and generates (restores) metadata. This metadata may include the vertex connection information and vertex connection reconstruction parameters mentioned above in the section <2. Transmission of Vertex Connection Information> (including the section <Vertex Connection Information> and the section <Parameters for Vertex Connection Reconstruction>).

[0209] In step S403, the 2D decoding unit 414 decodes the encoded data of the occupancy image extracted from the bitstream in step S401 and generates (restores) the occupancy image.

[0210] In step S404, the 2D decoding unit 413 decodes the encoded data of the geometry image extracted from the bitstream in step S401 and generates (restores) the geometry image.

[0211] In step S405, the 2D decoding unit 415 decodes the encoded data of the texture image extracted from the bitstream in step S401 and generates (restores) a texture image (texture 454).

[0212] In step S406, the vertex connection reconstruction unit 416 reconstructs (adds) vertices and connections that are not included in the vertex connection information generated (restored) in step S402 to the geometry image generated (restored) in step S404, and sets the UV coordinates of each vertex. At that time, the vertex connection reconstruction unit 416 reconstructs vertices and connections according to the vertex connection reconstruction parameters, as described in <Vertex Connection Reconstruction Parameters>.

[0213] In step S407, the patch reconstruction unit 417 uses the vertex connection information generated (restored) in step S402, the occupancy image generated (restored) in step S403, and the vertex and connection information reconstructed in step S406 to extract a patch image from the geometry image generated (restored) in step S404, and reconstructs the patch corresponding to that patch image in three-dimensional space.

[0214] In step S408, the vertex information reconstruction unit 418 uses the patch reconstructed in step S407, the vertex connection information generated (restored) in step S402, and the vertex and connection information reconstructed in step S406 to reconstruct the vertices included in the patch in three-dimensional space and generate vertex information 453.

[0215] Once step S408 is completed, the decryption process is finished.

[0216] <Reconstruction Process Flow 1> Next, an example of the vertex connection reconstruction process performed in step S406 of Figure 36 will be explained with reference to the flowchart in Figure 37. This flowchart shows an example of the process flow when the method described above is applied in <Vertex Connection Reconstruction Method 1>.

[0217] When the vertex connection reconstruction process is started, in step S421, the vertex connection reconstruction unit 416 linearly interpolates boundary vertices to the vertices indicated in the vertex connection information in the geometry image, as explained with reference to, for example, Figures 11 and 12, and determines the area within the patch.

[0218] In step S422, the vertex connection reconstruction unit 416 generates polygons in the geometry image, as described with reference to, for example, Figure 13.

[0219] In step S423, the vertex connection reconstruction unit 416 deletes unnecessary polygons outside the patch region, as explained with reference to, for example, Figure 14.

[0220] In step S424, the vertex connection reconstruction unit 416 performs mesh reduction and merges polygons within the patch region, as explained with reference to Figure 15, for example. At that time, the vertex connection reconstruction unit 416 merges the polygons without deleting the internal vertices indicated in the vertex connection information, as explained with reference to Figure 16, for example.

[0221] In this way, the connections between vertices (boundary vertices and internal vertices) and each vertex are reconstructed (added). When step S424 is completed, the process returns to Figure 36.

[0222] <Reconstruction Process Flow 2> Next, another example of the vertex connection reconstruction process flow performed in step S406 of Figure 36 will be described with reference to the flowchart in Figure 38. This flowchart shows an example of the process flow when the method described above is applied in <Vertex Connection Reconstruction Method 2>.

[0223] When the vertex connection reconstruction process is started, in step S441, the vertex connection reconstruction unit 416 corrects the patch boundaries in the geometry image, as described with reference to, for example, Figures 20 to 23.

[0224] In step S442, the vertex connection reconstruction unit 416 divides the patch region into a triangular region and a rectangular region, as described with reference to, for example, Figures 17 to 19.

[0225] In step S443, the vertex connection reconstruction unit 416 arranges the internal vertices in a rectangular region, as described with reference to, for example, Figures 24 and 25.

[0226] In step S444, the vertex connection reconstruction unit 416 determines the connectivity of the rectangular region, as described with reference to, for example, Figures 26 and 27.

[0227] In step S445, the vertex connection reconstruction unit 416 determines the connectivity of the triangular region, as described with reference to, for example, Figures 28 to 32. At this time, the vertex connection reconstruction unit 416 may add vertices as needed.

[0228] In this way, the connections between vertices (boundary vertices and internal vertices) and each vertex are reconstructed (added). When step S445 is completed, the process returns to Figure 36.

[0229] By performing each process as described above, the decoding device 400 can reconstruct vertices and connections in the decoded geometry image. In other words, the decoding device 400 can suppress the reduction in the number of vertices in the mesh due to encoding and decoding. Therefore, the decoding device 400 can suppress the reduction in mesh quality caused by the encoder reducing the number of vertices in the vertex connection information. Thus, a reduction in the number of vertices in the vertex connection information becomes practically possible. In other words, the decoding device 400 can suppress the reduction in encoding efficiency caused by encoding the vertex connection information.

[0230] <5. Application Examples> <Adding internal vertices> The method for adding vertices and connections not included in the vertex connection information in the decoding device (reconstruction process) is arbitrary and may be other than the methods described in <Vertex Connection Reconstruction Method 1> and <Vertex Connection Reconstruction Method 2>. For example, in the decoding device 400 of Figure 35, the vertex connection reconstruction unit 416 may subdivide the polygon (triangle) indicated by the vertex connection information. For example, the vertex connection reconstruction unit 416 may add vertices and connections by subdividing a polygon, which is shown by a dotted line on the left side of Figure 39, with vertices and connections included in the vertex connection information, so that it is shown by a dotted line on the right side of Figure 39.

[0231] The method of subdivision is arbitrary. For example, existing mesh subdivision methods, such as the one described in Hartmut Prautzsch, Qi Chen, "Analyzing Midpoint Subdivision", arXiv:0911.5157v3 [cs.GR] 27 Apr 2011, may be applied. Alternatively, as mentioned above in <Parameters for Vertex Connection Reconstruction>, a method may be applied in which vertices are added to the extreme values ​​(maximum or minimum values) of the depth value in the peripheral region of the geometry image, and subdivision is performed using those vertices.

[0232] <Vertex connection information> Furthermore, the vertex connection information may include a boundary vertex list, which is a list of patch vertices (also called boundary vertices) that lie on the boundary of the patch. The boundary vertex list includes identification information for the boundary vertices and positional information for the boundary vertices in the geometry image, the positional information including the difference in coordinates with other adjacent boundary vertices.

[0233] For example, as shown in Figure 40, the vertex connection information shows the position information of boundary vertices 521-1 to 521-8, and boundary vertices 521-1, 521-2, 521-3, 521-4, and 521-5 are connected in this order to form a loop, and boundary vertices 521-6, 521-5, 521-7, and 521-8 are connected in this order to form a loop. Boundary vertices 521-1 and 521-6 are considered the starting points of their respective loops. When there is no need to distinguish between boundary vertices 521-1 to 521-8, they are referred to as boundary vertex 521. The vertex connection information shows the position information (coordinates (U, V) on the geometry image) of each boundary vertex 521. However, for boundary vertices 521-1 and 521-6, their respective coordinates (U, V) are shown, whereas for the other boundary vertices 521, the difference in coordinates (deltaU, deltaV) with the previous boundary vertex 521 in the loop is shown. For example, the position information of boundary vertex 521-2 is shown as the difference between the coordinates of boundary vertex 521-2 and boundary vertex 521-1. Similarly, the position information of boundary vertex 521-3 is shown as the difference between the coordinates of boundary vertex 521-3 and boundary vertex 521-2. The same applies to the other boundary vertices.

[0234] The bit length of the coordinates of the first vertex in the loop is set according to the patch width (PatchWidth) and patch height (PatchHeight). The bit length of the coordinates of the second and subsequent vertices in the loop is set according to the maximum difference (maxDeltaU, maxDeltaV). By doing this, the increase in the amount of data for vertex connection information can be suppressed compared to when the position information of each boundary vertex is indicated by its coordinates (U, V).

[0235] Furthermore, in the decoding device 400 shown in Figure 35, the vertex connection reconstruction unit 416 may derive the coordinates of the boundary vertices by adding the difference between the coordinates of this vertex connection information and the coordinates of other adjacent boundary vertices. For example, the vertex connection reconstruction unit 416 can derive the coordinates (U, V) of boundary vertex 521-2 by adding the position information (deltaU, deltaV) of boundary vertex 521-2 to the position information (U, V) of boundary vertex 521-1. Similarly, the vertex connection reconstruction unit 416 can derive the coordinates (U, V) of boundary vertex 521-3 by adding the position information (deltaU, deltaV) of boundary vertex 521-3 to the position information (U, V) of boundary vertex 521-2. The same applies to other points. In this way, the vertex connection reconstruction unit 416 can obtain the position information (U, V) of each boundary vertex based on the vertex connection information described above.

[0236] <Generating occupancy images based on vertex connection information> Instead of transmitting the occupancy image from the encoding device to the decoding device, the decoding device may generate the occupancy image based on vertex connection information. For example, suppose the vertex connection information indicates a patch boundary as shown by the solid line 531 in Figure 41, based on boundary vertices and connections. In that case, an occupancy image with all pixel values ​​of "0" may be generated, and in that occupancy image, for example, pixel 532 that overlaps with the solid line 531 may be designated as the pixel corresponding to the patch boundary, and its pixel value may be set to "1". Then, based on the pixel corresponding to the patch boundary set in this way, the pixel corresponding to the inside of the patch may be determined, and its pixel value may be set to "1".

[0237] Furthermore, the decoder may generate an occupancy image based on vertex connection information as well as information indicating whether the left or right side of the connection between boundary vertices shown in the vertex connection information is the inner region of the patch. For example, when facing the direction of loop progression, flag information indicating whether the inside of the patch is to the right or left of the patch boundary may be transmitted from the encoder to the decoder, and the decoder may generate an occupancy image based on that flag information.

[0238] For example, suppose the flag information indicates that a polygon 533 is formed on the upper side of the patch boundary (solid line 531) in the figure, as shown in Figure 42 (i.e., the upper side of the figure is inside the patch). In this case, the decoder may set the pixel 532, shown by the diagonal line pattern in Figure 42, to correspond to this patch boundary (solid line 531) in the occupancy image, and set its pixel value to "1".

[0239] Furthermore, for example, suppose the flag information indicates that a polygon 533 is formed on the lower side of the patch boundary (solid line 531) in the figure, as shown in Figure 43 (i.e., the lower side of the figure is inside the patch). In this case, the decoder may set the pixel 532, shown by the diagonal line pattern in Figure 43, to be the pixel corresponding to this patch boundary (solid line 531) in the occupancy image, and set its pixel value to "1".

[0240] As shown in the examples in Figure 42 and Figure 43, the positions of pixels corresponding to the same patch boundary may differ depending on the flag information (information indicating whether the left or right side of the connection between boundary vertices shown in the vertex connection information is the inner region of the patch). By doing so, the decoder can generate an occupancy image that more accurately corresponds to the shape of the patch boundary.

[0241] In addition to the vertex connection information, the decoding device may generate an occupancy image based on information indicating the front and back of the polygon indicated by the vertex connection information. There are two possible directions of progression (rotation direction) for the loop of boundary vertices: clockwise and counterclockwise. For example, in the polygon with vertices A, B, and C as shown in FIG. 44, the cross product vector of vector AB and vector BC (the vector orthogonal to side AB and side BC) is vector 551 as shown on the left side of FIG. 44 if it is counterclockwise, and vector 552 as shown on the right side of FIG. 44 if it is clockwise. That is, the direction of the cross product vector is reversed depending on the rotation direction. In other words, the front and back of the polygon are reversed when the rotation direction is reversed. Put another way, the rotation direction depends on whether the polygon is the front or the back. Also, the rotation direction depends on the graphic API (Application Program Interface) used. For example, in the case of OpenGL (registered trademark) in the right-handed coordinate system, the counterclockwise direction as viewed from the viewpoint is the front. In the case of DirectX (registered trademark) in the left-handed coordinate system, the clockwise direction as viewed from the viewpoint is the front. Therefore, flag information indicating the front and back of the polygon indicated by the vertex connection information may be transmitted from the encoding device to the decoding device, and when generating the occupancy image, the decoding device may identify the direction of progression (rotation direction) of the loop based on the flag information.

[0242] In addition, the occupancy image generated based on the vertex connection information does not always match the occupancy image corresponding to the geometry image. That is, the patch shape estimated based on the vertex connection information may have an error with respect to the patch shape in the geometry image. Therefore, Omap correction information for correcting the occupancy image generated based on the vertex connection information may be transmitted from the encoding device to the decoding device. Then, the decoding device may use the Omap correction information to correct the occupancy image generated based on the vertex connection information. By doing so, the error with respect to the occupancy image corresponding to the geometry image can be reduced.

[0243] The content of this Omap correction information is arbitrary. For example, the Omap correction information may include the difference between the occupancy image generated based on the vertex connection information and the occupancy image corresponding to the geometry image. For example, the Omap correction information may include, as the difference, a list of pixels with different pixel values.

[0244] For example, like the Omap correction information 561 shown in FIG. 45, pixels (also referred to as corrected pixels) with different pixel values between the occupancy image generated based on the vertex connection information and the occupancy image corresponding to the geometry image may be indicated by their position information (UV coordinates). In this Omap correction information 561, U[K] indicates the U coordinate of the K-th corrected pixel, and V[K] indicates the V coordinate of that corrected pixel.

[0245] Also, like the Omap correction information 562 shown in FIG. 45, the corrected pixels may be indicated by a predetermined index. In the case of Omap correction information 562, Index[K] indicates the index value of the K-th corrected pixel. This Index[K] is calculated as follows using the coordinates (U[K] and V[K]). Of course, the index value may be any value as long as it can identify each pixel, and is not limited to this example.

[0246] Index[K] = U[K] + V[K] * width

[0247] <Encoding device> A main configuration example of the encoding device when transmitting such Omap correction information is shown in FIG. 46. The encoding device 600 shown in FIG. 46 is the same device as the encoding device 300 in FIG. 33. However, the encoding device 600 transmits Omap correction information instead of transmitting the occupancy image corresponding to the geometry image.

[0248] As shown in Figure 46, the encoding device 600 has the same configuration as the encoding device 300 in Figure 33, except for the occupancy image generation unit 315 and the 2D encoding unit 319. The encoding device 600 also has an Omap correction information generation unit 611 and an Omap correction information encoding unit 612. In this encoding device 600, the image generation unit 331 includes a geometry image generation unit 314 and a texture image generation unit 316. The encoding unit 332 includes a metadata encoding unit 317, a 2D encoding unit 318, a 2D encoding unit 320, and an Omap correction information encoding unit 612.

[0249] The patch generation unit 312 supplies the generated patch image to the Omap correction information generation unit 611. The Omap correction information generation unit 611 acquires the patch image and uses it to generate an occupancy image (also referred to as a second occupancy image) corresponding to the geometry image. The vertex connection update unit 313 also supplies the generated vertex connection information to the Omap correction information generation unit 611. The Omap correction information generation unit 611 acquires the vertex connection information and generates an occupancy image (also referred to as a first occupancy image) based on that vertex connection information.

[0250] The Omap correction information generation unit 611 generates Omap correction information for correcting the first occupancy image using the generated first occupancy image and the second occupancy image. For example, the Omap correction information generation unit 611 may derive the difference between the generated first occupancy image and the second occupancy image and generate Omap correction information including that difference. For example, the Omap correction information generation unit 611 may generate Omap correction information including a list of position information of the correction pixels as that difference. The position information may be UV coordinates or index values.

[0251] The Omap correction information generation unit 611 supplies the generated Omap correction information to the Omap correction information encoding unit 612. The Omap correction information encoding unit 612 receives the Omap correction information and encodes it. The encoding method is arbitrary. The Omap correction information encoding unit 612 supplies the encoded data of the generated Omap correction information to the multiplexing unit 321. The multiplexing unit 321 multiplexes the encoded data of the Omap correction information with other encoded data to generate a single bitstream. The multiplexing unit 321 outputs the generated bitstream to the outside of the encoding device 600. This bitstream is transmitted to, for example, a decoding device. In other words, the Omap correction information is transmitted to the decoding device. The method of transmitting this bitstream (Omap correction information) is arbitrary. For example, this bitstream may be transmitted to the decoding device via any transmission path. Alternatively, this bitstream may be recorded on any recording medium and transmitted to the decoding device via that recording medium.

[0252] Furthermore, the metadata encoded by the metadata encoding unit 317 may include information indicating whether the left or right side of the connection between boundary vertices shown in the vertex connection information is the inner region of the patch, or information indicating the front or back of the polygon shown in the vertex connection information.

[0253] <Encoding process flow> An example of the encoding process flow performed by this encoding device 600 will be explained with reference to the flowchart in Figure 47. When the encoding process starts, each of the processes in steps S601 to S606 is performed in the same way as the processes in steps S301 to S306 in Figure 34.

[0254] In step S607, the Omap correction information generation unit 611 executes the Omap correction information generation process and generates Omap correction information. When the process in step S607 is completed, the process proceeds to step S608. Each of the processes in steps S608 to S611 is executed in the same manner as the processes in steps S308 to S311 in Figure 34.

[0255] In step S612, the Omap correction information encoding unit 612 encodes the Omap correction information generated in step S607. When the processing of step S612 ends, the process proceeds to step S613. The processing of step S613 is executed in the same manner as the processing of step S313 in FIG. 34.

[0256] In step S614, the multiplexing unit 321 multiplexes the encoded data of the meta information generated in step S610, the encoded data of the geometry image generated in step S611, the encoded data of the Omap correction information generated in step S612, and the encoded data of the texture image generated in step S613 to generate a single bit stream. The multiplexing unit 321 outputs the generated bit stream to the outside of the encoding device 600. When the processing of step S614 ends, the encoding process ends.

[0257] <Flow of Omap correction information generation process> An example of the flow of the Omap correction information generation process executed in step S607 of this encoding process will be described with reference to the flowchart of FIG. 48. When the Omap correction information generation process is started, the Omap correction information generation unit 611 generates an occupancy image (second occupancy image) corresponding to the geometry image based on the patch image in step S631.

[0258] In step S632, the Omap correction information generation unit 611 generates an occupancy image (first occupancy image) based on the vertex connection information.

[0259] In step S633, the Omap correction information generation unit 611 compares the first occupancy image and the second occupancy image to generate Omap correction information. When the Omap correction information is generated, the Omap correction information generation process ends, and the process returns to the encoding process of FIG. 47.

[0260] <Decoder> Furthermore, Figure 49 shows a typical configuration of a decoding device when transmitting Omap correction information in this manner. The decoding device 700 shown in Figure 49 is similar to the decoding device 400 in Figure 35. However, instead of acquiring an occupancy image corresponding to a geometry image, the decoding device 700 generates an occupancy image based on vertex connection information. The decoding device 700 also acquires the transmitted Omap correction information and uses that Omap correction information to correct the generated occupancy image. The decoding device 700 corresponds to the encoding device 600 (Figure 46) and can decode the bitstream generated by the encoding device 600 and reconstruct the 3D data.

[0261] As shown in Figure 49, the decoding device 700 has the same configuration as the decoding device 400 in Figure 35, except for the 2D decoding unit 414. The decoding device 700 also has an Omap correction information decoding unit 711. In the case of this encoding device 600, the decoding unit 431 includes a metadata decoding unit 412, a 2D decoding unit 413, a 2D decoding unit 415, and an Omap correction information decoding unit 711.

[0262] The demultiplexing unit 411 extracts encoded data of Omap correction information from the bitstream and supplies it to the Omap correction information decoding unit 711. The Omap correction information decoding unit 711 acquires the encoded data of Omap correction information. The Omap correction information decoding unit 711 decodes the encoded data and generates Omap correction information. This decoding method corresponds to the encoding method applied by the Omap correction information encoding unit 612 (Figure 46) of the encoding device 600 when encoding the Omap correction information. The Omap correction information decoding unit 711 supplies the generated Omap correction information to the patch reconstruction unit 417.

[0263] The vertex connection reconstruction unit 416 supplies vertex connection information to the patch reconstruction unit 417. The metadata may include information indicating whether the left or right side of the connection between boundary vertices shown in the vertex connection information is the inner region of the patch, or information indicating the front and back sides of the polygon shown in the vertex connection information. In that case, the vertex connection reconstruction unit 416 may supply this information to the patch reconstruction unit 417.

[0264] The patch reconstruction unit 417 acquires its vertex connection information and generates an occupancy image based on the acquired vertex connection information. Note that the patch reconstruction unit 417 may acquire information indicating which side (left or right) of the connection between the boundary vertices shown in the vertex connection information supplied from the vertex connection reconstruction unit 416 is the inner area of the patch, and use this information when generating the occupancy image. Also, the patch reconstruction unit 417 may acquire information indicating the front and back of the polygon shown in the vertex connection information supplied from the vertex connection reconstruction unit 416, and use this information when generating the occupancy image.

[0265] Also, the patch reconstruction unit 417 may acquire Omap correction information for correcting the occupancy image supplied from the Omap correction information decoder unit 711, and correct the occupancy image generated based on the vertex connection information based on the Omap correction information. For example, the patch reconstruction unit 417 may change the pixel value of the corrected pixel shown in the Omap correction information of the occupancy image generated based on the vertex connection information from "0" to "1", or from "1" to "0".

[0266] The patch reconstruction unit 417 extracts a patch image from the geometry image using the occupancy image generated as described above, and reconstructs the patch corresponding to the extracted patch image. The patch reconstruction unit 417 supplies the reconstructed patch and information regarding vertices and connections to the vertex information reconstruction unit 418.

[0267] <Flow of decoding process> An example of the decoding process performed by the decoding device 700 will be explained with reference to the flowchart in Figure 50. When the decoding process is started, the demultiplexing unit 411 demultiplexes the bitstream input to the decoding device 700 in step S701. Through this demultiplexing, the demultiplexing unit 411 extracts encoded metadata data from the bitstream. The demultiplexing unit 411 also extracts encoded geometry image data from the bitstream. Furthermore, the demultiplexing unit 411 extracts encoded Omap correction information data from the bitstream. The demultiplexing unit 411 also extracts encoded texture image data from the bitstream. In step S702, the metadata decoding unit 412 decodes the encoded metadata data extracted from the bitstream in step S701 and generates (restores) the metadata.

[0268] In step S703, the Omap correction information decoding unit 711 decodes the encoded data of the Omap correction information extracted from the bitstream in step S701 and generates (restores) the Omap correction information. Then, each process in steps S704 to S706 is executed in the same manner as the processes in steps S404 to S406 in Figure 36.

[0269] In step S707, the patch reconstruction unit 417 performs occupancy image generation processing and generates an occupancy image based on vertex connection information. Then, steps S708 and S709 are performed in the same manner as steps S407 and S408 in Figure 36. When step S709 is completed, the decoding process is completed.

[0270] <Occupancy Image Generation Process Flow> An example of the flow of the occupancy image generation process performed in step S707 of Figure 50 will be explained with reference to the flowchart in Figure 51. When the occupancy image generation process starts, the patch reconstruction unit 417 generates an occupancy image based on vertex connection information in step S731. In step S732, the patch reconstruction unit 417 corrects the generated occupancy image based on Omap correction information. When the process in step S732 is completed, the occupancy image generation process ends, and the process returns to the decoding process in Figure 50.

[0271] As described above, the decoding device generates an occupancy image based on vertex connection information, thereby eliminating the need to transmit the occupancy image. Consequently, the reduction in the encoding efficiency of 3D data can be suppressed.

[0272] <Patch generation> As described above, when generating an occupancy image based on vertex connection information, the method for generating patches (the portion corresponding to the patch image of the geometry image) in the occupancy image is arbitrary. For example, the patch reconstruction unit 417 may place an internal region indicating that it is inside the patch inside the boundary of the patch shown by the boundary vertices and connections of the vertex connection information, deform the internal region by moving each vertex of the internal region to a nearby boundary vertex, and further correct the internal region based on the vertex connection information.

[0273] For example, as shown in Figure 52, an internal region 802 is placed inside the patch boundary 801 indicated by the vertex connection information. This internal region 802 is composed of rectangular areas. Then, as shown by the dotted arrow 803, each vertex of the internal region 802 is moved to a nearby boundary vertex. This deforms the internal region 802. In the example shown in Figure 52, each vertex of the internal region 802 is moved to all boundary vertices, so the outline of the internal region 802 matches the patch boundary 801. By setting the pixel value of this internal region to "1", it is possible to form an area in the occupancy image that corresponds to the patch of geometry indicated by the vertex connection information (i.e., generate a patch).

[0274] However, this method does not always guarantee that the outline of the internal region 802 will always match the patch boundary 801. For example, in Figure 53, the vertices of the internal region 802 are not moved to boundary vertices 813 and 814. As a result, the outline of the internal region 802 becomes like the dotted lines 811 and 812, and boundary vertices 813 and 814 are not included in the patch. Examples of boundary vertices not being included in a patch include cases where a single boundary vertex is not included in the patch, as in Figure 53, and cases where multiple consecutive boundary vertices (e.g., two points) are not included in the patch.

[0275] Furthermore, in the case of Figure 54, the outline of the internal region 802 becomes as shown by dotted lines 821 and 822, and the boundary edge 823 is not included in the patch.

[0276] Thus, if a boundary vertex or boundary edge is not included in the patch (internal region 802), the patch reconstruction unit 417 may further correct the shape of the patch (internal region) based on vertex connection information. This correction method is arbitrary. For example, if a boundary vertex is not included in the patch, a polygon containing that boundary vertex may be added to the patch. For example, as shown in Figure 55, boundary vertices 831-1 and 831-2, and connections 832-1 to 832-3 are considered the boundary, and the lower region 833 (shown in gray) of the boundary in the figure is the internal region of the patch, and boundary vertex 831-3 is not included in that patch. In this case, if a single boundary vertex is not included in the patch, the patch reconstruction unit 417 may add a region 834 (polygon) enclosed by the connections connecting boundary vertices 831-1 to 831-3 to this patch. By doing so, the outline of the patch is corrected so that boundary vertex 831-3 is included. Therefore, an occupancy image corresponding to the vertex connection information is obtained.

[0277] For example, as shown in Figure 56, boundary vertices 841-1 and 841-4, and connections 842-1 to 842-3 are defined as a boundary, and the lower region 843 (shown in gray) of that boundary is the interior region of the patch, while boundary vertices 841-2 and 841-3 are not included in that patch. In cases where multiple consecutive boundary vertices are not included in a patch in this way, a polygon containing those boundary vertices may be added to the patch.

[0278] However, in this case, there are two possible ways to add the patch. For example, as shown on the left side of Figure 56, one possible case is to add the region 844-1 (polygon) enclosed by the connection between boundary vertices 841-1 to 841-3, and the region 844-2 (polygon) enclosed by the connection between boundary vertices 841-1, 841-3, and 841-4 to the patch. Alternatively, as shown on the right side of Figure 56, one possible case is to add the region 846-1 (polygon) enclosed by the connection between boundary vertices 841-2, 841-3, and 841-4, and the region 846-2 (polygon) enclosed by the connection between boundary vertices 841-1, 841-3, and 841-4 to the patch. In such cases, one of these cases will be selected and applied.

[0279] This selection method is arbitrary. For example, the selection may be made based on whether the midpoint between boundary vertex 841-1 included in the patch and a boundary vertex not included in the patch that is not connected to boundary vertex 841-1 is included in the corrected patch. For example, in the case shown on the left of Figure 56, the midpoint 845 between boundary vertices 841-1 and 841-3 is included in the corrected patch. In contrast, in the case shown on the right of Figure 56, the midpoint 847 between boundary vertices 841-1 and 841-2 is not included in the corrected patch. Either case may be selected depending on whether this midpoint is included in the patch indicated by the boundary vertices and connections included in the vertex connection information.

[0280] For example, in Figure 57, the area shown in gray is considered the interior area of ​​the patch, and the boundary edge (connection 851) is not included in that patch. In cases where the boundary edge is not included in the patch, a polygon containing that edge may be added to the patch. First, connections are found where one vertex is at each end of the boundary vertices of connection 851 (boundary vertices 852 and 853). Connections where one vertex is boundary vertex 852 are connections 854-1 to 854-3. Connections where one vertex is boundary vertex 853 are connections 855-1 to 855-3. Then, connections where the other vertex is the same are found among these connections. Connections 854-3 and 855-1 are connected to the same vertex. Therefore, the area 856 (polygon) enclosed by connections 851, 854-3, and 855-1 is added to the patch.

[0281] <6. Addendum> The above describes the encoding of 3D data using meshes by extending the VPCC standard. However, V3C (Visual Volumetric Video-based Coding) or MIV (metadata immersive video) can also be applied instead of VPCC. V3C and MIV are standards that use encoding techniques almost identical to VPCC, and can be extended to encode 3D data using meshes, just as with VPCC. Therefore, the above-described technique can also be applied when applying V3C or MIV to encode 3D data using meshes.

[0282] <3D data> The above describes the application of this technology to mesh encoding and decoding, but this technology is not limited to these examples and can be applied to the encoding and decoding of 3D data of any standard. In other words, as long as it does not contradict the technology described above, the encoding and decoding methods and other processing methods, as well as the specifications of various data such as 3D data and metadata, are arbitrary. Furthermore, some of the processing and specifications described above may be omitted as long as they do not contradict the technology.

[0283] <Computer> The series of processes described above can be executed by hardware or by software. When the series of processes are executed by software, the programs that make up that software are installed on a computer. Here, "computer" includes computers built into dedicated hardware, as well as general-purpose personal computers, for example, that can perform various functions by installing various programs.

[0284] Figure 58 is a block diagram showing an example of the hardware configuration of a computer that executes the series of processes described above by a program.

[0285] In the computer 900 shown in Figure 58, the CPU (Central Processing Unit) 901, ROM (Read Only Memory) 902, and RAM (Random Access Memory) 903 are interconnected via a bus 904.

[0286] An input / output interface 910 is also connected to the bus 904. The input / output interface 910 is connected to an input unit 911, an output unit 912, a storage unit 913, a communication unit 914, and a drive 915.

[0287] The input unit 911 consists of, for example, a keyboard, mouse, microphone, touch panel, and input terminals. The output unit 912 consists of, for example, a display, speaker, and output terminals. The storage unit 913 consists of, for example, a hard disk, RAM disk, and non-volatile memory. The communication unit 914 consists of, for example, a network interface. The drive 915 drives removable media 921 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0288] In a computer configured as described above, the CPU 901 loads, for example, a program stored in the memory unit 913 into the RAM 903 via the input / output interface 910 and the bus 904, and executes it, thereby performing the series of processes described above. The RAM 903 also stores data necessary for the CPU 901 to perform various processes as appropriate.

[0289] The program to be executed by the computer can be recorded and applied, for example, on removable media 921 such as a package medium. In this case, the program can be installed in the storage unit 913 via the input / output interface 910 by inserting the removable media 921 into the drive 915.

[0290] Furthermore, this program can also be provided via wired or wireless transmission media such as a local area network, the internet, or digital satellite broadcasting. In that case, the program can be received by the communication unit 914 and installed in the storage unit 913.

[0291] Additionally, this program can be pre-installed on ROM902 or memory unit913.

[0292] <Applicability of this technology> This technology can be applied to any configuration. For example, this technology can be applied to various electronic devices.

[0293] Furthermore, this technology can also be implemented as part of a device, such as a processor (e.g., a video processor) as a system LSI (Large Scale Integration), a module using multiple processors (e.g., a video module), a unit using multiple modules (e.g., a video unit), or a set with additional functions added to a unit (e.g., a video set).

[0294] Furthermore, this technology can also be applied to network systems composed of multiple devices. For example, this technology may be implemented as cloud computing, where multiple devices share and collaborate on processing via a network. For example, this technology may be implemented in a cloud service that provides image (video) related services to any terminal such as computers, AV (Audio Visual) equipment, portable information processing terminals, and IoT (Internet of Things) devices.

[0295] In this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all components are located in the same enclosure. Therefore, multiple devices housed in separate enclosures and connected via a network, and a single device containing multiple modules within a single enclosure, are both considered systems.

[0296] <Fields and applications where this technology can be applied> Systems, devices, and processing units incorporating this technology can be used in any field, such as transportation, healthcare, security, agriculture, livestock farming, mining, beauty, factories, home appliances, weather, and nature monitoring. Furthermore, their applications are entirely arbitrary.

[0297] <Other> In this specification, "flag" refers to information used to identify multiple states, and includes not only information used to identify two states, true (1) or false (0), but also information capable of identifying three or more states. Therefore, the values ​​that this "flag" can take are, for example, two values, 1 / 0, or three or more values. In other words, the number of bits that constitute this "flag" is arbitrary, and can be one bit or multiple bits. Furthermore, identification information (including flags) can be included not only in the form of the identification information itself in the bitstream, but also in the form of differential information of the identification information relative to a certain reference information in the bitstream. Therefore, in this specification, "flag" and "identification information" include not only the information itself, but also differential information relative to the reference information.

[0298] Furthermore, various types of information (metadata, etc.) related to encoded data (bitstream) may be transmitted or recorded in any form as long as they are associated with the encoded data. Here, the term "associate" means, for example, making it possible to use (link) one piece of data when processing the other. In other words, associated data may be combined into a single piece of data, or they may be individual pieces of data. For example, information associated with encoded data (image) may be transmitted on a different transmission path than the encoded data (image). Also, for example, information associated with encoded data (image) may be recorded on a different recording medium (or a different recording area on the same recording medium) than the encoded data (image). Note that this "association" does not have to be with the entire data, but only with a part of the data. For example, an image and the information corresponding to that image may be associated with each other in any unit, such as multiple frames, one frame, or a part within a frame.

[0299] In this specification, terms such as "combine," "multiplex," "add," "integrate," "include," "store," "insert," "insert," and "place" mean combining multiple things into one, such as combining encoded data and metadata into a single data, and represent one method of "associating" as described above.

[0300] Furthermore, the embodiments of this technology are not limited to those described above, and various modifications are possible without departing from the spirit of this technology.

[0301] For example, the configuration described as a single device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, the configurations described above as multiple devices (or processing units) may be combined and configured as a single device (or processing unit). Furthermore, it is also possible to add configurations other than those described above to the configuration of each device (or each processing unit). In addition, if the overall system configuration and operation are substantially the same, a part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0302] Furthermore, for example, the program described above may be executed on any device. In that case, the device should have the necessary functions (such as functional blocks) and be able to obtain the necessary information.

[0303] Furthermore, for example, each step of a flowchart may be executed by one device, or it may be divided among multiple devices. Additionally, if a single step includes multiple processes, these processes may be executed by one device, or they may be divided among multiple devices. In other words, multiple processes included in a single step can be executed as multiple steps. Conversely, processes described as multiple steps can be combined and executed as a single step.

[0304] Furthermore, for example, a program executed by a computer may be structured so that the steps of the program are executed chronologically in the order described herein, or they may be executed in parallel or individually at necessary times, such as when a call is made. In other words, the steps may be executed in an order different from the order described above, as long as no inconsistencies arise. Moreover, the steps of this program may be executed in parallel with the processing of other programs, or in combination with the processing of other programs.

[0305] Furthermore, for example, multiple technologies relating to this technology can be implemented independently, as long as they do not create a contradiction. Of course, any multiple technologies can also be implemented in combination. For example, some or all of the technologies described in one embodiment can be implemented in combination with some or all of the technologies described in another embodiment. Also, some or all of the above-mentioned technologies can be implemented in combination with other technologies not mentioned above.

[0306] Furthermore, this technology can also be configured as follows. (1) A vertex connection information generation unit that deletes at least some of the internal vertices, which are vertices of a mesh representing a three-dimensional object, that are located outside the boundaries of the geometric patch, and generates vertex connection information that indicates the vertices of the mesh and the connections between those vertices, An encoding unit that encodes the vertex connection information and An information processing device equipped with the following features. (2) The vertex connection information includes a boundary vertex list, which is a list of boundary vertices that are the vertices located at the boundary of the patch. (1) The information processing device described above. (3) The boundary vertex list includes an inclusive list in which the region enclosed by the boundary vertices and connections between the boundary vertices that make up the list is included in the patch. (2) The information processing device described in (2). (4) The boundary vertex list further includes an exclusive list in which the region enclosed by the boundary vertices and the connections between the boundary vertices that make up the list is not included in the patch. (3) The information processing device described above. (5) The boundary vertex list includes identification information of the boundary vertices and positional information of the boundary vertices in a geometry image which is a two-dimensional plane on which the patches are placed. (2) The information processing device described in any of (4) to (4). (6) The boundary vertex list indicates the connections between the boundary vertices in the order in which the boundary vertices are arranged. (2) The information processing device described in any of (5). (7) The vertex connection information further includes an internal vertex list which is a list of internal vertices. (2) The information processing device described in any of (6) above. (8) The internal vertex list includes identification information of the internal vertex and positional information of the internal vertex in a geometry image which is a two-dimensional plane on which the patch is placed. (7) The information processing device described above. (9) The vertex connection information further includes pair information indicating the correspondence between the boundary vertices between the patches. (2) The information processing device described in any of (8). (10) The vertex connection information generation unit further generates parameters used when reconstructing the vertices and the connections. (1) The information processing device described in any of (9) above. (11) The vertex connection information generation unit generates a flag as a parameter indicating whether at least a portion of the internal vertices of the mesh have been deleted. (10) The information processing device described above. (12) The vertex connection information generation unit generates information regarding the density of the vertices during reconstruction as the parameters. The information processing device described in (10) or (11). (13) The vertex connection information generation unit generates a flag information as a parameter indicating whether to correct the position of the vertex to be reconstructed. (10) to (12) the information processing device described in any of these. (14) The vertex connection information generation unit generates, as parameters, information regarding the geometric features that are the target of correction for the vertices to be reconstructed. (10) to (13) the information processing device described in any of these three terms. (15) The vertex connection information generation unit generates, as parameters, information regarding the search range of the target for correction of the vertices to be reconstructed. (10) to (14) the information processing device described in any of these. (16) The vertex connection information generation unit generates information regarding the corrected minimum interval of the vertices to be reconstructed as parameters. (10) to (15) the information processing device described in any of these. (17) The vertex connection information generation unit generates information regarding the method of connecting the vertices to be reconstructed as parameters. (10) to (16) the information processing device described in any of these. (18) The vertex connection information generation unit generates, as parameters, information regarding the method of merging the polygon composed of the vertices to be reconstructed. (10) to (17) the information processing device described in any of these. (19) The vertex connection information generation unit determines whether to delete the internal vertices of the mesh, and if it determines to delete them, it deletes at least a portion of the internal vertices of the mesh and generates the vertex connection information. (1) to (18) the information processing device described in any of these. (20) The vertex connection information generation unit determines whether to delete the internal vertices of the mesh based on the result of comparing the number of internal vertices with the number of boundary vertices which are vertices located at the boundary of the patch. (19) The information processing device described above. (21) The vertex connection information generation unit determines whether to delete the internal vertices of the mesh based on the magnitude of the distortion caused by deleting at least a portion of the internal vertices of the mesh and reconstructing the internal vertices. (19) The information processing device described above. (22) The vertex connection information includes a boundary vertex list which is a list of boundary vertices which are vertices located at the boundary of the patch, The boundary vertex list includes identification information of the boundary vertices and positional information of the boundary vertices in a geometry image which is a two-dimensional plane on which the patches are placed. The position information includes the difference in coordinates with other adjacent boundary vertices. (1) The information processing device described in any of (21). (23) The unit further comprises an Omap correction information generation unit that generates Omap correction information for correcting the occupancy image generated based on the vertex connection information, The encoding unit further encodes the Omap correction information. (1) The information processing device described in any of (22). (24) Delete at least some of the internal vertices, which are vertices of a mesh representing a three-dimensional object, that are located outside the boundaries of the geometric patch, and generate vertex connection information that shows the vertices of the mesh and the connections between the vertices, Encode the aforementioned vertex connection information. Information processing methods.

[0307] (31) A decoding unit that decodes encoded data of vertex connection information indicating connections between boundary vertices, which are vertices of a mesh representing a three-dimensional structure object located at least at the boundary of a geometric patch, and the connections between said boundary vertices. A vertex connection reconstruction unit reconstructs the vertices located in the patch and the connections between the vertices using the vertex connection information obtained by decoding the encoded data by the decoding unit and a geometry image in which the patch is arranged in a two-dimensional plane. An information processing device equipped with the following features. (32) The vertex connection reconstruction unit reconstructs the vertices and connections located in the patch by linearly interpolating the boundary vertices, determining the patch region which is the area included in the patch, generating polygons, merging unnecessary polygons outside the patch region, and merging the polygons in the patch region based on the geometry. (31) The information processing device described above. (33) The vertex connection reconstruction unit further deletes the linearly interpolated boundary vertices. (32) The information processing device described above. (34) The vertex connection reconstruction unit reconstructs the vertices and connections located in the patch by dividing the patch into a triangular region and a rectangular region, placing the vertices in the rectangular region, determining the connections between the vertices in the rectangular region, and determining the connections between the vertices in the triangular region. (31) The information processing device described above. (35) The vertex connection reconstruction unit generates connectivity and UV maps of the mesh, including the reconstructed vertices and connections. (31) to (34) the information processing device described in any of these. (36) Further comprising a patch reconstruction unit for reconstructing the patch from the geometric image. (31) The information processing device described above. (37) The system further comprises a vertex information reconstruction unit that reconstructs the vertex information of the mesh based on the patch reconstructed by the patch reconstruction unit. (36) The information processing device described above. (38) The vertex connection reconstruction unit subdivides the polygon indicated by the vertex connection information. (31) The information processing device described above. (39) The vertex connection information includes a boundary vertex list which is a list of boundary vertices which are vertices located at the boundary of the patch, The boundary vertex list includes identification information for the boundary vertices and positional information for the boundary vertices in the geometry image. The position information includes the difference in coordinates with other adjacent boundary vertices. The vertex connection reconstruction unit derives the coordinates of the boundary vertex by adding the difference with the coordinates of other adjacent boundary vertices. (31) The information processing device described above. (40) The patch reconstruction unit further comprises generating an occupancy image based on the vertex connection information and reconstructing the patch from the geometry image using the occupancy image. (31) The information processing device described above. (41) The patch reconstruction unit further generates the occupancy image based on information indicating which side of the connection between the boundary vertices shown in the vertex connection information is the inner region of the patch. (40) The information processing device described above. (42) The patch reconstruction unit further generates the occupancy image based on information indicating the front and back sides of the polygons shown in the vertex connection information. (40) The information processing device described above. (43) The decoding unit further decodes encoded data of Omap correction information for correcting the occupancy image, The patch reconstruction unit modifies the occupancy image based on the Omap correction information, The Omap correction information indicates the difference between the first occupancy image generated based on the vertex connection information and the second occupancy image corresponding to the geometry image. (40) The information processing device described above. (44) The patch reconstruction unit places an internal region inside the boundary of the patch, which is indicated by the boundary vertices and connections between the boundary vertices in the vertex connection information, and deforms the internal region by moving each vertex of the internal region to a nearby boundary vertex, and corrects the internal region based on the vertex connection information. (40) The information processing device described above. (45) Decode the encoded data of vertex connection information indicating the connections between boundary vertices, which are vertices of a mesh representing a three-dimensional structure object located at least at the boundary of a patch of geometry, and the connections between said boundary vertices. Using the vertex connection information obtained by decoding the encoded data and the geometry image in which the patch is arranged in a two-dimensional plane, the vertices located in the patch and the connections between the vertices are reconstructed. Information processing methods. [Explanation of Symbols]

[0308] 300 Encoding unit, 311 Mesh voxelization unit, 312 Patch generation unit, 313 Vertex connection update unit, 314 Geometry image generation unit, 315 Occupancy image generation unit, 316 Texture image generation unit, 317 Metadata encoding unit, 318 to 320 2D encoding unit, 321 Multiplexing unit, 322 Vertex connection reconstruction unit, 331 Image generation unit, 332 Encoding unit, 400 Decoding unit, 411 Demultiplexing unit, 412 Metadata decoding unit, 413 to 415 2D decoding unit, 416 Vertex connection reconstruction unit, 417 Patch reconstruction unit, 418 Vertex information reconstruction unit, 600 Encoding unit, 611 Omap correction information generation unit, 612 Omap correction information encoding unit, 700 Decoding unit, 711 Omap Correction Information Decoding Unit, 900 Computers< / mesh> < / vpcc>

Claims

1. A vertex connection information generation unit deletes at least some of the internal vertices, which are vertices of a mesh representing a three-dimensional object, that are located outside the boundaries of a geometric patch, and generates vertex connection information indicating the vertices of the mesh and the connections between those vertices. An encoding unit that encodes the vertex connection information and An information processing device equipped with the following features.

2. The vertex connection information includes a boundary vertex list, which is a list of boundary vertices, which are the vertices located at the boundary of the patch. The information processing apparatus according to claim 1.

3. The boundary vertex list includes an inclusive list in which the region enclosed by the boundary vertices and connections between the boundary vertices that make up the list is included in the patch. The information processing apparatus according to claim 2.

4. The boundary vertex list further includes an exclusive list in which the region enclosed by the boundary vertices and the connections between the boundary vertices that make up the list is not included in the patch. The information processing apparatus according to claim 3.

5. The boundary vertex list includes identification information for the boundary vertices and positional information for the boundary vertices in a geometry image which is a two-dimensional plane on which the patches are placed. The information processing apparatus according to claim 2.

6. The boundary vertex list indicates the connections between the boundary vertices in the order in which they are arranged. The information processing apparatus according to claim 2.

7. The vertex connection information further includes an internal vertex list, which is a list of the internal vertices. The information processing apparatus according to claim 2.

8. The vertex connection information further includes pair information indicating the correspondence between the boundary vertices between the patches. The information processing apparatus according to claim 2.

9. The vertex connection information generation unit further generates parameters used when reconstructing the vertices and connections. The information processing apparatus according to claim 1.

10. The vertex connection information generation unit generates flag information as a parameter indicating whether at least a portion of the internal vertices of the mesh have been deleted. The information processing apparatus according to claim 9.

11. The vertex connection information generation unit generates information regarding the density of the vertices during reconstruction as the parameters. The information processing apparatus according to claim 9.

12. The vertex connection information generation unit generates flag information as a parameter indicating whether to correct the position of the vertex to be reconstructed. The information processing apparatus according to claim 9.

13. The vertex connection information generation unit generates, as parameters, information regarding the geometric features that are the target of correction for the vertices to be reconstructed. The information processing apparatus according to claim 9.

14. The vertex connection information generation unit generates, as parameters, information regarding the search range for the target of correction of the vertices to be reconstructed. The information processing apparatus according to claim 9.

15. The vertex connection information generation unit determines whether to delete the internal vertices of the mesh, and if it determines to delete them, it deletes at least a portion of the internal vertices of the mesh and generates the vertex connection information. The information processing apparatus according to claim 1.

16. At least some of the internal vertices, which are vertices of a mesh representing a three-dimensional object, located outside the boundaries of the geometric patch are deleted, and vertex connection information is generated that shows the vertices of the mesh and the connections between those vertices. Encode the aforementioned vertex connection information. Information processing methods.

17. A decoding unit that decodes encoded data of vertex connection information indicating the connections between boundary vertices, which are vertices of a mesh representing a three-dimensional object located at least at the boundary of a geometric patch, and the connections between said boundary vertices. Vertex connection reconstruction unit reconstructs the vertices located in the patch and the connections between them by using the vertex connection information obtained by decoding the encoded data by the decoding unit and a geometry image in which the patch is arranged in a two-dimensional plane, linearly interpolating the boundary vertices, determining the patch region which is the area included in the patch, generating polygons, merging unnecessary polygons outside the patch region, and merging the polygons in the patch region based on the geometry. An information processing device equipped with the following features.

18. Decode the encoded data of vertex connection information indicating the connections between boundary vertices, which are vertices of a mesh representing a three-dimensional object located at least at the boundary of a geometric patch, and the connections between said boundary vertices. Using the vertex connection information obtained by decoding the encoded data and the geometric image in which the patch is arranged in a two-dimensional plane, the boundary vertices are linearly interpolated, the patch region which is the area included in the patch is determined, polygons are generated, unnecessary polygons outside the patch region are merged, and the polygons within the patch region are merged based on the geometry to reconstruct the vertices located in the patch and the connections between those vertices. Information processing methods.

19. A decoding unit that decodes encoded data of vertex connection information indicating the connections between boundary vertices, which are vertices of a mesh representing a three-dimensional object located at least at the boundary of a geometric patch, and the connections between said boundary vertices. A vertex connection reconstruction unit reconstructs the vertices located in the patch and the connections between them by using the vertex connection information obtained by decoding the encoded data by the decoding unit and a geometric image in which the patch is arranged in a two-dimensional plane, dividing the patch into a triangular region and a rectangular region, placing the vertices in the rectangular region, determining the connections between the vertices in the rectangular region, and determining the connections between the vertices in the triangular region. An information processing device equipped with the following features.

20. Decode the encoded data of vertex connection information indicating the connections between boundary vertices, which are vertices of a mesh representing a three-dimensional object located at least at the boundary of a geometric patch, and the connections between said boundary vertices. Using the vertex connection information obtained by decoding the encoded data and the geometric image in which the patch is arranged in a two-dimensional plane, the patch is divided into a triangular region and a rectangular region, the vertices are placed in the rectangular region, the connections between the vertices in the rectangular region are determined, and the connections between the vertices in the triangular region are determined, thereby reconstructing the vertices located in the patch and the connections between those vertices. Information processing methods.

Images