Encoding method, decoding method, encoding device, and decoding device

Abstract

In an encoding method according to one aspect of the present disclosure, a plurality of items of encoded data are generated (S761) by encoding a plurality of sub-meshes, threshold value information indicating a threshold value that is used in combination processing for combining a plurality of three-dimensional points included in the plurality of sub-meshes is generated (S762) on the basis of the locations of the plurality of three-dimensional points included in the plurality of sub-meshes, and a bitstream including the plurality of items of encoded data, the threshold value information, and a plurality of items of sub-mesh identification information uniquely indicating each of the plurality of sub-meshes is generated (S763).

Description

Encoding method, decoding method, encoding device, and decoding device

[0001] The present disclosure relates to encoding methods and the like.

[0002] In US Pat. No. 6,299,549 a method and apparatus for encoding and decoding three-dimensional mesh data is proposed.

[0003] Japanese Patent Application Laid-Open No. 2006-187015

[0004] Further improvements are desired in the encoding or decoding process for three-dimensional data. The present disclosure aims to improve the encoding or decoding process for three-dimensional data.

[0005] An encoding method according to one aspect of the present disclosure generates multiple encoded data by encoding multiple submeshes, generates threshold information indicating a threshold used in a combining process for combining multiple three-dimensional points included in the multiple submeshes based on the positions of multiple three-dimensional points included in the multiple submeshes, and generates a bitstream including the multiple encoded data, the threshold information, and multiple submesh identification information that uniquely identifies each of the multiple submeshes.

[0006] These comprehensive or specific aspects may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0007] The present disclosure may contribute to improvements in encoding processes and the like related to three-dimensional data.

[0008] 1 is a conceptual diagram showing a three-dimensional mesh according to an embodiment. FIG. 2 is a conceptual diagram showing basic elements of a three-dimensional mesh according to an embodiment. FIG. 3 is a conceptual diagram showing mapping according to an embodiment. FIG. 4 is a block diagram showing a configuration example of an encoding / decoding system according to an embodiment. FIG. 5 is a block diagram showing a configuration example of an encoding device according to an embodiment. FIG. 6 is a block diagram showing another configuration example of an encoding device according to an embodiment. FIG. 7 is a block diagram showing another configuration example of a decoding device according to an embodiment. FIG. 8 is a block diagram showing another configuration example of a decoding device according to an embodiment. FIG. 9 is a block diagram showing another configuration example of a decoding device according to an embodiment. FIG. 10 is a conceptual diagram showing another configuration example of a bit stream according to an embodiment. FIG. 11 is a conceptual diagram showing yet another configuration example of a bit stream according to an embodiment. FIG. 12 is a block diagram showing a specific example of an encoding / decoding system according to an embodiment. FIG. 13 is a conceptual diagram showing an example configuration of point cloud data according to an embodiment. FIG. 14 is a conceptual diagram showing an example data file of point cloud data according to an embodiment. FIG. 15 is a conceptual diagram showing an example configuration of mesh data according to an embodiment. FIG. 16 is a conceptual diagram showing an example data file of mesh data according to an embodiment. FIG. 17 is a conceptual diagram showing types of three-dimensional data according to an embodiment. FIG. 18 is a block diagram showing an example configuration of a three-dimensional data encoder according to an embodiment. FIG. 19 is a block diagram showing an example configuration of a three-dimensional data decoder according to an embodiment. FIG. 19 is a block diagram showing another configuration example of a three-dimensional data encoder according to an embodiment. FIG. 19 is a block diagram showing another configuration example of a three-dimensional data decoder according to an embodiment. FIG. 1 is a conceptual diagram showing a specific example of encoding processing according to an embodiment. FIG. 2 is a conceptual diagram showing a specific example of decoding processing according to an embodiment. FIG. 3 is a block diagram showing an implementation example of an encoding device according to an embodiment. FIG. 4 is a block diagram showing an implementation example of a decoding device according to an embodiment. FIG. 5 is a block diagram showing another configuration example of an encoding / decoding system according to an embodiment. FIG. 6 is a block diagram showing another configuration example of an encoding device according to an embodiment. FIG. 7 is a block diagram showing another configuration example of a decoding device according to an embodiment. FIG. 8 is a block diagram showing yet another configuration example of an encoding device according to an embodiment. FIG. 9 is a block diagram showing yet another configuration example of a decoding device according to an embodiment. FIG. 10 is a flow diagram showing processing of an encoding device according to an embodiment. FIG. 11 is an explanatory diagram conceptually showing encoding of a mesh frame according to an embodiment.1 is a flow diagram showing processing of a decoding device according to an embodiment; FIG. 2 is an explanatory diagram conceptually showing decoding of a mesh frame according to an embodiment; FIG. 3 is a block diagram showing an example configuration of a decoding device according to an embodiment; FIG. 4 is a block diagram showing an example configuration of an encoding device according to an embodiment; FIG. 5 is a block diagram showing an example configuration of a decoding device according to an embodiment; FIG. 6 is an explanatory diagram showing an example of subdivision according to an embodiment; FIG. 7 is an explanatory diagram showing an example of displacement of vertices after displacement after subdivision according to an embodiment; FIG. 8 is an explanatory diagram showing example vertices of an original mesh according to an embodiment; FIG. 9 is an explanatory diagram showing an example mesh according to an embodiment; FIG. 10 is an explanatory diagram showing an example of division of a mesh into sub-meshes according to an embodiment; FIG. 11 is a first explanatory diagram showing an example of packing of displacement information into an image frame according to an embodiment; FIG. 12 is a second explanatory diagram showing an example of packing of displacement information into an image frame according to an embodiment; FIG. 13 is a third explanatory diagram showing an example of packing of displacement information into an image frame according to an embodiment; FIG. 14 is a block diagram showing another example configuration of an encoding device according to an embodiment; FIG. 15 is a block diagram showing another example configuration of a decoding device according to an embodiment; FIG. 16 is a block diagram showing a specific example configuration of a decoding device according to an embodiment; FIG. 17 is a diagram showing an example of two subdivided sub-meshes having boundary edges according to an embodiment; FIG. 18 is a block diagram showing example configurations of an encoding device and a decoding device according to an embodiment; FIG. 19 is a flow diagram showing processing of searching for matching points according to an embodiment. FIG. 1 is a diagram for explaining a positional relationship of a plurality of submeshes according to an embodiment. FIG. 2 is a diagram for explaining a search process for matching points of a plurality of submeshes according to an embodiment. FIG. 3 is a diagram for explaining a search process for matching points of a plurality of submeshes according to an embodiment. FIG. 4 is a flow diagram showing a search process for matching points according to an embodiment. FIG. 5 is a diagram for explaining a search process for matching points of a plurality of submeshes according to an embodiment. FIG. 6 is a diagram for explaining a search process for matching points of a plurality of submeshes according to an embodiment. FIG. 7 is a diagram for explaining an example of a syntax in which submesh combining metadata according to an embodiment is signaled. FIG. 8 is a diagram for explaining an example of a syntax in which submesh combining metadata according to an embodiment is signaled.1 is a diagram for explaining an example of a syntax in which submesh combining metadata according to an embodiment is signaled. FIG. 2 is a diagram for explaining an example of a syntax in which information indicating a combination of submeshes to be combined according to an embodiment is signaled. FIG. 3 is a diagram for explaining a method of assigning submesh IDs according to an embodiment. FIG. 4 is a diagram for explaining a submesh ID of a decoded submesh according to an embodiment. FIG. 5 is a diagram for explaining a submesh ID of a submesh in each processing procedure according to an embodiment. FIG. 6 is a diagram for explaining an example of a syntax in which submesh combining metadata according to an embodiment is signaled. FIG. 7 is a diagram for explaining an example of a syntax in which decoded data according to an embodiment is signaled. FIG. 8 is a diagram for explaining an example of a syntax in which submesh combining metadata according to an embodiment. FIG. 9 is a flow diagram showing a processing procedure when a three-dimensional mesh is presented by an application according to an embodiment. FIG. 10 is a flow diagram showing a submesh combining process according to an embodiment. FIG. 11 is a flow diagram showing an example of a basic encoding process according to an embodiment. FIG. 12 is a flow diagram showing an example of a basic decoding process according to an embodiment.

[0009] Introduction Three-dimensional (3D) meshes are used in computer graphics images, which may be composed of multiple temporally distinct frames, each of which may be represented by a 3D mesh.

[0010] A 3D mesh is composed of vertex information indicating the positions of each of the vertices in 3D space, connectivity information indicating the connections between the vertices, and attribute information indicating the attributes of each vertex or face. Each face is constructed according to the connectivity between the vertices. Various computer graphics images can be expressed using such 3D meshes.

[0011] Furthermore, for transmission and storage of the 3D mesh, efficient encoding and decoding of the 3D mesh is expected. For efficient encoding and decoding of the 3D mesh, arithmetic coding and decoding may be used.

[0012] Further improvements are desired in the encoding or decoding process for three-dimensional data. The present disclosure aims to improve the encoding or decoding process for three-dimensional data.

[0013] Below, examples of inventions that can be obtained from the disclosure of this specification will be given, and the effects and the like that can be obtained from these inventions will be explained.

[0014] The encoding method of Example 1 generates multiple encoded data by encoding multiple submeshes, generates threshold information indicating a threshold used in a combining process to combine multiple three-dimensional points included in the multiple submeshes based on the positions of multiple three-dimensional points included in the multiple submeshes, and generates a bitstream including the multiple encoded data, the threshold information, and multiple submesh identification information that uniquely identifies each of the multiple submeshes.

[0015] According to this, the decoding device that has acquired the bitstream can correctly determine the submeshes to be joined based on the submesh identification information and perform joining processing.

[0016] The encoding method of Example 2 is the encoding method of Example 1, and the threshold information may indicate a threshold determined for each combination of two submeshes selected from the plurality of submeshes.

[0017] For example, if a threshold is set based only on the distance between 3D points, when there are multiple submeshes to be combined with one submesh, there is a possibility that a 3D point belonging to a submesh other than the submesh containing the 3D point that should be combined may be combined. Therefore, by setting a threshold for each combination of submeshes, 3D points can be appropriately combined. Furthermore, for two submeshes that are not combined, a specific value, such as 0, can be set as the threshold information, allowing the decoding device to omit the process of determining whether or not to combine each 3D point for those two submeshes. This reduces the amount of processing required by the decoding device.

[0018] The encoding method of Example 3 is the encoding method of Example 1, wherein the threshold information may indicate a threshold value determined for each of the plurality of sub-meshes.

[0019] According to this, the decoding device that has acquired the bitstream can correctly determine the submeshes to be joined based on the submesh identification information and perform joining processing.

[0020] The decoding method of Example 4 acquires a bitstream containing multiple encoded data generated by encoding multiple submeshes, threshold information indicating one or more thresholds used in a joining process to join multiple three-dimensional points included in the multiple submeshes, and multiple submesh identification information that uniquely identifies each of the multiple submeshes, performs a decoding process to decode the multiple encoded data, and performs the joining process based on the threshold information and two or more submesh identification information from the multiple submesh identification information.

[0021] This allows the sub-meshes to be joined to be correctly determined based on the sub-mesh identification information and the joining process to be performed.

[0022] The decoding method of Example 5 is the decoding method of Example 4, further comprising: after the decoding process, a generation process is performed to generate output data including two or more submeshes among the plurality of submeshes for which the combining process is performed using the threshold information, and two or more submesh identification information among the plurality of submesh identification information that uniquely indicates each of the two or more submeshes, based on the plurality of submesh identification information; and in the combining process, the combining process may be performed based on the threshold information and the two or more submesh identification information included in the output data.

[0023] That is, the combining process uses the submesh identification information included in the output data generated by the generation process. For example, the combining process does not assume the submesh identification numbers of two or more submeshes, but instead uses the submesh identification information included in the output data to determine the submesh identification numbers of two or more submeshes. For example, if the combining process assumes that the order in which the encoded data of multiple submeshes is decoded is the order of the submesh identification numbers of the multiple submeshes, if some of the encoded data is lost when the bitstream is transmitted from the encoding device to the decoding device, a submesh different from the intended target submesh may be selected. In this case, the combining process will not be performed properly. Therefore, after the decoding process, the submesh identification numbers of two or more submeshes are clearly determined before the combining process is performed, so that the submeshes to be combined can be correctly determined and the combining process can be performed.

[0024] A decoding method of Example 6 is the decoding method of Example 5, wherein the output data may further include information indicating the number of the two or more sub-meshes.

[0025] This allows the process of using two or more sub-meshes in the combining process to be carried out appropriately.

[0026] The encoding device of Example 7 includes a processor and a memory, and the processor uses the memory to generate multiple encoded data by encoding multiple submeshes, generates threshold information indicating a threshold used in a combining process to combine multiple three-dimensional points included in the multiple submeshes based on the positions of multiple three-dimensional points included in the multiple submeshes, and generates a bitstream including the multiple encoded data, the threshold information, and multiple submesh identification information that uniquely identifies each of the multiple submeshes.

[0027] This provides the same effects as the encoding method according to the first technique.

[0028] The decoding device of Example 8 includes a processor and a memory, and the processor uses the memory to acquire a bitstream including multiple encoded data generated by encoding multiple submeshes, threshold information indicating one or more thresholds used in a joining process to join multiple three-dimensional points included in the multiple submeshes, and multiple submesh identification information that uniquely identifies each of the multiple submeshes, performs a decoding process to decode the multiple encoded data, and performs the joining process based on the threshold information and two or more submesh identification information of the multiple submeshes.

[0029] This provides the same effect as the decoding method according to Technique 4.

[0030] Furthermore, these comprehensive or specific aspects may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0031] <Expressions and Terms> The following expressions and terms are used herein.

[0032] (1) Three-dimensional Mesh A three-dimensional mesh is a collection of multiple faces, and represents, for example, a three-dimensional object. A three-dimensional mesh is mainly composed of vertex information, connectivity information, and attribute information. A three-dimensional mesh may be expressed as a polygon mesh or a mesh. A three-dimensional mesh may also vary over time. A three-dimensional mesh may include metadata related to the vertex information, connectivity information, and attribute information, and may also include other additional information.

[0033] (2) Vertex Information Vertex information is information indicating a vertex. For example, the vertex information indicates the position of a vertex in a three-dimensional space. Furthermore, a vertex corresponds to a vertex of a face that constitutes a three-dimensional mesh. Vertex information may be expressed as "geometry." Furthermore, vertex information may be expressed as position information.

[0034] (3) Connection Information Connection information is information that indicates connections between vertices. For example, connection information indicates connections for forming faces or edges of a three-dimensional mesh. Connection information may be expressed as "Connectivity." Connection information may also be expressed as face information.

[0035] (4) Attribute Information Attribute information is information that indicates attributes of a vertex or a face. For example, attribute information indicates attributes such as a color, an image, and a normal vector associated with a vertex or a face. Attribute information may be expressed as "texture."

[0036] (5) Faces A face is an element that makes up a three-dimensional mesh. Specifically, a face is a polygon on a plane in three-dimensional space. For example, a face can be defined as a triangle in three-dimensional space.

[0037] (6) Plane A plane is a two-dimensional plane in a three-dimensional space. For example, a polygon is formed on a plane, and multiple polygons are formed on multiple planes.

[0038] (7) Bitstream: A bitstream corresponds to coded information. A bitstream may also be referred to as a stream, a coded bitstream, a compressed bitstream, or a coded signal.

[0039] (8) Encoding and Decoding The term encoding may be substituted with terms such as storing, including, writing, describing, signaling, sending, notifying, saving, or compressing, and these terms may be interchangeable. For example, encoding information may mean including the information in a bitstream. Also, encoding information into a bitstream may mean encoding the information to generate a bitstream that includes the encoded information.

[0040] Additionally, the term "decode" may be replaced with terms such as "read," "decode," "read," "load," "derive," "obtain," "receive," "extract," "reconstruct," "reconstruct," "decompress," or "decompress," and these terms may be interchangeable. For example, decoding information may mean obtaining information from a bitstream. Decoding information from a bitstream may mean decoding the bitstream to obtain information contained in the bitstream.

[0041] (9) Ordinal Numbers In the description, ordinal numbers such as first and second may be assigned to components, etc. These ordinal numbers may be changed as appropriate. Furthermore, new ordinal numbers may be assigned to components, etc., or removed. Furthermore, these ordinal numbers may be assigned to elements in order to identify them, and may not correspond to a meaningful order.

[0042] <Three-dimensional mesh> Fig. 1 is a conceptual diagram showing a three-dimensional mesh according to this embodiment. A three-dimensional mesh is composed of multiple faces. For example, each face is a triangle. The vertices of these triangles are defined in three-dimensional space. The three-dimensional mesh then represents a three-dimensional object. Each face may have a color or an image.

[0043] FIG. 2 is a conceptual diagram showing the basic elements of a three-dimensional mesh according to this embodiment. A three-dimensional mesh is composed of vertex information, connection information, and attribute information. The vertex information indicates the positions of the vertices of a face in three-dimensional space. The connection information indicates the connections between the vertices. A face can be identified by the vertex information and connection information. In other words, a colorless three-dimensional object is formed in three-dimensional space by the vertex information and connection information.

[0044] The attribute information may be associated with a vertex or a face. The attribute information associated with a vertex may be expressed as "Attribute Per Point." The attribute information associated with a vertex may indicate an attribute of the vertex itself, or may indicate an attribute of a face connected to the vertex.

[0045] For example, a color may be associated with a vertex as attribute information. The color associated with a vertex may be the color of the vertex itself, or the color of a face connected to the vertex. The color of a face may be the average of multiple colors associated with multiple vertices of the face. Furthermore, a normal vector may be associated with a vertex or a face as attribute information. Such a normal vector can represent the front and back of a face.

[0046] A two-dimensional image may be associated with a surface as attribute information. The two-dimensional image associated with a surface is also expressed as a texture image or an "Attribute Map." Information indicating mapping between the surface and the two-dimensional image may be associated with the surface as attribute information. Information indicating such mapping may be expressed as mapping information, vertex information of a texture image, texture coordinates, or "Attribute UV Coordinate."

[0047] Furthermore, information such as color, image, and moving image used as attribute information may be expressed as "parametric space."

[0048] The attribute information allows texture to be reflected on the three-dimensional object. That is, a three-dimensional object having color is formed in three-dimensional space based on the vertex information, connection information, and attribute information.

[0049] In the above, the attribute information is associated with the vertices or faces, but it may also be associated with the edges.

[0050] 3 is a conceptual diagram illustrating mapping according to this embodiment. For example, a region of a two-dimensional image on a two-dimensional plane can be mapped onto a surface of a three-dimensional mesh in three-dimensional space. Specifically, coordinate information of the region in the two-dimensional image is associated with the surface of the three-dimensional mesh. As a result, an image of the mapped region in the two-dimensional image is reflected on the surface of the three-dimensional mesh.

[0051] By using the mapping, the 2D image used as attribute information can be separated from the 3D mesh. For example, in encoding the 3D mesh, the 2D image may be encoded by an image encoding method or a video encoding method.

[0052] <System Configuration> Fig. 4 is a block diagram showing an example of the configuration of a coding / decoding system according to this embodiment. In Fig. 4, the coding / decoding system includes a coding device 100 and a decoding device 200.

[0053] For example, the encoding device 100 obtains a three-dimensional mesh and encodes the three-dimensional mesh into a bitstream. Then, the encoding device 100 outputs the bitstream to the network 300. For example, the bitstream includes the encoded three-dimensional mesh and control information for decoding the encoded three-dimensional mesh. By encoding the three-dimensional mesh, information about the three-dimensional mesh is compressed.

[0054] The network 300 transmits a bitstream from the encoding device 100 to the decoding device 200. The network 300 may be the Internet, a wide area network (WAN), a local area network (LAN), or a combination of these. The network 300 is not necessarily limited to bidirectional communication, and may be a unidirectional communication network for terrestrial digital broadcasting, satellite broadcasting, or the like.

[0055] Furthermore, the network 300 can be replaced by a recording medium such as a DVD (Digital Versatile Disc) or a BD (Blu-Ray Disc (registered trademark)).

[0056] The decoding device 200 obtains a bitstream and decodes a three-dimensional mesh from the bitstream. By decoding the three-dimensional mesh, information about the three-dimensional mesh is expanded. For example, the decoding device 200 decodes the three-dimensional mesh according to a decoding method corresponding to the encoding method used by the encoding device 100 to encode the three-dimensional mesh. That is, the encoding device 100 and the decoding device 200 perform encoding and decoding according to encoding methods and decoding methods that correspond to each other.

[0057] The 3D mesh before encoding may also be referred to as an original 3D mesh, and the 3D mesh after decoding may also be referred to as a reconstructed 3D mesh.

[0058] 5 is a block diagram showing an example of the configuration of a coding device 100 according to this embodiment. For example, the coding device 100 includes a vertex information encoder 101, a connection information encoder 102, and an attribute information encoder 103.

[0059] The vertex information encoder 101 is an electrical circuit that encodes vertex information. For example, the vertex information encoder 101 encodes the vertex information into a bitstream according to a format defined for the vertex information.

[0060] The connection information encoder 102 is an electrical circuit that encodes the connection information, for example, the connection information encoder 102 encodes the connection information into a bitstream according to a format defined for the connection information.

[0061] The attribute information encoder 103 is an electric circuit that encodes the attribute information. For example, the attribute information encoder 103 encodes the attribute information into a bit stream in accordance with a format defined for the attribute information.

[0062] The vertex information, connectivity information, and attribute information may be coded using variable-length coding or fixed-length coding, such as Huffman coding or context-adaptive binary arithmetic coding (CABAC).

[0063] The vertex information encoder 101, the connection information encoder 102, and the attribute information encoder 103 may be integrated together, or each of the vertex information encoder 101, the connection information encoder 102, and the attribute information encoder 103 may be further subdivided into multiple components.

[0064] 6 is a block diagram showing another example of the configuration of the encoding device 100 according to this embodiment. For example, the encoding device 100 includes a pre-processor 104 and a post-processor 105 in addition to the configuration shown in FIG.

[0065] The preprocessor 104 is an electrical circuit that performs processing before encoding the vertex information, connectivity information, and attribute information. For example, the preprocessor 104 may perform a conversion process, a separation process, a multiplexing process, or the like on the 3D mesh before encoding. More specifically, for example, the preprocessor 104 may separate the vertex information, connectivity information, and attribute information from the 3D mesh before encoding.

[0066] The post-processor 105 is an electrical circuit that performs processing after the vertex information, connection information, and attribute information are encoded. For example, the post-processor 105 may perform conversion processing, separation processing, multiplexing processing, or the like on the encoded vertex information, connection information, and attribute information. More specifically, for example, the post-processor 105 may multiplex the encoded vertex information, connection information, and attribute information into a bitstream. Furthermore, for example, the post-processor 105 may further perform variable-length coding on the encoded vertex information, connection information, and attribute information.

[0067] 7 is a block diagram showing an example of the configuration of a decoding device 200 according to this embodiment. For example, the decoding device 200 includes a vertex information decoder 201, a connection information decoder 202, and an attribute information decoder 203.

[0068] The vertex information decoder 201 is an electrical circuit that decodes vertex information. For example, the vertex information decoder 201 decodes vertex information from a bitstream according to a format defined for the vertex information.

[0069] The connection information decoder 202 is an electrical circuit that decodes the connection information, for example, the connection information decoder 202 decodes the connection information from the bitstream according to a format defined for the connection information.

[0070] The attribute information decoder 203 is an electric circuit that decodes the attribute information. For example, the attribute information decoder 203 decodes the attribute information from the bitstream in accordance with a format defined for the attribute information.

[0071] The vertex information, connection information, and attribute information may be decoded using variable length decoding or fixed length decoding, which may correspond to Huffman coding, context-adaptive binary arithmetic coding (CABAC), or the like.

[0072] The vertex information decoder 201, the connection information decoder 202, and the attribute information decoder 203 may be integrated together, or each of the vertex information decoder 201, the connection information decoder 202, and the attribute information decoder 203 may be further subdivided into multiple components.

[0073] 8 is a block diagram showing another example of the configuration of the decoding device 200 according to this embodiment. For example, the decoding device 200 includes a pre-processor 204 and a post-processor 205 in addition to the configuration shown in FIG.

[0074] The preprocessor 204 is an electrical circuit that performs processing before decoding the vertex information, connection information, and attribute information. For example, the preprocessor 204 may perform conversion processing, separation processing, multiplexing processing, or the like on the bitstream before decoding the vertex information, connection information, and attribute information.

[0075] More specifically, for example, the preprocessor 204 may separate a sub-bitstream corresponding to vertex information, a sub-bitstream corresponding to connectivity information, and a sub-bitstream corresponding to attribute information from the bitstream. Also, for example, the preprocessor 204 may perform variable-length decoding on the bitstream in advance before decoding the vertex information, connectivity information, and attribute information.

[0076] The post-processor 205 is an electrical circuit that performs processing after the vertex information, connection information, and attribute information are decoded. For example, the post-processor 205 may perform conversion processing, separation processing, multiplexing processing, or the like on the decoded vertex information, connection information, and attribute information. More specifically, for example, the post-processor 205 may multiplex the decoded vertex information, connection information, and attribute information onto a three-dimensional mesh.

[0077] <Bitstream> Vertex information, connection information, and attribute information are coded and stored in a bitstream. The relationship between this information and the bitstream is shown below.

[0078] 9 is a conceptual diagram showing an example of the configuration of a bitstream according to this embodiment. In this example, connection information, vertex information, and attribute information are integrated in the bitstream. For example, the connection information, vertex information, and attribute information may be included in a single file.

[0079] Furthermore, multiple portions of this information may be stored sequentially, such as a first portion of connection information, a first portion of vertex information, a first portion of attribute information, a second portion of connection information, a second portion of vertex information, a second portion of attribute information, etc. These multiple portions may correspond to multiple portions that are different in time, multiple portions that are different in space, or multiple different faces.

[0080] Furthermore, the storage order of the connection information, vertex information, and attribute information is not limited to the above example, and a storage order different from the above example may be used.

[0081] 10 is a conceptual diagram showing another example of the configuration of a bitstream according to this embodiment. In this example, a plurality of files are included in the bitstream, and connection information, vertex information, and attribute information are stored in different files. Here, a file containing connection information, a file containing vertex information, and a file containing attribute information are shown, but the storage format is not limited to this example. For example, two types of information among the connection information, vertex information, and attribute information may be included in one file, and the remaining type of information may be included in another file.

[0082] Alternatively, the information may be split and stored in more files. For example, multiple pieces of connectivity information may be stored in multiple files, multiple pieces of vertex information may be stored in multiple files, or multiple pieces of attribute information may be stored in multiple files. These multiple pieces may correspond to multiple temporally different pieces, multiple spatially different pieces, or multiple different faces.

[0083] Furthermore, the storage order of the connection information, vertex information, and attribute information is not limited to the above example, and a storage order different from the above example may be used.

[0084] 11 is a conceptual diagram showing another example of the configuration of a bitstream according to this embodiment. In this example, the bitstream is composed of multiple separable sub-bitstreams, and connection information, vertex information, and attribute information are stored in different sub-bitstreams.

[0085] Here, a sub-bitstream containing connection information, a sub-bitstream containing vertex information, and a sub-bitstream containing attribute information are shown, but the storage format is not limited to this example.

[0086] For example, two types of information among the connection information, vertex information, and attribute information may be included in one sub-bitstream, and the remaining type of information may be included in another sub-bitstream. Specifically, attribute information of a two-dimensional image or the like may be stored in a sub-bitstream that complies with an image coding method, separate from the sub-bitstreams of the connection information and vertex information.

[0087] Each sub-bitstream may also include multiple files, and multiple pieces of connectivity information may be stored in multiple files, multiple pieces of vertex information may be stored in multiple files, or multiple pieces of attribute information may be stored in multiple files.

[0088] 9, 10, and 11, and a storage order different from the above examples may be used. For example, the vertex information, connection information, and attribute information may be stored in the bitstream in this order. Alternatively, the connection information, connection information, and attribute information may be stored in the bitstream in any of the following orders: connection information, attribute information, and vertex information; vertex information, attribute information, and connection information; attribute information, connection information, and vertex information; or attribute information, vertex information, and connection information.

[0089] Furthermore, each of the connection information, vertex information, and attribute information may be divided into a plurality of data, and the plurality of data may be stored in a cyclical or random order within the bitstream.

[0090] 12 is a block diagram showing a specific example of an encoding / decoding system according to this embodiment. In FIG. 12, the encoding / decoding system includes a three-dimensional data encoding system 110, a three-dimensional data decoding system 210, and an external connector 310.

[0091] The three-dimensional data encoding system 110 includes a controller 111, an input / output processor 112, a three-dimensional data encoder 113, a three-dimensional data generator 115, and a system multiplexer 114. The three-dimensional data decoding system 210 includes a controller 211, an input / output processor 212, a three-dimensional data decoder 213, a system demultiplexer 214, a presenter 215, and a user interface 216.

[0092] In the three-dimensional data encoding system 110, sensor data is input from a sensor terminal to a three-dimensional data generator 115. The three-dimensional data generator 115 generates three-dimensional data, such as point cloud data or mesh data, from the sensor data and inputs it to a three-dimensional data encoder 113.

[0093] For example, the three-dimensional data generator 115 generates vertex information, and generates connection information and attribute information corresponding to the vertex information. The three-dimensional data generator 115 may process the vertex information when generating the connection information and attribute information. For example, the three-dimensional data generator 115 may reduce the amount of data by deleting duplicate vertices, or may transform the vertex information (such as by shifting its position, rotating it, or normalizing it). The three-dimensional data generator 115 may also render the attribute information.

[0094] Furthermore, although the three-dimensional data generator 115 is a component of the three-dimensional data encoding system 110 in FIG. 12, it may be arranged externally and independently of the three-dimensional data encoding system 110.

[0095] The sensor terminal that provides the sensor data for generating the three-dimensional data may be, for example, a moving body such as an automobile, a flying object such as an airplane, a mobile terminal, a camera, etc. Furthermore, a distance sensor such as a LIDAR, a millimeter wave radar, an infrared sensor, or a range finder, a stereo camera, or a combination of multiple monocular cameras may also be used as the sensor terminal.

[0096] The sensor data may be the distance (position) of the object, monocular camera images, stereo camera images, color, reflectance, sensor attitude, orientation, gyro, sensing position (GPS information or altitude), speed, acceleration, sensing time, temperature, air pressure, humidity, or magnetism.

[0097] The three-dimensional data encoder 113 corresponds to the encoding device 100 shown in FIG. 5 and other figures. For example, the three-dimensional data encoder 113 encodes three-dimensional data to generate encoded data. The three-dimensional data encoder 113 also generates control information when encoding the three-dimensional data. The three-dimensional data encoder 113 then inputs the encoded data together with the control information to the system multiplexer 114.

[0098] The encoding method for the three-dimensional data may be an encoding method using geometry or an encoding method using a video codec. Here, the encoding method using geometry may also be referred to as a geometry-based encoding method. The encoding method using a video codec may also be referred to as a video-based encoding method.

[0099] The system multiplexer 114 multiplexes the encoded data and control information input from the 3D data encoder 113 to generate multiplexed data using a specified multiplexing method. The system multiplexer 114 may multiplex other media such as video, audio, subtitles, application data, or document files, or reference time information, along with the encoded data and control information of the 3D data. Furthermore, the system multiplexer 114 may multiplex attribute information related to the sensor data or the 3D data.

[0100] For example, the multiplexed data may have a file format for storage or a packet format for transmission. As these formats, ISOBMFF or a format based on ISOBMFF may be used. Also, MPEG-DASH, MMT, MPEG-2 TS Systems, RTP, or the like may be used.

[0101] The multiplexed data is then output as a transmission signal to the external connector 310 by the input / output processor 112. The multiplexed data may be transmitted as a transmission signal by wire or wirelessly. Alternatively, the multiplexed data is stored in an internal memory or a storage device. The multiplexed data may be transmitted to a cloud server via the Internet or may be stored in an external storage device.

[0102] For example, the transmission or storage of the multiplexed data is performed by a method according to the medium for transmission or storage, such as broadcasting or communication. The communication protocol may be http, ftp, TCP, UDP, IP, or a combination thereof. Furthermore, a pull-type communication method or a push-type communication method may be used.

[0103] For wired transmission, Ethernet (registered trademark), USB, RS-232C, HDMI (registered trademark), coaxial cable, etc. may be used. For wireless transmission, 3GPP (registered trademark), 3G / 4G / 5G defined by IEEE, wireless LAN, Wi-Fi, Bluetooth, or millimeter wave may be used. For broadcasting, for example, DVB-T2, DVB-S2, DVB-C2, ATSC3.0, or ISDB-S3 may be used.

[0104] The sensor data may be input to the three-dimensional data generator 115 or the system multiplexer 114. The three-dimensional data or encoded data may be output as a transmission signal directly to the external connector 310 via the input / output processor 112. The transmission signal output from the three-dimensional data encoding system 110 is input to the three-dimensional data decoding system 210 via the external connector 310.

[0105] Furthermore, each operation of the three-dimensional data encoding system 110 may be controlled by a controller 111 that executes an application program.

[0106] In the three-dimensional data decoding system 210, a transmission signal is input to an input / output processor 212. The input / output processor 212 decodes multiplexed data having a file format or a packet format from the transmission signal and inputs the multiplexed data to a system demultiplexer 214. The system demultiplexer 214 obtains coded data and control information from the multiplexed data and inputs them to a three-dimensional data decoder 213. The system demultiplexer 214 may extract other media or reference time information from the multiplexed data.

[0107] The three-dimensional data decoder 213 corresponds to the decoding device 200 shown in Fig. 7 etc. For example, the three-dimensional data decoder 213 decodes three-dimensional data from the encoded data based on a predefined encoding method. The three-dimensional data is then presented to the user by the presenter 215.

[0108] Additionally, additional information such as sensor data may be input to the presenter 215. The presenter 215 may present three-dimensional data based on the additional information. Additionally, a user instruction may be input from a user terminal to the user interface 216. Then, the presenter 215 may present three-dimensional data based on the input instruction.

[0109] The input / output processor 212 may acquire the three-dimensional data and the encoded data from the external connector 310 .

[0110] Furthermore, each operation of the three-dimensional data decoding system 210 may be controlled by a controller 211 that executes an application program.

[0111] 13 is a conceptual diagram showing an example of the configuration of point cloud data according to this embodiment. The point cloud data is data of a group of points representing a three-dimensional object.

[0112] Specifically, a point cloud is made up of a plurality of points, and has position information indicating the three-dimensional coordinate position of each point and attribute information indicating the attribute of each point. The position information is also expressed as geometry.

[0113] The type of attribute information may be, for example, color, reflectance, etc. One point may be associated with attribute information of one type, one point may be associated with attribute information of multiple different types, or one point may be associated with attribute information having multiple values ​​for the same type.

[0114] 14 is a conceptual diagram showing an example of a data file of point cloud data according to this embodiment. This example shows a case where there is a one-to-one correspondence between position information items and attribute information items, and shows position information and attribute information for N points that make up the point cloud data. In this example, the position information is information indicating a three-dimensional coordinate position using three axes, x, y, and z, and the attribute information is information indicating a color using RGB. A PLY file or the like can be used as a representative data file for point cloud data.

[0115] 15 is a conceptual diagram showing an example of the configuration of mesh data according to this embodiment. Mesh data is data used in CG (Computer Graphics) and the like, and is three-dimensional mesh data that shows the three-dimensional shape of an object using multiple surfaces. Each surface is also expressed as a polygon, and has a polygonal shape such as a triangle or a rectangle.

[0116] Specifically, a 3D mesh is composed of a plurality of points constituting a point cloud, as well as a plurality of edges and a plurality of faces. Each point is also expressed as a vertex or a position. Each edge corresponds to a line segment connected by two vertices. Each face corresponds to an area surrounded by three or more edges.

[0117] Furthermore, a three-dimensional mesh has position information indicating the three-dimensional coordinate positions of vertices. The position information is also expressed as vertex information or geometry. A three-dimensional mesh also has connection information indicating the relationship between multiple vertices that make up an edge or a face. The connection information is also expressed as connectivity. A three-dimensional mesh also has attribute information indicating the attributes of the vertices, edges, or faces. The attribute information in a three-dimensional mesh is also expressed as texture.

[0118] For example, the attribute information may indicate the color, reflectance, or normal vector for a vertex, edge, or face. The direction of the normal vector may represent the front and back of the face.

[0119] The mesh data may be stored in a data file format such as an object file.

[0120] 16 is a conceptual diagram showing an example of a data file of mesh data according to this embodiment. In this example, the data file includes position information G(1) to G(N) of N vertices that make up the three-dimensional mesh, and attribute information A1(1) to A1(N) of the N vertices. Also, in this example, M pieces of attribute information A2(1) to A2(M) are included. The attribute information items do not need to correspond one-to-one to vertices or faces. Furthermore, attribute information need not exist.

[0121] The connection information is represented by a combination of vertex indices. n[1, 3, 4] indicates a triangular face formed by three vertices, n=1, n=3, and n=4. Also, m[2, 4, 6] indicates that the attribute information of m=2, m=4, and m=6 corresponds to the three vertices, respectively.

[0122] Furthermore, the actual contents of the attribute information may be written in a separate file. A pointer to that content may be associated with a vertex, a face, or the like. For example, attribute information indicating an image for a face may be stored in a two-dimensional attribute map file. The file name of the attribute map and two-dimensional coordinate values ​​in the attribute map may be written in attribute information A2(1) to A2(M). The method of specifying attribute information for a face is not limited to these methods, and any method may be used.

[0123] 17 is a conceptual diagram showing types of three-dimensional data according to this embodiment. Point cloud data and mesh data may represent static objects or dynamic objects. A static object is an object that does not change over time, and a dynamic object is an object that changes over time. A static object may correspond to three-dimensional data for any point in time.

[0124] For example, point cloud data for a given point in time may be referred to as a PCC frame, mesh data for a given point in time may be referred to as a mesh frame, and PCC frames and mesh frames may be simply referred to as frames.

[0125] The area of ​​the object may be limited to a certain range, as in normal video data, or may not be limited, as in map data. The density of points or surfaces may be determined in various ways. Sparse point cloud data or sparse mesh data may be used, or dense point cloud data or dense mesh data may be used.

[0126] Next, encoding and decoding of a point cloud or a three-dimensional mesh will be described. The device, process, or syntax for encoding and decoding vertex information of a three-dimensional mesh in the present disclosure may be applied to encoding and decoding of a point cloud. The device, process, or syntax for encoding and decoding of a point cloud in the present disclosure may be applied to encoding and decoding vertex information of a three-dimensional mesh.

[0127] Furthermore, a device, process, or syntax for encoding and decoding attribute information of a point cloud in the present disclosure may be applied to encoding and decoding connectivity information or attribute information of a three-dimensional mesh.Furthermore, a device, process, or syntax for encoding and decoding connectivity information or attribute information of a three-dimensional mesh in the present disclosure may be applied to encoding and decoding attribute information of a point cloud.

[0128] Furthermore, at least some of the processing may be shared between the encoding and decoding of point cloud data and the encoding and decoding of mesh data, thereby reducing the scale of the circuit and software program.

[0129] 18 is a block diagram showing an example configuration of a three-dimensional data encoder 113 according to this embodiment. In this example, the three-dimensional data encoder 113 includes a vertex information encoder 121, an attribute information encoder 122, a metadata encoder 123, and a multiplexer 124. The vertex information encoder 121, the attribute information encoder 122, and the multiplexer 124 may correspond to the vertex information encoder 101, the attribute information encoder 103, the post-processor 105, etc. in FIG.

[0130] In this example, the three-dimensional data encoder 113 encodes the three-dimensional data according to a geometry-based encoding method, which takes into account the three-dimensional structure. In addition, in the geometry-based encoding method, attribute information is encoded using configuration information obtained in encoding the vertex information.

[0131] Specifically, first, vertex information, attribute information, and metadata included in three-dimensional data generated from sensor data are input to a vertex information encoder 121, an attribute information encoder 122, and a metadata encoder 123, respectively. Here, connectivity information included in the three-dimensional data may be treated in the same way as attribute information. In addition, in the case of point cloud data, position information may be treated as vertex information.

[0132] The vertex information encoder 121 encodes the vertex information into compressed vertex information and outputs the compressed vertex information as encoded data to the multiplexer 124. The vertex information encoder 121 also generates metadata for the compressed vertex information and outputs it to the multiplexer 124. The vertex information encoder 121 also generates configuration information and outputs it to the attribute information encoder 122.

[0133] The attribute information encoder 122 uses the configuration information generated by the vertex information encoder 121 to encode the attribute information into compressed attribute information and outputs the compressed attribute information as encoded data to the multiplexer 124. The attribute information encoder 122 also generates metadata of the compressed attribute information and outputs it to the multiplexer 124.

[0134] The metadata encoder 123 encodes compressible metadata into compressed metadata and outputs the compressed metadata as encoded data to the multiplexer 124. The metadata encoded by the metadata encoder 123 may be used to encode vertex information and attribute information.

[0135] The multiplexer 124 multiplexes the compressed vertex information, the compressed vertex information metadata, the compressed attribute information, the compressed attribute information metadata, and the compressed metadata into a bitstream, and then inputs the bitstream to the system layer.

[0136] 19 is a block diagram showing an example configuration of a three-dimensional data decoder 213 according to this embodiment. In this example, the three-dimensional data decoder 213 includes a vertex information decoder 221, an attribute information decoder 222, a metadata decoder 223, and a demultiplexer 224. The vertex information decoder 221, the attribute information decoder 222, and the demultiplexer 224 may correspond to the vertex information decoder 201, the attribute information decoder 203, the preprocessor 204, and the like in FIG.

[0137] In this example, the three-dimensional data decoder 213 decodes three-dimensional data according to a geometry-based encoding method. The three-dimensional structure is taken into consideration in the decoding according to the geometry-based encoding method. Furthermore, in the decoding according to the geometry-based encoding method, attribute information is decoded using configuration information obtained in decoding vertex information.

[0138] Specifically, first, a bitstream is input from the system layer to a demultiplexer 224. The demultiplexer 224 separates compressed vertex information, compressed vertex information metadata, compressed attribute information, compressed attribute information metadata, and compressed metadata from the bitstream. The compressed vertex information and compressed vertex information metadata are input to a vertex information decoder 221. The compressed attribute information and compressed attribute information metadata are input to an attribute information decoder 222. The metadata is input to a metadata decoder 223.

[0139] The vertex information decoder 221 decodes vertex information from the compressed vertex information using metadata of the compressed vertex information. The vertex information decoder 221 also generates configuration information and outputs it to the attribute information decoder 222. The attribute information decoder 222 decodes attribute information from the compressed attribute information using the configuration information generated by the vertex information decoder 221 and the metadata of the compressed attribute information. The metadata decoder 223 decodes metadata from the compressed metadata. The metadata decoded by the metadata decoder 223 may be used to decode the vertex information and the attribute information.

[0140] Thereafter, the vertex information, attribute information, and metadata are output as three-dimensional data from the three-dimensional data decoder 213. Note that, for example, this metadata is metadata of the vertex information and attribute information, and can be used in an application program.

[0141] 20 is a block diagram showing another example configuration of the three-dimensional data encoder 113 according to the present embodiment. In this example, the three-dimensional data encoder 113 includes a vertex image generator 131, an attribute image generator 132, a metadata generator 133, a video encoder 134, a metadata encoder 123, and a multiplexer 124. The vertex image generator 131, the attribute image generator 132, and the video encoder 134 may correspond to the vertex information encoder 101 and the attribute information encoder 103 in FIG. 6 , etc.

[0142] In this example, the 3D data encoder 113 encodes the 3D data according to a video-based encoding method. In encoding according to the video-based encoding method, multiple 2D images are generated from the 3D data, and the multiple 2D images are encoded according to a video encoding method. Here, the video encoding method may be High Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), or the like.

[0143] Specifically, first, vertex information and attribute information included in three-dimensional data generated from sensor data are input to a metadata generator 133. The vertex information and attribute information are then input to a vertex image generator 131 and an attribute image generator 132, respectively. The metadata included in the three-dimensional data is then input to a metadata encoder 123. Here, connectivity information included in the three-dimensional data may be treated in the same way as attribute information. In the case of point cloud data, position information may be treated as vertex information.

[0144] The metadata generator 133 generates map information of a plurality of two-dimensional images from the vertex information and attribute information, and inputs the map information to the vertex image generator 131, the attribute image generator 132, and the metadata encoder 123.

[0145] The vertex image generator 131 generates a vertex image based on the vertex information and map information, and inputs the generated image to the video encoder 134. The attribute image generator 132 generates an attribute image based on the attribute information and map information, and inputs the generated image to the video encoder 134.

[0146] The video encoder 134 encodes the vertex images and attribute images into compressed vertex information and compressed attribute information, respectively, in accordance with a video encoding method, and outputs the compressed vertex information and compressed attribute information as encoded data to the multiplexer 124. The video encoder 134 also generates metadata for the compressed vertex information and metadata for the compressed attribute information, and outputs them to the multiplexer 124.

[0147] The metadata encoder 123 encodes the compressible metadata into compressed metadata and outputs the compressed metadata as encoded data to the multiplexer 124. The compressible metadata includes map information. The metadata encoded by the metadata encoder 123 may also be used to encode vertex information and attribute information.

[0148] The multiplexer 124 multiplexes the compressed vertex information, the compressed vertex information metadata, the compressed attribute information, the compressed attribute information metadata, and the compressed metadata into a bitstream, and then inputs the bitstream to the system layer.

[0149] 21 is a block diagram showing another example configuration of the 3D data decoder 213 according to this embodiment. In this example, the 3D data decoder 213 includes a vertex information generator 231, an attribute information generator 232, a video decoder 234, a metadata decoder 223, and a demultiplexer 224. The vertex information generator 231, the attribute information generator 232, and the video decoder 234 may correspond to the vertex information decoder 201 and the attribute information decoder 203 in FIG. 8, etc.

[0150] In this example, the 3D data decoder 213 decodes the 3D data according to a video-based coding method. In the decoding according to the video-based coding method, a plurality of 2D images are decoded according to a video coding method, and 3D data is generated from the plurality of 2D images. Here, the video coding method may be High Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), or the like.

[0151] Specifically, first, a bitstream is input from the system layer to the demultiplexer 224. The demultiplexer 224 separates compressed vertex information, compressed vertex information metadata, compressed attribute information, compressed attribute information metadata, and compressed metadata from the bitstream. The compressed vertex information, compressed vertex information metadata, compressed attribute information, and compressed attribute information metadata are input to the video decoder 234. The compressed metadata is input to the metadata decoder 223.

[0152] The video decoder 234 decodes the vertex images in accordance with the video encoding method. At this time, the video decoder 234 decodes the vertex images from the compressed vertex information using the metadata of the compressed vertex information. Then, the video decoder 234 inputs the vertex images to the vertex information generator 231. The video decoder 234 also decodes the attribute images in accordance with the video encoding method. At this time, the video decoder 234 decodes the attribute images from the compressed attribute information using the metadata of the compressed attribute information. Then, the video decoder 234 inputs the attribute images to the attribute information generator 232.

[0153] The metadata decoder 223 decodes metadata from the compressed metadata. The metadata decoded by the metadata decoder 223 includes map information used to generate vertex information and attribute information. The metadata decoded by the metadata decoder 223 may also be used to decode vertex images and attribute images.

[0154] The vertex information generator 231 reproduces vertex information from the vertex image in accordance with the map information included in the metadata decoded by the metadata decoder 223. The attribute information generator 232 reproduces attribute information from the attribute image in accordance with the map information included in the metadata decoded by the metadata decoder 223.

[0155] Thereafter, the vertex information, attribute information, and metadata are output as three-dimensional data from the three-dimensional data decoder 213. Note that, for example, this metadata is metadata of the vertex information and attribute information, and can be used in an application program.

[0156] Fig. 22 is a conceptual diagram showing a specific example of encoding processing according to this embodiment. Fig. 22 shows a three-dimensional data encoder 113 and a description encoder 148. In this example, the three-dimensional data encoder 113 includes a two-dimensional data encoder 141 and a mesh data encoder 142. The two-dimensional data encoder 141 includes a texture encoder 143. The mesh data encoder 142 includes a vertex information encoder 144 and a connection information encoder 145.

[0157] The vertex information encoder 144, the connection information encoder 145, and the texture encoder 143 may correspond to the vertex information encoder 101, the connection information encoder 102, and the attribute information encoder 103 in FIG.

[0158] For example, the two-dimensional data encoder 141 operates as a texture encoder 143 and generates a texture file by encoding the texture corresponding to the attribute information as two-dimensional data according to an image encoding method or a video encoding method.

[0159] The mesh data encoder 142 also operates as a vertex information encoder 144 and a connectivity information encoder 145, and generates a mesh file by encoding the vertex information and connectivity information. The mesh data encoder 142 may further encode mapping information for textures. The encoded mapping information may then be included in the mesh file.

[0160] The description encoder 148 also generates a description file by encoding a description corresponding to metadata such as text data. The description encoder 148 may encode the description at the system layer. For example, the description encoder 148 may be included in the system multiplexer 114 of FIG. 12 .

[0161] The above operations generate a bitstream containing texture files, mesh files, and description files, which may be multiplexed into the bitstream in file formats such as glTF (Graphics Language Transmission Format) or USD (Universal Scene Description).

[0162] The three-dimensional data encoder 113 may include two mesh data encoders as the mesh data encoder 142. For example, one mesh data encoder encodes vertex information and connectivity information of a static three-dimensional mesh, and the other mesh data encoder encodes vertex information and connectivity information of a dynamic three-dimensional mesh.

[0163] Correspondingly, two mesh files may then be included in the bitstream, for example one mesh file corresponding to a static 3D mesh and another mesh file corresponding to a dynamic 3D mesh.

[0164] Furthermore, the static three-dimensional mesh may be a three-dimensional mesh of an intraframe coded using intraprediction, and the dynamic three-dimensional mesh may be a three-dimensional mesh of an interframe coded using interprediction. Furthermore, information on the dynamic three-dimensional mesh may be differential information between vertex information or connectivity information of the three-dimensional mesh of an intraframe and vertex information or connectivity information of the three-dimensional mesh of an interframe.

[0165] Fig. 23 is a conceptual diagram showing a specific example of the decoding process according to this embodiment. Fig. 23 shows a three-dimensional data decoder 213, a description decoder 248, and a renderer 247. In this example, the three-dimensional data decoder 213 includes a two-dimensional data decoder 241, a mesh data decoder 242, and a mesh reconstructor 246. The two-dimensional data decoder 241 includes a texture decoder 243. The mesh data decoder 242 includes a vertex information decoder 244 and a connectivity information decoder 245.

[0166] The vertex information decoder 244, the connection information decoder 245, the texture decoder 243, and the mesh reconstructor 246 may correspond to the vertex information decoder 201, the connection information decoder 202, the attribute information decoder 203, and the post-processor 205 in Fig. 8. The presenter 247 may correspond to the presenter 215 in Fig. 12.

[0167] For example, the two-dimensional data decoder 241 operates as a texture decoder 243, and decodes the texture corresponding to the attribute information from the texture file as two-dimensional data in accordance with an image coding method or a video coding method.

[0168] The mesh data decoder 242 also operates as a vertex information decoder 244 and a connectivity information decoder 245 to decode vertex information and connectivity information from the mesh file. The mesh data decoder 242 may further decode mapping information for textures from the mesh file.

[0169] The description decoder 248 also decodes descriptions corresponding to metadata such as text data from the description file. The description decoder 248 may decode the descriptions at the system layer. For example, the description decoder 248 may be included in the system demultiplexer 214 of FIG. 12 .

[0170] The mesh reconstructor 246 reconstructs a 3D mesh from the vertex information, connectivity information, and textures according to the description. The renderer 247 renders and outputs the 3D mesh according to the description.

[0171] Through the above operations, a 3D mesh is reconstructed and output from a bitstream containing a texture file, a mesh file, and a description file.

[0172] The three-dimensional data decoder 213 may include two mesh data decoders as the mesh data decoder 242. For example, one mesh data decoder decodes vertex information and connectivity information of a static three-dimensional mesh, and the other mesh data decoder decodes vertex information and connectivity information of a dynamic three-dimensional mesh.

[0173] Correspondingly, two mesh files may then be included in the bitstream, for example one mesh file corresponding to a static 3D mesh and another mesh file corresponding to a dynamic 3D mesh.

[0174] Furthermore, the static three-dimensional mesh may be a three-dimensional mesh of an intraframe coded using intraprediction, and the dynamic three-dimensional mesh may be a three-dimensional mesh of an interframe coded using interprediction. Furthermore, information on the dynamic three-dimensional mesh may be differential information between vertex information or connectivity information of the three-dimensional mesh of an intraframe and vertex information or connectivity information of the three-dimensional mesh of an interframe.

[0175] A dynamic 3D mesh coding method is sometimes called DMC (Dynamic Mesh Coding), and a video-based dynamic 3D mesh coding method is sometimes called V-DMC (Video-based Dynamic Mesh Coding).

[0176] The point cloud encoding method is sometimes called PCC (Point Cloud Compression). The point cloud video-based encoding method is sometimes called V-PCC (Video-based Point Cloud Compression). The point cloud geometry-based encoding method is sometimes called G-PCC (Geometry-based Point Cloud Compression).

[0177] <Implementation Example> Fig. 24 is a block diagram showing an implementation example of the encoding device 100 according to this embodiment. The encoding device 100 includes a circuit 151 and a memory 152. For example, multiple components of the encoding device 100 shown in Fig. 5 etc. are implemented by the circuit 151 and memory 152 shown in Fig. 24.

[0178] The circuit 151 is a circuit that performs information processing and is a circuit that can access the memory 152. For example, the circuit 151 is a dedicated or general-purpose electric circuit that encodes a three-dimensional mesh. The circuit 151 may be a processor such as a CPU. Alternatively, the circuit 151 may be a collection of multiple electric circuits.

[0179] The memory 152 is a dedicated or general-purpose memory that stores information used by the circuit 151 to encode the three-dimensional mesh. The memory 152 may be an electric circuit and may be connected to the circuit 151. The memory 152 may also be included in the circuit 151. The memory 152 may also be a collection of multiple electric circuits. The memory 152 may also be a magnetic disk, an optical disk, or the like, and may also be expressed as a storage, a recording medium, or the like. The memory 152 may also be a non-volatile memory or a volatile memory.

[0180] For example, the memory 152 may store a three-dimensional mesh or a bitstream, or may store a program for the circuit 151 to encode the three-dimensional mesh.

[0181] Note that the encoding device 100 does not necessarily have to implement all of the components shown in Figure 5 and the like, and does not necessarily have to perform all of the processes shown here. Some of the components shown in Figure 5 and the like may be included in another device, and some of the processes shown here may be executed by another device. Furthermore, the encoding device 100 may implement any combination of the components of the present disclosure, and may perform any combination of the processes of the present disclosure.

[0182] Fig. 25 is a block diagram showing an example implementation of a decoding device 200 according to this embodiment. The decoding device 200 includes a circuit 251 and a memory 252. For example, multiple components of the decoding device 200 shown in Fig. 7 and other figures are implemented by the circuit 251 and memory 252 shown in Fig. 25.

[0183] The circuit 251 is a circuit that performs information processing and is a circuit that can access the memory 252. For example, the circuit 251 is a dedicated or general-purpose electric circuit that decodes a three-dimensional mesh. The circuit 251 may be a processor such as a CPU. Alternatively, the circuit 251 may be a collection of multiple electric circuits.

[0184] The memory 252 is a dedicated or general-purpose memory that stores information for the circuit 251 to decode the 3D mesh. The memory 252 may be an electric circuit and may be connected to the circuit 251. The memory 252 may also be included in the circuit 251. The memory 252 may also be a collection of multiple electric circuits. The memory 252 may also be a magnetic disk, an optical disk, or the like, and may also be expressed as a storage, a recording medium, or the like. The memory 252 may also be a non-volatile memory or a volatile memory.

[0185] For example, the memory 252 may store a three-dimensional mesh or a bitstream, or may store a program for the circuit 251 to decode the three-dimensional mesh.

[0186] Note that the decoding device 200 does not necessarily have to implement all of the components shown in Figure 7 and the like, and does not necessarily have to perform all of the processes shown here. Some of the components shown in Figure 7 and the like may be included in another device, and some of the processes shown here may be executed by another device. Furthermore, the decoding device 200 may implement any combination of the components of the present disclosure, and may perform any combination of the processes of the present disclosure.

[0187] The encoding method and the decoding method including the steps performed by each component of the encoding device 100 and the decoding device 200 of the present disclosure may be executed by any device or system. For example, part or all of the encoding method and the decoding method may be executed by a computer including a processor, a memory, an input / output circuit, etc. In this case, the encoding method and the decoding method may be executed by the computer executing a program for causing the computer to execute the encoding method and the decoding method.

[0188] Alternatively, the program or the bitstream may be recorded on a non-transitory computer-readable recording medium such as a CD-ROM.

[0189] An example of a program may be a bitstream. For example, a bitstream including an encoded three-dimensional mesh includes syntax elements for causing the decoding device 200 to decode the three-dimensional mesh. The bitstream then causes the decoding device 200 to decode the three-dimensional mesh according to the syntax elements included in the bitstream. Thus, the bitstream may play a role similar to that of a program.

[0190] The bitstream may be an encoded bitstream containing the encoded 3D mesh, or may be a multiplexed bitstream containing the encoded 3D mesh and other information.

[0191] Furthermore, each component of the encoding device 100 and the decoding device 200 may be configured with dedicated hardware, general-purpose hardware that executes the above-mentioned programs, or a combination of these. The general-purpose hardware may be configured with a memory in which the programs are recorded and a general-purpose processor that reads and executes the programs from the memory. Here, the memory may be a semiconductor memory or a hard disk, and the general-purpose processor may be a CPU.

[0192] Furthermore, the dedicated hardware may be configured with a memory, a dedicated processor, etc. For example, the dedicated processor may execute the encoding method and the decoding method by referring to a memory for recording data.

[0193] Furthermore, as described above, each component of the encoding device 100 and the decoding device 200 may be an electric circuit. These electric circuits may form a single electric circuit as a whole, or each may be a separate electric circuit. Furthermore, these electric circuits may correspond to dedicated hardware, or may correspond to general-purpose hardware that executes the above-mentioned programs, etc. Furthermore, the encoding device 100 and the decoding device 200 may be implemented as an integrated circuit.

[0194] The encoding device 100 may also be a transmitting device that transmits a three-dimensional mesh, and the decoding device 200 may also be a receiving device that receives a three-dimensional mesh.

[0195] Displacement Encoding and Decoding The following terminology is used here by way of example:

[0196] (1) Image An image is a data unit made up of a set of pixels, and includes a picture or a block smaller than a picture. Images include both moving images and still images.

[0197] (2) Picture A picture is a unit of image processing that is made up of a set of pixels, and is also called a frame or field.

[0198] (3) Block A block is a processing unit consisting of a specific number of pixels. The term shown in the following example is also used for a block. The shape of a block is not particularly limited. A block may be, for example, a rectangular shape of M×N pixels or a square shape of M×M pixels. A block may also be a triangular shape, a circular shape, or another shape. Examples of blocks are as follows:

[0199] Slice, tile, or brick CTU, superblock, or basic division unit VPDU, processing division unit for hardware CU, processing block unit, prediction block unit (PU), or orthogonal transform block unit (TU) Sub-block

[0200] (4) Pixel or Sample A pixel or sample is the smallest point of an image, in other words, the smallest unit. Pixels or samples include not only pixels at integer positions, but also pixels at sub-pixel positions generated based on pixels at integer positions.

[0201] (5) Pixel Value or Sample Value: A pixel value or sample value is a unique value of a pixel. The pixel value or sample value may include a luma value, a chroma value, or an RGB gradation level, and may also include a depth value or a binary value of 0 or 1.

[0202] (6) Flags A flag indicates one or more bits. A flag is, for example, a parameter or index represented by two or more bits. A flag may indicate not only a value represented by a binary number, but also a value represented by a number other than a binary number.

[0203] (7) Signal: A signal is something that is symbolized or coded to transmit information. A signal includes a discrete digital signal or a continuous analog signal.

[0204] (8) Stream or Bit Stream A stream or bit stream is a digital data sequence that indicates the flow of digital data. A stream or bit stream may be a single stream, or may be configured to include multiple streams with multiple layers. A stream or bit stream may be transmitted by serial communication using a single transmission path, or may be transmitted by packet communication using multiple transmission paths.

[0205] (9) Difference: For scalar quantities, the difference can include simple difference (x-y) and difference calculations. The difference can include absolute difference (|x-y|), squared difference (x^2-y^2), square root difference (√(x-y)), weighted difference (ax-by, where a and b are constants), or offset difference (x-y+a, where a is an offset).

[0206] (10) Sum: For scalar quantities, sums can include simple sum (x + y) and addition calculations. The sum can also include absolute sum (|x + y|), sum of squares (x^2 + y^2), square root of the sum (√(x + y)), weighted sum (ax + by, where a and b are constants), or offset sum (x + y + a, where a is an offset).

[0207] (11) "Based on" The expression "based on something" means that something other than that "something" may be taken into consideration. Also, "based on" can be used both when a direct result is obtained and when a result is obtained through an intermediate result.

[0208] (12) "Used" or "Using" The phrases "something was used" or "used something" mean that something other than the "something" may be taken into consideration. The phrases "used" or "used" may be used both in cases where a direct result is obtained and in cases where a result is obtained via an intermediate result.

[0209] (13) Prohibition "Prohibit" can be rephrased as "not permitted." Also, "not prohibited / prohibited" or "permitted / permitted" does not necessarily mean "obligation."

[0210] (14) "Restriction" or "Limitation" "Restriction" or "Limitation" can be rephrased as "not permitted / not allowed" or "not permitted / permitted." Furthermore, "prohibited / not prohibited" or "not permitted / permitted" does not necessarily mean "obligation." Furthermore, what is prohibited quantitatively or qualitatively may be either partial or total.

[0211] (15) Chroma The term chroma is an adjective, represented by the symbols Cb or Cr, that indicates that a sample array or a single sample represents one of the two color difference signals associated with a primary color. The term chroma is sometimes used instead of the term chrominance.

[0212] (16) Luma The term luma is an adjective, denoted by the symbols or subscripts Y or L, that indicates that a sample array or a single sample represents a monochrome signal for a primary color. The term luma is sometimes used instead of the term luminance.

[0213] The encoding / decoding system of this embodiment will be described below.

[0214] A typical three-dimensional model (also called a 3D model) digitally represents an object so that a user can explore the model using zoom, pan, and rotation in all three dimensions while it is rendered over time. One way to construct such a representation is to build a 3D mesh using triangles. The model stores the positions of the triangle vertices, their connectivity to each other, and their associated attributes (such as normals or UV patches).

[0215] Storing all this information in uncompressed form requires a very large storage space and therefore a very large bandwidth for transmission. The triangles that form the mesh often have repeating patterns and similar properties, especially in temporal and spatial neighborhoods. These repetitions can be exploited to develop efficient encoding and decoding methods for storage and transmission. One such encoding and decoding method is Video-based Dynamic Mesh Coding (V-DMC).

[0216] 26 is a block diagram showing another example of the configuration of the encoding / decoding system according to this embodiment. As shown in FIG. 26, the encoding / decoding system includes an encoding device 100 and a decoding device 200.

[0217] The encoding / decoding system accepts input three-dimensional meshes (also called 3D meshes) in the form of three-dimensional coordinates of vertices (vertex information), connectivity (connection information) and associated attributes (attribute information), which may include texture maps as well as geometry.

[0218] The encoding device 100 takes an input 3D mesh (also referred to as an input 3D mesh or input mesh) in the form of 3D coordinates of vertices, connectivity, and associated attributes. The encoding device 100 encodes all associated information into a stream. The stream may consist of a single bitstream or multiple bitstreams.

[0219] The network 300 transmits the stream generated by the encoding device 100 to the decoding device 200. The network 300 may be the Internet, a wide area network (WAN), a local area network (LAN), or any combination thereof. Furthermore, the network 300 is not necessarily limited to a two-way communication network, but may also be a one-way communication network that transmits broadcast waves such as terrestrial digital broadcasting or satellite broadcasting. Instead of the network 300, a recording medium such as a digital versatile disc (DVD) or a blue-ray disc (BD) on which a stream is recorded may be used.

[0220] The stream is transmitted to a decoding device 200 via a network 300. The decoding device 200 decodes the bitstream and generates a 3D mesh using the 3D coordinates, connectivity, and associated attributes of the decoded vertices. The decoding device 200 outputs the generated 3D mesh (also referred to as an output 3D mesh or output mesh).

[0221] FIG. 27 is a diagram showing another example of the configuration of the encoding device 100.

[0222] As shown in FIG. 27, the encoding device 100 includes a preprocessor 1103 and a compressor 1106 .

[0223] The encoding device 100 reads an input mesh 1101 and an attribute map 1102 and passes them to a preprocessor 1103. The preprocessor 1103 processes the input mesh to extract a base mesh 1104 and displacement data 1105. The attribute map 1102, along with the extracted base mesh 1104 and displacement data 1105, are passed to a compressor 1106.

[0224] The compressor 1106 also compresses the base mesh 1104, the displacement data 1105, and the attribute map 1102 to generate a bitstream 1107. The compressor 1106 can transmit additional information to the decoding device 200 by further including metadata 1108 in the bitstream 1107.

[0225] FIG. 28 is a diagram showing another example of the configuration of the decoding device 200.

[0226] As shown in FIG. 28, the decoding device 200 includes a decompressor 2102 and a post-processor 2106 .

[0227] The decoding device 200 reads a bitstream 2101 and passes it to a decompressor 2102. The decompressor 2102 decompresses a base mesh 2103, displacement data 2104, and an attribute map 2108 from the bitstream 2101 and passes them to a post-processor 2106. An example of the displacement data 2104 is a displacement vector.

[0228] The post-processor 2106 also processes the base mesh 2103 according to the displacement data 2104 and the attribute map 2108 to generate an output mesh 2107. The post-processor 2106 may further use information from the metadata 2105 to generate the output mesh 2107.

[0229] FIG. 29 is a block diagram showing yet another example configuration of the encoding device 100 according to this embodiment.

[0230] In this example, the encoding device 100 comprises a volumetric capturer 511, a projector 512, a base mesh encoder 513, a displacement encoder 514, an attribute encoder 515, and optionally one or more other type encoders 516.

[0231] The volumetric capturer 511 captures content and outputs the captured content to the projector 512 .

[0232] The projector 512 projects the content onto an input mesh (a 3D mesh frame) that includes geometry coordinates (vertex coordinates indicating the positions of the vertices), texture coordinates, and connectivity (connectivity information). The data is output to a base mesh encoder 513, a displacement encoder 514, an attribute encoder 515, and optionally one or more other type encoders 516. Each encoder compresses the data into a bitstream.

[0233] FIG. 30 is a block diagram showing yet another example configuration of the decoding device 200 according to this embodiment.

[0234] In this example, the decoding device 200 comprises a base mesh decoder 613 , a displacement decoder 614 , an attribute decoder 615 , one or more other type decoders 616 , and a 3D reconstructor 617 .

[0235] The bitstream is sent to a base mesh decoder 613, a displacement decoder 614, an attribute decoder 615, and optionally one or more other type decoders 616. These decoders decode the bitstream to generate data (decoded data) including geometry coordinates, texture coordinates, connectivity, etc. The decoded data is then sent to a 3D reconstructor 617, which reconstructs an output mesh (a 3D mesh frame).

[0236] The encoding process performed by the encoding device 100 will be described in detail below.

[0237] Fig. 31 is a flow diagram showing the processing of the encoding device 100. Fig. 32 is an explanatory diagram conceptually showing the encoding of mesh frames. The processing of the encoding device 100 will be described with reference to Figs. 31 and 32.

[0238] In step S101, the encoding device 100 reads a 3D mesh frame, which is an input mesh frame, and its attributes. The input mesh frame is a mesh frame input to the encoding device 100. An example of the 3D mesh frame that is an input mesh frame is shown as mesh frame 1301 (see FIG. 32 ).

[0239] In step S102, the encoding device 100 performs a decimation process on the input mesh frame read in step S101 to generate a base mesh frame having fewer vertices than the input mesh frame. The base mesh frame generated by decimating the mesh frame 1301 is shown as a base mesh frame 1302 (see FIG. 32).

[0240] In step S103, the encoding device 100 calculates displacement information that the decoding device 200 uses to reconstruct a mesh frame. The displacement information corresponds to a displacement vector directed from a vertex of the base mesh frame generated in step S102 to a vertex of the input mesh frame. One method for calculating the displacement information is to subtract the coordinates of the vertex of the base mesh frame from the coordinates of the vertex of the input mesh frame. The displacement information calculated from the mesh frame 1301 and the base mesh frame 1302 is shown as displacement information 1303 (see FIG. 32). The displacement information 1303 is in vector format, in other words, expressed as a displacement vector.

[0241] In step S104, the encoding device 100 encodes the base mesh frame generated in step S102, the displacement information generated in step S103, and the attributes of the input mesh frame into a bitstream (corresponding to a compressed bitstream). An example of the bitstream is shown as bitstream 1304 (see FIG. 32).

[0242] Specifically, the bitstream 1304 includes vertex coordinates and connectivity information for vertices A, C, E, and F, displacement information, a video bitstream including texture data, and a compressed attribute map (see FIG. 32). The displacement information includes displacement information for displacing vertices based on vertex coordinates obtained from the subdivided base mesh frame. The compressed attribute map is texture coordinates for applying texture data to a mesh frame reconstructed using the base mesh frame and the displacement information.

[0243] The decoding process performed by the decoding device 200 will be described in detail below.

[0244] Fig. 33 is a flow diagram showing the processing of the decoding device 200. Fig. 34 is an explanatory diagram conceptually showing the decoding of a mesh frame (3D mesh). The processing of the decoding device 200 will be described with reference to Figs. 33 and 34.

[0245] In step S201, the decoding device 200 decodes a base mesh frame and attributes from a bitstream (corresponding to a compressed bitstream). An example of the decoded base mesh frame (corresponding to a decoded base mesh frame) is shown as a decoded base mesh frame 2301 (see FIG. 34).

[0246] In step S202, the decoding device 200 generates subdivided vertices by performing a subdivision process on the base mesh frame decoded in step S201. An example of a base mesh frame (mesh frame) including subdivided vertices is shown as a base mesh frame 2302 (see FIG. 34).

[0247] In step S203, the decoding device 200 decodes the disparity information from the bitstream (corresponding to the compressed bitstream). An example of the decoded disparity information is shown as disparity information 2303 (see FIG. 34). The disparity information 2303 is in vector format, in other words, expressed as a disparity vector.

[0248] In step S204, the decoding device 200 reconstructs the shape of the mesh frame by moving the vertices of the base mesh frame, including the subdivided vertices, to new positions using the displacement information, and then restores the mesh frame by applying attribute information. An example of the attribute is texture. An example of the reconstructed mesh frame is shown as mesh frame 2304 (see FIG. 34 ).

[0249] FIG. 35 is a block diagram showing an example of the configuration of a decoding device according to this embodiment.

[0250] FIG. 35 shows an example of a block diagram of a general intra-decoding system.

[0251] The decoding device shown in FIG. 35 comprises a demultiplexer 1231, a switch 1232, a static mesh decoder 1233, a mesh buffer 1234, a motion decoder 1235, a base mesh reconstructor 1236, an inverse quantizer 1237, a video decoder 1238, an image unpacker 1239, an inverse quantizer 1240, an inverse wavelet transformer 1241, a reconstructor 1242, a video decoder 1243, and a color converter 1244.

[0252] The demultiplexer 1231 receives the compressed bitstream and separates it into compressed data for the base mesh, video containing displacement data (also called displacement bitstream), and video containing attribute data (also called attribute bitstream). The compressed data for the base mesh is passed to a switch 1232. The switch 1232 determines whether to perform intra-decoding or inter-decoding based on parameters in the bitstream.

[0253] If an intra-decoding process is selected, the bitstream is passed to a static mesh decoder 1233, which generates a quantized base mesh. The static mesh decoder 1233 is, for example, a decoder that uses an edge breaker algorithm to decode 3D mesh data. The static mesh decoder 1233 generates a quantized base mesh from the bitstream. The quantized base mesh generated by the static mesh decoder 1233 is stored in a mesh buffer 1234 for reference when an inter-decoding process is selected.

[0254] If inter-decoding is selected, switch 1232 passes compressed data for the base mesh to motion decoder 1235. Motion decoder 1235 receives a previously decoded quantized base mesh and decodes motion data representing the differences in vertex coordinates between the quantized base mesh stored in mesh buffer 1234 and the current quantized base mesh. The motion data and the quantized base mesh stored in mesh buffer 1234 are used by base mesh reconstructor 1236 to reconstruct the current quantized base mesh. The quantized base mesh resulting from either inter-decoding or intra-decoding is passed to inverse quantizer 1237 to obtain a decoded base mesh.

[0255] The video containing the displacement data is passed to a video decoder 1238, since the bitstream contains the displacement data in an image format with two chroma information and one luma information. The video decoder 1238 decodes the data using a video frame decompression method. Alternatively, the displacement data can be decoded using an arithmetic decoder. This decompressed data is passed to an image unpacker 1239, which extracts wavelet coefficients associated with each vertex from the image-format decompressed data. An inverse quantizer 1240 dequantizes the quantized wavelet coefficients into the three components associated with each vertex. An inverse wavelet transformer 1241 inversely transforms the result to finally obtain decoded displacement data. The decoded displacement data and the decoded base mesh are passed to a reconstructor 1242, which performs edge refinement on the decoded base mesh and displaces the vertices using the decoded displacement data to obtain a decoded mesh.

[0256] The video containing the attribute data is passed to another video decoder 1243 to obtain a decoded attribute bitstream, which is further processed in a color converter 1244 for color space and color format conversion to obtain a decoded attribute map.

[0257] FIG. 36 is a block diagram showing an example of the configuration of an encoding device according to this embodiment.

[0258] First, the encoding device obtains the base mesh bitstream, the displacement bitstream, and the attribute bitstream resulting from the 3D mesh preprocessing step.

[0259] The encoding device shown in FIG. 36 comprises a quantizer 1261, a switch 1262, a static mesh encoder 1263, a mesh buffer 1264, a motion encoder 1265, a base mesh reconstructor 1266, a displacement data updater 1267, a wavelet transformer 1268, a quantizer 1269, an image packer 1270, a video encoder 1271, a color converter 1272, and a video encoder 1273.

[0260] The base mesh (specifically, the position information of the multiple vertices that make up the base mesh) is first quantized by a quantizer 1261. The quantized base mesh (base mesh data) is output to a switch 1262, which determines whether intra-coding or inter-coding is to be performed. If intra-coding is selected, the quantized base mesh (base mesh bitstream) is output to a static mesh encoder 1263, which generates a quantized base mesh. An example of the static mesh encoder 1263 is an encoder that uses an edge breaker algorithm to encode 3D mesh data. This encoded (quantized) base mesh is stored in a mesh buffer 1264 for reference when inter-coding is selected. If inter-coding is selected, the switch 1262 outputs compressed data related to the base mesh to a motion encoder 1265. The motion data and static 3D mesh in the mesh buffer 1264 are used by a base mesh reconstructor 1266 to reconstruct the currently quantized base mesh.

[0261] The displacement data is output to a displacement data updater 1267, where it is updated based on the quantized static base mesh or the reconstructed inter-coded base mesh. Next, a wavelet transformer 1268 performs a transformation process, followed by quantization in a quantizer 1269. The quantized displacement data is packed into an image in an image packer 1270, and finally encoded in a video coder 1271. The encoded displacement data is output to a multiplexer 1274.

[0262] The attribute information (e.g., attribute map) is output to a color converter 1272 for color space and color format conversion, and the converted attribute information is coded by a video coder 1273 and output to a multiplexer 1274.

[0263] The multiplexer 1274 acquires data related to the encoded base mesh (compressed data related to the base mesh), video data including encoded displacement data, and video data including attribute information such as an encoded attribute map, and generates a bitstream (compressed bitstream) including these acquired data. The generated compressed bitstream is output to, for example, a decoding device.

[0264] FIG. 37 is a block diagram showing an example of the configuration of a decoding device according to this embodiment.

[0265] FIG. 37 illustrates an example of a reconstructor that obtains a decoded 3D mesh 1256 from a decoded base mesh 1251 and decoded displacement data 1254.

[0266] The decoded base mesh 1251 is passed to a subdivider 1252 .

[0267] The subdivision unit 1252 subdivides any two connected vertices in the entire 3D mesh by adding a new vertex between them. This process can be repeated several times to include vertices created in previous subdivision steps to generate a predefined number of vertices. Each subdivision iteration across the 3D mesh generates a new level of detail (LoD). The subdivided mesh 1253 and the decoded displacement data 1254 are passed to a displacer 1255. The displacer 1255 generates a decoded 3D mesh 1256 by moving each vertex to a new position according to the corresponding displacement data.

[0268] The subdivision is described below and is performed, for example, by subdivision unit 1252.

[0269] FIG. 38 is an explanatory diagram showing an example of subdivision.

[0270] The base mesh shown in FIG. 38(a) includes vertices A, B, and C and connectivity information indicating their connectivity.

[0271] 38(b) shows a mesh generated by the first subdivision, in other words, the mesh after the first subdivision. In the first subdivision, the subdivider generates vertices D, E, and F and connectivity information indicating their connectivity. The mesh generated by the subdivider is also referred to as LoD1 or first LoD.

[0272] Vertex D of the mesh after the first subdivision is a vertex generated by subdivision based on vertices A and B. Similarly, vertex F is a vertex generated by subdivision based on vertices B and C. Vertex E is a vertex generated by subdivision based on vertices A and C.

[0273] As an example, vertex D may be the midpoint of line segment AB (in other words, side AB) connecting vertices A and B that were the basis for its generation. Similarly, vertex E may be the midpoint of line segment AC. Vertex F may be the midpoint of line segment BC.

[0274] 38(c) shows the mesh generated by the second subdivision, i.e., the mesh after the second subdivision. In the second subdivision, the subdivider generates vertices G, H, I, J, K, L, M, N, and O and connectivity information indicating their connectivity. The mesh generated by the subdivider is also called LoD2 or second LoD.

[0275] Vertex G of the mesh after the second subdivision is a vertex generated by subdivision based on vertices A and D. Similarly, vertex H is a vertex generated by subdivision based on vertices A and E. Vertex I is a vertex generated by subdivision based on vertices B and D. Vertex J is a vertex generated by subdivision based on vertices D and F. Vertex K is a vertex generated by subdivision based on vertices E and F. Vertex L is a vertex generated by subdivision based on vertices C and E. Vertex M is a vertex generated by subdivision based on vertices B and F. Vertex N is a vertex generated by subdivision based on vertices C and F. Vertex O is a vertex generated by subdivision based on vertices D and E.

[0276] As an example, vertex G may be the midpoint of line segment AD (in other words, side AD) connecting vertices A and D, which were the source of its generation. Similarly, vertex H may be the midpoint of line segment AE. vertex I may be the midpoint of line segment BD. vertex J may be the midpoint of line segment DF. vertex K may be the midpoint of line segment EF. vertex L may be the midpoint of line segment CE. vertex M may be the midpoint of line segment BF. vertex N may be the midpoint of line segment CF. vertex O may be the midpoint of line segment DE.

[0277] The displacement of vertices will be described below with reference to Figures 39 and 40. The displacement of vertices is performed by the reconstructor.

[0278] Fig. 39 is an explanatory diagram showing an example of displacement of vertices after subdivision, and Fig. 40 is an explanatory diagram showing an example of vertices of an original mesh.

[0279] The base mesh shown in FIG. 39(a) includes vertices A, B, C, and Z, and connectivity information indicating their connectivity.

[0280] 39(b) shows a mesh generated by the first subdivision, in other words, a mesh after the first subdivision (i.e., the first LoD). In the first subdivision, the subdivider generates vertices S, T, U, X, or Y and connectivity information indicating their connectivity. The vertices S, T, U, X, or Y are similar to the vertices D, E, and F shown in FIG. 38(b).

[0281] 39(c) shows a mesh generated by the second subdivision, in other words, a mesh after the second subdivision (i.e., the second LoD). In the second subdivision, the subdivider generates vertices D, E, F, G, and H and connectivity information indicating their connectivity. Vertices D, E, F, G, and H are similar to vertices G, H, I, J, K, L, M, N, and O shown in FIG. 38(c).

[0282] Figure 39(d) shows a mesh including the vertices after they have been displaced after subdivision, with vertices A, B, C, D, E, F, G, H, S, T, U, X, Y, and Z shown in Figure 39(d) being located at positions displaced using displacement information from the positions of the vertices shown in Figure 39(c).

[0283] The original mesh shown in FIG. 40 is an example of the mesh input to the encoding device 100, that is, the mesh before encoding.

[0284] The mesh shown in Fig. 39 has a shape similar to that of the original mesh shown in Fig. 40. The displacement information is generated by the displacement vector calculator 1207 of the encoding device 100 as information indicating the displacement from the vertices of the base mesh to the vertices of the original mesh, and therefore, by reconstructing the mesh using the displacement information thus generated, a mesh having a shape similar to that of the original mesh is generated.

[0285] The decoding device 200 can output the mesh shown in FIG.

[0286] Next, the division of a mesh into sub-meshes will be described with reference to FIGS.

[0287] A mesh can be divided into smaller parts and coded separately, with the vertices of the mesh being divided in such a way that the coordinates and connectivity of the vertices in each part can be coded independently.

[0288] Fig. 41 is an explanatory diagram showing an example of a mesh, and Fig. 42 is an explanatory diagram showing an example of dividing a mesh into sub-meshes.

[0289] The mesh shown in FIG. 41 is the original mesh, which is sometimes called a full mesh in contrast to a sub-mesh.

[0290] Figure 42 shows how the full mesh shown in Figure 41 is divided into two sub-meshes. For vertices A, B, and C of the full mesh (see Figure 41), vertex A is duplicated to vertices A1 and A2, vertex B is duplicated to vertices B1 and B2, and vertex C is duplicated to vertices C1 and C2, thereby creating two sub-meshes (i.e., a first sub-mesh and a second sub-mesh) from the full mesh. The first sub-mesh and the second sub-mesh are each independently decodable meshes.

[0291] Packing of displacement information into image frames will be described below with reference to FIGS. 43, 44 and 45.

[0292] 43, 44, and 45 are explanatory diagrams showing examples of packing displacement information into image frames. Note that image frames can also be called video frames.

[0293] The vertex displacement data is encoded as image frame data by being mapped to each component of a YUV format image frame (i.e., each of the Y component (Y Plane), U component (U Plane), and V component (V Plane)). This case will be described below as an example. As another example, the vertex displacement data may be encoded as image frame data by being mapped to each component of an RGB format image frame (each of the R component, G component, and B component).

[0294] The decoding device 200 can use an image encoding module to extract the displacement data. The displacement data can be in the form of X, Y, or Z components in a global coordinate system (e.g., a Cartesian coordinate system), or normal, tangential, or both tangential components in a local coordinate system. Methods for mapping the displacement data to an image frame include the following:

[0295] For example, in the first method, the displacement data is arranged in the image frame in scan order. An example of packing the displacement data in this case is shown in Figure 43. The displacement data is directly mapped to the image frame according to a predefined scan order.

[0296] Note that since an image frame has a fixed height and width, it may happen that the displacement data does not fit perfectly in the frame, in which case the remaining part of the image frame is padded with padding data (see Figure 43).

[0297] For example, in the second method, the displacement data is separated into multiple LoDs and mapped to the Y, U, and V components of the image frame. An example of packing of the displacement data in this case is shown in Figure 44. Here, the displacement data of the image frame of the next LoD starts immediately after the displacement data of the previous LoD ends. As in the first method, if the displacement data does not fit exactly into the image frame, padding is performed at the end of the image frame (see Figure 44).

[0298] For example, in the third method, displacement data corresponding to the LoD is mapped to the Y component, U component, and V component of the image frame in a manner different from that in the second method. An example of packing of the displacement data in this case is shown in Figure 45. In this way, each LoD can be decoded independently. In the third method, middle padding is performed on the displacement data of each LoD, and CTU alignment is performed together with padding at the end of the video frame (see Figure 45).

[0299] Next, the encoding device 100 and the decoding device 200 when a mesh is divided into a plurality of sub-meshes will be described.

[0300] Fig. 46 is a diagram showing another example of the configuration of the encoding device 100 according to an embodiment. Specifically, Fig. 46 is a diagram showing the configuration of a submesh encoding device that performs encoding processing when an input mesh 1101 is divided into divided meshes (plurality of submeshes). For example, the submesh encoding device includes a plurality of encoding devices 100.

[0301] An input mesh 1101 (full mesh) input to the submesh encoding device is divided into multiple meshes (submeshes). The multiple submeshes are input to, for example, multiple encoding devices 100. Each of the multiple submeshes may be input to one of the multiple encoding devices 100. For example, the submesh encoding device divides the input mesh 1101 into multiple submeshes and inputs the divided multiple submeshes to the multiple encoding devices 100.

[0302] After the image is divided into a plurality of sub-meshes, coding processing is performed on the boundaries of the sub-meshes (processing of overlapping sub-meshes).

[0303] For example, for each sub-mesh, preprocessing is performed by the preprocessor 1103, and a base mesh, displacement data, and metadata are generated and encoded.

[0304] The encoding device 100 may be realized by such a submesh encoding device configuration. That is, the encoding device 100 may be configured to include a plurality of preprocessors 1103 and compressors 1106, and may perform a predetermined process on the submesh for each of a plurality of pairs of preprocessors 1103 and compressors 1106. Furthermore, the number of pairs of preprocessors 1103 and compressors 1106 included in the encoding device 100 may be any number and is not particularly limited.

[0305] FIG. 47 is a diagram illustrating an example of the configuration of the preprocessor 1103 according to the embodiment.

[0306] The preprocessor 1103 includes, for example, a base mesh generator 1401 , a subdivision unit 1402 , and a displacement data generator 1403 .

[0307] First, in the preprocessor 1103, a base mesh is generated by the base mesh generator 1401.

[0308] The base mesh is then subdivided in a predetermined manner by a subdivider 1402 to generate a subdivided mesh (or a subdivided base mesh) that is the subdivided base mesh.

[0309] Displacement data is generated by a displacement data generator 1403 from the sub-divided mesh and the sub-mesh that is the input mesh 1101 after division.

[0310] The displacement data is, for example, a difference vector between the input mesh 1101 and the sub-division mesh.

[0311] The same subdivision method as that used in encoding is used in decoding as well.

[0312] Furthermore, for example, the encoding device 100 may transmit to the decoding device 200 a subdivision method for encoding and parameters used in the subdivision.

[0313] Fig. 48 is a diagram showing another example of the configuration of the decoding device 200 according to the embodiment. Specifically, Fig. 48 is a diagram showing the configuration of a submesh decoding device, which is a device that performs decoding processing when a bitstream 2101 includes multiple submeshes. For example, the submesh decoding device includes multiple decoding devices 200. For example, the submesh decoding device includes multiple decoding devices 200 and a combiner 2109.

[0314] The coded data for each submesh included in the bit stream 2101 is input to the decompressor 2102 of each decoding device 200 .

[0315] Also, for example, the post-processor 2106 performs the processing of the reconstructor described above.

[0316] In the post-processor 2106, for each decoded sub-mesh, the base mesh is subdivided and the displacement vector is added to the subdivided base mesh to reconstruct the sub-mesh.

[0317] That is, the post-processor 2106 performs the above-described reconstructor processing for each sub-mesh.

[0318] The combiner 2109 combines (merges) the sub-meshes restored by the respective decoding devices 200 to reconstruct the full mesh (output mesh 2107) before division.

[0319] Note that the decoding device 200 may be realized by such a submesh decoding device configuration. That is, the decoding device 200 may be configured to include a plurality of decompressors 2102 and post-processors 2106, and each of the plurality of pairs of decompressors 2102 and post-processors 2106 may perform a predetermined process on the encoded data for each submesh. Furthermore, the number of pairs of decompressors 2102 and post-processors 2106 included in the decoding device 200 may be any number and is not particularly limited.

[0320] Fig. 49 is a diagram showing a specific example of the configuration of the decoding device 200 according to the embodiment. Specifically, Fig. 49 shows a specific configuration of the post-decoder 2306 out of the decoder 2305 and post-decoder 2306 included in the decoding device 200.

[0321] The post-decoder 2306 comprises a pre-reconstructor 2307 , a reconstructor 2308 , a post-reconstructor 2309 and an adaptor 2310 .

[0322] The processing in the post-decoder 2306 is optional depending on the application. An example of the processing in the post-decoder 2306 (post-decoding processing) is conversion of the decoded data to a nominal format, such as video conversion from YUV space to RGB space. The post-decoding processing can be encapsulated into multiple processes, such as pre-reconstruction, reconstruction, post-reconstruction, and adaptation.

[0323] The pre-reconstructor 2307 performs a pre-reconstruction process, for example, upscaling the normalized texture coordinates to match the dimensions of the texture image in the context of video-based dynamic mesh coding.

[0324] The reconstructor 2308 performs a reconstruction process, which is invoked, for example, on the decoded atlas frame, the decoded base mesh frame, the decoded video frame, and syntax elements associated with the same mesh sequence. The output of the reconstruction process is a series of reconstructed mesh frames prior to a post-reconstruction process.

[0325] The post-reconstructor 2309 performs post-reconstruction operations, e.g., in the context of video-based dynamic mesh coding, which perform a number of smoothing operations on the reconstructed mesh frames, such as collapsing edges in the mesh or adding new vertices in the mesh.

[0326] The adaptor 2310 performs a fitting process, which may be applied by some application to fit the reconstructed mesh to a given scenario. For example, the vertices of the reconstructed mesh are transformed from the 3D model coordinate system to the 3D world coordinate system. The adaptor 2310 outputs a final reconstructed mesh frame (final 3D mesh frame 2311).

[0327] The details of subdivision are explained below.

[0328] <Subdivision> Figure 50 is a diagram showing an example of two subdivided submeshes having boundary edges according to an embodiment. Specifically, (a) of Figure 50 shows an example of a submesh, and (b) of Figure 50 shows another example of a submesh. (c) of Figure 50 shows a 3D mesh in which the submesh shown in (a) of Figure 50 and the submesh shown in (b) of Figure 50 are merged (i.e., combined).

[0329] The decoding device 200 subdivides each edge of the submesh based on position information of each vertex of the submesh and connection information indicating the connection relationship between the vertices, i.e., information on the multiple edges of the submesh. In the subdivision, for example, new vertices (three-dimensional points) are generated on the edges. The generated vertices are connected by new edges, for example. As a result, for example, each of the multiple faces included in the submesh is subdivided into multiple faces. For example, a triangular face enclosed by three vertices included in the submesh is subdivided into four faces.

[0330] The decoding device 200 performs subdivision for each submesh and merges the resulting submeshes to reconstruct a three-dimensional mesh corresponding to the original mesh.

[0331] Here, the encoding device 100 generates multiple submeshes from an original mesh and encodes, for each submesh, the position information, displacement information, etc., of the vertices constituting each submesh. Therefore, the position information, displacement information, etc., of the vertices constituting each submesh may be encoded using different encoding parameters for each submesh. As a result, when the decoding device 200 performs subdivision based on this encoded information, an edge (also called a boundary edge) shared by two or more submeshes may be subdivision at a different position for each submesh, or the number of times each submesh is subdivision may differ. This may result in a problem in which the submeshes cannot be merged properly.

[0332] For example, in the example shown in Figure 50, side CD is a boundary edge. When the submesh shown in Figure 50(a) is subdivided, for example, vertex F is generated on side CD. Also, when the submesh shown in Figure 50(b) is subdivided, for example, vertex M is generated on side CD. If the positions of vertex F and vertex M are different, or if the number of vertices generated on side CD is different, that is, if the number of times side CD is subdivided is different, when the submesh shown in Figure 50(a) and the submesh shown in Figure 50(b) are merged, gaps will be created around side CD, and the submesh will not be able to merge properly.

[0333] To solve this problem caused by using different subdivision schemes in adjacent submeshes (i.e., submeshes having the same boundary edge), the present application imposes a constraint that all boundary edges have the same type of subdivision scheme and the same number of iterations. That is, for example, when subdividing a boundary edge, the subdivision is performed in each submesh using the same subdivision method and the same number of subdivisions. As a result, if the displacement information (i.e., the value of the displacement vector) of vertex F and vertex M is the same, the vertices generated by the subdivision are displaced, and after the submeshes are merged, there can be no gaps around the boundary edge, as shown in (c) of FIG.

[0334] Note that the subdivision of the boundary edge may be performed using a predetermined subdivision method and / or a predetermined number of subdivisions, or the encoding device 100 may determine the predetermined subdivision method and / or the predetermined number of subdivisions and signal the determined information in the bitstream. Also, for example, the alignment of vertex F with vertex M may be performed when subdividing a submesh or when merging multiple submeshes.

[0335] <Summary of the Disclosure> Next, an example of an overview of the technology obtained from the disclosure of this specification will be given.

[0336] For example, in the decoding method disclosed herein, for multiple submeshes that make up a three-dimensional mesh, position information of vertices included in polygons that make up the submeshes and connection information regarding the connection relationships of the vertices are obtained from the encoded bitstream (i.e., decoding of information), the polygons are generated using the position information and the connection information (i.e., decoding of faces), a determination is made as to whether an edge that makes up the polygon is a boundary of the submesh (i.e., edge condition determination), a division process for the edge is determined based on the determination result (i.e., determination of subdivision process based on the determination result), and the edge is divided (subdivision) using the division process.

[0337] Specific examples and variations of the division process are described below.

[0338] In the division process, a method for dividing the edge (division method) may be specified.

[0339] In the division process, the number of times to divide the edge (number of divisions) may be specified.

[0340] In the division process, a method and number of times to divide the edge (division method and number of divisions) may be specified.

[0341] The division process may be a process of generating a new vertex based on position information of a plurality of vertices that form the edge (another definition of the division process).

[0342] The decoding process may decode parameters specifying the segmentation process from the coded bitstream (segmentation process signaling).

[0343] At least one of the division processes may be defined in advance (predetermined division process).

[0344] The splitting process includes not splitting the edge (no split option).

[0345] Specific examples and modifications of the determination process will be described below.

[0346] In the determination process, it may be determined whether or not one of the sides constituting the polygon to be processed is the boundary of the sub-mesh (determination process for each side).

[0347] The determination process may determine whether or not any of the edges constituting the polygon is a boundary edge of the sub-mesh (determining whether or not the polygon includes a boundary edge).

[0348] The submesh boundary may be an edge that includes at both ends a plurality of vertices that constitute a plurality of submeshes (definition of boundary).

[0349] A specific example of the relationship between the determination result and the division process will be described below.

[0350] In the division process, when the side to be processed is the submesh boundary, the side may be divided using a first division process.

[0351] In the division process, when the side to be processed is not the submesh boundary, the side may be divided using a second division process.

[0352] In the division process, when the polygon includes an edge that is the submesh boundary, the edge that is the submesh boundary may be divided using a first division process, and the edge that is not the submesh boundary may be divided using a second division process.

[0353] In the division process, when the polygon does not include an edge that is the boundary of the sub-mesh, all edges included in the polygon may be divided using a second division process.

[0354] In the division process, when the polygon includes an edge that is the submesh boundary, it may be determined whether the first division process and the second division process have a predetermined relationship, and the division process may be determined based on the determination result. For example, the division process may be switched based on a result of comparing the number of divisions specified by the first division process and the second division process.

[0355] Specific examples of the first division process (boundary division process) and the second division process (non-boundary division process) will be described below.

[0356] The first division process and the second division process may be different processes (different division processes may be selected).

[0357] The first division process may be selected from a first division process group, and the second division process may be selected from a second division process group. The first division process group and the second division process group may include different division processes (selected from a plurality of division processes, with different options).

[0358] The first division process may select the same process common to a plurality of sub-meshes, or the division process may select processes from the same group for a plurality of sub-meshes.

[0359] The first division process may be determined on a sequence-by-sequence or frame-by-frame basis. Furthermore, parameters used in the first division process may be coded into the coded bitstream.

[0360] The second division process may be determined for each sub-mesh. Furthermore, parameters used in the second division process may be coded into the coded bitstream.

[0361] Note that "different processing" may mean processing in which at least one of the number of divisions or the division method is different.

[0362] Also, for example, an encoding device of the present disclosure includes a circuit and a memory connected to the circuit, wherein the circuit, during operation, encodes a first submesh to generate first encoded data, encodes a second submesh to generate second encoded data, and in a decoding device, generates control information used to select a set of vertices including a first vertex included in the first submesh and a second vertex included in the second submesh, and generates a bitstream including the first encoded data, the second encoded data, and the control information, wherein the first vertex and the second vertex have the same level of detail (LoD) index value.

[0363] Also, for example, a decoding device of the present disclosure includes a circuit and a memory connected to the circuit, wherein the circuit, during operation, generates a first submesh and a second submesh, selects a set of vertices including a first vertex included in the first submesh and a second vertex included in the second submesh, modifies the positions of the first vertex and the second vertex to modified positions generated from the set of vertices, and the first vertex and the second vertex have the same level of detail (LoD) index value.

[0364] Also, for example, the encoding method of the present disclosure encodes a first submesh to generate first encoded data, encodes a second submesh to generate second encoded data, generates control information in a decoding device used to select a set of vertices including a first vertex included in the first submesh and a second vertex included in the second submesh, and generates a bitstream including the first encoded data, the second encoded data, and the control information, wherein the first vertex and the second vertex have the same level of detail (LoD) index value.

[0365] Also, for example, the decoding method of the present disclosure generates a first submesh and a second submesh, selects a set of vertices including a first vertex included in the first submesh and a second vertex included in the second submesh, modifies the positions of the first vertex and the second vertex to modified positions generated from the set of vertices, and the first vertex and the second vertex have the same level of detail (LoD) index value.

[0366] In this disclosure, to solve the problem of using different subdivision methods in adjacent sub-meshes, we enforce the constraint to have the same type of subdivision method and the same number of subdivisions along all boundary edges, as shown in Figure 50. Therefore, after the subdivided vertices are displaced and the sub-meshes are joined, there are no holes around the boundary edges, as shown in Figure 50(c), where the values ​​of the displacement vectors of vertex F and vertex M are the same.

[0367] A predetermined subdivision method and / or a predetermined number of subdivisions may be used for subdivision of the boundary edge, or the encoding device may determine the subdivision method and / or the number of subdivisions and signal information in the bitstream indicating the determined subdivision method and / or the number of subdivisions.

[0368] In the field of multimedia data coding technology, it is desirable to propose new methods for improving coding efficiency, improving image quality, and reducing circuit scale.

[0369] Each of the embodiments, some of the components, and each of the methods in the present disclosure enables at least one of, for example, improved coding efficiency, improved image quality, reduced encoding / decoding processing volume, reduced circuit size, and improved encoding / decoding processing speed. Alternatively, each of the embodiments, some of the components, and each of the methods in the present disclosure enables appropriate selection of any of elements or operations, such as filters, block sizes, motion vectors, reference pictures, and reference blocks, in encoding and decoding. The present disclosure includes disclosure of configurations and methods that can provide advantages other than those described above. Examples of such configurations and methods include configurations and methods that improve coding efficiency while suppressing an increase in processing volume.

[0370] Additional value and advantages of aspects of the present disclosure will become apparent from the specification and drawings, which may be obtained individually through various embodiments and features of the specification and drawings, not all of which need be provided to obtain one or more of such value and / or advantages.

[0371] These general or specific aspects may be implemented using a system, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or any combination of a system, a method, an integrated circuit, a computer program, or a recording medium.

[0372] <Submesh Combination Threshold> When zipping multiple submeshes, specifically when zipping the submeshes (more specifically, when zipping the vertices of the submeshes (more specifically, when zipping the boundary vertices) of the submeshes), the decoding device 200 searches for the vertices of each submesh and determines whether they are vertices to be combined (also called matching points). In this case, for each vertex of one submesh, the decoding device 200 searches for multiple vertices of the other submesh and determines whether the distance from the vertex of the one submesh is within a predetermined threshold range.

[0373] Note that a boundary vertex is a vertex on a boundary edge, in other words, a vertex located on a boundary edge. For example, boundary vertices are vertices located on both ends of a boundary edge. A boundary edge is an edge (side) that overlaps between sub-meshes. Specifically, a boundary edge is an edge that is common to two sub-meshes when a three-dimensional mesh is divided into two sub-meshes.

[0374] The threshold may be set per sequence, per frame, per submesh, and / or per vertex.

[0375] Here, the threshold value is set to, for example, the distance from a vertex of a certain submesh.

[0376] However, when there are two or more submeshes to be combined with a certain submesh, that is, when a certain submesh is combined with multiple submeshes, it may not be possible to set an optimal threshold. For example, the optimal threshold may differ for each vertex. In such a case, if a threshold is set for each submesh, the decoding device 200 may not be able to properly search for the vertex to be combined and may make an incorrect combination, resulting in a failure to obtain a high-quality 3D mesh shape.

[0377] Therefore, for example, the encoding device 100 transmits information (combination information) indicating a combination of submeshes to be combined (submesh-pair) to the decoding device 200. In addition, the encoding device 100 transmits information (threshold information) indicating a threshold for each combination of submeshes to be combined to the decoding device 200.

[0378] By transmitting information indicating the threshold value for each combination of submeshes to be combined to the decoding device 200, even when a submesh is combined with multiple submeshes, the decoding device 200 can properly search for the vertices to be combined, thereby preventing incorrect combinations and obtaining a high-quality three-dimensional mesh shape.

[0379] In particular, if the coding parameters, such as base mesh quantization, displacement vector quantization, and lifting transform, are different for each sub-mesh, the distance between the sub-meshes will change, which will result in a change in the optimal threshold for each vertex. Even in such cases, a high-quality connected 3D mesh can be reproduced.

[0380] Furthermore, by setting a threshold for each combination, it is not necessary to set a threshold for each vertex, and therefore the amount of information can be reduced.

[0381] Furthermore, when a large number of submeshes are combined into one submesh, information indicating the optimal threshold for each combination is transmitted to the decoding device 200, thereby enabling a high-quality three-dimensional mesh shape to be obtained.

[0382] In addition, by transmitting the submesh combination information to the decoding device 200, it is possible to omit transmitting threshold information corresponding to submeshes that are not connected to any submeshes, thereby reducing the amount of information in the threshold information.

[0383] Furthermore, by transmitting the submesh combination information to the decoding device 200, the decoding device 200 can search only the vertices of the submeshes to be combined. This reduces the amount of processing required for the vertex search process by the decoding device 200. It also makes it possible to avoid erroneous combination with submeshes other than the combination target.

[0384] The transmission of threshold information and the process of searching for matching points using a threshold in combining sub-meshes will be described below.

[0385] FIG. 51 is a block diagram showing an example of the configuration of the encoding device 100 and the decoding device 200 according to the embodiment.

[0386] The encoding device 100 includes, for example, a submesh divider 2401 and a submesh encoder 2402 .

[0387] The submesh divider 2401 divides an input 3D mesh into a plurality of submeshes. The submesh divider 2401 also generates hint information (submesh hint information metadata) that indicates useful hints for combining submeshes with high quality in the decoding device 200. The hint information is metadata that includes, for example, threshold information (distance information).

[0388] The submesh divider 2401, for example, calculates a threshold value for searching for vertices between submeshes when combining the submeshes, and stores threshold information indicating the calculated threshold value in metadata such as SEI.

[0389] The sub-mesh encoder 2402 encodes the SEI, and includes (stores) the encoded SEI in a bitstream together with encoded data that encodes information about the sub-mesh, and transmits the encoded SEI to the decoding device 200.

[0390] The decoding device 200 includes, for example, a sub-mesh decoder 2403 and a sub-mesh combiner 2404 .

[0391] The submesh decoder 2403 decodes the coded data contained in the bitstream (more specifically, information relating to the coded submeshes).

[0392] The submesh combiner 2404 obtains threshold information from metadata such as SEI contained in the bitstream, searches for vertices to be combined with vertices of a certain submesh based on the threshold information, and combines the found vertices with vertices of a certain submesh.

[0393] 52 is a flow diagram showing a search process for matching points according to an embodiment. Before the search process is performed, the decoding device 200, for example, decodes coded data included in the bitstream to obtain multiple submeshes (specifically, position information for each boundary vertex of the multiple submeshes). In this example, a threshold is set for each boundary vertex, and the threshold information for each boundary vertex is included in the bitstream. In this example, the threshold for boundary vertices that are not connected to other boundary vertices is set to 0.

[0394] First, the decoding device 200 determines whether maxDistance, which is the maximum value of the threshold indicated by each piece of threshold information included in the bitstream, is 0 (S701). In other words, the decoding device 200 determines whether there is a matching point at each boundary vertex of the multiple sub-meshes based on maxDistance. For example, if maxDistance = 0, the decoding device 200 determines that there is no matching point at any of the boundary vertices of the multiple sub-meshes. In this case, the decoding device 200 does not perform the matching process described below. On the other hand, for example, if maxDistance = 0, the decoding device 200 determines that there is a matching point at at least one of the boundary vertices of the multiple sub-meshes.

[0395] If the decoding device 200 determines that the maxDistance included in the bitstream is 0 (Yes in S701), it determines that there is no matching point (S709) and ends the process.

[0396] On the other hand, if the decoding device 200 determines that the maxDistance included in the bitstream is not 0 (No in S701), it determines that a matching point exists and starts matching processing (S702). Specifically, it executes the processing from step S703 onwards for each boundary vertex of multiple sub-meshes.

[0397] The decoding device 200 calculates a threshold value for each boundary vertex in the frame (for example, multiple sub-meshes) based on the SEI included in the bitstream (S703).

[0398] Next, the decoding device 200 performs processing for each boundary vertex in the frame (S704).

[0399] Next, the decoding device 200 calculates the distances to all boundary vertices of all submeshes other than the submesh containing the boundary vertex (target boundary vertex) for which a matching point is searched (S705).

[0400] Next, the decoding device 200 determines whether the distance between the target boundary vertex and each boundary vertex is less than a threshold value (S706).

[0401] If the decoding device 200 determines that the distance between the target boundary vertex and each boundary vertex is not less than the threshold value (No in S706), that is, if it determines that the distances between the target boundary vertex and each boundary vertex are all greater than or equal to the threshold value, it determines that no matching point exists at the target boundary vertex (S709).

[0402] On the other hand, if the decoding device 200 determines that the distance between the target boundary vertex and each boundary vertex is less than the threshold (Yes in S706), that is, if it determines that the distances between the target boundary vertex and each boundary vertex include distances less than the threshold, it determines the boundary vertex with the smallest distance among the distances less than the threshold as the matching point (S707).

[0403] Furthermore, the decoding device 200 determines not to search for a matching point for the boundary vertex determined as the matching point (S708).

[0404] After searching for a matching point for the target boundary vertex, the decoding device 200 performs a joining process between the target boundary vertex and the matching point. For example, when joining two boundary vertices that are far apart, the decoding device 200 calculates the position coordinates of the joining point (a new vertex generated by joining the two boundary vertices) using a predetermined method. The calculated position coordinates (position coordinates of the joining point) may be, for example, the midpoint or center of gravity of the two points, or the position coordinates may be brought closer to one of the boundary vertices using a weighting factor. For example, a three-dimensional point is generated at the position coordinates calculated in this way, and the two boundary vertices are deleted, thereby joining the two boundary vertices.

[0405] FIG. 53 is a diagram for explaining the positional relationship of a plurality of sub-meshes according to an embodiment.

[0406] In the example shown in Figure 53, multiple sub-meshes are generated by cutting and dividing a 3D mesh in a certain direction (horizontally in this example) (linear segmentation). With this division method, the maximum number of sub-meshes adjacent to one sub-mesh is two. For example, sub-mesh 1 is not adjacent to sub-mesh 3.

[0407] In the encoding process, the positions (coordinates) of the vertices of the submeshes may be moved. For example, suppose that the division of the three-dimensional mesh results in the maximum distance between the boundary vertices of submeshes 1 and 2 being 5 units, and the maximum distance between the boundary vertices of submeshes 2 and 3 being 20 units. Note that the unit of distance may be any unit and is not particularly limited.

[0408] For example, when a threshold is set for each submesh, for the three submeshes (submeshes 1, 2, and 3) shown in Figure 53, the threshold for submesh 1 is set to 5 (Threshold(submesh_1) = 5), the threshold for submesh 2 is set to 20 (Threshold(submesh_2) = 20), and the threshold for submesh 3 is set to 20 (Threshold(submesh_3) = 20).

[0409] When a division method such as the example shown in Fig. 53 is used and it is guaranteed that the submeshes can be processed in, for example, the encoding order, submeshes 1 and 3 are farther apart than submeshes 1 and 2. Therefore, even if threshold information indicating a threshold for each submesh is transmitted, for example, the threshold for submesh 1 becomes the threshold used in the joining process between submeshes 1 and 2, and the threshold for submesh 2 becomes the threshold used in the joining process between submeshes 2 and 3. Therefore, even if a threshold is set for each submesh, the set threshold may be treated as the threshold for each combination of submeshes.

[0410] However, if submeshes 1 and 3 are located close to each other, there is a possibility that submeshes 1 and 3 will be joined even if submeshes 1 are not a target for joining with submeshes 3.

[0411] Here, for example, the threshold value for submeshes 1 and 2 is set to 5, the threshold value for submeshes 2 and 3 is set to 20, and the threshold value for submeshes 1 and 3 is set to 0. In other words, a threshold value is set for each combination of submeshes. For example, if the threshold value for a combination is 0, the decoding device 200 does not combine the submeshes of that combination. In this example, the decoding device 200 does not combine submeshes 1 and 3.

[0412] In this way, by including in the bit stream information indicating that submesh 1 is not a target for joining with submesh 3, it is possible to prevent erroneous joining.

[0413] FIG. 54 is a diagram for explaining the process of searching for matching points of a plurality of sub-meshes according to an embodiment.

[0414] In this example, submesh 1 contains boundary vertices A, B, C, L, M, and N; submesh 2 contains boundary vertices D, E, F, G, H, and K; and submesh 3 contains boundary vertices X, Y, and Z.

[0415] The algorithm for merging boundary vertices (the merge algorithm) uses a set threshold (threshold distance) to search for matching points for boundary vertices. Figure 54 shows the search radius associated with the search for matching points for each boundary vertex of submesh 2. The search radius corresponds, for example, to a threshold value set for each boundary vertex. Other boundary vertices within a circle whose distance (radius) from the target boundary vertex is the threshold value become the search range for matching points for the target boundary vertex.

[0416] 54 indicates the search radius of boundary vertex E. The dashed line in FIG. 54 indicates the search radius of boundary vertex D. The dashed line in FIG. 54 indicates the search radius of boundary vertex H. The dotted line in FIG. 54 indicates the search radius of boundary vertex F.

[0417] Boundary vertices D, H, F, and E are boundary vertices of submesh 2 that are connected to boundary vertices of other submeshes. Boundary vertex D is connected to boundary vertex A, boundary vertex H is connected to boundary vertex L, boundary vertex F is connected to boundary vertex B, and boundary vertex E is connected to boundary vertex X. On the other hand, boundary vertices G and K of submesh 2 are not connected to any boundary vertices of other submeshes.

[0418] For example, if a threshold is set for each submesh, the maximum value of the distance between each target boundary vertex of the submesh and each matching point is set as the threshold for the submesh. Therefore, for submesh 2, the threshold associated with boundary vertex E, which has the largest search radius, is set.

[0419] When a threshold is set for each submesh, in this example, the distance between submeshes 1 and 2 is close, but the distance between submeshes 2 and 3 is far, so in order to join boundary vertex E and boundary vertex X, the distance between submeshes 2 and 3 (i.e., the search radius of boundary vertex E) needs to be set as the threshold.

[0420] 55 is a diagram for explaining the process of searching for matching points of a plurality of sub-meshes according to an embodiment. Note that the two-dot chain line in FIG. 55 indicates the search radius of the boundary vertex G.

[0421] The decoding device 200, using the threshold set by the encoding device 100, searches for matching points for all boundary vertices of submesh 2.

[0422] Because the set threshold (distance) is greater than or equal to the threshold (distance) that is originally sufficient for the boundary vertices D, H, F, and E, the boundary vertices D, H, F, and E are combined with the appropriate matching points.

[0423] For example, when a threshold (search radius) set for boundary vertex G is used, boundary vertex A of submesh 1 is found within the search range. Therefore, boundary vertex G is combined with boundary vertex A. Then, boundary vertices A, D, and G may be combined to form a degenerate triangle of face DGH.

[0424] Furthermore, for example, even if a threshold value (search radius) set for boundary vertex K is used, boundary vertices of submeshes 1 and 3 may not be found within the range in which matching points are searched.

[0425] Therefore, for example, the decoding device 200 performs the following process in searching for matching points.

[0426] 56 is a flow diagram showing a search process for matching points according to an embodiment. Before the search process is performed, the decoding device 200, for example, obtains multiple submeshes (specifically, position information of each boundary vertex of the multiple submeshes) by decoding encoded data included in the bitstream. In this example, a threshold is set for each combination of submeshes, and the threshold information for each combination is included in the bitstream. In this example, the threshold for combinations of submeshes that are not combined is set to 0.

[0427] First, the decoding device 200 determines whether maxDistance, which is the maximum value of the thresholds indicated by the threshold information included in the bitstream, is 0 (S711). In other words, the decoding device 200 determines whether there is a matching point at each boundary vertex of the multiple sub-meshes based on maxDistance. For example, if maxDistance = 0, the decoding device 200 determines that there is no matching point at any of the boundary vertices of the multiple sub-meshes. In this case, the decoding device 200 does not perform the matching process described below. On the other hand, for example, if maxDistance = 0, the decoding device 200 determines that there is a matching point at at least one of the boundary vertices of the multiple sub-meshes.

[0428] If the decoding device 200 determines that the maxDistance included in the bitstream is 0 (Yes in S711), it determines that there is no matching point (S719) and ends the process.

[0429] On the other hand, if the decoding device 200 determines that the maxDistance included in the bitstream is not 0 (No in S711), it determines that a matching point exists and starts matching processing (S712). Specifically, it executes the processing from step S713 onwards for each boundary vertex of multiple sub-meshes.

[0430] The decoding device 200 calculates a threshold for each combination of submeshes (connected pairs of submeshes) within a frame (e.g., multiple submeshes) based on the SEI contained in the bitstream, and calculates a threshold for each boundary vertex based on the threshold for each combination (S713).

[0431] Next, the decoding device 200 performs processing for each boundary vertex in the frame (S714).

[0432] Next, the decoding device 200 calculates the distances to all boundary vertices of all submeshes for which combinations are defined (S715).

[0433] Next, the decoding device 200 determines whether the distance between the target boundary vertex and each boundary vertex is less than a threshold value (S716).

[0434] If the decoding device 200 determines that the distance between the target boundary vertex and each boundary vertex is not less than the threshold value (No in S716), that is, if it determines that the distances between the target boundary vertex and each boundary vertex are all greater than or equal to the threshold value, it determines that no matching point exists at the target boundary vertex (S719).

[0435] On the other hand, if the decoding device 200 determines that the distance between the target boundary vertex and each boundary vertex is less than the threshold (Yes in S716), that is, if it determines that the distances between the target boundary vertex and each boundary vertex include distances less than the threshold, it determines the boundary vertex with the smallest distance among the distances less than the threshold as the matching point (S717).

[0436] Furthermore, the decoding device 200 determines not to search for a matching point for the boundary vertex determined as the matching point (S718).

[0437] Figure 57 is a diagram for explaining the search process for matching points of multiple sub-meshes according to an embodiment. Note that the two-dot chain line in Figure 57 indicates the search radius of boundary vertex E. The dashed line in Figure 57 indicates the search radius of boundary vertex D. The one-dot chain line in Figure 57 indicates the search radius of boundary vertex H. The dotted line in Figure 57 indicates the search radius of boundary vertex F.

[0438] When a threshold is set for each combination (pair) of submeshes as described above, as shown in Figure 57, if the search radius of boundary vertex H is larger than the search radius of boundary vertices D and F, the threshold for the combination of submesh 1 and submesh 2 is set to the search radius of boundary vertex H.

[0439] 58 is a diagram for explaining the process of searching for matching points in multiple submeshes according to an embodiment. Note that the dashed-dotted line in Fig. 58 indicates the search radius of the boundary vertex G used for submesh 1. Also, the dashed-two-dotted line in Fig. 58 indicates the search radius of the boundary vertex G used for submesh 3.

[0440] The decoding device 200 searches for boundary vertices of submesh 1 that are included in the search range for matching points of boundary vertex G, for example, using a threshold value associated with the combination of submesh 1 and submesh 2.

[0441] In addition, the decoding device 200 searches for boundary vertices of submesh 3 that are included in the search range of matching points for boundary vertex G, for example, using a threshold value associated with the combination of submesh 2 and submesh 3.

[0442] Boundary vertex A is included in submesh 1, and boundary vertex G is included in submesh 2. Furthermore, the distance between boundary vertex G and boundary vertex A is closer than the search radius of boundary vertex G used for submesh 3, but is farther than the search radius of boundary vertex G used for submesh 1. Therefore, boundary vertex G is not connected to boundary vertex A.

[0443] Fig. 59 is a diagram for explaining an example of syntax in which sub-mesh combining metadata according to an embodiment is signaled. Specifically, Fig. 59 is a diagram showing syntax for transmitting threshold information indicating a threshold (maximum distance threshold for searching for matching points in combining processing) to the decoding device 200. Fig. 59 is a first example of the syntax.

[0444] The sub-mesh combining metadata exists for each instance and is a syntax indicating the kth (kth integer) metadata. That is, each sub-mesh combining metadata has k elements. In this example, the index k is omitted in each syntax.

[0445] In this example, the metadata for submesh combination includes distance_per_submesh_flag (a flag (flag information) indicating whether or not to indicate a threshold value for each submesh), and when distance_per_submesh_flag = 1 (i.e., when a threshold value for each submesh is indicated), it includes information indicating the threshold value for each submesh.

[0446] On the other hand, if distance_per_submesh_flag=0 (that is, if a threshold value for each submesh is not indicated), the submesh combining metadata includes distance_per_submesh_pair_flag (a flag indicating whether or not a threshold value for each combination of submeshes is indicated).

[0447] If distance_per_submesh_pair_flag=1 (that is, if a threshold value for each combination of submeshes is indicated), the submesh combining metadata includes threshold value information indicating the threshold value for each combination of submeshes.

[0448] Here, if the threshold value indicated by the threshold value information is a predetermined value (for example, 0), the sub-meshes may be considered not to be adjacent to each other.

[0449] 60 is a diagram for explaining an example of syntax in which sub-mesh combining metadata according to an embodiment is signaled, and is a second example of the syntax.

[0450] In this example, the type of sub-mesh division method (division type / segmentation_type) is also shown.

[0451] For example, if the division method of a 3D mesh is linear segmentation (linear_segmentation), it indicates that the 3D mesh is divided by cutting in a certain direction (for example, the horizontal direction).

[0452] The division method may be indicated by a flag instead of the division type.

[0453] Furthermore, if a division method other than linear segmentation exists, the method of transmitting threshold information may be switched depending on the division method. For example, when the division method of 3D meshes is linear segmentation, the encoding device 100 does not include threshold information indicating the threshold for each combination of submeshes in the bitstream, but includes threshold information for the number of submeshes or the number of submeshes minus 1 in the bitstream.

[0454] Note that distance_per_submesh_flag and distance_per_submesh_pair_flag may be included independently in the bitstream.

[0455] Furthermore, distance_per_submesh_pair may be included in the bitstream as a one-dimensional array instead of a two-dimensional array.

[0456] In a syntax loop indicating a combination of submeshes, the second loop indicates information about submeshes from the pth submesh (p is an integer greater than or equal to 1) in the first loop onwards, thereby reducing the amount of information.

[0457] For example, distance_per_submesh_flag[k] = 1 indicates that for multiple submeshes (zippering instances) corresponding to the metadata of index k, a threshold for each combination of submeshes is set to the boundary vertex. For example, distance_per_submesh_flag[k] = 0 indicates that for multiple submeshes (zippering instances) corresponding to the metadata of index k, a threshold for each combination of submeshes is not set to the boundary vertex. The default value of distance_per_submesh_pair_flag[k] is, for example, 0.

[0458] Also, for example, segmentation_type[k]=1 indicates that submeshes having an index (submeshIdx, for example, a submesh identification number) in the range from 1 to MaxNumSubmeshes[frameIdx]-2 can have mutual boundary vertices (i.e., have boundary vertices that can be joined) only with submeshes having an index of submeshIdx-1 and an index of submeshIdx+1. For example, when segmentation_type[k]=0, a submesh having an index of 0 and a submesh having an index of MaxNumSubmeshes[frameIdx]-1 can have mutual boundary vertices only with a submesh having an index of 1 and a submesh having an index of MaxNumSubmeshes[frameIdx]-2, respectively. Also, for example, segmentation_type[k] = 0 indicates that each submesh can have a mutual boundary vertex with a submesh different from the above. In other words, when segmentation_type[k] = 0, it indicates that a submesh having an index of submeshIdx may have a mutual boundary vertex with submeshes other than those having an index of submeshIdx-1 and an index of submeshIdx+1. The default value of segmentation_type[k] is, for example, 1.

[0459] Also, for example, segmentation_type[k] = 1 indicates that a submesh having an index (submeshIdx) ranging from 0 to MaxNumSubmeshes[frameIdx] - 1 can have mutual boundary vertices with the submeshes with indices submeshIdx - 1 and submeshIdx + 1 when submeshIdx - 1 is 0 or greater and submeshIdx + 1 is MaxNumSubmeshes[frameIdx] - 2 or less. When segmentation_type[k] = 0, it indicates that each submesh may have mutual boundary vertices with a submesh different from the above. The default value of segmentation_type[k] is, for example, 1.

[0460] distance_per_submesh_pair_flag[k][p][n] indicates the value of a variable (zipperingMaxMatchDistancePerPatchPair[k][p][n]) used to process a combination of a submesh with index p and a submesh with index n+p+1 that is different from the submesh with index p and that is divided from the same 3D mesh as the submesh (zippering instance) for multiple submeshes (zippering instances) corresponding to the metadata of index [k] when a zipping process is used. For example, the length of the syntax element distance_per_submesh_pair_flag[k][p][n] is Ceil(Log2(max_match_distance[k])) bits.

[0461] A specific example of syntax setting using an example of a threshold determination method in the encoding device 100 will be described.

[0462] For example, suppose the values ​​of each variable are given as follows:

[0463] 1. distance_per_submesh_pair_flag[k]=True 2. distance_per_submesh_flag[k]=False 3. number_of_submesh[k]=4 4. segmentation_type[k]=1 (Linear Segmentation)

[0464] These variables cause the following code to be executed:

[0465] for (p=0; p<numSubmeshes-1; p++) { distance_per_submesh_pair_flag[k][p][0]}

[0466] If numSubmeshes is replaced with a signed value, the above code can be written as follows:

[0467] for (p=0; p<3; p++) { distance_per_submesh_pair_flag[k][p][0]}

[0468] Then, from the bitstream, (i) distance_per_submesh_pair_flag[k][0][0] indicating the threshold (distance) between submesh 0 and submesh 1, (ii) distance_per_submesh_pair_flag[k][1][0] indicating the threshold (distance) between submesh 1 and submesh 2, and (iii) distance_per_submesh_pair_flag[k][2][0] indicating the threshold (distance) between submesh 2 and submesh 3 are obtained.

[0469] Here, the threshold value for the combination of submeshes 1 and 2 is the same as the threshold value for the combination of submeshes 2 and 1, so two pieces of information about the same value do not need to be included in the bit stream.

[0470] Next, another example of syntax setting using an example of a threshold determination method in the encoding device 100 will be described.

[0471] For example, suppose the values ​​of each variable are given as follows:

[0472] 1. distance_per_submesh_pair_flag[k]=True 2. distance_per_submesh_flag[k]=False 3. number_of_submesh[k]=4 4. segmentation_type[k]=0

[0473] These variables cause the following code to be executed:

[0474] for (p=0; p<numSubmeshes-1; p++) { for (n=p+1; n<numSubmeshes; n++) { distance_per_submesh_pair_flag[k][p][n-p-1]}}

[0475] If numSubmeshes is replaced with a signed value, the above code can be written as follows:

[0476] for (p=0; p<3; p++) { for (n=p+1; n<4; n++) { distance_per_submesh_pair_flag[k][p][n-p-1]}}

[0477] Then, from the bitstream, (i) distance_per_submesh_pair_flag[k][0][0] indicating the threshold (distance) between submesh 0 and submesh 1, (ii) distance_per_submesh_pair_flag[k][0][1] indicating the threshold (distance) between submesh 0 and submesh 2, and (iii) distance_per_submesh_pair_flag[k][0] indicating the threshold (distance) between submesh 0 and submesh 3. [2], (iv) distance_per_submesh_pair_flag[k][1][0] indicating the threshold (distance) between submesh 1 and submesh 2, (v) distance_per_submesh_pair_flag[k][1][1] indicating the threshold (distance) between submesh 1 and submesh 3, and (vi) distance_per_submesh_pair_flag[k][2][0] indicating the threshold (distance) between submesh 2 and submesh 3 are obtained.

[0478] Here, the threshold value for the combination of submeshes 1 and 3 is the same as the threshold value for the combination of submeshes 3 and 0, so two pieces of information about the same value do not need to be included in the bit stream.

[0479] Next, another example of the syntax of the sub-mesh connection metadata will be shown.

[0480] Figure 61 is a diagram for explaining an example of a syntax in which submesh combination metadata according to an embodiment is signaled. Figure 62 is a diagram for explaining an example of a syntax in which information indicating a combination of submeshes to be combined according to an embodiment is signaled. Specifically, Figure 62 shows a syntax (connected_submesh_info) indicating combination information indicating whether multiple submeshes are combined.

[0481] The information indicated by connected_submesh_info may be included in the submesh combining metadata, or may be included in an SEI different from the submesh combining metadata and transmitted to the decoding device 200 via the bitstream.

[0482] Based on combination information indicating whether the sub-meshes are combined or not, threshold information indicating the threshold value of the combination is transmitted if the combination is combined, and if the combination is not combined, the threshold information does not need to be transmitted.

[0483] As described above, threshold information indicating the threshold for each submesh may be included in the bit stream, or threshold information indicating the threshold for each combination of submeshes may be included in the bit stream.

[0484] The information indicating the threshold value may be signaled in the bitstream for each submesh, or for each combination of vertices (specifically, boundary vertices) to be combined. Information on whether a vertex is a boundary vertex may also be signaled in the bitstream.

[0485] For example, the encoding device 100 determines a method for setting the threshold, includes information indicating the determined method and threshold information in metadata, and transmits a bitstream including the metadata to the decoding device 200.

[0486] The decoding device 200 performs a search for matching points in the combination of sub-meshes based on, for example, information indicating the method and threshold information included in the bitstream.

[0487] When threshold information indicating a threshold for each combination of sub-meshes is transmitted, the decoding device 200 may determine that the sub-meshes are not combined and skip the search process for matching points using the threshold if, for example, the threshold indicated by the threshold information is 0. In other words, in this case, the search process for matching points does not need to be performed.

[0488] Furthermore, for example, if the threshold value indicated by the threshold value information is other than 0, the decoding device 200 determines that the sub-meshes are to be combined, and executes a search process using the threshold value.

[0489] Furthermore, the encoding device 100 may transmit, for example, for each combination of submeshes, a bitstream including information indicating whether the submeshes are combined to the decoding device 200. Furthermore, when the submeshes are combined, the encoding device 100 may transmit, to the decoding device 200, a bitstream including threshold information.

[0490] Furthermore, in the process of searching for matching points using a threshold value, the decoding device 200 executes the search process in two submeshes to be combined using a threshold value corresponding to the combination.

[0491] Furthermore, the decoding device 200 determines whether or not submeshes are to be combined, for example, based on threshold information or information indicating whether or not the submeshes are to be combined, and performs processing on combinations of submeshes that are determined to be combined, and does not perform processing on combinations of submeshes that are determined not to be combined. Note that the processing referred to here may be a search process for matching points, other processing in combining submeshes, or a combining process of attribute information of the submeshes and attribute information of the faces of the submeshes.

[0492] Furthermore, the encoding device 100 may switch between transmitting a bitstream including threshold information for each submesh and transmitting a bitstream including threshold information for each combination of submeshes, depending on the submesh division method.

[0493] Also, a third threshold transmission method may be used depending on the sub-mesh division method.

[0494] <Handling of Some Submeshes> The encoding device 100 divides a three-dimensional mesh into a plurality of submeshes, encodes each of the submeshes (specifically, vertex information and connection information of each of the plurality of submeshes (more specifically, base mesh information of each of the plurality of submeshes), and information about the plurality of submeshes such as displacement vector information), and includes identification information of the submeshes (specifically, information indicating a submesh ID that uniquely identifies each submesh) in a bitstream and transmits the bitstream to the decoding device 200. The submesh ID is assigned to, for example, the vertex information and connection information of the submesh, and information about the submesh such as displacement vector information.

[0495] When dividing a three-dimensional mesh, the encoding device 100 generates information indicating the number of all submeshes contained in the frame to which the three-dimensional mesh belongs, and submesh combining metadata, which is information necessary for the decoding device 200 to combine the submeshes.

[0496] The metadata for sub-mesh merging includes, for example, information (profile information) indicating whether the decoding device 200 will perform a merging process, threshold information indicating the threshold (distance) for the decoding device 200 to search for sub-mesh combinations (merging pairs), and information specifying the content of the merging process when the merging process is executed.

[0497] The submesh combining metadata includes, for example, information on all submeshes included in one frame that are the subject of combining processing, or information on submeshes divided from a three-dimensional mesh.

[0498] The decoding device 200 decodes some or all of the submeshes and the submesh identification information of the some or all of the submeshes, and combines the some or all of the submeshes.

[0499] Specifically, the submesh decoder 2403 included in the decoding device 200 decodes, for example, some or all of the submeshes and the submesh identification information of the some or all of the submeshes, and outputs the some or all of the submeshes to the submesh combiner 2404 included in the decoding device 200. The submesh decoder 2403 generates, for example, information indicating the number of submeshes to be output to the submesh combiner 2404, as well as submesh identification information, vertex information, and face information for each submesh to be output to the submesh combiner 2404, and outputs the generated information to the submesh combiner 2404.

[0500] The submesh combiner 2404 obtains from the input data some or all of the submeshes, information indicating the number of some or all of the input submeshes (i.e., the number of submeshes to be combined), and submesh identification information that uniquely indicates the submeshes to be combined, and identifies the submeshes to be combined.

[0501] The submesh combiner 2404 also acquires the combining information of the submeshes to be combined from the input submesh combining metadata based on the submesh identification information. The combining information includes, for example, the threshold information described above. The submesh combiner 2404 combines the submeshes to be combined based on the combining information.

[0502] When a bitstream is transmitted from the encoding device 100 to the decoding device 200 via a network, the order of the multiple submeshes may be changed from the encoded order, or some of the multiple submeshes may be missing. Also, for example, in encoding and decoding of submeshes, the order of the multiple submeshes may be changed from the encoded order, or some of the multiple submeshes may be missing.

[0503] Here, as in the above configuration, by using the submesh identification information, the decoding device 200 can identify the submeshes to be combined. Therefore, the decoding device 200 can correctly combine (restore) the submeshes to be combined using the submesh combining metadata.

[0504] Furthermore, the above configuration enables parallel processing in encoding, enables parallel processing in decoding, and enables the decoding device 200 to combine partial decoded data (specifically, some submeshes among multiple submeshes).

[0505] Furthermore, with the above configuration, it is sufficient that only information on the sub-meshes to be combined is stored in the sub-mesh combining metadata, and the amount of information can be reduced.

[0506] FIG. 63 is a diagram illustrating a method of assigning submesh IDs according to an embodiment. Specifically, FIG. 63 is a diagram illustrating a method in which the encoding device 100 assigns submesh IDs to submeshes. (a) of FIG. 63 illustrates a first example of the assignment method, and (b) of FIG. 63 illustrates a second example of the assignment method. Note that each rectangle in FIG. 63 schematically illustrates a submesh, and the numbers within the rectangle indicate the submesh ID of the submesh. Furthermore, the multiple submeshes arranged horizontally in a row in FIG. 63 are coded, for example, from left to right by the encoding device 100. In the example shown in (a) of FIG. 63, the submeshes having submesh IDs set thereto are coded by the encoding device 100 in the order of submesh IDs = 0, 1, 2, 3, 4, 5, 6, and 7. For example, the multiple submeshes are stored in this order in the bitstream.

[0507] The header of the coded data of a submesh stores information indicating a submesh ID for uniquely identifying the submesh.

[0508] As a method for the encoding device 100 to assign submesh IDs to submeshes, the following two methods, a first example and a second example, are conceivable.

[0509] 63(a), in the first example, the encoding device 100 assigns a submesh ID to each of the multiple submeshes, with the submesh ID of the first submesh being 0 and numbers (numbers) increasing by 1 in the encoding order from the first submesh. Here, the first submesh is, for example, the first submesh of a frame or the first submesh of a tile in the encoding order (specifically, the first submesh to be encoded among the multiple submeshes).

[0510] In the first example, the decoding device 200 acquires information indicating a submesh ID from the header of the submesh. Note that the decoding device 200 may assign a submesh ID to each of the submeshes, for example, a submesh ID of the first submesh (e.g., the submesh that is decoded first among the multiple encoded submeshes) as 0, and assign numbers that increase by 1 from the first submesh in the decoding order to each of the multiple submeshes.

[0511] In the second example shown in FIG. 63(b), for example, the encoding device 100 assigns a submesh ID of an arbitrary value to each of a plurality of submeshes.

[0512] In the second example, the decoding device 200 acquires information indicating the submesh ID from the header of the submesh.

[0513] For example, a flag (submesh_id_signal_flag) indicating whether the first or second example signal assignment method was used is stored in metadata (parameter set) common to each submesh. The metadata common to each submesh in which the flag is stored may be determined arbitrarily. The metadata common to each submesh may be, for example, a parameter set related to the base mesh or a parameter set related to the displacement vector.

[0514] 64 is a diagram for explaining the submesh IDs of decoded submeshes according to an embodiment. Specifically, (a) to (d) in FIG. 64 each show a specific example of the order of the submeshes.

[0515] Note that each rectangle in Figure 64 schematically represents a submesh, and the numbers within the rectangle indicate the submesh ID of that submesh. Furthermore, the multiple submeshes arranged in a horizontal row in Figure 64 indicate, for example, the order of the multiple coded submeshes in the bitstream when the bitstream transmitted from the coding device 100 is acquired by the decoding device 200. Specifically, (a) and (b) of Figure 64 show the order of the multiple coded submeshes in the bitstream acquired by the decoding device 200 when the decoding device 200 acquires (receives) via a network a bitstream containing multiple coded submeshes in the order shown in (a) of Figure 63. Furthermore, (c) and (d) of Figure 64 show the order of the multiple coded submeshes in the bitstream acquired by the decoding device 200 when the decoding device 200 acquires (receives) via a network a bitstream containing multiple coded submeshes in the order shown in (b) of Figure 63.

[0516] Moreover, a submesh generated by decoding the coded data included in the bitstream is also called a decoded submesh (decoded submesh).

[0517] The bitstream sent from the encoding device 100 to the decoding device 200 may cause the order of multiple submeshes included in the bitstream to be changed.

[0518] In a network, if the order of multiple submeshes is changed due to retransmission of a bit stream (packet), etc., the order of the coded submeshes may be changed when they are decoded in parallel.

[0519] In the example shown in Figure 64(a), the second submesh (submesh with submesh ID = 1) and the third submesh (submesh with submesh ID = 2) are swapped from the example shown in Figure 63(a).

[0520] In the example shown in FIG. 64(c), the second submesh (submesh with submesh ID = 6) and the third submesh (submesh with submesh ID = 9) are swapped from the example shown in FIG. 63(b).

[0521] Here, in the example shown in (a) of Figure 64, since the submesh IDs are assigned to the submeshes in ascending order, the decoding device 200 can detect (determine) whether the order of the submeshes is appropriate using the submesh IDs. Specifically, the decoding device 200 can detect whether the order of the submeshes has changed.

[0522] On the other hand, in the example shown in FIG. 64(c), the submesh IDs are assigned discontinuously to the submeshes, so the decoding device 200 cannot detect that the order of the submeshes has been changed.

[0523] Also, transmission of the bitstream may result in sub-meshes being missing from the bitstream.

[0524] Such a loss occurs, for example, when there is a loss in the network, or when data of some sub-meshes is intentionally extracted and transmitted.

[0525] Furthermore, some sub-meshes may be intentionally extracted and decoded, resulting in missing sub-meshes after decoding.

[0526] In the example shown in FIG. 64(b), the third submesh (the submesh with submesh ID=2) is missing from the example shown in FIG. 63(a).

[0527] In the example shown in FIG. 64(d), the third submesh (the submesh with submesh ID=9) is missing from the example shown in FIG. 63(b).

[0528] In the example shown in FIG. 64(b), submesh IDs are assigned to the submeshes in ascending order, so the decoding device 200 can detect missing submeshes.

[0529] On the other hand, in the example shown in FIG. 64(d), the submesh IDs are assigned discontinuously to the submeshes, so the decoding device 200 cannot detect missing submeshes.

[0530] If the submeshes are coded independently and can be decoded independently, decoding is possible without any problem even if the order of the submeshes is changed or some submeshes are missing.

[0531] In addition, an independent submesh (independent submesh), such as an independently coded submesh, is, for example, a submesh that is coded without any dependency on other submeshes and is randomly accessible, that is, a submesh that can be decoded independently and can be the start of decoding.

[0532] In addition, a non-independent submesh (dependent submesh), such as a submesh that is coded depending on the coding results of other submeshes, refers to a submesh whose decoding depends on the decoding results of other submeshes and which cannot be decoded independently.

[0533] Context information may be used in encoding and decoding a submesh. The content of the context information changes during encoding and decoding. Therefore, for example, a submesh encoded using continuous context information can be decoded only after the decoding of the previous submesh is completed. Therefore, the decoding device 200 cannot independently decode a submesh encoded using continuous context information. When the context continuation function is used, a dependency may exist between submeshes. For example, a submesh encoded with initialized context information is an independent submesh because it only needs to be decoded with the context information initialized. On the other hand, a submesh encoded using continuous context information is a dependent submesh because it can only be decoded after the decoding of the previous submesh is completed.

[0534] 65 is a diagram for explaining the submesh IDs of submeshes in each processing procedure according to the embodiment. Specifically, (a), (b), (c), and (d) of FIG. 65 schematically show the order of submeshes acquired (input) or output to the submesh decoder 2403 and the submesh combiner 2404 provided in the decoding device 200, respectively. (a) of FIG. 65 shows an example in which coded data in which submesh IDs are assigned to the submeshes in ascending order is input to the submesh decoder 2403. (b) of FIG. 65 shows an example in which coded data in which the order of the submesh IDs assigned to the submeshes has been changed is input to the submesh decoder 2403. (c) of FIG. 65 shows an example in which one or more submeshes (in this example, three submeshes) are input to the submesh combiner 2404 when some submeshes are missing from the coded data. FIG. 65(d) shows an example in which coded data in which submesh IDs are assigned to the submeshes in a non-ascending order is input to the submesh decoder 2403.

[0535] Each rectangle in Fig. 65 is a schematic representation of a submesh, and the numerical value in the rectangle indicates the submesh ID of the submesh. Furthermore, the multiple submeshes (specifically, three or four submeshes) arranged in a horizontal row in Fig. 65 are acquired by the submesh decoder 2403 or the submesh combiner 2404, for example, in order from the right.

[0536] In the example shown in (a) of Figure 65, submeshes are acquired by the submesh decoder 2403 in the order of submesh IDs = 0, 1, 2, 3. In the example shown in (b) of Figure 65, submeshes are acquired by the submesh decoder 2403 in the order of submesh IDs = 0, 1, 3, 2. In the example shown in (c) of Figure 65, submeshes are acquired by the submesh decoder 2403 in the order of submesh IDs = 0, 1, 2, 3. In the example shown in (d) of Figure 65, submeshes are acquired by the submesh decoder 2403 in the order of submesh IDs = 2, 6, 9, 4.

[0537] FIG. 66 is a diagram illustrating an example of syntax in which sub-mesh combining metadata according to an embodiment is signaled.

[0538] The submesh combining metadata includes, for example, distance_per_submesh_flag and number_of_submesh. The submesh combining metadata also includes, for example, information for each submesh such as distance_per_submesh, or information for each combination of submeshes such as distance_per_submesh_pair_flag and distance_per_submesh_pair.

[0539] In the coded data, for example, information indicating the submesh ID is stored in the header, so that the submesh decoder 2403 can execute processing according to the submesh ID.

[0540] On the other hand, if a submesh ID is not indicated in the decoded submesh (decoded submesh), specifically, if the submesh decoder 2403 outputs the decoded submesh to the submesh combiner 2404 without assigning a submesh ID to it, the submesh combiner 2404 does not know the submesh ID of each decoded submesh (submesh ID = ? shown in Figure 65). In this case, the submesh combiner 2404, for example, determines the submesh ID itself and assigns a submesh ID to each decoded submesh. For example, the submesh combiner 2404 assigns a submesh ID to each decoded submesh in the order they are input to the submesh combiner 2404. In addition, the submesh combiner 2404 sets the number of submeshes input to the submesh combiner 2404 as the number of submeshes (number_of_submesh).

[0541] On the other hand, the submesh combining metadata stores the number of submeshes (specifically, the number of decoded submeshes input to the submesh combiner 2404) and combining information for each submesh (e.g., threshold information, etc.). The submesh combining metadata may also include, for example, combining information (threshold information) for each combination of submeshes to be combined.

[0542] Here, the submesh combiner 2404 performs processing by determining that the decoded submesh information is stored in order, assuming that the submesh IDs are in ascending order, for example. In this case, when combining certain submeshes, the submesh combiner 2404 obtains combining information corresponding to the submesh IDs generated (assigned) by the submesh combiner 2404 from the submesh combining metadata, and performs the submesh combining process.

[0543] In the example shown in (a) of Figure 65, since submesh IDs are assigned to each submesh in the coded data input to the submesh decoder 2403 in ascending order, the submesh combiner 2404 also assigns the correct submesh IDs to the four submeshes in the order of submesh ID = 0, 1, 2, 3.

[0544] However, in the case of such a method, in any of the following cases: (i) when the order of the submeshes is reversed (as in the example shown in (b) of Figure 65); (ii) when a submesh is missing or partially decoded (as in the example shown in (c) of Figure 65); and (iii) when the submesh IDs are not in ascending order (as in the example shown in (d) of Figure 65), the submesh combiner 2404 will assign an incorrect submesh ID to the submesh, rather than the correct submesh ID.

[0545] In the example shown in (b) of Figure 65, since submesh IDs are not assigned to each submesh in the encoded data input to the submesh decoder 2403 in ascending order, the submesh IDs = 2 and 3 are swapped in the submesh combiner 2404 and submesh IDs are assigned to the submeshes.

[0546] In the example shown in (c) of Figure 65, although submesh IDs are assigned in ascending order to each submesh in the coded data input to the submesh decoder 2403, only a portion of each submesh in the coded data input to the submesh decoder 2403 is input from the submesh decoder 2403 to the submesh combiner 2404. In this example, the submesh with submesh ID = 1 is not input to the submesh combiner 2404, and only three of the four submeshes input to the submesh decoder 2403 are input. Therefore, in the submesh combiner 2404, submesh IDs = 2 and 3 are assigned to the submeshes that should originally have submesh IDs = 3 and 4.

[0547] In the example shown in (d) of Figure 65, since submesh IDs are not assigned to each submesh in the encoded data input to the submesh decoder 2403 in ascending order, the submesh is assigned an incorrect submesh ID in the submesh combiner 2404.

[0548] Furthermore, if a submesh is missing or partially decoded, that is, in the example of (c) in Figure 65, the number of submeshes calculated by the submesh combiner 2404 will differ from the number of submeshes stored in the metadata by the encoding device 100, and the correspondence between the submeshes and the combining information will no longer be consistent.

[0549] Figure 67 is a diagram for explaining an example of syntax in which decoded data (decoded_frame) according to an embodiment is signaled. Specifically, Figure 67 shows the decoded sub-mesh data (output data) input to the sub-mesh combiner 2404.

[0550] The data includes submeshID, number_of_vertex, vertex_coordinate, uv_coordinate, number_of_face, and connectivity.

[0551] The number_of_reconstruction_submesh is information indicating the number of submeshes to be combined, i.e., the number_of_reconstruction_submesh indicates the number of submeshes input to the submesh combiner 2404.

[0552] Also, the submesh ID is information indicating the submesh ID of the submesh.

[0553] Also, number_of_vertex is information indicating the number of vertices (three-dimensional points) included in the submesh.

[0554] Furthermore, vertex_coordinate is information indicating the position (coordinates) of a vertex. vertex_coordinate_x, vertex_coordinate_y, and vertex_coordinate_z indicate the x-coordinate, y-coordinate, and z-coordinate of the vertex.

[0555] Furthermore, uv_coordinate is information indicating the UV coordinates of the texture map corresponding to the vertex, and uv_coordinate_u and uv_coordinate_v indicate the U and V coordinates of the texture map corresponding to the vertex.

[0556] Also, number_of_face is information indicating the number of faces included in the submesh.

[0557] The connectivity is information indicating the connectivity of the vertices included in the submesh, which corresponds to the number of faces included in the submesh. The connectivity information is information indicating, for example, whether the vertex is connected to other vertices.

[0558] In addition, submesh index numbers are assigned to vertex_coordinate, uv_coordinate, and connectivity, and the submesh ID of index number i is indicated as submeshID[i]. In other words, the index number indicates the submesh ID, and the submesh ID is assigned to this information.

[0559] The submesh combiner 2404 can accept input of a partial submesh in addition to input of a submesh for one frame that includes all the submeshes.

[0560] This syntax configuration allows the submesh ID to be associated with the decoded submesh data input to the submesh combiner 2404 .

[0561] Furthermore, even if the decoded submeshes input to the submesh combiner 2404 do not include all submeshes contained in one frame (e.g., all submeshes input to the submesh decoder 2403), and only some of the submeshes are input to the submesh combiner 2404 (e.g., when a submesh is missing and / or partially decoded), the submesh combiner 2404 can correctly combine the submeshes.

[0562] Furthermore, for example, decoded_frame may store (i) a flag indicating whether the decoded data obtained by decoding the encoded data is complete data for one frame or whether it is incomplete and some submeshes may not exist, (ii) a flag indicating that the order of the submeshes after decoding may be reversed from the order of the encoded data, and / or (iii) a flag indicating whether the submeshes are assigned in ascending order in the encoding device 100. The decoding device 200 may detect (determine) these pieces of information during decoding.

[0563] Depending on the communication standard, there is a possibility that information may be lost or reordered in the network. Therefore, for example, the decoding device 200 may determine, based on information indicating the communication standard used for communication with the encoding device 100, whether the decoded data obtained by decoding the encoded data is complete data for one frame and whether the order of the submeshes after decoding may be reordered from the order of the encoded data. The information indicating the communication standard may be stored in advance in a memory provided in the decoding device 200. Furthermore, these flags may be included in the bitstream.

[0564] For example, the submesh decoder 2403 may input data to the submesh combiner 2404 in a syntax structure that does not indicate a submesh ID if (i) the decoded data obtained by decoding the encoded data is complete data for one frame, (ii) there is no possibility that the order of the submeshes after decoding has been changed from the order of the encoded data, and (iii) the submeshes are assigned in ascending order in the encoding device 100, and in other cases, the data may be in a syntax structure that indicates a submesh ID.

[0565] 68 is a diagram for explaining an example of syntax in which sub-mesh combining metadata according to an embodiment is signaled. Specifically, FIG. 68 shows another example of syntax of sub-mesh combining metadata.

[0566] The number_of_connected_submesh is information indicating the number of submeshes that are connected to any other submeshes among the submeshes that make up the frame (specifically, the multiple submeshes included in one frame).

[0567] In addition, in this example, submesh_id indicates the submesh ID of a submesh that is to be combined with any other submesh.

[0568] The number_of_connected_submesh2 is information indicating the number of submeshes to be connected, for which connection information is indicated, among one or more submeshes to be connected to the submesh of submesh_id.

[0569] In addition, the number of submeshes to which the submesh of submesh_id is combined may be indicated, or a number smaller than the number of submeshes to which the submesh of submesh_id is combined (for example, the number of submeshes to which the submesh of submesh_id is combined - 1) may be indicated.

[0570] The submesh_id2 indicates the submesh ID of the submesh to be combined, for which the combining information is indicated, among one or more submeshes to be combined with the submesh of the submesh_id.

[0571] In this way, by indicating the submesh ID, the submesh decoder 2403 can indicate the combination information of only the combination of submeshes to be combined.

[0572] Furthermore, since only the combination information of the combination of sub-meshes to be combined is displayed, the amount of metadata information input to the sub-mesh combiner 2404 can be reduced.

[0573] For example, if there are 10 submeshes, and the combined information is displayed for all combinations, 10 x 10 = 100 pieces of combined information will be displayed. Here, if the combined submesh with submesh ID = A and the combined submesh with submesh ID = B are expressed as [A, B], when the number of combined submeshes is small, for example, when the combined combinations of the 10 submeshes are [A, B] = [1, 2], [1, 10], and [2, 3], only three pieces of combined information are required, and the amount of information in the metadata can be reduced.

[0574] FIG. 69 is a flowchart showing a processing procedure when a three-dimensional mesh is presented by an application according to an embodiment.

[0575] First, the decoding device 200 acquires instruction information indicating an instruction for a submesh to be restored from an application (S741). A user inputs an instruction for a submesh to be restored using the application by operating an operating device such as a touch panel of a computer device such as a smartphone. As a result, the decoding device 200 acquires the instruction information from the application. The submesh to be restored is, for example, a submesh (specifically, a three-dimensional mesh formed by combining multiple submeshes) displayed as an image on a display device such as a display of a computer device such as a smartphone used by the user.

[0576] Next, the decoding device 200 decodes and combines some or all of the specified submeshes (S742). For example, the decoding device 200 acquires two or more pieces of coded data including some or all of the specified submeshes from the bitstream, decodes the acquired two or more pieces of coded data to generate two or more submeshes, and combines the generated two or more submeshes to generate a three-dimensional mesh. In this manner, the decoding device 200 may decode only some of the coded submeshes included in the bitstream. Alternatively, the decoding device 200 may generate a three-dimensional mesh in which all of the submeshes are combined, or may generate a partial three-dimensional mesh in which some of the submeshes are combined.

[0577] Next, the decoding device 200 outputs the 3D mesh to the application (S743). Specifically, the decoding device 200 outputs information about the 3D mesh to a computer device that uses the application.

[0578] As a result, the decoding device 200 causes the application to present (display) the three-dimensional mesh on a display device or the like of the computer device (S744).

[0579] The decoding device 200 may be included in the computer device, or may not be included in the computer device but may be communicably connected to the computer device.

[0580] 70 is a flowchart showing the sub-mesh combining process according to the embodiment, specifically showing the details of the process of step S743 in FIG.

[0581] First, the decoding device 200 decodes some or all of the specified submeshes (S751). For example, the submesh decoder 2403 acquires two or more pieces of coded data including some or all of the specified submeshes from the bitstream, and generates two or more submeshes (decoded submeshes) by decoding the acquired two or more pieces of coded data.

[0582] Next, the decoding device 200 starts the process of outputting the decoded submesh (S752). Specifically, the decoding device 200 executes the processes from step S753 onwards.

[0583] The decoding device 200 determines whether to execute submesh combining processing (S753). For example, the submesh decoder 2403 determines whether to combine submeshes by determining whether submesh combining metadata has been acquired (e.g., whether submesh combining metadata is included in the bitstream). For example, the submesh decoder 2403 determines to combine submeshes if submesh combining metadata has been acquired, and determines not to combine submeshes if submesh combining metadata has not been acquired. The submesh decoder 2403 may also determine whether to combine submeshes based on information indicating whether to combine submeshes. The information indicating whether to combine submeshes may be included in the bitstream or may be acquired from an application or the like. The submesh decoder 2403 may also determine whether to combine submeshes based on whether submesh combining metadata has been acquired and information indicating whether to combine submeshes. The submesh decoder 2403 may, for example, perform the submesh combining process only when it has acquired metadata for submesh combining and the information indicating whether to combine the submeshes indicates that the submeshes are to be combined.

[0584] When the decoding device 200 determines not to perform the submesh combining process (No in S753), it generates output data that does not include information indicating the submesh ID (S757). Specifically, the submesh decoder 2403 generates output data that does not include information indicating the submesh ID of the decoded submesh, and outputs the generated output data to the submesh combiner 2404.

[0585] On the other hand, when the decoding device 200 determines to execute the submesh combining process (Yes in S753), it generates output data including information indicating the submesh ID and information indicating the number of submeshes (S754). Specifically, the submesh decoder 2403 generates output data including information indicating the submesh ID of the decoded submesh to be input to the submesh combiner 2404 and information indicating the number of decoded submeshes to be input to the submesh combiner 2404, and outputs the generated output data to the submesh combiner 2404.

[0586] Next, the decoding device 200 acquires information indicating the submesh to be processed (for example, the submesh including the target boundary vertex) and the submesh ID of the submesh from the output data (S755). Specifically, the submesh combiner 2404 acquires information indicating the decoded submesh to be processed and the submesh ID of the decoded submesh from the output data.

[0587] Next, the decoding device 200 obtains the submesh of the submesh ID to be processed from the metadata included in the bitstream, and combines the submeshes using the obtained combination information (S756). Specifically, the submesh combiner 2404 obtains the submesh of the submesh ID to be processed from the metadata included in the bitstream, and combines the decoded submeshes using the obtained combination information. When the bitstream includes combination information for each combination of submeshes, the submesh combiner 2404 obtains, for example, the submesh of the submesh ID to be processed and the submesh of the submesh ID to be combined with the submesh to be processed.

[0588] The encoding device 100 may include information indicating whether to perform sub-mesh combining in metadata (e.g., a parameter set or SEI). The information indicating whether to perform sub-mesh combining may be indicated, for example, by a flag indicating whether to perform sub-mesh reconstruction processing in a profile that defines a group of tools supported in decoding, or may indicate a flag indicating whether to perform sub-mesh combining.

[0589] In addition, the encoding device 100 may include in the metadata, for example, a flag indicating whether or not sub-mesh combining may be performed, or a flag indicating whether or not sub-mesh combining is likely to be performed, rather than whether or not sub-mesh combining will be performed.

[0590] For example, when the encoding device 100 includes information indicating that a sub-mesh reconstruction process is to be performed in the metadata, the encoding device 100 transmits a bitstream including the sub-mesh combination metadata.

[0591] Furthermore, the encoding device 100 may store in the bitstream (i) a flag indicating whether the decoded data obtained by decoding the encoded data is complete data for one frame or is incomplete and some submeshes may be missing, (ii) a flag indicating whether the order of the submeshes after decoding may be reversed from the order in the encoded data, and / or (iii) a flag indicating whether the submeshes are assigned in ascending order in the encoding device 100. For example, the encoding device 100 may determine whether the decoded data obtained by decoding the encoded data is complete data for one frame and whether the order of the submeshes after decoding may be reversed from the order in the encoded data, based on information indicating the communication standard used for communication with the decoding device 200. The information indicating the communication standard may be stored in advance in a memory provided in the encoding device 100.

[0592] Furthermore, the submesh decoder 2403 may decode some or all of the coded submeshes in the coded data included in the bitstream, and then rearrange some or all of the decoded submeshes in coding order before outputting them to the submesh combiner 2404. In this case, the submesh decoder 2403 may rewrite each of the above flags and output the rewritten flags to the submesh combiner 2404.

[0593] <Representative Example> Fig. 71 is a flow diagram showing an example of basic encoding processing according to this embodiment. For example, the encoding device 100 shown in Fig. 24 includes a circuit 151 and a memory 152 connected to the circuit 151. In the encoding device 100, the circuit 151 performs the following processing during operation.

[0594] First, the encoding device 100 generates a plurality of encoded data by encoding a plurality of submeshes (S761). The encoding device 100 encodes the plurality of submeshes by encoding, for example, vertex information and connection information of each of the plurality of submeshes (specifically, base mesh information of each of the plurality of submeshes), as well as information about the plurality of submeshes, such as displacement vector information.

[0595] Next, the encoding device 100 generates threshold information indicating a threshold used in a combining process for combining multiple three-dimensional points included in multiple submeshes based on the positions of multiple three-dimensional points included in the multiple submeshes (S762).

[0596] The three-dimensional point is, for example, the boundary vertex described above. The threshold information is, for example, the submesh_id2 described above or the distance_per_submesh_pair described above.

[0597] Next, the encoding device 100 generates a bitstream including a plurality of pieces of encoded data, threshold information, and a plurality of pieces of submesh identification information that uniquely indicate each of a plurality of submeshes (S763).

[0598] The submesh identification information is, for example, information indicating the submesh ID assigned to the above-mentioned submesh (specifically, the submesh in the coded data).

[0599] According to this, the decoding device 200 that has acquired the bitstream can correctly determine the submeshes to be joined based on the submesh identification information and perform joining processing.

[0600] Furthermore, for example, the threshold information indicates a threshold determined for each combination of two submeshes selected from a plurality of submeshes.

[0601] For example, if a threshold is set based only on the distance between three-dimensional points, when there are multiple submeshes to be combined with one submesh, there is a possibility that a three-dimensional point belonging to a submesh other than the submesh containing the three-dimensional point that should be combined may be combined. Therefore, by setting a threshold for each combination of submeshes, three-dimensional points can be appropriately combined. Furthermore, for two submeshes that are not combined, a specific value, such as 0, can be set as the threshold information. This allows the decoding device 200 to omit the process of determining whether or not to combine each three-dimensional point for those two submeshes. This reduces the amount of processing by the decoding device 200.

[0602] Furthermore, for example, the threshold information indicates a threshold determined for each of one or more sub-meshes.

[0603] According to this, the decoding device 200 that has acquired the bitstream can correctly determine the submeshes to be joined based on the submesh identification information and perform joining processing.

[0604] Fig. 72 is a flow diagram showing an example of basic decoding processing according to this embodiment. For example, the decoding device 200 shown in Fig. 25 includes a circuit 251 and a memory 252 connected to the circuit 251. In the decoding device 200, the circuit 251 performs the following processing during operation.

[0605] First, the decoding device 200 acquires a bitstream (S771) including multiple encoded data generated by encoding multiple submeshes, threshold information indicating one or more thresholds used in a joining process to join multiple three-dimensional points included in the multiple submeshes, and multiple submesh identification information that uniquely identifies each of the multiple submeshes.

[0606] Next, the decoding device 200 performs a decoding process to decode the plurality of coded data (S772), thereby generating a plurality of sub-meshes (specifically, vertex information and connectivity information of each of the plurality of sub-meshes (specifically, base mesh information of each of the plurality of sub-meshes), and information related to the plurality of sub-meshes such as displacement vector information).

[0607] Next, the decoding device 200 performs a combining process based on threshold information and two or more pieces of submesh identification information from among the plurality of pieces of submesh identification information (S773). Specifically, the decoding device 200 combines the plurality of submeshes generated by the decoding process based on threshold information and two or more pieces of submesh identification information. Note that the number of two or more pieces of submesh identification information may be determined arbitrarily and is not particularly limited. The two or more pieces of submesh identification information may be all or part of the plurality of submesh identification information included in the bitstream. The two or more pieces of submesh identification information are, for example, the submesh IDs included in the above-mentioned decoded_frame.

[0608] This allows the sub-meshes to be joined to be correctly determined based on the sub-mesh identification information and the joining process to be performed.

[0609] Furthermore, for example, the decoding device 200 further performs a generation process after the decoding process to generate output data based on the submesh identification information, the output data including two or more submeshes among the multiple submeshes for which a combining process is performed using threshold information and two or more submesh identification information among the multiple submesh identification information, each of which uniquely identifies one of the two or more submeshes. In the combining process, the combining process is performed based on the threshold information and the two or more submesh identification information included in the output data. For example, one processor in the decoding device 200 generates output data including submesh identification information indicating the multiple submeshes based on the submesh identification information included in the bitstream, and outputs the generated output data to another processor. The other processor performs the combining process based on the submesh identification information included in the output data.

[0610] That is, the combining process uses submesh identification information included in the output data generated by the generation process. The bit stream includes, for example, submesh identification information indicating each of the multiple submeshes included in the encoded data, in other words, the multiple encoded submeshes. In the generation process, output data including submesh identification numbers (second submesh identification information) uniquely indicating each of the multiple submeshes generated by the decoding process, i.e., the multiple decoded submeshes (more specifically, two or more submeshes to be combined among the multiple decoded submeshes), is generated based on the submesh identification information (first submesh identification information) included in the bit stream. As a result, in the combining process, for example, the submesh identification numbers of the multiple decoded submeshes are determined using the submesh identification information included in the output data, without performing a process assuming the submesh identification numbers of the multiple decoded submeshes. For example, if the combining process is performed assuming that the order in which the encoded data of multiple submeshes are decoded is the order of the submesh identification numbers of the multiple submeshes, if some of the encoded data is missing when the bitstream is transmitted from the encoding device 100 to the decoding device 200, a submesh different from the intended target submesh may be selected. In this case, the combining process will not be performed properly. Therefore, after the decoding process, the submesh identification numbers of the multiple submeshes are clearly determined before the combining process is performed, so that the submeshes to be combined can be correctly determined and the combining process can be performed.

[0611] Moreover, for example, the output data further includes information indicating the number of sub-meshes, which may be two or more.

[0612] The information indicating the number of sub-meshes, which is two or more, is, for example, the above-mentioned number_of_reconstruction_submesh.

[0613] This allows the process of using two or more sub-meshes in the combining process to be carried out appropriately.

[0614] <Other Examples> Although aspects of the encoding device 100 and the decoding device 200 have been described above according to the embodiments, the aspects of the encoding device 100 and the decoding device 200 are not limited to the embodiments. Modifications conceivable by those skilled in the art may be applied to the embodiments, and multiple components in the embodiments may be combined in any manner.

[0615] For example, a process performed by a specific component in the embodiment may be performed by another component instead of the specific component. Also, the order of multiple processes may be changed, or multiple processes may be performed in parallel.

[0616] Furthermore, as described above, at least some of the configurations of the present disclosure may be implemented as an integrated circuit. At least some of the processes of the present disclosure may be used as an encoding method or a decoding method. A program for causing a computer to execute the encoding method or the decoding method may be used. A non-transitory computer-readable recording medium on which the program is recorded may be used. A bitstream for causing the decoding device 200 to perform a decoding process may be used.

[0617] Furthermore, at least some of the configurations and processes of the present disclosure may be used as a transmitting device, a receiving device, a transmitting method, or a receiving method. A program for causing a computer to execute the transmitting method or the receiving method may be used. Furthermore, a non-transitory computer-readable recording medium on which the program is recorded may be used.

[0618] The present disclosure is useful, for example, in encoding devices, decoding devices, transmitting devices, receiving devices, etc. related to three-dimensional meshes, and is applicable to computer graphics systems, three-dimensional data display systems, etc.

[0619] 100 Encoding device 101, 121, 144 Vertex information encoder 102, 145 Connection information encoder 103, 122 Attribute information encoder 104, 204, 1103 Preprocessor 105, 205, 2106 Postprocessor 110 Three-dimensional data encoding system 111, 211 Controller 112, 212 Input / output processor 113 Three-dimensional data encoder 114 System multiplexer 115 Three-dimensional data generator 123 Metadata encoder 124, 1274 Multiplexer 131 Vertex image generator 132 Attribute image generator 133 Metadata generator 134 Video encoder 141 Two-dimensional data encoder 142 Mesh data encoder 143 Texture encoder 148 Description encoder 151, 251 Circuit 152, 252 Memory 200 Decoding device 201, 221, 244 Vertex information decoder 202, 245 Connection information decoder 203, 222 Attribute information decoder 210 3D data decoding system 213 3D data decoder 214 System demultiplexer 215, 247 Presentation device 216 User interface 223 Metadata decoder 224 Demultiplexer 231 Vertex information generator 232 Attribute information generator 234 Video decoder 241 2D data decoder 242 Mesh data decoder 243 Texture decoder 246 Mesh reconstructor 248 Description decoder 300 Network 310 External connector 511 Volumetric capture device 512 Projector 513 Base mesh encoder 514 Displacement encoder 515 Attribute Encoder 516 Other Type Encoder 613 Base Mesh Decoder 614 Displacement Decoder 615 Attribute Decoder 616 Other Type Decoder 617 3D Reconstructor 1101 Input Mesh 1102, 2108 Attribute Map 1104, 2103 Base Mesh 1105, 2104 Displacement Data 1106 Compressor 1107, 1304, 2101 Bitstream1108, 2105 Metadata 1231 Demultiplexer 1232, 1262 Switch 1233 Static Mesh Decoder 1234, 1264 Mesh Buffer 1235 Motion Decoder 1236, 1266 Base Mesh Reconstructor 1237, 1240 Inverse Quantizer 1238, 1243 Video Decoder 1239 Image Unpacker 1241 Inverse Wavelet Transformer 1242 Reconstructor 1244, 1272 Color Transformer 1251 Decoded Base Mesh 1252, 1402 Subdivider 1253 Subdivided Mesh 1254 Decoded Displacement Data 1255 Displacer 1256 Decoded 3D Mesh 1261, 1269 Quantizer 1263 Static Mesh Encoder 1265 Motion Encoder 1267 Displacement data updater 1268 Wavelet transformer 1270 Image packer 1271, 1273 Video encoder 1301, 2304 Mesh frame 1302, 2301, 2302 Base mesh frame 1303, 2303 Displacement information 1401 Base mesh generator 1403 Displacement data generator 2102 Decompressor 2107 Output mesh 2109 Combiner 2305 Decoder 2306 Post-decoder 2307 Pre-reconstructor 2308 Reconstructor 2309 Post-reconstructor 2310 Fitter 2311 Final 3D mesh frame 2401 Sub-mesh divider 2402 Sub-mesh encoder 2403 Sub-mesh decoder 2404 Sub-mesh combiner

Claims

1. An encoding method comprising: generating a plurality of encoded data by encoding a plurality of submeshes; generating threshold information indicating a threshold used in a combining process for combining a plurality of three-dimensional points included in the plurality of submeshes based on the positions of a plurality of three-dimensional points included in the plurality of submeshes; and generating a bitstream including the plurality of encoded data, the threshold information, and a plurality of submesh identification information uniquely identifying each of the plurality of submeshes.

2. The encoding method according to claim 1, wherein the threshold information indicates a threshold determined for each combination of two submeshes selected from the plurality of submeshes.

3. The encoding method according to claim 1, wherein the threshold information indicates a threshold determined for each of the plurality of sub-meshes.

4. A decoding method comprising: acquiring a bitstream including a plurality of coded data generated by coding a plurality of submeshes, threshold information indicating one or more thresholds used in a joining process for joining a plurality of three-dimensional points included in the plurality of submeshes, and a plurality of submesh identification information uniquely identifying each of the plurality of submeshes; performing a decoding process to decode the plurality of coded data; and performing the joining process based on the threshold information and two or more submesh identification information among the plurality of submesh identification information.

5. The decoding method according to claim 4, further comprising: after the decoding process, a generation process is performed to generate output data based on the plurality of submesh identification information, the output data including two or more submeshes among the plurality of submeshes for which the combining process is performed using the threshold information, and the two or more submesh identification information among the plurality of submesh identification information that uniquely indicates each of the two or more submeshes; and in the combining process, the combining process is performed based on the threshold information and the two or more submesh identification information included in the output data.

6. The decoding method according to claim 5, wherein the output data further includes information indicating the number of the two or more sub-meshes.

7. An encoding device comprising: a processor; and a memory, wherein the processor uses the memory to generate a plurality of encoded data by encoding a plurality of submeshes; generate threshold information indicating a threshold used in a combining process for combining a plurality of three-dimensional points included in the plurality of submeshes based on the positions of a plurality of three-dimensional points included in the plurality of submeshes; and generate a bit stream including the plurality of encoded data, the threshold information, and a plurality of submesh identification information uniquely identifying each of the plurality of submeshes.

8. A decoding device comprising: a processor; and a memory, wherein the processor uses the memory to obtain a bit stream including a plurality of coded data generated by coding a plurality of submeshes, threshold information indicating one or more thresholds used in a joining process for joining a plurality of three-dimensional points included in the plurality of submeshes, and a plurality of submesh identification information uniquely identifying each of the plurality of submeshes, performs a decoding process to decode the plurality of coded data, and performs the joining process based on the threshold information and two or more submesh identification information among the plurality of submesh identification information.

Images