Encoding device, decoding device, encoding method, and decoding method

Abstract

An encoding device according to one aspect of the present disclosure comprises a circuit, and a memory connected to the circuit. During operation, the circuit determines whether a first vertex, among a plurality of vertices comprised by a first sub-mesh divided from a three-dimensional mesh, which is not positioned on a side constituting the outer periphery of the first sub-mesh overlaps with a vertex comprised by a second sub-mesh divided from the three-dimensional mesh (S401), and on the basis of the determination result, encodes first information indicating whether the first vertex overlaps with the vertex comprised by the second sub-mesh into a bitstream (S402).

Description

Encoding device, decoding device, encoding method, and decoding method

[0001] The present disclosure relates to an encoding device and the like.

[0002] In US Pat. No. 6,299,549 a method and apparatus for encoding and decoding three-dimensional mesh data is proposed.

[0003] Japanese Patent Application Laid-Open No. 2006-187015

[0004] Further improvements are desired in the encoding or decoding process for three-dimensional data. The present disclosure aims to improve the encoding or decoding process for three-dimensional data.

[0005] An encoding device according to one aspect of the present disclosure comprises a circuit and a memory connected to the circuit, wherein during operation the circuit determines whether a first vertex, among a plurality of vertices of a first submesh divided from a three-dimensional mesh, that is not located on an edge that constitutes the periphery of the first submesh, overlaps with a vertex of a second submesh divided from the three-dimensional mesh, and, based on the result of the determination, encodes first information indicating whether the first vertex overlaps with a vertex of the second submesh into a bitstream.

[0006] These comprehensive or specific aspects may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0007] The present disclosure may contribute to improvements in encoding processes and the like related to three-dimensional data.

[0008] 1 is a conceptual diagram showing a three-dimensional mesh according to an embodiment. FIG. 2 is a conceptual diagram showing basic elements of a three-dimensional mesh according to an embodiment. FIG. 3 is a conceptual diagram showing mapping according to an embodiment. FIG. 4 is a block diagram showing a configuration example of an encoding / decoding system according to an embodiment. FIG. 5 is a block diagram showing a configuration example of an encoding device according to an embodiment. FIG. 6 is a block diagram showing another configuration example of an encoding device according to an embodiment. FIG. 7 is a block diagram showing another configuration example of a decoding device according to an embodiment. FIG. 8 is a block diagram showing another configuration example of a decoding device according to an embodiment. FIG. 9 is a block diagram showing another configuration example of a decoding device according to an embodiment. FIG. 10 is a conceptual diagram showing another configuration example of a bit stream according to an embodiment. FIG. 11 is a conceptual diagram showing yet another configuration example of a bit stream according to an embodiment. FIG. 12 is a block diagram showing a specific example of an encoding / decoding system according to an embodiment. FIG. 13 is a conceptual diagram showing an example configuration of point cloud data according to an embodiment. FIG. 14 is a conceptual diagram showing an example data file of point cloud data according to an embodiment. FIG. 15 is a conceptual diagram showing an example configuration of mesh data according to an embodiment. FIG. 16 is a conceptual diagram showing an example data file of mesh data according to an embodiment. FIG. 17 is a conceptual diagram showing types of three-dimensional data according to an embodiment. FIG. 18 is a block diagram showing an example configuration of a three-dimensional data encoder according to an embodiment. FIG. 19 is a block diagram showing an example configuration of a three-dimensional data decoder according to an embodiment. FIG. 19 is a block diagram showing another configuration example of a three-dimensional data encoder according to an embodiment. FIG. 19 is a block diagram showing another configuration example of a three-dimensional data decoder according to an embodiment. FIG. 1 is a conceptual diagram showing a specific example of encoding processing according to an embodiment. FIG. 2 is a conceptual diagram showing a specific example of decoding processing according to an embodiment. FIG. 3 is a block diagram showing an implementation example of an encoding device according to an embodiment. FIG. 4 is a block diagram showing an implementation example of a decoding device according to an embodiment. FIG. 5 is a block diagram showing another configuration example of an encoding / decoding system according to an embodiment. FIG. 6 is a block diagram showing another configuration example of an encoding device according to an embodiment. FIG. 7 is a block diagram showing another configuration example of a decoding device according to an embodiment. FIG. 8 is a block diagram showing a detailed configuration example of an encoding device according to an embodiment. FIG. 9 is a block diagram showing a modified example of the detailed configuration of an encoding device according to an embodiment. FIG. 10 is a flow diagram showing processing of an encoding device according to an embodiment. FIG. 11 is an explanatory diagram conceptually showing encoding of a mesh frame according to an embodiment.1 is a block diagram showing a detailed configuration example of a decoding device according to an embodiment; FIG. 2 is a block diagram showing a modified example of the detailed configuration of a decoding device according to an embodiment; FIG. 3 is a flow diagram showing processing by a decoding device according to an embodiment; FIG. 4 is an explanatory diagram conceptually showing decoding of a mesh frame according to an embodiment; FIG. 5 is an explanatory diagram showing an example of subdivision according to an embodiment; FIG. 6 is an explanatory diagram showing an example of displacement of vertices after displacement after subdivision according to an embodiment; FIG. 7 is an explanatory diagram showing an example of vertices of an original mesh according to an embodiment; FIG. 8 is an explanatory diagram showing an example of a mesh according to an embodiment; FIG. 9 is an explanatory diagram showing an example of division of a mesh into sub-meshes according to an embodiment; FIG. 10 is a first explanatory diagram showing an example of packing of displacement information into an image frame according to an embodiment; FIG. 11 is a second explanatory diagram showing an example of packing of displacement information into an image frame according to an embodiment; FIG. 12 is a third explanatory diagram showing an example of packing of displacement information into an image frame according to an embodiment; FIG. 13 is a block diagram showing another example of the configuration of an encoding device according to an embodiment; FIG. 14 is a block diagram showing an example of the configuration of a preprocessor according to an embodiment; FIG. 15 is a block diagram showing another example of the configuration of a decoding device according to an embodiment; FIG. 16 is a block diagram showing a specific example of the configuration of a decoding device according to an embodiment; FIG. 17 is a flow diagram showing processing for sub-mesh boundaries in an encoding device according to an embodiment; FIG. 18 is a flow diagram showing an example of processing for sub-mesh boundaries in a decoding device according to an embodiment; 1 is a flow diagram showing processing for a submesh of a decoding device according to an embodiment; FIG. 2 is a diagram showing an example of a submesh according to an embodiment; FIG. 3 is a diagram showing another example of a submesh according to an embodiment; FIG. 4 is a diagram showing another example of a submesh according to an embodiment; FIG. 5 is a diagram showing an example of syntax related to signaling of boundary information according to an embodiment; FIG. 6 is a diagram showing another example of syntax related to signaling of boundary information according to an embodiment; FIG. 7 is a diagram explaining an example of a method of using syntax according to an embodiment; FIG. 8 is a diagram explaining an order in which vertices are processed according to an embodiment; FIG. 9 is a diagram explaining an example of an order in which vertices are processed according to an embodiment; FIG. 10 is a diagram explaining another example of an order in which vertices are processed according to an embodiment.FIG. 1 is a diagram for explaining another example of the order in which vertices are processed according to an embodiment; FIG. 2 is a diagram showing an example of syntax related to signaling the number of sub-meshes according to an embodiment; FIG. 3 is a diagram showing another example of syntax related to signaling boundary information according to an embodiment; FIG. 4 is a flow diagram showing an example of a basic encoding process according to an embodiment; and FIG. 5 is a flow diagram showing an example of a basic decoding process according to an embodiment.

[0009] Introduction Three-dimensional (3D) meshes are used in computer graphics images, which may be composed of multiple temporally distinct frames, each of which may be represented by a 3D mesh.

[0010] A 3D mesh is composed of vertex information indicating the positions of each of the vertices in 3D space, connectivity information indicating the connections between the vertices, and attribute information indicating the attributes of each vertex or face. Each face is constructed according to the connectivity between the vertices. Various computer graphics images can be expressed using such 3D meshes.

[0011] Furthermore, for transmission and storage of the 3D mesh, efficient encoding and decoding of the 3D mesh is expected. For efficient encoding and decoding of the 3D mesh, arithmetic coding and decoding may be used.

[0012] Further improvements are desired in the encoding or decoding process for three-dimensional data. The present disclosure aims to improve the encoding or decoding process for three-dimensional data.

[0013] Below, examples of inventions that can be obtained from the disclosure of this specification will be given, and the effects and the like that can be obtained from these inventions will be explained.

[0014] The encoding device of Example 1 comprises a circuit and a memory connected to the circuit, and during operation, the circuit determines whether a first vertex, among a plurality of vertices of a first submesh divided from a three-dimensional mesh, that is not located on an edge that constitutes the periphery of the first submesh, overlaps with a vertex of a second submesh divided from the three-dimensional mesh, and, based on the result of the determination, encodes first information indicating whether the first vertex overlaps with a vertex of the second submesh into a bitstream.

[0015] According to this, a decoding device that receives a bitstream can determine whether or not there is overlap by simply checking the first information, without having to perform a process of decoding the first submesh and the second submesh to determine whether or not there is overlap, thereby reducing the amount of processing.

[0016] The encoding device of Example 2 is the encoding device of Example 1, and may further encode into the bitstream second information indicating a second vertex, among two or more vertices among the plurality of vertices, on which a predetermined process is performed first, among two or more vertices on which a predetermined process is performed in sequence.

[0017] According to this, the decoding device that receives the bitstream can first perform a predetermined process on the second vertex among the two or more vertices based on the second information, and therefore can execute the predetermined process on the two or more vertices in an appropriate order.

[0018] The encoding device of Example 3 is the encoding device of Example 1 or Example 2, and may further encode into the bitstream third information indicating a third vertex, which is one of two or more vertices among the plurality of vertices and on which a predetermined process is performed in sequence, and which is the last vertex on which the predetermined process is performed.

[0019] According to this, the decoding device that has received the bitstream can perform the predetermined processing on the third vertex of the two or more vertices last based on the third information, and therefore can execute the predetermined processing on the two or more vertices in an appropriate order.

[0020] The encoding device of Example 4 is any of the encoding devices of Examples 1 to 3, and may further encode fourth information into the bitstream, the fourth information indicating the order of the plurality of vertices when a predetermined process is performed on each of the plurality of vertices.

[0021] This allows the decoding device that receives the bitstream to perform predetermined processing on two or more vertices in order based on the fourth information, thereby enabling the decoding device to perform predetermined processing on two or more vertices in an appropriate order.

[0022] The decoding device of Example 5 comprises a circuit and a memory connected to the circuit, and during operation, the circuit decodes from a bit stream first information indicating whether a first vertex, among multiple vertices of a first submesh divided from a three-dimensional mesh, that is not located on an edge that forms the periphery of the first submesh, overlaps with a second submesh divided from the three-dimensional mesh, and determines whether the first vertex overlaps with the second submesh based on the first information.

[0023] According to this, the decoding device can determine whether or not there is an overlap by simply checking the first information, without performing a process of decoding the first submesh and the second submesh to determine whether or not there is an overlap, for example, thereby reducing the amount of processing.

[0024] The decoding device of Example 6 is the decoding device of Example 5, and may further decode from the bitstream second information indicating a second vertex, which is one of two or more vertices among the plurality of vertices and on which a predetermined process is performed in sequence, and on which the predetermined process is performed first.

[0025] This allows the decoding device to first perform the predetermined process on the second vertex among the two or more vertices based on the second information, thereby enabling the decoding device to execute the predetermined process on the two or more vertices in an appropriate order.

[0026] The decoding device of Example 7 is the decoding device of Example 5 or Example 6, and may further decode from the bitstream third information indicating a third vertex, which is one of two or more vertices among the plurality of vertices and on which a predetermined process is performed last, among the two or more vertices on which a predetermined process is performed in sequence.

[0027] This allows the decoding device to perform the predetermined process on the third vertex last among the two or more vertices based on the third information, thereby enabling the decoding device to execute the predetermined process on the two or more vertices in an appropriate order.

[0028] The decoding device of Example 8 is any of the decoding devices of Examples 5 to 7, and may further decode fourth information from the bitstream that indicates the order of the multiple vertices when a predetermined processing is performed on each of the multiple vertices.

[0029] This allows the decoding device to perform predetermined processing on two or more vertices in order based on the fourth information, thereby enabling the decoding device to execute predetermined processing on two or more vertices in an appropriate order.

[0030] The encoding method of Example 9 determines whether a first vertex, among multiple vertices of a first submesh divided from a three-dimensional mesh, that is not located on an edge that constitutes the outer periphery of the first submesh, overlaps with a vertex of a second submesh divided from the three-dimensional mesh, and, based on the result of the determination, encodes first information indicating whether the first vertex overlaps with a vertex of the second submesh into a bitstream.

[0031] This provides the same effect as the encoding device of Example 1.

[0032] The decoding method of Example 10 decodes, from a bitstream, first information indicating whether a first vertex, among multiple vertices of a first submesh divided from a three-dimensional mesh, that is not located on an edge that forms the outer periphery of the first submesh, overlaps with a vertex of a second submesh divided from the three-dimensional mesh, and determines, based on the first information, whether the first vertex overlaps with a vertex of the second submesh.

[0033] This provides the same effect as the decoding device of Example 5.

[0034] Furthermore, these comprehensive or specific aspects may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0035] <Expressions and Terms> The following expressions and terms are used herein.

[0036] (1) Three-dimensional Mesh A three-dimensional mesh is a collection of multiple faces, and represents, for example, a three-dimensional object. A three-dimensional mesh is mainly composed of vertex information, connectivity information, and attribute information. A three-dimensional mesh may be expressed as a polygon mesh or a mesh. A three-dimensional mesh may also vary over time. A three-dimensional mesh may include metadata related to the vertex information, connectivity information, and attribute information, and may also include other additional information.

[0037] (2) Vertex Information Vertex information is information indicating a vertex. For example, the vertex information indicates the position of a vertex in a three-dimensional space. Furthermore, a vertex corresponds to a vertex of a face that constitutes a three-dimensional mesh. Vertex information may be expressed as "geometry." Furthermore, vertex information may be expressed as position information.

[0038] (3) Connection Information Connection information is information that indicates connections between vertices. For example, connection information indicates connections for forming faces or edges of a three-dimensional mesh. Connection information may be expressed as "Connectivity." Connection information may also be expressed as face information.

[0039] (4) Attribute Information Attribute information is information that indicates attributes of a vertex or a face. For example, attribute information indicates attributes such as a color, an image, and a normal vector associated with a vertex or a face. Attribute information may be expressed as "texture."

[0040] (5) Faces A face is an element that makes up a three-dimensional mesh. Specifically, a face is a polygon on a plane in three-dimensional space. For example, a face can be defined as a triangle in three-dimensional space.

[0041] (6) Plane A plane is a two-dimensional plane in a three-dimensional space. For example, a polygon is formed on a plane, and multiple polygons are formed on multiple planes.

[0042] (7) Bitstream: A bitstream corresponds to coded information. A bitstream may also be referred to as a stream, a coded bitstream, a compressed bitstream, or a coded signal.

[0043] (8) Encoding and Decoding The term encoding may be substituted with terms such as storing, including, writing, describing, signaling, sending, notifying, saving, or compressing, and these terms may be interchangeable. For example, encoding information may mean including the information in a bitstream. Also, encoding information into a bitstream may mean encoding the information to generate a bitstream that includes the encoded information.

[0044] Additionally, the term "decode" may be replaced with terms such as "read," "decode," "read," "load," "derive," "obtain," "receive," "extract," "reconstruct," "reconstruct," "decompress," or "decompress," and these terms may be interchangeable. For example, decoding information may mean obtaining information from a bitstream. Decoding information from a bitstream may mean decoding the bitstream to obtain information contained in the bitstream.

[0045] (9) Ordinal Numbers In the description, ordinal numbers such as first and second may be assigned to components, etc. These ordinal numbers may be changed as appropriate. Furthermore, new ordinal numbers may be assigned to components, etc., or removed. Furthermore, these ordinal numbers may be assigned to elements in order to identify them, and may not correspond to a meaningful order.

[0046] <Three-dimensional mesh> Fig. 1 is a conceptual diagram showing a three-dimensional mesh according to this embodiment. A three-dimensional mesh is composed of multiple faces. For example, each face is a triangle. The vertices of these triangles are defined in three-dimensional space. The three-dimensional mesh then represents a three-dimensional object. Each face may have a color or an image.

[0047] FIG. 2 is a conceptual diagram showing the basic elements of a three-dimensional mesh according to this embodiment. A three-dimensional mesh is composed of vertex information, connection information, and attribute information. The vertex information indicates the positions of the vertices of a face in three-dimensional space. The connection information indicates the connections between the vertices. A face can be identified by the vertex information and connection information. In other words, a colorless three-dimensional object is formed in three-dimensional space by the vertex information and connection information.

[0048] The attribute information may be associated with a vertex or a face. The attribute information associated with a vertex may be expressed as "Attribute Per Point." The attribute information associated with a vertex may indicate an attribute of the vertex itself, or may indicate an attribute of a face connected to the vertex.

[0049] For example, a color may be associated with a vertex as attribute information. The color associated with a vertex may be the color of the vertex itself, or the color of a face connected to the vertex. The color of a face may be the average of multiple colors associated with multiple vertices of the face. Furthermore, a normal vector may be associated with a vertex or a face as attribute information. Such a normal vector can represent the front and back of a face.

[0050] A two-dimensional image may be associated with a surface as attribute information. The two-dimensional image associated with a surface is also expressed as a texture image or an "Attribute Map." Information indicating mapping between the surface and the two-dimensional image may be associated with the surface as attribute information. Information indicating such mapping may be expressed as mapping information, vertex information of a texture image, texture coordinates, or "Attribute UV Coordinate."

[0051] Furthermore, information such as color, image, and moving image used as attribute information may be expressed as "parametric space."

[0052] The attribute information allows texture to be reflected on the three-dimensional object. That is, a three-dimensional object having color is formed in three-dimensional space based on the vertex information, connection information, and attribute information.

[0053] In the above, the attribute information is associated with the vertices or faces, but it may also be associated with the edges.

[0054] 3 is a conceptual diagram illustrating mapping according to this embodiment. For example, a region of a two-dimensional image on a two-dimensional plane can be mapped onto a surface of a three-dimensional mesh in three-dimensional space. Specifically, coordinate information of the region in the two-dimensional image is associated with the surface of the three-dimensional mesh. As a result, an image of the mapped region in the two-dimensional image is reflected on the surface of the three-dimensional mesh.

[0055] By using the mapping, the 2D image used as attribute information can be separated from the 3D mesh. For example, in encoding the 3D mesh, the 2D image may be encoded by an image encoding method or a video encoding method.

[0056] <System Configuration> Fig. 4 is a block diagram showing an example of the configuration of a coding / decoding system according to this embodiment. In Fig. 4, the coding / decoding system includes a coding device 100 and a decoding device 200.

[0057] For example, the encoding device 100 obtains a three-dimensional mesh and encodes the three-dimensional mesh into a bitstream. Then, the encoding device 100 outputs the bitstream to the network 300. For example, the bitstream includes the encoded three-dimensional mesh and control information for decoding the encoded three-dimensional mesh. By encoding the three-dimensional mesh, information about the three-dimensional mesh is compressed.

[0058] The network 300 transmits a bitstream from the encoding device 100 to the decoding device 200. The network 300 may be the Internet, a wide area network (WAN), a local area network (LAN), or a combination of these. The network 300 is not necessarily limited to bidirectional communication, and may be a unidirectional communication network for terrestrial digital broadcasting, satellite broadcasting, or the like.

[0059] Furthermore, the network 300 can be replaced by a recording medium such as a DVD (Digital Versatile Disc) or a BD (Blu-Ray Disc (registered trademark)).

[0060] The decoding device 200 obtains a bitstream and decodes a three-dimensional mesh from the bitstream. By decoding the three-dimensional mesh, information about the three-dimensional mesh is expanded. For example, the decoding device 200 decodes the three-dimensional mesh according to a decoding method corresponding to the encoding method used by the encoding device 100 to encode the three-dimensional mesh. That is, the encoding device 100 and the decoding device 200 perform encoding and decoding according to encoding methods and decoding methods that correspond to each other.

[0061] The 3D mesh before encoding may also be referred to as an original 3D mesh, and the 3D mesh after decoding may also be referred to as a reconstructed 3D mesh.

[0062] 5 is a block diagram showing an example of the configuration of a coding device 100 according to this embodiment. For example, the coding device 100 includes a vertex information encoder 101, a connection information encoder 102, and an attribute information encoder 103.

[0063] The vertex information encoder 101 is an electrical circuit that encodes vertex information. For example, the vertex information encoder 101 encodes the vertex information into a bitstream according to a format defined for the vertex information.

[0064] The connection information encoder 102 is an electrical circuit that encodes the connection information, for example, the connection information encoder 102 encodes the connection information into a bitstream according to a format defined for the connection information.

[0065] The attribute information encoder 103 is an electric circuit that encodes the attribute information. For example, the attribute information encoder 103 encodes the attribute information into a bit stream in accordance with a format defined for the attribute information.

[0066] The vertex information, connectivity information, and attribute information may be coded using variable-length coding or fixed-length coding, such as Huffman coding or context-adaptive binary arithmetic coding (CABAC).

[0067] The vertex information encoder 101, the connection information encoder 102, and the attribute information encoder 103 may be integrated together, or each of the vertex information encoder 101, the connection information encoder 102, and the attribute information encoder 103 may be further subdivided into multiple components.

[0068] 6 is a block diagram showing another example of the configuration of the encoding device 100 according to this embodiment. For example, the encoding device 100 includes a pre-processor 104 and a post-processor 105 in addition to the configuration shown in FIG.

[0069] The preprocessor 104 is an electrical circuit that performs processing before encoding the vertex information, connectivity information, and attribute information. For example, the preprocessor 104 may perform a conversion process, a separation process, a multiplexing process, or the like on the 3D mesh before encoding. More specifically, for example, the preprocessor 104 may separate the vertex information, connectivity information, and attribute information from the 3D mesh before encoding.

[0070] The post-processor 105 is an electrical circuit that performs processing after the vertex information, connection information, and attribute information are encoded. For example, the post-processor 105 may perform conversion processing, separation processing, multiplexing processing, or the like on the encoded vertex information, connection information, and attribute information. More specifically, for example, the post-processor 105 may multiplex the encoded vertex information, connection information, and attribute information into a bitstream. Furthermore, for example, the post-processor 105 may further perform variable-length coding on the encoded vertex information, connection information, and attribute information.

[0071] 7 is a block diagram showing an example of the configuration of a decoding device 200 according to this embodiment. For example, the decoding device 200 includes a vertex information decoder 201, a connection information decoder 202, and an attribute information decoder 203.

[0072] The vertex information decoder 201 is an electrical circuit that decodes vertex information. For example, the vertex information decoder 201 decodes vertex information from a bitstream according to a format defined for the vertex information.

[0073] The connection information decoder 202 is an electrical circuit that decodes the connection information, for example, the connection information decoder 202 decodes the connection information from the bitstream according to a format defined for the connection information.

[0074] The attribute information decoder 203 is an electric circuit that decodes the attribute information. For example, the attribute information decoder 203 decodes the attribute information from the bitstream in accordance with a format defined for the attribute information.

[0075] The vertex information, connection information, and attribute information may be decoded using variable length decoding or fixed length decoding, which may correspond to Huffman coding, context-adaptive binary arithmetic coding (CABAC), or the like.

[0076] The vertex information decoder 201, the connection information decoder 202, and the attribute information decoder 203 may be integrated together, or each of the vertex information decoder 201, the connection information decoder 202, and the attribute information decoder 203 may be further subdivided into multiple components.

[0077] 8 is a block diagram showing another example of the configuration of the decoding device 200 according to this embodiment. For example, the decoding device 200 includes a pre-processor 204 and a post-processor 205 in addition to the configuration shown in FIG.

[0078] The preprocessor 204 is an electrical circuit that performs processing before decoding the vertex information, connection information, and attribute information. For example, the preprocessor 204 may perform conversion processing, separation processing, multiplexing processing, or the like on the bitstream before decoding the vertex information, connection information, and attribute information.

[0079] More specifically, for example, the preprocessor 204 may separate a sub-bitstream corresponding to vertex information, a sub-bitstream corresponding to connectivity information, and a sub-bitstream corresponding to attribute information from the bitstream. Also, for example, the preprocessor 204 may perform variable-length decoding on the bitstream in advance before decoding the vertex information, connectivity information, and attribute information.

[0080] The post-processor 205 is an electrical circuit that performs processing after the vertex information, connection information, and attribute information are decoded. For example, the post-processor 205 may perform conversion processing, separation processing, multiplexing processing, or the like on the decoded vertex information, connection information, and attribute information. More specifically, for example, the post-processor 205 may multiplex the decoded vertex information, connection information, and attribute information onto a three-dimensional mesh.

[0081] <Bitstream> Vertex information, connection information, and attribute information are coded and stored in a bitstream. The relationship between this information and the bitstream is shown below.

[0082] 9 is a conceptual diagram showing an example of the configuration of a bitstream according to this embodiment. In this example, connection information, vertex information, and attribute information are integrated in the bitstream. For example, the connection information, vertex information, and attribute information may be included in a single file.

[0083] Furthermore, multiple portions of this information may be stored sequentially, such as a first portion of connection information, a first portion of vertex information, a first portion of attribute information, a second portion of connection information, a second portion of vertex information, a second portion of attribute information, etc. These multiple portions may correspond to multiple portions that are different in time, multiple portions that are different in space, or multiple different faces.

[0084] Furthermore, the storage order of the connection information, vertex information, and attribute information is not limited to the above example, and a storage order different from the above example may be used.

[0085] 10 is a conceptual diagram showing another example of the configuration of a bitstream according to this embodiment. In this example, a plurality of files are included in the bitstream, and connection information, vertex information, and attribute information are stored in different files. Here, a file containing connection information, a file containing vertex information, and a file containing attribute information are shown, but the storage format is not limited to this example. For example, two types of information among the connection information, vertex information, and attribute information may be included in one file, and the remaining type of information may be included in another file.

[0086] Alternatively, the information may be split and stored in more files. For example, multiple pieces of connectivity information may be stored in multiple files, multiple pieces of vertex information may be stored in multiple files, or multiple pieces of attribute information may be stored in multiple files. These multiple pieces may correspond to multiple temporally different pieces, multiple spatially different pieces, or multiple different faces.

[0087] Furthermore, the storage order of the connection information, vertex information, and attribute information is not limited to the above example, and a storage order different from the above example may be used.

[0088] 11 is a conceptual diagram showing another example of the configuration of a bitstream according to this embodiment. In this example, the bitstream is composed of multiple separable sub-bitstreams, and connection information, vertex information, and attribute information are stored in different sub-bitstreams.

[0089] Here, a sub-bitstream containing connection information, a sub-bitstream containing vertex information, and a sub-bitstream containing attribute information are shown, but the storage format is not limited to this example.

[0090] For example, two types of information among the connection information, vertex information, and attribute information may be included in one sub-bitstream, and the remaining type of information may be included in another sub-bitstream. Specifically, attribute information of a two-dimensional image or the like may be stored in a sub-bitstream that complies with an image coding method, separate from the sub-bitstreams of the connection information and vertex information.

[0091] Each sub-bitstream may also include multiple files, and multiple pieces of connectivity information may be stored in multiple files, multiple pieces of vertex information may be stored in multiple files, or multiple pieces of attribute information may be stored in multiple files.

[0092] 9, 10, and 11, and a storage order different from the above examples may be used. For example, the vertex information, connection information, and attribute information may be stored in the bitstream in this order. Alternatively, the connection information, connection information, and attribute information may be stored in the bitstream in any of the following orders: connection information, attribute information, and vertex information; vertex information, attribute information, and connection information; attribute information, connection information, and vertex information; or attribute information, vertex information, and connection information.

[0093] Furthermore, each of the connection information, vertex information, and attribute information may be divided into a plurality of data, and the plurality of data may be stored in a cyclical or random order within the bitstream.

[0094] 12 is a block diagram showing a specific example of an encoding / decoding system according to this embodiment. In FIG. 12, the encoding / decoding system includes a three-dimensional data encoding system 110, a three-dimensional data decoding system 210, and an external connector 310.

[0095] The three-dimensional data encoding system 110 includes a controller 111, an input / output processor 112, a three-dimensional data encoder 113, a three-dimensional data generator 115, and a system multiplexer 114. The three-dimensional data decoding system 210 includes a controller 211, an input / output processor 212, a three-dimensional data decoder 213, a system demultiplexer 214, a presenter 215, and a user interface 216.

[0096] In the three-dimensional data encoding system 110, sensor data is input from a sensor terminal to a three-dimensional data generator 115. The three-dimensional data generator 115 generates three-dimensional data, such as point cloud data or mesh data, from the sensor data and inputs it to a three-dimensional data encoder 113.

[0097] For example, the three-dimensional data generator 115 generates vertex information, and generates connection information and attribute information corresponding to the vertex information. The three-dimensional data generator 115 may process the vertex information when generating the connection information and attribute information. For example, the three-dimensional data generator 115 may reduce the amount of data by deleting duplicate vertices, or may transform the vertex information (such as by shifting its position, rotating it, or normalizing it). The three-dimensional data generator 115 may also render the attribute information.

[0098] Furthermore, although the three-dimensional data generator 115 is a component of the three-dimensional data encoding system 110 in FIG. 12, it may be arranged externally and independently of the three-dimensional data encoding system 110.

[0099] The sensor terminal that provides the sensor data for generating the three-dimensional data may be, for example, a moving body such as an automobile, a flying object such as an airplane, a mobile terminal, a camera, etc. Furthermore, a distance sensor such as a LIDAR, a millimeter wave radar, an infrared sensor, or a range finder, a stereo camera, or a combination of multiple monocular cameras may also be used as the sensor terminal.

[0100] The sensor data may be the distance (position) of the object, monocular camera images, stereo camera images, color, reflectance, sensor attitude, orientation, gyro, sensing position (GPS information or altitude), speed, acceleration, sensing time, temperature, air pressure, humidity, or magnetism.

[0101] The three-dimensional data encoder 113 corresponds to the encoding device 100 shown in FIG. 5 and other figures. For example, the three-dimensional data encoder 113 encodes three-dimensional data to generate encoded data. The three-dimensional data encoder 113 also generates control information when encoding the three-dimensional data. The three-dimensional data encoder 113 then inputs the encoded data together with the control information to the system multiplexer 114.

[0102] The encoding method for the three-dimensional data may be an encoding method using geometry or an encoding method using a video codec. Here, the encoding method using geometry may also be referred to as a geometry-based encoding method. The encoding method using a video codec may also be referred to as a video-based encoding method.

[0103] The system multiplexer 114 multiplexes the encoded data and control information input from the 3D data encoder 113 to generate multiplexed data using a specified multiplexing method. The system multiplexer 114 may multiplex other media such as video, audio, subtitles, application data, or document files, or reference time information, along with the encoded data and control information of the 3D data. Furthermore, the system multiplexer 114 may multiplex attribute information related to the sensor data or the 3D data.

[0104] For example, the multiplexed data may have a file format for storage or a packet format for transmission. As these formats, ISOBMFF or a format based on ISOBMFF may be used. Also, MPEG-DASH, MMT, MPEG-2 TS Systems, RTP, or the like may be used.

[0105] The multiplexed data is then output as a transmission signal to the external connector 310 by the input / output processor 112. The multiplexed data may be transmitted as a transmission signal by wire or wirelessly. Alternatively, the multiplexed data is stored in an internal memory or a storage device. The multiplexed data may be transmitted to a cloud server via the Internet or may be stored in an external storage device.

[0106] For example, the transmission or storage of the multiplexed data is performed by a method according to the medium for transmission or storage, such as broadcasting or communication. The communication protocol may be http, ftp, TCP, UDP, IP, or a combination thereof. Furthermore, a pull-type communication method or a push-type communication method may be used.

[0107] For wired transmission, Ethernet (registered trademark), USB, RS-232C, HDMI (registered trademark), coaxial cable, etc. may be used. For wireless transmission, 3GPP (registered trademark), 3G / 4G / 5G defined by IEEE, wireless LAN, Wi-Fi, Bluetooth, or millimeter wave may be used. For broadcasting, for example, DVB-T2, DVB-S2, DVB-C2, ATSC3.0, or ISDB-S3 may be used.

[0108] The sensor data may be input to the three-dimensional data generator 115 or the system multiplexer 114. The three-dimensional data or encoded data may be output as a transmission signal directly to the external connector 310 via the input / output processor 112. The transmission signal output from the three-dimensional data encoding system 110 is input to the three-dimensional data decoding system 210 via the external connector 310.

[0109] Furthermore, each operation of the three-dimensional data encoding system 110 may be controlled by a controller 111 that executes an application program.

[0110] In the three-dimensional data decoding system 210, a transmission signal is input to an input / output processor 212. The input / output processor 212 decodes multiplexed data having a file format or a packet format from the transmission signal and inputs the multiplexed data to a system demultiplexer 214. The system demultiplexer 214 obtains coded data and control information from the multiplexed data and inputs them to a three-dimensional data decoder 213. The system demultiplexer 214 may extract other media or reference time information from the multiplexed data.

[0111] The three-dimensional data decoder 213 corresponds to the decoding device 200 shown in Fig. 7 etc. For example, the three-dimensional data decoder 213 decodes three-dimensional data from the encoded data based on a predefined encoding method. The three-dimensional data is then presented to the user by the presenter 215.

[0112] Additionally, additional information such as sensor data may be input to the presenter 215. The presenter 215 may present three-dimensional data based on the additional information. Additionally, a user instruction may be input from a user terminal to the user interface 216. Then, the presenter 215 may present three-dimensional data based on the input instruction.

[0113] The input / output processor 212 may acquire the three-dimensional data and the encoded data from the external connector 310 .

[0114] Furthermore, each operation of the three-dimensional data decoding system 210 may be controlled by a controller 211 that executes an application program.

[0115] 13 is a conceptual diagram showing an example of the configuration of point cloud data according to this embodiment. The point cloud data is data of a group of points representing a three-dimensional object.

[0116] Specifically, a point cloud is made up of a plurality of points, and has position information indicating the three-dimensional coordinate position of each point and attribute information indicating the attribute of each point. The position information is also expressed as geometry.

[0117] The type of attribute information may be, for example, color, reflectance, etc. One point may be associated with attribute information of one type, one point may be associated with attribute information of multiple different types, or one point may be associated with attribute information having multiple values ​​for the same type.

[0118] 14 is a conceptual diagram showing an example of a data file of point cloud data according to this embodiment. This example shows a case where there is a one-to-one correspondence between position information items and attribute information items, and shows position information and attribute information for N points that make up the point cloud data. In this example, the position information is information indicating a three-dimensional coordinate position using three axes, x, y, and z, and the attribute information is information indicating a color using RGB. A PLY file or the like can be used as a representative data file for point cloud data.

[0119] 15 is a conceptual diagram showing an example of the configuration of mesh data according to this embodiment. Mesh data is data used in CG (Computer Graphics) and the like, and is three-dimensional mesh data that shows the three-dimensional shape of an object using multiple surfaces. Each surface is also expressed as a polygon, and has a polygonal shape such as a triangle or a rectangle.

[0120] Specifically, a 3D mesh is composed of a plurality of points constituting a point cloud, as well as a plurality of edges and a plurality of faces. Each point is also expressed as a vertex or a position. Each edge corresponds to a line segment connected by two vertices. Each face corresponds to an area surrounded by three or more edges.

[0121] Furthermore, a three-dimensional mesh has position information indicating the three-dimensional coordinate positions of vertices. The position information is also expressed as vertex information or geometry. A three-dimensional mesh also has connection information indicating the relationship between multiple vertices that make up an edge or a face. The connection information is also expressed as connectivity. A three-dimensional mesh also has attribute information indicating the attributes of the vertices, edges, or faces. The attribute information in a three-dimensional mesh is also expressed as texture.

[0122] For example, the attribute information may indicate the color, reflectance, or normal vector for a vertex, edge, or face. The direction of the normal vector may represent the front and back of the face.

[0123] The mesh data may be stored in a data file format such as an object file.

[0124] 16 is a conceptual diagram showing an example of a data file of mesh data according to this embodiment. In this example, the data file includes position information G(1) to G(N) of N vertices that make up the three-dimensional mesh, and attribute information A1(1) to A1(N) of the N vertices. Also, in this example, M pieces of attribute information A2(1) to A2(M) are included. The attribute information items do not need to correspond one-to-one to vertices or faces. Furthermore, attribute information need not exist.

[0125] The connection information is represented by a combination of vertex indices. n[1, 3, 4] indicates a triangular face formed by three vertices, n=1, n=3, and n=4. Also, m[2, 4, 6] indicates that the attribute information of m=2, m=4, and m=6 corresponds to the three vertices, respectively.

[0126] Furthermore, the actual contents of the attribute information may be written in a separate file. A pointer to that content may be associated with a vertex, a face, or the like. For example, attribute information indicating an image for a face may be stored in a two-dimensional attribute map file. The file name of the attribute map and two-dimensional coordinate values ​​in the attribute map may be written in attribute information A2(1) to A2(M). The method of specifying attribute information for a face is not limited to these methods, and any method may be used.

[0127] 17 is a conceptual diagram showing types of three-dimensional data according to this embodiment. Point cloud data and mesh data may represent static objects or dynamic objects. A static object is an object that does not change over time, and a dynamic object is an object that changes over time. A static object may correspond to three-dimensional data for any point in time.

[0128] For example, point cloud data for a given point in time may be referred to as a PCC frame, mesh data for a given point in time may be referred to as a mesh frame, and PCC frames and mesh frames may be simply referred to as frames.

[0129] The area of ​​the object may be limited to a certain range, as in normal video data, or may not be limited, as in map data. The density of points or surfaces may be determined in various ways. Sparse point cloud data or sparse mesh data may be used, or dense point cloud data or dense mesh data may be used.

[0130] Next, encoding and decoding of a point cloud or a three-dimensional mesh will be described. The device, process, or syntax for encoding and decoding vertex information of a three-dimensional mesh in the present disclosure may be applied to encoding and decoding of a point cloud. The device, process, or syntax for encoding and decoding of a point cloud in the present disclosure may be applied to encoding and decoding vertex information of a three-dimensional mesh.

[0131] Furthermore, a device, process, or syntax for encoding and decoding attribute information of a point cloud in the present disclosure may be applied to encoding and decoding connectivity information or attribute information of a three-dimensional mesh.Furthermore, a device, process, or syntax for encoding and decoding connectivity information or attribute information of a three-dimensional mesh in the present disclosure may be applied to encoding and decoding attribute information of a point cloud.

[0132] Furthermore, at least some of the processing may be shared between the encoding and decoding of point cloud data and the encoding and decoding of mesh data, thereby reducing the scale of the circuit and software program.

[0133] 18 is a block diagram showing an example configuration of a three-dimensional data encoder 113 according to this embodiment. In this example, the three-dimensional data encoder 113 includes a vertex information encoder 121, an attribute information encoder 122, a metadata encoder 123, and a multiplexer 124. The vertex information encoder 121, the attribute information encoder 122, and the multiplexer 124 may correspond to the vertex information encoder 101, the attribute information encoder 103, the post-processor 105, etc. in FIG.

[0134] In this example, the three-dimensional data encoder 113 encodes the three-dimensional data according to a geometry-based encoding method, which takes into account the three-dimensional structure. In addition, in the geometry-based encoding method, attribute information is encoded using configuration information obtained in encoding the vertex information.

[0135] Specifically, first, vertex information, attribute information, and metadata included in three-dimensional data generated from sensor data are input to a vertex information encoder 121, an attribute information encoder 122, and a metadata encoder 123, respectively. Here, connectivity information included in the three-dimensional data may be treated in the same way as attribute information. In addition, in the case of point cloud data, position information may be treated as vertex information.

[0136] The vertex information encoder 121 encodes the vertex information into compressed vertex information and outputs the compressed vertex information as encoded data to the multiplexer 124. The vertex information encoder 121 also generates metadata for the compressed vertex information and outputs it to the multiplexer 124. The vertex information encoder 121 also generates configuration information and outputs it to the attribute information encoder 122.

[0137] The attribute information encoder 122 uses the configuration information generated by the vertex information encoder 121 to encode the attribute information into compressed attribute information and outputs the compressed attribute information as encoded data to the multiplexer 124. The attribute information encoder 122 also generates metadata of the compressed attribute information and outputs it to the multiplexer 124.

[0138] The metadata encoder 123 encodes compressible metadata into compressed metadata and outputs the compressed metadata as encoded data to the multiplexer 124. The metadata encoded by the metadata encoder 123 may be used to encode vertex information and attribute information.

[0139] The multiplexer 124 multiplexes the compressed vertex information, the compressed vertex information metadata, the compressed attribute information, the compressed attribute information metadata, and the compressed metadata into a bitstream, and then inputs the bitstream to the system layer.

[0140] 19 is a block diagram showing an example configuration of a three-dimensional data decoder 213 according to this embodiment. In this example, the three-dimensional data decoder 213 includes a vertex information decoder 221, an attribute information decoder 222, a metadata decoder 223, and a demultiplexer 224. The vertex information decoder 221, the attribute information decoder 222, and the demultiplexer 224 may correspond to the vertex information decoder 201, the attribute information decoder 203, the preprocessor 204, and the like in FIG.

[0141] In this example, the three-dimensional data decoder 213 decodes three-dimensional data according to a geometry-based encoding method. The three-dimensional structure is taken into consideration in the decoding according to the geometry-based encoding method. Furthermore, in the decoding according to the geometry-based encoding method, attribute information is decoded using configuration information obtained in decoding vertex information.

[0142] Specifically, first, a bitstream is input from the system layer to a demultiplexer 224. The demultiplexer 224 separates compressed vertex information, compressed vertex information metadata, compressed attribute information, compressed attribute information metadata, and compressed metadata from the bitstream. The compressed vertex information and compressed vertex information metadata are input to a vertex information decoder 221. The compressed attribute information and compressed attribute information metadata are input to an attribute information decoder 222. The metadata is input to a metadata decoder 223.

[0143] The vertex information decoder 221 decodes vertex information from the compressed vertex information using metadata of the compressed vertex information. The vertex information decoder 221 also generates configuration information and outputs it to the attribute information decoder 222. The attribute information decoder 222 decodes attribute information from the compressed attribute information using the configuration information generated by the vertex information decoder 221 and the metadata of the compressed attribute information. The metadata decoder 223 decodes metadata from the compressed metadata. The metadata decoded by the metadata decoder 223 may be used to decode the vertex information and the attribute information.

[0144] Thereafter, the vertex information, attribute information, and metadata are output as three-dimensional data from the three-dimensional data decoder 213. Note that, for example, this metadata is metadata of the vertex information and attribute information, and can be used in an application program.

[0145] 20 is a block diagram showing another example configuration of the three-dimensional data encoder 113 according to the present embodiment. In this example, the three-dimensional data encoder 113 includes a vertex image generator 131, an attribute image generator 132, a metadata generator 133, a video encoder 134, a metadata encoder 123, and a multiplexer 124. The vertex image generator 131, the attribute image generator 132, and the video encoder 134 may correspond to the vertex information encoder 101 and the attribute information encoder 103 in FIG. 6 , etc.

[0146] In this example, the 3D data encoder 113 encodes the 3D data according to a video-based encoding method. In encoding according to the video-based encoding method, multiple 2D images are generated from the 3D data, and the multiple 2D images are encoded according to a video encoding method. Here, the video encoding method may be High Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), or the like.

[0147] Specifically, first, vertex information and attribute information included in three-dimensional data generated from sensor data are input to a metadata generator 133. The vertex information and attribute information are then input to a vertex image generator 131 and an attribute image generator 132, respectively. The metadata included in the three-dimensional data is then input to a metadata encoder 123. Here, connectivity information included in the three-dimensional data may be treated in the same way as attribute information. In the case of point cloud data, position information may be treated as vertex information.

[0148] The metadata generator 133 generates map information of a plurality of two-dimensional images from the vertex information and attribute information, and inputs the map information to the vertex image generator 131, the attribute image generator 132, and the metadata encoder 123.

[0149] The vertex image generator 131 generates a vertex image based on the vertex information and map information, and inputs the generated image to the video encoder 134. The attribute image generator 132 generates an attribute image based on the attribute information and map information, and inputs the generated image to the video encoder 134.

[0150] The video encoder 134 encodes the vertex images and attribute images into compressed vertex information and compressed attribute information, respectively, in accordance with a video encoding method, and outputs the compressed vertex information and compressed attribute information as encoded data to the multiplexer 124. The video encoder 134 also generates metadata for the compressed vertex information and metadata for the compressed attribute information, and outputs them to the multiplexer 124.

[0151] The metadata encoder 123 encodes the compressible metadata into compressed metadata and outputs the compressed metadata as encoded data to the multiplexer 124. The compressible metadata includes map information. The metadata encoded by the metadata encoder 123 may also be used to encode vertex information and attribute information.

[0152] The multiplexer 124 multiplexes the compressed vertex information, the compressed vertex information metadata, the compressed attribute information, the compressed attribute information metadata, and the compressed metadata into a bitstream, and then inputs the bitstream to the system layer.

[0153] 21 is a block diagram showing another example configuration of the 3D data decoder 213 according to this embodiment. In this example, the 3D data decoder 213 includes a vertex information generator 231, an attribute information generator 232, a video decoder 234, a metadata decoder 223, and a demultiplexer 224. The vertex information generator 231, the attribute information generator 232, and the video decoder 234 may correspond to the vertex information decoder 201 and the attribute information decoder 203 in FIG. 8, etc.

[0154] In this example, the 3D data decoder 213 decodes the 3D data according to a video-based coding method. In the decoding according to the video-based coding method, a plurality of 2D images are decoded according to a video coding method, and 3D data is generated from the plurality of 2D images. Here, the video coding method may be High Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), or the like.

[0155] Specifically, first, a bitstream is input from the system layer to the demultiplexer 224. The demultiplexer 224 separates compressed vertex information, compressed vertex information metadata, compressed attribute information, compressed attribute information metadata, and compressed metadata from the bitstream. The compressed vertex information, compressed vertex information metadata, compressed attribute information, and compressed attribute information metadata are input to the video decoder 234. The compressed metadata is input to the metadata decoder 223.

[0156] The video decoder 234 decodes the vertex images in accordance with the video encoding method. At this time, the video decoder 234 decodes the vertex images from the compressed vertex information using the metadata of the compressed vertex information. Then, the video decoder 234 inputs the vertex images to the vertex information generator 231. The video decoder 234 also decodes the attribute images in accordance with the video encoding method. At this time, the video decoder 234 decodes the attribute images from the compressed attribute information using the metadata of the compressed attribute information. Then, the video decoder 234 inputs the attribute images to the attribute information generator 232.

[0157] The metadata decoder 223 decodes metadata from the compressed metadata. The metadata decoded by the metadata decoder 223 includes map information used to generate vertex information and attribute information. The metadata decoded by the metadata decoder 223 may also be used to decode vertex images and attribute images.

[0158] The vertex information generator 231 reproduces vertex information from the vertex image in accordance with the map information included in the metadata decoded by the metadata decoder 223. The attribute information generator 232 reproduces attribute information from the attribute image in accordance with the map information included in the metadata decoded by the metadata decoder 223.

[0159] Thereafter, the vertex information, attribute information, and metadata are output as three-dimensional data from the three-dimensional data decoder 213. Note that, for example, this metadata is metadata of the vertex information and attribute information, and can be used in an application program.

[0160] Fig. 22 is a conceptual diagram showing a specific example of encoding processing according to this embodiment. Fig. 22 shows a three-dimensional data encoder 113 and a description encoder 148. In this example, the three-dimensional data encoder 113 includes a two-dimensional data encoder 141 and a mesh data encoder 142. The two-dimensional data encoder 141 includes a texture encoder 143. The mesh data encoder 142 includes a vertex information encoder 144 and a connection information encoder 145.

[0161] The vertex information encoder 144, the connection information encoder 145, and the texture encoder 143 may correspond to the vertex information encoder 101, the connection information encoder 102, and the attribute information encoder 103 in FIG.

[0162] For example, the two-dimensional data encoder 141 operates as a texture encoder 143 and generates a texture file by encoding the texture corresponding to the attribute information as two-dimensional data according to an image encoding method or a video encoding method.

[0163] The mesh data encoder 142 also operates as a vertex information encoder 144 and a connectivity information encoder 145, and generates a mesh file by encoding the vertex information and connectivity information. The mesh data encoder 142 may further encode mapping information for textures. The encoded mapping information may then be included in the mesh file.

[0164] The description encoder 148 also generates a description file by encoding a description corresponding to metadata such as text data. The description encoder 148 may encode the description at the system layer. For example, the description encoder 148 may be included in the system multiplexer 114 of FIG. 12 .

[0165] The above operations generate a bitstream containing texture files, mesh files, and description files, which may be multiplexed into the bitstream in file formats such as glTF (Graphics Language Transmission Format) or USD (Universal Scene Description).

[0166] The three-dimensional data encoder 113 may include two mesh data encoders as the mesh data encoder 142. For example, one mesh data encoder encodes vertex information and connectivity information of a static three-dimensional mesh, and the other mesh data encoder encodes vertex information and connectivity information of a dynamic three-dimensional mesh.

[0167] Correspondingly, two mesh files may then be included in the bitstream, for example one mesh file corresponding to a static 3D mesh and another mesh file corresponding to a dynamic 3D mesh.

[0168] Furthermore, the static three-dimensional mesh may be a three-dimensional mesh of an intraframe coded using intraprediction, and the dynamic three-dimensional mesh may be a three-dimensional mesh of an interframe coded using interprediction. Furthermore, information on the dynamic three-dimensional mesh may be differential information between vertex information or connectivity information of the three-dimensional mesh of an intraframe and vertex information or connectivity information of the three-dimensional mesh of an interframe.

[0169] Fig. 23 is a conceptual diagram showing a specific example of the decoding process according to this embodiment. Fig. 23 shows a three-dimensional data decoder 213, a description decoder 248, and a renderer 247. In this example, the three-dimensional data decoder 213 includes a two-dimensional data decoder 241, a mesh data decoder 242, and a mesh reconstructor 246. The two-dimensional data decoder 241 includes a texture decoder 243. The mesh data decoder 242 includes a vertex information decoder 244 and a connectivity information decoder 245.

[0170] The vertex information decoder 244, the connection information decoder 245, the texture decoder 243, and the mesh reconstructor 246 may correspond to the vertex information decoder 201, the connection information decoder 202, the attribute information decoder 203, and the post-processor 205 in Fig. 8. The presenter 247 may correspond to the presenter 215 in Fig. 12.

[0171] For example, the two-dimensional data decoder 241 operates as a texture decoder 243, and decodes the texture corresponding to the attribute information from the texture file as two-dimensional data in accordance with an image coding method or a video coding method.

[0172] The mesh data decoder 242 also operates as a vertex information decoder 244 and a connectivity information decoder 245 to decode vertex information and connectivity information from the mesh file. The mesh data decoder 242 may further decode mapping information for textures from the mesh file.

[0173] The description decoder 248 also decodes descriptions corresponding to metadata such as text data from the description file. The description decoder 248 may decode the descriptions at the system layer. For example, the description decoder 248 may be included in the system demultiplexer 214 of FIG. 12 .

[0174] The mesh reconstructor 246 reconstructs a 3D mesh from the vertex information, connectivity information, and textures according to the description. The renderer 247 renders and outputs the 3D mesh according to the description.

[0175] Through the above operations, a 3D mesh is reconstructed and output from a bitstream containing a texture file, a mesh file, and a description file.

[0176] The three-dimensional data decoder 213 may include two mesh data decoders as the mesh data decoder 242. For example, one mesh data decoder decodes vertex information and connectivity information of a static three-dimensional mesh, and the other mesh data decoder decodes vertex information and connectivity information of a dynamic three-dimensional mesh.

[0177] Correspondingly, two mesh files may then be included in the bitstream, for example one mesh file corresponding to a static 3D mesh and another mesh file corresponding to a dynamic 3D mesh.

[0178] Furthermore, the static three-dimensional mesh may be a three-dimensional mesh of an intraframe coded using intraprediction, and the dynamic three-dimensional mesh may be a three-dimensional mesh of an interframe coded using interprediction. Furthermore, information on the dynamic three-dimensional mesh may be differential information between vertex information or connectivity information of the three-dimensional mesh of an intraframe and vertex information or connectivity information of the three-dimensional mesh of an interframe.

[0179] A dynamic 3D mesh coding method is sometimes called DMC (Dynamic Mesh Coding), and a video-based dynamic 3D mesh coding method is sometimes called V-DMC (Video-based Dynamic Mesh Coding).

[0180] The point cloud encoding method is sometimes called PCC (Point Cloud Compression). The point cloud video-based encoding method is sometimes called V-PCC (Video-based Point Cloud Compression). The point cloud geometry-based encoding method is sometimes called G-PCC (Geometry-based Point Cloud Compression).

[0181] <Implementation Example> Fig. 24 is a block diagram showing an implementation example of the encoding device 100 according to this embodiment. The encoding device 100 includes a circuit 151 and a memory 152. For example, multiple components of the encoding device 100 shown in Fig. 5 etc. are implemented by the circuit 151 and memory 152 shown in Fig. 24.

[0182] The circuit 151 is a circuit that performs information processing and is a circuit that can access the memory 152. For example, the circuit 151 is a dedicated or general-purpose electric circuit that encodes a three-dimensional mesh. The circuit 151 may be a processor such as a CPU. Alternatively, the circuit 151 may be a collection of multiple electric circuits.

[0183] The memory 152 is a dedicated or general-purpose memory that stores information used by the circuit 151 to encode the three-dimensional mesh. The memory 152 may be an electric circuit and may be connected to the circuit 151. The memory 152 may also be included in the circuit 151. The memory 152 may also be a collection of multiple electric circuits. The memory 152 may also be a magnetic disk, an optical disk, or the like, and may also be expressed as a storage, a recording medium, or the like. The memory 152 may also be a non-volatile memory or a volatile memory.

[0184] For example, the memory 152 may store a three-dimensional mesh or a bitstream, or may store a program for the circuit 151 to encode the three-dimensional mesh.

[0185] Note that the encoding device 100 does not necessarily have to implement all of the components shown in Figure 5 and the like, and does not necessarily have to perform all of the processes shown here. Some of the components shown in Figure 5 and the like may be included in another device, and some of the processes shown here may be executed by another device. Furthermore, the encoding device 100 may implement any combination of the components of the present disclosure, and may perform any combination of the processes of the present disclosure.

[0186] Fig. 25 is a block diagram showing an example implementation of a decoding device 200 according to this embodiment. The decoding device 200 includes a circuit 251 and a memory 252. For example, multiple components of the decoding device 200 shown in Fig. 7 and other figures are implemented by the circuit 251 and memory 252 shown in Fig. 25.

[0187] The circuit 251 is a circuit that performs information processing and is a circuit that can access the memory 252. For example, the circuit 251 is a dedicated or general-purpose electric circuit that decodes a three-dimensional mesh. The circuit 251 may be a processor such as a CPU. Alternatively, the circuit 251 may be a collection of multiple electric circuits.

[0188] The memory 252 is a dedicated or general-purpose memory that stores information for the circuit 251 to decode the 3D mesh. The memory 252 may be an electric circuit and may be connected to the circuit 251. The memory 252 may also be included in the circuit 251. The memory 252 may also be a collection of multiple electric circuits. The memory 252 may also be a magnetic disk, an optical disk, or the like, and may also be expressed as a storage, a recording medium, or the like. The memory 252 may also be a non-volatile memory or a volatile memory.

[0189] For example, the memory 252 may store a three-dimensional mesh or a bitstream, or may store a program for the circuit 251 to decode the three-dimensional mesh.

[0190] Note that the decoding device 200 does not necessarily have to implement all of the components shown in Figure 7 and the like, and does not necessarily have to perform all of the processes shown here. Some of the components shown in Figure 7 and the like may be included in another device, and some of the processes shown here may be executed by another device. Furthermore, the decoding device 200 may implement any combination of the components of the present disclosure, and may perform any combination of the processes of the present disclosure.

[0191] The encoding method and the decoding method including the steps performed by each component of the encoding device 100 and the decoding device 200 of the present disclosure may be executed by any device or system. For example, part or all of the encoding method and the decoding method may be executed by a computer including a processor, a memory, an input / output circuit, etc. In this case, the encoding method and the decoding method may be executed by the computer executing a program for causing the computer to execute the encoding method and the decoding method.

[0192] Alternatively, the program or the bitstream may be recorded on a non-transitory computer-readable recording medium such as a CD-ROM.

[0193] An example of a program may be a bitstream. For example, a bitstream including an encoded three-dimensional mesh includes syntax elements for causing the decoding device 200 to decode the three-dimensional mesh. The bitstream then causes the decoding device 200 to decode the three-dimensional mesh according to the syntax elements included in the bitstream. Thus, the bitstream may play a role similar to that of a program.

[0194] The bitstream may be an encoded bitstream containing the encoded 3D mesh, or may be a multiplexed bitstream containing the encoded 3D mesh and other information.

[0195] Furthermore, each component of the encoding device 100 and the decoding device 200 may be configured with dedicated hardware, general-purpose hardware that executes the above-mentioned programs, or a combination of these. The general-purpose hardware may be configured with a memory in which the programs are recorded and a general-purpose processor that reads and executes the programs from the memory. Here, the memory may be a semiconductor memory or a hard disk, and the general-purpose processor may be a CPU.

[0196] Furthermore, the dedicated hardware may be configured with a memory, a dedicated processor, etc. For example, the dedicated processor may execute the encoding method and the decoding method by referring to a memory for recording data.

[0197] Furthermore, as described above, each component of the encoding device 100 and the decoding device 200 may be an electric circuit. These electric circuits may form a single electric circuit as a whole, or each may be a separate electric circuit. Furthermore, these electric circuits may correspond to dedicated hardware, or may correspond to general-purpose hardware that executes the above-mentioned programs, etc. Furthermore, the encoding device 100 and the decoding device 200 may be implemented as an integrated circuit.

[0198] Furthermore, the encoding device 100 may be a transmitting device that transmits the three-dimensional mesh, and the decoding device 200 may be a receiving device that receives the three-dimensional mesh.

[0199] Displacement Encoding and Decoding The following terminology is used here by way of example:

[0200] (1) Image An image is a data unit made up of a set of pixels, and includes a picture or a block smaller than a picture. Images include both moving images and still images.

[0201] (2) Picture A picture is a unit of image processing that is made up of a set of pixels, and is also called a frame or field.

[0202] (3) Block A block is a processing unit consisting of a specific number of pixels. The term shown in the following example is also used for a block. The shape of a block is not particularly limited. A block may be, for example, a rectangular shape of M×N pixels or a square shape of M×M pixels. A block may also be a triangular shape, a circular shape, or another shape. Examples of blocks are as follows:

[0203] Slice, tile, or brick CTU, superblock, or basic division unit VPDU, processing division unit for hardware CU, processing block unit, prediction block unit (PU), or orthogonal transform block unit (TU) Sub-block

[0204] (4) Pixel or Sample A pixel or sample is the smallest point of an image, in other words, the smallest unit. Pixels or samples include not only pixels at integer positions, but also pixels at sub-pixel positions generated based on pixels at integer positions.

[0205] (5) Pixel Value or Sample Value: A pixel value or sample value is a unique value of a pixel. The pixel value or sample value may include a luma value, a chroma value, or an RGB gradation level, and may also include a depth value or a binary value of 0 or 1.

[0206] (6) Flags A flag indicates one or more bits. A flag is, for example, a parameter or index represented by two or more bits. A flag may indicate not only a value represented by a binary number, but also a value represented by a number other than a binary number.

[0207] (7) Signal: A signal is something that is symbolized or coded to transmit information. A signal includes a discrete digital signal or a continuous analog signal.

[0208] (8) Stream or Bit Stream A stream or bit stream is a digital data sequence that indicates the flow of digital data. A stream or bit stream may be a single stream, or may be configured to include multiple streams with multiple layers. A stream or bit stream may be transmitted by serial communication using a single transmission path, or may be transmitted by packet communication using multiple transmission paths.

[0209] (9) Difference: For scalar quantities, the difference can include simple difference (x-y) and difference calculations. The difference can include absolute difference (|x-y|), squared difference (x^2-y^2), square root difference (√(x-y)), weighted difference (ax-by, where a and b are constants), or offset difference (x-y+a, where a is an offset).

[0210] (10) Sum: For scalar quantities, sums can include simple sum (x + y) and addition calculations. The sum can also include absolute sum (|x + y|), sum of squares (x^2 + y^2), square root of the sum (√(x + y)), weighted sum (ax + by, where a and b are constants), or offset sum (x + y + a, where a is an offset).

[0211] (11) "Based on" The expression "based on something" means that something other than that "something" may be taken into consideration. Also, "based on" can be used both when a direct result is obtained and when a result is obtained through an intermediate result.

[0212] (12) "Used" or "Using" The phrases "something was used" or "used something" mean that something other than the "something" may be taken into consideration. The phrases "used" or "used" may be used both in cases where a direct result is obtained and in cases where a result is obtained via an intermediate result.

[0213] (13) Prohibition "Prohibit" can be rephrased as "not permitted." Also, "not prohibited / prohibited" or "permitted / permitted" does not necessarily mean "obligation."

[0214] (14) "Restriction" or "Limitation" "Restriction" or "Limitation" can be rephrased as "not permitted / not allowed" or "not permitted / permitted." Furthermore, "prohibited / not prohibited" or "not permitted / permitted" does not necessarily mean "obligation." Furthermore, what is prohibited quantitatively or qualitatively may be either partial or total.

[0215] (15) Chroma The term chroma is an adjective, represented by the symbols Cb or Cr, that indicates that a sample array or a single sample represents one of the two color difference signals associated with a primary color. The term chroma is sometimes used instead of the term chrominance.

[0216] (16) Luma The term luma is an adjective, denoted by the symbols or subscripts Y or L, that indicates that a sample array or a single sample represents a monochrome signal for a primary color. The term luma is sometimes used instead of the term luminance.

[0217] The encoding / decoding system of this embodiment will be described below.

[0218] A typical three-dimensional model (also called a 3D model) digitally represents an object so that a user can explore the model using zoom, pan, and rotation in all three dimensions while it is rendered over time. One way to construct such a representation is to build a 3D mesh using triangles. The model stores the positions of the triangle vertices, their connectivity to each other, and their associated attributes (such as normals or UV patches).

[0219] Storing all this information in uncompressed form requires a very large storage space and therefore a very large bandwidth for transmission. The triangles that form the mesh often have repeating patterns and similar properties, especially in temporal and spatial neighborhoods. These repetitions can be exploited to develop efficient encoding and decoding methods for storage and transmission. One such encoding and decoding method is Video-based Dynamic Mesh Coding (V-DMC).

[0220] 26 is a block diagram showing another example of the configuration of the encoding / decoding system according to this embodiment. As shown in FIG. 26, the encoding / decoding system includes an encoding device 100 and a decoding device 200.

[0221] The encoding / decoding system accepts input three-dimensional meshes (also called 3D meshes) in the form of three-dimensional coordinates of vertices (vertex information), connectivity (connection information) and associated attributes (attribute information), which may include texture maps as well as geometry.

[0222] The encoding device 100 takes an input 3D mesh (also referred to as an input 3D mesh or input mesh) in the form of 3D coordinates of vertices, connectivity, and associated attributes. The encoding device 100 encodes all associated information into a stream. The stream may consist of a single bitstream or multiple bitstreams.

[0223] The network 300 transmits the stream generated by the encoding device 100 to the decoding device 200. The network 300 may be the Internet, a wide area network (WAN), a local area network (LAN), or any combination thereof. Furthermore, the network 300 is not necessarily limited to a two-way communication network, but may also be a one-way communication network that transmits broadcast waves such as terrestrial digital broadcasting or satellite broadcasting. Instead of the network 300, a recording medium such as a digital versatile disc (DVD) or a blue-ray disc (BD) on which a stream is recorded may be used.

[0224] The stream is transmitted to a decoding device 200 via a network 300. The decoding device 200 decodes the bitstream and generates a 3D mesh using the 3D coordinates, connectivity, and associated attributes of the decoded vertices. The decoding device 200 outputs the generated 3D mesh (also referred to as an output 3D mesh or output mesh).

[0225] FIG. 27 is a diagram showing another example of the configuration of the encoding device 100.

[0226] As shown in FIG. 27, the encoding device 100 includes a preprocessor 1103 and a compressor 1106 .

[0227] The encoding device 100 reads an input mesh 1101 and an attribute map 1102 and passes them to a preprocessor 1103. The preprocessor 1103 processes the input mesh to extract a base mesh 1104 and displacement data 1105. The attribute map 1102, along with the extracted base mesh 1104 and displacement data 1105, are passed to a compressor 1106.

[0228] The compressor 1106 also compresses the base mesh 1104, the displacement data 1105, and the attribute map 1102 to generate a bitstream 1107. The compressor 1106 can transmit additional information to the decoding device 200 by further including metadata 1108 in the bitstream 1107.

[0229] FIG. 28 is a diagram showing another example of the configuration of the decoding device 200.

[0230] As shown in FIG. 28, the decoding device 200 includes a decompressor 2102 and a post-processor 2106 .

[0231] The decoding device 200 reads a bitstream 2101 and passes it to a decompressor 2102. The decompressor 2102 decompresses a base mesh 2103, displacement data 2104, and an attribute map 2108 from the bitstream 2101 and passes them to a post-processor 2106. An example of the displacement data 2104 is a displacement vector.

[0232] The post-processor 2106 also processes the base mesh 2103 according to the displacement data 2104 and the attribute map 2108 to generate an output mesh 2107. The post-processor 2106 may further use information from the metadata 2105 to generate the output mesh 2107.

[0233] The detailed configuration of the encoding device 100 will be described below.

[0234] FIG. 29 is a block diagram showing a detailed configuration example of the encoding device 100.

[0235] As shown in FIG. 29, the encoding device 100 includes a decimator 1201, a quantizer 1202, a base mesh encoder 1203, a base mesh decoder 1204, an inverse quantizer 1205, a subdivision unit 1206, a displacement vector calculator 1207, a wavelet transformer 1208, a quantizer 1209, an image packer 1210, a video encoder 1211, a color converter 1212, a video encoder 1213, and a multiplexer 1214.

[0236] The decimator 1201 acquires the mesh input to the encoding device 100 (corresponding to the input mesh 1101) as an original mesh, and generates a base mesh by performing a decimation process (in other words, a thinning process) on the acquired original mesh. The decimation process is a process of deleting (in other words, thinning) some of the vertices included in the original mesh. The decimation process may include a process of changing the positions of at least some of the vertices included in the original mesh, or a process of changing the connectivity of at least some of the vertices included in the original mesh. The decimation process is also simply referred to as decimation.

[0237] The base mesh generated by the decimation process has fewer vertices than the original mesh. The vertices of the base mesh may be located at different positions than the vertices of the original mesh. Also, the vertex connectivity of the base mesh may be different from the vertex connectivity of the original mesh. The decimator 1201 provides the generated base mesh to the quantizer 1202.

[0238] The quantizer 1202 quantizes the base mesh generated by the decimator 1201. The quantizer 1202 provides the quantized base mesh to the base mesh encoder 1203.

[0239] The base mesh encoder 1203 encodes the base mesh quantized by the quantizer 1202 into a bitstream (also called a base mesh bitstream) (in other words, generates a base mesh bitstream). The base mesh encoder 1203 provides the base mesh bitstream to the base mesh decoder 1204 and the multiplexer 1214.

[0240] The base mesh decoder 1204 obtains a quantized base mesh by decoding the base mesh bitstream provided by the base mesh encoder 1203. The base mesh decoder 1204 provides the quantized base mesh to the inverse quantizer 1205.

[0241] The inverse quantizer 1205 generates a base mesh (also referred to as a decoded base mesh) by inverse quantizing the quantized base mesh provided by the base mesh decoder 1204. The inverse quantizer 1205 provides the decoded base mesh to the subdivision unit 1206. There may be differences between the decoded base mesh generated by the inverse quantizer 1205 and the base mesh generated by the decimator 1201 due to the quantization and inverse quantization processes.

[0242] The subdivider 1206 performs a subdivision process on the decoded base mesh generated by the inverse quantizer 1205. The subdivision process may be a process of subdividing the faces of the decoded base mesh to make it smaller. The subdivider 1206 provides the subdivided decoded base mesh to the displacement vector calculator 1207.

[0243] Specifically, the subdivider 1206 subdivides the mesh by generating a new vertex between two connected vertices in the mesh. By repeating this process, the number of vertices in the mesh can be increased to a predetermined number. By repeating the subdivision process throughout the mesh (i.e., by performing the subdivision multiple times), multiple levels of detail (LoD) are generated.

[0244] The displacement vector calculator 1207 receives the original mesh obtained by the encoding device 100 and also receives the subdivided decoded base mesh from the subdivider 1206. The displacement vector calculator 1207 calculates a vector from a vertex of the subdivided decoded base mesh to a vertex, face, or edge of the original mesh as a displacement vector. The displacement vector calculator 1207 provides the displacement vector to the wavelet transformer 1208.

[0245] The wavelet transformer 1208 obtains wavelet coefficients by performing wavelet transform processing on the displacement vectors calculated by the displacement vector calculator 1207. The wavelet transformer 1208 provides the wavelet coefficients to the quantizer 1209. In the wavelet transform, the wavelet transformer 1208 assigns vertices to multiple LoD layers and applies, for example, a lifting transform to the displacement vectors of the vertices, thereby calculating wavelet coefficients that represent various components from low-frequency components to high-frequency components.

[0246] The quantizer 1209 quantizes the wavelet coefficients acquired by the wavelet transformer 1208. The quantizer 1209 can quantize the wavelet coefficients for each LoD layer. The quantizer 1209 provides the quantized wavelet coefficients to the image packer 1210.

[0247] The image packer 1210 generates an image containing wavelet coefficients quantized by the quantizer 1209. The image packer 1210 can generate the image by mapping the wavelet coefficients quantized by the quantizer 1209 to pixels in a two-dimensional image format. The image packer 1210 provides the generated image to the video encoder 1211. The process of mapping the quantized wavelet coefficients to pixels in the two-dimensional image format can use mapping information that represents the assignment of the quantized wavelet coefficients to pixels in the two-dimensional image format.

[0248] The video encoder 1211 encodes the image generated by the image packer 1210 into a bitstream (also called a displacement bitstream) (in other words, generates a displacement bitstream). The video encoder 1211 provides the displacement bitstream to the multiplexer 1214. The displacement bitstream may be a bitstream containing displacement information in an image format. The image format may be, for example, a format containing two chroma information pieces and one luma information piece.

[0249] The color converter 1212 obtains the attribute map obtained by the encoding device 100 as an original attribute map and performs color conversion processing on the original attribute map. The color conversion processing may include conversion processing of the color representation format or color space. The color converter 1212 provides the attribute map after color conversion processing to the video encoder 1213. Note that, although the description here takes as an example a case where the original attribute map is input to the color converter 1212, if the number or positions of vertices differ between the decoded mesh and the original mesh, the feature map may be converted to match the structure of the decoded mesh.

[0250] The video encoder 1213 encodes the attribute map converted by the color converter 1212 into a bitstream (also called an attribute bitstream) (in other words, generates an attribute bitstream). The video encoder 1213 provides the attribute bitstream to the multiplexer 1214.

[0251] The multiplexer 1214 obtains the base mesh bitstream from the base mesh encoder 1203, the displacement bitstream from the video encoder 1211, and the attribute bitstream from the video encoder 1213, and multiplexes these bitstreams to generate and output a compressed bitstream. The output of the compressed bitstream by the multiplexer 1214 may correspond to the output of the bitstream by the encoding device 100.

[0252] The process of encoding wavelet coefficients into a displacement bitstream, which is performed by the image packer 1210 and the video encoder 1211, may be performed by arithmetic coding. Alternatively, the image packer 1210 and the video encoder 1211 may be configured to select whether the process is performed by the image packer 1210 and the video encoder 1211 (also referred to as video coding) or by arithmetic coding. An example of such a configuration is described below.

[0253] Fig. 30 is a block diagram showing a modification of the detailed configuration of the encoding device 100. Fig. 30 shows a modification of the functional blocks enclosed by the dashed line in Fig. 29.

[0254] The displacement vector calculator 1207, wavelet transformer 1208, quantizer 1209, image packer 1210, and video encoder 1211 shown in FIG. 30 are the same as those shown in FIG.

[0255] As shown in FIG. 30, the encoding device 100 further includes a switch 1221, a switch 1222, and an arithmetic encoder 1223.

[0256] The switch 1221 and the switch 1222 are switchers that respectively switch whether the process of encoding wavelet coefficients into a displacement bitstream is performed by the image packer 1210 and the video coder 1211, or by the arithmetic coder 1223.

[0257] The switches 1221 and 1222 may dynamically switch the components that perform the above processing between the image packer 1210 and the video encoder 1211, and the arithmetic encoder 1223. Furthermore, the switches 1221 and 1222 may always (in other words, fixedly) use the image packer 1210 and the video encoder 1211 as the components that perform the above processing, or may always (in other words, fixedly) use the arithmetic encoder 1223.

[0258] The arithmetic coder 1223 performs the process of encoding the wavelet coefficients into a displacement bitstream by arithmetic coding.

[0259] The encoding device 100 may add information to the header information indicating whether the process of encoding the wavelet coefficients into the displacement bitstream was performed by the image packer 1210 and the video encoder 1211 (in other words, by a video encoding process) or by the arithmetic encoder 1223 (in other words, by an arithmetic encoding process). In this way, the decoding device 200 that receives the bitstream encoded as described above can appropriately decode the bitstream by referring to the header information and switching the decoding method for decoding the bitstream.

[0260] The encoding process performed by the encoding device 100 will be described in detail below.

[0261] Fig. 31 is a flow diagram showing the processing of the encoding device 100. Fig. 32 is an explanatory diagram conceptually showing the encoding of mesh frames. The processing of the encoding device 100 will be described with reference to Figs. 31 and 32.

[0262] In step S101, the encoding device 100 reads a 3D mesh frame, which is an input mesh frame, and its attributes. The input mesh frame is a mesh frame input to the encoding device 100. An example of the 3D mesh frame that is an input mesh frame is shown as mesh frame 1301 (see FIG. 32 ).

[0263] In step S102, the encoding device 100 performs a decimation process on the input mesh frame read in step S101 to generate a base mesh frame having fewer vertices than the input mesh frame. The base mesh frame generated by decimating the mesh frame 1301 is shown as a base mesh frame 1302 (see FIG. 32).

[0264] In step S103, the encoding device 100 calculates displacement information that the decoding device 200 uses to reconstruct a mesh frame. The displacement information corresponds to a displacement vector directed from a vertex of the base mesh frame generated in step S102 to a vertex of the input mesh frame. One method for calculating the displacement information is to subtract the coordinates of the vertex of the base mesh frame from the coordinates of the vertex of the input mesh frame. The displacement information calculated from the mesh frame 1301 and the base mesh frame 1302 is shown as displacement information 1303 (see FIG. 32). The displacement information 1303 is in vector format, in other words, expressed as a displacement vector.

[0265] In step S104, the encoding device 100 encodes the base mesh frame generated in step S102, the displacement information generated in step S103, and the attributes of the input mesh frame into a bitstream (corresponding to a compressed bitstream). An example of the bitstream is shown as bitstream 1304 (see FIG. 32).

[0266] Specifically, the bitstream 1304 includes vertex coordinates and connectivity information for vertices A, C, E, and F, displacement information, a video bitstream including texture data, and a compressed attribute map (see FIG. 32). The displacement information includes displacement information for displacing vertices based on vertex coordinates obtained from the subdivided base mesh frame. The compressed attribute map is texture coordinates for applying texture data to a mesh frame reconstructed using the base mesh frame and the displacement information.

[0267] The detailed configuration of the decoding device 200 will be described below.

[0268] FIG. 33 is a block diagram showing a detailed configuration example of the decoding device 200.

[0269] As shown in Figure 33, the decoding device 200 comprises a demultiplexer 2201, a base mesh decoder 2202, an inverse quantizer 2203, a subdivision unit 2204, a video decoder 2205, an image unpacker 2206, an inverse quantizer 2207, an inverse wavelet transformer 2208, a reconstructor 2209, a video decoder 2210, and a color transformer 2211.

[0270] The demultiplexer 2201 receives the compressed bitstream input to the decoding device 200 and separates it into a base mesh bitstream, a displacement bitstream, and an attribute bitstream. The demultiplexer 2201 provides the base mesh bitstream to the base mesh decoder 2202, the displacement bitstream to the video decoder 2205, and the attribute bitstream to the video decoder 2210. The compressed bitstream input to the decoding device 200 may be, for example, a compressed bitstream output by the encoding device 100, and this case will be described as an example.

[0271] The base mesh decoder 2202 obtains a quantized base mesh by decoding the base mesh bitstream provided by the demultiplexer 2201. The base mesh decoder 2202 provides the quantized base mesh to the inverse quantizer 2203.

[0272] The inverse quantizer 2203 generates a base mesh (also called a decoded base mesh) by inverse quantizing the quantized base mesh provided by the base mesh decoder 2202. The inverse quantizer 2203 provides the decoded base mesh to the subdivision unit 2204.

[0273] The subdivider 2204 performs a subdivision process on the decoded base mesh generated by the inverse quantizer 2203. The subdivision process is similar to the subdivision process performed by the subdivider 1206. The subdivider 2204 provides the subdivided decoded base mesh to the reconstructor 2209.

[0274] The video decoder 2205 decodes the displacement bitstream provided by the demultiplexer 2201 into an image, which may be an image stored by mapping quantized wavelet coefficients to pixels in a two-dimensional image format, and provides the image to the image unpacker 2206.

[0275] The image unpacker 2206 extracts quantized wavelet coefficients from the image provided by the video decoder 2205. The process of extracting the quantized wavelet coefficients from the image may use a mapping that represents the assignment of the quantized wavelet coefficients to pixels in a two-dimensional image format. The image unpacker 2206 provides the quantized wavelet coefficients extracted from the image to the inverse quantizer 2207.

[0276] The inverse quantizer 2207 generates wavelet coefficients by inverse quantizing the quantized wavelet coefficients provided by the image unpacker 2206 .

[0277] The inverse wavelet transformer 2208 generates a displacement vector (corresponding to a decoded displacement vector) by performing an inverse wavelet transform process on the wavelet coefficients provided by the inverse quantizer 2207. The inverse wavelet transform process corresponds to the inverse transform of the wavelet transform process performed by the wavelet transformer 1208. The inverse wavelet transformer 2208 provides the generated decoded displacement vector to the reconstructor 2209.

[0278] The reconstructor 2209 reconstructs a mesh (corresponding to a decoded mesh frame) using the subdivided decoded base mesh provided by the subdivider 2204 and the decoded displacement vectors provided by the inverse wavelet transformer 2208. The reconstructor 2209 also outputs the reconstructed decoded mesh as the output mesh 2107.

[0279] The video decoder 2210 decodes the attribute bitstream provided by the demultiplexer 2201 into an attribute map (corresponding to a decoded attribute map). The video decoder 2210 provides the decoded attribute map to a color converter 2211.

[0280] The color converter 2211 performs color conversion processing on the decoded attribute map provided by the video decoder 2210. The color conversion processing corresponds to the inverse conversion of the color conversion processing performed by the color converter 1212, and may include conversion processing of color representation formats or color spaces. The color converter 2211 outputs the decoded attribute map after the color conversion processing.

[0281] The process of decoding the displacement bitstream into wavelet coefficients, performed by the video decoder 2205 and the image unpacker 2206, may be performed by arithmetic coding. Furthermore, the configuration may be such that it is selectable whether the process is performed by the video decoder 2205 and the image unpacker 2206 (also referred to as video decoding) or by arithmetic coding. An example of such a configuration is described below.

[0282] Fig. 34 is a block diagram showing a modification of the detailed configuration of the decoding device 200. Fig. 34 shows a modification of the functional blocks enclosed by the dashed line in Fig. 33.

[0283] The video decoder 2205, image unpacker 2206, inverse quantizer 2207, inverse wavelet transformer 2208, and reconstructor 2209 shown in FIG. 34 are the same as those shown in FIG.

[0284] As shown in FIG. 34, the decoding device 200 further includes a switch 2221, a switch 2222, and an arithmetic decoder 2223.

[0285] The switch 2221 and the switch 2222 are switchers that respectively switch whether the process of decoding the displacement bitstream into wavelet coefficients is performed by the video decoder 2205 and image unpacker 2206, or by the arithmetic decoder 2223.

[0286] The switch 2221 and the switch 2222 may dynamically switch the components that perform the above processing between the video decoder 2205 and the image unpacker 2206 and the arithmetic decoder 2223. Furthermore, the switch 2221 and the switch 2222 may always (in other words, fixedly) use the video decoder 2205 and the image unpacker 2206 as the components that perform the above processing, or may always (in other words, fixedly) use the arithmetic decoder 2223.

[0287] The arithmetic decoder 2223 performs arithmetic decoding to decode the displaced bitstream into wavelet coefficients.

[0288] Note that the header information may include information indicating whether the process of decoding the displaced bitstream into wavelet coefficients was performed by the video decoder 2205 and the image unpacker 2206 (in other words, by a video decoding process) or by the arithmetic decoder 2223 (in other words, by an arithmetic decoding process). In this case, the decoding device 200 can appropriately decode the bitstream by switching the decoding method for the bitstream by referring to the header information.

[0289] The decoding process performed by the decoding device 200 will be described in detail below.

[0290] Fig. 35 is a flow diagram showing the processing of the decoding device 200. Fig. 36 is an explanatory diagram conceptually showing the decoding of a mesh frame (3D mesh). The processing of the decoding device 200 will be described with reference to Figs. 35 and 36.

[0291] In step S201, the decoding device 200 decodes a base mesh frame and attributes from a bitstream (corresponding to a compressed bitstream). An example of the decoded base mesh frame (corresponding to a decoded base mesh frame) is shown as a decoded base mesh frame 2301 (see FIG. 36).

[0292] In step S202, the decoding device 200 generates subdivided vertices by performing a subdivision process on the base mesh frame decoded in step S201. An example of a base mesh frame (mesh frame) including subdivided vertices is shown as a base mesh frame 2302 (see FIG. 36).

[0293] In step S203, the decoding device 200 decodes the disparity information from the bitstream (corresponding to the compressed bitstream). An example of the decoded disparity information is shown as disparity information 2303 (see FIG. 36). The disparity information 2303 is in vector format, in other words, expressed as a disparity vector.

[0294] In step S204, the decoding device 200 reconstructs the shape of the mesh frame by moving the vertices of the base mesh frame, including the subdivided vertices, to new positions using the displacement information, and then restores the mesh frame by applying attribute information. An example of the attribute is texture. An example of the reconstructed mesh frame is shown as mesh frame 2304 (see FIG. 36 ).

[0295] The subdivision is described below and is performed by a subdivider (specifically subdivider 1206 or subdivider 2204).

[0296] FIG. 37 is an explanatory diagram showing an example of subdivision.

[0297] The base mesh shown in FIG. 37(a) includes vertices A, B, and C and connectivity information indicating their connectivity.

[0298] 37(b) shows a mesh generated by the first subdivision, in other words, the mesh after the first subdivision. In the first subdivision, the subdivider generates vertices D, E, and F and connectivity information indicating their connectivity. The mesh generated by the subdivider is also referred to as LoD1 or first LoD.

[0299] Vertex D of the mesh after the first subdivision is a vertex generated by subdivision based on vertices A and B. Similarly, vertex F is a vertex generated by subdivision based on vertices B and C. Vertex E is a vertex generated by subdivision based on vertices A and C.

[0300] As an example, vertex D may be the midpoint of line segment AB (in other words, side AB) connecting vertices A and B that were the basis for its generation. Similarly, vertex E may be the midpoint of line segment AC. Vertex F may be the midpoint of line segment BC.

[0301] 37(c) shows the mesh generated by the second subdivision, i.e., the mesh after the second subdivision. In the second subdivision, the subdivider generates vertices G, H, I, J, K, L, M, N, and O and connectivity information indicating their connectivity. The mesh generated by the subdivider is also called LoD2 or second LoD.

[0302] Vertex G of the mesh after the second subdivision is a vertex generated by subdivision based on vertices A and D. Similarly, vertex H is a vertex generated by subdivision based on vertices A and E. Vertex I is a vertex generated by subdivision based on vertices B and D. Vertex J is a vertex generated by subdivision based on vertices D and F. Vertex K is a vertex generated by subdivision based on vertices E and F. Vertex L is a vertex generated by subdivision based on vertices C and E. Vertex M is a vertex generated by subdivision based on vertices B and F. Vertex N is a vertex generated by subdivision based on vertices C and F. Vertex O is a vertex generated by subdivision based on vertices D and E.

[0303] As an example, vertex G may be the midpoint of line segment AD (in other words, side AD) connecting vertices A and D, which were the source of its generation. Similarly, vertex H may be the midpoint of line segment AE. vertex I may be the midpoint of line segment BD. vertex J may be the midpoint of line segment DF. vertex K may be the midpoint of line segment EF. vertex L may be the midpoint of line segment CE. vertex M may be the midpoint of line segment BF. vertex N may be the midpoint of line segment CF. vertex O may be the midpoint of line segment DE.

[0304] The displacement of vertices will be described below with reference to Figures 38 and 39. The displacement of vertices is performed by the reconstructor 2209.

[0305] Fig. 38 is an explanatory diagram showing an example of displacement of vertices after subdivision, and Fig. 39 is an explanatory diagram showing an example of vertices of an original mesh.

[0306] The base mesh shown in FIG. 38(a) includes vertices A, B, C, and Z and connectivity information indicating their connectivity.

[0307] 38(b) shows a mesh generated by the first subdivision, in other words, a mesh after the first subdivision (i.e., the first LoD). In the first subdivision, the subdivider generates vertices S, T, U, X, or Y and connectivity information indicating their connectivity. The vertices S, T, U, X, or Y are similar to the vertices D, E, and F shown in FIG. 37(b).

[0308] 38(c) shows a mesh generated by the second subdivision, in other words, a mesh after the second subdivision (i.e., the second LoD). In the second subdivision, the subdivider generates vertices D, E, F, G, and H and connectivity information indicating their connectivity. Vertices D, E, F, G, and H are similar to vertices G, H, I, J, K, L, M, N, and O shown in FIG. 37(c).

[0309] Figure 38(d) shows a mesh including the vertices after they have been displaced after subdivision, with vertices A, B, C, D, E, F, G, H, S, T, U, X, Y, and Z shown in Figure 38(d) being located at positions displaced using displacement information from the positions of the vertices shown in Figure 38(c).

[0310] The original mesh shown in FIG. 39 is an example of the mesh input to the encoding device 100, that is, the mesh before encoding.

[0311] The mesh shown in Fig. 38 has a shape similar to that of the original mesh shown in Fig. 39. The displacement information is generated by the displacement vector calculator 1207 of the encoding device 100 as information indicating the displacement from the vertices of the base mesh to the vertices of the original mesh, and therefore, by reconstructing the mesh using the displacement information thus generated, a mesh having a shape similar to that of the original mesh is generated.

[0312] The decoding device 200 can output the mesh shown in FIG.

[0313] Next, the division of a mesh into sub-meshes will be described with reference to FIGS.

[0314] A mesh can be divided into smaller parts and coded separately, with the vertices of the mesh being divided in such a way that the coordinates and connectivity of the vertices in each part can be coded independently.

[0315] Fig. 40 is an explanatory diagram showing an example of a mesh, and Fig. 41 is an explanatory diagram showing an example of dividing a mesh into sub-meshes.

[0316] The mesh shown in FIG. 40 is the original mesh, which is sometimes called a full mesh in contrast to a sub-mesh.

[0317] Figure 41 shows how the full mesh shown in Figure 40 is divided into two sub-meshes. For vertices A, B, and C of the full mesh (see Figure 40), vertex A is duplicated to vertices A1 and A2, vertex B is duplicated to vertices B1 and B2, and vertex C is duplicated to vertices C1 and C2, thereby creating two sub-meshes (i.e., a first sub-mesh and a second sub-mesh) from the full mesh. The first sub-mesh and the second sub-mesh are each independently decodable meshes.

[0318] Packing of displacement information into image frames will be described below with reference to FIGS.

[0319] 42, 43 and 44 are explanatory diagrams showing examples of packing of displacement information into image frames. Note that image frames can also be called video frames.

[0320] The vertex displacement data is encoded as image frame data by being mapped to each component of a YUV format image frame (i.e., each of the Y component (Y Plane), U component (U Plane), and V component (V Plane)). This case will be described below as an example. As another example, the vertex displacement data may be encoded as image frame data by being mapped to each component of an RGB format image frame (each of the R component, G component, and B component).

[0321] The decoding device 200 can use an image encoding module to extract the displacement data. The displacement data can be in the form of X, Y, or Z components in a global coordinate system (e.g., a Cartesian coordinate system), or normal, tangential, or both tangential components in a local coordinate system. Methods for mapping the displacement data to an image frame include the following:

[0322] For example, in the first method, the displacement data is arranged in the image frame in scan order, and an example of packing the displacement data in this case is shown in Figure 42. The displacement data is directly mapped onto the image frame according to a predefined scan order.

[0323] Note that since an image frame has a fixed height and width, it may happen that the displacement data does not fit perfectly in the frame, in which case the remaining part of the image frame is padded with padding data (see Figure 42).

[0324] For example, in the second method, the displacement data is separated into multiple LoDs and mapped to the Y, U, and V components of the image frame. An example of packing of the displacement data in this case is shown in Figure 43. Here, the displacement data of the image frame of the next LoD starts immediately after the displacement data of the previous LoD ends. As in the first method, if the displacement data does not fit exactly into the image frame, padding is performed at the end of the image frame (see Figure 43).

[0325] For example, in the third method, displacement data corresponding to the LoD is mapped to the Y component, U component, and V component of the image frame in a manner different from that in the second method. An example of packing of the displacement data in this case is shown in Figure 44. In this way, each LoD can be decoded independently. In the third method, middle padding is performed on the displacement data of each LoD, and CTU alignment is performed together with padding at the end of the video frame (see Figure 44).

[0326] Next, the encoding device 100 and the decoding device 200 when a mesh is divided into a plurality of sub-meshes will be described.

[0327] Fig. 45 is a diagram showing another example of the configuration of the encoding device 100 according to an embodiment. Specifically, Fig. 45 is a diagram showing the configuration of a submesh encoding device that performs encoding processing when an input mesh 1101 is divided into divided meshes (plurality of submeshes). For example, the submesh encoding device includes a plurality of encoding devices 100.

[0328] An input mesh 1101 (full mesh) input to the submesh encoding device is divided into multiple meshes (submeshes). The multiple submeshes are input to, for example, multiple encoding devices 100. Each of the multiple submeshes may be input to one of the multiple encoding devices 100. For example, the submesh encoding device divides the input mesh 1101 into multiple submeshes and inputs the divided multiple submeshes to the multiple encoding devices 100.

[0329] After the image is divided into a plurality of sub-meshes, coding processing is performed on the boundaries of the sub-meshes (processing of overlapping sub-meshes).

[0330] For example, for each sub-mesh, preprocessing is performed by the preprocessor 1103, and a base mesh, displacement data, and metadata are generated and encoded.

[0331] The encoding device 100 may be realized by such a submesh encoding device configuration. That is, the encoding device 100 may be configured to include a plurality of preprocessors 1103 and compressors 1106, and may perform a predetermined process on the submesh for each of a plurality of pairs of preprocessors 1103 and compressors 1106. Furthermore, the number of pairs of preprocessors 1103 and compressors 1106 included in the encoding device 100 may be any number and is not particularly limited.

[0332] FIG. 46 is a diagram illustrating an example of the configuration of the preprocessor 1103 according to the embodiment.

[0333] The preprocessor 1103 includes, for example, a base mesh generator 1401 , a subdivision unit 1402 , and a displacement data generator 1403 .

[0334] First, in the preprocessor 1103, a base mesh is generated by the base mesh generator 1401.

[0335] The base mesh is then subdivided in a predetermined manner by a subdivider 1402 to generate a subdivided mesh (or a subdivided base mesh) that is the subdivided base mesh.

[0336] Displacement data is generated by a displacement data generator 1403 from the sub-divided mesh and the sub-mesh that is the input mesh 1101 after division.

[0337] The displacement data is, for example, a difference vector between the input mesh 1101 and the sub-division mesh.

[0338] The same subdivision method as that used in encoding is used in decoding as well.

[0339] Furthermore, for example, the encoding device 100 may transmit to the decoding device 200 a subdivision method for encoding and parameters used in the subdivision.

[0340] Fig. 47 is a diagram showing another example of the configuration of the decoding device 200 according to the embodiment. Specifically, Fig. 47 is a diagram showing the configuration of a submesh decoding device, which is a device that performs decoding processing when a bitstream 2101 includes multiple submeshes. For example, the submesh decoding device includes multiple decoding devices 200. For example, the submesh decoding device includes multiple decoding devices 200 and a combiner 2109.

[0341] The coded data for each submesh included in the bit stream 2101 is input to the decompressor 2102 of each decoding device 200 .

[0342] Also, for example, the post-processor 2106 executes the processing of the reconstructor 2209 described above.

[0343] In the post-processor 2106, for each decoded sub-mesh, the base mesh is subdivided and the displacement vector is added to the subdivided base mesh to reconstruct the sub-mesh.

[0344] That is, the post-processor 2106 performs the processing of the reconstructor 2209 for each sub-mesh.

[0345] The combiner 2109 combines (merges) the sub-meshes restored by the respective decoding devices 200 to reconstruct the full mesh (output mesh 2107) before division.

[0346] Note that the decoding device 200 may be realized by such a submesh decoding device configuration. That is, the decoding device 200 may be configured to include a plurality of decompressors 2102 and post-processors 2106, and each of the plurality of pairs of decompressors 2102 and post-processors 2106 may perform a predetermined process on the encoded data for each submesh. Furthermore, the number of pairs of decompressors 2102 and post-processors 2106 included in the decoding device 200 may be any number and is not particularly limited.

[0347] Fig. 48 is a diagram showing a specific example of the configuration of the decoding device 200 according to the embodiment. Specifically, Fig. 48 shows a specific configuration of the post-decoder 2306 out of the decoder 2305 and the post-decoder 2306 included in the decoding device 200.

[0348] The post-decoder 2306 comprises a pre-reconstructor 2307 , a reconstructor 2308 , a post-reconstructor 2309 and an adaptor 2310 .

[0349] The processing in the post-decoder 2306 is optional depending on the application. An example of the processing in the post-decoder 2306 (post-decoding processing) is conversion of the decoded data to a nominal format, such as video conversion from YUV space to RGB space. The post-decoding processing can be encapsulated into multiple processes, such as pre-reconstruction, reconstruction, post-reconstruction, and adaptation.

[0350] The pre-reconstructor 2307 performs a pre-reconstruction process, for example, upscaling the normalized texture coordinates to match the dimensions of the texture image in the context of video-based dynamic mesh coding.

[0351] The reconstructor 2308 performs a reconstruction process, which is invoked, for example, on the decoded atlas frame, the decoded base mesh frame, the decoded video frame, and syntax elements associated with the same mesh sequence. The output of the reconstruction process is a series of reconstructed mesh frames prior to a post-reconstruction process.

[0352] The post-reconstructor 2309 performs post-reconstruction operations, e.g., in the context of video-based dynamic mesh coding, which perform a number of smoothing operations on the reconstructed mesh frames, such as collapsing edges in the mesh or adding new vertices in the mesh.

[0353] The adaptor 2310 performs a fitting process, which may be applied by some application to fit the reconstructed mesh to a given scenario. For example, the vertices of the reconstructed mesh are transformed from the 3D model coordinate system to the 3D world coordinate system. The adaptor 2310 outputs a final reconstructed mesh frame (final 3D mesh frame 2311).

[0354] Below, we will explain a method for detecting overlapping vertices of submeshes or overlapping edges connecting each overlapping vertex of a submesh in the decoding device 200 when multiple submeshes overlap, and a method for generating and transmitting metadata (parameters for each pair of boundary vertices described later) for detecting overlapping vertices or overlapping edges by the encoding device 100.

[0355] The overlapping edges and overlapping vertices are sometimes called boundary edges and boundary vertices, respectively.

[0356] In order to optimize the bitstream size and decoding complexity, and to detect the boundary vertices of the sub-mesh in an efficient way, a new method is described below that requires fewer bits and reduces the decoding time.

[0357] A boundary vertex is a vertex on an edge in a 3D mesh that is connected to only one face. A boundary vertex is a vertex that constitutes a first submesh (also called a vertex owned by the first submesh, a vertex that belongs to the first submesh, or a vertex that is included in the first submesh), and can also be a vertex located in a similar or identical position to a vertex that constitutes a second submesh. A boundary vertex can have multiple faces connected to it. In other words, a boundary vertex contacts multiple faces in the submesh.

[0358] A parameter for each pair of boundary vertices may be signaled to indicate whether a different third vertex of the pair is a boundary vertex.

[0359] For example, the decoding device 200 first identifies the outermost vertices (also called outermost vertices) of a submesh by checking the connectivity information of the three-dimensional mesh. The outermost vertices are a subset of the boundary vertices.

[0360] Next, the decoding device 200 decodes the parameters for each pair of outermost vertices connected via an edge, and further determines the boundary vertices.

[0361] 49 is a flow diagram showing processing for a submesh boundary by the encoding device 100 according to an embodiment. Note that the encoding device 100 may transmit overlap information between submeshes to the decoding device 200 by signaling.

[0362] A submesh boundary is, for example, a position in a submesh where it joins with other submeshes, and is a position that includes vertices and edges that form the outer periphery (outermost periphery) of the submesh. A submesh boundary is, for example, a portion of a submesh that is formed by boundary vertices and boundary edges.

[0363] First, the encoding device 100 determines whether submeshes overlap (S301). For example, the encoding device 100 determines, for a submesh among a plurality of submeshes, whether the submesh overlaps with any of the other submeshes among the plurality of submeshes. Submeshes overlap when one submesh is combined with another submesh, meaning that at least a portion of the vertices, edges, and / or faces of the one submesh overlap with the other submesh. Submeshes overlap when one submesh shares a vertex, edge, and / or face with another submesh in the original mesh (full mesh), such as a boundary edge. For example, a first submesh and a second submesh overlap when they contain vertices, edges, or faces with the same coordinates.

[0364] When the encoding device 100 determines that there is an overlap between the submeshes (Yes in S301), it determines whether only the vertices and edges of the boundaries of the submeshes (also simply referred to as submesh boundaries) overlap (S302). That is, the encoding device 100 determines whether the boundary of one submesh overlaps (shares) with another submesh, and whether the portion of the one submesh other than the boundary (also referred to as the inside of the submesh boundaries or simply the inside) does not overlap (share) with another submesh.

[0365] Depending on the result of the determination in step S302, submesh overlap area determination is performed in steps S303 to S304 or steps S306 to S308 to determine (detect) submesh overlap areas, which are areas that overlap between submeshes (e.g., vertices, edges, and / or faces of submeshes), and processing of the submesh overlap areas is performed in step S305 or step S309.

[0366] When the encoding device 100 determines that only the vertices and edges of the submesh boundaries overlap (Yes in S302), it detects overlapping vertices and overlapping edges of the submesh boundaries (S303).

[0367] Next, the encoding device 100 signals overlap information including information about overlapping submesh boundaries to the bitstream (S304). The overlap information is, for example, information indicating overlapping vertices and overlapping edges.

[0368] Next, the encoding device 100 performs processing using overlapping vertices and overlapping edges on the submesh boundaries (S305).

[0369] On the other hand, if the encoding device 100 determines that not only the vertices and edges of the submesh boundary are overlapping, that is, that in addition to the vertices and edges of the submesh boundary, at least one of the vertices, edges, and faces within the submesh boundary is also overlapping (No in S302), it first detects overlapping vertices and overlapping edges of the submesh boundary (S306).

[0370] Next, the encoding device 100 detects overlapping vertices and overlapping edges within the submesh boundaries (S307).

[0371] Next, the encoding device 100 signals overlap information, including information about submesh boundaries and overlaps within the submesh boundaries, into the bitstream (S308).

[0372] Next, the encoding device 100 performs processing using either or both of the overlapping vertices and overlapping edges on the submesh boundary and the overlapping vertices and overlapping edges inside the submesh boundary (S309).

[0373] If the encoding device 100 determines that there is no overlap between the sub-meshes (No in S301), it does not perform processing related to the sub-mesh overlap.

[0374] The detection of submesh overlapping areas is performed using, for example, a divided mesh such as a submesh. The detection of submesh overlapping areas may be performed at any timing after the original mesh is divided into submeshes. For example, the encoding device 100 may determine which vertices to overlap when dividing the original mesh into submeshes and store information indicating the determined vertices in a memory or the like. Alternatively, the encoding device 100 may examine all vertices of the divided submeshes after dividing the original mesh into submeshes and determine overlapping vertices and edges.

[0375] Furthermore, as described above, the encoding device 100 determines overlapping areas of submesh boundaries and switches processing depending on whether the overlapping areas are (A) vertices and edges of the submesh boundaries, or (B) vertices and edges of the submesh boundaries and within the submesh boundaries.

[0376] In the case of (A) above, the encoding device 100 detects overlapping areas of submesh boundaries using a predetermined method, and transmits overlap information indicating the detection results to the decoding device 200 by including the overlap information in the bitstream.

[0377] In addition, in the above case (B), the encoding device 100 detects overlapping areas at submesh boundaries and overlapping areas within submesh boundaries using a predetermined method, and includes overlapping information indicating the detection results in a bitstream and transmits it to the decoding device 200.

[0378] Note that if the decoding device 200 can detect overlapping areas of sub-meshes in the same manner as the encoding device 100, the encoding device 100 does not need to signal overlapping information in the bitstream.

[0379] For example, if the decoding device 200 is capable of detecting submesh boundaries, the encoding device 100 may signal redundant information containing only information within the submesh boundaries to the bitstream when performing processing in case (B) above.

[0380] Furthermore, the predetermined method for detecting overlapping regions of submesh boundaries and the method for detecting overlapping regions inside submesh boundaries may be determined arbitrarily and are not particularly limited.

[0381] FIG. 50 is a flowchart showing an example of processing performed on a submesh boundary by the decoding device 200 according to the embodiment.

[0382] First, the decoding device 200 analyzes the metadata (S311). Specifically, the decoding device 200 obtains the duplication information, which is an example of the metadata included in the bitstream received from the encoding device 100.

[0383] Next, the decoding device 200 determines whether or not there is overlap between the sub-meshes based on the metadata (specifically, the overlap information) (S312).

[0384] If the decoding device 200 determines that there is overlap between the sub-meshes (Yes in S312), it further determines, based on the metadata, whether only the vertices and edges of the boundaries of the sub-meshes overlap (S313).

[0385] Depending on the result of the determination in step S313, step S314 or steps S316 to S318 for determining a submesh overlapping area, and processing for the submesh overlapping area in step S315 or step S319 are executed.

[0386] When the decoding device 200 determines that only the vertices and edges of the submesh boundaries overlap (Yes in S313), it detects overlapping vertices and overlapping edges of the submesh boundaries (S314).

[0387] Next, the decoding device 200 performs processing using overlapping vertices and overlapping edges on the submesh boundaries (S315).

[0388] On the other hand, if the decoding device 200 determines that not only the vertices and edges of the submesh boundary are overlapping, that is, that in addition to the vertices and edges of the submesh boundary, at least one of the vertices, edges, and faces within the submesh boundary is also overlapping (No in S313), it first detects overlapping vertices and overlapping edges of the submesh boundary (S316).

[0389] Next, the decoding device 200 analyzes the metadata of the overlap information of the overlapping vertices and overlapping edges inside the submesh boundaries (S317). Specifically, the decoding device 200 acquires (extracts) the overlap information including the information about the overlapping vertices and overlapping edges inside the submesh boundaries from the metadata included in the bitstream.

[0390] Next, the decoding device 200 detects overlapping vertices and overlapping edges within the submesh boundaries based on the analyzed metadata (S318).

[0391] Next, the encoding device 100 performs processing using either or both of the overlapping vertices and overlapping edges on the submesh boundary and the overlapping vertices and overlapping edges inside the submesh boundary (S319).

[0392] If the decoding device 200 determines that there is no overlap between the sub-meshes (No in S312), it does not execute processing related to the sub-mesh overlap.

[0393] As described above, the decoding device 200 determines the overlapping area of ​​the submesh boundary and switches processing depending on whether the overlapping area is (A) the vertices and edges of the submesh boundary, or (B) the boundary of the submesh and the vertices and edges within the submesh boundary.

[0394] 50 is an example in which the encoding device 100 and the decoding device 200 can detect overlapping vertices and overlapping edges at submesh boundaries using a predetermined method, and only information about the interior of the submesh boundaries is signaled in the bitstream. In the above case (B), the decoding device 200 can detect overlapping vertices and overlapping edges at the interior of the submesh boundaries by analyzing overlap information (metadata) that includes information about overlaps within the submesh boundaries.

[0395] FIG. 51 is a flowchart showing another example of processing performed on a submesh boundary by the decoding device 200 according to the embodiment.

[0396] This example differs from the example shown in Fig. 50 in the processing for sub-mesh overlapping areas. In the processing for the sub-mesh overlapping areas, step S321 is performed, followed by step S322 or step S323.

[0397] After step S314 or step S318, the decoding device 200 determines whether or not there is an overlap between the edges of the sub-meshes (S321).

[0398] When the decoding device 200 determines that an edge between submeshes overlaps (Yes in S321), it executes processing using a subdivision method and a subdivision count that are common to the plurality of submeshes (S322). That is, the decoding device 200 performs subdivision of the submeshes using a subdivision method and a subdivision count that are common to the submeshes.

[0399] On the other hand, if the decoding device 200 determines that the edges between the submeshes do not overlap (No in S321), it performs processing for each of the multiple submeshes using an individual subdivision method and subdivision count (S323). That is, the decoding device 200 performs subdivision of the submeshes using the individual subdivision method and subdivision count for each submesh.

[0400] 52 is a flowchart showing processing for a submesh by the decoding device 200 according to the embodiment. Specifically, FIG. 52 is a flowchart showing an example of switching processing by determining the submesh boundary and the vertices and edges within the submesh boundary, as in (B) above.

[0401] First, the decoding device 200 decodes a first vertex and a second vertex from the coded bitstream (S331). The first vertex and the second vertex are, for example, two points constituting an edge (i.e., the vertices at both ends of the edge).

[0402] Next, the decoding device 200 determines whether the first vertex and the second vertex are boundary vertices (S322). That is, the decoding device 200 determines whether the edge formed by the first vertex and the second vertex is a boundary edge.

[0403] When the decoding device 200 determines that the first vertex and the second vertex are boundary vertices (Yes in S322), it decodes a first parameter from the bitstream (S333). Specifically, the decoding device 200 acquires the first parameter, which is information for detecting an overlapping area within a submesh boundary, from metadata included in the bitstream. The first parameter is, for example, the above-mentioned overlapping information.

[0404] Next, the decoding device 200 determines whether the first parameter indicates that the third vertex is a boundary vertex (S334).

[0405] If the decoding device 200 determines that the first parameter indicates that the third vertex is a boundary vertex (Yes in S334), that is, if it determines that the first vertex, the second vertex, and the third vertex are all boundary vertices, it executes the first mesh decoding process (S335).

[0406] On the other hand, if the decoding device 200 determines that the first parameter does not indicate that the third vertex is a boundary vertex (No in S334), or if it determines that the first vertex and the second vertex are not boundary vertices (No in S322), that is, if it determines that at least one of the first vertex, the second vertex, and the third vertex is not a boundary vertex, it performs a second mesh decoding process that is different from the first mesh decoding process (S336).

[0407] This allows the process (specifically, the decoding process of the submesh) to be switched based on whether or not an overlapping edge is included inside the submesh boundary.

[0408] The first mesh decoding process and the second mesh decoding process may each be any process content and are not particularly limited.

[0409] As described above, the decoding device 200, for example, decodes the first vertex and the second vertex from a bit stream (encoded bit stream), determines whether the first vertex and the second vertex are boundary vertices, decodes a first parameter from the bit stream, determines whether the first parameter indicates that the third vertex is a boundary vertex, and if it is determined that the first vertex, the second vertex, and the third vertex are boundary vertices, performs a first mesh decoding process, and if it is determined that at least one of the first vertex, the second vertex, and the third vertex is not a boundary vertex, performs a second mesh decoding process that is different from the first mesh decoding process.

[0410] Also, for example, the first parameter is signaled in a base mesh sub-bitstream, an atlas frame parameter set, an atlas style header, a patch data unit, or SEI (Supplemental Enhancement Information).

[0411] Also, for example, a first vertex is connected to a second vertex via an edge.

[0412] Also, for example, the edge connecting the first vertex and the second vertex is part of only one face.

[0413] Furthermore, for example, when it is determined that the first vertex and the second vertex are boundary vertices, the decoding device 200 decodes the first parameter.

[0414] Furthermore, for example, the first vertex, the second vertex, and the third vertex belong to the same face in the submesh, i.e., the first vertex, the second vertex, and the third vertex are vertices that constitute a common face among the multiple faces in the submesh.

[0415] Furthermore, for example, the first vertex, the second vertex, and the third vertex are connected to each other.

[0416] Also, for example, when the decoding device 200 determines that the first vertex, the second vertex, and the third vertex are boundary vertices, it performs a first mesh decoding process, and the first mesh decoding process includes a post-decoding process, a reconstruction process, a pre-reconstruction process, a post-reconstruction process, or an adaptation process.

[0417] Also, for example, when the decoding device 200 determines that at least one of the first vertex, the second vertex, and the third vertex is not a boundary vertex, it performs a second mesh decoding process that is different from the first mesh decoding process, and the second mesh decoding process includes a post-decoding process, a reconstruction process, a pre-reconstruction process, a post-reconstruction process, or an adaptation process.

[0418] Fig. 53 is a diagram showing an example of a submesh according to an embodiment. Of the multiple vertices included in the submesh shown in Fig. 53, vertices A, B, C, D, E, and F are outermost vertices, and vertices G, H, and K are inner vertices. The inner vertices are vertices other than the outermost vertex among the multiple vertices included in the submesh.

[0419] In step S331, the decoding device 200 decodes a first vertex and a second vertex from the bitstream. The first vertex and the second vertex may belong to the same submesh. The first vertex and the second vertex may be connected via an edge.

[0420] [Method for detecting submesh boundary vertices] In addition, in step S332, the decoding device 200 determines whether the first and second vertices are boundary vertices. For example, the decoding device 200 determines whether the vertices are boundary vertices of the submesh by identifying the outermost vertices based on the connectivity information of the submesh.

[0421] An outermost vertex is a vertex whose connecting edges form the boundary of a submesh. For example, if the first vertex and the second vertex are both outermost vertices, the edge connecting a first vertex and a second vertex is part of only one of the faces in the submesh. For example, the outermost vertex is part of at least one edge that forms only one triangle together with other vertices included in the manifold in the submesh. A manifold represents a shape that has no self-intersections and no edges shared by two or more faces. For example, vertex A is part of edge AB that forms only one triangle ABK. On the other hand, for example, all edges tangent to vertex K are part of two triangles. Therefore, vertex K is not an outermost vertex. For example, whether or not a vertex is a boundary edge is determined based on whether or not the edge belongs to only one triangle in the submesh. For example, if the edge is determined to belong to only one triangle, the edge is determined to be a boundary edge, and if the edge is not determined to belong to only one triangle, the edge is determined not to be a boundary edge. Furthermore, for example, among one or more boundary edges, a boundary edge that overlaps (is shared with) another sub-mesh is determined to be an overlapping edge, and a boundary edge that does not overlap (is shared with) another sub-mesh is determined not to be an overlapping edge. For example, the vertices that make up a boundary edge are also called boundary vertices, and the vertices that make up an overlapping edge are also called overlapping vertices.

[0422] Fig. 54 is a diagram showing another example of a submesh according to an embodiment. Of the multiple vertices included in the submesh shown in Fig. 54, vertices A, B, C, D, E, F, K, and L are outermost vertices, and vertices G and H are inner vertices.

[0423] In step S332 above, for a non-manifold in a submesh such as that shown in Fig. 54, the decoding device 200 reads one or more parameters from the bitstream to identify which vertices in the non-manifold are the outermost vertices. To detect edges in the non-manifold, the decoding device 200 determines, for example, whether three or more faces are connected to the edge.

[0424] For example, vertex K shown in Figure 54 forms three triangles: triangle ABK, triangle BCK, and triangle LBK. Therefore, vertex K is part of edge BK of the non-manifold. To identify whether vertex K is the outermost vertex, decoding device 200 may decode one or more parameters.

[0425] [Method for detecting vertices inside a submesh boundary based on a first parameter signaled by a decoding device] Figure 55 is a diagram showing another example of a submesh according to an embodiment. Among the multiple vertices included in the submesh shown in Figure 55, vertices A, B, C, D, E, F, and H are boundary vertices, and vertices G and K are non-boundary vertices (vertices that are not boundary vertices).

[0426] In step S333, the decoding device 200 decodes the first parameter from the bitstream. The first parameter may be a flag.

[0427] In step S334, the decoding device 200 determines whether the first parameter indicates that the third vertex is a boundary vertex. The first vertex, the second vertex, and the third vertex may belong to the same submesh. The first vertex, the second vertex, and the third vertex may belong to the same face in the submesh. In other words, the first parameter is a parameter for each triangle that has at least two boundary vertices.

[0428] As shown in FIG. 55 , for example, a parameter indicating that triangle EFH has at least two boundary vertices (specifically, vertices E and F) is decoded. Based on the decoded parameter, vertex H is determined to be a boundary vertex. Similarly, for triangle AGF, a parameter indicating whether vertex G is a boundary vertex may be decoded because triangle AGF has at least two boundary vertices (specifically, vertices A and F). Based on the value of the decoded parameter, vertex G is determined to be a non-boundary vertex. For example, in step S333, such parameters are decoded from the bitstream. Also, in step S334, based on the decoded parameter, it is determined whether the third vertex is a boundary vertex. For example, if vertices E and F are boundary vertices in triangle EFH, the parameter indicates 1 (parameter=1), and therefore the third vertex, vertex H, is determined to be a boundary vertex. Furthermore, in the triangle AGF, when the vertices A and F are boundary vertices, the parameter indicates 0 (parameter=0), and therefore the third vertex, vertex G, is determined not to be a boundary vertex.

[0429] [Example of Process Switching Based on Whether or Not a Boundary is Included Within a Submesh Boundary] In step S335 described above, if the decoding device 200 determines that the first, second, and third vertices are boundary vertices, it performs a first mesh decoding process. For example, as the first mesh decoding process, the decoding device 200 performs subdivision (midpoint subdivision) to form a vertex at the midpoint of the edge connecting two boundary vertices. For example, the decoding device 200 performs an inverse linear wavelet transform on the edge connecting the two boundary vertices. For example, the decoding device 200 traverses the coefficients of the displacement vectors of the boundary vertices from low frequency to high frequency.

[0430] Furthermore, in step S336, if it is determined that at least one of the first vertex, the second vertex, and the third vertex is a non-boundary vertex, the decoding device 200 performs a second mesh decoding process that is different from the first mesh decoding process. For example, as the second mesh decoding process, the decoding device 200 performs butterfly subdivision on an edge that does not connect two boundary vertices. For example, the decoding device 200 performs an inverse butterfly wavelet transform on an edge that connects two boundary vertices. For example, the decoding device 200 traverses the coefficients of the displacement vectors of the non-boundary vertices from high frequencies to low frequencies.

[0431] The first mesh decoding process and the second sub-mesh decoding process may include additional processes such as a post-decoding process, a reconstruction process, a pre-reconstruction process, a post-reconstruction process, and an adaptation process. The first mesh decoding process may include at least one of processes such as a subdivision process, an inverse quantization process, an inverse image packing process, an inverse wavelet transform process, and a texture coordinate derivation process. For example, the decoding device 200 applies the subdivision process three times iteratively to an edge connecting two boundary vertices.

[0432] For example, in step S335, the subdivision process is repeated three times in the first mesh decoding process. Also, for example, in step S336, the subdivision process is repeated once in the second submesh decoding process to the edges that do not connect two boundary vertices. The number of subdivisions performed in steps S335 and S336 may be the same or different.

[0433] Also, for example, the decoding device 200 may apply midpoint subdivision to edges connecting boundary vertices in the first mesh decoding process, and may apply a subdivision different from midpoint subdivision, such as butterfly subdivision, to edges connecting non-boundary vertices in the second mesh decoding process.

[0434] As described above, for example, in step S335, in a surface that includes a submesh boundary and overlaps between submeshes, the decoding device 200 performs a first mesh decoding process if the third vertex is a boundary vertex (i.e., if all edges are boundary edges).

[0435] For example, in the first mesh decoding process, the decoding device 200 divides the boundary edge at its midpoint in the subdivision process. Also, for example, in the first mesh decoding process, the decoding device 200 applies a linear wavelet inverse transform. Also, for example, the decoding device 200 traverses the coefficients of the displacement vectors of the boundary vertices from low frequency to high frequency.

[0436] Furthermore, for example, in step S336, if the third vertex is not a boundary vertex (that is, if it includes an edge that is not a boundary edge), the decoding device 200 executes the second mesh decoding process.

[0437] For example, in the second mesh decoding process, the decoding device 200 applies butterfly division to boundary edges in the subdivision process. Also, for example, in the second mesh decoding process, the decoding device 200 applies an inverse butterfly wavelet transform. Also, for example, in the second mesh decoding process, the decoding device 200 traverses coefficients of displacement vectors of non-boundary vertices from low frequency to high frequency.

[0438] For example, in the subdivision process, if all edges in a submesh are overlapping edges (e.g., step S335), the decoding device 200 applies a common subdivision count to all overlapping edges across multiple submeshes. On the other hand, if a submesh includes an edge that is not an overlapping edge (e.g., step S336), the decoding device 200 applies a common subdivision count to the overlapping edges across the submeshes, and applies individual subdivision counts to the edges that are not overlapping edges for each submesh.

[0439] Fig. 56 is a diagram showing an example of syntax related to signaling of boundary information according to an embodiment. Specifically, Fig. 56 is a diagram showing an example of syntax elements required for signaling of boundary information. Boundary information is an example of overlap information.

[0440] apply_border_detection_flag, when its value is equal to 1, indicates that border detection is enabled and parameters for border vertex pairs are present in the bitstream.

[0441] Note that if the apply_border_detection_flag is not present in the bitstream, its value is assumed to be, for example, 0.

[0442] The value of num_border_flags_minus1 plus 1 specifies the number of border_flags[i] that are signaled in the bitstream. For example, num_border_flags_minus1 is predetermined to range from 0 to the maximum number of vertices.

[0443] If border_flag[i] is equal to 1, it indicates that the vertex associated with the i-th parameter is a border vertex. If border_flag[i] is equal to 0, it indicates that the vertex is not a border vertex.

[0444] Fig. 57 is a diagram illustrating another example of syntax related to signaling of boundary information according to an embodiment. Specifically, Fig. 57 illustrates an example of syntax elements necessary for signaling boundary information.

[0445] starting_vertex_id indicates the id of the first vertex (also called the starting vertex) referenced by border_flags[0].

[0446] ending_vertex_id indicates the id of the last vertex (also called the ending vertex) referenced by border_flags[num_border_flags_minus1].

[0447] The direction_flag indicates the direction in which the vertices are traversed. For example, if the value of the direction_flag is 0, it indicates that the vertices are traversed in a clockwise direction. If the value of the direction_flag is 1, it indicates that the vertices are traversed in a counterclockwise direction. If the direction_flag is not present, its value is assumed to be, for example, 0. In other words, the direction_flag indicates the order in which the vertices are processed.

[0448] Figure 58 is a diagram for explaining an example of how to use the syntax according to the embodiment. Specifically, Figure 58 is a diagram for explaining how to use the syntax shown in Figure 57. Of the multiple vertices included in the submesh shown in Figure 58, vertices A, B, C, D, E, F, and H are boundary vertices, and vertices G and K are non-boundary vertices. Furthermore, the numerical values ​​written in triangles AGF and EFH indicate the possible values ​​of border_flag_[i]. In this example, border_flag_[i] = 0 for triangle AGF, and border_flag_[i] = 1 for triangle EHF.

[0449] For example, suppose starting_vertex_id indicates vertex E (i.e., the starting vertex is vertex E). Also, suppose direction_flag = 0. In this case, traversal is performed clockwise in FIG. 58, and the next vertex is selected. In this example, when direction_flag = 0, the next vertex after vertex E is vertex F.

[0450] On the other hand, for example, if direction_flag = 1, then the next vertex is selected by traversing counterclockwise in Figure 58. In this example, when direction_flag = 1, the next vertex after vertex E is vertex D.

[0451] Also, for example, if ending_vertex_id indicates vertex B (that is, the ending vertex is vertex B), traversal is performed in order from vertex E to vertex B in the direction indicated by direction_flag.

[0452] In this way, for example, the encoding device 100 determines an arbitrary vertex of the submesh as the starting vertex, and processes each vertex at the submesh boundary in a predetermined direction (clockwise or counterclockwise).

[0453] The starting vertex and the predetermined direction are determined, for example, by an arbitrary method by the encoding device 100. For example, the encoding device 100 signals information indicating the starting vertex and the predetermined direction in a bitstream using starting_vertex_id and direction_flag, and transmits the signal to the decoding device 200.

[0454] In addition, the encoding device 100 may transmit information indicating the order of vertices to be processed corresponding to a specified direction to the decoding device 200 by signaling information indicating an identifier (id) indicating the second vertex in the processing order in addition to the starting vertex to the bitstream.

[0455] For example, the encoding device 100 uses border_flag[0] to include information indicating whether vertex H is a border vertex based on the starting vertex and the next vertices (vertices E and F in this example) in the bitstream. For example, the encoding device 100 repeats this process in order up to the ending vertex. The ending vertex may be determined by the encoding device 100 using any method. For example, the encoding device 100 signals the information indicating the ending vertex in the bitstream using ending_vertex_id and transmits it to the decoding device 200.

[0456] As described above, the encoding device 100 determines whether, for example, a vertex located outside the boundary of a submesh divided from an original mesh (e.g., a vertex not located on an edge constituting the periphery of the submesh, i.e., the above-mentioned inner vertex) overlaps with any of the vertices of another submesh divided from the original mesh. Furthermore, based on the determination result, the encoding device 100 encodes information such as the border_flag[i] into a bitstream. Furthermore, for example, the encoding device 100 further encodes into the bitstream a starting_vertex_id indicating the vertex on which a predetermined process, such as the first mesh decoding process or the second mesh decoding process, is first performed on the vertices of the submesh, a direction_flag indicating the order in which the predetermined process is performed on the vertices, and an ending_vertex_id indicating the vertex on which the predetermined process is last performed. Furthermore, based on the information coded (signaled) into the bitstream as described above, for example, the decoding device 200 determines whether there is overlap between submeshes, and decodes each submesh based on the determination result.

[0457] Fig. 59 is a diagram for explaining the order in which vertices are processed according to the embodiment. Fig. 60 to Fig. 62 are diagrams for explaining an example of the order in which vertices are processed according to the embodiment.

[0458] 59 to 62 indicate vertices of the sub-meshes. The solid lines in Fig. 59 to 62 indicate lines (boundaries) corresponding to the sub-mesh boundaries. Specifically, the solid lines in Fig. 59 to 62 connect the vertices in the order of processing.

[0459] When the submesh boundary (= overlapping boundary) shown in Figure 59 is circular, for example, Figures 60 to 62 are diagrams showing the boundary lines and overlapping lines in an expanded form. The overlapping boundary is, for example, the overlapping area described above, and the overlapping line is, for example, a line indicating the overlapping area. In Figures 60 to 62, the boundary lines (for example, boundary vertices and boundary edges) are shown with dotted lines, and the overlapping lines (for example, overlapping vertices and overlapping edges) are shown with solid lines.

[0460] 60 shows an example where all sub-mesh boundaries have overlapping vertices. In this example, start_vertex_id and end_vertex_id are identifiers that indicate the start and end vertices of the boundary line, and also the start and end vertices of the overlapping line.

[0461] The example shown in Figure 61 is a case where some of the sub-mesh boundaries have overlapping vertices. In this example, start_vertex_id and end_vertex_id are identifiers of the start and end vertices of the overlapping line, not the start and end vertices of the boundary line.

[0462] 62 shows an example in which there are multiple overlapping regions among the submesh boundaries. In this example, for example, the encoding device 100 signals num_submesh, which is information indicating the number of submeshes, to the bitstream, and indicates information for each submesh in the bitstream.

[0463] As shown in Figures 61 and 62, boundary lines and overlapping lines may be distinguished. In other words, boundary edges and overlapping edges may be distinguished. For example, a boundary line may be an edge that belongs to only one triangle in a submesh, and an overlapping line may be one of the edges that is shared with other submeshes. Furthermore, a boundary vertex may be a vertex that constitutes a boundary line, and an overlapping vertex may be one of the vertices that is shared with other submeshes.

[0464] Figure 63 is a diagram showing an example of syntax for signaling the number of sub-meshes in an embodiment.

[0465] For example, the encoding device 100 signals, to the bitstream, num_duplicate_border, which is information indicating the number of overlapping regions between submeshes (overlapping regions as indicated by the overlapping lines above). Furthermore, for example, the encoding device 100 signals, to the bitstream, information indicating the start and end vertices for each of the multiple overlapping regions. Information indicating the traversed direction (the above direction[k]) may be common to the multiple overlapping regions. Furthermore, for example, the encoding device 100 may signal, to the bitstream, an identifier indicating the second vertex to be processed for each overlapping region.

[0466] Furthermore, the encoding device 100 may signal, to the bitstream, information indicating the submesh to which the overlap region overlaps (e.g., duplicate_submesh_id). This signaling may allow, for example, information regarding multiple overlapping edges to be transmitted to the decoding device 200. For example, when one submesh overlaps with another submesh in two overlapping regions, the encoding device 100 signals, to the bitstream, information indicating num_duplicate_border=2 and the same submesh_id for duplicate_submesh_id. Also, for example, when one submesh overlaps with two different submeshes in one overlapping region each, the encoding device 100 signals to the bitstream that num_duplicate_border = 2 and that duplicate_submesh_id indicates the submesh_id of each different overlap destination.

[0467] Note that num_border_flags_minus1 does not need to be transmitted if it can be detected (calculated) in the processing of the decoding device 200.

[0468] Fig. 64 is a diagram showing another example of syntax related to signaling of boundary information according to an embodiment. Specifically, Fig. 64 shows another example of syntax elements necessary for signaling boundary information.

[0469] The starting_vertex_id indicates the identifier of the first vertex (start vertex) from which scanning (processing) starts.

[0470] The number_of_outermost_border_vertices indicates the number (total number) of vertices detected as outermost vertices.

[0471] If the value of direction_flag is 0, it indicates that the vertices are traversed in a clockwise direction. If the value of direction_flag is 1, it indicates that the vertices are traversed in a counterclockwise direction. If direction_flag is not present, its value is assumed to be 0, for example.

[0472] Figure 65 is a diagram showing another example of syntax related to signaling boundary information according to an embodiment. Specifically, Figure 65 shows another example of syntax elements required for signaling boundary information. Each vertex is associated with a border_flag. In the example shown in Figure 65, a flag is signaled for each face in a submesh.

[0473] apply_border_detection_flag indicates, if its value is equal to 1, that border detection is enabled (in other words, border detection is possible) and parameters for border vertex pairs are present in the bitstream. If apply_border_detection_flag is not present, its value is inferred, for example, to be 0.

[0474] If border_flag[i] is equal to 1, it indicates that the vertex associated with the i-th parameter is a border vertex, and if border_flag[i] is equal to 0, it indicates that the vertex is not a border vertex.

[0475] For example, for a submesh boundary, which is a boundary where submeshes overlap (an overlapping region), whether or not vertices touching the submesh boundary overlap is indicated (signaled) in the bitstream, thereby indicating whether or not faces of the submesh boundary overlap. Information indicating whether vertices touching the submesh boundary overlap is not limited to overlapping regions, and may be used as information indicating edges and faces inside the submesh boundary that are connected to the submesh boundary in a specific submesh. Such information may also be used as information indicating edges and faces outside the submesh boundary. Information indicating whether vertices, edges, and faces are inside or outside the submesh boundary may also be indicated in the bitstream. Either or both of information inside the submesh boundary and information outside the submesh boundary may also be indicated in the bitstream.

[0476] Furthermore, the information indicating whether vertices adjacent to a submesh boundary overlap may be used not only for submesh boundaries, but also as information indicating the edges and faces connected to a specific edge for a set of edges connected to each other in a specific mesh.

[0477] Such information may be used, for example, for overlapping boundaries between sub-meshes, or may be signaled in the bitstream, or for non-overlapping boundaries in sub-meshes, or for boundaries of sub-meshes, whether overlapping or not, or for portions of overlapping boundaries.

[0478] As described above, the signaling method and syntax configuration described above may be used to notify information about edges that constitute a submesh boundary, or may be used to notify information about edges that are arranged consecutively in a predetermined order in any three-dimensional mesh. Furthermore, the information notified using the signaling method and syntax configuration described above does not have to be information indicating whether or not there is an overlap with another submesh, or information indicating processing to be performed in the overlapping area. The notified information may be, for example, information indicating the edge, or information indicating the state or attributes of the mesh that includes the edge. The notified information may also be, for example, information indicating processing to be performed on the edge or the mesh that includes the edge.

[0479] Note that "submeshes overlap" means, for example, that the relationship between a first submesh and a second submesh is such that a first mesh included in the first submesh shares (i) vertices, (ii) vertices and edges, or (iii) vertices, edges, and faces with a second mesh included in the second submesh. Also, when viewed in terms of one submesh, "submeshes overlap" means that a mesh included in the one submesh shares (i) vertices, (ii) vertices and edges, or (iii) vertices, edges, and faces with a mesh included in another submesh.

[0480] In addition, if there is a constraint in generating sub-meshes so that faces are not shared, the objects to be determined as to whether or not there is overlap between sub-meshes may be only the vertices and edges out of the vertices, edges, and faces.

[0481] Although the above description uses an example in which the encoding device 100 divides mesh data into multiple sub-meshes and encodes them, at least a portion of the present disclosure may also be applied to data other than mesh data. For example, when dividing point cloud data into multiple sub-point cloud data and encoding them, the encoding device 100 may be applied to sub-point cloud data that includes a boundary with other sub-point cloud data. Specifically, if different processes are performed on different sides of a boundary, resulting in a boundary mismatch, the processes performed in the area around the boundary may be the same, or a correction method (modification method) to be performed when a boundary mismatch occurs may be specified. Furthermore, embodiments of the present disclosure may be applied to a method for signaling, in a bitstream, processes to be performed in an area that includes a boundary and processes to be performed in an area that does not include a boundary, and a method for signaling, in a bitstream, information indicating whether a boundary is included or not. Furthermore, embodiments of the present disclosure may be applied as a syntax for implementing such signaling.

[0482] In other words, the aspects directly or indirectly shown in the embodiments of the present disclosure can be widely used when encoding or decoding multiple sub-data generated by spatially dividing one piece of data.

[0483] <Summary of the Disclosure> Next, an example of an overview of the technology obtained from the disclosure of this specification will be given.

[0484] For example, in the decoding method of the present disclosure, for multiple submeshes that make up an overall image three-dimensional mesh, position information of vertices included in polygons that make up the submeshes and connection information regarding the connection relationships of the vertices are obtained from the encoded bitstream (i.e., decoding information), the polygons are generated using the position information and the connection information (i.e., decoding faces), a determination is made as to whether or not an edge that makes up the polygon is a boundary of the submesh (i.e., edge condition determination), a division process for the edge is determined based on the determination result (i.e., determination of subdivision process based on the determination result), and the edge is divided (subdivision) using the division process.

[0485] Specific examples and variations of the division process are described below.

[0486] In the division process, a method for dividing the edge (division method) may be specified.

[0487] In the division process, the number of times to divide the edge (number of divisions) may be specified.

[0488] In the division process, a method and number of times to divide the edge (division method and number of divisions) may be specified.

[0489] The division process may be a process of generating a new vertex based on position information of a plurality of vertices that form the edge (another definition of the division process).

[0490] The decoding process may decode parameters specifying the segmentation process from the coded bitstream (segmentation process signaling).

[0491] At least one of the division processes may be defined in advance (predetermined division process).

[0492] The splitting process includes not splitting the edge (no split option).

[0493] Specific examples and modifications of the determination process will be described below.

[0494] In the determination process, it may be determined whether or not one of the sides constituting the polygon to be processed is the boundary of the sub-mesh (determination process for each side).

[0495] The determination process may determine whether or not any of the edges constituting the polygon is a boundary edge of the sub-mesh (determining whether or not the polygon includes a boundary edge).

[0496] The submesh boundary may be an edge that includes at both ends a plurality of vertices that constitute a plurality of submeshes (definition of boundary).

[0497] A specific example of the relationship between the determination result and the division process will be described below.

[0498] In the division process, when the side to be processed is the submesh boundary, the side may be divided using a first division process.

[0499] In the division process, when the side to be processed is not the submesh boundary, the side may be divided using a second division process.

[0500] In the division process, when the polygon includes an edge that is the submesh boundary, the edge that is the submesh boundary may be divided using a first division process, and the edge that is not the submesh boundary may be divided using a second division process.

[0501] In the division process, when the polygon does not include an edge that is the boundary of the sub-mesh, all edges included in the polygon may be divided using a second division process.

[0502] In the division process, when the polygon includes an edge that is the submesh boundary, it may be determined whether the first division process and the second division process have a predetermined relationship, and the division process may be determined based on the determination result. For example, the division process may be switched based on a result of comparing the number of divisions specified by the first division process and the second division process.

[0503] Specific examples of the first division process (boundary division process) and the second division process (non-boundary division process) will be described below.

[0504] The first division process and the second division process may be different processes (different division processes may be selected).

[0505] The first division process may be selected from a first division process group, and the second division process may be selected from a second division process group. The first division process group and the second division process group may include different division processes (selected from a plurality of division processes, with different options).

[0506] The first division process may select the same process common to a plurality of sub-meshes, or the division process may select processes from the same group in a plurality of sub-meshes.

[0507] The first division process may be determined on a sequence-by-sequence or frame-by-frame basis. Furthermore, parameters used in the first division process may be coded into the coded bitstream.

[0508] The second division process may be determined for each sub-mesh. Furthermore, parameters used in the second division process may be coded into the coded bitstream.

[0509] Note that "different processing" may mean processing in which at least one of the number of divisions or the division method is different.

[0510] Also, for example, an encoding device of the present disclosure includes a circuit and a memory connected to the circuit, wherein during operation, the circuit encodes a first vertex and a second vertex into a bitstream, the first vertex being connected to the second vertex via an edge, determines whether the edge is a boundary edge, and if the edge is the boundary edge, derives a third vertex from only the first vertex and the second vertex, and the position of the third vertex is interpolated based only on the positions of the first vertex and the second vertex, and if the edge is not the boundary edge, derives a fourth vertex from the first vertex, the second vertex, and at least one fifth vertex different from the first vertex and the second vertex, and the position of the fourth vertex is interpolated based on the positions of the first vertex, the second vertex, and at least one fifth vertex.

[0511] Furthermore, for example, a coding device according to the present disclosure includes an input terminal to which at least one image is input, a block divider that divides a first image included in the at least one image into a plurality of blocks, an intra predictor that predicts a block included in the first image using a reference image included in the first image, an inter predictor that predicts a block included in the first image using a reference block included in a second image different from the first image, a loop filter that filters the block included in the first image, a converter that converts a prediction error between an original signal and a prediction signal generated by the intra predictor or the inter predictor to generate a transform coefficient, a quantizer that quantizes the transform coefficient to generate a quantized coefficient, and an entropy coding circuit that performs variable-length coding of the quantized coefficient to generate a coded bitstream. and an output terminal to which a bitstream including coded quantized coefficients and control information is output, wherein the encoder encodes the first vertex and the second vertex into the bitstream, the first vertex being connected to the second vertex via an edge, determines whether the edge is a boundary edge, and if the edge is the boundary edge, derives a third vertex from only the first vertex and the second vertex, and the position of the third vertex is interpolated based only on the positions of the first vertex and the second vertex; and if the edge is not the boundary edge, derives a fourth vertex from the first vertex, the second vertex, and at least one fifth vertex different from the first vertex and the second vertex, and the position of the fourth vertex is interpolated based on the positions of the first vertex, the second vertex, and at least one fifth vertex.

[0512] Also, for example, a decoding device of the present disclosure includes a circuit and a memory connected to the circuit, wherein during operation, the circuit decodes a first vertex and a second vertex from a bitstream, the first vertex being connected to a second vertex via an edge, determines whether the edge is a boundary edge, and if the edge is a boundary edge, derives a third vertex from only the first vertex and the second vertex, and the position of the third vertex is interpolated based only on the position of the first vertex and the position of the second vertex, and if the edge is not the boundary edge, derives a fourth vertex from the first vertex, the second vertex, and at least one fifth vertex different from the first vertex and the second vertex, and the position of the fourth vertex is interpolated based on the position of the first vertex, the position of the second vertex, and the position of at least one fifth vertex.

[0513] Furthermore, for example, a decoding device of the present disclosure includes an input terminal to which a bitstream is input, a decoder that decodes the bitstream and outputs quantized coefficients, an inverse quantizer that dequantizes the quantized coefficients and outputs transform coefficients, an inverse transformer that inversely transforms the transform coefficients and outputs prediction errors, an intra predictor that predicts blocks included in an image using a reference image included in a first image, an inter predictor that predicts blocks included in the first image using a reference block included in a second image different from the first image, a loop filter that filters blocks included in the first image, and an output terminal that outputs an image including the first image, wherein the decoder A first vertex and a second vertex are decoded from the bitstream, the first vertex being connected to a second vertex via an edge; determining whether the edge is a boundary edge; if the edge is the boundary edge, deriving a third vertex from only the first vertex and the second vertex, and the position of the third vertex is interpolated based only on the position of the first vertex and the position of the second vertex; if the edge is not the boundary edge, deriving a fourth vertex from the first vertex, the second vertex, and at least one fifth vertex different from the first vertex and the second vertex, and the position of the fourth vertex is interpolated based on the position of the first vertex, the position of the second vertex, and the position of at least one fifth vertex.

[0514] Furthermore, for example, in the encoding method of the present disclosure, a first vertex and a second vertex are encoded into a bitstream, the first vertex is connected to the second vertex via an edge, and it is determined whether the edge is a boundary edge. If the edge is the boundary edge, a third vertex is derived from only the first vertex and the second vertex, and the position of the third vertex is interpolated based only on the positions of the first vertex and the second vertex. If the edge is not the boundary edge, a fourth vertex is derived from the first vertex, the second vertex, and at least one fifth vertex different from the first vertex and the second vertex, and the position of the fourth vertex is interpolated based on the positions of the first vertex, the second vertex, and at least one fifth vertex.

[0515] Also, for example, in the decoding method of the present disclosure, a first vertex and a second vertex are decoded from a bitstream, the first vertex is connected to a second vertex via an edge, and it is determined whether the edge is a boundary edge. If the edge is the boundary edge, a third vertex is derived from only the first vertex and the second vertex, and the position of the third vertex is interpolated based only on the position of the first vertex and the position of the second vertex. If the edge is not the boundary edge, a fourth vertex is derived from the first vertex, the second vertex, and at least one fifth vertex different from the first vertex and the second vertex, and the position of the fourth vertex is interpolated based on the position of the first vertex, the position of the second vertex, and the position of at least one fifth vertex.

[0516] Also, for example, whether the edge is the boundary edge is determined based on the number of connections to the edge. Also, for example, whether the edge is the boundary edge is determined using one or more parameters decoded from the bitstream. Also, for example, the one or more parameters are decoded from a header of the bitstream. Also, for example, the first vertex, the second vertex, the third vertex, the fourth vertex, and the fifth vertex are vertices of a three-dimensional mesh frame. Also, for example, the first vertex, the second vertex, the third vertex, the fourth vertex, and the fifth vertex are vertices of a three-dimensional sub-mesh.

[0517] In the field of multimedia data coding technology, it is desirable to propose new methods for improving coding efficiency, improving image quality, and reducing circuit scale.

[0518] Each of the embodiments, some of the components, and each of the methods in the present disclosure enables at least one of, for example, improved coding efficiency, improved image quality, reduced encoding / decoding processing volume, reduced circuit size, and improved encoding / decoding processing speed. Alternatively, each of the embodiments, some of the components, and each of the methods in the present disclosure enables appropriate selection of any of elements or operations, such as filters, block sizes, motion vectors, reference pictures, and reference blocks, in encoding and decoding. The present disclosure includes disclosure of configurations and methods that can provide advantages other than those described above. Examples of such configurations and methods include configurations and methods that improve coding efficiency while suppressing an increase in processing volume.

[0519] Additional value and advantages of aspects of the present disclosure will become apparent from the specification and drawings, which may be obtained individually through various embodiments and features of the specification and drawings, not all of which need be provided to obtain one or more of such value and / or advantages.

[0520] These general or specific aspects may be implemented using a system, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or any combination of a system, a method, an integrated circuit, a computer program, or a recording medium.

[0521] <Representative Example> Figure 66 is a flow diagram showing an example of basic encoding processing according to this embodiment. For example, the encoding device 100 shown in Figure 24 includes a circuit 151 and a memory 152 connected to the circuit 151. In the encoding device 100, the circuit 151 performs the following processing during operation.

[0522] First, the encoding device 100 determines whether a first vertex that is not located on an edge that forms the outer periphery of a first submesh divided from a three-dimensional mesh overlaps with a vertex that is located in a second submesh divided from the three-dimensional mesh (S401).

[0523] Specifically, the encoding device 100 determines whether a first vertex of a first submesh is the same as one of multiple vertices of a second submesh (e.g., vertices with the same information, such as position information and attribute information). An overlapping state refers to a state in which multiple submeshes share the same vertex. For example, the encoding device 100 determines whether a given vertex (first vertex) in a three-dimensional mesh is included in both the first submesh and the second submesh. That is, the encoding device 100 determines whether the first submesh and the second submesh share the same first vertex in the three-dimensional mesh. In other words, when the first submesh and the second submesh are generated by dividing a three-dimensional mesh, the encoding device 100 determines whether both the first submesh and the second submesh included the first vertex in the three-dimensional mesh.

[0524] Furthermore, the edges that form the periphery of the first submesh are, for example, the submesh boundaries. In other words, the edges that form the periphery of the first submesh are edges that connect the outermost vertices. The first vertex is, for example, the inner vertex. In other words, the first vertex is any vertex other than the outermost vertex among the multiple vertices that form the first submesh. The first vertex is, for example, the overlapping vertex located inside the submesh boundary, or the boundary vertex located inside the submesh boundary. The three-dimensional mesh is, for example, the original mesh. The first submesh and the second submesh are, for example, submeshes divided from the original mesh.

[0525] Next, based on the result of the determination (S401), the encoding device 100 encodes first information indicating whether the first vertex overlaps with a vertex of the second submesh into a bitstream (S402). That is, the encoding device 100 generates a bitstream including the first information. The encoding device 100 transmits the generated bitstream to the decoding device 200.

[0526] The first information is, for example, the overlap information or the information included in the boundary information. Specifically, the first information is, for example, the border_flag[i].

[0527] According to this, the decoding device 200 that has received the bitstream can determine whether or not there is an overlap by simply checking the first information, without performing a process of decoding the first submesh and the second submesh to determine whether or not there is an overlap, for example, thereby reducing the amount of processing.

[0528] Furthermore, for example, the encoding device 100 further encodes into the bitstream second information indicating a second vertex at which a predetermined processing is performed first, among two or more vertices among the plurality of vertices, where the predetermined processing is performed in sequence.

[0529] The second information is, for example, the above-mentioned starting_vertex_id. The two or more vertices are, for example, all boundary vertices. The predetermined process is, for example, the above-mentioned first mesh decoding process.

[0530] According to this, the decoding device 200 that has received the bitstream can first perform a predetermined process on the second vertex among the two or more vertices based on the second information, and therefore can perform the predetermined process on the two or more vertices in an appropriate order.

[0531] The two or more vertices may be, for example, vertices other than boundary vertices, and the predetermined process may be, for example, the second mesh decoding process described above.

[0532] Furthermore, for example, the encoding device 100 further encodes into the bitstream third information indicating a third vertex, which is two or more of the multiple vertices at which a specified processing is performed last, among the two or more vertices at which a specified processing is performed in sequence.

[0533] The third information is, for example, ending_vertex_id.

[0534] According to this, the decoding device 200 that has received the bitstream can perform the predetermined process on the third vertex of the two or more vertices last based on the third information, and therefore can perform the predetermined process on the two or more vertices in an appropriate order.

[0535] Furthermore, for example, the encoding device 100 further encodes, into the bitstream, fourth information indicating the order of the multiple vertices when a predetermined process is performed on each of the multiple vertices.

[0536] According to this, the decoding device 200 that has received the bitstream can perform predetermined processing on two or more vertices in order based on the fourth information. Therefore, the decoding device 200 can perform predetermined processing on two or more vertices in an appropriate order.

[0537] Fig. 67 is a flow diagram showing an example of basic decoding processing according to this embodiment. For example, the decoding device 200 shown in Fig. 25 includes a circuit 251 and a memory 252 connected to the circuit 251. In the decoding device 200, the circuit 251 performs the following processing during operation.

[0538] First, the decoding device 200 decodes from the bitstream first information indicating whether a first vertex, among the multiple vertices of a first submesh divided from a three-dimensional mesh, that is not located on an edge that forms the outer periphery of the first submesh overlaps with a vertex of a second submesh divided from the three-dimensional mesh (S411).

[0539] Next, the decoding device 200 determines whether the first vertex overlaps with a vertex of the second submesh based on the first information (S412).

[0540] According to this, the decoding device 200 can determine whether or not there is an overlap by simply checking the first information, without performing a process of decoding the first submesh and the second submesh to determine whether or not there is an overlap, for example, thereby reducing the amount of processing.

[0541] Furthermore, for example, the decoding device 200 further decodes from the bitstream second information indicating a second vertex at which a predetermined processing is performed first, among two or more vertices among the plurality of vertices, where the predetermined processing is performed in sequence.

[0542] This allows the decoding device 200 to first perform a predetermined process on the second vertex among the two or more vertices based on the second information, thereby enabling the decoding device 200 to execute the predetermined process on the two or more vertices in an appropriate order.

[0543] Furthermore, for example, the decoding device 200 further decodes from the bitstream third information indicating a third vertex, which is two or more of the multiple vertices at which a specified processing is performed last, among the two or more vertices at which a specified processing is performed in sequence.

[0544] This allows the decoding device 200 to perform the predetermined process on the third vertex of the two or more vertices last based on the third information, and therefore the decoding device 200 can execute the predetermined process on the two or more vertices in an appropriate order.

[0545] Furthermore, for example, the decoding device 200 further decodes, from the bitstream, fourth information indicating the order of the multiple vertices when a predetermined process is performed on each of the multiple vertices.

[0546] This allows the decoding device 200 to perform predetermined processing on two or more vertices in order based on the fourth information. Therefore, the decoding device 200 can execute predetermined processing on two or more vertices in an appropriate order.

[0547] <Other Examples> Although aspects of the encoding device 100 and the decoding device 200 have been described above according to the embodiments, the aspects of the encoding device 100 and the decoding device 200 are not limited to the embodiments. Modifications conceivable by those skilled in the art may be applied to the embodiments, and multiple components in the embodiments may be combined in any manner.

[0548] For example, a process performed by a specific component in the embodiment may be performed by another component instead of the specific component. Also, the order of multiple processes may be changed, or multiple processes may be performed in parallel.

[0549] Furthermore, as described above, at least some of the configurations of the present disclosure may be implemented as an integrated circuit. At least some of the processes of the present disclosure may be used as an encoding method or a decoding method. A program for causing a computer to execute the encoding method or the decoding method may be used. A non-transitory computer-readable recording medium on which the program is recorded may be used. A bitstream for causing the decoding device 200 to perform a decoding process may be used.

[0550] Furthermore, at least some of the configurations and processes of the present disclosure may be used as a transmitting device, a receiving device, a transmitting method, or a receiving method. A program for causing a computer to execute the transmitting method or the receiving method may be used. Furthermore, a non-transitory computer-readable recording medium on which the program is recorded may be used.

[0551] The present disclosure is useful, for example, in encoding devices, decoding devices, transmitting devices, receiving devices, etc. related to three-dimensional meshes, and is applicable to computer graphics systems, three-dimensional data display systems, etc.

[0552] 100 Encoding device 101, 121, 144 Vertex information encoder 102, 145 Connection information encoder 103, 122 Attribute information encoder 104, 204, 1103 Preprocessor 105, 205, 2106 Postprocessor 110 Three-dimensional data encoding system 111, 211 Controller 112, 212 Input / output processor 113 Three-dimensional data encoder 114 System multiplexer 115 Three-dimensional data generator 123 Metadata encoder 124 Multiplexer 131 Vertex image generator 132 Attribute image generator 133 Metadata generator 134 Video encoder 141 Two-dimensional data encoder 142 Mesh data encoder 143 Texture encoder 148 Description encoder 151, 251 Circuit 152, 252 Memory 200 Decoding device 201, 221, 244 Vertex information decoder 202, 245 Connection information decoder 203, 222 Attribute information decoder 210 3D data decoding system 213 3D data decoder 214 System demultiplexer 215, 247 Presentation device 216 User interface 223 Metadata decoder 224 Demultiplexer 231 Vertex information generator 232 Attribute information generator 234 Video decoder 241 2D data decoder 242 Mesh data decoder 243 Texture decoder 246 Mesh reconstructor 248 Description decoder 300 Network 310 External connector 1101 Input mesh 1102, 2108 Attribute map 1104, 2103 Base mesh 1105, 2104 Displacement data 1106 Compressor 1107, 1304, 2101 Bitstream 1108, 2105 Metadata 1201 Decimator 1202, 1209 Quantizer 1203 Base mesh encoder 1204, 2202 Base mesh decoder 1205, 2203, 2207 Inverse quantizer 1206, 1402, 2204 Subdivider 1207 Displacement vector calculator 1208 Wavelet transformer1210 Image packer 1211, 1213 Video encoder 1212, 2211 Color converter 1214 Multiplexer 1221, 1222, 2221, 2222 Switcher 1223 Arithmetic encoder 1301, 2304 Mesh frame 1302, 2301, 2302 Base mesh frame 1303, 2303 Displacement information 1401 Base mesh generator 1403 Displacement data generator 2102 Decompressor 2107 Output mesh 2109 Combiner 2201 Demultiplexer 2205, 2210 Video decoder 2206 Image unpacker 2208 Inverse wavelet transformer 2209 Reconstructor 2223 Arithmetic decoder 2305 Decoder 2306 Post-decoder 2307 Pre-reconstructor 2308 Reconstructor 2309 Post-reconstructor 2310 Fitter 2311 Final 3D mesh frame

Claims

1. An encoding device comprising: a circuit; and a memory connected to the circuit, wherein during operation, the circuit determines whether a first vertex, among a plurality of vertices of a first submesh divided from a three-dimensional mesh, which is not located on an edge constituting the periphery of the first submesh, overlaps with a vertex of a second submesh divided from the three-dimensional mesh; and based on the result of the determination, encodes first information indicating whether the first vertex overlaps with a vertex of the second submesh into a bit stream.

2. The encoding device according to claim 1, further comprising: encoding second information into the bit stream, the second information indicating a second vertex on which a predetermined process is performed first among two or more vertices among the plurality of vertices, the second vertex being one of the two or more vertices on which a predetermined process is performed in sequence.

3. The encoding device according to claim 1, further comprising: encoding third information into the bit stream, the third information indicating a third vertex which is one of two or more vertices among the plurality of vertices for which a predetermined process is performed last among the two or more vertices for which the predetermined process is performed in sequence.

4. The encoding device according to any one of claims 1 to 3, further comprising: encoding into the bit stream fourth information indicating an order of the plurality of vertices when a predetermined processing is performed on each of the plurality of vertices.

5. A decoding device comprising: a circuit; and a memory connected to the circuit, wherein during operation, the circuit decodes from a bit stream first information indicating whether a first vertex, among a plurality of vertices of a first submesh divided from a three-dimensional mesh, that is not located on an edge that constitutes the periphery of the first submesh, overlaps with a vertex of a second submesh divided from the three-dimensional mesh; and determines, based on the first information, whether the first vertex overlaps with a vertex of the second submesh.

6. The decoding device according to claim 5, further comprising: decoding second information from the bit stream, the second information indicating a second vertex on which a predetermined process is performed first among two or more vertices among the plurality of vertices, the second vertex being one on which a predetermined process is performed in sequence.

7. The decoding device according to claim 5, further comprising: decoding third information from the bit stream, the third information indicating a third vertex which is one of two or more vertices among the plurality of vertices and which is the last vertex to be subjected to a predetermined processing among the two or more vertices for which a predetermined processing is performed in sequence.

8. A decoding device according to any one of claims 5 to 7, further comprising: decoding fourth information from the bit stream, the fourth information indicating an order of the plurality of vertices when a predetermined process is performed on each of the plurality of vertices.

9. An encoding method comprising: determining whether or not a first vertex, among a plurality of vertices of a first submesh divided from a three-dimensional mesh, that is not located on an edge that constitutes the outer periphery of the first submesh, overlaps with a vertex of a second submesh divided from the three-dimensional mesh; and encoding, based on the result of the determination, first information indicating whether or not the first vertex overlaps with a vertex of the second submesh into a bit stream.

10. A decoding method comprising: decoding first information from a bit stream indicating whether a first vertex, among multiple vertices of a first submesh divided from a three-dimensional mesh, that is not located on an edge that constitutes the outer periphery of the first submesh, overlaps with a vertex of a second submesh divided from the three-dimensional mesh; and determining whether the first vertex overlaps with a vertex of the second submesh based on the first information.

Images