Encoding method, decoding method, encoding device, and decoding device

Abstract

Provided is an encoding method executed by an encoding device, wherein: a plurality of vertices included in a three-dimensional mesh are acquired (S1621); vertex information relating to one vertex among the plurality of vertices is predicted and encoded (S1622); a bitstream including total number information indicating the total number of prediction residuals obtained through the prediction encoding is generated (S1623); and when the total number indicated by the total number information is greater than zero, a prediction residual having a code amount greater than zero among the prediction residuals obtained through the prediction encoding is included in the bitstream.

Description

Encoding method, decoding method, encoding device, and decoding device

[0001] The present disclosure relates to encoding methods and the like.

[0002] In US Pat. No. 6,299,549 a method and apparatus for encoding and decoding three-dimensional mesh data is proposed.

[0003] Japanese Patent Application Laid-Open No. 2006-187015

[0004] Further improvements are desired for the encoding or decoding process for three-dimensional meshes.The present disclosure aims to improve the encoding or decoding process for three-dimensional meshes.

[0005] An encoding method according to one aspect of the present disclosure is an encoding method executed by an encoding device, which obtains a plurality of vertices included in a three-dimensional mesh, predictively encodes vertex information relating to one of the plurality of vertices, generates a bitstream including total number information indicating the total number of prediction residuals obtained by the predictive encoding, and if the total number indicated by the total number information is greater than 0, includes in the bitstream the prediction residuals obtained by the predictive encoding that have a code amount greater than 0.

[0006] Another aspect of the present disclosure is an encoding method executed by an encoding device, which obtains multiple vertices included in a three-dimensional mesh, predictively encodes information corresponding to one of the multiple vertices, generates a bitstream including total number information indicating the total number of prediction residuals obtained by the predictive encoding, and code amount information indicating the code amount of the prediction residuals, and if the total number indicated by the total number information is greater than 0 and the code amount indicated by the code amount information is 0, does not encode the prediction residual.

[0007] A decoding method according to one aspect of the present disclosure is a decoding method executed by a decoding device, which obtains a bitstream including total number information indicating the total number of prediction residuals obtained by predictive coding, obtains a plurality of vertices included in a three-dimensional mesh, predictively decodes vertex information related to one of the plurality of vertices, and, if the total number indicated by the total number information is greater than 0, decodes the prediction residuals obtained by the predictive coding that have a code amount greater than 0.

[0008] A decoding method according to another aspect of the present disclosure is a decoding method executed by a decoding device, which acquires a bitstream including total number information indicating the total number of prediction residuals obtained by predictive coding and code amount information indicating the code amount of the prediction residuals, acquires multiple vertices included in a three-dimensional mesh, predictively decodes information corresponding to one of the multiple vertices, and, if the total number indicated by the total number information is greater than 0 and the code amount indicated by the code amount information is 0, decodes the prediction residual to a specific value.

[0009] These comprehensive or specific aspects may be realized as a system, an apparatus, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or may be realized as any combination of a system, an apparatus, an integrated circuit, a computer program, and a recording medium.

[0010] The present disclosure may contribute to improvements in encoding processes and the like related to three-dimensional meshes.

[0011] FIG. 1 is a conceptual diagram showing a three-dimensional mesh according to the first embodiment. FIG. 2 is a conceptual diagram showing basic elements of a three-dimensional mesh according to the first embodiment. FIG. 3 is a conceptual diagram showing mapping according to the first embodiment. FIG. 4 is a block diagram showing a configuration example of an encoding / decoding system according to the first embodiment. FIG. 5 is a block diagram showing a configuration example of an encoding device according to the first embodiment. FIG. 6 is a block diagram showing another configuration example of an encoding device according to the first embodiment. FIG. 7 is a block diagram showing another configuration example of a decoding device according to the first embodiment. FIG. 8 is a block diagram showing another configuration example of a decoding device according to the first embodiment. FIG. 9 is a conceptual diagram showing another configuration example of a bit stream according to the first embodiment. FIG. 10 is a conceptual diagram showing another configuration example of a bit stream according to the first embodiment. FIG. 11 is a block diagram showing a specific example of an encoding / decoding system according to the first embodiment. FIG. 12 is a conceptual diagram showing an example of a configuration of point cloud data according to the first embodiment. FIG. 13 is a conceptual diagram showing an example of a data file of point cloud data according to the first embodiment. FIG. 14 is a conceptual diagram showing an example of a configuration of mesh data according to the first embodiment. FIG. 15 is a conceptual diagram showing an example of a data file of mesh data according to the first embodiment. FIG. 16 is a conceptual diagram showing types of three-dimensional data according to the first embodiment. FIG. 17 is a block diagram showing an example of a configuration of a three-dimensional data encoder according to the first embodiment. FIG. 18 is a block diagram showing an example of a configuration of a three-dimensional data decoder according to the first embodiment. FIG. 19 is a block diagram showing another configuration example of a three-dimensional data encoder according to the first embodiment. FIG. 1 is a block diagram showing another example configuration of a three-dimensional data decoder according to embodiment 1. FIG. 2 is a conceptual diagram showing a specific example of encoding processing according to embodiment 1. FIG. 3 is a conceptual diagram showing a specific example of decoding processing according to embodiment 1. FIG. 4 is a block diagram showing an implementation example of an encoding device according to embodiment 1. FIG. 5 is a block diagram showing an implementation example of a decoding device according to embodiment 1. FIG. 6 is a block diagram showing another example configuration of an encoding / decoding system according to embodiment 1. FIG. 7 is a block diagram showing another example configuration of an encoding device according to embodiment 1. FIG. 8 is a block diagram showing another example configuration of a decoding device according to embodiment 1. FIG. 9 is a block diagram showing another example configuration of an encoding device according to embodiment 1. FIG. 10 is a block diagram showing another example configuration of a decoding device according to embodiment 1. FIG. 11 is a flow diagram showing processing of an encoding device according to embodiment 1.1 is an explanatory diagram conceptually illustrating encoding of a mesh frame according to a first embodiment; a flow diagram illustrating processing by a decoding device according to the first embodiment; an explanatory diagram conceptually illustrating decoding of a mesh frame according to the first embodiment; a block diagram illustrating an example of a configuration of a decoding device according to the first embodiment; a block diagram illustrating an example of a configuration of a decoding device according to the first embodiment; an explanatory diagram illustrating an example of subdivision according to the first embodiment; an explanatory diagram illustrating an example of displacement of vertices after displacement after subdivision according to the first embodiment; an explanatory diagram illustrating an example of vertices of an original mesh according to the first embodiment; an explanatory diagram illustrating an example of a mesh according to the first embodiment; an explanatory diagram illustrating an example of division of a mesh into sub-meshes according to the first embodiment; a first explanatory diagram illustrating an example of packing of displacement information into an image frame according to the first embodiment; a second explanatory diagram illustrating an example of packing of displacement information into an image frame according to the first embodiment; a third explanatory diagram illustrating an example of packing of displacement information into an image frame according to the first embodiment; a diagram for explaining an example of attribute information in mesh data according to the first embodiment; a diagram for explaining an example of attribute information in mesh data according to the first embodiment; a diagram for explaining an example of a method for encoding attribute information; a diagram for explaining an example of a method for encoding attribute information. 1 is a diagram for explaining a specific example of a method for encoding attribute information. 2 is a diagram for explaining a specific example of a method for encoding attribute information. 3 is a diagram for explaining a specific example of a method for encoding attribute information. 4 is a diagram for explaining a specific example of a method for encoding attribute information. 5 is a diagram for explaining a specific example of a method for encoding attribute information. 6 is a diagram showing an example of a flowchart for encoding attribute information of three-dimensional points according to embodiment 1. 7 is a flowchart showing an example of a calculation process for a predicted value of a third prediction mode according to embodiment 1. 8 is a diagram showing an example of a flowchart for decoding attribute information of three-dimensional points according to embodiment 1. 9 is a diagram showing an example of the syntax of mesh_attribute_coding according to embodiment 1. 10 is a diagram showing an example of the syntax of mesh_attribute_header according to embodiment 1. 11 is a diagram showing an example of the syntax of mesh_attribute_data according to embodiment 1.1 is a diagram showing an example of attribute information according to a modification of the first embodiment. FIG. 2 is a diagram showing an example of the syntax of mesh_attribute_data according to the modification of the first embodiment. FIG. 3 is a diagram showing an example of a geometry map according to the first embodiment. FIG. 4 is a diagram showing an example of a texture map according to the first embodiment. FIG. 5 is a diagram showing an example of the syntax of mesh_position_coding according to the first embodiment. FIG. 6 is a diagram showing an example of the syntax of mesh_position_header according to the first embodiment. FIG. 7 is a diagram showing an example of the syntax of mesh_position_data according to the first embodiment. FIG. 8 is a diagram showing an example of the configuration of an encoding device according to the first embodiment. FIG. 9 is a flowchart showing an example of an encoding method by the encoding device according to the first embodiment. FIG. 10 is a diagram showing an example of the configuration of a decoding device according to the first embodiment. FIG. 11 is a flowchart showing an example of a decoding method by the decoding device according to the first embodiment. FIG. 12 is a diagram showing an example of the syntax of mesh_attribute_data according to the second embodiment. FIG. 13 is a flowchart showing an example of an encoding process based on coarse prediction by the encoding device according to the second embodiment. FIG. 14 is a flowchart showing an example of a decoding process based on coarse prediction by the decoding device according to the second embodiment. FIG. 15 is a diagram showing an example of the syntax of mesh_attribute_data according to the modification of the second embodiment. Fig. 1 is a diagram showing another example of the syntax of mesh_attribute_data according to a modified example of embodiment 2. Fig. 2 is a diagram showing an example of the configuration of an encoding device in embodiment 2. Fig. 3 is a flowchart showing an example of an encoding method by an encoding device in embodiment 2. Fig. 4 is a diagram showing an example of the configuration of a decoding device in embodiment 2. Fig. 5 is a flowchart showing an example of a decoding method by a decoding device in embodiment 2.

[0012] <Summary of the Disclosure> Three-dimensional (3D) meshes are used in computer graphics images, for example, which may be composed of multiple temporally distinct frames, each of which may be represented by a 3D mesh.

[0013] A 3D mesh is composed of vertex information indicating the positions of each of the vertices in 3D space, connectivity information indicating the connections between the vertices, and attribute information indicating the attributes of each vertex or face. Each face is constructed according to the connectivity between the vertices. Various computer graphics images can be expressed using such 3D meshes.

[0014] Furthermore, for transmission and storage of the 3D mesh, efficient encoding and decoding of the 3D mesh is expected. For efficient encoding and decoding of the 3D mesh, arithmetic coding and decoding may be used.

[0015] Further improvements are desired in the encoding or decoding process for three-dimensional data. The present disclosure aims to improve the encoding or decoding process for three-dimensional data.

[0016] Below, examples of inventions that can be obtained from the disclosure of this specification will be given, and the effects and the like that can be obtained from these inventions will be explained.

[0017] An encoding method according to a first aspect of the present disclosure is an encoding method executed by an encoding device, which obtains a plurality of vertices included in a three-dimensional mesh, predictively encodes vertex information relating to one of the plurality of vertices, generates a bitstream including total number information indicating the total number of prediction residuals obtained by the predictive encoding, and, if the total number indicated by the total number information is greater than 0, includes in the bitstream the prediction residuals obtained by the predictive encoding that have a code amount greater than 0.

[0018] According to this, by generating a bitstream that further includes prediction residuals when the total number is greater than 0 based on total number information indicating the total number of prediction residuals obtained by predictive coding, it is possible to suppress the coding of unnecessary information. This makes it possible to generate bitstreams efficiently, thereby improving the compression efficiency of three-dimensional meshes.

[0019] An encoding method according to a second aspect of the present disclosure is an encoding method according to the first aspect, wherein the bitstream further includes code amount information indicating the code amount of the prediction residual, and when the total number indicated by the total number information is greater than 0, the code amount indicated by the code amount information is greater than 0.

[0020] This allows explicit management of the amount of information to be coded by including prediction residual coding amount information in the bitstream, which reduces redundancy in the coding process while maintaining consistency with the decoding side, thereby improving overall bit efficiency.

[0021] An encoding method according to a third aspect of the present disclosure is the encoding method according to the first or second aspect, wherein the predictive encoding is performed based on one vertex that has already been processed.

[0022] This simplifies the calculation process by making predictions based on one vertex that has already been processed, which reduces the calculation load of the prediction process while ensuring a certain degree of prediction accuracy, thereby achieving effective coding even in a low-computational-load environment.

[0023] An encoding method according to a fourth aspect of the present disclosure is an encoding method according to any one of the first to third aspects, in which the predictive encoding is performed based on two or more vertices that have already been processed.

[0024] This method uses multiple processed vertices for prediction, resulting in higher prediction accuracy. As a result, the prediction residual is smaller, which reduces the amount of bits required for encoding and enables high compression of 3D meshes.

[0025] An encoding method according to a fifth aspect of the present disclosure is the encoding method according to any one of the first to fourth aspects, wherein the vertex information includes attribute information of the one vertex.

[0026] This method effectively compresses attribute information such as color information and normal vector information by using predictive coding for the attribute information of each vertex, thereby reducing the data size while maintaining the visual quality of the 3D mesh.

[0027] An encoding method according to a sixth aspect of the present disclosure is the encoding method according to any one of the first to fifth aspects, wherein the vertex information includes position information of the one vertex.

[0028] This makes it possible to compress the position information of each vertex by using predictive coding, which has the effect of improving the efficiency of storage capacity and communication bandwidth while preserving the structural information of the 3D mesh.

[0029] An encoding method according to a seventh aspect of the present disclosure is an encoding method executed by an encoding device, which obtains a plurality of vertices included in a three-dimensional mesh, predictively encodes information corresponding to one of the plurality of vertices, generates a bitstream including total number information indicating the total number of prediction residuals obtained by the predictive encoding, and code amount information indicating the code amount of the prediction residuals, and if the total number indicated by the total number information is greater than 0 and the code amount indicated by the code amount information is 0, does not encode the prediction residual.

[0030] According to this, when the code amount of a prediction residual is 0, the prediction residual is not coded, so that the coded data in the bitstream can be reduced, resulting in an effect of improving coding efficiency.

[0031] A decoding method according to an eighth aspect of the present disclosure is a decoding method executed by a decoding device, which obtains a bitstream including total number information indicating the total number of prediction residuals obtained by predictive coding, obtains a plurality of vertices included in a three-dimensional mesh, predictively decodes vertex information related to one of the plurality of vertices, and, if the total number indicated by the total number information is greater than 0, decodes the prediction residuals obtained by the predictive coding that have a code amount greater than 0.

[0032] According to this, decoding of unnecessary information can be suppressed by performing decoding only when the total number is greater than 0, based on total number information indicating the total number of prediction residuals obtained by predictive decoding. This enables efficient bitstream processing, and has the effect of improving the efficiency of the restoration process of 3D mesh data.

[0033] A decoding method according to a ninth aspect of the present disclosure is a decoding method according to the eighth aspect, wherein the bitstream further includes code amount information indicating the code amount of the prediction residual, and when the total number indicated by the total number information is greater than 0, the code amount indicated by the code amount information is greater than 0.

[0034] By including the coding amount information of the prediction residual in the bitstream, it is possible to determine whether or not the decoding process can be performed under the same conditions as the encoding process. As a result, it is possible to avoid unnecessary data processing while ensuring the consistency and integrity of the decoding process.

[0035] A decoding method according to a tenth aspect of the present disclosure is the decoding method according to the eighth or ninth aspect, wherein the predictive decoding performs predictive decoding based on one vertex that has already been processed.

[0036] This method reduces the computational load of the decoding process by performing predictive decoding based on one processed vertex. This simplifies the process while maintaining a certain level of restoration accuracy, which has the effect of making the method applicable to lightweight decoding environments.

[0037] A decoding method according to an eleventh aspect of the present disclosure is a decoding method according to any one of the eighth to tenth aspects, in which the predictive decoding is performed based on two or more vertices that have already been processed.

[0038] This method improves prediction accuracy during decoding by using multiple processed vertices for prediction, thereby reducing errors in the reconstructed 3D mesh. This enables mesh reconstruction with high data quality, resulting in highly accurate decoding.

[0039] A decoding method according to a twelfth aspect of the present disclosure is the decoding method according to any one of the eighth to eleventh aspects, wherein the vertex information includes attribute information of the one vertex.

[0040] This method improves the reproducibility of attribute information such as color information and normal information by performing predictive decoding on vertex attribute information, thereby enabling highly accurate reproduction from compressed data without impairing the visual quality of the 3D mesh.

[0041] A decoding method according to a thirteenth aspect of the present disclosure is the decoding method according to any one of the eighth to twelfth aspects, wherein the vertex information includes position information of the one vertex.

[0042] This method allows accurate reproduction of information by predictive decoding of vertex position information, which makes it possible to restore the spatial structure of the 3D mesh while maintaining its accuracy, thereby improving the accuracy of model reconstruction.

[0043] A decoding method according to a fourteenth aspect of the present disclosure is a decoding method executed by a decoding device, which acquires a bitstream including total number information indicating the total number of prediction residuals obtained by predictive encoding and code amount information indicating the code amount of the prediction residuals, acquires a plurality of vertices included in a three-dimensional mesh, predictively decodes information corresponding to one of the plurality of vertices, and, if the total number indicated by the total number information is greater than 0 and the code amount indicated by the code amount information is 0, decodes the prediction residual to a specific value.

[0044] This method decodes the prediction residuals to specific values ​​based on the total number information and the code amount information, allowing prediction residuals omitted on the encoding side to be processed in a consistent manner, thereby enabling accurate data restoration while avoiding unnecessary calculations and inconsistencies.

[0045] A coding device according to a fifteenth aspect of the present disclosure comprises a circuit and a memory connected to the circuit, wherein the circuit, in operation, acquires a plurality of vertices included in a three-dimensional mesh, predictively encodes vertex information relating to one of the plurality of vertices, generates a bitstream including total number information indicating the total number of prediction residuals obtained by the predictive encoding, and, if the total number indicated by the total number information is greater than 0, includes in the bitstream, among the prediction residuals obtained by the predictive encoding, the prediction residuals having a code amount greater than 0.

[0046] According to this, by encoding prediction residuals based on total number information indicating the total number of prediction residuals obtained by predictive coding when the total number is greater than 0, it is possible to suppress the encoding of unnecessary information. This makes it possible to generate bitstreams efficiently and improve the compression efficiency of three-dimensional meshes.

[0047] A decoding device according to a sixteenth aspect of the present disclosure includes a circuit and a memory connected to the circuit, and in operation, the circuit acquires a plurality of vertices included in a three-dimensional mesh, acquires a bitstream including total number information indicating the total number of prediction residuals obtained by predictive coding, predictively decodes vertex information related to one of the plurality of vertices, and if the total number indicated by the total number information is greater than 0, decodes the prediction residuals obtained by the predictive coding that have a code amount greater than 0.

[0048] According to this, decoding of unnecessary information can be suppressed by performing decoding only when the total number is greater than 0, based on total number information indicating the total number of prediction residuals obtained by predictive decoding. This enables efficient bitstream processing, and has the effect of improving the efficiency of the restoration process of 3D mesh data.

[0049] These comprehensive or specific aspects may be realized as a system, an apparatus, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM, or may be realized as any combination of a system, an apparatus, an integrated circuit, a computer program, or a recording medium.

[0050] Hereinafter, the embodiments will be specifically described with reference to the drawings.

[0051] The embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, component placement and connection configurations, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present invention. Furthermore, among the components in the following embodiments, components that are not described in the independent claims that represent the highest concepts are described as optional components.

[0052] (First Embodiment) In this embodiment, an encoding method, a decoding method, etc. will be described.

[0053] <Expressions and Terms> The following expressions and terms are used herein.

[0054] (1) Three-dimensional Mesh A three-dimensional mesh is a collection of multiple faces, and represents, for example, a three-dimensional object. A three-dimensional mesh is mainly composed of vertex information, connectivity information, and attribute information. A three-dimensional mesh may be expressed as a polygon mesh or a mesh. A three-dimensional mesh may also vary over time. A three-dimensional mesh may include metadata related to the vertex information, connectivity information, and attribute information, and may also include other additional information.

[0055] (2) Vertex Information Vertex information is information indicating a vertex. For example, the vertex information indicates the position of a vertex in a three-dimensional space. Furthermore, a vertex corresponds to a vertex of a face that constitutes a three-dimensional mesh. Vertex information may be expressed as "geometry." Furthermore, vertex information may be expressed as position information.

[0056] (3) Connection Information Connection information is information that indicates connections between vertices. For example, connection information indicates connections for forming faces or edges of a three-dimensional mesh. Connection information may be expressed as "Connectivity." Connection information may also be expressed as face information.

[0057] (4) Attribute Information Attribute information is information that indicates attributes of a vertex or a face. For example, attribute information indicates attributes such as a color, an image, and a normal vector associated with a vertex or a face. Attribute information may be expressed as "texture."

[0058] (5) Faces A face is an element that makes up a three-dimensional mesh. Specifically, a face is a polygon on a plane in three-dimensional space. For example, a face can be defined as a triangle in three-dimensional space.

[0059] (6) Plane A plane is a two-dimensional plane in a three-dimensional space. For example, a polygon is formed on a plane, and multiple polygons are formed on multiple planes.

[0060] (7) Bitstream: A bitstream corresponds to coded information. A bitstream may also be referred to as a stream, a coded bitstream, a compressed bitstream, or a coded signal.

[0061] (8) Encoding and Decoding The term encoding may be substituted with terms such as storing, including, writing, describing, signaling, sending, notifying, saving, or compressing, and these terms may be interchangeable. For example, encoding information may mean including the information in a bitstream. Also, encoding information into a bitstream may mean encoding the information to generate a bitstream that includes the encoded information.

[0062] Additionally, the term "decode" may be replaced with terms such as "read," "decode," "read," "load," "derive," "obtain," "receive," "extract," "reconstruct," "reconstruct," "decompress," or "decompress," and these terms may be interchangeable. For example, decoding information may mean obtaining information from a bitstream. Decoding information from a bitstream may mean decoding the bitstream to obtain information contained in the bitstream.

[0063] (9) Ordinal Numbers In the description, ordinal numbers such as first and second may be assigned to components, etc. These ordinal numbers may be changed as appropriate. Furthermore, new ordinal numbers may be assigned to components, etc., or removed. Furthermore, these ordinal numbers may be assigned to elements in order to identify them, and may not correspond to a meaningful order.

[0064] <Three-dimensional mesh> Fig. 1 is a conceptual diagram showing a three-dimensional mesh according to this embodiment. A three-dimensional mesh is composed of multiple faces. For example, each face is a triangle. The vertices of these triangles are defined in three-dimensional space. The three-dimensional mesh then represents a three-dimensional object. Each face may have a color or an image.

[0065] FIG. 2 is a conceptual diagram showing the basic elements of a three-dimensional mesh according to this embodiment. A three-dimensional mesh is composed of vertex information, connection information, and attribute information. The vertex information indicates the positions of the vertices of a face in three-dimensional space. The connection information indicates the connections between the vertices. A face can be identified by the vertex information and connection information. In other words, a colorless three-dimensional object is formed in three-dimensional space by the vertex information and connection information.

[0066] The attribute information may be associated with a vertex or a face. The attribute information associated with a vertex may be expressed as "Attribute Per Point." The attribute information associated with a vertex may indicate an attribute of the vertex itself, or may indicate an attribute of a face connected to the vertex.

[0067] For example, a color may be associated with a vertex as attribute information. The color associated with a vertex may be the color of the vertex itself, or the color of a face connected to the vertex. The color of a face may be the average of multiple colors associated with multiple vertices of the face. Furthermore, a normal vector may be associated with a vertex or a face as attribute information. Such a normal vector can represent the front and back of a face.

[0068] A two-dimensional image may be associated with a surface as attribute information. The two-dimensional image associated with a surface is also expressed as a texture image or an "Attribute Map." Information indicating mapping between the surface and the two-dimensional image may be associated with the surface as attribute information. Information indicating such mapping may be expressed as mapping information, vertex information of a texture image, texture coordinates, or "Attribute UV Coordinate."

[0069] Furthermore, information such as color, image, and moving image used as attribute information may be expressed as "parametric space."

[0070] The attribute information allows texture to be reflected on the three-dimensional object. That is, a three-dimensional object having color is formed in three-dimensional space based on the vertex information, connection information, and attribute information.

[0071] In the above, the attribute information is associated with the vertices or faces, but it may also be associated with the edges.

[0072] 3 is a conceptual diagram illustrating mapping according to this embodiment. For example, a region of a two-dimensional image on a two-dimensional plane can be mapped onto a surface of a three-dimensional mesh in three-dimensional space. Specifically, coordinate information of the region in the two-dimensional image is associated with the surface of the three-dimensional mesh. As a result, an image of the mapped region in the two-dimensional image is reflected on the surface of the three-dimensional mesh.

[0073] By using the mapping, the 2D image used as attribute information can be separated from the 3D mesh. For example, in encoding the 3D mesh, the 2D image may be encoded by an image encoding method or a video encoding method.

[0074] <System Configuration> Fig. 4 is a block diagram showing an example of the configuration of a coding / decoding system according to this embodiment. In Fig. 4, the coding / decoding system includes a coding device 100 and a decoding device 200.

[0075] For example, the encoding device 100 obtains a three-dimensional mesh and encodes the three-dimensional mesh into a bitstream. Then, the encoding device 100 outputs the bitstream to the network 300. For example, the bitstream includes the encoded three-dimensional mesh and control information for decoding the encoded three-dimensional mesh. By encoding the three-dimensional mesh, information about the three-dimensional mesh is compressed.

[0076] The network 300 transmits a bitstream from the encoding device 100 to the decoding device 200. The network 300 may be the Internet, a wide area network (WAN), a local area network (LAN), or a combination of these. The network 300 is not necessarily limited to bidirectional communication, and may be a unidirectional communication network for terrestrial digital broadcasting, satellite broadcasting, or the like.

[0077] Furthermore, the network 300 can be replaced by a recording medium such as a DVD (Digital Versatile Disc) or a BD (Blu-Ray Disc (registered trademark)).

[0078] The decoding device 200 obtains a bitstream and decodes a three-dimensional mesh from the bitstream. By decoding the three-dimensional mesh, information about the three-dimensional mesh is expanded. For example, the decoding device 200 decodes the three-dimensional mesh according to a decoding method corresponding to the encoding method used by the encoding device 100 to encode the three-dimensional mesh. That is, the encoding device 100 and the decoding device 200 perform encoding and decoding according to encoding methods and decoding methods that correspond to each other.

[0079] The 3D mesh before encoding may also be referred to as an original 3D mesh, and the 3D mesh after decoding may also be referred to as a reconstructed 3D mesh.

[0080] 5 is a block diagram showing an example of the configuration of a coding device 100 according to this embodiment. For example, the coding device 100 includes a vertex information encoder 101, a connection information encoder 102, and an attribute information encoder 103.

[0081] The vertex information encoder 101 is an electrical circuit that encodes vertex information. For example, the vertex information encoder 101 encodes the vertex information into a bitstream according to a format defined for the vertex information.

[0082] The connection information encoder 102 is an electrical circuit that encodes the connection information, for example, the connection information encoder 102 encodes the connection information into a bitstream according to a format defined for the connection information.

[0083] The attribute information encoder 103 is an electric circuit that encodes the attribute information. For example, the attribute information encoder 103 encodes the attribute information into a bit stream in accordance with a format defined for the attribute information.

[0084] The vertex information, connectivity information, and attribute information may be coded using variable-length coding or fixed-length coding, such as Huffman coding or context-adaptive binary arithmetic coding (CABAC).

[0085] The vertex information encoder 101, the connection information encoder 102, and the attribute information encoder 103 may be integrated together, or each of the vertex information encoder 101, the connection information encoder 102, and the attribute information encoder 103 may be further subdivided into multiple components.

[0086] 6 is a block diagram showing another example of the configuration of the encoding device 100 according to this embodiment. For example, the encoding device 100 includes a pre-processor 104 and a post-processor 105 in addition to the configuration shown in FIG.

[0087] The preprocessor 104 is an electrical circuit that performs processing before encoding the vertex information, connectivity information, and attribute information. For example, the preprocessor 104 may perform a conversion process, a separation process, a multiplexing process, or the like on the 3D mesh before encoding. More specifically, for example, the preprocessor 104 may separate the vertex information, connectivity information, and attribute information from the 3D mesh before encoding.

[0088] The post-processor 105 is an electrical circuit that performs processing after the vertex information, connection information, and attribute information are encoded. For example, the post-processor 105 may perform conversion processing, separation processing, multiplexing processing, or the like on the encoded vertex information, connection information, and attribute information. More specifically, for example, the post-processor 105 may multiplex the encoded vertex information, connection information, and attribute information into a bitstream. Furthermore, for example, the post-processor 105 may further perform variable-length coding on the encoded vertex information, connection information, and attribute information.

[0089] 7 is a block diagram showing an example of the configuration of a decoding device 200 according to this embodiment. For example, the decoding device 200 includes a vertex information decoder 201, a connection information decoder 202, and an attribute information decoder 203.

[0090] The vertex information decoder 201 is an electrical circuit that decodes vertex information. For example, the vertex information decoder 201 decodes vertex information from a bitstream according to a format defined for the vertex information.

[0091] The connection information decoder 202 is an electrical circuit that decodes the connection information, for example, the connection information decoder 202 decodes the connection information from the bitstream according to a format defined for the connection information.

[0092] The attribute information decoder 203 is an electric circuit that decodes the attribute information. For example, the attribute information decoder 203 decodes the attribute information from the bitstream in accordance with a format defined for the attribute information.

[0093] The vertex information, connection information, and attribute information may be decoded using variable length decoding or fixed length decoding, which may correspond to Huffman coding, context-adaptive binary arithmetic coding (CABAC), or the like.

[0094] The vertex information decoder 201, the connection information decoder 202, and the attribute information decoder 203 may be integrated together, or each of the vertex information decoder 201, the connection information decoder 202, and the attribute information decoder 203 may be further subdivided into multiple components.

[0095] 8 is a block diagram showing another example of the configuration of the decoding device 200 according to this embodiment. For example, the decoding device 200 includes a pre-processor 204 and a post-processor 205 in addition to the configuration shown in FIG.

[0096] The preprocessor 204 is an electrical circuit that performs processing before decoding the vertex information, connection information, and attribute information. For example, the preprocessor 204 may perform conversion processing, separation processing, multiplexing processing, or the like on the bitstream before decoding the vertex information, connection information, and attribute information.

[0097] More specifically, for example, the preprocessor 204 may separate a sub-bitstream corresponding to vertex information, a sub-bitstream corresponding to connectivity information, and a sub-bitstream corresponding to attribute information from the bitstream. Also, for example, the preprocessor 204 may perform variable-length decoding on the bitstream in advance before decoding the vertex information, connectivity information, and attribute information.

[0098] The post-processor 205 is an electrical circuit that performs processing after the vertex information, connection information, and attribute information are decoded. For example, the post-processor 205 may perform conversion processing, separation processing, multiplexing processing, or the like on the decoded vertex information, connection information, and attribute information. More specifically, for example, the post-processor 205 may multiplex the decoded vertex information, connection information, and attribute information onto a three-dimensional mesh.

[0099] <Bitstream> Vertex information, connection information, and attribute information are coded and stored in a bitstream. The relationship between this information and the bitstream is shown below.

[0100] 9 is a conceptual diagram showing an example of the configuration of a bitstream according to this embodiment. In this example, connection information, vertex information, and attribute information are integrated in the bitstream. For example, the connection information, vertex information, and attribute information may be included in a single file.

[0101] Furthermore, multiple portions of this information may be stored sequentially, such as a first portion of connection information, a first portion of vertex information, a first portion of attribute information, a second portion of connection information, a second portion of vertex information, a second portion of attribute information, etc. These multiple portions may correspond to multiple portions that are different in time, multiple portions that are different in space, or multiple different faces.

[0102] Furthermore, the storage order of the connection information, vertex information, and attribute information is not limited to the above example, and a storage order different from the above example may be used.

[0103] 10 is a conceptual diagram showing another example of the configuration of a bitstream according to this embodiment. In this example, a plurality of files are included in the bitstream, and connection information, vertex information, and attribute information are stored in different files. Here, a file containing connection information, a file containing vertex information, and a file containing attribute information are shown, but the storage format is not limited to this example. For example, two types of information among the connection information, vertex information, and attribute information may be included in one file, and the remaining type of information may be included in another file.

[0104] Alternatively, the information may be split and stored in more files. For example, multiple pieces of connectivity information may be stored in multiple files, multiple pieces of vertex information may be stored in multiple files, or multiple pieces of attribute information may be stored in multiple files. These multiple pieces may correspond to multiple temporally different pieces, multiple spatially different pieces, or multiple different faces.

[0105] Furthermore, the storage order of the connection information, vertex information, and attribute information is not limited to the above example, and a storage order different from the above example may be used.

[0106] 11 is a conceptual diagram showing another example of the configuration of a bitstream according to this embodiment. In this example, the bitstream is composed of multiple separable sub-bitstreams, and connection information, vertex information, and attribute information are stored in different sub-bitstreams.

[0107] Here, a sub-bitstream containing connection information, a sub-bitstream containing vertex information, and a sub-bitstream containing attribute information are shown, but the storage format is not limited to this example.

[0108] For example, two types of information among the connection information, vertex information, and attribute information may be included in one sub-bitstream, and the remaining type of information may be included in another sub-bitstream. Specifically, attribute information of a two-dimensional image or the like may be stored in a sub-bitstream that complies with an image coding method, separate from the sub-bitstreams of the connection information and vertex information.

[0109] Also, each sub-bitstream may include multiple files, and multiple pieces of connectivity information may be stored in multiple files, multiple pieces of vertex information may be stored in multiple files, or multiple pieces of attribute information may be stored in multiple files.

[0110] 9, 10, and 11, and a storage order different from the above examples may be used. For example, the vertex information, connection information, and attribute information may be stored in the bitstream in this order. Alternatively, the connection information, connection information, and attribute information may be stored in the bitstream in any of the following orders: connection information, attribute information, and vertex information; vertex information, attribute information, and connection information; attribute information, connection information, and vertex information; or attribute information, vertex information, and connection information.

[0111] Furthermore, each of the connection information, vertex information, and attribute information may be divided into a plurality of data, and the plurality of data may be stored in a cyclical or random order within the bitstream.

[0112] 12 is a block diagram showing a specific example of an encoding / decoding system according to this embodiment. In FIG. 12, the encoding / decoding system includes a three-dimensional data encoding system 110, a three-dimensional data decoding system 210, and an external connector 310.

[0113] The three-dimensional data encoding system 110 includes a controller 111, an input / output processor 112, a three-dimensional data encoder 113, a three-dimensional data generator 115, and a system multiplexer 114. The three-dimensional data decoding system 210 includes a controller 211, an input / output processor 212, a three-dimensional data decoder 213, a system demultiplexer 214, a presenter 215, and a user interface 216.

[0114] In the three-dimensional data encoding system 110, sensor data is input from a sensor terminal to a three-dimensional data generator 115. The three-dimensional data generator 115 generates three-dimensional data, such as point cloud data or mesh data, from the sensor data and inputs it to a three-dimensional data encoder 113.

[0115] For example, the three-dimensional data generator 115 generates vertex information, and generates connection information and attribute information corresponding to the vertex information. The three-dimensional data generator 115 may process the vertex information when generating the connection information and attribute information. For example, the three-dimensional data generator 115 may reduce the amount of data by deleting duplicate vertices, or may transform the vertex information (such as by shifting its position, rotating it, or normalizing it). The three-dimensional data generator 115 may also render the attribute information.

[0116] Furthermore, although the three-dimensional data generator 115 is a component of the three-dimensional data encoding system 110 in FIG. 12, it may be arranged externally and independently of the three-dimensional data encoding system 110.

[0117] The sensor terminal that provides the sensor data for generating the three-dimensional data may be, for example, a moving body such as an automobile, a flying object such as an airplane, a mobile terminal, a camera, etc. Furthermore, a distance sensor such as a LIDAR, a millimeter wave radar, an infrared sensor, or a range finder, a stereo camera, or a combination of multiple monocular cameras may also be used as the sensor terminal.

[0118] The sensor data may be the distance (position) of the object, monocular camera images, stereo camera images, color, reflectance, sensor attitude, orientation, gyro, sensing position (GPS information or altitude), speed, acceleration, sensing time, temperature, air pressure, humidity, or magnetism.

[0119] The three-dimensional data encoder 113 corresponds to the encoding device 100 shown in FIG. 5 and other figures. For example, the three-dimensional data encoder 113 encodes three-dimensional data to generate encoded data. The three-dimensional data encoder 113 also generates control information when encoding the three-dimensional data. The three-dimensional data encoder 113 then inputs the encoded data together with the control information to the system multiplexer 114.

[0120] The encoding method for the three-dimensional data may be an encoding method using geometry or an encoding method using a video codec. Here, the encoding method using geometry may also be referred to as a geometry-based encoding method. The encoding method using a video codec may also be referred to as a video-based encoding method.

[0121] The system multiplexer 114 multiplexes the encoded data and control information input from the 3D data encoder 113 to generate multiplexed data using a specified multiplexing method. The system multiplexer 114 may multiplex other media such as video, audio, subtitles, application data, or document files, or reference time information, along with the encoded data and control information of the 3D data. Furthermore, the system multiplexer 114 may multiplex attribute information related to the sensor data or the 3D data.

[0122] For example, the multiplexed data may have a file format for storage or a packet format for transmission. As these formats, ISOBMFF or a format based on ISOBMFF may be used. Also, MPEG-DASH, MMT, MPEG-2 TS Systems, RTP, or the like may be used.

[0123] The multiplexed data is then output as a transmission signal to the external connector 310 by the input / output processor 112. The multiplexed data may be transmitted as a transmission signal by wire or wirelessly. Alternatively, the multiplexed data is stored in an internal memory or a storage device. The multiplexed data may be transmitted to a cloud server via the Internet or may be stored in an external storage device.

[0124] For example, the transmission or storage of the multiplexed data is performed by a method according to the medium for transmission or storage, such as broadcasting or communication. The communication protocol may be http, ftp, TCP, UDP, IP, or a combination thereof. Furthermore, a pull-type communication method or a push-type communication method may be used.

[0125] For wired transmission, Ethernet (registered trademark), USB, RS-232C, HDMI (registered trademark), coaxial cable, etc. may be used. For wireless transmission, 3GPP (registered trademark), 3G / 4G / 5G defined by IEEE, wireless LAN, Wi-Fi, Bluetooth, or millimeter wave may be used. For broadcasting, for example, DVB-T2, DVB-S2, DVB-C2, ATSC3.0, or ISDB-S3 may be used.

[0126] The sensor data may be input to the three-dimensional data generator 115 or the system multiplexer 114. The three-dimensional data or encoded data may be output as a transmission signal directly to the external connector 310 via the input / output processor 112. The transmission signal output from the three-dimensional data encoding system 110 is input to the three-dimensional data decoding system 210 via the external connector 310.

[0127] Furthermore, each operation of the three-dimensional data encoding system 110 may be controlled by a controller 111 that executes an application program.

[0128] In the three-dimensional data decoding system 210, a transmission signal is input to an input / output processor 212. The input / output processor 212 decodes multiplexed data having a file format or a packet format from the transmission signal and inputs the multiplexed data to a system demultiplexer 214. The system demultiplexer 214 obtains coded data and control information from the multiplexed data and inputs them to a three-dimensional data decoder 213. The system demultiplexer 214 may extract other media or reference time information from the multiplexed data.

[0129] The three-dimensional data decoder 213 corresponds to the decoding device 200 shown in Fig. 7 etc. For example, the three-dimensional data decoder 213 decodes three-dimensional data from the encoded data based on a predefined encoding method. The three-dimensional data is then presented to the user by the presenter 215.

[0130] Additionally, additional information such as sensor data may be input to the presenter 215. The presenter 215 may present three-dimensional data based on the additional information. Additionally, a user instruction may be input from a user terminal to the user interface 216. Then, the presenter 215 may present three-dimensional data based on the input instruction.

[0131] The input / output processor 212 may acquire the three-dimensional data and the encoded data from the external connector 310 .

[0132] Furthermore, each operation of the three-dimensional data decoding system 210 may be controlled by a controller 211 that executes an application program.

[0133] 13 is a conceptual diagram showing an example of the configuration of point cloud data according to this embodiment. The point cloud data is data of a group of points representing a three-dimensional object.

[0134] Specifically, a point cloud is made up of a plurality of points, and has position information indicating the three-dimensional coordinate position of each point and attribute information indicating the attribute of each point. The position information is also expressed as geometry.

[0135] The type of attribute information may be, for example, color, reflectance, etc. One point may be associated with attribute information of one type, one point may be associated with attribute information of multiple different types, or one point may be associated with attribute information having multiple values ​​for the same type.

[0136] 14 is a conceptual diagram showing an example of a data file of point cloud data according to this embodiment. This example shows a case where there is a one-to-one correspondence between position information items and attribute information items, and shows position information and attribute information for N points that make up the point cloud data. In this example, the position information is information indicating a three-dimensional coordinate position using three axes, x, y, and z, and the attribute information is information indicating a color using RGB. A PLY file or the like can be used as a representative data file for point cloud data.

[0137] 15 is a conceptual diagram showing an example of the configuration of mesh data according to this embodiment. Mesh data is data used in CG (Computer Graphics) and the like, and is three-dimensional mesh data that shows the three-dimensional shape of an object using multiple surfaces. Each surface is also expressed as a polygon, and has a polygonal shape such as a triangle or a rectangle.

[0138] Specifically, a 3D mesh is composed of a plurality of points constituting a point cloud, as well as a plurality of edges and a plurality of faces. Each point is also expressed as a vertex or a position. Each edge corresponds to a line segment connected by two vertices. Each face corresponds to an area surrounded by three or more edges.

[0139] Furthermore, a three-dimensional mesh has position information indicating the three-dimensional coordinate positions of vertices. The position information is also expressed as vertex information or geometry. A three-dimensional mesh also has connection information indicating the relationship between multiple vertices that make up an edge or a face. The connection information is also expressed as connectivity. A three-dimensional mesh also has attribute information indicating the attributes of the vertices, edges, or faces. The attribute information in a three-dimensional mesh is also expressed as texture.

[0140] For example, the attribute information may indicate the color, reflectance, or normal vector for a vertex, edge, or face. The direction of the normal vector may represent the front and back of the face.

[0141] The mesh data may be stored in a data file format such as an object file.

[0142] 16 is a conceptual diagram showing an example of a data file of mesh data according to this embodiment. In this example, the data file includes position information G(1) to G(N) of N vertices that make up the three-dimensional mesh, and attribute information A1(1) to A1(N) of the N vertices. Also, in this example, M pieces of attribute information A2(1) to A2(M) are included. The attribute information items do not need to correspond one-to-one to vertices or faces. Furthermore, attribute information need not exist.

[0143] The connection information is represented by a combination of vertex indices. n[1, 3, 4] indicates a triangular face formed by three vertices, n=1, n=3, and n=4. Also, m[2, 4, 6] indicates that the attribute information of m=2, m=4, and m=6 corresponds to the three vertices, respectively.

[0144] Furthermore, the actual contents of the attribute information may be written in a separate file. A pointer to that content may be associated with a vertex, a face, or the like. For example, attribute information indicating an image for a face may be stored in a two-dimensional attribute map file. The file name of the attribute map and two-dimensional coordinate values ​​in the attribute map may be written in attribute information A2(1) to A2(M). The method of specifying attribute information for a face is not limited to these methods, and any method may be used.

[0145] 17 is a conceptual diagram showing types of three-dimensional data according to this embodiment. Point cloud data and mesh data may represent static objects or dynamic objects. A static object is an object that does not change over time, and a dynamic object is an object that changes over time. A static object may correspond to three-dimensional data for any point in time.

[0146] For example, point cloud data for a given point in time may be referred to as a PCC frame, mesh data for a given point in time may be referred to as a mesh frame, and PCC frames and mesh frames may be simply referred to as frames.

[0147] The area of ​​the object may be limited to a certain range, as in normal video data, or may not be limited, as in map data. The density of points or surfaces may be determined in various ways. Sparse point cloud data or sparse mesh data may be used, or dense point cloud data or dense mesh data may be used.

[0148] Next, encoding and decoding of a point cloud or a three-dimensional mesh will be described. The device, process, or syntax for encoding and decoding vertex information of a three-dimensional mesh in the present disclosure may be applied to encoding and decoding of a point cloud. The device, process, or syntax for encoding and decoding of a point cloud in the present disclosure may be applied to encoding and decoding vertex information of a three-dimensional mesh.

[0149] Furthermore, a device, process, or syntax for encoding and decoding attribute information of a point cloud in the present disclosure may be applied to encoding and decoding connectivity information or attribute information of a three-dimensional mesh.Furthermore, a device, process, or syntax for encoding and decoding connectivity information or attribute information of a three-dimensional mesh in the present disclosure may be applied to encoding and decoding attribute information of a point cloud.

[0150] Furthermore, at least some of the processing may be shared between the encoding and decoding of point cloud data and the encoding and decoding of mesh data, thereby reducing the scale of the circuit and software program.

[0151] 18 is a block diagram showing an example configuration of a three-dimensional data encoder 113 according to this embodiment. In this example, the three-dimensional data encoder 113 includes a vertex information encoder 121, an attribute information encoder 122, a metadata encoder 123, and a multiplexer 124. The vertex information encoder 121, the attribute information encoder 122, and the multiplexer 124 may correspond to the vertex information encoder 101, the attribute information encoder 103, the post-processor 105, etc. in FIG.

[0152] In this example, the three-dimensional data encoder 113 encodes the three-dimensional data according to a geometry-based encoding method, which takes into account the three-dimensional structure. In addition, in the geometry-based encoding method, attribute information is encoded using configuration information obtained in encoding the vertex information.

[0153] Specifically, first, vertex information, attribute information, and metadata included in three-dimensional data generated from sensor data are input to a vertex information encoder 121, an attribute information encoder 122, and a metadata encoder 123, respectively. Here, connectivity information included in the three-dimensional data may be treated in the same way as attribute information. In addition, in the case of point cloud data, position information may be treated as vertex information.

[0154] The vertex information encoder 121 encodes the vertex information into compressed vertex information and outputs the compressed vertex information as encoded data to the multiplexer 124. The vertex information encoder 121 also generates metadata for the compressed vertex information and outputs it to the multiplexer 124. The vertex information encoder 121 also generates configuration information and outputs it to the attribute information encoder 122.

[0155] The attribute information encoder 122 uses the configuration information generated by the vertex information encoder 121 to encode the attribute information into compressed attribute information and outputs the compressed attribute information as encoded data to the multiplexer 124. The attribute information encoder 122 also generates metadata of the compressed attribute information and outputs it to the multiplexer 124.

[0156] The metadata encoder 123 encodes compressible metadata into compressed metadata and outputs the compressed metadata as encoded data to the multiplexer 124. The metadata encoded by the metadata encoder 123 may be used to encode vertex information and attribute information.

[0157] The multiplexer 124 multiplexes the compressed vertex information, the compressed vertex information metadata, the compressed attribute information, the compressed attribute information metadata, and the compressed metadata into a bitstream, and then inputs the bitstream to the system layer.

[0158] 19 is a block diagram showing an example configuration of a three-dimensional data decoder 213 according to this embodiment. In this example, the three-dimensional data decoder 213 includes a vertex information decoder 221, an attribute information decoder 222, a metadata decoder 223, and a demultiplexer 224. The vertex information decoder 221, the attribute information decoder 222, and the demultiplexer 224 may correspond to the vertex information decoder 201, the attribute information decoder 203, the preprocessor 204, and the like in FIG.

[0159] In this example, the three-dimensional data decoder 213 decodes three-dimensional data according to a geometry-based encoding method. The three-dimensional structure is taken into consideration in the decoding according to the geometry-based encoding method. Furthermore, in the decoding according to the geometry-based encoding method, attribute information is decoded using configuration information obtained in decoding vertex information.

[0160] Specifically, first, a bitstream is input from the system layer to a demultiplexer 224. The demultiplexer 224 separates compressed vertex information, compressed vertex information metadata, compressed attribute information, compressed attribute information metadata, and compressed metadata from the bitstream. The compressed vertex information and compressed vertex information metadata are input to a vertex information decoder 221. The compressed attribute information and compressed attribute information metadata are input to an attribute information decoder 222. The metadata is input to a metadata decoder 223.

[0161] The vertex information decoder 221 decodes vertex information from the compressed vertex information using metadata of the compressed vertex information. The vertex information decoder 221 also generates configuration information and outputs it to the attribute information decoder 222. The attribute information decoder 222 decodes attribute information from the compressed attribute information using the configuration information generated by the vertex information decoder 221 and the metadata of the compressed attribute information. The metadata decoder 223 decodes metadata from the compressed metadata. The metadata decoded by the metadata decoder 223 may be used to decode the vertex information and the attribute information.

[0162] Thereafter, the vertex information, attribute information, and metadata are output as three-dimensional data from the three-dimensional data decoder 213. Note that, for example, this metadata is metadata of the vertex information and attribute information, and can be used in an application program.

[0163] 20 is a block diagram showing another example configuration of the three-dimensional data encoder 113 according to the present embodiment. In this example, the three-dimensional data encoder 113 includes a vertex image generator 131, an attribute image generator 132, a metadata generator 133, a video encoder 134, a metadata encoder 123, and a multiplexer 124. The vertex image generator 131, the attribute image generator 132, and the video encoder 134 may correspond to the vertex information encoder 101 and the attribute information encoder 103 in FIG. 6 , etc.

[0164] In this example, the 3D data encoder 113 encodes the 3D data according to a video-based encoding method. In encoding according to the video-based encoding method, multiple 2D images are generated from the 3D data, and the multiple 2D images are encoded according to a video encoding method. Here, the video encoding method may be High Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), or the like.

[0165] Specifically, first, vertex information and attribute information included in three-dimensional data generated from sensor data are input to a metadata generator 133. The vertex information and attribute information are then input to a vertex image generator 131 and an attribute image generator 132, respectively. The metadata included in the three-dimensional data is then input to a metadata encoder 123. Here, connectivity information included in the three-dimensional data may be treated in the same way as attribute information. In the case of point cloud data, position information may be treated as vertex information.

[0166] The metadata generator 133 generates map information of a plurality of two-dimensional images from the vertex information and attribute information, and inputs the map information to the vertex image generator 131, the attribute image generator 132, and the metadata encoder 123.

[0167] The vertex image generator 131 generates a vertex image based on the vertex information and map information, and inputs the generated image to the video encoder 134. The attribute image generator 132 generates an attribute image based on the attribute information and map information, and inputs the generated image to the video encoder 134.

[0168] The video encoder 134 encodes the vertex images and attribute images into compressed vertex information and compressed attribute information, respectively, in accordance with a video encoding method, and outputs the compressed vertex information and compressed attribute information as encoded data to the multiplexer 124. The video encoder 134 also generates metadata for the compressed vertex information and metadata for the compressed attribute information, and outputs them to the multiplexer 124.

[0169] The metadata encoder 123 encodes the compressible metadata into compressed metadata and outputs the compressed metadata as encoded data to the multiplexer 124. The compressible metadata includes map information. The metadata encoded by the metadata encoder 123 may also be used to encode vertex information and attribute information.

[0170] The multiplexer 124 multiplexes the compressed vertex information, the compressed vertex information metadata, the compressed attribute information, the compressed attribute information metadata, and the compressed metadata into a bitstream, and then inputs the bitstream to the system layer.

[0171] 21 is a block diagram showing another example configuration of the 3D data decoder 213 according to this embodiment. In this example, the 3D data decoder 213 includes a vertex information generator 231, an attribute information generator 232, a video decoder 234, a metadata decoder 223, and a demultiplexer 224. The vertex information generator 231, the attribute information generator 232, and the video decoder 234 may correspond to the vertex information decoder 201 and the attribute information decoder 203 in FIG. 8, etc.

[0172] In this example, the 3D data decoder 213 decodes the 3D data according to a video-based coding method. In the decoding according to the video-based coding method, a plurality of 2D images are decoded according to a video coding method, and 3D data is generated from the plurality of 2D images. Here, the video coding method may be High Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), or the like.

[0173] Specifically, first, a bitstream is input from the system layer to the demultiplexer 224. The demultiplexer 224 separates compressed vertex information, compressed vertex information metadata, compressed attribute information, compressed attribute information metadata, and compressed metadata from the bitstream. The compressed vertex information, compressed vertex information metadata, compressed attribute information, and compressed attribute information metadata are input to the video decoder 234. The compressed metadata is input to the metadata decoder 223.

[0174] The video decoder 234 decodes the vertex images in accordance with the video encoding method. At this time, the video decoder 234 decodes the vertex images from the compressed vertex information using the metadata of the compressed vertex information. Then, the video decoder 234 inputs the vertex images to the vertex information generator 231. The video decoder 234 also decodes the attribute images in accordance with the video encoding method. At this time, the video decoder 234 decodes the attribute images from the compressed attribute information using the metadata of the compressed attribute information. Then, the video decoder 234 inputs the attribute images to the attribute information generator 232.

[0175] The metadata decoder 223 decodes metadata from the compressed metadata. The metadata decoded by the metadata decoder 223 includes map information used to generate vertex information and attribute information. The metadata decoded by the metadata decoder 223 may also be used to decode vertex images and attribute images.

[0176] The vertex information generator 231 reproduces vertex information from the vertex image in accordance with the map information included in the metadata decoded by the metadata decoder 223. The attribute information generator 232 reproduces attribute information from the attribute image in accordance with the map information included in the metadata decoded by the metadata decoder 223.

[0177] Thereafter, the vertex information, attribute information, and metadata are output as three-dimensional data from the three-dimensional data decoder 213. Note that, for example, this metadata is metadata of the vertex information and attribute information, and can be used in an application program.

[0178] Fig. 22 is a conceptual diagram showing a specific example of encoding processing according to this embodiment. Fig. 22 shows a three-dimensional data encoder 113 and a description encoder 148. In this example, the three-dimensional data encoder 113 includes a two-dimensional data encoder 141 and a mesh data encoder 142. The two-dimensional data encoder 141 includes a texture encoder 143. The mesh data encoder 142 includes a vertex information encoder 144 and a connection information encoder 145.

[0179] The vertex information encoder 144, the connection information encoder 145, and the texture encoder 143 may correspond to the vertex information encoder 101, the connection information encoder 102, and the attribute information encoder 103 in FIG.

[0180] For example, the two-dimensional data encoder 141 operates as a texture encoder 143 and generates a texture file by encoding the texture corresponding to the attribute information as two-dimensional data according to an image encoding method or a video encoding method.

[0181] The mesh data encoder 142 also operates as a vertex information encoder 144 and a connectivity information encoder 145, and generates a mesh file by encoding the vertex information and connectivity information. The mesh data encoder 142 may further encode mapping information for textures. The encoded mapping information may then be included in the mesh file.

[0182] The description encoder 148 also generates a description file by encoding a description corresponding to metadata such as text data. The description encoder 148 may encode the description at the system layer. For example, the description encoder 148 may be included in the system multiplexer 114 of FIG. 12 .

[0183] The above operations generate a bitstream containing texture files, mesh files, and description files, which may be multiplexed into the bitstream in file formats such as glTF (Graphics Language Transmission Format) or USD (Universal Scene Description).

[0184] The three-dimensional data encoder 113 may include two mesh data encoders as the mesh data encoder 142. For example, one mesh data encoder encodes vertex information and connectivity information of a static three-dimensional mesh, and the other mesh data encoder encodes vertex information and connectivity information of a dynamic three-dimensional mesh.

[0185] Correspondingly, two mesh files may then be included in the bitstream: for example, one mesh file corresponding to a static 3D mesh and another mesh file corresponding to a dynamic 3D mesh.

[0186] Furthermore, the static three-dimensional mesh may be a three-dimensional mesh of an intraframe coded using intraprediction, and the dynamic three-dimensional mesh may be a three-dimensional mesh of an interframe coded using interprediction. Furthermore, information on the dynamic three-dimensional mesh may be differential information between vertex information or connectivity information of the three-dimensional mesh of an intraframe and vertex information or connectivity information of the three-dimensional mesh of an interframe.

[0187] Fig. 23 is a conceptual diagram showing a specific example of the decoding process according to this embodiment. Fig. 23 shows a three-dimensional data decoder 213, a description decoder 248, and a renderer 247. In this example, the three-dimensional data decoder 213 includes a two-dimensional data decoder 241, a mesh data decoder 242, and a mesh reconstructor 246. The two-dimensional data decoder 241 includes a texture decoder 243. The mesh data decoder 242 includes a vertex information decoder 244 and a connectivity information decoder 245.

[0188] The vertex information decoder 244, the connection information decoder 245, the texture decoder 243, and the mesh reconstructor 246 may correspond to the vertex information decoder 201, the connection information decoder 202, the attribute information decoder 203, and the post-processor 205 in Fig. 8. The presenter 247 may correspond to the presenter 215 in Fig. 12.

[0189] For example, the two-dimensional data decoder 241 operates as a texture decoder 243, and decodes the texture corresponding to the attribute information from the texture file as two-dimensional data in accordance with an image coding method or a video coding method.

[0190] The mesh data decoder 242 also operates as a vertex information decoder 244 and a connectivity information decoder 245 to decode vertex information and connectivity information from the mesh file. The mesh data decoder 242 may further decode mapping information for textures from the mesh file.

[0191] The description decoder 248 also decodes descriptions corresponding to metadata such as text data from the description file. The description decoder 248 may decode the descriptions at the system layer. For example, the description decoder 248 may be included in the system demultiplexer 214 of FIG. 12 .

[0192] The mesh reconstructor 246 reconstructs a 3D mesh from the vertex information, connectivity information, and textures according to the description. The renderer 247 renders and outputs the 3D mesh according to the description.

[0193] Through the above operations, a 3D mesh is reconstructed and output from a bitstream containing a texture file, a mesh file, and a description file.

[0194] The three-dimensional data decoder 213 may include two mesh data decoders as the mesh data decoder 242. For example, one mesh data decoder decodes vertex information and connectivity information of a static three-dimensional mesh, and the other mesh data decoder decodes vertex information and connectivity information of a dynamic three-dimensional mesh.

[0195] Correspondingly, two mesh files may then be included in the bitstream: for example, one mesh file corresponding to a static 3D mesh and another mesh file corresponding to a dynamic 3D mesh.

[0196] Furthermore, the static three-dimensional mesh may be a three-dimensional mesh of an intraframe coded using intraprediction, and the dynamic three-dimensional mesh may be a three-dimensional mesh of an interframe coded using interprediction. Furthermore, information on the dynamic three-dimensional mesh may be differential information between vertex information or connectivity information of the three-dimensional mesh of an intraframe and vertex information or connectivity information of the three-dimensional mesh of an interframe.

[0197] A dynamic 3D mesh coding method is sometimes called DMC (Dynamic Mesh Coding), and a video-based dynamic 3D mesh coding method is sometimes called V-DMC (Video-based Dynamic Mesh Coding).

[0198] The point cloud encoding method is sometimes called PCC (Point Cloud Compression). The point cloud video-based encoding method is sometimes called V-PCC (Video-based Point Cloud Compression). The point cloud geometry-based encoding method is sometimes called G-PCC (Geometry-based Point Cloud Compression).

[0199] <Implementation Example> Fig. 24 is a block diagram showing an implementation example of the encoding device 100 according to this embodiment. The encoding device 100 includes a circuit 151 and a memory 152. For example, multiple components of the encoding device 100 shown in Fig. 5 etc. are implemented by the circuit 151 and memory 152 shown in Fig. 24.

[0200] The circuit 151 is a circuit that performs information processing and is a circuit that can access the memory 152. For example, the circuit 151 is a dedicated or general-purpose electric circuit that encodes a three-dimensional mesh. The circuit 151 may be a processor such as a CPU. Alternatively, the circuit 151 may be a collection of multiple electric circuits.

[0201] The memory 152 is a dedicated or general-purpose memory that stores information used by the circuit 151 to encode the three-dimensional mesh. The memory 152 may be an electric circuit and may be connected to the circuit 151. The memory 152 may also be included in the circuit 151. The memory 152 may also be a collection of multiple electric circuits. The memory 152 may also be a magnetic disk, an optical disk, or the like, and may also be expressed as a storage, a recording medium, or the like. The memory 152 may also be a non-volatile memory or a volatile memory.

[0202] For example, the memory 152 may store a three-dimensional mesh or a bitstream, or may store a program for the circuit 151 to encode the three-dimensional mesh.

[0203] Note that the encoding device 100 does not necessarily have to implement all of the components shown in Figure 5 and the like, and does not necessarily have to perform all of the processes shown here. Some of the components shown in Figure 5 and the like may be included in another device, and some of the processes shown here may be executed by another device. Furthermore, the encoding device 100 may implement any combination of the components of the present disclosure, and may perform any combination of the processes of the present disclosure.

[0204] Fig. 25 is a block diagram showing an example implementation of a decoding device 200 according to this embodiment. The decoding device 200 includes a circuit 251 and a memory 252. For example, multiple components of the decoding device 200 shown in Fig. 7 and other figures are implemented by the circuit 251 and memory 252 shown in Fig. 25.

[0205] The circuit 251 is a circuit that performs information processing and is a circuit that can access the memory 252. For example, the circuit 251 is a dedicated or general-purpose electric circuit that decodes a three-dimensional mesh. The circuit 251 may be a processor such as a CPU. Alternatively, the circuit 251 may be a collection of multiple electric circuits.

[0206] The memory 252 is a dedicated or general-purpose memory that stores information for the circuit 251 to decode the 3D mesh. The memory 252 may be an electric circuit and may be connected to the circuit 251. The memory 252 may also be included in the circuit 251. The memory 252 may also be a collection of multiple electric circuits. The memory 252 may also be a magnetic disk, an optical disk, or the like, and may also be expressed as a storage, a recording medium, or the like. The memory 252 may also be a non-volatile memory or a volatile memory.

[0207] For example, the memory 252 may store a three-dimensional mesh or a bitstream, or may store a program for the circuit 251 to decode the three-dimensional mesh.

[0208] Note that the decoding device 200 does not necessarily have to implement all of the components shown in Figure 7 and the like, and does not necessarily have to perform all of the processes shown here. Some of the components shown in Figure 7 and the like may be included in another device, and some of the processes shown here may be executed by another device. Furthermore, the decoding device 200 may implement any combination of the components of the present disclosure, and may perform any combination of the processes of the present disclosure.

[0209] The encoding method and the decoding method including the steps performed by each component of the encoding device 100 and the decoding device 200 of the present disclosure may be executed by any device or system. For example, part or all of the encoding method and the decoding method may be executed by a computer including a processor, a memory, an input / output circuit, etc. In this case, the encoding method and the decoding method may be executed by the computer executing a program for causing the computer to execute the encoding method and the decoding method.

[0210] Furthermore, the program or the bitstream may be recorded on a non-transitory computer-readable recording medium such as a CD-ROM.

[0211] An example of a program may be a bitstream. For example, a bitstream including an encoded three-dimensional mesh includes syntax elements for causing the decoding device 200 to decode the three-dimensional mesh. The bitstream then causes the decoding device 200 to decode the three-dimensional mesh according to the syntax elements included in the bitstream. Thus, the bitstream may play a role similar to that of a program.

[0212] The bitstream may be an encoded bitstream containing the encoded 3D mesh, or may be a multiplexed bitstream containing the encoded 3D mesh and other information.

[0213] Furthermore, each component of the encoding device 100 and the decoding device 200 may be configured with dedicated hardware, general-purpose hardware that executes the above-mentioned programs, or a combination of these. The general-purpose hardware may be configured with a memory in which the programs are recorded and a general-purpose processor that reads and executes the programs from the memory. Here, the memory may be a semiconductor memory or a hard disk, and the general-purpose processor may be a CPU.

[0214] Furthermore, the dedicated hardware may be configured with a memory, a dedicated processor, etc. For example, the dedicated processor may execute the encoding method and the decoding method by referring to a memory for recording data.

[0215] Furthermore, as described above, each component of the encoding device 100 and the decoding device 200 may be an electric circuit. These electric circuits may form a single electric circuit as a whole, or each may be a separate electric circuit. Furthermore, these electric circuits may correspond to dedicated hardware, or may correspond to general-purpose hardware that executes the above-mentioned programs, etc. Furthermore, the encoding device 100 and the decoding device 200 may be implemented as an integrated circuit.

[0216] Furthermore, the encoding device 100 may be a transmitting device that transmits the three-dimensional mesh, and the decoding device 200 may be a receiving device that receives the three-dimensional mesh.

[0217] Displacement Encoding and Decoding The following terminology is used here by way of example:

[0218] (1) Image An image is a data unit made up of a set of pixels, and includes a picture or a block smaller than a picture. Images include both moving images and still images.

[0219] (2) Picture A picture is a unit of image processing that is made up of a set of pixels, and is also called a frame or field.

[0220] (3) Block A block is a processing unit consisting of a specific number of pixels. The term "block" shown in the following example is also used. The shape of a block is not particularly limited. A block may be, for example, a rectangular shape of M x N pixels or a square shape of M x M pixels. A block may also be a triangular shape, a circular shape, or another shape. Examples of blocks are as follows:

[0221] Slice, tile, or brick CTU, superblock, or basic division unit VPDU, processing division unit for hardware CU, processing block unit, prediction block unit (PU), or orthogonal transform block unit (TU) Sub-block

[0222] (4) Pixel or Sample A pixel or sample is the smallest point of an image, in other words, the smallest unit. Pixels or samples include not only pixels at integer positions, but also pixels at sub-pixel positions generated based on pixels at integer positions.

[0223] (5) Pixel Value or Sample Value: A pixel value or sample value is a unique value of a pixel. The pixel value or sample value may include a luma value, a chroma value, or an RGB gradation level, and may also include a depth value or a binary value of 0 or 1.

[0224] (6) Flags A flag indicates one or more bits. A flag is, for example, a parameter or index represented by two or more bits. A flag may indicate not only a value represented by a binary number, but also a value represented by a number other than a binary number.

[0225] (7) Signal: A signal is something that is symbolized or coded to transmit information. A signal includes a discrete digital signal or a continuous analog signal.

[0226] (8) Stream or Bit Stream A stream or bit stream is a digital data sequence that indicates the flow of digital data. A stream or bit stream may be a single stream, or may be configured to include multiple streams with multiple layers. A stream or bit stream may be transmitted by serial communication using a single transmission path, or may be transmitted by packet communication using multiple transmission paths.

[0227] (9) Difference: For scalar quantities, difference can include simple difference (x - y) and difference calculations, such as absolute difference (|x - y|), squared difference (x^2 - y^2), square root difference (√(x - y)), weighted difference (ax - b, where a and b are constants), or offset difference (x - y + a, where a is an offset).

[0228] (10) Sum. For scalar quantities, sums can include simple sum (x + y) and addition operations. Sum can also include absolute sum (|x + y|), sum of squares (x^2 + y^2), square root of sum (√(x + y)), weighted sum (ax + by, where a and b are constants), or offset sum (x + y + a, where a is an offset).

[0229] (11) "Based on" The expression "based on something" means that something other than that "something" may be taken into consideration. Also, "based on" can be used both when a direct result is obtained and when a result is obtained through an intermediate result.

[0230] (12) "Used" or "Using" The phrases "something was used" or "used something" mean that something other than the "something" may be taken into consideration. The phrases "used" or "used" may be used both in cases where a direct result is obtained and in cases where a result is obtained via an intermediate result.

[0231] (13) Prohibition "Prohibit" can be rephrased as "not permitted." Also, "not prohibited / prohibited" or "permitted / permitted" does not necessarily mean "obligation."

[0232] (14) "Restriction" or "Limitation" "Restriction" or "Limitation" can be rephrased as "not permitted / not allowed" or "not permitted / permitted." Furthermore, "prohibited / not prohibited" or "not permitted / permitted" does not necessarily mean "obligation." Furthermore, what is prohibited quantitatively or qualitatively may be either partial or total.

[0233] (15) Chroma The term chroma is an adjective, represented by the symbols Cb or Cr, that indicates that a sample array or a single sample represents one of the two color-difference signals associated with a primary color. The term chroma is sometimes used instead of the term chrominance.

[0234] (16) Luma The term luma is an adjective, represented by the symbols or subscripts Y or L, that indicates that a sample array or a single sample represents a monochrome signal for a primary color. The term luma is sometimes used instead of the term luminance.

[0235] The encoding / decoding system of this embodiment will be described below.

[0236] A typical three-dimensional model (also called a 3D model) digitally represents an object so that a user can explore the model using zoom, pan, and rotation in all three dimensions while it is rendered over time. One way to construct such a representation is to build a 3D mesh using triangles. The model stores the positions of the triangle vertices, their connectivity to each other, and their associated attributes (such as normals or UV patches).

[0237] Storing all this information in uncompressed form requires a very large storage space and therefore a very large bandwidth for transmission. The triangles that form the mesh often have repeating patterns and similar properties, especially in temporal and spatial neighborhoods. These repetitions can be exploited to develop efficient encoding and decoding methods for storage and transmission. One such encoding and decoding method is Video-based Dynamic Mesh Coding (V-DMC).

[0238] 26 is a block diagram showing another example of the configuration of the encoding / decoding system according to this embodiment. As shown in FIG. 26, the encoding / decoding system includes an encoding device 100 and a decoding device 200.

[0239] The encoding / decoding system accepts input three-dimensional meshes (also called 3D meshes) in the form of three-dimensional coordinates of vertices (vertex information), connectivity (connection information) and associated attributes (attribute information), which may include texture maps as well as geometry.

[0240] The encoding device 100 takes an input 3D mesh (also referred to as an input 3D mesh or input mesh) in the form of three-dimensional coordinates of vertices, connectivity, and associated attributes. The encoding device 100 encodes all associated information into a stream. The stream may consist of a single bitstream or multiple bitstreams.

[0241] The network 300 transmits the stream generated by the encoding device to the decoding device 200. The network 300 may be the Internet, a wide area network (WAN), a local area network (LAN), or any combination thereof. Furthermore, the network 300 is not necessarily limited to a two-way communication network, but may also be a one-way communication network that transmits broadcast waves such as terrestrial digital broadcasting or satellite broadcasting. Instead of the network 300, a recording medium such as a digital versatile disc (DVD) or a blue-ray disc (BD) on which a stream is recorded may be used.

[0242] The stream is transmitted to a decoding device 200 via a network 300. The decoding device 200 decodes the bitstream and generates a 3D mesh using the 3D coordinates, connectivity, and associated attributes of the decoded vertices. The decoding device 200 outputs the generated 3D mesh (also referred to as an output 3D mesh or output mesh).

[0243] FIG. 27 is a diagram showing another example of the configuration of the encoding device 100.

[0244] As shown in FIG. 27, the encoding device 100 includes a preprocessor 1103 and a compressor 1106 .

[0245] The encoding device 100 reads an input mesh 1101 and an attribute map 1102 and passes them to a preprocessor 1103. The preprocessor 1103 processes the input mesh to extract a base mesh 1104 and displacement data 1105. The attribute map 1102, along with the extracted base mesh 1104 and displacement data 1105, are passed to a compressor 1106.

[0246] The compressor 1106 also compresses the base mesh 1104, the displacement data 1105, and the attribute map 1102 to generate a bitstream 1107. The compressor 1106 can transmit additional information to the decoding device 200 by further including metadata 1108 in the bitstream 1107.

[0247] FIG. 28 is a diagram showing another example of the configuration of the decoding device 200.

[0248] As shown in FIG. 28, the decoding device 200 includes a decompressor 2102 and a post-processor 2106 .

[0249] The decoding device 200 reads a bitstream 2101 and passes it to a decompressor 2102. The decompressor 2102 decompresses a base mesh 2103, displacement data 2104, and an attribute map 2108 from the bitstream 2101 and passes them to a post-processor 2106. An example of the displacement data 2104 is a displacement vector.

[0250] The post-processor 2106 also processes the base mesh 2103 according to the displacement data 2104 and the attribute map 2108 to generate an output mesh 2107. The post-processor 2106 may further use information from the metadata 2105 to generate the output mesh 2107.

[0251] FIG. 29 is a block diagram showing yet another example configuration of the encoding device 100 according to this embodiment.

[0252] In this example, the encoding device 100 comprises a volumetric capturer 511, a projector 512, a base mesh encoder 513, a displacement encoder 514, an attribute encoder 515, and optionally one or more other type encoders 516.

[0253] The volumetric capturer 511 captures content and outputs the captured content to the projector 512 .

[0254] The projector 512 projects the content onto a 3D mesh frame containing vertex geometry coordinates, texture coordinates, and connectivity data. The data is output to a base mesh encoder 513, a displacement encoder 514, an attribute encoder 515, and optionally one or more other type encoders 516. Each encoder compresses the data into a bitstream.

[0255] FIG. 30 is a block diagram showing yet another example configuration of the decoding device 200 according to this embodiment.

[0256] In this example, the decoding device 200 comprises a base mesh decoder 613 , a displacement decoder 614 , an attribute decoder 615 , one or more other type decoders 616 , and a 3D reconstructor 617 .

[0257] The bitstream is sent to a base mesh decoder 613, a displacement decoder 614, an attribute decoder 615, and optionally one or more other type decoders 616. These decoders decode the bitstream to generate decoded data including vertex geometry coordinates, texture coordinates, and connectivity data. The decoded data is then sent to a 3D reconstructor 617, which reconstructs a 3D mesh frame.

[0258] The encoding process performed by the encoding device 100 will be described in detail below.

[0259] Fig. 31 is a flow diagram showing the processing of the encoding device 100. Fig. 32 is an explanatory diagram conceptually showing the encoding of mesh frames. The processing of the encoding device 100 will be described with reference to Figs. 31 and 32.

[0260] In step S101, the encoding device 100 reads a 3D mesh frame, which is an input mesh frame, and its attributes. The input mesh frame is a mesh frame input to the encoding device 100. An example of the 3D mesh frame that is an input mesh frame is shown as mesh frame 1301 (see FIG. 32 ).

[0261] In step S102, the encoding device 100 performs a decimation process on the input mesh frame read in step S101 to generate a base mesh frame having fewer vertices than the input mesh frame. The base mesh frame generated by decimating the mesh frame 1301 is shown as a base mesh frame 1302 (see FIG. 32).

[0262] In step S103, the encoding device 100 calculates displacement information that the decoding device 200 uses to reconstruct a mesh frame. The displacement information corresponds to a displacement vector directed from a vertex of the base mesh frame generated in step S102 to a vertex of the input mesh frame. One method for calculating the displacement information is to subtract the coordinates of the vertex of the base mesh frame from the coordinates of the vertex of the input mesh frame. The displacement information calculated from the mesh frame 1301 and the base mesh frame 1302 is shown as displacement information 1303 (see FIG. 32). The displacement information 1303 is in vector format, in other words, expressed as a displacement vector.

[0263] In step S104, the encoding device 100 encodes the base mesh frame generated in step S102, the displacement information generated in step S103, and the attributes of the input mesh frame into a bitstream (corresponding to a compressed bitstream). An example of the bitstream is shown as bitstream 1304 (see FIG. 32).

[0264] Specifically, the bitstream 1304 includes vertex coordinates and connectivity information for vertices A, C, E, and F, displacement information, a video bitstream including texture data, and a compressed attribute map (see FIG. 32). The displacement information includes displacement information for displacing vertices based on vertex coordinates obtained from the subdivided base mesh frame. The compressed attribute map is texture coordinates for applying texture data to a mesh frame reconstructed using the base mesh frame and the displacement information.

[0265] The decoding process performed by the decoding device 200 will be described in detail below.

[0266] Fig. 33 is a flow diagram showing the processing of the decoding device 200. Fig. 34 is an explanatory diagram conceptually showing the decoding of a 3D mesh. The processing of the decoding device 200 will be described with reference to Figs. 33 and 34.

[0267] In step S201, the decoding device 200 decodes a base mesh frame and attributes from a bitstream (corresponding to a compressed bitstream). An example of the decoded base mesh frame (corresponding to a decoded base mesh frame) is shown as a decoded base mesh frame 2301 (see FIG. 34).

[0268] In step S202, the decoding device 200 generates subdivided vertices by performing a subdivision process on the base mesh frame decoded in step S201. An example of a base mesh frame including subdivided vertices is shown as base mesh frame 2302 (see FIG. 34).

[0269] In step S203, the decoding device 200 decodes the disparity information from the bitstream (corresponding to the compressed bitstream). An example of the decoded disparity information is shown as disparity information 2303 (see FIG. 34). The disparity information 2303 is in vector format, in other words, expressed as a disparity vector.

[0270] In step S204, the decoding device 200 reconstructs the shape of the mesh frame by moving the vertices of the base mesh frame, including the subdivided vertices, to new positions using the displacement information, and then restores the mesh frame by applying attribute information. An example of the attribute is texture. An example of the reconstructed mesh frame is shown as mesh frame 2304 (see FIG. 34 ).

[0271] FIG. 35 is a block diagram showing an example of the configuration of a decoding device according to this embodiment.

[0272] FIG. 35 shows an example of a block diagram of a general intra-decoding system.

[0273] The decoding device shown in FIG. 35 comprises a demultiplexer 1231, a switch 1232, a static mesh decoder 1233, a mesh buffer 1234, a motion decoder 1235, a base mesh reconstructor 1236, an inverse quantizer 1237, a video decoder 1238, an image unpacker 1239, an inverse quantizer 1240, an inverse wavelet transformer 1241, a reconstructor 1242, a video decoder 1243, and a color converter 1244.

[0274] The demultiplexer 1231 receives the compressed bitstream and separates it into compressed data for the base mesh, video containing displacement data (also called displacement bitstream), and video containing attribute data (also called attribute bitstream). The compressed data for the base mesh is passed to a switch 1232. The switch 1232 determines whether to perform intra-decoding or inter-decoding based on parameters in the bitstream.

[0275] If an intra-decoding process is selected, the bitstream is passed to a static mesh decoder 1233, which generates a quantized base mesh. The static mesh decoder 1233 is, for example, a decoder that uses an edge breaker algorithm to decode 3D mesh data. The static mesh decoder 1233 generates a quantized base mesh from the bitstream. The quantized base mesh generated by the static mesh decoder 1233 is stored in a mesh buffer 1234 for reference when an inter-decoding process is selected.

[0276] If inter-decoding is selected, switch 1232 passes compressed data for the base mesh to motion decoder 1235. Motion decoder 1235 receives a previously decoded quantized base mesh and decodes motion data representing the differences in vertex coordinates between the quantized base mesh stored in mesh buffer 1234 and the current quantized base mesh. The motion data and the quantized base mesh stored in mesh buffer 1234 are used by base mesh reconstructor 1236 to reconstruct the current quantized base mesh. The quantized base mesh resulting from either inter-decoding or intra-decoding is passed to inverse quantizer 1237 to obtain a decoded base mesh.

[0277] The video containing the displacement data is passed to a video decoder 1238, since the bitstream contains the displacement data in an image format with two chroma information and one luma information. The video decoder 1238 decodes the data using a video frame decompression method. Alternatively, the displacement data can be decoded using an arithmetic decoder. This decompressed data is passed to an image unpacker 1239, which extracts wavelet coefficients associated with each vertex from the image-format decompressed data. An inverse quantizer 1240 dequantizes the quantized wavelet coefficients into the three components associated with each vertex. An inverse wavelet transformer 1241 inversely transforms the result to finally obtain decoded displacement data. The decoded displacement data and the decoded base mesh are passed to a reconstructor 1242, which performs edge refinement on the decoded base mesh and displaces the vertices using the decoded displacement data to obtain a decoded mesh.

[0278] The video including the attribute data is passed to another video decoder 1243 to obtain a decoded attribute bitstream, which is further processed in a color converter 1244 for color space and color format conversion to obtain a decoded attribute map.

[0279] FIG. 36 is a block diagram showing an example of the configuration of a decoding device according to this embodiment.

[0280] FIG. 36 illustrates an example of a reconstructor that obtains a decoded 3D mesh 1256 from a decoded base mesh 1251 and decoded displacement data 1254.

[0281] The decoded base mesh 1251 is passed to a subdivider 1252 .

[0282] The subdivision unit 1252 subdivides any two connected vertices in the entire 3D mesh by adding a new vertex between them. This process can be repeated several times to include vertices created in previous subdivision steps to generate a predefined number of vertices. Each subdivision iteration across the 3D mesh generates a new level of detail (LoD). The subdivided mesh 1253 and the decoded displacement data 1254 are passed to a displacer 1255. The displacer 1255 generates a decoded 3D mesh 1256 by moving each vertex to a new position according to the corresponding displacement data.

[0283] The subdivision is described below and is performed by a subdivider (specifically subdivider 1206 or subdivider 2204).

[0284] FIG. 37 is an explanatory diagram showing an example of subdivision.

[0285] The base mesh shown in FIG. 37(a) includes vertices A, B, and C and connectivity information indicating their connectivity.

[0286] 37(b) shows a mesh generated by the first subdivision, in other words, the mesh after the first subdivision. In the first subdivision, the subdivider generates vertices D, E, and F and connectivity information indicating their connectivity. The mesh generated by the subdivider is also referred to as LoD1 or first LoD.

[0287] Vertex D of the mesh after the first subdivision is a vertex generated by subdivision based on vertices A and B. Similarly, vertex E is a vertex generated by subdivision based on vertices B and C. Vertex F is a vertex generated by subdivision based on vertices A and C.

[0288] As an example, vertex D may be the midpoint of line segment AB (in other words, side AB) connecting vertices A and B that were the basis for its generation. Similarly, vertex E may be the midpoint of line segment AC. Vertex F may be the midpoint of line segment BC.

[0289] 37(c) shows the mesh generated by the second subdivision, i.e., the mesh after the second subdivision. In the second subdivision, the subdivider generates vertices G, H, I, J, K, L, M, N, and O and connectivity information indicating their connectivity. The mesh generated by the subdivider is also called LoD2 or second LoD.

[0290] Vertex G of the mesh after the second subdivision is a vertex generated by subdivision based on vertices A and D. Similarly, vertex H is a vertex generated by subdivision based on vertices A and E. Vertex I is a vertex generated by subdivision based on vertices B and D. Vertex J is a vertex generated by subdivision based on vertices D and F. Vertex K is a vertex generated by subdivision based on vertices E and F. Vertex L is a vertex generated by subdivision based on vertices C and E. Vertex M is a vertex generated by subdivision based on vertices B and F. Vertex N is a vertex generated by subdivision based on vertices C and F. Vertex O is a vertex generated by subdivision based on vertices D and E.

[0291] As an example, vertex G may be the midpoint of line segment AD (in other words, side AD) connecting vertices A and D, which were the source of its generation. Similarly, vertex H may be the midpoint of line segment AE. vertex I may be the midpoint of line segment BD. vertex J may be the midpoint of line segment DF. vertex K may be the midpoint of line segment EF. vertex L may be the midpoint of line segment CE. vertex M may be the midpoint of line segment BF. vertex N may be the midpoint of line segment CF. vertex O may be the midpoint of line segment DE.

[0292] The displacement of vertices will be described below with reference to Figures 38 and 39. The displacement of vertices is performed by the reconstructor 2209.

[0293] Fig. 38 is an explanatory diagram showing an example of displacement of vertices after subdivision, and Fig. 39 is an explanatory diagram showing an example of vertices of an original mesh.

[0294] The base mesh shown in FIG. 38(a) includes vertices A, B, C, and Z and connectivity information indicating their connectivity.

[0295] 38(b) shows a mesh generated by the first subdivision, in other words, a mesh after the first subdivision (i.e., the first LoD). In the first subdivision, the subdivider generates vertices S, T, U, X, or Y and connectivity information indicating their connectivity. The vertices S, T, U, X, or Y are similar to the vertices D, E, and F shown in FIG. 37(b).

[0296] 38(c) shows a mesh generated by the second subdivision, in other words, a mesh after the second subdivision (i.e., the second LoD). In the second subdivision, the subdivider generates vertices D, E, F, G, and H and connectivity information indicating their connectivity. Vertices D, E, F, G, and H are the same as vertices G, H, I, J, K, L, M, N, and O shown in FIG. 37(c).

[0297] Figure 38(d) shows a mesh including the vertices after they have been displaced after subdivision, with vertices A, B, C, D, E, F, G, H, S, T, U, X, Y, and Z shown in Figure 38(d) being located at positions displaced using displacement information from the positions of the vertices shown in Figure 38(c).

[0298] The original mesh shown in FIG. 39 is an example of the mesh input to the encoding device 100, that is, the mesh before encoding.

[0299] The mesh shown in Fig. 38 has a shape similar to that of the original mesh shown in Fig. 39. The displacement information is generated by the displacement vector calculator 1207 of the encoding device 100 as information indicating the displacement from the vertices of the base mesh to the vertices of the original mesh, and therefore, by reconstructing the mesh using the displacement information thus generated, a mesh having a shape similar to that of the original mesh is generated.

[0300] The decoding device 200 can output the mesh shown in FIG.

[0301] Next, the division of a mesh into sub-meshes will be described with reference to FIGS.

[0302] A mesh can be divided into smaller parts and coded separately, with the mesh vertices being divided in such a way that the coordinates and connectivity of the vertices in each part can be coded independently.

[0303] Fig. 40 is an explanatory diagram showing an example of a mesh, and Fig. 41 is an explanatory diagram showing an example of dividing a mesh into sub-meshes.

[0304] The mesh shown in FIG. 40 is the original mesh, which is sometimes called a full mesh in contrast to a sub-mesh.

[0305] Figure 41 shows how the full mesh shown in Figure 40 is divided into two sub-meshes. For vertices A, B, and C of the full mesh (see Figure 40), vertex A is duplicated to vertices A1 and A2, vertex B is duplicated to vertices B1 and B2, and vertex C is duplicated to vertices C1 and C2, thereby creating two sub-meshes (i.e., a first sub-mesh and a second sub-mesh) from the full mesh. The first sub-mesh and the second sub-mesh are each independently decodable meshes.

[0306] Packing of displacement information into image frames will be described below with reference to FIGS.

[0307] 42, 43 and 44 are explanatory diagrams showing examples of packing of displacement information into image frames. Note that image frames can also be called video frames.

[0308] The vertex displacement data is encoded as image frame data by being mapped to each component of a YUV format image frame (i.e., each of the Y component (Y Plane), U component (U Plane), and V component (V Plane)). This case will be described below as an example. As another example, the vertex displacement data may be encoded as image frame data by being mapped to each component of an RGB format image frame (each of the R component, G component, and B component).

[0309] The decoding device 200 can use an image encoding module to extract the displacement data. The displacement data can be in the form of X, Y, or Z components in a global coordinate system (e.g., a Cartesian coordinate system), or normal, tangential, or both tangential components in a local coordinate system. Methods for mapping the displacement data to an image frame include the following:

[0310] For example, in the first method, the displacement data is arranged in the image frame in scan order, and an example of packing the displacement data in this case is shown in Figure 42. The displacement data is directly mapped onto the image frame according to a predefined scan order.

[0311] Note that since an image frame has a fixed height and width, it may happen that the displacement data does not fit perfectly in the frame, in which case the remaining part of the image frame is padded with padding data (see Figure 42).

[0312] For example, in the second method, the displacement data is separated into multiple LoDs and mapped to the Y, U, and V components of the image frame. An example of packing of the displacement data in this case is shown in Figure 43. Here, the displacement data of the image frame of the next LoD starts immediately after the displacement data of the previous LoD ends. As in the first method, if the displacement data does not fit exactly into the image frame, padding is performed at the end of the image frame (see Figure 43).

[0313] For example, in the third method, displacement data corresponding to the LoD is mapped to the Y component, U component, and V component of the image frame in a manner different from that in the second method. An example of packing of the displacement data in this case is shown in Figure 44. In this way, each LoD can be decoded independently. In the third method, middle padding is performed on the displacement data of each LoD, and CTU alignment is performed together with padding at the end of the video frame (see Figure 44).

[0314] <Other Examples> Although the aspects of the encoding device and the decoding device have been described above according to the embodiments, the aspects of the encoding device and the decoding device are not limited to the embodiments. Modifications that a person skilled in the art can conceive of may be applied to the embodiments, and multiple components in the embodiments may be combined in any manner.

[0315] For example, a process performed by a specific component in the embodiment may be performed by another component instead of the specific component. Also, the order of multiple processes may be changed, or multiple processes may be performed in parallel.

[0316] Furthermore, as described above, at least some of the configurations of the present disclosure may be implemented as an integrated circuit. At least some of the processes of the present disclosure may be used as an encoding method or a decoding method. A program for causing a computer to execute the encoding method or the decoding method may be used. A non-transitory computer-readable recording medium on which the program is recorded may be used. A bitstream for causing a decoding device to perform a decoding process may be used.

[0317] Furthermore, at least some of the configurations and processes of the present disclosure may be used as a transmitting device, a receiving device, a transmitting method, or a receiving method. A program for causing a computer to execute the transmitting method or the receiving method may be used. Furthermore, a non-transitory computer-readable recording medium on which the program is recorded may be used.

[0318] In the above embodiments, each component may be configured with dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory. Here, the software that realizes the encoding device and the like in the above embodiments is the following program.

[0319] In other words, the software is a program that causes a computer to execute an encoding method that obtains a first frame to be encoded, encodes first data having one or more layers contained in the first frame by referencing second data having one or more layers contained in a second frame, and when encoding the first data, determines one of a plurality of processes for each of the one or more layers contained in the first data using a value indicating the layer and a value related to the one or more layers contained in the second data, and executes the determined one process on the first data of that layer.

[0320] The software for realizing the decoding device and the like according to the above-described embodiment is the following program.

[0321] In other words, the software is a program that causes a computer to execute a decoding method in which it obtains a first frame to be decoded, decodes first data having one or more layers contained in the first frame by referring to second data having one or more layers contained in a second frame, and when decoding the first data, determines one of a plurality of processes for each of the one or more layers contained in the first data using a value indicating the layer and a value related to the one or more layers contained in the second data, and executes the determined one process on the first data of that layer.

[0322] Next, examples of attribute information in mesh data will be described. Here, "device" refers to an encoding device or a decoding device. Figures 45 to 47 are diagrams for explaining examples of attribute information in mesh data according to an embodiment.

[0323] As shown in Fig. 45, multiple three-dimensional points G (vertices) in mesh data are projected onto a certain plane X, and the two-dimensional coordinates on plane X after the projection are sometimes set as attribute information of these three-dimensional points. For example, if the two-dimensional coordinates after projection of a three-dimensional point (x, y, z) onto plane X are (u, v), the two-dimensional coordinates (u, v) are set as attribute information of the three-dimensional point (x, y, z). Fig. 45 shows an example in which a set 1501 of vertex coordinates of a three-dimensional mesh represented by mesh data is projected onto plane X and converted into a set 1502 of two-dimensional coordinates on plane X.

[0324] Furthermore, as shown in FIG. 46 , multiple 3D points in mesh data are projected onto plane X, and plane information to which the two-dimensional coordinates on plane X after the projection belong is sometimes set as attribute information of the 3D point. For example, if the two-dimensional coordinates (u, v) after the projection of a 3D point (x, y, z) onto plane X belong to plane information face0, the plane information face0 is set as attribute information of the 3D point (x, y, z). In this case, by setting (u, v) = (face0, 0), the 3D point (x, y, z) and the plane information face0 are associated as UV coordinates. This method allows the relationship between the 3D point (x, y, z) and the plane information face0 to be defined using the same mechanism as UV coordinates.

[0325] Furthermore, when projecting 3D points with connectivity onto a 2D image, attribute information for the 3D points can be added by setting the same plane information. For example, the following processing can be performed.

[0326] In the case of Figure 46, face0 is set as attribute information for a 3D point that has connectivity between 3D point G and 3D point G. Similarly, face1 is set as attribute information for a 3D point that has connectivity between 3D point G' and 3D point G'. Using this method, the decoding device can obtain the set plane information by decoding the attribute information of the 3D points. Then, using the plane information, it can obtain, for example, related projection information.

[0327] Furthermore, as shown in Fig. 47, a 3D point in mesh data or a normal vector corresponding to a plane to which the 3D point belongs may be set as attribute information. For example, the device can set a normal vector (nx, ny, nz) for a 3D point (x, y, z) as attribute information. It is also possible to set a 3D point G or a normal vector (nx, ny, nz) corresponding to a plane to which the 3D point G belongs as attribute information for G.

[0328] Note that the attribute information in the mesh data is not limited to this. The attribute information can include any information, such as color information of the three-dimensional point, reflectance, or the group ID to which the three-dimensional point belongs. The attribute information may also include information with various dimensions, such as two-dimensional information such as UV coordinates, one-dimensional information such as plane information face0, and three-dimensional information such as normal vectors.

[0329] Furthermore, it is possible to assign multiple pieces of attribute information to a single 3D point. For example, the device can assign multiple pieces of attribute information, such as UV coordinates and a normal vector, to a 3D point (x, y, z). Also, there may be 3D points to which no attribute information is assigned. In this way, by assigning various types of attribute information to 3D points, mesh data with higher expressiveness can be generated.

[0330] In this embodiment, an example of a method for encoding or decoding attribute information by a device is shown. The example shown here describes the case where one piece of attribute information assigned to a three-dimensional point is encoded or decoded, but is not limited to this. For example, if two or more pieces of attribute information are assigned to a three-dimensional point, it is also possible to encode or decode each piece of attribute information individually using this embodiment. This allows the attribute information to be efficiently encoded or decoded even for a three-dimensional point to which multiple pieces of attribute information are assigned.

[0331] In this embodiment, an example is shown in which uv coordinates are used as attribute information, but other attribute information can also be used. Figures 48 to 53 are diagrams for explaining specific examples of methods for encoding attribute information.

[0332] First, as shown in FIG. 48, the device selects a first initial 3D point v0 and encodes its UV coordinates without prediction. For example, in the following case, the device may encode the value of (u0, v0) as is. Note that the order in which the 3D points for which attribute information is encoded may be arbitrary; for example, the device may select them in the same order as when the position information of the 3D points was encoded. This allows the device to apply predictive encoding while reducing the amount of processing required to select 3D points.

[0333] Next, as shown in FIG. 49, the device selects a 3D point v1. If the 3D point v1 has connectivity with an already-encoded 3D point v, i.e., if there is an edge, the device can differentially encode the UV coordinates of the 3D point v1 using the UV coordinates of the 3D point v0 as predicted values. For example, in the following case, the device can encode (u1 - u0, v1 - v0). This allows the amount of coding to be reduced when the UV coordinate values ​​of the 3D points v0 and v1 are close. Note that this method of predictive encoding using the UV coordinates of a single 3D point with connectivity is called "coarse prediction."

[0334] Next, as shown in FIG. 50, the device selects 3D point v2. If 3D point v2 has connectivity with already-encoded 3D points v0 and v1, i.e., if 3D point v2 has edges with 3D point v1 and 3D point v2 has edges with 3D point v0, the device can generate a predicted value using the UV coordinates of 3D point v0 and 3D point v1, and differentially encode the UV coordinates of 3D point v2. For example, in the following case, the device can encode (u2 - predx, v2 - predy). This allows for highly accurate generation of a predicted value using 3D point v0 and 3D point v1, thereby reducing the amount of coding. Note that this method of predictive encoding using the UV coordinates of two or more connected 3D points is called "fine prediction."

[0335] Next, the device selects 3D point v3 as shown in Fig. 51. If 3D point v3 has connectivity with already-encoded 3D point v1 and 3D point v2, i.e., if 3D point v3 has edges with 3D point v1 and 3D point v3 with 3D point v2, the device can generate predicted values ​​by fine prediction using the UV coordinates of 3D point v1 and 3D point v2, and differentially encode the UV coordinates of 3D point v3.

[0336] Next, as shown in FIG. 52 , the device selects a 3D point v4. If the 3D point v4 has connectivity with the already-encoded 3D points v0, v1, and v3, i.e., if the 3D points v4 and v3, and the 3D points v4 and v1, and v4 and v3 have edges, the device can generate a predicted value by fine prediction using the UV coordinates of the 3D points v0, v1, and v3, and differentially encode the UV coordinates of the 3D point v4. In this case, for example, the device may calculate a fine predicted value 1 using the UV coordinates of the 3D points v0 and v1, calculate a fine predicted value 2 using the UV coordinates of the 3D points v1 and v3, and use the average of the fine predicted values ​​1 and 2 as the predicted value of the UV coordinate of the 3D point v4. In this way, if two or more fine predicted values ​​can be calculated, using the average as the predicted value can improve prediction accuracy and reduce the amount of coding.

[0337] Next, as shown in Figure 53, the device selects 3D point v5. If 3D point v5 has no connectivity with any of the previously encoded 3D points, the device selects 3D point v5 as the second initial 3D point and can directly encode the value of (u5, v5) without prediction, as in the first process. Similarly, the device can reduce the amount of code for the UV coordinates by applying predictive coding using each process thereafter.

[0338] Next, examples of prediction modes for encoding attribute information will be described.

[0339] First, the no-prediction mode will be described. The no-prediction mode is an example of a first prediction mode. The no-prediction mode is applied when the 3D point to be coded is an initial 3D point. Here, the initial 3D point refers to a 3D point to be processed that has already been coded and does not have any other coded 3D point that has connectivity with that 3D point.

[0340] In the no-prediction mode, the device can directly encode the attribute information values. For example, if the uv coordinates are (100, 60), the device can encode the u value as 100 and the v value as 60. The device may assume fixed-length encoding using n bits (n is an integer value greater than or equal to 0). For example, if the uv coordinates are (100, 60), the device can perform fixed-length encoding of each component using 7 bits.

[0341] When a device applies n-bit fixed-length coding, the device may add a value n to the header of the bitstream, so that a decoding device can correctly decode the attribute information of the initial 3D point coded in n-bit fixed length by decoding the value n of the header.

[0342] The device may change the bit length used for fixed-length encoding for each component of the attribute information. For example, if the u and v coordinates are (100, 60), the device can reduce the amount of code by applying 7-bit fixed-length encoding to u and 6-bit fixed-length encoding to v. In this case, the device may add the bit lengths used for the fixed-length encoding of each component to the header of the bitstream. This allows the decoding device to correctly decode the attribute information of the initial 3D point that was encoded using different bit lengths for each component by decoding the bit length for each component in the header.

[0343] The device can set the bit length of each component to 0 bits. A 0-bit value may indicate that the component is not included in the bitstream. If the bit length added to the header is 0 bits, the decoding device can determine that the component is not included in the bitstream and estimate the value of the component as 0. For example, if the attribute information includes a component whose value is always 0, the device can reduce the amount of code by setting the bit length of the fixed-length coding of the component to 0 bits and not encoding the value 0. Furthermore, if the bit length added to the header is 0 bits, the decoding device can determine that the component is not included in the bitstream and estimate the value of the component as 0.

[0344] For example, as described with reference to FIG. 46 , the device may project a three-dimensional point G(x, y, z) onto the xz plane, and use the plane face0 to which the two-dimensional coordinates (u, v) on the xy plane after projection belong as attribute information for G. In this case, the device can associate G with face0 as uv coordinates by setting (u, v) = (face0, 0). In this case, because v is always a value of 0, the device sets the bit length of the fixed-length coding of v to 0 bits, and does not include the value 0 for v in the bitstream, thereby reducing the amount of code.

[0345] The device may encode the attribute information of the initial 3D points using variable-length coding such as unary coding or exponential golomb coding. The device may also reduce the amount of code by assigning a context to each bit after binarization and applying arithmetic coding.

[0346] Note that the no-prediction mode does not necessarily need to be applied to the initial 3D points, and the device may apply other prediction methods. For example, the device may calculate a representative value of the initial 3D points and predictively encode the attribute information of the initial 3D points using the representative value as a predicted value. The representative value may be, for example, the average value of the initial 3D points. In this case, the device may add the representative value to the header, allowing the decoding device to decode the header, obtain the predicted value of the initial 3D points, and correctly decode the data. Furthermore, the device may use the attribute information of the initial 3D points that was encoded or decoded immediately before in the encoding or decoding order as the predicted value of the attribute information of the initial 3D points. This allows the device to reduce the amount of code by predictively encoding the attribute information of the initial 3D points.

[0347] Next, the coarse prediction mode will be described. The coarse prediction mode is an example of a second prediction mode. The coarse prediction mode is applied when, among the coded 3D points, there is one 3D point that has connectivity with the 3D point to be coded.

[0348] When applying coarse prediction, the device can use attribute information of one 3D point that has been coded and has connectivity as a predicted value and encode the prediction residual. Note that, since the tendency of the prediction residual may differ from that in fine prediction, the device may use different methods for encoding or decoding the prediction residual in coarse prediction and fine prediction.

[0349] For example, the device may binarize the prediction residual using unary coding, exponential golomb coding, or the like, assign a context to each bit, and use different contexts for coarse prediction and fine prediction when applying arithmetic coding. This allows the amount of coding to be reduced even when the trends in the values ​​of the prediction residual differ between coarse prediction and fine prediction. Furthermore, the device may further reduce the amount of coding by using different binarization methods for coarse prediction and fine prediction.

[0350] Next, the fine prediction mode will be described. The fine prediction mode is an example of a third prediction mode. The fine prediction mode is applied when, among the coded 3D points, there are two or more 3D points that have connectivity with the 3D point to be coded.

[0351] When applying fine prediction, the device can calculate a predicted value using attribute information of two or more 3D points that have already been coded and have connectivity, and encode the prediction residual. Note that when a 3D point having attribute information to be coded belongs to the vertices of two or more triangles composed of coded 3D points (such as 3D point v4 in FIG. 52), the device may be able to generate two or more fine predicted values ​​using two or more triangles. In this case, the device may use the average value of two or more fine predicted values ​​as the predicted value (average fine prediction).

[0352] For example, if the device can generate count fine predicted values, the device may calculate the sum of the count fine predicted values, divide that sum by count to calculate an average fine predicted value, and use that average fine predicted value as the predicted value of the attribute information to be coded. This allows the device to improve the accuracy of the predicted values ​​obtained by fine prediction and reduce the amount of coding.

[0353] The device may also limit the value of count to a power of 2. For example, by limiting the count of fine predictions used in the average fine prediction to a power of 2, such as 1, 2, 4, or 8, the device can use a right bit shift instead of division when calculating the average value, thereby reducing the amount of calculation. For example, by applying a 1-bit right shift when count = 2 and a 2-bit right shift when count = 4, the average fine prediction value can be generated without using division.

[0354] In this case, average fine prediction using a count that is not a power of two may be prohibited, and the device may omit its calculation. This allows the device to reduce the amount of processing required to calculate the average fine prediction value. The device may also limit the number of counts used to calculate the average fine prediction. For example, the device may set a fineMaxcount indicating the maximum value of count, and limit the number of fine predictions used to calculate the average fine prediction value to fineMaxcount or less. This allows the device to reduce the amount of processing required to calculate the average fine prediction.

[0355] The device may add fineMaxcount to the header of the bitstream, so that the decoding device can calculate the average fine prediction value using the same maximum count as the encoding device by decoding the fineMaxcount in the header, and can correctly decode the bitstream.

[0356] Furthermore, by setting fineMaxcount = 1, the device can limit the number of fine predictions used to calculate the average fine prediction value to 1, effectively eliminating the need to calculate the average value. This reduces the amount of processing required to calculate the average value and prevents decimal points from appearing when calculating the average value.

[0357] Furthermore, the device can set fineMaxcount = 0 to prevent fine prediction. In this case, (1) no prediction mode or (2) coarse prediction mode, etc., is used as the prediction mode. For example, if the attribute information of all 3D points following the initial 3D point is the same, the device can apply no prediction mode to the initial 3D point and apply coarse prediction to subsequent 3D points, thereby always keeping the prediction residual at 0, and there are cases where fine prediction does not need to be applied. In such cases, the device can set fineMaxcount = 0 to prevent fine prediction from being applied, thereby reducing the amount of processing required for encoding or decoding.

[0358] Furthermore, if there is additional information required for fine prediction, the device sets fineMaxcount=0 and does not apply fine prediction, thereby reducing the additional information and the amount of code.

[0359] The above predictive coding can also be applied to predictive decoding in the same way. In other words, by replacing the above "coding" with "decoding," the above explanation of predictive coding can be used as an explanation of predictive decoding.

[0360] FIG. 54 is a diagram showing an example of a flowchart for encoding attribute information of a three-dimensional point according to this embodiment.

[0361] The encoding device obtains a plurality of 3D points (vertices) of a 3D mesh (S1501).

[0362] The encoding device determines whether there are any 3D points for which attribute information has not been coded (S1502). If there are any 3D points for which attribute information has not been coded (Yes in S1502), the encoding device proceeds to step S1503; if not (No in S1502), the encoding device ends the process.

[0363] The encoding device selects a 3D point to be encoded from among the plurality of 3D points that have not yet been encoded (S1503).

[0364] The encoding device determines whether there are two or more encoded 3D points that have connectivity with the 3D point to be encoded (S1504). If there are two or more encoded 3D points that have connectivity with the 3D point to be encoded (Yes in S1504), the encoding device proceeds to step S1505, and if not (No in S1504), the encoding device proceeds to step S1508.

[0365] The encoding device calculates a predicted value using attribute information of two or more 3D points (S1505).

[0366] The encoding device determines whether count is greater than 0 (S1506). If count is greater than 0 (Yes in S1506), the encoding device proceeds to step S1507, and if not (No in S1506), the encoding device proceeds to step S1508.

[0367] The encoding device encodes the attribute information of the 3D point to be encoded using the calculated predicted value (S1507). That is, the encoding device calculates a residual between the attribute information of the 3D point to be encoded and a predicted value calculated using attribute information of two or more 3D points, and encodes the calculated residual. Step S1507 is predictive encoding in a third prediction mode.

[0368] The encoding device determines whether there is one encoded 3D point that has connectivity with the 3D point to be encoded (S1508). If there is one encoded 3D point that has connectivity with the 3D point to be encoded (Yes in S1508), the encoding device proceeds to step S1509, and if not (No in S1508), the encoding device proceeds to step S1510.

[0369] The encoding device encodes the attribute information of the 3D point to be encoded using the attribute information of one 3D point as a predicted value (S1509). That is, the encoding device calculates a residual between the attribute information of the 3D point to be encoded and the attribute information of one 3D point, and encodes the calculated residual. Step S1509 is predictive encoding in the second prediction mode.

[0370] The encoding device encodes the attribute information of the 3D point to be encoded in a prediction-free mode (S1510). That is, the encoding device encodes the attribute information of the 3D point to be encoded without using a predicted value. Step S1510 is predictive encoding in a first prediction mode.

[0371] FIG. 55 is a flowchart showing an example of a calculation process of a predicted value in the third prediction mode according to this embodiment.

[0372] The encoding device sets count to 0 (S1511).

[0373] The encoding device determines whether count is less than fineMaxcount (S1512). If count is less than fineMaxcount (Yes in S1512), the encoding device proceeds to step S1513; if not (No in S1512), the encoding device proceeds to step S1515. Note that the encoding device can turn off fine prediction by setting fineMaxcount=0. The encoding device may also add the set value of fineMaxcount to the bitstream.

[0374] Furthermore, if the value of fineMaxcount decoded from the bitstream is 0, the decoder may decode the attribute information without applying fine prediction.

[0375] The encoding device calculates the count-th fine predicted value (S1513).

[0376] The encoding device adds 1 to count to update it (S1514), and returns to step S1512.

[0377] The encoding device determines whether or not count is greater than 0 (S1515). If count is greater than 0 (Yes in S1515), the encoding device proceeds to step S1516, and if not (No in S1515), the encoding device proceeds to step S1517.

[0378] The encoding device calculates an average fine predicted value (S1516). That is, the encoding device calculates the average of one or more fine predicted values ​​as the average fine predicted value. Note that, when there is one fine predicted value, the encoding device may calculate one fine predicted value as the average fine predicted value. Note that the one or more fine predicted values ​​are the count fine predicted values ​​obtained in steps S1511 to S1514.

[0379] The encoding device outputs count (S1517).

[0380] FIG. 56 is a diagram showing an example of a flowchart for decoding attribute information of a 3D point according to this embodiment.

[0381] The decoding device obtains a plurality of 3D points (vertices) of the 3D mesh (S1521).

[0382] The decoding device determines whether there are any 3D points for which attribute information has not been decoded (S1522). If there are any 3D points for which attribute information has not been decoded (Yes in S1522), the decoding device proceeds to step S1523; if not (No in S1522), the decoding device ends the process.

[0383] The decoding device selects a 3D point to be decoded from among the plurality of 3D points that have not been decoded (S1523).

[0384] The decoding device determines whether there are two or more decoded 3D points that have connectivity with the 3D point to be decoded (S1524). If there are two or more decoded 3D points that have connectivity with the 3D point to be decoded (Yes in S1524), the decoding device proceeds to step S1525, and if not (No in S1524), the decoding device proceeds to step S1528.

[0385] The decoding device calculates a predicted value using attribute information of two or more 3D points (S1525).

[0386] The decoding device determines whether or not count is greater than 0 (S1526). If count is greater than 0 (Yes in S1526), ​​the decoding device proceeds to step S1527, and if not (No in S1526), ​​the decoding device proceeds to step S1528.

[0387] The decoding device decodes the attribute information of the 3D point to be decoded using the calculated predicted value (S1527). That is, the decoding device calculates a residual between the attribute information of the 3D point to be decoded and a predicted value calculated using attribute information of two or more 3D points, and decodes the calculated residual. Step S1527 is predictive decoding in the third prediction mode.

[0388] The decoding device determines whether there is one decoded 3D point that has connectivity with the 3D point to be decoded (S1528). If there is one decoded 3D point that has connectivity with the 3D point to be decoded (Yes in S1528), the decoding device proceeds to step S1529, and if not (No in S1528), the decoding device proceeds to step S1530.

[0389] The decoding device decodes the attribute information of the 3D point to be decoded using the attribute information of one 3D point as a predicted value (S1529). That is, the decoding device calculates a residual between the attribute information of the 3D point to be decoded and the attribute information of one 3D point, and decodes the calculated residual. Step S1529 is predictive decoding in the second prediction mode.

[0390] The decoding device decodes the attribute information of the 3D point to be decoded in a prediction-free mode (S1530). That is, the decoding device decodes the attribute information of the 3D point to be decoded without using a predicted value. Step S1530 is predictive decoding in the first prediction mode.

[0391] Next, mesh_attribute_coding, which is a unit for encoding attribute information of 3D points, will be described. Fig. 57 is a diagram showing an example of the syntax of mesh_attribute_coding according to this embodiment. Fig. 58 is a diagram showing an example of the syntax of mesh_attribute_header according to this embodiment.

[0392] "mesh_attribute_coding" indicates a unit for encoding attribute information of a 3D point, "mesh_attribute_header" indicates header information related to the attribute information, and "mesh_attribute_data" indicates encoded data related to the attribute information.

[0393] AttributeBitDepthMinus1 is information indicating the bit precision of the attribute information coded in the bitstream. The bit precision of the attribute information is calculated by adding 1 to AttributeBitDepthMinus1.

[0394] NumComponent is information indicating the number of components of the attribute information. For example, if the attribute information is UV coordinates, the value of NumComponent is set to 2, and if the attribute information is normal vectors, the value of NumComponent is set to 3.

[0395] AttributeStartDeltaBitDepth[j] is information indicating the bit precision of the initial 3D point of the attribute information coded in the bit stream. The bit precision AttributeStartBitDepth[j] of the j-th component of the initial 3D point is calculated as follows:

[0396] AttributeStartBitDepth[j] = AttributeBitDepthMinus1 + 1 - AttributeStartDeltaBitDepth[j]

[0397] For example, if AttributeBitDepthMinus1 is 9, AttributeStartDeltaBitDepth[0] is 0, and AttributeStartDeltaBitDepth[1] is 2: AttributeStartBitDepth[0] = 9 + 1 - 0 = 10 AttributeStartBitDepth[1] = 9 + 1 - 2 = 8 This indicates that the bit precision of the 0th component of the initial 3D point is 10-bit precision, and the bit precision of the 1st component is 8-bit precision.

[0398] In this way, by providing AttributeStartDeltaBitDepth for each component of the initial 3D point, different bit precisions can be set for each component. Furthermore, by setting the bit precision of the initial 3D point separately from AttributeBitDepthMinus1, which is the bit precision of the attribute information encoded in the bit stream, for example, if the range of values ​​that each component of the initial 3D point can take is smaller than the range of values ​​that each component of the attribute information encoded in the bit stream can take, the value of AttributeStartBitDepth may be smaller than the value obtained by adding 1 to AttributeBitDepthMinus1. In such cases, the amount of code can be reduced by encoding AttributeStart, the value of each component of the initial 3D point (described later), using the value of AttributeStartBitDepth rather than the value obtained by adding 1 to AttributeBitDepthMinus1.

[0399] Furthermore, in this embodiment, the difference between the value obtained by adding 1 to the bit precision AttributeBitDepthMinus1 of the attribute information encoded in the bit stream and the bit precision of the j-th component of the initial 3D point may be encoded as AttributeStartDeltaBitDepth. As a result, if the bit precision of the attribute information and the bit precision of the initial 3D point are the same, the difference value added to the bit stream will be 0, thereby reducing the amount of code.

[0400] Furthermore, the value of AttributeStartBitDepth[j] may be set to 0 by setting AttributeStartDeltaBitDepth[j] to the same value as AttributeBitDepthMinus1 plus 1. In this case, the j-th component for which AttributeStartBitDepth[j] is 0 does not need to be coded as 0 in the bitstream. The decoder can estimate that the j-th component for which AttributeStartBitDepth[j] is 0 is 0. This allows the amount of coding to be reduced by not adding the component for which AttributeStartBitDepth[j] is 0 to the bitstream.

[0401] In this embodiment, an example is shown in which the difference value AttributeStartDeltaBitDepth[j] is added to the bitstream, but this is not necessarily limited to this. For example, AttributeStartBitDepth[j] may be directly coded into the bitstream. This method can reduce the amount of processing required to calculate the difference value.

[0402] Furthermore, the bit length of AttributeStartDeltaBitDepth[j] can be calculated based on the value of AttributeBitDepthMinus1. Specifically, the bit length of AttributeStartDeltaBitDepth[j] can be calculated as ceil(log2(AttributeBitDepthMinus1 + 2)). Here, ceil(x) is a function that outputs an integer value obtained by rounding up the value x. In this way, by determining the bit length of AttributeStartDeltaBitDepth[j] according to the value of AttributeBitDepthMinus1, the amount of code can be reduced.

[0403] AttributePredMode is information indicating a method for encoding attribute information. For example, as shown in FIG. 54, a prediction method "methodA" including fine prediction, coarse prediction, and no-prediction mode may be indicated. Note that the prediction method is not limited to methodA. For example, a prediction method combining two of the three prediction modes of methodA, or a prediction method using only one prediction mode, may be used. Specifically, a prediction method "methodB" including coarse prediction and no-prediction mode may be considered. By providing methodB as the prediction method, a predicted value can be generated without applying division or arithmetic processing requiring decimal point precision, thereby reducing the amount of processing and applying lossless encoding.

[0404] fineMaxcount is information indicating the maximum number of fine predictions used when calculating an average fine predicted value. When AttributePredMode includes fine prediction, for example, when AttributePredMode is methodA, fineMaxcount may be added to the bitstream.

[0405] Note that by setting fineMaxcount to 1, the number of fine predictions used to calculate the average fine prediction value can be limited to 1, making it possible to practically eliminate the need to calculate the average value. This reduces the amount of processing required to calculate the average value. It also prevents decimal points from being generated when calculating the average value.

[0406] Furthermore, fineMaxcount may be set to 0 to prevent the application of fine prediction. In this case, coarse prediction or no prediction mode is used as the prediction mode. For example, if the attribute information of the 3D points following the initial 3D point is all the same, the no prediction mode can be applied to the initial 3D point and coarse prediction can be applied to the subsequent 3D points to reduce the prediction residual to 0, thereby eliminating the need to apply fine prediction. In such a case, setting fineMaxcount to 0 prevents the application of fine prediction, thereby reducing the amount of processing required for encoding or decoding.

[0407] Furthermore, even when additional information necessary for fine prediction exists, fineMaxcount can be set to 0 to prevent fine prediction from being applied, thereby reducing the amount of additional information and the amount of coding.

[0408] AttributeBitDepthMinus1, AttributePredMode, and fineMaxcount may be binarized using Exponential Golomb coding or unary coding, and then variable-length coded. Coding efficiency can be improved by assigning a context to each bit and coding while updating the probability of occurrence based on the frequency of occurrence of 0 and 1. Alternatively, this information may be coded at a fixed length, which reduces the amount of processing.

[0409] Furthermore, fineMaxcount may be set to a different value for each sequence, frame, basemesh, submesh, or 3D point. This allows the number used for average fine prediction to be controlled in an appropriate unit, thereby reducing the amount of processing.

[0410] FIG. 59 is a diagram showing an example of the syntax of mesh_attribute_data according to this embodiment.

[0411] AttributeStartCount is information indicating the number of initial 3D points. AttributeStart[c][j] is information indicating the value of the jth component of the attribute information related to the cth initial 3D point. For example, if the attribute information is UV coordinates, AttributeStart[c][0] may indicate the u value of the cth initial 3D point, and AttributeStart[c][1] may indicate the v value of the cth initial 3D point. AttributeStart[c][j] may be fixed-length coded using the bit length calculated as AttributeStartBitDepth described in FIG. 58. For example, if AttributeStartBitDepth[0] is 10 and AttributeStartBitDepth[1] is 8, this indicates that AttributeStart[c][0], the 0th component of the cth initial 3D point, is coded with 10-bit precision, and AttributeStart[c][1], the 1st component, is coded with 8-bit precision. The amount of code can be reduced by coding the 0th component with 10 bits and the 1st component with 8 bits.

[0412] Furthermore, if the value of AttributeStartBitDepth[j] is 0, AttributeStart[c][j] does not need to be coded into the bitstream. By confirming that the value of AttributeStartBitDepth[j] is 0, the decoder can infer that the value of AttributeStart[c][j] is 0. In this way, by not including components whose value of AttributeStartBitDepth[j] is 0 in the bitstream, the amount of coding can be reduced.

[0413] Furthermore, AttributeStart may be subjected to variable length coding using unary coding, exponential golomb coding, etc. Also, after binarization, a context may be assigned to each bit, and the amount of code may be reduced by applying arithmetic coding.

[0414] Also, if the value of AttributeStartCount is 0, there is no need to encode AttributeStart. In this case, the decoder can confirm that AttributeStartCount is 0 and infer that the value of AttributeStart is 0. This method can reduce the amount of coding.

[0415] Furthermore, if it is guaranteed that there is always at least one initial 3D point, AttributeStartCountMinus1 can be added to the bitstream instead of AttributeStartCount. In this case, the decoder can calculate AttributeStartCount by adding 1 to AttributeStartCountMinus1. This makes it possible to further reduce the amount of code.

[0416] AttributeFineResiCount is information indicating the number of prediction residuals of attribute information predictively coded by fine prediction. In particular, when the value of fineMaxcount is 0, it is assumed that AttributeFineResiCount is set to 0, which is the most efficient case.

[0417] AttributeFineResiSize represents information indicating the code size of AttributeFineResi. For example, it may indicate the code size when AttributeFineResi is binarized and arithmetically coded. In this way, by notifying the decoder of the code size using AttributeFineResiSize before AttributeFineResi, the decoder can pre-read the bits related to AttributeFineResi from the bitstream. Specifically, while the decoder is performing arithmetic decoding on AttributeFineResi, it can perform decoding processing on the following AttributeCoarseResi in parallel, thereby reducing the decoding time.

[0418] AttributeFineResi[c][j] is information indicating the prediction residual for the jth component of attribute information predictively encoded by the cth fine prediction. AttributeFineResi[c][j] can be binarized and variable-length coded using Exponential Golomb coding, unary coding, or the like. Furthermore, coding efficiency can be improved by assigning a context to each bit and updating the probability of occurrence based on the frequency of occurrence of 0s and 1s while coding.

[0419] Furthermore, the context used for arithmetic coding of each bit may be shared with the context used for AttributeCoarseResi, which will be described later. This method allows the number of contexts to be reduced. On the other hand, the context used for arithmetic coding of each bit can also be set to be different from the context used for AttributeCoarseResi, which will be described later. By using different contexts in this way, coding efficiency can be improved even when the trends of prediction residuals differ between fine prediction and coarse prediction. Furthermore, using different contexts allows AttributeFineResi and AttributeCoarseResi to be decoded in parallel, further improving the efficiency of the decoding process.

[0420] AttributeCoarseResiCount is information indicating the number of prediction residuals of attribute information predictively coded by coarse prediction, and plays an important role in coding AttributeCoarseResi, which will be described later.

[0421] AttributeCoarseResiSize represents information indicating the code size of AttributeCoarseResi. For example, it may indicate the code size when AttributeCoarseResi is binarized and arithmetically coded. In this way, by informing the decoder of AttributeCoarseResiSize prior to coding AttributeCoarseResi, the decoder can pre-read bits related to AttributeCoarseResi from the bitstream. Specifically, while the decoder is performing arithmetic decoding related to AttributeCoarseResi, it can perform decoding processing of the subsequent bitstream in parallel, thereby reducing decoding time.

[0422] AttributeCoarseResi[c][j] is information indicating the prediction residual for the jth component of attribute information predictively coded by the cth coarse prediction. AttributeCoarseResi[c][j] can be binarized and variable-length coded using Exponential Golomb coding, unary coding, or the like. Furthermore, coding efficiency can be improved by assigning a context to each bit and updating the probability of occurrence based on the frequency of occurrence of 0 and 1 while coding.

[0423] Furthermore, the context used for arithmetic coding of each bit may be shared with the context used in the above-mentioned AttributeFineResi. This method allows the number of contexts to be reduced. On the other hand, the context used for arithmetic coding of each bit can also be set to be different from the context used in the above-mentioned AttributeFineResi. By using different contexts in this way, coding efficiency can be improved even when the trends of prediction residuals differ between fine prediction and coarse prediction.

[0424] Furthermore, AttributeStartCount, AttributeFineResiCount, AttributeFineResiSize, AttributeFineResi, AttributeCoarseResiCount, AttributeCoarseResiSize, and AttributeCoarseResi may be binarized using Exponential Golomb coding or unary coding and then variable-length coded. Also, coding efficiency can be improved by assigning a context to each bit and coding while updating the occurrence probability based on the occurrence frequency of 0 and 1. Furthermore, this information can also be coded at a fixed length, thereby reducing the amount of processing.

[0425] In the description of this embodiment, an example related to encoding or decoding of attribute information of three-dimensional points will be mainly shown, but this is not necessarily limited to this. For example, this embodiment may be applied to encoding or decoding of position information of three-dimensional points. An example of syntax when applied to position information of three-dimensional points will be described later.

[0426] In this embodiment, for example, as shown in FIG. 46 , when an apparatus projects a three-dimensional point G(x, y, z) onto the x-z plane and sets the plane face0 to which the two-dimensional coordinates (u, v) on the x-y plane after projection belong as attribute information for G, an example is shown in which G and face0 are associated as UV coordinates by setting (u, v) = (face0, 0), but this is not necessarily limited to this. For example, when encoding or decoding face0 as UV coordinates, the attribute information is UV coordinates, but by setting attribute information with NumComponent = 1, only u of the UV coordinates may be encoded. In this way, the amount of code can be reduced by encoding face0 as u and not encoding v.

[0427] Connected 3D points (connected components) may be projected onto a 2D image, and the same plane information may be added as attribute information of the 3D points, and the attribute information may be encoded. This allows a decoder to obtain plane information associated with each 3D point by decoding the attribute information of the 3D points, and use this value to obtain, for example, associated projection information. A method of generating UV coordinates of 3D points on the decoder side using this projection information may be called an "ortho atlas."

[0428] When plane information is encoded as UV coordinates using ortho atlas, the attribute information of three-dimensional points having connectivity all have the same value (face0, 0), for example, as shown in Fig. 60. Fig. 60 is a diagram showing an example of attribute information according to a modified example of this embodiment.

[0429] In this case, the device encodes the initial 3D points in a no-prediction mode, and applies coarse encoding to subsequent 3D points, thereby reducing the prediction residual to zero. Therefore, when using an ortho atlas, setting fineMaxcount=0 as described in this embodiment prevents the application of fine prediction, thereby reducing the amount of processing required for encoding or decoding. Furthermore, if additional information required for fine prediction exists, setting fineMaxcount=0 prevents the application of fine prediction, thereby reducing the additional information and the amount of code.

[0430] Furthermore, in the uv coordinates, v is always 0. Therefore, when using an ortho atlas, AttributeStartBitDepth for the v component of the initial 3D point is set to 0 using AttributeBitDepthMinus1 and AttributeStartDeltaBitDepth described in this embodiment, and the bit length of the fixed-length coding of the v component is set to 0 bit. By not encoding the value 0 of v, the amount of code can be reduced.

[0431] Furthermore, when using an ortho atlas, the prediction residual can be set to 0 by applying coarse prediction or fine prediction, so the amount of code may be reduced by not encoding at least one of the information about the prediction residual, AttributeFineResiCount, AttributeFineResiSize, AttributeFineResi, AttributeCoarseResiCount, AttributeCoarseResiSize, and AttributeCoarseResi. The decoder can estimate the value of 0 for the information that is not encoded.

[0432] Furthermore, if fine prediction is not applied when using an ortho atlas, the amount of coding can be reduced by not encoding additional information related to fine prediction.

[0433] FIG. 61 is a diagram showing an example of the syntax of mesh_attribute_data according to a modification of this embodiment.

[0434] If AttributeFineResiCount or AttributeCoarseResiCount is not coded, the number of 3D points that have connectivity with the initial 3D point, OtherAttributeCount, may be added to the bitstream by the number of initial 3D points. Figure 60 shows an example of this syntax. In this case, the decoder decodes the UV coordinates of the initial 3D point c and copies them to the UV coordinates of OtherAttributeCount[c] 3D points that have connectivity with the initial 3D point c, thereby correctly decoding the UV coordinates.

[0435] The OtherAttributeCount may be binarized by Exponential Golomb coding, unary coding, or the like, and then variable-length coded. The coding efficiency can be improved by assigning a context to each bit and coding while updating the occurrence probability based on the occurrence frequency of 0 and 1. Alternatively, the OtherAttributeCount may be coded at a fixed length. In this case, the amount of processing can be reduced.

[0436] Furthermore, an AttributePredMode for ortho atlas may be defined, and when an encoder encodes attribute information using the method described above, the encoder may add information to the bitstream indicating that the AttributePredMode is a prediction method for ortho atlas. This allows a decoder to determine whether the bitstream was encoded using the above method by decoding the AttributePredMode. If the determination result is Yes, the decoder can decode the attribute information using the above method.

[0437] In this embodiment, the encoding or decoding method for attribute information corresponding to three-dimensional points such as UV coordinates has been described, but this is not necessarily limited to this. For example, the encoding or decoding method described in this embodiment may also be applied to attribute information corresponding to the surface of a polygon made up of three-dimensional points.

[0438] Specifically, face_id, which is face information corresponding to the faces of a polygon, can be encoded or decoded as attribute information with NumComponent = 1. In this way, this embodiment can be used to reduce the amount of code for attribute information corresponding to three-dimensional points or attribute information corresponding to polygons made up of three-dimensional points.

[0439] Fig. 62 is a diagram showing an example of a geometry map according to this embodiment, and Fig. 63 is a diagram showing an example of a texture map according to this embodiment.

[0440] Texture mapping is a common technique used to add visual appearance by projecting an image onto the surface of a three-dimensional model. This technique allows for the modification of visual detail without directly modifying the model itself, thereby improving performance when rendering is required. Information stored in a two-dimensional texture image may include color, smoothness, transparency, etc. The relationship between the three-dimensional mesh and the two-dimensional texture image is established using a UV map. A UV map is a deformation of the three-dimensional mesh onto a two-dimensional plane. Attribute values ​​from the image are then projected onto the three-dimensional model during rendering by overlaying the texture image onto the UV map. This necessitates encoding and decoding techniques that can efficiently store and stream UV maps.

[0441] Different prediction schemes are available for encoding texture coordinates in UV maps. These prediction schemes include those based on the parametrization techniques used in projecting a 3D mesh onto a 2D map, or those based on the relationship between the mesh's geometric data and texture coordinates. In conformal parametrization, triangle edge lengths maintain approximately the same ratio in both 3D space and UV coordinates. Therefore, texture coordinates on a UV map can be predicted using the ratio obtained from the corresponding edges in 3D space.

[0442] As shown in Figure 62, texture coordinates A2, B2, and C2 correspond to A1, B1, and C1 on the geometry map, respectively. The ratio between edges A1B1 and A2B2 in 3D space is calculated as (A1C1) / (C1B1)=R. Finally, the coordinates of A2 in UV space are predicted as A2C2=R*C2B2 and A2=C2+A2C2.

[0443] As shown in Fig. 63, the device can calculate a predicted value for UV coordinates A2 when a 3D point A1 is projected onto a 2D image (texture map). The device may calculate this predicted value using already encoded or decoded C2 and B2, and position information of 3D points A1, B1, and C1 corresponding to A2, B2, and C2.

[0444] More specifically, the device assumes that triangle A1B1C1 and triangle A2B2C2 have a similarity relationship, and calculates A2C2 by multiplying C2B2, whose value has already been determined, by the similarity ratio R, calculated as (A1C1) / (C1B1)=R. The device can also predict the position of A2 using C2+A2C2. As a result, when triangles formed by corresponding 3D points in the geometry map and texture map have a similarity relationship, the device can accurately estimate the predicted value of the attribute information of the 3D point to be encoded or decoded using the 3D point information in the encoded or decoded geometry map and the UV coordinates in the encoded or decoded texture map, thereby reducing the amount of coding.

[0445] In addition, the method in which the device calculates a predicted value of the attribute information of the three-dimensional point to be encoded or decoded using three-dimensional point information in an encoded or decoded geometry map and UV coordinates in an encoded or decoded texture map may also be referred to as fine prediction in this embodiment.

[0446] Next, mesh_position_coding, which is a unit for encoding position information of a three-dimensional point, will be described. Fig. 64 is a diagram showing an example of the syntax of mesh_position_coding according to this embodiment. Fig. 65 is a diagram showing an example of the syntax of mesh_position_header according to this embodiment.

[0447] Mesh_position_coding is a unit for encoding the position information of three-dimensional points, and performs processing related to encoding and decoding of position information. Mesh_position_header indicates header information related to position information, and mesh_position_data indicates encoded data related to position information.

[0448] PositionBitDepthMinus1 is information indicating the bit precision of the position information coded in the bit stream. The bit precision of the position information can be calculated by adding 1 to PositionBitDepthMinus1.

[0449] NumComponent is information indicating the number of components of the position information. For example, if the position information is an xyz coordinate, the value of NumComponent can be set to 3.

[0450] PositionStartDeltaBitDepth[j] is information indicating the bit precision of the initial 3D point of the position information coded in the bit stream. The bit precision PositionStartBitDepth[j] of the j-th component of the initial 3D point can be calculated as follows:

[0451] This can be determined by adding 1 to PositionStartBitDepth[j] = PositionBitDepthMinus1 and then subtracting PositionStartDeltaBitDepth[j]. For example, if PositionBitDepthMinus1 is 9, PositionStartDeltaBitDepth[0] is 0, PositionStartDeltaBitDepth[1] is 2, and PositionStartDeltaBitDepth[2] is 4, the value of PositionStartBitDepth[0] is 9 + 1 - 0 = 10, the value of PositionStartBitDepth[1] is 9 + 1 - 2 = 8, and the value of PositionStartBitDepth[2] is 9 + 1 - 4 = 6. This indicates that the 0th component of the initial 3D point has 10-bit precision, the first component has 8-bit precision, and the second component has 6-bit precision.

[0452] In this way, by setting PositionStartDeltaBitDepth for each component of the initial 3D point, it is possible to set different bit precisions for each component. Also, by making it possible to set the bit precision of the initial 3D point separately from PositionBitDepthMinus1, which is the bit precision of the position information encoded in the bit stream, for example, if the range of values ​​that each component of the initial 3D point can take is smaller than the range that each component of the position information encoded in the bit stream can take, the value of PositionStartBitDepth may be smaller than the value obtained by adding 1 to PositionBitDepthMinus1.

[0453] In such a case, the amount of code can be reduced by encoding PositionStart, which is the value of each component of the initial 3D point (described later), using the value of PositionStartBitDepth instead of the value obtained by adding 1 to PositionBitDepthMinus1. This setting also makes it possible to improve coding efficiency by avoiding the specification of unnecessary bit precision.

[0454] In this embodiment, the difference between the value obtained by adding 1 to PositionBitDepthMinus1, which is the bit precision of the position information encoded in the bit stream, and the bit precision of the j-th component of the initial 3D point can be encoded as PositionStartDeltaBitDepth. With this method, if the value obtained by adding 1 to PositionBitDepthMinus1, which is the bit precision of the position information, is the same as the bit precision of the j-th component of the initial 3D point, the difference value added to the bit stream will be 0, making it possible to reduce the amount of code.

[0455] Furthermore, by setting PositionStartDeltaBitDepth[j] to the same value as PositionBitDepthMinus1 plus 1, the value of PositionStartBitDepth[j] for the j-th component can be set to 0. In this case, the j-th component for which PositionStartBitDepth[j] is 0 does not need to be coded as 0 in the bitstream. The decoder can infer that the j-th component for which PositionStartBitDepth[j] is 0 is 0. This allows the amount of code to be reduced by not adding components for which PositionStartBitDepth[j] is 0 to the bitstream.

[0456] In this embodiment, a method of adding the difference value PositionStartDeltaBitDepth[j] to the bitstream is shown, but this method is not necessarily limited to this. For example, a method of directly encoding PositionStartBitDepth[j] into the bitstream can also be used. This method can reduce the amount of processing required to calculate the difference value.

[0457] Furthermore, the bit length of PositionStartDeltaBitDepth[j] can be determined based on the value of PositionBitDepthMinus1. Specifically, the bit length of PositionStartDeltaBitDepth[j] can be calculated as ceil(log2(PositionBitDepthMinus1+2)), where ceil(x) is a function that outputs an integer value obtained by rounding up the value x. In this way, by determining the bit length of PositionStartDeltaBitDepth[j] according to the value of PositionBitDepthMinus1, the amount of code can be reduced efficiently.

[0458] PositionPredMode is information indicating a method for encoding position information. For example, method A may be used as a prediction method including fine prediction, coarse prediction, and no prediction mode, as shown in the encoding flowchart. The prediction method is not limited to method A, and a prediction method combining two of the three prediction modes, or a prediction method consisting of one prediction mode, may also be used. A specific example is method B, a prediction method combining coarse prediction and no prediction mode. By using method B, a predicted value can be generated without applying division or arithmetic processing requiring decimal point precision, thereby reducing the amount of processing and achieving lossless encoding. Furthermore, in the case of position information, a method such as parallelogram prediction may be used as fine prediction.

[0459] "fineMaxcount" is information indicating the maximum number of fine predictions used when calculating an average fine predicted value. When "PositionPredMode" includes fine prediction, for example, when "PositionPredMode" is "methodA," "fineMaxcount" can be added to the bitstream. By setting "fineMaxcount" to 1, the number of fine predictions used to calculate an average fine predicted value can be limited to one, making it practically unnecessary to calculate an average value. This reduces the amount of processing required to calculate the average value and prevents the generation of numerical values ​​below the decimal point precision due to the average value calculation.

[0460] Furthermore, fine prediction can be prevented from being applied by setting fineMaxcount to 0. In this case, the prediction mode can be coarse prediction or no prediction mode. For example, if the position information of all 3D points following the initial 3D point is the same, the no prediction mode can be applied to the initial 3D point, and coarse prediction can be applied to the subsequent 3D points, thereby making the prediction residual always 0. In such a case, setting fineMaxcount to 0 prevents the application of fine prediction, thereby reducing the amount of processing required for encoding or decoding.

[0461] Furthermore, if there is additional information required for fine prediction, by setting fineMaxcount to 0, the amount of code can be reduced by not applying fine prediction and not encoding the additional information.

[0462] FIG. 66 is a diagram showing an example of the syntax of mesh_position_data according to this embodiment.

[0463] PositionStartCount is information indicating the number of initial 3D points. PositionStart[c][j] is information indicating the value of the jth component of the position information related to the cth initial 3D point. For example, if the position information is an xyz coordinate, PositionStart[c][0] may indicate the x value of the cth initial 3D point, PositionStart[c][1] may indicate the y value of the cth initial 3D point, and PositionStart[c][2] may indicate the z value of the cth initial 3D point.

[0464] Note that PositionStart[c][j] may be fixed-length coded using the bit length calculated using the PositionStartBitDepth described above. For example, if PositionStartBitDepth[0] is 10, PositionStartBitDepth[1] is 8, and PositionStartBitDepth[2] is 6, then PositionStart[c][0], the 0th component of the cth initial 3D point, has 10-bit precision, PositionStart[c][1], the 1st component, has 8-bit precision, and PositionStart[c][2], the 2nd component, has 6-bit precision. The 0th component may be fixed-length coded using 10 bits, the 1st component may be fixed-length coded using 8 bits, and the 2nd component may be fixed-length coded using 6 bits. In this way, the amount of code can be reduced by setting an appropriate bit length for each component of the initial 3D point and coding it using that bit length.

[0465] Furthermore, if PositionStartBitDepth[j] is 0, PositionStart[c][j] does not need to be coded into the bitstream, and the decoder may estimate the value of PositionStart[c][j] to be 0 when PositionStartBitDepth[j] is 0. This method reduces the amount of coding by not adding the j-th component for which PositionStartBitDepth[j] is 0 to the bitstream.

[0466] Note that PositionStart may use variable-length coding such as unary coding or Exponential Golomb coding. Alternatively, a context may be assigned to each bit after binarization, and arithmetic coding may be applied. This reduces the amount of code.

[0467] Furthermore, if PositionStartCount is 0, PositionStart does not need to be coded, and the decoder may estimate the value of PositionStart to be 0 when PositionStartCount is 0. This makes it possible to reduce the amount of coding.

[0468] Furthermore, if there is always one or more initial 3D points, PositionStartCountMinus1 may be added to the bitstream instead of PositionStartCount. In this case, the decoder may calculate PositionStartCount by adding 1 to PositionStartCountMinus1.

[0469] PositionFineResiCount is information indicating the number of prediction residuals of position information predictively coded by fine prediction. PositionFineResiSize is information indicating the code amount of PositionFineResi, and may indicate, for example, the code amount when PositionFineResi is binarized and arithmetically coded.

[0470] In this way, by informing the decoder of the code amount by PositionFineResiSize before PositionFineResi, the decoder can pre-read the bits related to PositionFineResi from the bitstream. For example, while arithmetic decoding related to PositionFineResi is in progress, decoding processing for the following PositionCoarseResi can be performed in parallel, thereby reducing the decoding time.

[0471] PositionFineResi[c][j] is information indicating the prediction residual of the j-th component of the position information predictively coded by the c-th fine prediction. PositionFineResi[c][j] may be binarized and variable-length coded using Exponential Golomb coding, unary coding, or the like. Alternatively, a context may be assigned to each bit, and coding may be performed while updating the occurrence probability based on the occurrence frequency of 0 and 1. This can improve coding efficiency.

[0472] Furthermore, the context used for arithmetic coding of each bit may be shared with the context used in PositionCoarseResi, which will be described later. This method allows the number of contexts to be reduced. Also, the context used for arithmetic coding of each bit may be different from the context used in PositionCoarseResi, which will be described later. This improves coding efficiency when the trends of prediction residuals differ between fine prediction and coarse prediction. Furthermore, using different contexts also allows PositionFineResi and PositionCoarseResi to be decoded in parallel.

[0473] PositionCoarseResiCount is information indicating the number of prediction residuals of position information predictively coded by coarse prediction. PositionCoarseResiSize is information indicating the code amount of PositionCoarseResi, and may indicate, for example, the code amount when PositionCoarseResi is binarized and arithmetically coded.

[0474] In this way, by informing the decoder of the code amount by PositionCoarseResiSize before PositionCoarseResi, the decoder can pre-read the bits related to PositionCoarseResi from the bitstream. For example, while arithmetic decoding related to PositionCoarseResi is in progress, decoding processing of the following bitstream can be performed in parallel, thereby reducing the decoding time.

[0475] PositionCoarseResi[c][j] is information indicating the prediction residual of the j-th component of the position information predictively coded by the c-th coarse prediction. PositionCoarseResi[c][j] may be binarized and variable-length coded using Exponential Golomb coding, unary coding, or the like. Alternatively, a context may be assigned to each bit, and coding may be performed while updating the probability of occurrence based on the frequency of occurrence of 0 and 1. This can improve coding efficiency.

[0476] Furthermore, the context used for arithmetic coding of each bit may be shared with the context used in the above-described PositionFineResi. This method can reduce the number of contexts. Furthermore, the context used for arithmetic coding of each bit may be different from the context used in the above-described PositionFineResi. This can improve coding efficiency when the trends of prediction residuals differ between fine prediction and coarse prediction.

[0477] Furthermore, it is possible to decode PositionFineResi and PositionCoarseResi in parallel by using different contexts.

[0478] Note that PositionStartCount, PositionFineResiCount, PositionFineResiSize, PositionFineResi, PositionCoarseResiCount, PositionCoarseResiSize, and PositionCoarseResi may be binarized and variable-length coded using Exponential Golomb coding, unary coding, or the like. Alternatively, a context may be assigned to each bit, and coding may be performed while updating the occurrence probability based on the occurrence frequency of 0 and 1. This can improve coding efficiency. Furthermore, this information can also be coded at a fixed length, reducing the amount of processing.

[0479] (Configuration Example) Fig. 67 is a diagram showing an example of the configuration of a coding device in this embodiment. Fig. 68 is a flowchart showing an example of a coding method by the coding device in this embodiment.

[0480] The encoding device 1530 includes a circuit 1531 and a memory 1532 connected to the circuit 1531. The encoding device 1530 may perform the processes described in FIGS.

[0481] The circuit 1531 performs the following operations.

[0482] The circuit 1531 acquires multiple vertices included in a three-dimensional mesh (S1541). The circuit 1531 encodes attribute information of one of the multiple vertices using one of at least three prediction modes (S1542). The three prediction modes include a first prediction mode that does not make a prediction, a second prediction mode that performs predictive encoding based on one processed vertex, and a third prediction mode that performs predictive encoding based on two or more processed vertices. If the number of components of the attribute information is set to 1, in encoding (S1542), the attribute information is encoded using a prediction mode different from the third prediction mode.

[0483] This reduces the load on the calculation process by not applying the third prediction mode (predictive coding based on two or more vertices that have already been processed) when the number of components is set to 1. Furthermore, eliminating redundant prediction calculations improves the efficiency of the coding process.

[0484] For example, if one vertex has connectivity with another vertex among the plurality of vertices, the number of components of the attribute information is set to one.

[0485] According to this, by setting the condition that connectivity exists when the number of components is 1, it becomes possible to apply an efficient encoding method that assumes connectivity, thereby improving the compression efficiency of encoded data.

[0486] For example, the attribute information is specific attribute information.

[0487] According to this, by encoding specific attribute information, it is possible to easily clarify the encoding target and perform optimization to increase encoding efficiency.

[0488] For example, the specific attribute information is plane information that indicates the area to which one vertex belongs when one vertex is projected onto a two-dimensional plane.

[0489] This allows for efficient representation of three-dimensional information by encoding based on projection onto a two-dimensional plane. Also, encoding planar information as attribute information simplifies data and improves encoding efficiency.

[0490] For example, the attribute information of one vertex and the attribute information of another vertex are common.

[0491] Therefore, when multiple vertices with connectivity share the same attribute information, redundant data can be omitted, thereby reducing the amount of code.

[0492] For example, when one vertex is coded in the second prediction mode, the prediction residual is 0.

[0493] Therefore, since the prediction residual is 0 when the prediction in the second prediction mode is a perfect match, it is possible to further improve the coding efficiency.

[0494] Fig. 69 is a diagram showing an example of the configuration of a decoding device according to this embodiment, and Fig. 70 is a flowchart showing an example of a decoding method performed by the decoding device according to this embodiment.

[0495] The decoding device 1540 includes a circuit 1541 and a memory 1542 connected to the circuit 1541. The decoding device 1540 may perform the processing described in FIG.

[0496] The circuit 1541 performs the following operations.

[0497] The circuit 1541 acquires multiple vertices included in a three-dimensional mesh (S1551). The circuit 1541 decodes attribute information of one of the multiple vertices using one of at least three prediction modes (S1552). The three prediction modes include a first prediction mode in which no prediction is made, a second prediction mode in which predictive decoding is performed based on one processed vertex, and a third prediction mode in which predictive decoding is performed based on two or more processed vertices. If the number of components of the attribute information is set to 1, in the decoding (S1552), the attribute information is decoded using a prediction mode different from the third prediction mode.

[0498] According to this, the efficiency of the decoding process can be improved by not using the third prediction mode when the number of components during decoding is 1. Furthermore, by omitting redundant processing, the calculation load of the decoding process can be reduced.

[0499] For example, if one vertex has connectivity with another vertex among the plurality of vertices, the number of components of the attribute information is set to one.

[0500] According to this, by defining the number of components as 1 when connectivity exists, efficient decoding processing based on connectivity becomes possible, and the processing load can be reduced.

[0501] For example, the attribute information is specific attribute information.

[0502] According to this, by performing a decoding process on specific attribute information, it is possible to clarify the decoding target and improve the efficiency of decoding.

[0503] For example, the specific attribute information is plane information that indicates the area to which one vertex belongs when one vertex is projected onto a two-dimensional plane.

[0504] According to this, by performing a decoding process based on two-dimensional plane information, it is possible to realize efficient decoding of attribute information for three-dimensional meshes.

[0505] For example, the attribute information of one vertex and the attribute information of another vertex are common.

[0506] Therefore, when common attribute information is used, redundant decoding of attribute information for multiple vertices can be omitted, improving decoding efficiency.

[0507] For example, when one vertex is decoded in the second prediction mode, the prediction residual is 0.

[0508] This reduces the load of the decoding process and improves the efficiency of data decoding.

[0509] The encoding device and decoding device of the present disclosure may also be implemented by combining at least a part of other aspects. Furthermore, the configuration or processing of the encoding device and decoding device of the present disclosure may be realized by combining a part of the processing shown in any of the flowcharts according to this aspect, a part of the configuration of any of the devices, a part of the syntax, or the like with other aspects.

[0510] Furthermore, the processing performed by the decoding device of the present disclosure may also be performed in the encoding device in the same manner.

[0511] The circuit 1531 of the encoding device 1530 may perform the following operation: This operation example is a second example.

[0512] The circuit 1531 acquires a plurality of vertices included in a three-dimensional mesh (S1541). The circuit 1531 arithmetically encodes one of the plurality of vertices using a first prediction mode (the second prediction mode in the above embodiment) or a second prediction mode (the third prediction mode in the above embodiment) (S1542). The first context used in the arithmetic encoding of the first prediction mode and the second context used in the arithmetic encoding of the second prediction mode are the same.

[0513] According to this, by using a common context in different prediction modes (first prediction mode and second prediction mode), the efficiency of context management can be improved. This reduces the processing load required for initializing or updating the context, and improves the efficiency of the encoding process. Furthermore, by sharing the context, if the data has similar tendencies in arithmetic encoding, even if different parameters are used, the encoding efficiency may be improved. By sharing the context, the use of redundant context memory can be reduced, and encoding efficiency can be improved.

[0514] For example, the first prediction mode is a prediction mode that performs predictive coding based on one vertex that has already been processed, and the second prediction mode is a prediction mode that performs predictive coding based on two or more vertices that have already been processed.

[0515] According to this, by using different prediction modes, namely, a first prediction mode (prediction based on one processed vertex) and a second prediction mode (prediction based on two or more processed vertices), prediction accuracy can be improved depending on the situation. In particular, prediction based on two or more processed vertices can achieve more accurate coding. Furthermore, by sharing contexts for the residuals of attribute information generated by different predictions, the number of contexts can be reduced while improving the coding efficiency of arithmetic coding.

[0516] For example, the arithmetic coding of one vertex includes the arithmetic coding of the position information of one vertex.

[0517] According to this, by performing coding targeting position information, it is possible to code the structure or vertex arrangement of a 3D mesh with high efficiency. By applying context sharing to position information, it is possible to further improve the efficiency of arithmetic coding. In particular, by sharing context for residuals of position information generated by different prediction modes, efficient coding is possible.

[0518] For example, the arithmetic coding of one vertex includes the arithmetic coding of the attribute information of one vertex.

[0519] This method contributes to improving the rendering or display quality of 3D meshes by including coding for attribute information (e.g., color, normal vector, etc.). Applying a common context to attribute information can also improve coding efficiency. Furthermore, sharing a context for the residuals of attribute information generated by separate predictions can further improve coding efficiency.

[0520] For example, multiple vertices are included in one group based on a three-dimensional mesh, that is, multiple vertices are included in a group of multiple vertices that form the same three-dimensional mesh or a group of multiple vertices that form the same sub-mesh.

[0521] This allows for the use of a common context when encoding vertices in the same group, thereby improving the prediction accuracy of related vertices. Furthermore, by sharing a context across meshes or sub-meshes, the number of contexts can be reduced, further improving encoding efficiency.

[0522] Furthermore, the circuit 1541 of the decoding device 1540 may perform the following operation: This operation example is a second example.

[0523] The circuit 1541 acquires a plurality of vertices included in a three-dimensional mesh (S1551). The circuit 1541 arithmetically decodes one of the plurality of vertices using a first prediction mode (the second prediction mode in the above embodiment) or a second prediction mode (the third prediction mode in the above embodiment) (S1552). The first context used in the arithmetic decoding of the first prediction mode and the second context used in the arithmetic decoding of the second prediction mode are the same.

[0524] According to this, by using a common context in different prediction modes, it is possible to improve the efficiency of the decoding process. By sharing the context, it is possible to reduce the memory usage and processing load during arithmetic decoding. Furthermore, by using a common context, it is possible to improve the decoding efficiency when different parameters have the same tendency.

[0525] For example, the first prediction mode is a prediction mode in which predictive decoding is performed based on one vertex that has already been processed, and the second prediction mode is a prediction mode in which predictive decoding is performed based on two or more vertices that have already been processed.

[0526] According to this, by using a first prediction mode (predictive decoding based on one processed vertex) and a second prediction mode (predictive decoding based on two or more processed vertices), efficient decoding corresponding to the prediction method used during encoding is possible. Furthermore, by supporting multiple prediction methods, the accuracy of the decoding process can be improved. By sharing context, the amount of calculation and memory consumption in the decoding process can be reduced.

[0527] For example, arithmetic decoding of one vertex includes arithmetic decoding of the position information of one vertex.

[0528] This allows for the encoding of position information during encoding, and allows for efficient restoration of position information during decoding. Application of a common context to position information can improve decoding efficiency. Furthermore, use of a common context in different prediction modes for position information can reduce processing load.

[0529] For example, arithmetic decoding of one vertex includes arithmetic decoding of the attribute information of one vertex.

[0530] This allows for the attribute information to be coded during encoding, and the attribute information to be efficiently restored during decoding. Applying a common context to the attribute information can improve decoding efficiency. Furthermore, using a common context for the residuals of attribute information generated in different prediction modes enables efficient decoding.

[0531] For example, multiple vertices are included in one group based on a three-dimensional mesh, that is, multiple vertices are included in a group of multiple vertices that form the same three-dimensional mesh or a group of multiple vertices that form the same sub-mesh.

[0532] This method improves the efficiency of decoding by using a common context for each mesh or sub-mesh. In particular, when processing multiple vertices as a group, it is possible to reduce redundant contexts and achieve efficient decoding.

[0533] Furthermore, not all of the components described in this embodiment are always necessary, and only some of the components of the encoding device and decoding device of the present disclosure may be included.

[0534] (First Embodiment) FIG. 71 is a diagram showing an example of the syntax of mesh_attribute_data according to this embodiment.

[0535] AttributeStartCount is information indicating the total number of initial 3D points. AttributeStart[c][j] is information indicating the value of the jth component of the attribute information related to the cth initial 3D point. For example, if the attribute information is UV coordinates, AttributeStart[c][0] may indicate the u value of the cth initial 3D point, and AttributeStart[c][1] may indicate the v value of the cth initial 3D point. The encoding device may perform fixed-length encoding of AttributeStart[c][j] using the bit length calculated as AttributeStartBitDepth described in the above embodiment. For example, if AttributeStartBitDepth[0] is 10 and AttributeStartBitDepth[1] is 8, this may indicate that AttributeStart[c][0], the 0th component of the cth initial 3D point, is encoded with 10-bit precision, and AttributeStart[c][1], the first component, is encoded with 8-bit precision. The amount of code can be reduced by performing fixed-length encoding of the 0th component with 10 bits and the first component with 8 bits.

[0536] Furthermore, if the value of AttributeStartBitDepth[j] is 0, the encoding device may not encode AttributeStart[c][j] into the bitstream. The decoding device may confirm that the value of AttributeStartBitDepth[j] is 0, and thereby infer that the value of AttributeStart[c][j] is 0. In this way, by not including components whose value of AttributeStartBitDepth[j] is 0 in the bitstream, the amount of coding can be reduced.

[0537] Furthermore, the encoding device may apply variable length encoding such as unary encoding or exponential golomb encoding to the AttributeStart. Also, after binarization, a context is assigned to each bit, and arithmetic encoding is applied to reduce the amount of code.

[0538] Alternatively, the encoding device may not encode AttributeStart if the value of AttributeStartCount is 0. In this case, the decoding device may confirm that AttributeStartCount is 0 and infer that the value of AttributeStart is 0. This method can reduce the amount of coding.

[0539] Furthermore, if it is guaranteed that there is always at least one initial 3D point, a configuration may be adopted in which AttributeStartCountMinus1 is added to the bitstream instead of AttributeStartCount. In this case, the decoding device may calculate AttributeStartCount by adding 1 to AttributeStartCountMinus1. This makes it possible to further reduce the amount of code.

[0540] AttributeFineResiCount is information indicating the total number of prediction residuals of attribute information predictively coded based on fine prediction. In particular, when the value of fineMaxcount is 0, AttributeFineResiCount may be set to 0 as the most efficient case.

[0541] AttributeFineResiSize is information indicating the code amount of AttributeFineResi. For example, it may indicate the bit length corresponding to when AttributeFineResi is binarized and arithmetically coded. In this way, the encoding device may notify the decoding device of the code amount in advance by adding AttributeFineResiSize to the bitstream before AttributeFineResi. This allows the decoding device to pre-read the bits related to AttributeFineResi from the bitstream while performing decoding processing of the following AttributeCoarseResi in parallel, thereby reducing the decoding time.

[0542] AttributeFineResi[c][j] is information indicating a prediction residual for the jth component of attribute information predictively encoded based on the cth fine prediction. The encoding device may binarize AttributeFineResi[c][j] using Exponential Golomb encoding, unary encoding, or the like, and then perform variable-length encoding. Furthermore, by assigning a context to each bit and performing arithmetic encoding while updating the occurrence probability based on the occurrence frequency of 0 and 1, encoding efficiency can be improved.

[0543] Furthermore, the context used for arithmetic coding of each bit may be shared with the context used in AttributeCoarseResi, which will be described later. This method can reduce the number of contexts. On the other hand, the context used for arithmetic coding of each bit may be set to be different from the context used in AttributeCoarseResi, which will be described later. By using different contexts in this way, coding efficiency can be improved even when the trends of prediction residuals differ between fine prediction and coarse prediction. Furthermore, by using different contexts, AttributeFineResi and AttributeCoarseResi can be decoded in parallel, further improving the efficiency of the decoding process.

[0544] As shown in Fig. 71, the encoding device may not encode AttributeFineResi[c][j] into the bitstream if the value of AttributeFineResiSize is 0. In this case, the decoding device may confirm that AttributeFineResi[c][j] is not included in the bitstream and then estimate the value to be 0. In other words, AttributeFineResiSize functions as control information indicating whether or not there is a code amount for each component of AttributeFineResi[c][j].

[0545] For example, if the encoding device determines that all of the values ​​of AttributeFineResiCount AttributeFineResi[c][j] are 0, it can reduce the amount of code by setting the value of AttributeFineResiSize to 0 and not encoding the AttributeFineResi[c][j] into the bitstream. On the other hand, the decoding device may confirm that the value of AttributeFineResiSize is 0, and then infer that the values ​​of the corresponding AttributeFineResi[c][j] are all 0 and perform decoding processing.

[0546] Furthermore, a constraint may be defined in a standard, profile, or the like that the value of AttributeFineResiSize must not be 0 when the value of AttributeFineResiCount is 1 or greater. Defining such a constraint makes it possible to avoid malfunctions that may occur when a decoding device attempts to decode AttributeFineResi[c][j] by arithmetic decoding even though AttributeFineResiSize is 0. For example, when such an inconsistency occurs, the decoding device may report the bitstream as violating the standard.

[0547] Alternatively, instead of directly encoding the value of AttributeFineResiSize into the bitstream, the encoding device may encode the value of AttributeFineResiSizeMinus1 = AttributeFineResiSize - 1 into the bitstream. In this case, the decoding device may restore AttributeFineResiSize by decoding the value of AttributeFineResiSizeMinus1 from the bitstream and adding 1 to that value. This makes it possible to reduce the amount of information added to the bitstream.

[0548] AttributeCoarseResiCount is information indicating the total number of prediction residuals of attribute information predictively coded based on coarse prediction. AttributeCoarseResiCount plays an important role in coding AttributeCoarseResi, which will be described later.

[0549] AttributeCoarseResiSize is information indicating the code amount of AttributeCoarseResi. For example, AttributeCoarseResi may indicate the code amount (bit length) corresponding to when binarized and arithmetically coded. The coding device may notify the decoding device of the code amount in advance by adding AttributeCoarseResiSize to the bitstream prior to coding AttributeCoarseResi. This allows the decoding device to pre-read bits related to AttributeCoarseResi from the bitstream while performing decoding processing of the subsequent bitstream in parallel, thereby reducing the decoding time.

[0550] AttributeCoarseResi[c][j] is information indicating a prediction residual for the j-th component of attribute information predictively coded based on the c-th coarse prediction. The coding device may binarize AttributeCoarseResi[c][j] using Exponential Golomb coding, unary coding, or the like, and apply variable-length coding to it. Furthermore, the coding device may improve coding efficiency by assigning a context to each bit and performing arithmetic coding while updating the occurrence probability based on the occurrence frequency of 0 and 1.

[0551] Furthermore, the context used for arithmetic coding of each bit may be shared with the context used in the above-described AttributeFineResi. In this way, the number of contexts can be reduced. On the other hand, the context used for arithmetic coding of each bit may be set to be different from the context used in the above-described AttributeFineResi. By using different contexts in this way, coding efficiency can be improved even when the trends of prediction residuals differ between fine prediction and coarse prediction.

[0552] As shown in FIG. 71, the encoding device may add AttributeCoarseResi[c][j] to the bitstream when AttributeCoarseResiSize>0, and may not add AttributeCoarseResi[c][j] to the bitstream when AttributeCoarseResiSize=0.

[0553] If AttributeCoarseResi[c][j] is not added to the bitstream, the decoding device may estimate the value of AttributeCoarseResi[c][j] to be 0. In other words, if the value of AttributeCoarseResiSize, which is information indicating the code amount of AttributeCoarseResi, is 0, AttributeCoarseResi[c][j] is not included in the bitstream, and the decoding device may estimate the values ​​of AttributeCoarseResiCount AttributeCoarseResi[c][j] to be 0.

[0554] For example, if all of the values ​​of AttributeCoarseResiCount AttributeCoarseResi[c][j] are 0, the encoding device adds AttributeCoarseResiSize=0 to the bitstream and does not add the AttributeCoarseResi[c][j] to the bitstream, thereby reducing the amount of code. If the decoding device confirms that AttributeCoarseResiSize=0, it can appropriately decode the bitstream by estimating that all of the values ​​of AttributeCoarseResiCount AttributeCoarseResi[c][j] are 0.

[0555] Furthermore, when AttributeCoarseResiCount>0, AttributeCoarseResiSize needs to be added to the bitstream, and therefore a constraint that the value of AttributeCoarseResiSize must be greater than 0 may be defined in a standard, profile, etc. In this case, when a stream with AttributeCoarseResiSize=0 is decoded, the decoding device may notify that the standard is violated.

[0556] By configuring AttributeCoarseResiSize to have a constraint as described above, it is possible to avoid errors and malfunctions that may occur, for example, when a decoding device performs arithmetic decoding to decode AttributeCoarseResi[c][j] even if the value of AttributeCoarseResiSize, which is information indicating the code size of AttributeCoarseResi, is 0.

[0557] Furthermore, if a standard, profile, or the like specifies that the value of AttributeCoarseResiSize is 1 or greater, the encoding device may add AttributeCoarseResiSizeMinus1, which is a value obtained by subtracting 1 from AttributeCoarseResiSize, to the bitstream. The decoding device may restore AttributeCoarseResiSize by decoding AttributeCoarseResiSizeMinus1 and then adding 1 to the value. This reduces the amount of bits.

[0558] Fig. 72 is a flowchart showing an example of a coding process based on coarse prediction by the coding device according to this embodiment. Fig. 72 shows a specific example of applying this method to coarse prediction using AttributeCoarseResiCount, AttributeCoarseResiSize, and AttributeCoarseResi.

[0559] At the start of processing, the encoding device first encodes AttributeCoarseResiCount, which is information indicating the total number of prediction residuals of attribute information predictively encoded by coarse prediction (S1601).

[0560] Next, the encoding device determines whether AttributeCoarseResiCount is greater than 0 (S1602).

[0561] If the encoding device determines that AttributeCoarseResiCount is greater than 0 (Yes in S1602), it determines whether all values ​​of AttributeCoarseResi[c][j] corresponding to the AttributeCoarseResiCount three-dimensional points are 0 (S1603).

[0562] If the encoding device determines that all AttributeCoarseResi[c][j] are 0 (Yes in S1603), it sets the value of AttributeCoarseResiSize to 0 (S1604).

[0563] Next, if the encoding device determines that any of AttributeCoarseResi[c][j] is not 0 (No in S1603), or after processing in step S1604, it encodes AttributeCoarseResiSize (S1605).

[0564] Thereafter, the encoding device determines whether AttributeCoarseResiSize is greater than 0 (S1606).

[0565] If the encoding device determines that AttributeCoarseResiSize is greater than 0 (Yes in S1606), it encodes AttributeCoarseResiCount number of AttributeCoarseResi[c][j] and adds them to the bitstream (S1607).

[0566] On the other hand, if the encoding device judges No in step S1602 (i.e., AttributeCoarseResiCount = 0) or if the encoding device judges No in step S1606 (i.e., AttributeCoarseResiSize = 0), it terminates processing.

[0567] If the determination in step S1606 is No (i.e., AttributeCoarseResiSize=0), the encoding device may end the process without encoding AttributeCoarseResi[c][j], thereby reducing the amount of code.

[0568] 72 is an example of applying this method to coarse prediction of attribute information, but a similar method may also be applied to AttributeFineResiCount, AttributeFineResiSize, and AttributeFineResi in fine prediction. Furthermore, this method is not limited to attribute information, and may also be applied to, for example, position information of three-dimensional points shown in the above embodiment.

[0569] 73 is a flowchart showing an example of a decoding process based on coarse prediction by the decoding device according to this embodiment. In FIG. 73, a specific example of applying this method to coarse prediction is shown using AttributeCoarseResiCount, AttributeCoarseResiSize, and AttributeCoarseResi.

[0570] At the start of processing, the decoding device first decodes AttributeCoarseResiCount, which is information indicating the total number of prediction residuals of attribute information predictively coded by coarse prediction (S1611).

[0571] Next, the decoding device determines whether AttributeCoarseResiCount is greater than 0 (S1612).

[0572] If the decoding device determines that AttributeCoarseResiCount is greater than 0 (Yes in S1612), it decodes AttributeCoarseResiSize (S1613).

[0573] Next, the decoding device determines whether AttributeCoarseResiSize is greater than 0 (S1614).

[0574] If the decoding device determines that AttributeCoarseResiSize is greater than 0 (Yes in S1614), it decodes AttributeCoarseResiCount pieces of AttributeCoarseResi[c][j] (S1615).

[0575] On the other hand, if the decoding device determines that AttributeCoarseResiSize is 0 (No in S1614), it estimates that all of the values ​​of AttributeCoarseResiCount AttributeCoarseResi[c][j] are 0 (S1616).

[0576] Furthermore, even if the determination in S1612 is No (i.e., AttributeCoarseResiCount=0), the decoding device may end the processing by estimating the values ​​of AttributeCoarseResiSize and AttributeCoarseResi to be 0.

[0577] This allows the decoding device to properly estimate the value of AttributeCoarseResi to be decoded even when AttributeCoarseResi is not included in the bitstream, and maintain the consistency of the decoding process.

[0578] 73 is an example of applying this method to coarse prediction of attribute information, but a similar method may also be applied to AttributeFineResiCount, AttributeFineResiSize, and AttributeFineResi in fine prediction. Furthermore, this method is not limited to attribute information, and may also be applied to, for example, position information of three-dimensional points shown in the above embodiment.

[0579] The syntax for realizing the present disclosure is not limited to the syntax of Fig. 71, and may be, for example, a syntax as shown in Fig. 74. Fig. 74 is a diagram showing an example of the syntax of mesh_attribute_data according to a modified example of the present embodiment.

[0580] The decoding device may use AttributeFineResiSize as the number of bits generated when AttributeFineResi is arithmetically coded (i.e., the code size of AttributeFineResi), thereby enabling the decoding device to know the number of bits generated by arithmetically coding AttributeFineResi.

[0581] Alternatively, the encoding device may set AttributeFineResiSize to a value obtained by adding the bits added for byte alignment by byte_align()_B to the bit amount generated when AttributeFineResi is arithmetically encoded, and converting the total bit amount to a byte amount. In other words, AttributeFineResiSize may be a value in bytes that includes the code amount of AttributeFineResi as well as the code amount added for byte alignment.

[0582] For example, if a coding device arithmetically codes AttributeFineResi, resulting in a code amount of 22 bits, and this is adjusted to 24 bits by byte_align()_B, a decoding device may convert these 24 bits to 3 bytes and add the value "3" to the bitstream as AttributeFineResiSize. By using bytes as the unit of AttributeFineResiSize in this way, the coding device can keep the absolute value of AttributeFineResiSize small, and as a result, the code amount of the entire bitstream can be reduced.

[0583] Furthermore, if a standard or profile specifies that the value of AttributeFineResiSize must be one byte or greater, the encoding device may add AttributeFineResiSizeMinus1, which is obtained by subtracting "1", to the bitstream. In this case, the decoding device may decode AttributeFineResiSize by adding "1" to AttributeFineResiSizeMinus1. This allows for further reduction in the amount of code.

[0584] Furthermore, in a configuration in which AttributeFineResiSize is added as a bit amount rather than a byte amount, if a standard or profile specifies that the value of AttributeFineResiSize must be 8 bits or more because it is necessary to include the code amount for byte alignment, the encoding device may add AttributeFineResiSizeMinus8, which is obtained by subtracting the value "8", to the bitstream. In this case, the decoding device may decode AttributeFineResiSize by adding the value "8" to AttributeFineResiSizeMinus8.

[0585] Furthermore, the standard or profile may be configured to set the value of AttributeFineResiSize to the sum (sum) of the code amount of AttributeFineResi and the code amount added by byte_align()_B. In this case, the decoding device may output a signal indicating a violation of the standard when decoding a bitstream in which the value of AttributeFineResiSize does not match sum. This makes it possible to avoid decoding errors or malfunctions that may occur when the decoding device performs arithmetic decoding using an invalid value of AttributeFineResiSize.

[0586] However, when AttributeFineResiSize=0, the decoding device does not need to perform arithmetic decoding of AttributeFineResi, so even if the value of AttributeFineResiSize does not match sum, it may be configured not to be considered a violation of the standard.

[0587] Note that, although this configuration shows an example using AttributeFineResiCount, AttributeFineResiSize, and AttributeFineResi related to fine prediction, the present invention is not limited to this, and a similar method may be applied using AttributeCoarseResiCount, AttributeCoarseResiSize, and AttributeCoarseResi related to coarse prediction, as shown in Fig. 74. Furthermore, this method is not limited to attribute information, and may also be applied to position information of three-dimensional points shown in the above embodiment.

[0588] In the present disclosure, for example, for information related to fine prediction, an example configuration is shown in which the encoding device adds AttributeFineResi[c][j] to the bitstream when AttributeFineResiSize>0, and does not add AttributeFineResi[c][j] to the bitstream when AttributeFineResiSize=0, thereby reducing the amount of code. Also shown is a configuration in which, when the decoding device cannot acquire AttributeFineResi[c][j] from the bitstream, the decoding device estimates the value to be 0 and appropriately executes decoding processing.

[0589] However, the present disclosure is not limited to this, and the present method may be applied to encoding and decoding of any information having a similar structure, as shown in Fig. 75. Fig. 75 is a diagram showing another example of the syntax of mesh_attribute_data according to a modification of the present embodiment.

[0590] For example, the encoding device may add AttributeXXXResi[c][j] to the bitstream when AttributeXXXResiSize>0, and may not add AttributeXXXResi[c][j] to the bitstream when AttributeXXXResiSize=0.

[0591] If AttributeXXXResi[c][j] is not added to the bitstream, the decoding device may estimate the value to be 0. In other words, if the value of AttributeXXXResiSize, which is information indicating the code amount of AttributeXXXResi, is 0, AttributeXXXResi[c][j] is not included in the bitstream, and the decoding device may estimate that the values ​​of all AttributeXXXResiCount AttributeXXXResi[c][j] are 0.

[0592] For example, if the encoding device determines that all of the values ​​of AttributeXXXResiCount AttributeXXXResi[c][j] are 0, it can add AttributeXXXResiSize=0 to the bitstream and not add AttributeXXXResi[c][j], thereby reducing the amount of code. By confirming that AttributeXXXResiSize=0, the decoding device can infer that the values ​​of the corresponding AttributeXXXResi[c][j] are all 0 and perform appropriate decoding processing.

[0593] Furthermore, if AttributeXXXResiCount>0, it is necessary to add AttributeXXXResiSize to the bitstream, and therefore a standard, profile, or the like may specify that the value of AttributeXXXResiSize is greater than 0. This constraint can prevent inconsistencies from occurring when the decoding device attempts to perform arithmetic decoding even when AttributeXXXResiSize=0.

[0594] Furthermore, when a decoding device decodes a stream in which AttributeXXXResiSize=0, if the above constraint is violated, the decoding device may notify the stream as violating the standard.

[0595] In this way, by configuring the device to be able to detect and notify violations of the AttributeXXXResiSize standard, it is possible to avoid errors and malfunctions that may occur when a decoding device performs arithmetic decoding to decode AttributeXXXResi[c][j] even though the value of AttributeXXXResiSize is 0.

[0596] If a standard or profile specifies a constraint that the value of AttributeXXXResiSize must be 1 or greater, the encoding device may add AttributeXXXResiSizeMinus1, which is the value obtained by subtracting 1 from AttributeXXXResiSize, to the bitstream. The decoding device may restore AttributeXXXResiSize by adding 1 to AttributeXXXResiSizeMinus1. This reduces the absolute value of the value to be encoded, thereby reducing the amount of code in the bitstream.

[0597] Note that this method is not limited to the modified examples of the encoding and decoding processes for attribute information shown in Figures 71, 74, and 75, but may also be applied, for example, to the position information of three-dimensional points shown in the above embodiments.

[0598] Specifically, this method may be applied to a process in which an encoding device uses PositionFineResiSize to encode PositionFineResiCount PositionFineResi for information related to fine prediction of position information, and a decoding device decodes the values.Also, this method may be applied to a process in which an encoding device uses PositionCoarseResiSize to encode PositionCoarseResiCount PositionCoarseResi for information related to coarse prediction of position information, and a decoding device decodes the values.

[0599] (Configuration Example) Fig. 76 is a diagram showing an example of the configuration of a coding device in this embodiment. Fig. 77 is a flowchart showing an example of a coding method by the coding device in this embodiment.

[0600] The encoding device 1600 includes a circuit 1601 and a memory 1602 connected to the circuit 1601. The encoding device 1600 may perform the processing described in FIG.

[0601] The circuit 1601 performs the following operations.

[0602] The circuit 1601 acquires multiple vertices included in a three-dimensional mesh (S1621). The circuit 1601 predictively encodes vertex information related to one of the multiple vertices (S1622). The circuit 1601 generates a bitstream including total number information indicating the total number of prediction residuals obtained by the predictive encoding (S1623). If the total number indicated by the total number information is greater than 0, the circuit 1601 includes in the bitstream prediction residuals obtained by the predictive encoding that have a code amount greater than 0. In other words, a bitstream is generated that includes, in addition to the total number information, prediction residuals that have a code amount greater than 0.

[0603] According to this, by generating a bitstream that further includes prediction residuals when the total number is greater than 0 based on total number information indicating the total number of prediction residuals obtained by predictive coding, it is possible to suppress the coding of unnecessary information. This makes it possible to generate bitstreams efficiently, thereby improving the compression efficiency of three-dimensional meshes.

[0604] For example, the bitstream further includes code amount information indicating the code amount of the prediction residual. When the total number indicated by the total number information is greater than 0, the code amount indicated by the code amount information is greater than 0. Note that the code amount information is, for example, AttributeFineResiSize when fine prediction is performed, and AttributeCoarseResiSize when coarse prediction is performed. The code amount information may indicate the code amount (packet size) of a packet of vertex information.

[0605] This allows explicit management of the amount of information to be coded by including prediction residual coding amount information in the bitstream, which reduces redundancy in the coding process while maintaining consistency with the decoding side, thereby improving overall bit efficiency.

[0606] For example, predictive coding is performed based on one vertex that has already been processed. This predictive coding is a coarse prediction, and is performed when the prediction mode is a value indicating coarse prediction.

[0607] This simplifies the calculation process by making predictions based on one vertex that has already been processed, which reduces the calculation load of the prediction process while ensuring a certain degree of prediction accuracy, thereby achieving effective coding even in a low-computational-load environment.

[0608] For example, predictive coding is performed based on two or more vertices that have already been processed. This predictive coding is fine prediction, and is performed when the prediction mode is a value indicating fine prediction.

[0609] This method uses multiple processed vertices for prediction, resulting in higher prediction accuracy. As a result, the prediction residual is smaller, which reduces the amount of bits required for encoding and enables high compression of 3D meshes.

[0610] For example, the vertex information includes attribute information of one vertex to which the vertex information corresponds.

[0611] This method effectively compresses attribute information such as color information and normal vector information by using predictive coding for the attribute information of each vertex, thereby reducing the data size while maintaining the visual quality of the 3D mesh.

[0612] For example, the vertex information includes position information of one vertex to which the vertex information corresponds.

[0613] This makes it possible to compress the position information of each vertex by using predictive coding, which has the effect of improving the efficiency of storage capacity and communication bandwidth while preserving the structural information of the 3D mesh.

[0614] The circuit 1601 may also perform the following operations.

[0615] The circuit 1601 acquires multiple vertices included in a three-dimensional mesh (S1621). The circuit 1601 predictively encodes information corresponding to one of the multiple vertices (S1622). The circuit 1601 generates a bitstream including total number information indicating the total number of prediction residuals obtained by the predictive encoding and code amount information indicating the code amount of the prediction residuals (S1623). If the total number indicated by the total number information is greater than 0 and the code amount indicated by the code amount information is 0, the circuit 1601 does not encode the prediction residual.

[0616] According to this, when the code amount of a prediction residual is 0, the prediction residual is not coded, so that the coded data in the bitstream can be reduced, resulting in an effect of improving coding efficiency.

[0617] Fig. 78 is a diagram showing an example of the configuration of a decoding device in this embodiment. Fig. 79 is a flowchart showing an example of a decoding method by the decoding device in this embodiment.

[0618] The decoding device 1610 includes a circuit 1611 and a memory 1612 connected to the circuit 1611. The decoding device 1610 may perform the processing described in FIG.

[0619] The circuit 1611 performs the following operations.

[0620] The circuit 1611 acquires a bitstream including total number information indicating the total number of prediction residuals obtained by predictive coding (S1631). The circuit 1611 acquires multiple vertices included in a three-dimensional mesh (S1632). The circuit 1611 predictively decodes vertex information related to one of the multiple vertices (S1633). If the total number indicated by the total number information is greater than 0, the circuit 1611 decodes prediction residuals having a code amount greater than 0, among the prediction residuals obtained by predictive coding.

[0621] According to this, decoding of unnecessary information can be suppressed by performing decoding only when the total number is greater than 0, based on total number information indicating the total number of prediction residuals obtained by predictive decoding. This enables efficient bitstream processing, and has the effect of improving the efficiency of the restoration process of 3D mesh data.

[0622] For example, the bitstream further includes code amount information indicating the code amount of the prediction residual. When the total number indicated by the total number information is greater than 0, the code amount indicated by the code amount information is greater than 0. Note that the code amount information is, for example, AttributeFineResiSize when fine prediction is performed, and AttributeCoarseResiSize when coarse prediction is performed. The code amount information may indicate the code amount (packet size) of a packet of vertex information.

[0623] By including the coding amount information of the prediction residual in the bitstream, it is possible to determine whether or not the decoding process can be performed under the same conditions as the encoding process. As a result, it is possible to avoid unnecessary data processing while ensuring the consistency and integrity of the decoding process.

[0624] For example, predictive decoding is performed based on one vertex that has already been processed. This predictive coding is a coarse prediction, and is performed when the prediction mode is a value indicating coarse prediction.

[0625] This method reduces the computational load of the decoding process by performing predictive decoding based on one processed vertex. This simplifies the process while maintaining a certain level of restoration accuracy, which has the effect of making the method applicable to lightweight decoding environments.

[0626] For example, predictive decoding is performed based on two or more processed vertices. This predictive coding is fine prediction and is performed when the prediction mode is a value indicating fine prediction.

[0627] This method improves prediction accuracy during decoding by using multiple processed vertices for prediction, thereby reducing errors in the reconstructed 3D mesh. This enables mesh reconstruction with high data quality, resulting in highly accurate decoding.

[0628] For example, the vertex information includes attribute information of one vertex to which the vertex information corresponds.

[0629] This method improves the reproducibility of attribute information such as color information and normal information by performing predictive decoding on vertex attribute information, thereby enabling highly accurate reproduction from compressed data without impairing the visual quality of the 3D mesh.

[0630] For example, the vertex information includes position information of one vertex to which the vertex information corresponds.

[0631] This method allows accurate reproduction of information by predictive decoding of vertex position information, which makes it possible to restore the spatial structure of the 3D mesh while maintaining its accuracy, thereby improving the accuracy of model reconstruction.

[0632] The circuit 1611 may also perform the following operations.

[0633] The circuit 1611 acquires a bitstream including total number information indicating the total number of prediction residuals obtained by predictive coding and code amount information indicating the code amount of the prediction residuals (S1631). The circuit 1611 acquires multiple vertices included in a three-dimensional mesh (S1632). The circuit 1611 predictively decodes information corresponding to one of the multiple vertices (S1633). If the total number indicated by the total number information is greater than 0 and the code amount indicated by the code amount information is 0, the circuit 1611 decodes the prediction residual to a specific value (e.g., 0).

[0634] This method decodes the prediction residuals to specific values ​​based on the total number information and the code amount information, allowing prediction residuals omitted on the encoding side to be processed in a consistent manner, thereby enabling accurate data restoration while avoiding unnecessary calculations and inconsistencies.

[0635] Note that, in predictive decoding, circuit 1611 decodes the prediction residual to a specific value (e.g., 0) when the total number indicated by the total number information is greater than 0 and the code amount indicated by the code amount information is 0, but this is not limited to this. For example, in predictive decoding, circuit 1611 may not perform predictive decoding when the total number indicated by the total number information is greater than 0 and the code amount indicated by the code amount information is 0. In this way, it is possible to simplify the decoding process, and there are effects such as reducing the processing load and shortening the decoding time.

[0636] [Others] In the above embodiments, each component may be configured with dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory. Here, the software that realizes the system, information terminal, server, camera device, etc. of the above embodiments is a program that causes a computer to execute each step included in the flowcharts shown in the figures.

[0637] The following cases are also included in this disclosure:

[0638] (1) Specifically, each of the above devices is a computer system consisting of a microprocessor, ROM, RAM, hard disk unit, display unit, keyboard, mouse, etc. A computer program is stored in the RAM or hard disk unit. Each device achieves its function when the microprocessor operates in accordance with the computer program. Here, a computer program is composed of a combination of multiple instruction codes that indicate commands to a computer to achieve a specified function.

[0639] (2) Some or all of the components constituting each of the above devices may be configured as a single system LSI (Large Scale Integration). A system LSI is an ultra-multifunctional LSI manufactured by integrating multiple components on a single chip, and specifically, is a computer system configured to include a microprocessor, ROM, RAM, etc. A computer program is stored in the RAM. The system LSI achieves its functions when the microprocessor operates in accordance with the computer program.

[0640] (3) Some or all of the components constituting each of the above devices may be configured as an IC card or a standalone module that can be attached to each device. The IC card or module may be a computer system configured with a microprocessor, ROM, RAM, etc. The IC card or module may include the above-mentioned ultra-multifunctional LSI. The IC card or module achieves its functions when the microprocessor operates in accordance with a computer program. The IC card or module may be tamper-resistant.

[0641] (4) The present disclosure may be embodied as the methods described above, a computer program that implements these methods on a computer, or a digital signal that includes the computer program.

[0642] The present disclosure may also be a computer program or a digital signal recorded on a computer-readable recording medium, such as a flexible disk, a hard disk, a CD-ROM, an MO, a DVD, a DVD-ROM, a DVD-RAM, a BD (Blu-ray (registered trademark) Disc), a semiconductor memory, etc. Alternatively, the present disclosure may be a digital signal recorded on such a recording medium.

[0643] Furthermore, the present disclosure may involve transmitting the computer program or the digital signal via a telecommunications line, a wireless or wired communication line, a network such as the Internet, data broadcasting, or the like.

[0644] The present disclosure may also be a computer system including a microprocessor and a memory, the memory storing the computer program, and the microprocessor operating in accordance with the computer program.

[0645] The program or the digital signal may also be implemented by another independent computer system by recording it on the recording medium and transferring it, or by transferring the program or the digital signal via the network or the like.

[0646] (5) The above-described embodiments and modifications may be combined with each other.

[0647] The present disclosure is useful, for example, in encoding devices, decoding devices, transmitting devices, receiving devices, etc. related to three-dimensional meshes, and is applicable to computer graphics systems, three-dimensional data display systems, etc.

[0648] 100 Encoding device 101, 121, 144 Vertex information encoder 102, 145 Connection information encoder 103, 122 Attribute information encoder 104, 204, 1103 Preprocessor 105, 205, 2106 Postprocessor 110 Three-dimensional data encoding system 111, 211 Controller 112, 212 Input / output processor 113 Three-dimensional data encoder 114 System multiplexer 115 Three-dimensional data generator 123 Metadata encoder 124 Multiplexer 131 Vertex image generator 132 Attribute image generator 133 Metadata generator 134 Video encoder 141 Two-dimensional data encoder 142 Mesh data encoder 143 Texture encoder 148 Description encoder 151, 251 Circuit 152, 252 Memory 200 Decoding device 201, 221, 244 Vertex information decoder 202, 245 Connection information decoder 203, 222 Attribute information decoder 210 3D data decoding system 213 3D data decoder 214 System demultiplexer 215, 247 Presenter 216 User interface 223 Metadata decoder 224 Demultiplexer 231 Vertex information generator 232 Attribute information generator 234 Video decoder 241 2D data decoder 242 Mesh data decoder 243 Texture decoder 246 Mesh reconstructor 248 Description decoder 300 Network 310 External connector 511 Volumetric capturer 512 Projector 513 Base mesh encoder 514 Displacement encoder 515 Attribute encoder 516 Other type encoder 613 Base mesh decoder 614 Displacement decoder 615 Attribute decoder 616 Other type decoder 617 3D reconstructor 1101 Input mesh 1102 Attribute map 1104 Base mesh 1105 Displacement data 1106 Compressor 1107 Bitstream 1108 Metadata 1206 Subdivider1207 Displacement Vector Calculator 1231 Demultiplexer 1232 Switch 1233 Static Mesh Decoder 1234 Mesh Buffer 1235 Motion Decoder 1236 Base Mesh Reconstructor 1237 Inverse Quantizer 1238, 1243 Video Decoder 1239 Image Unpacker 1240 Inverse Quantizer 1241 Inverse Wavelet Transformer 1242 Reconstructor 1244 Color Converter 1251 Decoded Base Mesh 1252 Subdivider 1253 Subdivided Mesh 1254 Decoded Displacement Data 1255 Displacer 1256 Decoded 3D Mesh 1301 Mesh Frame 1302 Base Mesh Frame 1303 Displacement Information 1304 Bitstream 1501 Set of Vertex Coordinates 1502 Set of 2D Coordinates 1530, 1600 Encoder 1531, 1541, 1601, 1611 Circuit 1532, 1542, 1602, 1612 Memory 1540, 1610 Decoder 2101 Bitstream 2102 Decompressor 2103 Base mesh 2104 Displacement data 2105 Metadata 2106 Postprocessor 2107 Output mesh 2108 Attribute map 2204 Subdivider 2209 Reconstructor 2301 Base mesh frame 2302 Base mesh frame 2303 Displacement information 2304 Mesh frame

Claims

1. An encoding method executed by an encoding device, comprising: obtaining a plurality of vertices included in a three-dimensional mesh; predictively encoding vertex information relating to one of the plurality of vertices; generating a bitstream including total number information indicating the total number of prediction residuals obtained by the predictive encoding; and, if the total number indicated by the total number information is greater than 0, including in the bitstream the prediction residuals obtained by the predictive encoding that have a code amount greater than 0.

2. The encoding method according to claim 1, wherein the bitstream further includes code amount information indicating the code amount of the prediction residual, and when the total number indicated by the total number information is greater than 0, the code amount indicated by the code amount information is greater than 0.

3. The encoding method according to claim 1 or 2, wherein the predictive encoding is performed based on one vertex that has already been processed.

4. The encoding method according to claim 1 or 2, wherein the predictive encoding is performed based on two or more vertices that have already been processed.

5. The encoding method according to claim 1 or 2, wherein the vertex information includes attribute information of the one vertex.

6. The encoding method according to claim 1 or 2, wherein the vertex information includes position information of the one vertex.

7. An encoding method executed by an encoding device, comprising: obtaining a plurality of vertices included in a three-dimensional mesh; predictively encoding information corresponding to one of the plurality of vertices; generating a bitstream including total number information indicating the total number of prediction residuals obtained by the predictive encoding and code amount information indicating the code amount of the prediction residuals; and not encoding the prediction residuals if the total number indicated by the total number information is greater than 0 and the code amount indicated by the code amount information is 0.

8. A decoding method executed by a decoding device, comprising: acquiring a bitstream including total number information indicating the total number of prediction residuals obtained by predictive coding; acquiring a plurality of vertices included in a three-dimensional mesh; predictively decoding vertex information related to one of the plurality of vertices; and, if the total number indicated by the total number information is greater than 0, decoding the prediction residuals obtained by the predictive coding that have a code amount greater than 0.

9. The decoding method according to claim 8, wherein the bitstream further includes code amount information indicating the code amount of the prediction residual, and when the total number indicated by the total number information is greater than 0, the code amount indicated by the code amount information is greater than 0.

10. The decoding method according to claim 8 or 9, wherein the predictive decoding is performed based on one vertex that has already been processed.

11. The decoding method according to claim 8 or 9, wherein the predictive decoding is performed based on two or more vertices that have already been processed.

12. The decoding method according to claim 8 or 9, wherein the vertex information includes attribute information of the one vertex.

13. The decoding method according to claim 8 or 9, wherein the vertex information includes position information of the one vertex.

14. A decoding method executed by a decoding device, comprising: acquiring a bitstream including total number information indicating the total number of prediction residuals obtained by predictive coding and code amount information indicating the code amount of the prediction residuals; acquiring multiple vertices included in a three-dimensional mesh; predictively decoding information corresponding to one of the multiple vertices; and, if the total number indicated by the total number information is greater than 0 and the code amount indicated by the code amount information is 0, decoding the prediction residual to a specific value.

15. An encoding device comprising: a circuit; and a memory connected to the circuit, wherein the circuit, in operation, obtains a plurality of vertices included in a three-dimensional mesh; predictively encodes vertex information related to one of the plurality of vertices; generates a bitstream including total number information indicating the total number of prediction residuals obtained by the predictive encoding; and, if the total number indicated by the total number information is greater than 0, includes in the bitstream, among the prediction residuals obtained by the predictive encoding, the prediction residuals having a code amount greater than 0.

16. A decoding device comprising: a circuit; and a memory connected to the circuit, wherein the circuit, in operation, obtains a bit stream including total number information indicating the total number of prediction residuals obtained by predictive coding; obtains a plurality of vertices included in a three-dimensional mesh; predictively decodes vertex information related to one of the plurality of vertices; and, if the total number indicated by the total number information is greater than 0, decodes the prediction residuals obtained by the predictive coding that have a code amount greater than 0.

Images