Encoding method, decoding method, encoding device, and decoding device

Abstract

An encoding method according to one aspect of the present disclosure encodes information on three-dimensional points in a target frame to be encoded, and involves selecting one or more reference three-dimensional points from among a plurality of three-dimensional points in a target frame (S201), and using first information on each of the one or more reference three-dimensional points to calculate a predicted value of second information on a target three-dimensional point to be encoded in the target frame (S202), the one or more reference three-dimensional points being selected on the basis of a distance between the target three-dimensional point and each of the plurality of three-dimensional points.

Description

Encoding method, decoding method, encoding device, and decoding device

[0001] The present disclosure relates to encoding methods and the like.

[0002] In US Pat. No. 6,299,549 a method and apparatus for encoding and decoding three-dimensional mesh data is proposed.

[0003] Japanese Patent Application Laid-Open No. 2006-187015

[0004] Further improvements are desired in the encoding or decoding process for three-dimensional data. The present disclosure aims to improve the encoding or decoding process for three-dimensional data.

[0005] An encoding method according to one aspect of the present disclosure is an encoding method for encoding information of a three-dimensional point in a target frame to be encoded, which selects one or more reference three-dimensional points from a plurality of three-dimensional points in the target frame, and calculates a predicted value of second information of a target three-dimensional point to be encoded in the target frame using first information of each of the one or more reference three-dimensional points, and in selecting the one or more reference three-dimensional points, selects the one or more reference three-dimensional points based on the distance between the target three-dimensional point and each of the plurality of three-dimensional points.

[0006] These comprehensive or specific aspects may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0007] The present disclosure may contribute to improvements in encoding processes and the like related to three-dimensional data.

[0008] 1 is a conceptual diagram showing a three-dimensional mesh according to an embodiment. FIG. 2 is a conceptual diagram showing basic elements of a three-dimensional mesh according to an embodiment. FIG. 3 is a conceptual diagram showing mapping according to an embodiment. FIG. 4 is a block diagram showing a configuration example of an encoding / decoding system according to an embodiment. FIG. 5 is a block diagram showing a configuration example of an encoding device according to an embodiment. FIG. 6 is a block diagram showing another configuration example of an encoding device according to an embodiment. FIG. 7 is a block diagram showing another configuration example of a decoding device according to an embodiment. FIG. 8 is a block diagram showing another configuration example of a decoding device according to an embodiment. FIG. 9 is a block diagram showing another configuration example of a decoding device according to an embodiment. FIG. 10 is a conceptual diagram showing another configuration example of a bit stream according to an embodiment. FIG. 11 is a conceptual diagram showing yet another configuration example of a bit stream according to an embodiment. FIG. 12 is a block diagram showing a specific example of an encoding / decoding system according to an embodiment. FIG. 13 is a conceptual diagram showing an example configuration of point cloud data according to an embodiment. FIG. 14 is a conceptual diagram showing an example data file of point cloud data according to an embodiment. FIG. 15 is a conceptual diagram showing an example configuration of mesh data according to an embodiment. FIG. 16 is a conceptual diagram showing an example data file of mesh data according to an embodiment. FIG. 17 is a conceptual diagram showing types of three-dimensional data according to an embodiment. FIG. 18 is a block diagram showing an example configuration of a three-dimensional data encoder according to an embodiment. FIG. 19 is a block diagram showing an example configuration of a three-dimensional data decoder according to an embodiment. FIG. 19 is a block diagram showing another configuration example of a three-dimensional data encoder according to an embodiment. FIG. 19 is a block diagram showing another configuration example of a three-dimensional data decoder according to an embodiment. FIG. 1 is a conceptual diagram showing a specific example of encoding processing according to an embodiment. FIG. 2 is a conceptual diagram showing a specific example of decoding processing according to an embodiment. FIG. 3 is a block diagram showing an implementation example of an encoding device according to an embodiment. FIG. 4 is a block diagram showing an implementation example of a decoding device according to an embodiment. FIG. 5 is a block diagram showing a configuration example of an encoding / decoding system according to an embodiment. FIG. 6 is a block diagram showing another configuration example of an encoding device according to an embodiment. FIG. 7 is a block diagram showing another configuration example of a decoding device according to an embodiment. FIG. 8 is a block diagram showing yet another configuration example of an encoding device according to an embodiment. FIG. 9 is a block diagram showing yet another configuration example of a decoding device according to an embodiment. FIG. 10 is a block diagram showing a detailed configuration example of a decoding device according to an embodiment. FIG. 11 is an explanatory diagram showing coordinates of vertices in a three-dimensional mesh according to an embodiment.FIG. 1 is an explanatory diagram showing prediction information according to an embodiment. FIG. 2 is an explanatory diagram showing an example of a mesh according to an embodiment. FIG. 3 is an explanatory diagram showing an example of dividing a mesh into sub-meshes according to an embodiment. FIG. 4 is an explanatory diagram showing a positional relationship of 3D points according to an embodiment. FIG. 5 is an explanatory diagram showing a distance between 3D points according to an embodiment. FIG. 6 is a flow diagram showing a process of selecting adjacent points according to an embodiment. FIG. 7 is an explanatory diagram showing a process of selecting adjacent points according to an embodiment. FIG. 8 is a diagram showing a first example of a reference destination of a motion group according to an embodiment. FIG. 9 is a diagram showing an example of syntax of a base mesh header according to an embodiment. FIG. 10 is a diagram showing a second example of a reference destination of a motion group according to an embodiment. FIG. 11 is a diagram showing a third example of a reference destination of a motion group according to an embodiment. FIG. 12 is an explanatory diagram showing a relationship between vertices constituting a mesh and motion groups according to an embodiment. FIG. 13 is an explanatory diagram showing a relationship between vertices constituting a sub-mesh and motion groups according to an embodiment. FIG. 14 is a flow diagram showing an example of a basic encoding process according to an embodiment. FIG. 15 is a flow diagram showing an example of a basic decoding process according to an embodiment.

[0009] <Summary of the Disclosure> Three-dimensional (3D) meshes are used in computer graphics images, for example, which may be composed of multiple temporally distinct frames, each of which may be represented by a 3D mesh.

[0010] A 3D mesh is composed of vertex information indicating the positions of each of the vertices in 3D space, connectivity information indicating the connections between the vertices, and attribute information indicating the attributes of each vertex or face. Each face is constructed according to the connectivity between the vertices. Various computer graphics images can be expressed using such 3D meshes.

[0011] Furthermore, for transmission and storage of the 3D mesh, efficient encoding and decoding of the 3D mesh is expected. For efficient encoding and decoding of the 3D mesh, arithmetic coding and decoding may be used.

[0012] Further improvements are desired in the encoding or decoding process for three-dimensional data. The present disclosure aims to improve the encoding or decoding process for three-dimensional data.

[0013] Hereinafter, examples of inventions that can be obtained from the disclosure of this specification will be given, and the effects and the like that can be obtained from these inventions will be explained.

[0014] The encoding method of Example 1 is an encoding method for encoding information of a three-dimensional point in a target frame to be encoded, which selects one or more reference three-dimensional points from a plurality of three-dimensional points in the target frame, and calculates a predicted value of second information of a target three-dimensional point to be encoded in the target frame using first information of each of the one or more reference three-dimensional points, and in selecting the one or more reference three-dimensional points, selects the one or more reference three-dimensional points based on the distance between the target three-dimensional point and each of the plurality of three-dimensional points.

[0015] It is considered that the closer the distance between the three-dimensional points, the closer the information about the three-dimensional points will be. Therefore, for example, it is considered that the prediction residual can be reduced by using a three-dimensional point that is close to the target three-dimensional point as a reference three-dimensional point to calculate a predicted value. If the prediction residual can be reduced, for example, the amount of code of a bitstream including information indicating the prediction residual can be reduced. Therefore, the amount of code can be reduced by selecting one or more reference three-dimensional points based on the distance between the target three-dimensional point and each of the multiple three-dimensional points.

[0016] The encoding method of Example 2 may be the encoding method of Example 1, further comprising: calculating a prediction residual, which is the difference between the value indicated by the second information and the predicted value; and generating a bitstream including prediction residual information indicating the calculated prediction residual.

[0017] This makes it possible to generate a bitstream with a reduced amount of coding.

[0018] The encoding method of Example 3 may be the encoding method of Example 1 or Example 2, wherein the first information for each of the one or more reference 3D points indicates a motion vector for each of the one or more reference 3D points, and the second information indicates a motion vector for the target 3D point.

[0019] This allows the motion vectors to be coded.

[0020] The encoding method of Example 4 is the encoding method of any one of Examples 1 to 3, and in calculating the predicted value, the predicted value may be calculated using inter prediction.

[0021] This allows a predicted value to be calculated.

[0022] The encoding method of Example 5 is any of the encoding methods of Examples 1 to 4, and in selecting the one or more reference three-dimensional points, the distance may be calculated by calculating a difference between the coordinates of the target three-dimensional point and the coordinates of each of the plurality of three-dimensional points.

[0023] This makes it possible to calculate the distance between the target 3D point and each of the multiple 3D points.

[0024] The encoding method of Example 6 is any of the encoding methods of Examples 1 to 5, and in selecting the one or more reference three-dimensional points, three-dimensional points whose distance is equal to or less than a predetermined value may be selected from the plurality of three-dimensional points as the one or more reference three-dimensional points.

[0025] This makes it possible to select a reference 3D point that is closest to the target 3D point from among a plurality of 3D points.

[0026] The encoding method of Example 7 is any of the encoding methods of Examples 1 to 6, and in selecting the one or more reference three-dimensional points, a predetermined number of three-dimensional points may be selected from the plurality of three-dimensional points in order of smallest distance, to be selected as the one or more reference three-dimensional points.

[0027] This allows an appropriate number of reference 3D points to be selected for calculating the predicted value.

[0028] The encoding method of Example 8 is the encoding method of Example 7, and may generate a bitstream including predetermined number information indicating the predetermined number.

[0029] This allows the decoding device to select a reference 3D point using a predetermined amount of information obtained from the bitstream.

[0030] The encoding method of Example 9 is any of the encoding methods of Examples 1 to 8, and in selecting the one or more reference 3D points, the distance may be calculated using coordinates of a 3D point in a reference frame that corresponds to the target 3D point.

[0031] This allows the decoding device to calculate the distance in the same way as the encoding device, without decoding the coordinates of the target 3D point in the target frame.

[0032] The encoding method of Example 10 is the encoding method of any one of Examples 1 to 9, and the reference frame may be a frame that precedes the target frame in display order.

[0033] This allows the target frame to be coded using a frame that is displayed on the display device earlier than the target frame, in other words, a past frame.

[0034] An encoding method of Example 11 is the encoding method of Example 9, wherein the reference frame may be a frame preceding the target frame in encoding order.

[0035] This allows the target frame to be coded using a coded frame.

[0036] The encoding method of Example 12 is any of the encoding methods of Examples 1 to 11, and the one or more reference three-dimensional points may be selected using the distance and information other than the distance.

[0037] According to this, by appropriately selecting information other than the distance, the amount of coding can be further reduced.

[0038] The encoding method of Example 13 is the encoding method of Example 12, wherein the information other than the distance is connectivity information indicating whether the target 3D point is connected to each of the plurality of 3D points, and in selecting the one or more reference 3D points, a 3D point connected to the target 3D point among the plurality of 3D points may be selected as the one or more reference 3D points.

[0039] In the case of connected 3D points, the information of the 3D points is considered to be closer than that of unconnected 3D points. Therefore, for example, by using a 3D point connected to the target 3D point as a reference 3D point to calculate a predicted value, it is considered that the prediction residual can be reduced, and the amount of code can be further reduced.

[0040] The decoding method of Example 14 is a decoding method for decoding information of a three-dimensional point in a target frame to be decoded, which selects one or more reference three-dimensional points from a plurality of three-dimensional points in the target frame, and calculates a predicted value of second information of a target three-dimensional point to be decoded in the target frame using first information of each of the one or more reference three-dimensional points, and in selecting the one or more reference three-dimensional points, selects the one or more reference three-dimensional points based on the distance between the target three-dimensional point and each of the plurality of three-dimensional points.

[0041] It is considered that the closer the distance between the three-dimensional points, the closer the information about the three-dimensional points will be. Therefore, for example, it is considered that the prediction residual can be reduced by using a three-dimensional point that is close to the target three-dimensional point as a reference three-dimensional point to calculate a predicted value. If the prediction residual can be reduced, for example, the amount of code of a bitstream including information indicating the prediction residual can be reduced. Therefore, by selecting one or more reference three-dimensional points based on the distance between the target three-dimensional point and each of the multiple three-dimensional points, the information about the three-dimensional point can be decoded using information with a reduced amount of code.

[0042] The decoding method of Example 15 is the decoding method of Example 14, wherein prediction residual information indicating a prediction residual is obtained from a bitstream, and the second information may be calculated based on the prediction residual and the predicted value.

[0043] This allows the information on the three-dimensional points to be decoded using the information on the bit stream with the reduced amount of coding.

[0044] The decoding method of Example 16 is the decoding method of Example 14 or Example 15, wherein the first information of each of the one or more reference 3D points may indicate a motion vector of each of the one or more reference 3D points, and the second information may indicate a motion vector of the target 3D point.

[0045] This allows the motion vectors to be decoded.

[0046] The decoding method of Example 17 is the decoding method of any one of Examples 14 to 16, wherein the predicted value may be calculated using inter prediction.

[0047] This allows a predicted value to be calculated.

[0048] The decoding method of Example 18 is any of the decoding methods of Examples 14 to 17, and in selecting the one or more reference three-dimensional points, the distance may be calculated by calculating a difference between the coordinates of the target three-dimensional point and the coordinates of each of the plurality of three-dimensional points.

[0049] This makes it possible to calculate the distance between the target 3D point and each of the multiple 3D points.

[0050] The decoding method of Example 19 is any of the decoding methods of Examples 14 to 18, and in selecting the one or more reference three-dimensional points, three-dimensional points whose distance is equal to or less than a predetermined value may be selected from the plurality of three-dimensional points as the one or more reference three-dimensional points.

[0051] This makes it possible to select a reference 3D point that is closest to the target 3D point from among a plurality of 3D points.

[0052] The decoding method of Example 20 is any of the decoding methods of Examples 14 to 19, and in selecting the one or more reference three-dimensional points, a predetermined number of three-dimensional points may be selected from the plurality of three-dimensional points in order of smallest distance, thereby selecting the one or more reference three-dimensional points.

[0053] This allows an appropriate number of reference 3D points to be selected for calculating the predicted value.

[0054] The decoding method of Example 21 is any one of the decoding methods of Examples 14 to 20, and may acquire predetermined number information indicating the predetermined number from a bitstream.

[0055] This allows a predetermined number of pieces of information obtained from the bitstream to be used to select an appropriate number of reference 3D points for calculating the predicted value.

[0056] The decoding method of Example 22 is any of the decoding methods of Examples 14 to 21, and in selecting the one or more reference 3D points, the distance may be calculated using coordinates of a 3D point in a reference frame that corresponds to the target 3D point.

[0057] This allows the decoding device to calculate the distance in the same way as the encoding device, without decoding the coordinates of the target 3D point in the target frame.

[0058] A decoding method of Example 23 is the decoding method of Example 22, in which the reference frame may be a frame that precedes the target frame in display order.

[0059] This allows the target frame to be decoded using a frame that is displayed on the display device earlier than the target frame, in other words, a past frame.

[0060] A decoding method of Example 24 is the decoding method of Example 22, in which the reference frame may be a frame that precedes the target frame in decoding order.

[0061] This allows the target frame to be decoded using a previously decoded frame.

[0062] The decoding method of Example 25 is any of the decoding methods of Examples 14 to 24, and in selecting the one or more reference three-dimensional points, the one or more reference three-dimensional points may be selected using the distance and information other than the distance.

[0063] According to this, by appropriately selecting information other than distance, it is possible to decode information on three-dimensional points using information with a further reduced amount of coding.

[0064] The decoding method of Example 26 is the decoding method of Example 25, wherein the information other than the distance is connectivity information indicating whether the target 3D point is connected to each of the plurality of 3D points, and in selecting the one or more reference 3D points, a 3D point connected to the target 3D point among the plurality of 3D points may be selected as the one or more reference 3D points.

[0065] In the case of connected 3D points, the information of the 3D points is considered to be closer than that of unconnected 3D points. Therefore, for example, by using a 3D point connected to the target 3D point as a reference 3D point to calculate a predicted value, it is considered that the prediction residual can be reduced, and the information of the 3D point can be decoded using information with a further reduced amount of coding.

[0066] The encoding device of Example 27 is an encoding device that encodes information about a three-dimensional point in a target frame to be encoded, and includes a memory and a circuit that can access the memory. In operation, the circuit selects one or more reference three-dimensional points from a plurality of three-dimensional points in the target frame, and calculates a predicted value of second information of a target three-dimensional point to be encoded in the target frame using first information about each of the one or more reference three-dimensional points. In selecting the one or more reference three-dimensional points, the one or more reference three-dimensional points are selected based on the distance between the target three-dimensional point and each of the plurality of three-dimensional points.

[0067] This provides the same effect as the encoding method of Example 1.

[0068] The decoding device of Example 28 is a decoding method for decoding information of a three-dimensional point in a target frame to be decoded, the decoding device comprising: a memory; and a circuit capable of accessing the memory, wherein the circuit, in operation, selects one or more reference three-dimensional points from among a plurality of three-dimensional points in the target frame, and calculates a predicted value of second information of a target three-dimensional point to be decoded in the target frame using first information of each of the one or more reference three-dimensional points, and selects the one or more reference three-dimensional points based on a distance between the target three-dimensional point and each of the plurality of three-dimensional points.

[0069] This provides the same effect as the decoding method of Example 14.

[0070] Furthermore, these comprehensive or specific aspects may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0071] <Expressions and Terms> The following expressions and terms are used herein.

[0072] (1) Three-dimensional Mesh A three-dimensional mesh is a collection of multiple faces, and represents, for example, a three-dimensional object. A three-dimensional mesh is mainly composed of vertex information, connectivity information, and attribute information. A three-dimensional mesh may be expressed as a polygon mesh or a mesh. A three-dimensional mesh may also vary over time. A three-dimensional mesh may include metadata related to the vertex information, connectivity information, and attribute information, and may also include other additional information.

[0073] (2) Vertex Information Vertex information is information indicating a vertex. For example, the vertex information indicates the position of a vertex in a three-dimensional space. Furthermore, a vertex corresponds to a vertex of a face that constitutes a three-dimensional mesh. Vertex information may be expressed as "geometry." Furthermore, vertex information may be expressed as position information.

[0074] (3) Connection Information Connection information is information that indicates connections between vertices. For example, connection information indicates connections for forming faces or edges of a three-dimensional mesh. Connection information may be expressed as "Connectivity." Connection information may also be expressed as face information.

[0075] (4) Attribute Information Attribute information is information that indicates attributes of a vertex or a face. For example, attribute information indicates attributes such as a color, an image, and a normal vector associated with a vertex or a face. Attribute information may be expressed as "texture."

[0076] (5) Faces A face is an element that constitutes a three-dimensional mesh. Specifically, a face is a polygon on a plane in three-dimensional space. For example, a face can be defined as a triangle in three-dimensional space.

[0077] (6) Plane A plane is a two-dimensional plane in a three-dimensional space. For example, a polygon is formed on a plane, and multiple polygons are formed on multiple planes.

[0078] (7) Bitstream: A bitstream corresponds to coded information. A bitstream may also be referred to as a stream, a coded bitstream, a compressed bitstream, or a coded signal.

[0079] (8) Encoding and Decoding The term encoding may be substituted with terms such as storing, including, writing, describing, signaling, sending, notifying, saving, or compressing, and these terms may be interchangeable. For example, encoding information may mean including the information in a bitstream. Also, encoding information into a bitstream may mean encoding the information to generate a bitstream that includes the encoded information.

[0080] Additionally, the term "decode" may be replaced with terms such as "read," "decode," "read," "load," "derive," "obtain," "receive," "extract," "reconstruct," "reconstruct," "decompress," or "decompress," and these terms may be interchangeable. For example, decoding information may mean obtaining information from a bitstream. Decoding information from a bitstream may mean decoding the bitstream to obtain information contained in the bitstream.

[0081] (9) Ordinal Numbers In the description, ordinal numbers such as first and second may be assigned to components, etc. These ordinal numbers may be changed as appropriate. Furthermore, new ordinal numbers may be assigned to components, etc., or removed. Furthermore, these ordinal numbers may be assigned to elements in order to identify them, and may not correspond to a meaningful order.

[0082] <Three-dimensional mesh> Fig. 1 is a conceptual diagram showing a three-dimensional mesh according to this embodiment. A three-dimensional mesh is composed of multiple faces. For example, each face is a triangle. The vertices of these triangles are defined in three-dimensional space. The three-dimensional mesh then represents a three-dimensional object. Each face may have a color or an image.

[0083] FIG. 2 is a conceptual diagram showing the basic elements of a three-dimensional mesh according to this embodiment. A three-dimensional mesh is composed of vertex information, connection information, and attribute information. The vertex information indicates the positions of the vertices of a face in three-dimensional space. The connection information indicates the connections between the vertices. A face can be identified by the vertex information and connection information. In other words, a colorless three-dimensional object is formed in three-dimensional space by the vertex information and connection information.

[0084] The attribute information may be associated with a vertex or a face. The attribute information associated with a vertex may be expressed as "Attribute Per Point." The attribute information associated with a vertex may indicate an attribute of the vertex itself, or may indicate an attribute of a face connected to the vertex.

[0085] For example, a color may be associated with a vertex as attribute information. The color associated with a vertex may be the color of the vertex itself, or the color of a face connected to the vertex. The color of a face may be the average of multiple colors associated with multiple vertices of the face. Furthermore, a normal vector may be associated with a vertex or a face as attribute information. Such a normal vector can represent the front and back of a face.

[0086] A two-dimensional image may be associated with a surface as attribute information. The two-dimensional image associated with a surface is also expressed as a texture image or an "Attribute Map." Information indicating mapping between the surface and the two-dimensional image may be associated with the surface as attribute information. Information indicating such mapping may be expressed as mapping information, vertex information of a texture image, texture coordinates, or "Attribute UV Coordinate."

[0087] Furthermore, information such as color, image, and moving image used as attribute information may be expressed as "parametric space."

[0088] The attribute information allows texture to be reflected on the three-dimensional object. That is, a three-dimensional object having color is formed in three-dimensional space based on the vertex information, connection information, and attribute information.

[0089] In the above, the attribute information is associated with the vertices or faces, but it may also be associated with the edges.

[0090] 3 is a conceptual diagram illustrating mapping according to this embodiment. For example, a region of a two-dimensional image on a two-dimensional plane can be mapped onto a surface of a three-dimensional mesh in three-dimensional space. Specifically, coordinate information of the region in the two-dimensional image is associated with the surface of the three-dimensional mesh. As a result, an image of the mapped region in the two-dimensional image is reflected on the surface of the three-dimensional mesh.

[0091] By using the mapping, the 2D image used as attribute information can be separated from the 3D mesh. For example, in encoding the 3D mesh, the 2D image may be encoded by an image encoding method or a video encoding method.

[0092] <System Configuration> Fig. 4 is a block diagram showing an example of the configuration of a coding / decoding system according to this embodiment. In Fig. 4, the coding / decoding system includes a coding device 100 and a decoding device 200.

[0093] For example, the encoding device 100 obtains a three-dimensional mesh and encodes the three-dimensional mesh into a bitstream. Then, the encoding device 100 outputs the bitstream to the network 300. For example, the bitstream includes the encoded three-dimensional mesh and control information for decoding the encoded three-dimensional mesh. By encoding the three-dimensional mesh, information about the three-dimensional mesh is compressed.

[0094] The network 300 transmits a bitstream from the encoding device 100 to the decoding device 200. The network 300 may be the Internet, a wide area network (WAN), a local area network (LAN), or a combination of these. The network 300 is not necessarily limited to bidirectional communication, and may be a unidirectional communication network for terrestrial digital broadcasting, satellite broadcasting, or the like.

[0095] Furthermore, the network 300 can be replaced by a recording medium such as a DVD (Digital Versatile Disc) or a BD (Blu-Ray Disc (registered trademark)).

[0096] The decoding device 200 obtains a bitstream and decodes a three-dimensional mesh from the bitstream. By decoding the three-dimensional mesh, information about the three-dimensional mesh is expanded. For example, the decoding device 200 decodes the three-dimensional mesh according to a decoding method corresponding to the encoding method used by the encoding device 100 to encode the three-dimensional mesh. That is, the encoding device 100 and the decoding device 200 perform encoding and decoding according to encoding methods and decoding methods that correspond to each other.

[0097] The 3D mesh before encoding may also be referred to as an original 3D mesh, and the 3D mesh after decoding may also be referred to as a reconstructed 3D mesh.

[0098] 5 is a block diagram showing an example of the configuration of a coding device 100 according to this embodiment. For example, the coding device 100 includes a vertex information encoder 101, a connection information encoder 102, and an attribute information encoder 103.

[0099] The vertex information encoder 101 is an electrical circuit that encodes vertex information. For example, the vertex information encoder 101 encodes the vertex information into a bitstream according to a format defined for the vertex information.

[0100] The connection information encoder 102 is an electrical circuit that encodes the connection information, for example, the connection information encoder 102 encodes the connection information into a bitstream according to a format defined for the connection information.

[0101] The attribute information encoder 103 is an electric circuit that encodes the attribute information. For example, the attribute information encoder 103 encodes the attribute information into a bit stream in accordance with a format defined for the attribute information.

[0102] The vertex information, connectivity information, and attribute information may be coded using variable-length coding or fixed-length coding, such as Huffman coding or context-adaptive binary arithmetic coding (CABAC).

[0103] The vertex information encoder 101, the connection information encoder 102, and the attribute information encoder 103 may be integrated together, or each of the vertex information encoder 101, the connection information encoder 102, and the attribute information encoder 103 may be further subdivided into multiple components.

[0104] 6 is a block diagram showing another example of the configuration of the encoding device 100 according to this embodiment. For example, the encoding device 100 includes a pre-processor 104 and a post-processor 105 in addition to the configuration shown in FIG.

[0105] The preprocessor 104 is an electrical circuit that performs processing before encoding the vertex information, connectivity information, and attribute information. For example, the preprocessor 104 may perform a conversion process, a separation process, a multiplexing process, or the like on the 3D mesh before encoding. More specifically, for example, the preprocessor 104 may separate the vertex information, connectivity information, and attribute information from the 3D mesh before encoding.

[0106] The post-processor 105 is an electrical circuit that performs processing after the vertex information, connection information, and attribute information are encoded. For example, the post-processor 105 may perform conversion processing, separation processing, multiplexing processing, or the like on the encoded vertex information, connection information, and attribute information. More specifically, for example, the post-processor 105 may multiplex the encoded vertex information, connection information, and attribute information into a bitstream. Furthermore, for example, the post-processor 105 may further perform variable-length coding on the encoded vertex information, connection information, and attribute information.

[0107] 7 is a block diagram showing an example of the configuration of a decoding device 200 according to this embodiment. For example, the decoding device 200 includes a vertex information decoder 201, a connection information decoder 202, and an attribute information decoder 203.

[0108] The vertex information decoder 201 is an electrical circuit that decodes vertex information. For example, the vertex information decoder 201 decodes vertex information from a bitstream according to a format defined for the vertex information.

[0109] The connection information decoder 202 is an electrical circuit that decodes the connection information, for example, the connection information decoder 202 decodes the connection information from the bitstream according to a format defined for the connection information.

[0110] The attribute information decoder 203 is an electric circuit that decodes the attribute information. For example, the attribute information decoder 203 decodes the attribute information from the bitstream in accordance with a format defined for the attribute information.

[0111] The vertex information, connection information, and attribute information may be decoded using variable length decoding or fixed length decoding, which may correspond to Huffman coding, context-adaptive binary arithmetic coding (CABAC), or the like.

[0112] The vertex information decoder 201, the connection information decoder 202, and the attribute information decoder 203 may be integrated together, or each of the vertex information decoder 201, the connection information decoder 202, and the attribute information decoder 203 may be further subdivided into multiple components.

[0113] 8 is a block diagram showing another example of the configuration of the decoding device 200 according to this embodiment. For example, the decoding device 200 includes a pre-processor 204 and a post-processor 205 in addition to the configuration shown in FIG.

[0114] The preprocessor 204 is an electrical circuit that performs processing before decoding the vertex information, connection information, and attribute information. For example, the preprocessor 204 may perform conversion processing, separation processing, multiplexing processing, or the like on the bitstream before decoding the vertex information, connection information, and attribute information.

[0115] More specifically, for example, the preprocessor 204 may separate a sub-bitstream corresponding to vertex information, a sub-bitstream corresponding to connectivity information, and a sub-bitstream corresponding to attribute information from the bitstream. Also, for example, the preprocessor 204 may perform variable-length decoding on the bitstream in advance before decoding the vertex information, connectivity information, and attribute information.

[0116] The post-processor 205 is an electrical circuit that performs processing after the vertex information, connection information, and attribute information are decoded. For example, the post-processor 205 may perform conversion processing, separation processing, multiplexing processing, or the like on the decoded vertex information, connection information, and attribute information. More specifically, for example, the post-processor 205 may multiplex the decoded vertex information, connection information, and attribute information onto a three-dimensional mesh.

[0117] <Bitstream> Vertex information, connection information, and attribute information are coded and stored in a bitstream. The relationship between this information and the bitstream is shown below.

[0118] 9 is a conceptual diagram showing an example of the configuration of a bitstream according to this embodiment. In this example, connection information, vertex information, and attribute information are integrated in the bitstream. For example, the connection information, vertex information, and attribute information may be included in a single file.

[0119] Furthermore, multiple portions of this information may be stored sequentially, such as a first portion of connection information, a first portion of vertex information, a first portion of attribute information, a second portion of connection information, a second portion of vertex information, a second portion of attribute information, etc. These multiple portions may correspond to multiple portions that are different in time, multiple portions that are different in space, or multiple different faces.

[0120] Furthermore, the storage order of the connection information, vertex information, and attribute information is not limited to the above example, and a storage order different from the above example may be used.

[0121] 10 is a conceptual diagram showing another example of the configuration of a bitstream according to this embodiment. In this example, a plurality of files are included in the bitstream, and connection information, vertex information, and attribute information are stored in different files. Here, a file containing connection information, a file containing vertex information, and a file containing attribute information are shown, but the storage format is not limited to this example. For example, two types of information among the connection information, vertex information, and attribute information may be included in one file, and the remaining type of information may be included in another file.

[0122] Alternatively, the information may be split and stored in more files. For example, multiple pieces of connectivity information may be stored in multiple files, multiple pieces of vertex information may be stored in multiple files, or multiple pieces of attribute information may be stored in multiple files. These multiple pieces may correspond to multiple temporally different pieces, multiple spatially different pieces, or multiple different faces.

[0123] Furthermore, the storage order of the connection information, vertex information, and attribute information is not limited to the above example, and a storage order different from the above example may be used.

[0124] 11 is a conceptual diagram showing another example of the configuration of a bitstream according to this embodiment. In this example, the bitstream is composed of multiple separable sub-bitstreams, and connection information, vertex information, and attribute information are stored in different sub-bitstreams.

[0125] Here, a sub-bitstream containing connection information, a sub-bitstream containing vertex information, and a sub-bitstream containing attribute information are shown, but the storage format is not limited to this example.

[0126] For example, two types of information among the connection information, vertex information, and attribute information may be included in one sub-bitstream, and the remaining type of information may be included in another sub-bitstream. Specifically, attribute information of a two-dimensional image or the like may be stored in a sub-bitstream that complies with an image coding method, separate from the sub-bitstreams of the connection information and vertex information.

[0127] Each sub-bitstream may also include multiple files, and multiple pieces of connectivity information may be stored in multiple files, multiple pieces of vertex information may be stored in multiple files, or multiple pieces of attribute information may be stored in multiple files.

[0128] 9, 10, and 11, and a storage order different from the above examples may be used. For example, the vertex information, connection information, and attribute information may be stored in the bitstream in this order. Alternatively, the connection information, connection information, and attribute information may be stored in the bitstream in any of the following orders: connection information, attribute information, and vertex information; vertex information, attribute information, and connection information; attribute information, connection information, and vertex information; or attribute information, vertex information, and connection information.

[0129] Furthermore, each of the connection information, vertex information, and attribute information may be divided into a plurality of data, and the plurality of data may be stored in a cyclical or random order within the bitstream.

[0130] 12 is a block diagram showing a specific example of an encoding / decoding system according to this embodiment. In FIG. 12, the encoding / decoding system includes a three-dimensional data encoding system 110, a three-dimensional data decoding system 210, and an external connector 310.

[0131] The three-dimensional data encoding system 110 includes a controller 111, an input / output processor 112, a three-dimensional data encoder 113, a three-dimensional data generator 115, and a system multiplexer 114. The three-dimensional data decoding system 210 includes a controller 211, an input / output processor 212, a three-dimensional data decoder 213, a system demultiplexer 214, a presenter 215, and a user interface 216.

[0132] In the three-dimensional data encoding system 110, sensor data is input from a sensor terminal to a three-dimensional data generator 115. The three-dimensional data generator 115 generates three-dimensional data, such as point cloud data or mesh data, from the sensor data and inputs it to a three-dimensional data encoder 113.

[0133] For example, the three-dimensional data generator 115 generates vertex information, and generates connection information and attribute information corresponding to the vertex information. The three-dimensional data generator 115 may process the vertex information when generating the connection information and attribute information. For example, the three-dimensional data generator 115 may reduce the amount of data by deleting duplicate vertices, or may transform the vertex information (such as by shifting its position, rotating it, or normalizing it). The three-dimensional data generator 115 may also render the attribute information.

[0134] Furthermore, although the three-dimensional data generator 115 is a component of the three-dimensional data encoding system 110 in FIG. 12, it may be arranged externally and independently of the three-dimensional data encoding system 110.

[0135] The sensor terminal that provides the sensor data for generating the three-dimensional data may be, for example, a moving body such as an automobile, a flying object such as an airplane, a mobile terminal, a camera, etc. Furthermore, a distance sensor such as a LIDAR, a millimeter wave radar, an infrared sensor, or a range finder, a stereo camera, or a combination of multiple monocular cameras may also be used as the sensor terminal.

[0136] The sensor data may be the distance (position) of the object, monocular camera images, stereo camera images, color, reflectance, sensor attitude, orientation, gyro, sensing position (GPS information or altitude), speed, acceleration, sensing time, temperature, air pressure, humidity, or magnetism.

[0137] The three-dimensional data encoder 113 corresponds to the encoding device 100 shown in FIG. 5 and other figures. For example, the three-dimensional data encoder 113 encodes three-dimensional data to generate encoded data. The three-dimensional data encoder 113 also generates control information when encoding the three-dimensional data. The three-dimensional data encoder 113 then inputs the encoded data together with the control information to the system multiplexer 114.

[0138] The encoding method for the three-dimensional data may be an encoding method using geometry or an encoding method using a video codec. Here, the encoding method using geometry may also be referred to as a geometry-based encoding method. The encoding method using a video codec may also be referred to as a video-based encoding method.

[0139] The system multiplexer 114 multiplexes the encoded data and control information input from the 3D data encoder 113 to generate multiplexed data using a specified multiplexing method. The system multiplexer 114 may multiplex other media such as video, audio, subtitles, application data, or document files, or reference time information, along with the encoded data and control information of the 3D data. Furthermore, the system multiplexer 114 may multiplex attribute information related to the sensor data or the 3D data.

[0140] For example, the multiplexed data may have a file format for storage or a packet format for transmission. As these formats, ISOBMFF or a format based on ISOBMFF may be used. Also, MPEG-DASH, MMT, MPEG-2 TS Systems, RTP, or the like may be used.

[0141] The multiplexed data is then output as a transmission signal to the external connector 310 by the input / output processor 112. The multiplexed data may be transmitted as a transmission signal by wire or wirelessly. Alternatively, the multiplexed data is stored in an internal memory or a storage device. The multiplexed data may be transmitted to a cloud server via the Internet or may be stored in an external storage device.

[0142] For example, the transmission or storage of the multiplexed data is performed by a method according to the medium for transmission or storage, such as broadcasting or communication. The communication protocol may be http, ftp, TCP, UDP, IP, or a combination thereof. Furthermore, a pull-type communication method or a push-type communication method may be used.

[0143] For wired transmission, Ethernet (registered trademark), USB, RS-232C, HDMI (registered trademark), coaxial cable, etc. may be used. For wireless transmission, 3GPP (registered trademark), 3G / 4G / 5G defined by IEEE, wireless LAN, Wi-Fi, Bluetooth, or millimeter wave may be used. For broadcasting, for example, DVB-T2, DVB-S2, DVB-C2, ATSC3.0, or ISDB-S3 may be used.

[0144] The sensor data may be input to the three-dimensional data generator 115 or the system multiplexer 114. The three-dimensional data or encoded data may be output as a transmission signal directly to the external connector 310 via the input / output processor 112. The transmission signal output from the three-dimensional data encoding system 110 is input to the three-dimensional data decoding system 210 via the external connector 310.

[0145] Furthermore, each operation of the three-dimensional data encoding system 110 may be controlled by a controller 111 that executes an application program.

[0146] In the three-dimensional data decoding system 210, a transmission signal is input to an input / output processor 212. The input / output processor 212 decodes multiplexed data having a file format or a packet format from the transmission signal and inputs the multiplexed data to a system demultiplexer 214. The system demultiplexer 214 obtains coded data and control information from the multiplexed data and inputs them to a three-dimensional data decoder 213. The system demultiplexer 214 may extract other media or reference time information from the multiplexed data.

[0147] The three-dimensional data decoder 213 corresponds to the decoding device 200 shown in Fig. 7 etc. For example, the three-dimensional data decoder 213 decodes three-dimensional data from the encoded data based on a predefined encoding method. The three-dimensional data is then presented to the user by the presenter 215.

[0148] Additionally, additional information such as sensor data may be input to the presenter 215. The presenter 215 may present three-dimensional data based on the additional information. Additionally, a user instruction may be input from a user terminal to the user interface 216. Then, the presenter 215 may present three-dimensional data based on the input instruction.

[0149] The input / output processor 212 may acquire the three-dimensional data and the encoded data from the external connector 310 .

[0150] Furthermore, each operation of the three-dimensional data decoding system 210 may be controlled by a controller 211 that executes an application program.

[0151] 13 is a conceptual diagram showing an example of the configuration of point cloud data according to this embodiment. The point cloud data is data of a group of points representing a three-dimensional object.

[0152] Specifically, a point cloud is made up of a plurality of points, and has position information indicating the three-dimensional coordinate position of each point and attribute information indicating the attribute of each point. The position information is also expressed as geometry.

[0153] The type of attribute information may be, for example, color, reflectance, etc. One point may be associated with attribute information of one type, one point may be associated with attribute information of multiple different types, or one point may be associated with attribute information having multiple values ​​for the same type.

[0154] 14 is a conceptual diagram showing an example of a data file of point cloud data according to this embodiment. This example shows a case where there is a one-to-one correspondence between position information items and attribute information items, and shows position information and attribute information for N points that make up the point cloud data. In this example, the position information is information indicating a three-dimensional coordinate position using three axes, x, y, and z, and the attribute information is information indicating a color using RGB. A PLY file or the like can be used as a representative data file for point cloud data.

[0155] 15 is a conceptual diagram showing an example of the configuration of mesh data according to this embodiment. Mesh data is data used in CG (Computer Graphics) and the like, and is three-dimensional mesh data that shows the three-dimensional shape of an object using multiple surfaces. Each surface is also expressed as a polygon, and has a polygonal shape such as a triangle or a rectangle.

[0156] Specifically, a 3D mesh is composed of a plurality of points constituting a point cloud, as well as a plurality of edges and a plurality of faces. Each point is also expressed as a vertex or a position. Each edge corresponds to a line segment connected by two vertices. Each face corresponds to an area surrounded by three or more edges.

[0157] Furthermore, a three-dimensional mesh has position information indicating the three-dimensional coordinate positions of vertices. The position information is also expressed as vertex information or geometry. A three-dimensional mesh also has connection information indicating the relationship between multiple vertices that make up an edge or a face. The connection information is also expressed as connectivity. A three-dimensional mesh also has attribute information indicating the attributes of the vertices, edges, or faces. The attribute information in a three-dimensional mesh is also expressed as texture.

[0158] For example, the attribute information may indicate the color, reflectance, or normal vector for a vertex, edge, or face. The direction of the normal vector may represent the front and back of the face.

[0159] The mesh data may be stored in a data file format such as an object file.

[0160] 16 is a conceptual diagram showing an example of a data file of mesh data according to this embodiment. In this example, the data file includes position information G(1) to G(N) of N vertices that make up the three-dimensional mesh, and attribute information A1(1) to A1(N) of the N vertices. Also, in this example, M pieces of attribute information A2(1) to A2(M) are included. The attribute information items do not need to correspond one-to-one to vertices or faces. Furthermore, attribute information need not exist.

[0161] The connection information is represented by a combination of vertex indices. n[1, 3, 4] indicates a triangular face formed by three vertices, n=1, n=3, and n=4. Also, m[2, 4, 6] indicates that the attribute information of m=2, m=4, and m=6 corresponds to the three vertices, respectively.

[0162] Furthermore, the actual contents of the attribute information may be written in a separate file. A pointer to that content may be associated with a vertex, a face, or the like. For example, attribute information indicating an image for a face may be stored in a two-dimensional attribute map file. The file name of the attribute map and two-dimensional coordinate values ​​in the attribute map may be written in attribute information A2(1) to A2(M). The method of specifying attribute information for a face is not limited to these methods, and any method may be used.

[0163] 17 is a conceptual diagram showing types of three-dimensional data according to this embodiment. Point cloud data and mesh data may represent static objects or dynamic objects. A static object is an object that does not change over time, and a dynamic object is an object that changes over time. A static object may correspond to three-dimensional data for any point in time.

[0164] For example, point cloud data for a given point in time may be referred to as a PCC frame, mesh data for a given point in time may be referred to as a mesh frame, and PCC frames and mesh frames may be simply referred to as frames.

[0165] The area of ​​the object may be limited to a certain range, as in normal video data, or may not be limited, as in map data. The density of points or surfaces may be determined in various ways. Sparse point cloud data or sparse mesh data may be used, or dense point cloud data or dense mesh data may be used.

[0166] Next, encoding and decoding of a point cloud or a three-dimensional mesh will be described. The device, process, or syntax for encoding and decoding vertex information of a three-dimensional mesh in the present disclosure may be applied to encoding and decoding of a point cloud. The device, process, or syntax for encoding and decoding of a point cloud in the present disclosure may be applied to encoding and decoding vertex information of a three-dimensional mesh.

[0167] Furthermore, a device, process, or syntax for encoding and decoding attribute information of a point cloud in the present disclosure may be applied to encoding and decoding connectivity information or attribute information of a three-dimensional mesh.Furthermore, a device, process, or syntax for encoding and decoding connectivity information or attribute information of a three-dimensional mesh in the present disclosure may be applied to encoding and decoding attribute information of a point cloud.

[0168] Furthermore, at least some of the processing may be shared between the encoding and decoding of point cloud data and the encoding and decoding of mesh data, thereby reducing the scale of the circuit and software program.

[0169] 18 is a block diagram showing an example configuration of a three-dimensional data encoder 113 according to this embodiment. In this example, the three-dimensional data encoder 113 includes a vertex information encoder 121, an attribute information encoder 122, a metadata encoder 123, and a multiplexer 124. The vertex information encoder 121, the attribute information encoder 122, and the multiplexer 124 may correspond to the vertex information encoder 101, the attribute information encoder 103, the post-processor 105, etc. in FIG.

[0170] In this example, the three-dimensional data encoder 113 encodes the three-dimensional data according to a geometry-based encoding method, which takes into account the three-dimensional structure. In addition, in the geometry-based encoding method, attribute information is encoded using configuration information obtained in encoding the vertex information.

[0171] Specifically, first, vertex information, attribute information, and metadata included in three-dimensional data generated from sensor data are input to a vertex information encoder 121, an attribute information encoder 122, and a metadata encoder 123, respectively. Here, connectivity information included in the three-dimensional data may be treated in the same way as attribute information. In addition, in the case of point cloud data, position information may be treated as vertex information.

[0172] The vertex information encoder 121 encodes the vertex information into compressed vertex information and outputs the compressed vertex information as encoded data to the multiplexer 124. The vertex information encoder 121 also generates metadata for the compressed vertex information and outputs it to the multiplexer 124. The vertex information encoder 121 also generates configuration information and outputs it to the attribute information encoder 122.

[0173] The attribute information encoder 122 uses the configuration information generated by the vertex information encoder 121 to encode the attribute information into compressed attribute information and outputs the compressed attribute information as encoded data to the multiplexer 124. The attribute information encoder 122 also generates metadata of the compressed attribute information and outputs it to the multiplexer 124.

[0174] The metadata encoder 123 encodes compressible metadata into compressed metadata and outputs the compressed metadata as encoded data to the multiplexer 124. The metadata encoded by the metadata encoder 123 may be used to encode vertex information and attribute information.

[0175] The multiplexer 124 multiplexes the compressed vertex information, the compressed vertex information metadata, the compressed attribute information, the compressed attribute information metadata, and the compressed metadata into a bitstream, and then inputs the bitstream to the system layer.

[0176] 19 is a block diagram showing an example configuration of a three-dimensional data decoder 213 according to this embodiment. In this example, the three-dimensional data decoder 213 includes a vertex information decoder 221, an attribute information decoder 222, a metadata decoder 223, and a demultiplexer 224. The vertex information decoder 221, the attribute information decoder 222, and the demultiplexer 224 may correspond to the vertex information decoder 201, the attribute information decoder 203, the preprocessor 204, and the like in FIG.

[0177] In this example, the three-dimensional data decoder 213 decodes three-dimensional data according to a geometry-based encoding method. The three-dimensional structure is taken into consideration in the decoding according to the geometry-based encoding method. Furthermore, in the decoding according to the geometry-based encoding method, attribute information is decoded using configuration information obtained in decoding vertex information.

[0178] Specifically, first, a bitstream is input from the system layer to a demultiplexer 224. The demultiplexer 224 separates compressed vertex information, compressed vertex information metadata, compressed attribute information, compressed attribute information metadata, and compressed metadata from the bitstream. The compressed vertex information and compressed vertex information metadata are input to a vertex information decoder 221. The compressed attribute information and compressed attribute information metadata are input to an attribute information decoder 222. The metadata is input to a metadata decoder 223.

[0179] The vertex information decoder 221 decodes vertex information from the compressed vertex information using metadata of the compressed vertex information. The vertex information decoder 221 also generates configuration information and outputs it to the attribute information decoder 222. The attribute information decoder 222 decodes attribute information from the compressed attribute information using the configuration information generated by the vertex information decoder 221 and the metadata of the compressed attribute information. The metadata decoder 223 decodes metadata from the compressed metadata. The metadata decoded by the metadata decoder 223 may be used to decode the vertex information and the attribute information.

[0180] Thereafter, the vertex information, attribute information, and metadata are output as three-dimensional data from the three-dimensional data decoder 213. Note that, for example, this metadata is metadata of the vertex information and attribute information, and can be used in an application program.

[0181] 20 is a block diagram showing another example configuration of the three-dimensional data encoder 113 according to the present embodiment. In this example, the three-dimensional data encoder 113 includes a vertex image generator 131, an attribute image generator 132, a metadata generator 133, a video encoder 134, a metadata encoder 123, and a multiplexer 124. The vertex image generator 131, the attribute image generator 132, and the video encoder 134 may correspond to the vertex information encoder 101 and the attribute information encoder 103 in FIG. 6 , etc.

[0182] In this example, the 3D data encoder 113 encodes the 3D data according to a video-based encoding method. In encoding according to the video-based encoding method, multiple 2D images are generated from the 3D data, and the multiple 2D images are encoded according to a video encoding method. Here, the video encoding method may be High Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), or the like.

[0183] Specifically, first, vertex information and attribute information included in three-dimensional data generated from sensor data are input to a metadata generator 133. The vertex information and attribute information are then input to a vertex image generator 131 and an attribute image generator 132, respectively. The metadata included in the three-dimensional data is then input to a metadata encoder 123. Here, connectivity information included in the three-dimensional data may be treated in the same way as attribute information. In the case of point cloud data, position information may be treated as vertex information.

[0184] The metadata generator 133 generates map information of a plurality of two-dimensional images from the vertex information and attribute information, and inputs the map information to the vertex image generator 131, the attribute image generator 132, and the metadata encoder 123.

[0185] The vertex image generator 131 generates a vertex image based on the vertex information and map information, and inputs the generated image to the video encoder 134. The attribute image generator 132 generates an attribute image based on the attribute information and map information, and inputs the generated image to the video encoder 134.

[0186] The video encoder 134 encodes the vertex images and attribute images into compressed vertex information and compressed attribute information, respectively, in accordance with a video encoding method, and outputs the compressed vertex information and compressed attribute information as encoded data to the multiplexer 124. The video encoder 134 also generates metadata for the compressed vertex information and metadata for the compressed attribute information, and outputs them to the multiplexer 124.

[0187] The metadata encoder 123 encodes the compressible metadata into compressed metadata and outputs the compressed metadata as encoded data to the multiplexer 124. The compressible metadata includes map information. The metadata encoded by the metadata encoder 123 may also be used to encode vertex information and attribute information.

[0188] The multiplexer 124 multiplexes the compressed vertex information, the compressed vertex information metadata, the compressed attribute information, the compressed attribute information metadata, and the compressed metadata into a bitstream, and then inputs the bitstream to the system layer.

[0189] 21 is a block diagram showing another example configuration of the 3D data decoder 213 according to this embodiment. In this example, the 3D data decoder 213 includes a vertex information generator 231, an attribute information generator 232, a video decoder 234, a metadata decoder 223, and a demultiplexer 224. The vertex information generator 231, the attribute information generator 232, and the video decoder 234 may correspond to the vertex information decoder 201 and the attribute information decoder 203 in FIG. 8, etc.

[0190] In this example, the 3D data decoder 213 decodes the 3D data according to a video-based coding method. In the decoding according to the video-based coding method, a plurality of 2D images are decoded according to a video coding method, and 3D data is generated from the plurality of 2D images. Here, the video coding method may be High Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), or the like.

[0191] Specifically, first, a bitstream is input from the system layer to the demultiplexer 224. The demultiplexer 224 separates compressed vertex information, compressed vertex information metadata, compressed attribute information, compressed attribute information metadata, and compressed metadata from the bitstream. The compressed vertex information, compressed vertex information metadata, compressed attribute information, and compressed attribute information metadata are input to the video decoder 234. The compressed metadata is input to the metadata decoder 223.

[0192] The video decoder 234 decodes the vertex images in accordance with the video encoding method. At this time, the video decoder 234 decodes the vertex images from the compressed vertex information using the metadata of the compressed vertex information. Then, the video decoder 234 inputs the vertex images to the vertex information generator 231. The video decoder 234 also decodes the attribute images in accordance with the video encoding method. At this time, the video decoder 234 decodes the attribute images from the compressed attribute information using the metadata of the compressed attribute information. Then, the video decoder 234 inputs the attribute images to the attribute information generator 232.

[0193] The metadata decoder 223 decodes metadata from the compressed metadata. The metadata decoded by the metadata decoder 223 includes map information used to generate vertex information and attribute information. The metadata decoded by the metadata decoder 223 may also be used to decode vertex images and attribute images.

[0194] The vertex information generator 231 reproduces vertex information from the vertex image in accordance with the map information included in the metadata decoded by the metadata decoder 223. The attribute information generator 232 reproduces attribute information from the attribute image in accordance with the map information included in the metadata decoded by the metadata decoder 223.

[0195] Thereafter, the vertex information, attribute information, and metadata are output as three-dimensional data from the three-dimensional data decoder 213. Note that, for example, this metadata is metadata of the vertex information and attribute information, and can be used in an application program.

[0196] Fig. 22 is a conceptual diagram showing a specific example of encoding processing according to this embodiment. Fig. 22 shows a three-dimensional data encoder 113 and a description encoder 148. In this example, the three-dimensional data encoder 113 includes a two-dimensional data encoder 141 and a mesh data encoder 142. The two-dimensional data encoder 141 includes a texture encoder 143. The mesh data encoder 142 includes a vertex information encoder 144 and a connection information encoder 145.

[0197] The vertex information encoder 144, the connection information encoder 145, and the texture encoder 143 may correspond to the vertex information encoder 101, the connection information encoder 102, and the attribute information encoder 103 in FIG.

[0198] For example, the two-dimensional data encoder 141 operates as a texture encoder 143 and generates a texture file by encoding the texture corresponding to the attribute information as two-dimensional data according to an image encoding method or a video encoding method.

[0199] The mesh data encoder 142 also operates as a vertex information encoder 144 and a connectivity information encoder 145, and generates a mesh file by encoding the vertex information and connectivity information. The mesh data encoder 142 may further encode mapping information for textures. The encoded mapping information may then be included in the mesh file.

[0200] The description encoder 148 also generates a description file by encoding a description corresponding to metadata such as text data. The description encoder 148 may encode the description at the system layer. For example, the description encoder 148 may be included in the system multiplexer 114 of FIG. 12 .

[0201] The above operations generate a bitstream containing texture files, mesh files, and description files, which may be multiplexed into the bitstream in file formats such as glTF (Graphics Language Transmission Format) or USD (Universal Scene Description).

[0202] The three-dimensional data encoder 113 may include two mesh data encoders as the mesh data encoder 142. For example, one mesh data encoder encodes vertex information and connectivity information of a static three-dimensional mesh, and the other mesh data encoder encodes vertex information and connectivity information of a dynamic three-dimensional mesh.

[0203] Correspondingly, two mesh files may then be included in the bitstream, for example one mesh file corresponding to a static 3D mesh and another mesh file corresponding to a dynamic 3D mesh.

[0204] Furthermore, the static three-dimensional mesh may be a three-dimensional mesh of an intraframe coded using intraprediction, and the dynamic three-dimensional mesh may be a three-dimensional mesh of an interframe coded using interprediction. Furthermore, information on the dynamic three-dimensional mesh may be differential information between vertex information or connectivity information of the three-dimensional mesh of an intraframe and vertex information or connectivity information of the three-dimensional mesh of an interframe.

[0205] Fig. 23 is a conceptual diagram showing a specific example of the decoding process according to this embodiment. Fig. 23 shows a three-dimensional data decoder 213, a description decoder 248, and a renderer 247. In this example, the three-dimensional data decoder 213 includes a two-dimensional data decoder 241, a mesh data decoder 242, and a mesh reconstructor 246. The two-dimensional data decoder 241 includes a texture decoder 243. The mesh data decoder 242 includes a vertex information decoder 244 and a connectivity information decoder 245.

[0206] The vertex information decoder 244, the connection information decoder 245, the texture decoder 243, and the mesh reconstructor 246 may correspond to the vertex information decoder 201, the connection information decoder 202, the attribute information decoder 203, and the post-processor 205 in Fig. 8. The presenter 247 may correspond to the presenter 215 in Fig. 12.

[0207] For example, the two-dimensional data decoder 241 operates as a texture decoder 243, and decodes the texture corresponding to the attribute information from the texture file as two-dimensional data in accordance with an image coding method or a video coding method.

[0208] The mesh data decoder 242 also operates as a vertex information decoder 244 and a connectivity information decoder 245 to decode vertex information and connectivity information from the mesh file. The mesh data decoder 242 may further decode mapping information for textures from the mesh file.

[0209] The description decoder 248 also decodes descriptions corresponding to metadata such as text data from the description file. The description decoder 248 may decode the descriptions at the system layer. For example, the description decoder 248 may be included in the system demultiplexer 214 of FIG. 12 .

[0210] The mesh reconstructor 246 reconstructs a 3D mesh from the vertex information, connectivity information, and textures according to the description. The renderer 247 renders and outputs the 3D mesh according to the description.

[0211] Through the above operations, a 3D mesh is reconstructed and output from a bitstream containing a texture file, a mesh file, and a description file.

[0212] The three-dimensional data decoder 213 may include two mesh data decoders as the mesh data decoder 242. For example, one mesh data decoder decodes vertex information and connectivity information of a static three-dimensional mesh, and the other mesh data decoder decodes vertex information and connectivity information of a dynamic three-dimensional mesh.

[0213] Correspondingly, two mesh files may then be included in the bitstream, for example one mesh file corresponding to a static 3D mesh and another mesh file corresponding to a dynamic 3D mesh.

[0214] Furthermore, the static three-dimensional mesh may be a three-dimensional mesh of an intraframe coded using intraprediction, and the dynamic three-dimensional mesh may be a three-dimensional mesh of an interframe coded using interprediction. Furthermore, information on the dynamic three-dimensional mesh may be differential information between vertex information or connectivity information of the three-dimensional mesh of an intraframe and vertex information or connectivity information of the three-dimensional mesh of an interframe.

[0215] A dynamic 3D mesh coding method is sometimes called DMC (Dynamic Mesh Coding), and a video-based dynamic 3D mesh coding method is sometimes called V-DMC (Video-based Dynamic Mesh Coding).

[0216] The point cloud encoding method is sometimes called PCC (Point Cloud Compression). The point cloud video-based encoding method is sometimes called V-PCC (Video-based Point Cloud Compression). The point cloud geometry-based encoding method is sometimes called G-PCC (Geometry-based Point Cloud Compression).

[0217] <Implementation Example> Fig. 24 is a block diagram showing an implementation example of the encoding device 100 according to this embodiment. The encoding device 100 includes a circuit 151 and a memory 152. For example, multiple components of the encoding device 100 shown in Fig. 5 etc. are implemented by the circuit 151 and memory 152 shown in Fig. 24.

[0218] The circuit 151 is a circuit that performs information processing and is a circuit that can access the memory 152. For example, the circuit 151 is a dedicated or general-purpose electric circuit that encodes a three-dimensional mesh. The circuit 151 may be a processor such as a CPU. Alternatively, the circuit 151 may be a collection of multiple electric circuits.

[0219] The memory 152 is a dedicated or general-purpose memory that stores information used by the circuit 151 to encode the three-dimensional mesh. The memory 152 may be an electric circuit and may be connected to the circuit 151. The memory 152 may also be included in the circuit 151. The memory 152 may also be a collection of multiple electric circuits. The memory 152 may also be a magnetic disk, an optical disk, or the like, and may also be expressed as a storage, a recording medium, or the like. The memory 152 may also be a non-volatile memory or a volatile memory.

[0220] For example, the memory 152 may store a three-dimensional mesh or a bitstream, or may store a program for the circuit 151 to encode the three-dimensional mesh.

[0221] Note that the encoding device 100 does not necessarily have to implement all of the components shown in Figure 5 and the like, and does not necessarily have to perform all of the processes shown here. Some of the components shown in Figure 5 and the like may be included in another device, and some of the processes shown here may be executed by another device. Furthermore, the encoding device 100 may implement any combination of the components of the present disclosure, and may perform any combination of the processes of the present disclosure.

[0222] Fig. 25 is a block diagram showing an example implementation of a decoding device 200 according to this embodiment. The decoding device 200 includes a circuit 251 and a memory 252. For example, multiple components of the decoding device 200 shown in Fig. 7 and other figures are implemented by the circuit 251 and memory 252 shown in Fig. 25.

[0223] The circuit 251 is a circuit that performs information processing and is a circuit that can access the memory 252. For example, the circuit 251 is a dedicated or general-purpose electric circuit that decodes a three-dimensional mesh. The circuit 251 may be a processor such as a CPU. Alternatively, the circuit 251 may be a collection of multiple electric circuits.

[0224] The memory 252 is a dedicated or general-purpose memory that stores information for the circuit 251 to decode the 3D mesh. The memory 252 may be an electric circuit and may be connected to the circuit 251. The memory 252 may also be included in the circuit 251. The memory 252 may also be a collection of multiple electric circuits. The memory 252 may also be a magnetic disk, an optical disk, or the like, and may also be expressed as a storage, a recording medium, or the like. The memory 252 may also be a non-volatile memory or a volatile memory.

[0225] For example, the memory 252 may store a three-dimensional mesh or a bitstream, or may store a program for the circuit 251 to decode the three-dimensional mesh.

[0226] Note that the decoding device 200 does not necessarily have to implement all of the components shown in Figure 7 and the like, and does not necessarily have to perform all of the processes shown here. Some of the components shown in Figure 7 and the like may be included in another device, and some of the processes shown here may be executed by another device. Furthermore, the decoding device 200 may implement any combination of the components of the present disclosure, and may perform any combination of the processes of the present disclosure.

[0227] The encoding method and the decoding method including the steps performed by each component of the encoding device 100 and the decoding device 200 of the present disclosure may be executed by any device or system. For example, part or all of the encoding method and the decoding method may be executed by a computer including a processor, a memory, an input / output circuit, etc. In this case, the encoding method and the decoding method may be executed by the computer executing a program for causing the computer to execute the encoding method and the decoding method.

[0228] Alternatively, the program or the bitstream may be recorded on a non-transitory computer-readable recording medium such as a CD-ROM.

[0229] An example of a program may be a bitstream. For example, a bitstream including an encoded three-dimensional mesh includes syntax elements for causing the decoding device 200 to decode the three-dimensional mesh. The bitstream then causes the decoding device 200 to decode the three-dimensional mesh according to the syntax elements included in the bitstream. Thus, the bitstream may play a role similar to that of a program.

[0230] The bitstream may be an encoded bitstream containing the encoded 3D mesh, or may be a multiplexed bitstream containing the encoded 3D mesh and other information.

[0231] Furthermore, each component of the encoding device 100 and the decoding device 200 may be configured with dedicated hardware, general-purpose hardware that executes the above-mentioned programs, or a combination of these. The general-purpose hardware may be configured with a memory in which the programs are recorded and a general-purpose processor that reads and executes the programs from the memory. Here, the memory may be a semiconductor memory or a hard disk, and the general-purpose processor may be a CPU.

[0232] Furthermore, the dedicated hardware may be configured with a memory, a dedicated processor, etc. For example, the dedicated processor may execute the encoding method and the decoding method by referring to a memory for recording data.

[0233] Furthermore, as described above, each component of the encoding device 100 and the decoding device 200 may be an electric circuit. These electric circuits may form a single electric circuit as a whole, or each may be a separate electric circuit. Furthermore, these electric circuits may correspond to dedicated hardware, or may correspond to general-purpose hardware that executes the above-mentioned programs, etc. Furthermore, the encoding device 100 and the decoding device 200 may be implemented as an integrated circuit.

[0234] Furthermore, the encoding device 100 may be a transmitting device that transmits the three-dimensional mesh, and the decoding device 200 may be a receiving device that receives the three-dimensional mesh.

[0235] Inter and Intra Prediction: A typical 3D model digitally represents an object so that a user can explore the 3D model by zooming, panning, and / or rotating in all three dimensions while rendering it temporally. One way to construct such a representation is to build a 3D mesh using triangles. In a 3D model, the positions of the triangle vertices, their connectivity to each other, and their associated attributes (such as normals and UV patches) are stored. Storing all this information in an uncompressed format requires significant storage space. Therefore, transmitting this information requires significant bandwidth. The triangles that form the 3D mesh often have repeating patterns and similar attributes, especially in temporal and spatial neighborhoods. These repetitions can be used to develop efficient encoding and decoding methods for storage and transmission.

[0236] FIG. 26 is a block diagram showing an example of the configuration of a coding / decoding system according to this embodiment.

[0237] The encoding / decoding system includes an encoding device 100 and a decoding device 200. The encoding / decoding system accepts a 3D mesh frame input in the form of 3D coordinates of vertices, connectivity information, and associated attributes. The encoding device 100 is responsible for encoding all of this associated information into a bitstream (compressed bitstream). The bitstream may consist of multiple bitstreams. The bitstream is transmitted to the decoding device 200 via a transmission channel. The decoding device 200 decodes the bitstream and generates a 3D model (3D mesh frame) using the decoded 3D coordinates of vertices, the decoded connectivity information, and the decoded associated attributes.

[0238] FIG. 27 is a block diagram showing another example configuration of the encoding device 100 according to this embodiment.

[0239] In this example, the encoding device 100 includes a preprocessor 521 and an encoding processor 522 .

[0240] The preprocessor 521 reads the input 3D mesh frame, processes the 3D mesh frame, extracts a base mesh, displacement information, and an attribute map, and outputs them to the encoding processor 522. An example of the displacement information is a displacement vector.

[0241] The encoding processor 522 compresses the base mesh, displacement information, and attribute map separately and combines them to generate a bitstream.

[0242] FIG. 28 is a block diagram showing another example of the configuration of the decoding device 200 according to this embodiment.

[0243] In this example, the decoding device 200 includes a decoding processor 622 and a post-processor 623 .

[0244] The decoding processor 622 reads the bitstream, separates the coded base mesh, the coded displacement information, and the coded attribute map from the read bitstream, decodes them individually, and outputs them to the post-processor 623. An example of the displacement information is a displacement vector.

[0245] The post-processor 623 processes the base mesh using the displacement information and the attribute map to generate a 3D mesh frame. The generated 3D mesh frame is displayed on a display, for example, by being output to the display. By repeating this process, the 3D mesh frame is repeatedly displayed on the display.

[0246] FIG. 29 is a block diagram showing yet another example configuration of the encoding device 100 according to this embodiment.

[0247] In this example, the encoding device 100 comprises a volumetric capturer 511, a projector 512, a base mesh encoder 513, a displacement encoder 514, an attribute encoder 515, and optionally one or more other type encoders 516.

[0248] The volumetric capturer 511 captures content and outputs the captured content to the projector 512 .

[0249] The projector 512 projects the content onto a 3D mesh frame containing vertex geometry coordinates, texture coordinates, and connectivity data. The data is output to a base mesh encoder 513, a displacement encoder 514, an attribute encoder 515, and optionally one or more other type encoders 516. Each encoder compresses the data into a bitstream.

[0250] FIG. 30 is a block diagram showing yet another example configuration of the decoding device 200 according to this embodiment.

[0251] In this example, the decoding device 200 comprises a base mesh decoder 613 , a displacement decoder 614 , an attribute decoder 615 , one or more other type decoders 616 , and a 3D reconstructor 617 .

[0252] The bitstream is sent to a base mesh decoder 613, a displacement decoder 614, an attribute decoder 615, and optionally one or more other type decoders 616. These decoders decode the bitstream to generate decoded data including vertex geometry coordinates, texture coordinates, and connectivity data. The decoded data is then sent to a 3D reconstructor 617, which reconstructs a 3D mesh frame.

[0253] Fig. 31 is a block diagram showing a detailed configuration example of the decoding device 200 according to this embodiment. Specifically, Fig. 31 shows an example of the configuration of a geometry coordinate decoder included in the decoding device 200.

[0254] In this example, the decoding device 200 comprises a frame header decoder 631 , a vertex geometry coordinate predictor 632 , a vertex geometry coordinate difference decoder 633 , and a reconstructor 634 .

[0255] The frame header decoder 631 reads the bitstream and decodes the frame headers in the bitstream to determine whether the frame data is to be intra-decoded (intra-predicted) or inter-decoded (inter-predicted).

[0256] If inter-decoding is selected, the frame data contained in the bitstream is output to a vertex geometry coordinate predictor 632 .

[0257] The vertex geometry coordinate predictor 632 outputs prediction information to the reconstructor 634. An example of prediction information is a motion vector.

[0258] The reconstructor 634 uses prediction information along with vertex coordinates from previously decoded frames to output the three-dimensional coordinates of the vertices (vertex geometry coordinates).

[0259] On the other hand, if intra-decoding is selected, the frame data included in the bitstream is output to the vertex geometry coordinate difference decoder 633 .

[0260] The vertex geometry coordinate differential decoder 633 decodes frame data encoded as differences between the coordinates of the vertices contained in the frame to generate vertex coordinates. Only one of the vertex geometry coordinates from the vertex geometry coordinate differential decoder 633 and the reconstructor 634 is used to generate the decoded 3D mesh frame.

[0261] 32 is an explanatory diagram showing the coordinates of vertices in a 3D mesh according to this embodiment. Specifically, FIG. 32 shows an example in which the entire 3D mesh frame is decoded using the coordinates (positions) of the actual vertices included in the bitstream.

[0262] The coordinates of vertex A included in the 3D mesh frame at time (t) are decoded as (6, 8, 9) using the Cartesian coordinate system (x, y, z) as shown in Figure 32(a). Similarly, the coordinates of vertex B are decoded as (10, 6, 7), and the coordinates of vertex C are decoded as (14, 8, 9). The same is true for vertices D to G.

[0263] Fig. 33 is an explanatory diagram showing prediction information according to this embodiment. Specifically, Fig. 33 shows another example in which the entire 3D mesh frame at time (t) is decoded using a frame (past frame) at time (t-1) and prediction information included in the bitstream.

[0264] The coordinates (6, 8, 9) of vertex A in the frame to be decoded (current frame) are decoded by adding the coordinates (4, 7, 8) of vertex A in the past frame to the value (2, 1, 1) for vertex A indicated by the prediction information. Similarly, the coordinates (10, 6, 7) of vertex B in the current frame are decoded by adding the coordinates (8, 6, 7) of vertex B in the past frame to the value (2, 0, 0) for vertex B indicated by the prediction information.

[0265] One way to encode a 3D mesh frame is to divide the original 3D mesh (original mesh) into several smaller meshes (sub-meshes) so that each sub-mesh can be coded independently. The vertices of the 3D mesh frame are divided so that the coordinates and connectivity information of the vertices within each partition can be coded independently. Each divided mesh is called a sub-mesh.

[0266] Fig. 34 is an explanatory diagram showing an example of a mesh (original mesh) according to this embodiment. Fig. 35 is an explanatory diagram showing an example of division of a mesh into sub-meshes according to this embodiment. Specifically, Fig. 35 is a diagram showing how the mesh shown in Fig. 34 is divided into two sub-meshes.

[0267] Here, vertices A, B, and C of the original mesh are duplicated to vertices A1, B1, and C1, and vertices A2, B2, and C2, and two submeshes (the first submesh and the second submesh) that can be encoded and decoded independently are created (generated). The first submesh and the second submesh are meshes that can be decoded independently.

[0268] In this way, a mesh can be divided into a plurality of smaller parts and coded for each part. When dividing the mesh, the vertices of the mesh are divided in such a way that the coordinates and connectivity information of the vertices in each part can be coded independently.

[0269] The mesh shown in FIG. 34 is an original mesh, which is sometimes called a full mesh in contrast to a sub-mesh.

[0270] <Calculation of Adjacent Points> Next, a method for encoding and decoding the prediction information output from the vertex geometry coordinate predictor 632 shown in FIG. 31 will be described in detail.

[0271] In the following description, an example will be given in which the prediction information is the motion vectors of the vertices (in other words, three-dimensional points) included in the base mesh.

[0272] The prediction information is not necessarily limited to a motion vector, but may be other information possessed by a three-dimensional point, such as position information (geometry) or attribute information (attribute) possessed by a three-dimensional point.

[0273] Here, the position information includes, for example, coordinates (x coordinate, y coordinate, z coordinate) based on a certain point. Also, the attribute information includes, for example, color information (RGB, YUV, etc.), reflectance, normal vector, etc. of each three-dimensional point. Note that the attribute information may be information expressed as a vector.

[0274] Furthermore, when the prediction information output by the vertex geometry coordinate predictor 632 is a motion vector, the vertex geometry coordinate predictor 632 may be referred to as a motion decoder.

[0275] In the following description, the motion vector (specifically, the value of the motion vector) is an integer value. For example, if the motion vector has 8-bit precision, the motion vector takes an integer value from 0 to 255, and if the motion vector has 10-bit precision, the motion vector takes an integer value from 0 to 1023.

[0276] If the bit precision of the motion vector is decimal precision, the motion vector may be multiplied by a scale value to obtain an integer value, and then rounded to an integer value.

[0277] This scale value may be added to the bitstream, such as in a header.

[0278] FIG. 36 is an explanatory diagram showing the positional relationship of three-dimensional points according to this embodiment.

[0279] A possible method for encoding a motion vector of a three-dimensional point is to calculate a predicted value of the motion vector of a certain three-dimensional point and encode the difference (prediction residual) between the value of the original motion vector and the predicted value. For example, if the value of the motion vector of a certain three-dimensional point p is Ap and the predicted value is Pp, the encoding device 100 encodes the absolute difference value Diffp = |Ap - Pp|, which indicates the absolute value of the difference. In this case, if the predicted value Pp can be generated with high accuracy, the value of the absolute difference value Diffp will be small. Therefore, for example, the encoding device 100 can reduce the amount of code by performing entropy encoding using a coding table in which the smaller the value, the smaller the number of generated bits.

[0280] As a method for the encoding device 100 to generate a predicted value of a motion vector, it is possible to use the motion vector of another 3D point that is located around the 3D point to be encoded. Here, a 3D point that is located around the 3D point refers to another 3D point that is located within a predetermined distance (within a predetermined range) from the 3D point. For example, if there are 3D points p=(x1, y1, z1) and 3D points q=(x2, y2, z2) that are the 3D points to be encoded, the Euclidean distance d(p, q) between the 3D points p and q is √((x1-y1) ２ +(x2-y2) ２ +(x3-y3) ２ ) is smaller than a certain threshold THd, the encoding device 100 determines that the position of the 3D point q is close to the position of the 3D point p, and determines to use the value of the motion vector of the 3D point q to generate a predicted value of the motion vector of the 3D point p.

[0281] The distance calculation method may be another method, such as Mahalanobis distance.

[0282] The predetermined distance may be determined arbitrarily and is not particularly limited.

[0283] Furthermore, for example, the encoding device 100 may determine that a 3D point that is farther away from the 3D point to be encoded than a predetermined distance (outside a predetermined range) is not to be used for prediction. For example, if a 3D point r exists and the distance d(p, r) between the 3D point p and the 3D point r is equal to or greater than a threshold THd, the encoding device 100 may determine that the 3D point r is not to be used for prediction.

[0284] The encoding device 100 may add the value of the threshold THd to the header of the bitstream.

[0285] For example, when encoding the motion vector of a three-dimensional point to be encoded using a predicted value, the encoding device 100 uses motion vectors of surrounding three-dimensional points to be used to generate the predicted value, and uses motion vectors that have already been encoded or motion vectors that have already been decoded.

[0286] In addition, when decoding the motion vector of the three-dimensional point to be decoded using a predicted value, the decoding device 200 uses motion vectors that have already been decoded when using the motion vectors of surrounding three-dimensional points to generate the predicted value.

[0287] This allows the same predicted value to be generated during encoding and decoding, thereby enabling the decoding device 200 to correctly decode the bit stream of 3D points generated by the encoding device 100.

[0288] Although it has been described that the points surrounding a 3D point refer to other 3D points within a predetermined range from the 3D point, this is not necessarily limited to this. For example, in the case of 3D point D (i.e., vertex D) shown in FIG. 33, the surrounding 3D points include 3D points A, B, C, E, F, and G. However, the surrounding 3D points (in other words, adjacent points) may be selected according to one or more of the following conditions A and B. In other words, adjacent points are points selected according to certain conditions and are points that are referenced to predict the information of the 3D point to be encoded. An adjacent point may also be referred to as a reference 3D point, a reference point, or a reference vertex.

[0289] Condition A: A 3D point that has connectivity with the target 3D point. Condition B: A 3D point that has been encoded or decoded before the target 3D point.

[0290] For example, when 3D points that satisfy the above conditions A and B are selected as adjacent points, and the 3D points are encoded or decoded in the order of 3D point A, 3D point B, 3D point C, 3D point D, 3D point E, 3D point F, and 3D point G, 3D point A and 3D point C may be selected as adjacent points of 3D point D. Because 3D point A and 3D point C have connectivity with 3D point D, it is highly likely that their motion vectors are also close in value. Furthermore, because 3D point A and 3D point C have been encoded or decoded before 3D point D, the motion vectors of 3D point A and 3D point C can be used to calculate a predicted value of the motion vector of 3D point D.

[0291] This makes it possible to improve the accuracy of the predicted value of the motion vector of the three-dimensional point D, thereby improving the coding efficiency.

[0292] In addition to the above conditions A and B, the number of neighboring points of a three-dimensional point may be limited to a predetermined value (NumNeiCnt) or less. For example, by setting NumNeiCnt = 3, the number of neighboring points of a three-dimensional point may be limited to 3 or less.

[0293] This makes it possible to reduce the memory capacity required to store information about adjacent points of a three-dimensional point, and also to reduce the amount of processing required to predict (calculate) a motion vector.

[0294] The predetermined value may be determined arbitrarily and is not particularly limited.

[0295] Furthermore, for example, the encoding device 100 may add the above-mentioned predetermined value, in other words, NumNeiCnt indicating the maximum number of adjacent points, to the header of the data unit and encode it to add it to the bitstream.

[0296] As a result, the decoding device 200 can properly decode a bitstream in which the maximum number of adjacent points is limited to NumNeiCnt or less by decoding the header of the bitstream.

[0297] In addition, when there are more three-dimensional points satisfying the above conditions A and B than NumNeiCnt as adjacent points, the adjacent points may be selected in order of proximity to the three-dimensional point to be encoded or decoded. For example, when NumNeiCnt = 3, there are five three-dimensional points satisfying the above conditions A and B as adjacent points of three-dimensional point D, namely, three-dimensional point A, three-dimensional point C, three-dimensional point H, three-dimensional point I, and three-dimensional point J, and when the distance to three-dimensional point D is closest in order of three-dimensional point A, three-dimensional point C, three-dimensional point H, three-dimensional point I, and three-dimensional point J, three-dimensional point A, three-dimensional point C, and three-dimensional point H may be selected as adjacent points of three-dimensional point D. Because three-dimensional point A, three-dimensional point C, and three-dimensional point H have connectivity with three-dimensional point D and are close to three-dimensional point D, the values ​​of their respective motion vectors are also likely to be close to the value of the motion vector of three-dimensional point D. Furthermore, 3D point A, 3D point C, and 3D point H have been coded or decoded before 3D point D. Therefore, the motion vectors of 3D point A, 3D point C, and 3D point H can be used to calculate a predicted value of the motion vector of 3D point D.

[0298] This makes it possible to improve the accuracy of the predicted value of the motion vector of the 3D point D. Furthermore, by limiting the number of adjacent points, it is possible to reduce the memory capacity for storing information on adjacent points of the 3D point and also reduce the amount of processing required when calculating (predicting) the motion vector.

[0299] Note that when using connectivity with a target 3D point as condition A when selecting adjacent points of a 3D point to be encoded or decoded (hereinafter also referred to as a target 3D point), it is not limited to using connectivity within the frame to be encoded or decoded (hereinafter also referred to as a target frame). For example, connectivity within a frame that has already been encoded or decoded may be used. For example, in the example shown in FIG. 33, when adjacent points of each 3D point (each target 3D point) in the frame at time (t) (current frame) are selected using the above condition A, the corresponding three-dimensional connectivity within the frame at time (t-1) (past frame) may be used. More specifically, when selecting adjacent points of 3D point D in the current frame using the above condition A, the encoding device 100 and the decoding device 200 may select 3D points A, C, and G with reference to the connectivity of 3D point D in the past frame, and select 3D points A and C that have already been encoded or decoded as adjacent points. In frames that have been coded or decoded before the target frame, such as past frames, the coding device 100 and the decoding device 200 can calculate the connectivity and distance between three-dimensional points, and can therefore appropriately calculate adjacent points of the target three-dimensional point using the above condition A or the distance (distance information) between three-dimensional points.

[0300] In this embodiment, an example has been shown in which a past frame is used as the frame before the target frame, but this is not necessarily limited to this, and any frame that has already been coded or decoded can be used.

[0301] This allows the encoding device 100 and the decoding device 200 to appropriately calculate the neighboring points of the target 3D point using connectivity and / or distance.

[0302] Note that this embodiment may be applied when there is correspondence between 3D points in a target frame and 3D points in frames that have already been encoded or decoded. For example, in the example shown in Fig. 33, there is correspondence between 3D points A, B, C, D, E, F, and G in the current frame and the previous frame, so neighboring points of the 3D points in the current frame may be calculated using the connectivity and / or distance in the previous frame shown in this embodiment.

[0303] In addition, if there is no correspondence between a three-dimensional point in the target frame and a three-dimensional point in a frame that has already been encoded or decoded, the encoding device 100 and the decoding device 200 may calculate (select) adjacent points using the connectivity of the three-dimensional points in the target frame without using distance.

[0304] This allows calculation of adjacent points even when there is no correspondence between each of the three-dimensional points in the coded or decoded frame.

[0305] In addition, the encoding device 100 may add information to the bitstream indicating whether or not there is a correspondence between each of the three-dimensional points in the frame to be encoded and each of the three-dimensional points in frames that have already been encoded or decoded.

[0306] This allows the decoding device 200 to know whether or not there is correspondence between 3D points in a frame to be coded (i.e., a frame coded by the coding device 100 and to be decoded by the decoding device 200) and 3D points in frames that have already been decoded. The decoding device 200 can switch the method of calculating adjacent points so that, for example, if there is correspondence between the 3D points, it calculates adjacent points of the 3D points in the frame to be coded using the connectivity and / or distance in the already decoded frame, and if there is no correspondence between the 3D points, it calculates adjacent points using the connectivity of the 3D points in the frame to be coded without using the distance.

[0307] In addition, when decoding, if the decoding device 200 is unable to calculate the distance between the three-dimensional point to be decoded in the frame to be decoded and an adjacent point before decoding the position information of the three-dimensional point to be decoded, it may calculate the adjacent point of the three-dimensional point to be decoded using the distance between three-dimensional points corresponding to the three-dimensional point to be decoded in a frame that has already been decoded.

[0308] FIG. 37 is an explanatory diagram showing the distance between three-dimensional points according to this embodiment.

[0309] For example, in the example shown in FIG. 37 , the distance between each 3D point in the current frame at time (t) and its adjacent point may be the distance between the corresponding 3D point in the past frame at time (t−1) and its adjacent point. More specifically, the distance between 3D point D in the current frame and adjacent point A, adjacent point C, and adjacent point G may be the distance between 3D point D in the past frame and adjacent point A, adjacent point C, and adjacent point G. Since the decoding device 200 can reliably calculate the distance between 3D points in frames that have been decoded before the frame to be decoded, such as past frames, it can use the distance to appropriately calculate the adjacent points of the 3D point to be decoded.

[0310] In this embodiment, an example has been shown in which a past frame is used as a frame preceding the frame to be decoded, but this is not necessarily limited to this, and any frame that has already been decoded may be used.

[0311] This allows the decoding device 200 to appropriately calculate adjacent points that are close to the 3D point to be decoded, thereby enabling the decoding device 200 to accurately calculate (predict) the 3D motion vector to be decoded, thereby improving coding efficiency.

[0312] In addition, when the decoding device 200 calculates, during decoding, the adjacent points of the three-dimensional point to be decoded using the distance between three-dimensional points corresponding to the three-dimensional point to be decoded within a frame that has already been decoded, the encoding device 100 may also calculate, during encoding, the adjacent points of the three-dimensional point to be encoded using the distance between three-dimensional points corresponding to the three-dimensional point to be encoded within a frame that has already been encoded.

[0313] This allows the method of calculating adjacent points to be the same during encoding and decoding, allowing the decoding device 200 to appropriately decode the bit stream generated by encoding.

[0314] In addition, the encoding device 100 and the decoding device 200 may also calculate the connectivity of a target 3D point using the connectivity between 3D points corresponding to the target 3D point within a frame that has already been encoded or decoded.

[0315] This allows the connectivity and distance to be calculated simultaneously using information on frames that have already been coded or decoded, thereby reducing the amount of processing.

[0316] In addition, the encoding device 100 and the decoding device 200 may select appropriate neighboring points by using connectivity within the target frame and the distance between 3D points corresponding to the target 3D point in an already encoded or decoded frame.

[0317] This allows the encoding device 100 and the decoding device 200 to calculate adjacent points that have connectivity within the target frame and are close to each other using information within frames that have already been encoded or decoded, thereby enabling the motion vector of the target 3D point to be calculated (predicted) with high accuracy and improving encoding efficiency.

[0318] When the adjacent points are calculated without using the distance, the adjacent points may be calculated using the connectivity within the target frame, which can reduce the amount of processing.

[0319] Furthermore, when distance is not used and the number of neighboring points is limited by NumNeiCnt, the encoding device 100 and the decoding device 200 may stop calculating neighboring points when the number of neighboring points reaches NumNeiCnt while adding neighboring points of the target 3D point. This reduces the amount of processing.

[0320] Furthermore, when the number of neighboring points reaches NumNeiCnt while adding neighboring points of the target 3D point, the encoding device 100 and the decoding device 200 may replace at least one of the neighboring points already stored as neighboring points with a newly found neighboring point in subsequent processing. This makes it possible to improve encoding efficiency while limiting the number of neighboring points.

[0321] Fig. 38 is a flow diagram showing the adjacent point selection process according to this embodiment. The flow shown in Fig. 38 is a specific example of a processing procedure executed by the encoding device 100 and the decoding device 200 when calculating adjacent points of a target 3D point.

[0322] First, the encoding device 100 and the decoding device 200 select, from among a plurality of 3D points included in a target frame, 3D points that have connectivity with a target 3D point as first adjacent point candidates (S101).

[0323] Next, the encoding device 100 and the decoding device 200 select, as a second adjacent point candidate, a 3D point that has been coded or decoded before the target 3D point from among the multiple first adjacent point candidates selected in step S101 (S102). For example, the encoding device 100 selects, as a second adjacent point candidate, a 3D point that has been coded before the target 3D point from among the multiple first adjacent point candidates. Also, for example, the decoding device 200 selects, as a second adjacent point candidate, a 3D point that has been decoded before the target 3D point from among the multiple first adjacent point candidates.

[0324] Next, the encoding device 100 and the decoding device 200 calculate the distance between the target 3D point and each of the multiple second adjacent point candidates selected in step S102 (S103).

[0325] Next, the encoding device 100 and the decoding device 200 select the adjacent point with the maximum number of adjacent points by selecting three-dimensional points with a number less than or equal to the maximum number of adjacent points (the above-mentioned NumNeiCnt) from the multiple second adjacent point candidates selected in step S102 in order of closest distance (S104).

[0326] FIG. 39 is an explanatory diagram showing the adjacent point selection process according to this embodiment. In the example shown in FIG. 39 , the target 3D point is 3D point f. Also, in the example shown in FIG. 39 , 3D point a, 3D point b, 3D point c, 3D point d, 3D point e, 3D point f, 3D point g, and 3D point h are encoded or decoded in this order. That is, in the example shown in FIG. 39 , 3D point a, 3D point b, 3D point c, 3D point d, and 3D point e are 3D points that have already been encoded or decoded. Also, in the example shown in FIG. 39 , 3D points that have connectivity are shown connected to each other by solid lines. Also, in the example shown in FIG. 39 , the distance between 3D point f and 3D point x (x is a, b, c, d, e, g, or h) is shown as D(x), and 3D point d, 3D point b, 3D point c, 3D point a, and 3D point e are closest to 3D point f in this order. In the example shown in FIG. 39, the maximum number of adjacent points (the above-mentioned NumNeiCnt) is three.

[0327] For example, in step S101, the encoding device 100 and the decoding device 200 select three-dimensional points a, b, c, d, e, g, and h as first adjacent point candidates, as shown in (a) of Figure 39.

[0328] Also, for example, in step S102, the encoding device 100 and the decoding device 200 select three-dimensional points a, b, c, d, and e as second adjacent point candidates, as shown in (b) of Figure 39.

[0329] Also, for example, in step S104, the encoding device 100 and the decoding device 200 select 3D points b, c, and d as first adjacent points, as shown in (c) of FIG.

[0330] 38 is an example, and the order in which steps S101 to S104 are executed may be arbitrarily interchanged. For example, if step S101 and step S102 are interchanged, the encoding device 100 and the decoding device 200 may select a 3D point that has been encoded or decoded before the target 3D point as a first adjacent point candidate by the processing of step S102, and may select a 3D point that has connectivity with the target 3D point from the first adjacent point candidates as a second adjacent point candidate by the processing of step S101. This improves the flexibility of implementation.

[0331] Furthermore, for example, some of the processes in steps S101 to S104 may be executed in parallel. For example, while the process in step S102 is being executed, the process in step S103 may be executed to calculate the distance between the target 3D point and the first adjacent point candidate in advance in parallel. This reduces the processing time.

[0332] Note that a motion group (MG) may be provided as a prediction unit according to the encoding order or decoding order. When encoding or decoding a motion vector of a three-dimensional point, the encoding device 100 and the decoding device 200 may encode or decode for each MG. For example, the number of three-dimensional points included in an MG (MGSize) may be defined, and the encoding device 100 and the decoding device 200 may divide the three-dimensional points into multiple MGs according to the encoding order or decoding order and encode or decode them.

[0333] Note that the coding order and decoding order of the motion vectors of the 3D points may be any order. For example, the encoding device 100 and the decoding device 200 may generate Level of Detail (hereinafter referred to as LoD) and encode or decode in order for each LoD layer. Alternatively, the encoding device 100 and the decoding device 200 may encode or decode the motion vectors in the coding order or decoding order of the position information of the 3D points (i.e., vertices) without generating LoD. Alternatively, the encoding device 100 and the decoding device 200 may generate Morton codes using the position information of the 3D points and encode or decode in the order of the Morton codes.

[0334] Next, specific examples of motion groups will be described using Figures 40 to 42. Note that in Figures 40, 42, and 43, MG0, MG1, and MGN are each examples of motion groups. Note that N is, for example, an integer equal to or greater than 2, but the number of motion groups may be two, or may be three or more. Furthermore, the multiple three-dimensional points (specifically, information on the three-dimensional points) indicated by circles in each figure are coded or decoded in order from left to right. In other words, the multiple three-dimensional points shown in each figure are coded or decoded in order starting from the three-dimensional point belonging to MG0. Furthermore, the three-dimensional points belonging to the same MG are coded or decoded in order from left to right.

[0335] FIG. 40 is a diagram showing a first example of a reference destination of a motion group according to this embodiment.

[0336] In the first example, 3D points belonging to the same motion group, in other words, 3D points within the same motion group, are defined as not being able to reference each other. That is, in the first example, the motion vectors of 3D points belonging to the same motion group as the target 3D point are not used to calculate the predicted value of the motion vector of the target 3D point. For example, 3D points within the same motion group are not added to adjacent points.

[0337] Furthermore, in the first example, the motion vector of a 3D point that belongs to a different motion group from the target 3D point is used to calculate the predicted value of the motion vector of the target 3D point. Specifically, in the first example, 3D points in a different motion group that have already been coded or decoded are defined as referenceable. In other words, in the first example, the motion vector of a 3D point that has already been coded or decoded, among 3D points that belong to a different motion group from the target 3D point, is used to calculate the predicted value of the motion vector of the target 3D point.

[0338] For example, in the example shown in Figure 40, in calculating the predicted value of the motion vector of the target 3D point belonging to MG1, the motion vector of the 3D point belonging to MG1 is not used, but the motion vector of the 3D point belonging to MG0 is used. Also, in the example shown in Figure 40, in calculating the predicted value of the motion vector of the target 3D point belonging to MG1, the motion vector of the 3D point belonging to MGN (specifically, MGN when N is an integer greater than or equal to 2) is not used.

[0339] Thus, for example, coded or decoded 3D points in different motion groups are added to their neighbors.

[0340] FIG. 41 is a diagram showing an example of the syntax of a base mesh header according to this embodiment.

[0341] As in the syntax shown in Fig. 41, the size (data size) of the motion group may be written in the header of the bitstream, etc. For example, when the size (MGSize) of the motion group is 16, the encoding device 100 may add MGSize = 16 to the header of the bitstream. ｎ (n: an integer equal to or greater than 0), and the encoding device 100 may add the value of n to the header of the bitstream.

[0342] The encoding device 100 and the decoding device 200 may encode or decode 3D points in the same motion group in parallel.

[0343] FIG. 42 is a diagram showing a second example of a reference destination of a motion group according to this embodiment.

[0344] In the second example, coded or decoded 3D points within the same motion group are defined as referable. Also in the second example, coded or decoded 3D points within different motion groups are defined as referable. Also in the second example, uncoded or undecoded 3D points are defined as unreferenceable. That is, in the second example, only motion vectors of 3D points that have already been coded or decoded are used to calculate the predicted value of the motion vector of the target 3D point. For example, coded or decoded 3D points within the same motion group may be added as neighboring points. Also, for example, coded or decoded 3D points within different motion groups may be added as neighboring points. On the other hand, for example, uncoded or undecoded 3D points cannot be added as neighboring points, whether within the same motion group or a different motion group.

[0345] For example, in the example shown in Figure 42, in calculating the predicted value of the motion vector of the target 3D point belonging to MG1, the motion vectors of the 3D points that have already been coded or decoded among the 3D points that belong to MG1 may be used, but the motion vectors of the 3D points that have not been coded or decoded are not used. Also, in the example shown in Figure 42, in calculating the predicted value of the motion vector of the target 3D point that belongs to MG1, the motion vectors of the 3D points that belong to MG0 may be used, but the motion vectors of the 3D points that belong to MGN (specifically, MGN when N is an integer greater than or equal to 2) are not used.

[0346] In the second example, the size of the motion group may also be written in the header of the bitstream. For example, if the size of the motion group (MGSize) is 16, the encoding device 100 may add MGSize=16 to the header of the bitstream. ｎ The encoding device 100 may add the value of n to the header of the bitstream.

[0347] In this way, by defining that already coded or decoded three-dimensional points can be referenced even for three-dimensional points within the same motion group, prediction accuracy can be improved, and coding efficiency can be improved.

[0348] FIG. 43 is a diagram showing a third example of a reference destination of a motion group according to this embodiment.

[0349] In the third example, coded or decoded 3D points within the same motion group are defined as referable. On the other hand, in the third example, uncoded or undecoded 3D points are defined as unreferenceable. For example, coded or decoded 3D points within the same motion group may be added as neighboring points. On the other hand, uncoded or undecoded 3D points within the same motion group may not be added as neighboring points.

[0350] Also, in the third example, 3D points in different motion groups are defined as not being able to be referenced, i.e., 3D points in different motion groups cannot be added to adjacent points.

[0351] For example, in the example shown in Figure 43, in calculating the predicted value of the motion vector of the target 3D point belonging to MG1, the motion vectors of the 3D points that have already been coded or decoded among the 3D points that belong to MG1 may be used, but the motion vectors of the 3D points that have not been coded or decoded are not used. Also, in the example shown in Figure 43, in calculating the predicted value of the motion vector of the target 3D point that belongs to MG1, the motion vectors of the 3D points that belong to motion groups other than MG1 are not used.

[0352] In the third example, the size of the motion group may also be written in the header of the bitstream. For example, if the size of the motion group (MGSize) is 16, the encoding device 100 may add MGSize=16 to the header of the bitstream. ｎ The encoding device 100 may add the value of n to the header of the bitstream.

[0353] In this way, by prohibiting references between motion groups and eliminating dependencies between motion groups, the encoding device 100 and the decoding device 200 can encode or decode information on three-dimensional points of multiple motion groups in parallel.

[0354] Furthermore, by defining three-dimensional points that are within the same motion group and that have already been coded or decoded as referenceable in this manner, prediction accuracy can be improved, and coding efficiency can be improved.

[0355] The number of 3D points belonging to each motion group may be determined arbitrarily and is not particularly limited. Furthermore, the number of 3D points belonging to each motion group may be the same for each group or may be different for each group.

[0356] In addition, when a full mesh, which is a mesh before being divided into one or more submeshes, is divided into one or more submeshes and then encoded or decoded, the encoding device 100 and the decoding device 200 may divide the three-dimensional points in each submesh into motion groups according to the encoding order or decoding order, and encode or decode the motion vector of each three-dimensional point for each motion group.

[0357] Fig. 44 is an explanatory diagram showing the relationship between the vertices constituting a mesh (original mesh) and motion groups according to this embodiment. Fig. 45 is an explanatory diagram showing the relationship between the vertices constituting submeshes (first submesh and second submesh) and motion groups according to this embodiment. Note that the first submesh and second submesh shown in Fig. 45 are meshes generated by dividing the original mesh shown in Fig. 44.

[0358] 44 and 45, the 3D points A, B, and C constituting the original mesh (full mesh) are divided into sub-meshes, and are duplicated into A1, B1, and C1 constituting the first sub-mesh and A2, B2, and C2 constituting the second sub-mesh, respectively. For example, the encoding device 100 may assign the motion vectors of the 3D points A1, B1, C1, A2, B2, and C2 to motion groups within each sub-mesh, and encode them using the method shown in the above example.

[0359] This allows the encoding device 100 to assign the motion vectors of the three-dimensional points of each submesh to motion groups within the submesh, while selecting appropriate adjacent points from the three-dimensional points within the submesh to encode the motion vectors.

[0360] Note that 3D points that belong to a submesh different from the 3D point to be encoded may not be included in the adjacent points. This allows each submesh to be encoded or decoded independently without referring to information between the submeshes.

[0361] Alternatively, 3D points belonging to different submeshes may not be included in the same motion group, which prevents reference to information between submeshes and allows the encoding device 100 and the decoding device 200 to encode or decode each submesh independently.

[0362] As described above, for example, in the process of predicting information about vertices (target 3D points) that make up a 3D mesh, the encoding device 100 and the decoding device 200 determine the vertices (adjacent points) to reference using distance information (information indicating the distance between 3D points).

[0363] Furthermore, for example, the vertex information may be a motion vector of the vertex coordinates. Note that the vertex information may be any information that a three-dimensional point has, such as position information or attribute information.

[0364] Furthermore, for example, the prediction process is an inter-prediction process.

[0365] Furthermore, for example, the encoding device 100 and the decoding device 200 determine combinations of a plurality of adjacent points.

[0366] Furthermore, for example, the distance information is a difference value between the coordinates of the processing target point (target three-dimensional point) and the coordinates of the adjacent point.

[0367] Furthermore, for example, the encoding device 100 and the decoding device 200 determine, as adjacent points, vertices whose difference values ​​are equal to or less than a predetermined value.

[0368] Furthermore, for example, the encoding device 100 and the decoding device 200 select a predetermined number of vertices in ascending order of difference value as adjacent points.

[0369] Also, for example, the predetermined number is coded into the bitstream.

[0370] Furthermore, for example, the distance information is calculated using information of the reference frame.

[0371] Furthermore, for example, the encoding device 100 and the decoding device 200 derive distance information using a point included in the reference frame and corresponding to the processing target point.

[0372] Also, for example, the reference frame is a frame that precedes the processing target frame in display order.

[0373] Also, for example, the reference frame is a frame that precedes the frame to be processed in coding order or decoding order.

[0374] Furthermore, for example, information other than the distance is derived using information on the processing target frame.

[0375] Furthermore, for example, the encoding device 100 and the decoding device 200 select points having connectivity using processing target points included in the processing target frame.

[0376] Furthermore, for example, the encoding device 100 and the decoding device 200 determine adjacent points using other information in addition to the distance information. Note that one or more pieces of other information used together with the distance information may be used in any combination.

[0377] Furthermore, for example, the encoding device 100 and the decoding device 200 determine, as adjacent points, vertices that have connectivity with the processing target point.

[0378] Furthermore, for example, the encoding device 100 and the decoding device 200 determine, as adjacent points, vertices that have been coded or decoded before the processing target point. For example, the encoding device 100 determines, as adjacent points, vertices that have been coded before the processing target point (the three-dimensional point to be coded). For example, the decoding device 200 determines, as adjacent points, vertices that have been decoded before the processing target point (the three-dimensional point to be decoded).

[0379] Furthermore, for example, the encoding device 100 and the decoding device 200 determine, as adjacent points, vertices that belong to the same submesh as the processing target point.

[0380] Furthermore, for example, the encoding device 100 and the decoding device 200 determine, as adjacent points, vertices that belong to the same motion group as the processing target point.

[0381] Furthermore, for example, the encoding device 100 and the decoding device 200 determine, as adjacent points, vertices that belong to a different motion group from the point to be processed.

[0382] Furthermore, for example, when the number of vertices that are candidates for adjacent points exceeds a predetermined number, the encoding device 100 and the decoding device 200 select a predetermined number of vertices from the candidate vertices using at least one of the above methods.

[0383] <Representative Example> Fig. 46 is a flow diagram showing an example of basic encoding processing according to this embodiment. For example, the circuit 151 of the encoding device 100 shown in Fig. 24 performs the encoding processing shown in Fig. 46 in operation.

[0384] The encoding device 100 executes an encoding method for encoding information on three-dimensional points in a target frame to be encoded.

[0385] First, the encoding device 100 selects one or more reference 3D points from among a plurality of 3D points in the target frame (S201).

[0386] Next, the encoding device 100 calculates a predicted value of the second information of the target 3D point to be encoded in the target frame, using the first information of each of the one or more reference 3D points (S202).

[0387] Here, in the step of selecting one or more reference 3D points (S201), encoding device 100 selects one or more reference 3D points based on the distance between the target 3D point and each of the plurality of 3D points.

[0388] The first information and the second information are, for example, information indicating a motion vector (specifically, prediction information). The first information and the second information may each be any information possessed by a 3D point, such as position information or attribute information. The reference 3D point is, for example, the adjacent point described above. The 3D point is, for example, the vertex described above. That is, the reference 3D point is a point in a 3D point cloud or a 3D mesh that is referenced to predict information about a 3D point (vertex) to be encoded, and is selected according to a condition. The condition may be, for example, the condition described above. A single condition or a combination of multiple conditions may be used to select the reference 3D point. The multiple 3D points in the target frame and the target 3D point are, for example, vertices constituting a 3D mesh included in the target frame and 3D points constituting a 3D point cloud, respectively. The target frame is, for example, the current frame described above. The information about the multiple 3D points and the target 3D point may not include connectivity information. That is, the 3D point cloud encoded by the encoding device 100 may or may not be a 3D mesh.

[0389] It is considered that the closer the distance between a 3D point and the target 3D point, the closer the information about the 3D point is. Therefore, for example, the distance from the target 3D point may be used as a condition for selecting a reference 3D point. For example, it is considered that the prediction residual can be reduced by using a 3D point that is close to the target 3D point as the reference 3D point to calculate a predicted value. If the prediction residual can be reduced, for example, the amount of code of a bitstream including information indicating the prediction residual can be reduced. Therefore, according to the encoding device 100, the amount of code can be reduced by selecting one or more reference 3D points based on the distance between the target 3D point and each of multiple 3D points.

[0390] Furthermore, for example, the encoding device 100 calculates a prediction residual, which is the difference between the value indicated by the second information and the predicted value, and generates a bitstream including prediction residual information indicating the calculated prediction residual. For example, the encoding device 100 calculates the prediction residual after performing step S202, and further generates a bitstream.

[0391] The prediction residual is, for example, the above-mentioned absolute difference Diffp, and the prediction residual information is, for example, information indicating the absolute difference Diffp.

[0392] This allows the encoding device 100 to generate a bitstream with a reduced amount of code.

[0393] Also, for example, the first information of each of the one or more reference 3D points indicates a motion vector of each of the one or more reference 3D points, and the second information indicates a motion vector of the target 3D point.

[0394] Specifically, the first information is a 3D point in a reference frame, and is information indicating a motion vector indicating the amount of change from the coordinates of a 3D point corresponding to the reference 3D point to the coordinates of the reference 3D point in the target frame. The second information is a 3D point in the reference frame, and is information indicating a motion vector indicating the amount of change from the coordinates of a 3D point corresponding to the target 3D point to the coordinates of the target 3D point in the target frame. The first information and the second information are, for example, the prediction information described above. The reference frame is, for example, the past frame described above.

[0395] This allows the encoding device 100 to encode motion vectors.

[0396] Furthermore, for example, in calculating the predicted value (S202), the encoding device 100 calculates the predicted value using inter prediction. That is, the encoding device 100 calculates the predicted value using information on a frame at a time different from that of the target frame.

[0397] This allows the encoding device 100 to calculate a predicted value.

[0398] Furthermore, for example, when selecting one or more reference 3D points, the encoding device 100 calculates the distance by calculating the difference between the coordinates of the target 3D point and the coordinates of each of the multiple 3D points.

[0399] This allows the encoding device 100 to calculate the distance between the target 3D point and each of the multiple 3D points.

[0400] Furthermore, for example, when selecting one or more reference 3D points, encoding device 100 selects, from among a plurality of 3D points, 3D points whose distance is equal to or less than a predetermined value as the one or more reference 3D points.

[0401] The predetermined value is, for example, the threshold value THd described above. The predetermined value may be determined arbitrarily in advance and is not particularly limited.

[0402] This allows the encoding device 100 to select a reference 3D point that is closest to the target 3D point from among a plurality of 3D points.

[0403] Also, for example, when selecting one or more reference three-dimensional points, the encoding device 100 selects a predetermined number of three-dimensional points from a plurality of three-dimensional points in order of shortest distance, thereby selecting them as one or more reference three-dimensional points.

[0404] This allows the encoding device 100 to select an appropriate number of reference 3D points for calculating the predicted value.

[0405] Furthermore, for example, the encoding device 100 generates a bitstream including predetermined number information indicating a predetermined number. For example, the encoding device 100 generates a bitstream including prediction residual information and the predetermined number information.

[0406] The predetermined number is, for example, the above-mentioned maximum number of adjacent points (NumNeiCnt).

[0407] This allows the decoding device 200 to select a reference 3D point using a predetermined amount of information acquired from the bitstream.

[0408] Furthermore, for example, when selecting one or more reference 3D points, the encoding device 100 calculates the distance using the coordinates of a 3D point in the reference frame that corresponds to the target 3D point.

[0409] This allows the decoding device 200 to calculate the distance in the same way as the encoding device 100, without decoding the coordinates of the target 3D point in the target frame.

[0410] Also, for example, the reference frame is a frame that precedes the target frame in display order.

[0411] This allows the encoding device 100 to encode the target frame using a frame that is displayed on the display before the target frame, in other words, a past frame.

[0412] Also, for example, the reference frame is a frame that precedes the target frame in coding order.

[0413] This allows the encoding device 100 to encode the current frame using an already encoded frame.

[0414] Furthermore, for example, the encoding device 100 selects one or more reference 3D points using distance and information other than distance. In other words, information other than the distance from the target 3D point may be used as a condition used for selecting a reference 3D point.

[0415] The information other than the distance may be, for example, connection information (connectivity). The information other than the distance may be, for example, the threshold value THd, the NumNeiCnt, and / or information about the motion group.

[0416] According to this, by appropriately selecting information other than the distance, the encoding device 100 can further reduce the amount of code.

[0417] Furthermore, for example, the information other than the distance is connection information indicating whether the target three-dimensional point is connected to each of the multiple three-dimensional points, and for example, when selecting one or more reference three-dimensional points, the encoding device 100 selects, from the multiple three-dimensional points, three-dimensional points that are connected to the target three-dimensional point as the one or more reference three-dimensional points.

[0418] In the case of connected 3D points, the information of the 3D points is considered to be closer than that of unconnected 3D points. Therefore, for example, by using a 3D point connected to the target 3D point as a reference 3D point to calculate a predicted value, it is considered that the prediction residual can be reduced, and therefore the encoding device 100 can further reduce the amount of code.

[0419] Fig. 47 is a flow diagram showing an example of basic decoding processing according to this embodiment. For example, the circuit 251 of the decoding device 200 shown in Fig. 25 performs the decoding processing shown in Fig. 47 in operation.

[0420] The decoding device 200 executes a decoding method for decoding information on three-dimensional points in a target frame to be decoded.

[0421] First, the decoding device 200 selects one or more reference 3D points from among a plurality of 3D points in the target frame (S301).

[0422] Next, the decoding device 200 calculates a predicted value of the second information of the target 3D point to be decoded in the target frame, using the first information of each of the one or more reference 3D points (S302).

[0423] Here, in the step of selecting one or more reference 3D points (S301), decoding device 200 selects one or more reference 3D points based on the distance between the target 3D point and each of the plurality of 3D points.

[0424] It is considered that the closer the distance between the three-dimensional points, the closer the information about the three-dimensional points will be. Therefore, for example, it is considered that the prediction residual can be reduced by using a three-dimensional point that is close to the target three-dimensional point as a reference three-dimensional point to calculate a predicted value. If the prediction residual can be reduced, for example, the amount of coding of a bitstream including information indicating the prediction residual can be reduced. Therefore, according to the decoding device 200, by selecting one or more reference three-dimensional points based on the distance between the target three-dimensional point and each of the multiple three-dimensional points, it is possible to decode the information about the three-dimensional point using information with a reduced amount of coding.

[0425] Furthermore, for example, the decoding device 200 acquires prediction residual information indicating a prediction residual from the bitstream, and calculates the second information based on the prediction residual and the predicted value. For example, the decoding device 200 calculates the second information using the prediction residual and the predicted value after step S302. The timing at which the decoding device 200 acquires the prediction residual information may be any timing as long as it is before the calculation of the second information.

[0426] This allows the decoding device 200 to decode information about three-dimensional points using information about a bitstream with a reduced amount of code.

[0427] Also, for example, the first information of each of the one or more reference 3D points indicates a motion vector of each of the one or more reference 3D points, and the second information indicates a motion vector of the target 3D point.

[0428] This allows the decoding device 200 to decode the motion vector.

[0429] Furthermore, for example, in calculating the predicted value (S302), the decoding device 200 calculates the predicted value using inter prediction.

[0430] This allows the decoding device 200 to calculate a predicted value.

[0431] Furthermore, for example, when selecting one or more reference 3D points, the decoding device 200 calculates the distance by calculating the difference between the coordinates of the target 3D point and the coordinates of each of the multiple 3D points.

[0432] This allows the decoding device 200 to calculate the distance between the target 3D point and each of the multiple 3D points.

[0433] Furthermore, for example, when selecting one or more reference 3D points, decoding device 200 selects, from among a plurality of 3D points, 3D points whose distance is equal to or less than a predetermined value as one or more reference 3D points.

[0434] This allows the decoding device 200 to select a reference 3D point that is closest to the target 3D point from among a plurality of 3D points.

[0435] Also, for example, when selecting one or more reference three-dimensional points, the decoding device 200 selects a predetermined number of three-dimensional points from a plurality of three-dimensional points in order of shortest distance, thereby selecting them as one or more reference three-dimensional points.

[0436] This allows the decoding device 200 to select an appropriate number of reference 3D points for calculating a predicted value.

[0437] Alternatively, for example, the decoding device 200 may obtain predetermined number information indicating the predetermined number from the bitstream. For example, the decoding device 200 obtains the predetermined number information from the bitstream before step S301.

[0438] This allows the decoding device 200 to select an appropriate number of reference 3D points for calculating a predicted value using the predetermined number of pieces of information acquired from the bitstream.

[0439] Furthermore, for example, when selecting one or more reference 3D points, the decoding device 200 calculates the distance using the coordinates of a 3D point in the reference frame that corresponds to the target 3D point.

[0440] This allows the decoding device 200 to calculate the distance in the same way as the encoding device 100, without decoding the coordinates of the target 3D point in the target frame.

[0441] Also, for example, the reference frame is a frame that precedes the target frame in display order.

[0442] This allows the decoding device 200 to decode the target frame using a frame that is displayed on the display device earlier than the target frame, in other words, a past frame.

[0443] Also, for example, the reference frame is a frame that precedes the target frame in decoding order.

[0444] This allows the decoding device 200 to decode the target frame using a previously decoded frame.

[0445] Furthermore, for example, when selecting one or more reference 3D points, the decoding device 200 selects one or more reference 3D points using distance and information other than distance.

[0446] According to this, by appropriately selecting information other than distance, the decoding device 200 can decode the information of the three-dimensional point using information with a further reduced amount of coding.

[0447] Furthermore, for example, the information other than the distance is connection information indicating whether the target 3D point is connected to each of the multiple 3D points, and for example, when selecting one or more reference 3D points, the decoding device 200 selects, from the multiple 3D points, a 3D point that is connected to the target 3D point as the one or more reference 3D points.

[0448] In the case of connected 3D points, the information of the 3D points is considered to be closer than that of unconnected 3D points. Therefore, for example, by using a 3D point connected to the target 3D point as a reference 3D point to calculate a predicted value, it is considered possible to reduce the prediction residual, and the decoding device 200 can decode the information of the 3D point using information with a further reduced amount of code.

[0449] <Other Examples> Although aspects of the encoding device 100 and the decoding device 200 have been described above according to the embodiments, the aspects of the encoding device 100 and the decoding device 200 are not limited to the embodiments. Modifications conceivable by those skilled in the art may be applied to the embodiments, and multiple components in the embodiments may be combined in any manner.

[0450] For example, a process performed by a specific component in the embodiment may be performed by another component instead of the specific component. Also, the order of multiple processes may be changed, or multiple processes may be performed in parallel.

[0451] Furthermore, as described above, at least some of the configurations of the present disclosure may be implemented as an integrated circuit. At least some of the processes of the present disclosure may be used as an encoding method or a decoding method. A program for causing a computer to execute the encoding method or the decoding method may be used. A non-transitory computer-readable recording medium on which the program is recorded may be used. A bitstream for causing the decoding device 200 to perform a decoding process may be used.

[0452] Furthermore, at least some of the configurations and processes of the present disclosure may be used as a transmitting device, a receiving device, a transmitting method, or a receiving method. A program for causing a computer to execute the transmitting method or the receiving method may be used. Furthermore, a non-transitory computer-readable recording medium on which the program is recorded may be used.

[0453] The present disclosure is useful, for example, in encoding devices, decoding devices, transmitting devices, receiving devices, etc. related to three-dimensional meshes, and is applicable to computer graphics systems, three-dimensional data display systems, etc.

[0454] 100 Encoding device 101, 121, 144 Vertex information encoder 102, 145 Connection information encoder 103, 122 Attribute information encoder 104, 204, 521 Preprocessor 105, 205, 623 Postprocessor 110 Three-dimensional data encoding system 111, 211 Controller 112, 212 Input / output processor 113 Three-dimensional data encoder 114 System multiplexer 115 Three-dimensional data generator 123 Metadata encoder 124 Multiplexer 131 Vertex image generator 132 Attribute image generator 133 Metadata generator 134 Video encoder 141 Two-dimensional data encoder 142 Mesh data encoder 143 Texture encoder 148 Description encoder 151, 251 Circuit 152, 252 Memory 200 Decoding device 201, 221, 244 Vertex information decoder 202, 245 Connection information decoder 203, 222 Attribute information decoder 210 Three-dimensional data decoding system 213 Three-dimensional data decoder 214 System demultiplexer 215, 247 Presentation device 216 User interface 223 Metadata decoder 224 Demultiplexer 231 Vertex information generator 232 Attribute information generator 234 Video decoder 241 Two-dimensional data decoder 242 Mesh data decoder 243 Texture decoder 246 Mesh reconstructor 248 Description decoder 300 Network 310 External connection device 631 Frame header decoder 632 Vertex geometry coordinate predictor 633 Vertex geometry coordinate difference decoder 634 Reconstructor

Claims

1. An encoding method for encoding information of a three-dimensional point in a target frame to be encoded, comprising the steps of: selecting one or more reference three-dimensional points from a plurality of three-dimensional points in the target frame; calculating a predicted value of second information of a target three-dimensional point to be encoded in the target frame using first information of each of the one or more reference three-dimensional points; and selecting the one or more reference three-dimensional points based on a distance between the target three-dimensional point and each of the plurality of three-dimensional points.

2. The encoding method according to claim 1, further comprising the steps of: calculating a prediction residual which is a difference between a value indicated by the second information and the predicted value; and generating a bitstream including prediction residual information indicating the calculated prediction residual.

3. The encoding method of claim 1, wherein the first information for each of the one or more reference three-dimensional points indicates a motion vector for each of the one or more reference three-dimensional points, and the second information indicates a motion vector for the target three-dimensional point.

4. The encoding method according to claim 1, wherein the calculation of the predicted value comprises calculating the predicted value using inter prediction.

5. The encoding method according to claim 1, wherein in selecting the one or more reference 3D points, the distance is calculated by calculating a difference between the coordinates of the target 3D point and the coordinates of each of the plurality of 3D points.

6. The encoding method according to claim 1, wherein in selecting the one or more reference three-dimensional points, a three-dimensional point whose distance is equal to or less than a predetermined value is selected from the plurality of three-dimensional points as the one or more reference three-dimensional points.

7. The encoding method according to claim 1, wherein the one or more reference three-dimensional points are selected by selecting a predetermined number of three-dimensional points from the plurality of three-dimensional points in order of smallest distance from the plurality of three-dimensional points as the one or more reference three-dimensional points.

8. The encoding method according to claim 7, further comprising the step of generating a bit stream including predetermined number information indicating said predetermined number.

9. The encoding method according to claim 1, wherein in selecting the one or more reference 3D points, the distance is calculated using coordinates of a 3D point in a reference frame that corresponds to the target 3D point.

10. The encoding method according to claim 9, wherein the reference frame is a frame preceding the target frame in display order.

11. The encoding method according to claim 9, wherein the reference frame is a frame preceding the target frame in encoding order.

12. The encoding method according to any one of claims 1 to 11, wherein the one or more reference 3D points are selected using the distance and information other than the distance.

13. The encoding method described in claim 12, wherein the information other than the distance is connectivity information indicating whether the target 3D point is connected to each of the plurality of 3D points, and in selecting the one or more reference 3D points, a 3D point connected to the target 3D point among the plurality of 3D points is selected as the one or more reference 3D points.

14. A decoding method for decoding information of a three-dimensional point in a target frame to be decoded, comprising the steps of: selecting one or more reference three-dimensional points from a plurality of three-dimensional points in the target frame; calculating a predicted value of second information of a target three-dimensional point to be decoded in the target frame using first information of each of the one or more reference three-dimensional points; and selecting the one or more reference three-dimensional points based on a distance between the target three-dimensional point and each of the plurality of three-dimensional points.

15. The decoding method according to claim 14, further comprising the steps of: obtaining prediction residual information indicating a prediction residual from a bit stream; and calculating the second information based on the prediction residual and the predicted value.

16. The decoding method of claim 14, wherein the first information for each of the one or more reference three-dimensional points indicates a motion vector for each of the one or more reference three-dimensional points, and the second information indicates a motion vector for the target three-dimensional point.

17. The decoding method according to claim 14, wherein the calculation of the predicted value includes calculating the predicted value using inter prediction.

18. The decoding method according to claim 14, wherein in selecting the one or more reference 3D points, the distance is calculated by calculating a difference between the coordinates of the target 3D point and the coordinates of each of the plurality of 3D points.

19. The decoding method according to claim 14, wherein in selecting the one or more reference three-dimensional points, a three-dimensional point whose distance is equal to or less than a predetermined value is selected from the plurality of three-dimensional points as the one or more reference three-dimensional points.

20. The decoding method according to claim 14, wherein the selection of the one or more reference three-dimensional points comprises selecting a predetermined number of three-dimensional points from the plurality of three-dimensional points in order of smallest distance, thereby selecting the one or more reference three-dimensional points.

21. The decoding method according to claim 20, further comprising obtaining predetermined number information indicating the predetermined number from a bit stream.

22. The decoding method according to claim 14, wherein in selecting the one or more reference 3D points, the distance is calculated using coordinates of a 3D point in a reference frame that corresponds to the target 3D point.

23. The decoding method according to claim 22, wherein the reference frame is a frame preceding the target frame in display order.

24. The decoding method according to claim 22, wherein the reference frame is a frame preceding the target frame in decoding order.

25. A decoding method according to any one of claims 14 to 24, wherein the one or more reference three-dimensional points are selected using the distance and information other than the distance.

26. The decoding method described in claim 25, wherein the information other than the distance is connectivity information indicating whether the target 3D point is connected to each of the plurality of 3D points, and in selecting the one or more reference 3D points, a 3D point connected to the target 3D point among the plurality of 3D points is selected as the one or more reference 3D points.

27. An encoding device that encodes information about a three-dimensional point in a target frame to be encoded, comprising: a memory; and a circuit capable of accessing the memory, wherein the circuit, in an operation, selects one or more reference three-dimensional points from among a plurality of three-dimensional points in the target frame, and calculates a predicted value of second information of a target three-dimensional point to be encoded in the target frame using first information of each of the one or more reference three-dimensional points, and in selecting the one or more reference three-dimensional points, selects the one or more reference three-dimensional points based on a distance between the target three-dimensional point and each of the plurality of three-dimensional points.

28. A decoding method for decoding information of a three-dimensional point in a target frame to be decoded, comprising: a memory; and a circuit capable of accessing the memory, wherein the circuit, in an operation, selects one or more reference three-dimensional points from a plurality of three-dimensional points in the target frame, calculates a predicted value of second information of a target three-dimensional point to be decoded in the target frame using first information of each of the one or more reference three-dimensional points, and selects the one or more reference three-dimensional points based on a distance between the target three-dimensional point and each of the plurality of three-dimensional points.

Images