Three-dimensional data encoding method, three-dimensional data decoding method, three-dimensional data encoding device, and three-dimensional data decoding device

Abstract

A three-dimensional data encoding method according to the present invention selects any one of a plurality of encoding methods including a first encoding method that is a video-based three-dimensional data encoding method and a second encoding method that is different from the first encoding method (S601), when the first encoding method is selected (first encoding method in S602), one or more first components are encoded using the first encoding method to generate first encoded data, (S603), when the second encoding method is selected (second encoding method in S602), the one or more first components are encoded using the second encoding method to generate second encoded data (S604), first information indicating the encoding method that was selected is stored in first metadata in which metadata of the first encoding method is extended (S605), and a bitstream including the first metadata and the first encoded data or the second encoded data is generated (S606).

Description

Three-dimensional data encoding method, three-dimensional data decoding method, three-dimensional data encoding device, and three-dimensional data decoding device

[0001] The present disclosure relates to a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, and a three-dimensional data decoding device.

[0002] In the future, devices and services that utilize 3D data are expected to become widespread in a wide range of fields, including computer vision for autonomous operation of automobiles or robots, map information, surveillance, infrastructure inspection, video distribution, etc. 3D data can be acquired in a variety of ways, including distance sensors such as range finders, stereo cameras, or a combination of multiple monocular cameras.

[0003] One method of representing three-dimensional data is a representation method called a point cloud, which represents the shape of a three-dimensional structure using a group of points in three-dimensional space. In a point cloud, the positions and colors of the points are stored. Point clouds are expected to become mainstream as a method of representing three-dimensional data, but point clouds have a very large amount of data. Therefore, when storing or transmitting three-dimensional data, it is essential to compress the amount of data by encoding, just as with two-dimensional moving images (examples include MPEG-4 AVC (Advanced Video Coding) or HEVC (High Efficiency Video Coding) standardized by MPEG).

[0004] Furthermore, compression of point clouds is partially supported by a public library (Point Cloud Library) that performs processing related to point clouds.

[0005] Furthermore, a technique is known in which three-dimensional map data is used to search for and display facilities located around a vehicle (see, for example, Patent Document 1).

[0006] International Publication No. 2014 / 020663

[0007] In encoding and decoding such three-dimensional data, it is desirable to be able to improve the encoding efficiency.

[0008] An object of the present disclosure is to provide a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, or a three-dimensional data decoding device that can improve encoding efficiency.

[0009] A three-dimensional data encoding method according to one aspect of the present disclosure is a three-dimensional data encoding method for encoding three-dimensional data including one or more first components, the method selecting one of a plurality of encoding methods including a first encoding method that is a video-based three-dimensional data encoding method and a second encoding method different from the first encoding method, and when the first encoding method is selected, generating first encoded data by encoding the one or more first components using the first encoding method, and when the second encoding method is selected, generating second encoded data by encoding the one or more first components using the second encoding method, storing first information indicating the selected encoding method in first metadata that is an extension of the metadata of the first encoding method, and generating a bitstream including the first metadata and the first encoded data or the second encoded data.

[0010] A three-dimensional data decoding method according to one aspect of the present disclosure is a three-dimensional data decoding method for decoding three-dimensional data including one or more first components, the method comprising: acquiring a bitstream including first metadata, which is an extension of the metadata of a first encoding method that is a video-based three-dimensional data encoding method, and encoded data; acquiring first information from the first metadata indicating an encoding method used to encode the one or more first components, from among a plurality of encoding methods including the first encoding method and a second encoding method different from the first encoding method; if the first information indicates the first encoding method, generating the one or more first components by decoding the encoded data using a first decoding method corresponding to the first encoding method; and if the first information indicates the second encoding method, generating the one or more first components by decoding the encoded data using a second decoding method corresponding to the second encoding method.

[0011] The present disclosure can provide a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, or a three-dimensional data decoding device that can improve encoding efficiency.

[0012] FIG. 1 is a diagram illustrating an example of the configuration of a 3D data encoding / decoding system according to an embodiment. FIG. 2 is a diagram illustrating an example of the configuration of point cloud data according to an embodiment. FIG. 3 is a diagram illustrating an example of the configuration of a data file in which point cloud data information according to an embodiment is described. FIG. 4 is a diagram illustrating an example of the configuration of mesh data according to an embodiment. FIG. 5 is a diagram illustrating an example of the configuration of a data file in which mesh data information according to an embodiment is described. FIG. 6 is a diagram illustrating types of 3D data according to an embodiment. FIG. 7 is a diagram illustrating the configuration of a first encoding unit according to an embodiment. FIG. 8 is a block diagram of a first encoding unit according to an embodiment. FIG. 9 is a diagram illustrating the configuration of a first decoding unit according to an embodiment. FIG. 10 is a block diagram of a first decoding unit according to an embodiment. FIG. 11 is a diagram illustrating the configuration of a second encoding unit according to an embodiment. FIG. 12 is a block diagram of a second encoding unit according to an embodiment. FIG. 13 is a diagram illustrating the configuration of a second decoding unit according to an embodiment. FIG. 14 is a block diagram of a second decoding unit according to an embodiment. FIG. 15 is a block diagram of a position information encoding unit according to an embodiment. FIG. 16 is a block diagram of a position information decoding unit according to an embodiment. FIG. 17 is a block diagram of an octree encoding unit according to an embodiment. FIG. 18 is a diagram showing an example of position information according to an embodiment. FIG. 19 is a diagram showing an example of an octree representation of position information according to an embodiment. FIG. 20 is a block diagram of an octree decoding unit according to an embodiment. FIG. 21 is a block diagram of an attribute information encoding unit according to an embodiment. FIG. 22 is a block diagram of an attribute information decoding unit according to an embodiment. FIG. 23 is a block diagram of an attribute information encoding unit according to an embodiment. FIG. 24 is a block diagram of an attribute information decoding unit according to an embodiment. FIG. 25 is a block diagram of a 3D data encoding device according to an embodiment. FIG. 26 is a block diagram of a 3D data decoding device according to an embodiment. FIG. 27 is a diagram showing the relationship between tiles and slices according to an embodiment. FIG. 28 is a diagram showing an example configuration of a bitstream according to an embodiment. FIG. 29 is a block diagram showing an example configuration of a 3D data encoding device according to an embodiment. FIG. 30 is a diagram showing example configurations of encoded data and NAL units according to an embodiment.FIG. 31 is a diagram showing example semantics of pcc_nal_unit_type according to an embodiment. FIG. 32 is a diagram showing part of processing in a 3D data generation system according to an embodiment. FIG. 33 is a diagram showing components of Gaussian data according to an embodiment. FIG. 34 is a diagram showing rendering processing of Gaussian data according to an embodiment. FIG. 35 is a diagram showing an example of the number of SH coefficients of a spherical harmonic function according to an embodiment. FIG. 36 is a diagram showing example input / output of a spherical harmonic function according to an embodiment. FIG. 37 is a diagram showing rendering processing according to an embodiment. FIG. 38 is a block diagram of an encoding device according to an embodiment. FIG. 39 is a block diagram of a decoding device according to an embodiment. FIG. 40 is a block diagram of an encoding device according to an embodiment. FIG. 41 is a block diagram of a decoding device according to an embodiment. FIG. 42 is a diagram showing example mapping of Gaussian data according to an embodiment. FIG. 43 is a diagram showing example mapping information according to an embodiment. FIG. 44 is a diagram showing an example configuration of conversion information according to an embodiment. FIG. 45 is a diagram showing an example configuration of encoded data according to an embodiment. FIG. 46 is a diagram showing an example G-PCC attribute type of Gaussian data according to an embodiment. FIG. 47 is a flowchart of processing by a decoding device according to an embodiment. FIG. 48 is a block diagram showing a system for encoding and decoding first Gaussian data according to an embodiment. FIG. 49 is a block diagram of a second pre-processing unit according to an embodiment. FIG. 50 is a diagram showing conversion processing according to an embodiment. FIG. 51 is a diagram showing a protocol stack of an encoding standard for encoding Gaussian data according to an embodiment. FIG. 52 is a block diagram of an encoding device according to an embodiment. FIG. 53 is a block diagram of a decoding device according to an embodiment. FIG. 54 is a flowchart of encoding processing according to an embodiment. FIG. 55 is a flowchart of decoding processing according to an embodiment. FIG. 56 is a diagram showing a basic data structure of a Gaussian codec according to an embodiment. FIG. 57 is a diagram showing an example of the structure of a 3D data unit according to an embodiment. FIG. 58 is a diagram showing an example of the structure of a 3DPS according to an embodiment. FIG. 59 is a diagram showing an example of the structure of Gaussian data component information according to an embodiment.FIG. 60 is a diagram showing an example of the structure of G-PCC mapping information according to an embodiment. FIG. 61 is a diagram showing an example of the structure of V3C mapping information according to an embodiment. FIG. 62 is a diagram showing an example of the structure of a Gaussian data unit according to an embodiment. FIG. 63 is a diagram showing a protocol stack of a standard for encoding a 3D Gaussian according to an embodiment. FIG. 64 is a diagram showing a protocol stack of a standard for encoding a 3D model according to an embodiment. FIG. 65 is a diagram showing an example of the structure of Gaussian component information according to an embodiment. FIG. 66 is a diagram showing a protocol stack of an encoding standard for encoding Gaussian data according to an embodiment. FIG. 67 is a block diagram of an encoding device according to an embodiment. FIG. 68 is a block diagram of a decoding device according to an embodiment. FIG. 69 is a flowchart of an encoding process according to an embodiment. FIG. 70 is a flowchart of a decoding process according to an embodiment. FIG. 71 is a diagram showing an example of the structure of an extended V3C unit according to an embodiment. FIG. 72 is a diagram showing an example of the structure of a V3C unit according to an embodiment. FIG. 73 is a diagram showing an example of the structure of a GCL NAL unit according to an embodiment. FIG. 74 is a diagram showing an example of the structure of an extended V3C parameter set according to an embodiment. Fig. 75 is a diagram showing an example of a configuration of Gaussian data component information according to an embodiment. Fig. 76 is a diagram showing an example of a configuration of video mapping information according to an embodiment. Fig. 77 is a diagram showing an example of a configuration of 3D mapping information according to an embodiment. Fig. 78 is a diagram showing a protocol stack of extended V3C according to an embodiment. Fig. 79 is a diagram showing a protocol stack of extended V3C according to an embodiment. Fig. 80 is a flowchart of encoding processing by an encoding device according to an embodiment. Fig. 81 is a block diagram of an encoding device according to an embodiment. Fig. 82 is a flowchart of decoding processing by a decoding device according to an embodiment. Fig. 83 is a block diagram of a decoding device according to an embodiment.

[0013] A three-dimensional data encoding method according to one aspect of the present disclosure is a three-dimensional data encoding method for encoding three-dimensional data including one or more first components, the method selecting one of a plurality of encoding methods including a first encoding method that is a video-based three-dimensional data encoding method and a second encoding method different from the first encoding method, and when the first encoding method is selected, generating first encoded data by encoding the one or more first components using the first encoding method, and when the second encoding method is selected, generating second encoded data by encoding the one or more first components using the second encoding method, storing first information indicating the selected encoding method in first metadata that is an extension of the metadata of the first encoding method, and generating a bitstream including the first metadata and the first encoded data or the second encoded data.

[0014] This allows the 3D data encoding method to selectively select an encoding method to be used for encoding 3D data from multiple encoding methods, potentially improving encoding efficiency. Furthermore, by extending the video-based 3D data encoding method, a system capable of easily implementing the above processing can be constructed.

[0015] For example, the three-dimensional data encoding method may further convert the one or more first components into one or more second components corresponding to the selected encoding method, and when the first encoding method is selected, generate the first encoded data by encoding the one or more second components using the first encoding method, and when the second encoding method is selected, generate the second encoded data by encoding the one or more second components using the second encoding method, and the first metadata may further include second information indicating a correspondence between the one or more first components and the one or more second components.

[0016] According to this, the three-dimensional data encoding method can encode the three-dimensional data using the selected encoding method by converting the data into a format corresponding to the selected encoding method, and the decoding device can convert the plurality of second components into the plurality of first components using the second information included in the bitstream.

[0017] For example, the second encoding scheme may be a geometry-based three-dimensional data encoding scheme. For example, the second encoding scheme may be a neural network-based three-dimensional data encoding scheme. For example, the second encoding scheme may be an AI-based three-dimensional data encoding scheme. For example, the three-dimensional data may be Gaussian data.

[0018] This three-dimensional data encoding method allows selective selection of an encoding method to be used for encoding Gaussian data from among a plurality of encoding methods, which may improve encoding efficiency.

[0019] A three-dimensional data decoding method according to one aspect of the present disclosure is a three-dimensional data decoding method for decoding three-dimensional data including one or more first components, the method comprising: acquiring a bitstream including first metadata, which is an extension of the metadata of a first encoding method that is a video-based three-dimensional data encoding method, and encoded data; acquiring first information from the first metadata indicating an encoding method used to encode the one or more first components, from among a plurality of encoding methods including the first encoding method and a second encoding method different from the first encoding method; if the first information indicates the first encoding method, generating the one or more first components by decoding the encoded data using a first decoding method corresponding to the first encoding method; and if the first information indicates the second encoding method, generating the one or more first components by decoding the encoded data using a second decoding method corresponding to the second encoding method.

[0020] This allows the encoding device to selectively select an encoding method to be used for encoding three-dimensional data from among multiple encoding methods. This potentially improves encoding efficiency. Furthermore, the three-dimensional data decoding method can appropriately decode encoded data generated by the encoding device. Furthermore, by extending the video-based three-dimensional data encoding method, a system capable of easily implementing the above processing can be constructed.

[0021] For example, the encoded data may be generated by encoding one or more second components converted from the one or more first components, and the three-dimensional data decoding method may further include obtaining, from the first metadata, second information indicating a correspondence between the one or more first components and the one or more second components, and if the first information indicates a first encoding method, generating the one or more second components by decoding the encoded data using the first decoding method, and if the first information indicates a second encoding method, generating the one or more second components by decoding the encoded data using the second decoding method, and converting the one or more second components into the one or more first components using the correspondence indicated by the second information.

[0022] According to this, the encoding device can encode the 3D data using the selected encoding method by converting the data into a format corresponding to the selected encoding method, and the 3D data decoding device can convert the plurality of second components into the plurality of first components using the second information included in the bitstream.

[0023] For example, the second encoding scheme may be a geometry-based three-dimensional data encoding scheme. For example, the second encoding scheme may be a neural network-based three-dimensional data encoding scheme. For example, the second encoding scheme may be an AI-based three-dimensional data encoding scheme.

[0024] For example, the three-dimensional data may be Gaussian data.

[0025] This allows the encoding device to selectively select an encoding method to be used for encoding Gaussian data from a plurality of encoding methods, which may improve encoding efficiency.

[0026] A three-dimensional data encoding device according to one aspect of the present disclosure is a three-dimensional data encoding device that encodes three-dimensional data including one or more first components, and is equipped with a processor and a memory. The processor uses the memory to select one of a plurality of encoding methods including a first encoding method that is a video-based three-dimensional data encoding method and a second encoding method different from the first encoding method. If the first encoding method is selected, the processor generates first encoded data by encoding the one or more first components using the first encoding method. If the second encoding method is selected, the processor generates second encoded data by encoding the one or more first components using the second encoding method. The processor stores first information indicating the selected encoding method in first metadata that is an extension of the metadata of the first encoding method. The processor generates a bitstream that includes the first metadata and the first encoded data or the second encoded data.

[0027] This allows the 3D data encoding device to selectively select an encoding method to be used for encoding 3D data from multiple encoding methods, which may improve encoding efficiency.Furthermore, by extending the video-based 3D data encoding method, a system capable of easily implementing the above processing can be constructed.

[0028] A three-dimensional data decoding device according to one aspect of the present disclosure is a three-dimensional data decoding device that decodes three-dimensional data including one or more first components, and is equipped with a processor and a memory. The processor uses the memory to acquire a bitstream that includes first metadata, which is extended metadata of a first encoding method that is a video-based three-dimensional data encoding method, and encoded data. The processor acquires first information from the first metadata that indicates an encoding method used to encode the one or more first components, from a plurality of encoding methods including the first encoding method and a second encoding method. If the first information indicates the first encoding method, the processor generates the one or more first components by decoding the encoded data using a first decoding method corresponding to the first encoding method. If the first information indicates the second encoding method, the processor generates the one or more first components by decoding the encoded data using a second decoding method corresponding to the second encoding method.

[0029] This allows the encoding device to selectively select an encoding method to be used for encoding three-dimensional data from among multiple encoding methods. This potentially improves encoding efficiency. Furthermore, the three-dimensional data decoding device can appropriately decode encoded data generated by the encoding device. Furthermore, by extending the video-based three-dimensional data encoding method, a system capable of easily implementing the above processing can be constructed.

[0030] These comprehensive or specific aspects may be realized as a system, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or may be realized as any combination of a system, a method, an integrated circuit, a computer program, and a recording medium.

[0031] Hereinafter, the embodiments will be described in detail with reference to the drawings. Note that each of the embodiments described below represents a specific example of the present disclosure. The numerical values, shapes, materials, components, component placement and connection configurations, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components not recited in independent claims will be described as optional components.

[0032] (Embodiment) [Three-dimensional data encoding / decoding system] First, the configuration of a three-dimensional data encoding / decoding system according to this embodiment will be described. Fig. 1 is a diagram showing an example of the configuration of a three-dimensional data encoding / decoding system according to this embodiment. As shown in Fig. 1, the three-dimensional data encoding / decoding system includes a three-dimensional data encoding system 101, a three-dimensional data decoding system 102, a sensor terminal 103, and an external connection unit 104.

[0033] The three-dimensional data encoding system 101 generates encoded data or multiplexed data by encoding three-dimensional data such as three-dimensional point cloud data or three-dimensional mesh data. The three-dimensional data encoding system 101 may be a three-dimensional data encoding device realized by a single device, or may be a system realized by multiple devices. The three-dimensional data encoding device may also include some of the multiple processing units included in the three-dimensional data encoding system 101.

[0034] The three-dimensional data encoding system 101 includes a three-dimensional data generation system 111, a presentation unit 112, an encoding unit 113, a multiplexing unit 114, an input / output unit 115, and a control unit 116. The three-dimensional data generation system 111 includes a sensor information acquisition unit 117 and a three-dimensional data generation unit 118.

[0035] The sensor information acquisition unit 117 acquires sensor information (sensor signals) from the sensor terminal 103 and outputs the sensor information to the three-dimensional data generation unit 118. The three-dimensional data generation unit 118 generates three-dimensional data from the sensor information and outputs the three-dimensional data to the encoding unit 113.

[0036] The presentation unit 112 presents the sensor information or the three-dimensional data to the user. For example, the presentation unit 112 displays information or an image based on the sensor information or the three-dimensional data.

[0037] The encoding unit 113 encodes (compresses) the three-dimensional data and outputs the resulting encoded data, control information obtained in the encoding process, and other additional information to the multiplexing unit 114. The additional information includes, for example, sensor information.

[0038] The multiplexing unit 114 generates multiplexed data by multiplexing the coded data, control information, and additional information input from the coding unit 113. The format of the multiplexed data is, for example, a file format for storage or a packet format for transmission.

[0039] The input / output unit 115 (e.g., a communication unit or an interface) outputs the multiplexed data to the outside. Alternatively, the multiplexed data is stored in a storage unit such as an internal memory. The control unit 116 (or application execution unit) controls each processing unit. In other words, the control unit 116 controls encoding, multiplexing, etc.

[0040] The sensor information may be input to the encoding unit 113 or the multiplexing unit 114. The input / output unit 115 may output the three-dimensional data or the encoded data directly to the outside.

[0041] The transmission signal (multiplexed data) output from the three-dimensional data encoding system 101 is input to the three-dimensional data decoding system 102 via the external connection unit 104 .

[0042] The three-dimensional data decoding system 102 generates three-dimensional data such as three-dimensional point cloud data or three-dimensional mesh data by decoding encoded data or multiplexed data. The three-dimensional data decoding system 102 may be a three-dimensional data decoding device realized by a single device, or may be a system realized by multiple devices. Furthermore, the three-dimensional data decoding device may include some of the multiple processing units included in the three-dimensional data decoding system 102.

[0043] The three-dimensional data decoding system 102 includes a sensor information acquisition unit 121 , an input / output unit 122 , a demultiplexing unit 123 , a decoding unit 124 , a presentation unit 125 , a user interface 126 , and a control unit 127 .

[0044] The sensor information acquisition unit 121 acquires sensor information (sensor signals) from the sensor terminal 103 .

[0045] The input / output unit 122 acquires a transmission signal, decodes multiplexed data (file format or packets) from the transmission signal, and outputs the multiplexed data to the demultiplexer 123 .

[0046] The demultiplexer 123 obtains the coded data, control information, and additional information from the multiplexed data, and outputs the coded data, control information, and additional information to the decoder 124 .

[0047] The decoding unit 124 reconstructs three-dimensional data by decoding the encoded data.

[0048] The presentation unit 125 presents the three-dimensional data to the user. For example, the presentation unit 125 displays information or an image based on the three-dimensional data. The user interface 126 acquires instructions based on user operations. The control unit 127 (or the application execution unit) controls each processing unit. That is, the control unit 127 controls demultiplexing, decoding, presentation, etc.

[0049] The input / output unit 122 may acquire the three-dimensional data or the encoded data directly from the outside. The presentation unit 125 may acquire additional information such as sensor information and present information based on the additional information. The presentation unit 125 may also present information based on a user instruction acquired by the user interface 126.

[0050] The sensor terminal 103 generates sensor information, which is information obtained by a sensor. The sensor terminal 103 is a terminal equipped with a sensor or a camera, and may be, for example, a mobile object such as an automobile, a flying object such as an airplane, a mobile terminal, or a camera.

[0051] Examples of sensor information that can be acquired by the sensor terminal 103 include (1) the distance between the sensor terminal 103 and an object or the reflectance of the object obtained from a LIDAR, millimeter-wave radar, or infrared sensor, and (2) the distance between a camera and an object or the reflectance of the object obtained from multiple monocular camera images or stereo camera images. The sensor information may also include the attitude, orientation, gyro (angular velocity), position (GPS information or altitude), speed, acceleration, etc. of the sensor. The sensor information may also include temperature, air pressure, humidity, magnetism, etc.

[0052] The external connection unit 104 is realized by an integrated circuit (LSI or IC), an external storage unit, communication with a cloud server via the Internet, broadcasting, or the like.

[0053] Next, three-dimensional point cloud data (hereinafter also referred to as point cloud data) will be described. Fig. 2 is a diagram showing the structure of point cloud data. Fig. 3 is a diagram showing an example of the structure of a data file in which information about point cloud data is written.

[0054] Point cloud data includes data on multiple points. Each point includes location information (three-dimensional coordinates) and attribute information for that location. A collection of multiple points is called a point cloud. For example, a point cloud can represent the three-dimensional shape of an object.

[0055] Position information such as three-dimensional coordinates is sometimes called geometry. Data for each point may include attribute information of multiple attribute types. The attribute types may be, for example, color or reflectance.

[0056] One piece of attribute information may be associated with one piece of location information, or multiple pieces of attribute information having different attribute types may be associated with one piece of location information, or multiple pieces of attribute information of the same attribute type may be associated with one piece of location information.

[0057] The data file configuration example shown in FIG. 3 is an example in which position information and attribute information correspond one-to-one, and shows the position information and attribute information of N points that make up the point cloud data.

[0058] The position information is, for example, information on three axes, x, y, and z. The attribute information is, for example, RGB color information. A typical data file is a ply file.

[0059] Next, three-dimensional mesh data (hereinafter also referred to as mesh data) will be described. Fig. 4 is a diagram showing an example of the structure of mesh data. Fig. 5 is a diagram showing an example of the structure of a data file in which information about the mesh data is written.

[0060] Mesh data is a data format used in CG (Computer Graphics). Mesh data represents the three-dimensional shape of an object using a collection of surface information. The surface information is a polygon such as a triangle or a quadrangle, and is also called a polygon or polygon mesh.

[0061] The components of mesh data are a three-dimensional point cloud (a set of points having three-dimensional position information and attribute information corresponding to that position information), as well as a set of three-dimensional points as vertices, edges connecting two vertices, and faces enclosed by the edges.

[0062] A vertex (also expressed as a "vertex" or "position") may have attribute information such as color information, reflectance, or normal vector for the point. Information indicating the relationship between the vertices that make up an edge or face is also called connectivity. The front and back of a face can be expressed by the direction of the normal vector for the point. Mesh data may also have attribute information for the face.

[0063] An example of the format of a mesh data file is an object file. The data file indicates the position information G(1) to G(N) of the N vertices that make up the mesh, and the attribute information A(1) to A(N) of the vertices. Note that the data file does not have to include the attribute information. Also, the attribute information does not have to correspond one-to-one to the vertices. Note that FIG. 5 shows an example in which the data file has 1 to M pieces of attribute information A2.

[0064] Face information is represented by a combination of vertex indices. n[1, 3, 4] indicates that the face is a triangular face consisting of three vertices, n=1, n=3, and n=4. Furthermore, m[2, 4, 6] indicates that the attribute information of m=1, m=4, and m=6 corresponds to the three vertices, respectively.

[0065] Alternatively, the attribute information may be stored in a file separate from the data file, with the data file indicating its pointer information. For example, the attribute information may be stored in a two-dimensional attribute map file, with the file name of the attribute map and two-dimensional coordinates in the attribute map being stored in attribute information A2 of the data file. Either method can be used to specify attribute information for a point.

[0066] Next, the types of three-dimensional data (point cloud data or mesh data) will be described. Fig. 6 is a diagram showing the types of three-dimensional data. As shown in Fig. 6, three-dimensional data includes static objects and dynamic objects.

[0067] A static object is three-dimensional data at any time (a certain time). A dynamic object is three-dimensional data that changes over time. Hereinafter, three-dimensional point cloud data at a certain time will be referred to as a PCC (Point Cloud Compression) frame or frame. Also, three-dimensional mesh data at a certain time will be referred to as a mesh frame or frame.

[0068] The multiple points that make up an object may be limited to a certain extent in terms of area, number of pixels, or number of points, as in normal video data, or the multiple points that make up an object may not be limited in terms of area, as in map information.

[0069] Furthermore, there may be point cloud data or mesh data of various densities, and there may be sparse point cloud data or mesh data and dense point cloud data or mesh data.

[0070] Each processing unit will be described in detail below. The sensor information is acquired by various methods, such as a distance sensor such as a LIDAR or a range finder, a stereo camera, or a combination of multiple monocular cameras. The three-dimensional data generation unit 118 generates three-dimensional data based on the sensor information acquired by the sensor information acquisition unit 117. The three-dimensional data generation unit 118 generates position information as three-dimensional data, and adds attribute information for the position information to the position information.

[0071] The three-dimensional data generation unit 118 may process the three-dimensional data when generating position information or adding attribute information. For example, the three-dimensional data generation unit 118 may reduce the amount of data by deleting point clouds with overlapping positions. The three-dimensional data generation unit 118 may also transform (position shift, rotation, normalization, etc.) the position information. The three-dimensional data generation unit 118 may also generate mesh data from the point cloud data. The three-dimensional data generation unit 118 may also render the attribute information.

[0072] In FIG. 1, the three-dimensional data generation system 111 is included in the three-dimensional data encoding system 101, but it may be provided independently outside the three-dimensional data encoding system 101.

[0073] The encoding unit 113 generates encoded data by encoding the three-dimensional data. There are the following encoding methods. The first is an encoding method using position information, which will be referred to hereinafter as the first encoding method. The second is an encoding method using a video codec, which will be referred to hereinafter as the second encoding method.

[0074] The decoding unit 124 decodes the encoded data to decode the point cloud data. The multiplexing unit 114 multiplexes the encoded data using an existing multiplexing method to generate multiplexed data. The generated multiplexed data is transmitted or stored. In addition to the PCC encoded data, the multiplexing unit 114 multiplexes other media such as video, audio, subtitles, applications, and files, or reference time information. The multiplexing unit 114 may also multiplex sensor information or attribute information related to the three-dimensional data.

[0075] Multiplexing methods or file formats include ISOBMFF, and ISOBMFF-based transmission methods such as MPEG-DASH, MMT, MPEG-2 TS Systems, and RTP.

[0076] The demultiplexer 123 extracts the coded data, other media, time information, and the like from the multiplexed data.

[0077] The input / output unit 115 transmits the multiplexed data using a method suited to the transmission medium or storage medium, such as broadcasting or communication. The input / output unit 115 may communicate with other devices via the Internet, or may communicate with a storage unit such as a cloud server.

[0078] The communication protocol used may be http, ftp, TCP, UDP, etc. A pull-type communication method or a push-type communication method may be used.

[0079] Either wired transmission or wireless transmission may be used. For wired transmission, Ethernet (registered trademark), USB, RS-232C, HDMI (registered trademark), coaxial cable, etc. are used. For wireless transmission, wireless LAN, Wi-Fi (registered trademark), Bluetooth (registered trademark), millimeter waves, etc. are used.

[0080] As a broadcasting system, for example, DVB-T2, DVB-S2, DVB-C2, ATSC3.0, or ISDB-S3 is used.

[0081] [First Encoding Method] Hereinafter, a method for encoding and decoding a three-dimensional point cloud or a three-dimensional mesh will be described. When the device, process, or syntax in this disclosure relates to encoding or decoding processing of point cloud data, it can also be applied to encoding or decoding of vertices in a three-dimensional mesh. Furthermore, the disclosure regarding encoding or decoding processing of vertices in a three-dimensional mesh can also be applied to encoding or decoding of point cloud data. Furthermore, the disclosure regarding encoding or decoding processing of attribute information of point cloud data may also be applied to encoding or decoding processing of face information, or attribute information for faces or vertices in a three-dimensional mesh. Furthermore, the processing may be shared between point cloud encoding and mesh encoding. This sharing may potentially reduce the scale of the circuit or software.

[0082] Fig. 7 is a diagram showing the configuration of a first encoding unit 130, which is an example of the encoding unit 113 that performs encoding using the first encoding method. Fig. 8 is a block diagram of the first encoding unit 130. The first encoding unit 130 generates encoded data (encoded stream) by encoding point cloud data using the first encoding method. This first encoding unit 130 includes a position information encoding unit 131, an attribute information encoding unit 132, an additional information encoding unit 133, and a multiplexing unit 134.

[0083] The first encoding unit 130 is characterized in that it performs encoding with consideration of a three-dimensional structure. The first encoding unit 130 is also characterized in that the attribute information encoding unit 132 performs encoding using information obtained from the position information encoding unit 131. The first encoding method is also called G-PCC (Geometry-based PCC).

[0084] The point cloud data is PCC point cloud data such as a PLY file or PCC point cloud data generated from sensor information, and includes position information (Position), attribute information (Attribute), and other additional information (MetaData). The position information is input to a position information encoding unit 131, the attribute information is input to an attribute information encoding unit 132, and the additional information is input to an additional information encoding unit 133.

[0085] When mesh data is encoded, the position information of the vertex information is input to a position information encoding unit 131 , and the face information or attribute information for the faces and vertices is input to an attribute information encoding unit 132 .

[0086] The position information encoding unit 131 encodes the position information to generate encoded position information (Compressed Geometry), which is encoded data. For example, the position information encoding unit 131 encodes the position information using an N-ary tree structure such as an octtree. Specifically, in an octtree, the target space is divided into eight nodes (subspaces), and 8-bit information (occupancy code) indicating whether or not a point cloud is included in each node is generated. Furthermore, the node including the point cloud is further divided into eight nodes, and 8-bit information indicating whether or not a point cloud is included in each of the eight nodes is generated. This process is repeated until the number of point clouds included in a predetermined layer or node falls below a threshold.

[0087] The attribute information encoding unit 132 generates encoded attribute information (Compressed Attribute) as encoded data by encoding using the configuration information generated by the position information encoding unit 131. For example, the attribute information encoding unit 132 determines a reference point (reference node) to be referenced when encoding a target point (target node) to be processed, based on the octree structure generated by the position information encoding unit 131. For example, the attribute information encoding unit 132 references a peripheral node or adjacent node whose parent node in the octree is the same as that of the target node. Note that the method of determining the reference relationship is not limited to this.

[0088] Furthermore, the encoding process of the attribute information may include at least one of a quantization process, a prediction process, and an arithmetic coding process. In this case, the reference means using a reference node to calculate a predicted value of the attribute information, or using the state of the reference node (e.g., occupancy information indicating whether the reference node includes a point group) to determine an encoding parameter. For example, the encoding parameter may be a quantization parameter in a quantization process, or a context in arithmetic coding.

[0089] The additional information encoding unit 133 generates encoded additional information (Compressed MetaData) that is encoded data by encoding compressible data from the additional information.

[0090] The multiplexing unit 134 multiplexes the encoding position information, the encoding attribute information, the encoding additional information, and other additional information to generate an encoded stream (Compressed Stream) that is encoded data. The generated encoded stream is output to a processing unit of a system layer (not shown).

[0091] Next, a first decoding unit 140, which is an example of the decoding unit 124 that performs decoding using the first encoding method, will be described. FIG. 9 is a diagram showing the configuration of the first decoding unit 140. FIG. 10 is a block diagram of the first decoding unit 140. The first decoding unit 140 generates point cloud data by decoding, using the first encoding method, coded data (coded stream). The first decoding unit 140 includes a demultiplexing unit 141, a position information decoding unit 142, an attribute information decoding unit 143, and an additional information decoding unit 144.

[0092] A coded stream (compressed stream) that is coded data is input to the first decoding unit 140 from a processing unit of a system layer (not shown).

[0093] The demultiplexer 141 separates the encoded position information (Compressed Geometry), the encoded attribute information (Compressed Attribute), the encoded additional information (Compressed MetaData), and other additional information from the encoded data.

[0094] The position information decoding unit 142 generates position information by decoding the encoded position information. For example, the position information decoding unit 142 restores position information of a point group represented by three-dimensional coordinates from the encoded position information represented by an N-ary tree structure such as an octree.

[0095] The attribute information decoding unit 143 decodes the encoded attribute information based on the configuration information generated by the position information decoding unit 142. For example, the attribute information decoding unit 143 determines a reference point (reference node) to be referenced in decoding the target point (target node) to be processed based on the octree structure obtained by the position information decoding unit 142. For example, the attribute information decoding unit 143 references a peripheral node or adjacent node whose parent node in the octree is the same as that of the target node. Note that the method of determining the reference relationship is not limited to this.

[0096] Furthermore, the attribute information decoding process may include at least one of an inverse quantization process, a prediction process, and an arithmetic decoding process. In this case, the reference means using a reference node to calculate a predicted value of the attribute information, or using the state of the reference node (e.g., occupancy information indicating whether the reference node includes a point group) to determine a decoding parameter. For example, the decoding parameter is a quantization parameter in an inverse quantization process, or a context in an arithmetic decoding process.

[0097] The additional information decoding unit 144 generates additional information by decoding the encoded additional information. The first decoding unit 140 uses the additional information necessary for decoding the position information and attribute information during decoding, and outputs the additional information necessary for the application to the outside.

[0098] [Second Encoding Method] Next, a second encoding unit 150, which is an example of the encoding unit 113 that performs encoding using the second encoding method, will be described. Fig. 11 is a diagram showing the configuration of the second encoding unit 150. Fig. 12 is a block diagram of the second encoding unit 150.

[0099] The second encoding unit 150 generates encoded data (encoded stream) by encoding the point cloud data using a second encoding method. The second encoding unit 150 includes an additional information generation unit 151, a position image generation unit 152, an attribute image generation unit 153, a video encoding unit 154, an additional information encoding unit 155, and a multiplexing unit 156.

[0100] The second encoding unit 150 generates a position image and an attribute image by projecting a three-dimensional structure onto a two-dimensional image, and encodes the generated position image and attribute image using an existing video encoding method. The second encoding method is also called VPCC (Video-based PCC).

[0101] The point cloud data is PCC point cloud data such as a PLY file or PCC point cloud data generated from sensor information, and includes position information (Position), attribute information (Attribute), and other additional information (MetaData).

[0102] The additional information generating unit 151 generates map information of a plurality of two-dimensional images by projecting a three-dimensional structure onto the two-dimensional images.

[0103] The position image generating unit 152 generates a position image (geometry image) based on the position information and the map information generated by the additional information generating unit 151. This position image is, for example, a distance image in which distance (depth) is indicated as pixel values. Note that this distance image may be an image in which multiple point clouds are viewed from a single viewpoint (an image in which multiple point clouds are projected onto a single two-dimensional plane), or may be multiple images in which multiple point clouds are viewed from multiple viewpoints, or may be a single image in which these multiple images are integrated.

[0104] The attribute image generation unit 153 generates an attribute image based on the attribute information and the map information generated by the additional information generation unit 151. This attribute image is, for example, an image in which attribute information (e.g., color (RGB)) is represented as pixel values. Note that this image may be an image in which multiple point clouds are viewed from a single viewpoint (an image in which multiple point clouds are projected onto a single two-dimensional plane), or multiple images in which multiple point clouds are viewed from multiple viewpoints, or a single image in which these multiple images are integrated.

[0105] When mesh data is encoded, the position information of the vertex information is input to the position image generating unit 152 , and the face information or attribute information for the faces and vertices is input to the attribute image generating unit 153 .

[0106] The video encoding unit 154 encodes the position image and the attribute image using a video encoding method to generate an encoded position image (Compressed Geometry Image) and an encoded attribute image (Compressed Attribute Image), which are encoded data. Note that any known encoding method may be used as the video encoding method. For example, the video encoding method is AVC, HEVC, or the like.

[0107] The additional information encoding unit 155 generates encoded additional information (Compressed MetaData) by encoding the additional information, map information, and the like included in the point cloud data.

[0108] The multiplexing unit 156 multiplexes the encoding position image, the encoding attribute image, the encoding additional information, and other additional information to generate an encoded stream (Compressed Stream) that is encoded data. The generated encoded stream is output to a processing unit of a system layer (not shown).

[0109] Next, a second decoding unit 160, which is an example of the decoding unit 124 that performs decoding using the second encoding method, will be described. FIG. 13 is a diagram showing the configuration of the second decoding unit 160. FIG. 14 is a block diagram of the second decoding unit 160. The second decoding unit 160 generates point cloud data by decoding, using the second encoding method, coded data (coded stream). The second decoding unit 160 includes a demultiplexing unit 161, a video decoding unit 162, an additional information decoding unit 163, a position information generation unit 164, and an attribute information generation unit 165.

[0110] A coded stream (compressed stream) that is coded data is input to the second decoding unit 160 from a processing unit of a system layer (not shown).

[0111] The demultiplexing unit 161 separates the encoded position image (Compressed Geometry Image), the encoded attribute image (Compressed Attribute Image), the encoded additional information (Compressed MetaData), and other additional information from the encoded data.

[0112] The video decoding unit 162 generates a position image and an attribute image by decoding the encoded position image and the encoded attribute image using a video encoding method. Note that any known encoding method may be used as the video encoding method. For example, the video encoding method may be AVC or HEVC.

[0113] The additional information decoding unit 163 decodes the encoded additional information to generate additional information including map information and the like.

[0114] The position information generating unit 164 generates position information using the position image and map information. The attribute information generating unit 165 generates attribute information using the attribute image and map information.

[0115] The second decoding unit 160 uses the additional information necessary for decoding during decoding, and outputs the additional information necessary for the application to the outside.

[0116] [Position Information Encoding in First Encoding Method] Fig. 15 is a block diagram showing an example configuration of the position information encoding unit 131. The position information encoding unit 131 includes an octree encoding unit 171 and a predictive tree encoding unit 172. The octree encoding unit 171 generates encoded position information and metadata by encoding the position information using an encoding method that uses an octree (octree encoding). The predictive tree encoding unit 172 generates encoded position information and metadata by encoding the position information using an encoding method that uses a predictive tree (predictive tree encoding).

[0117] The position information encoding unit 131 encodes the position information using either or both of octree encoding and predictive tree encoding. The position information encoding unit 131 may switch between these two encoding methods, may use an encoding method other than these two methods, or may include the position information in the bitstream as raw data without encoding it. Information indicating the encoding method of the encoded data is stored in metadata and notified to the decoding device.

[0118] 16 is a block diagram showing an example configuration of the position information decoding unit 142. The position information decoding unit 142 includes an octree decoding unit 173 and a predictive tree decoding unit 174. The octree decoding unit 173 generates position information by decoding the encoded position information using a decoding method that uses an octree (octree decoding). The predictive tree decoding unit 174 generates position information by decoding the encoded position information using a decoding method that uses a predictive tree (predictive tree decoding). Furthermore, the position information decoding unit 142 performs decoding using the encoding method notified by the metadata.

[0119] Next, an example of the configuration of the position information encoding unit will be described. Fig. 17 is a block diagram of an octree encoding unit 171 according to this embodiment. The octree encoding unit 171 includes an octree generation unit 181, a geometric information calculation unit 182, a coding table selection unit 183, and an entropy encoding unit 184.

[0120] The octree generation unit 181 generates, for example, an octree from the input position information and generates an occupancy code for each node in the octree. The geometric information calculation unit 182 acquires information indicating whether an adjacent node of the target node is an occupied node. For example, the geometric information calculation unit 182 calculates the occupancy information of the adjacent node (information indicating whether the adjacent node is an occupied node) from the occupancy code of the parent node to which the target node belongs. Alternatively, the geometric information calculation unit 182 may store encoded nodes in a list and search for adjacent nodes from the list. Note that the geometric information calculation unit 182 may switch adjacent nodes depending on the position of the target node within the parent node.

[0121] The coding table selection unit 183 selects a coding table to be used for entropy coding of the target node using the occupancy information of the adjacent nodes calculated by the geometric information calculation unit 182. For example, the coding table selection unit 183 may generate a bit string using the occupancy information of the adjacent nodes, and select a coding table for an index number generated from the bit string.

[0122] The entropy coding unit 184 generates the coding position information and the metadata by entropy coding the occupancy code of the target node using the coding table of the selected index number. The entropy coding unit 184 may add information indicating the selected coding table to the coding position information.

[0123] The octree representation and the scanning order of position information will be described below. Position information (position data) is converted into an octree structure (octreeing) and then encoded. The octree structure is composed of nodes and leaves. Each node has eight nodes or leaves, and each leaf has voxel (VXL) information. FIG. 18 is a diagram showing an example of the structure of position information containing multiple voxels. FIG. 19 is a diagram showing an example of the position information shown in FIG. 18 converted into an octree structure. Here, among the leaves shown in FIG. 19, leaves 1, 2, and 3 represent voxels VXL1, VXL2, and VXL3 shown in FIG. 18, respectively, and represent a VXL containing a point cloud (hereinafter referred to as a valid VXL).

[0124] Specifically, node 1 corresponds to the entire space containing the position information in Figure 18. The entire space corresponding to node 1 is divided into eight nodes, and of the eight nodes, a node containing a valid VXL is further divided into eight nodes or leaves, and this process is repeated for each level of the tree structure. Here, each node corresponds to a subspace, and has information (occupancy code) as node information indicating the position of the next node or leaf after division. In addition, the block at the bottom level is set as a leaf, and the number of point clouds contained in the leaf, etc., is held as leaf information.

[0125] Next, an example of the configuration of the position information decoding unit will be described. Fig. 20 is a block diagram of the octree decoding unit 173 according to this embodiment. The octree decoding unit 173 includes an octree generation unit 191, a geometric information calculation unit 192, a coding table selection unit 193, and an entropy decoding unit 194.

[0126] The octree generator 191 generates an octree of a certain space (node) using header information or metadata of a bitstream. For example, the octree generator 191 generates a large space (root node) using the sizes of the space in the x-axis, y-axis, and z-axis directions added to the header information, and then generates an octree by dividing the space into two in the x-axis, y-axis, and z-axis directions to generate eight small spaces A (nodes A0 to A7). Nodes A0 to A7 are set in order as target nodes.

[0127] The geometric information calculation unit 192 acquires occupancy information indicating whether an adjacent node of a target node is an occupied node. For example, the geometric information calculation unit 192 calculates the occupancy information of the adjacent node from the occupancy code of the parent node to which the target node belongs. Alternatively, the geometric information calculation unit 192 may store decoded nodes in a list and search for adjacent nodes from the list. Note that the geometric information calculation unit 192 may switch adjacent nodes depending on the position of the target node within the parent node.

[0128] The coding table selection unit 193 selects a coding table (decoding table) to be used for entropy decoding of the target node using the occupancy information of the adjacent nodes calculated by the geometric information calculation unit 192. For example, the coding table selection unit 193 may generate a bit string using the occupancy information of the adjacent nodes, and select a coding table for an index number generated from the bit string.

[0129] The entropy decoding unit 194 generates position information by entropy decoding the occupancy code of the target node using the selected coding table. Note that the entropy decoding unit 194 may obtain information on the selected coding table by decoding it from the bitstream, and entropy decode the occupancy code of the target node using the coding table indicated by the obtained information.

[0130] [Attribute Information Encoding in First Encoding Method] The configurations of the attribute information encoding unit and the attribute information decoding unit will be described below. Fig. 21 is a block diagram showing an example configuration of the attribute information encoding unit 132. The attribute information encoding unit may include multiple encoding units that execute different encoding methods. For example, the attribute information encoding unit may switch between the following two methods depending on the use case:

[0131] The attribute information encoding unit 132 includes an LoD attribute information encoding unit 201 and a conversion attribute information encoding unit 202. The LoD attribute information encoding unit 201 classifies each 3D point into multiple layers using the position information of the 3D point, predicts the attribute information of the 3D point belonging to each layer, and encodes the prediction residual. Here, each classified layer is called LoD (Level of Detail).

[0132] The transformed attribute information encoding unit 202 encodes the attribute information using RAHT (Region Adaptive Hierarchical Transform). Specifically, the transformed attribute information encoding unit 202 applies RAHT or Haar transform to each piece of attribute information based on the position information of the three-dimensional points to generate high-frequency components and low-frequency components for each layer, and encodes these values ​​using quantization, entropy coding, or the like.

[0133] 22 is a block diagram showing an example configuration of the attribute information decoding unit 143. The attribute information decoding unit may include multiple decoding units that execute different decoding methods. For example, the attribute information decoding unit may switch between the following two methods for decoding based on information included in the header or metadata.

[0134] The attribute information decoding unit 143 includes an LoD attribute information decoding unit 203 and a converted attribute information decoding unit 204. The LoD attribute information decoding unit 203 classifies each 3D point into multiple layers using the position information of the 3D points, and decodes the attribute values ​​while predicting the attribute information of the 3D points belonging to each layer.

[0135] The transformed attribute information decoding unit 204 decodes the attribute information using RAHT (Region Adaptive Hierarchical Transform). Specifically, the transformed attribute information decoding unit 204 decodes the attribute values ​​by applying an inverse RAHT or an inverse Haar transform to the high-frequency components and low-frequency components of each attribute value based on the position information of the three-dimensional points.

[0136] 23 is a block diagram of a transformed attribute information coding unit 202, which is an example of the transformed attribute information coding unit 202. The transformed attribute information coding unit 202 includes a sorting unit 211, a Haar transform unit 212, a quantization unit 213, an inverse quantization unit 214, an inverse Haar transform unit 215, a memory 216, and an arithmetic coding unit 217.

[0137] The sorting unit 211 generates a Morton code using the position information of the 3D points and sorts the multiple 3D points in Morton code order. The Haar transform unit 212 generates coding coefficients by applying a Haar transform to the attribute information. The quantization unit 213 quantizes the coding coefficients of the attribute information.

[0138] The inverse quantization unit 214 inversely quantizes the quantized coding coefficients. The inverse Haar transform unit 215 applies inverse Haar transform to the coding coefficients. The memory 216 stores values ​​of attribute information of multiple decoded 3D points. For example, the attribute information of the decoded 3D points stored in the memory 216 may be used for predicting uncoded 3D points.

[0139] The arithmetic coding unit 217 calculates ZeroCnt from the quantized coding coefficients and arithmetically codes the ZeroCnt. The arithmetic coding unit 217 also arithmetically codes the quantized non-zero coding coefficients. The arithmetic coding unit 217 may binarize the coding coefficients before arithmetic coding. The arithmetic coding unit 217 may also generate and code various header information.

[0140] 24 is a block diagram of the transformed attribute information decoding unit 204, which is an example of the transformed attribute information decoding unit 204. The transformed attribute information decoding unit 204 includes an arithmetic decoding unit 221, an inverse quantization unit 222, an inverse Haar transform unit 223, and a memory 224.

[0141] The arithmetic decoding unit 221 arithmetically decodes the ZeroCnt and the coding coefficients included in the bitstream. Note that the arithmetic decoding unit 221 may also decode various types of header information.

[0142] The inverse quantization unit 222 inverse quantizes the arithmetically decoded coding coefficients. The inverse Haar transform unit 223 applies inverse Haar transform to the inversely quantized coding coefficients. The memory 224 stores values ​​of attribute information of multiple decoded 3D points. For example, the attribute information of the decoded 3D points stored in the memory 224 may be used to predict undecoded 3D points.

[0143] [Slice Division] The encoding device may divide the three-dimensional data into one or more three-dimensional data and encode the divided three-dimensional point cloud. The divided three-dimensional point cloud is called a slice. A slice is a set of points each having position information (geometry) and attribute information (attribute).

[0144] 25 is a block diagram showing an example of the configuration of a three-dimensional data encoding device in this case. The three-dimensional data encoding device includes a data dividing unit 231 and an encoding unit 232.

[0145] The data dividing unit 231 divides the three-dimensional data to generate a plurality of divided three-dimensional data. Each divided three-dimensional data corresponds to a slice. The encoding unit 232 encodes the position information and attribute information of each of the plurality of divided three-dimensional data (slices) to generate encoded data.

[0146] 26 is a block diagram showing an example configuration of a three-dimensional data decoding device in this case. The three-dimensional data decoding device includes a decoding unit 233 and a data combining unit 234. The decoding unit 233 generates a plurality of divided three-dimensional data by decoding the encoded data. Each divided three-dimensional data includes position information and attribute information of a slice. The data combining unit 234 restores the three-dimensional data by combining the plurality of divided three-dimensional data.

[0147] In encoding, slices may or may not have dependencies. If there are no dependencies, slices can be encoded or decoded independently. Therefore, parallel processing of multiple slices can reduce processing time. In addition, partial decoding, which decodes only some slices, can reduce the amount of processing.

[0148] If there is a dependency, an identifier indicating the dependency is stored in the bitstream, and encoding or decoding is performed in order starting from the dependent data (referenced data).

[0149] The number of divisions or the division method may be any method. The three-dimensional data encoding device may determine the shape of an object and divide the point cloud for each object, or may perform division based on the number of points included in a slice. Furthermore, the three-dimensional data encoding device may use map information or location information to perform division based on whether the point cloud is included in the three-dimensional space (tile information).

[0150] Fig. 27 is a diagram showing the relationship between tiles and slices. As shown in Fig. 27, tiles correspond to three-dimensional space, and slices correspond to divided three-dimensional point clouds. Note that multiple tiles may overlap. This enables parallel processing in adaptive encoding or decoding according to content or object, improving the flexibility of the point cloud encoding system or point cloud decoding system.

[0151] [Structure of Encoded Data] The position information and attribute information for each slice are encoded. At least a part of the resulting encoded data is stored in the payload of a data unit. A header is also added to the payload.

[0152]

[0043] When multiple pieces of attribute information exist for each point, each of the multiple pieces of attribute information is stored in a data unit. FIG. 28 is a diagram showing an example of the configuration of a bit stream (encoded data). FIG. 28 shows encoded data of two frames of point clouds having two types of attribute information. The bit stream has a geometry (position information) data unit (Geometry Data Unit: Geom) and two attribute (attribute information) data units (Attribute Data Unit: Attr(0), Attr(1)) for each slice. For example, when two pieces of attribute information, color and reflectance, are present for each point, the encoded data for color is stored in Attr(0), and the encoded data for reflectance is stored in Attr(1). The header of the geometry data unit stores the slice identifier (slice_id). The header of the attribute data unit indicates the slice_id of the corresponding (referring to) geometry data unit. The data unit is sometimes called a slice, and the data unit header is sometimes called a slice header.

[0153] Furthermore, metadata related to encoding of position information is stored in a geometry (position information) parameter set (GPS). Metadata related to encoding of attribute information is stored in an attribute parameter set (APS). Metadata common to multiple PCC frames (PCC sequences) is stored in a sequence parameter set (SPS).

[0154] Each data unit and parameter set is converted into a NAL (Network Access Layer) unit or a TLV (Type Length Value) unit format, and a data sequence (stream) of the units is output.

[0155] 29 is a block diagram showing an example configuration of a three-dimensional data encoding device that outputs a stream of TLV units. The three-dimensional data encoding device includes an encoding unit 241 and a TLV storage unit 242. The encoding unit 241 generates encoded data by encoding point cloud data. The TLV storage unit 242 generates a TLV stream (a single stream of TLV units) by storing (encapsulating) the encoded data in multiple TLV units.

[0156] A unit includes a Type that indicates the data type, a Length that indicates length information, and a Value that stores the data indicated by the Type. Note that a unit may have a different format, such as not including length information.

[0157] Figure 28 shows a sequence of units that store data units and parameter sets. The arrows in Figure 28 indicate dependencies related to the decoding of encoded data. The source of the arrow depends on the data at the end of the arrow, and the header of the original data indicates an identifier for the data at the end of the arrow (reference destination). The three-dimensional data decoding device decodes the data at the end of the arrow and uses the decoded data to decode the data at the source of the arrow. For example, a geometry data unit (Geom) indicates the GPS ID and SPS ID corresponding to the geometry data unit. An attribute data unit (Attr) indicates the APS ID corresponding to the attribute data unit.

[0158] One GPS and one APS may be provided for each frame. Alternatively, if the encoding method is changed for each slice, one GPS and one APS may be provided for each slice. Furthermore, the GPS and the APS may be shared by multiple frames (sequences).

[0159] In addition, the APS may be shared by multiple pieces of attribute information. The parameter set of the reference destination is sent before the parameter set of the reference source.

[0160] When inter-frame prediction is used, for example, if there is a dependency between data units of frames, a GOF (Group of Frame) including multiple frames may be configured. A GOF is a random access unit, and the first slice of a GOF has no dependency and is the slice from which decoding starts. The stream may have a delimiter indicating a frame boundary or a GOF boundary, or a TLV unit indicating the boundary.

[0161] Next, the structure of the coded data and the method of storing the coded data in the NAL unit will be described.

[0162] For example, a data format is defined for each type of encoded data. Fig. 30 shows examples of encoded data and NAL units.

[0163] For example, as shown in Figure 30, the encoded data includes a header and a payload. The encoded data may include length information indicating the length (amount of data) of the encoded data, header, or payload. The encoded data may be in a TLV format including the length information. The encoded data may not include a header.

[0164] The header includes, for example, identification information for identifying the data, such as the data type or frame number.

[0165] The header includes, for example, identification information indicating a reference relationship. This identification information is stored in the header when, for example, there is a dependency relationship between data, and is information for referencing the reference destination from the reference source. For example, the header of the reference destination includes identification information for identifying the data. The header of the reference source includes identification information indicating the reference destination.

[0166] If the reference destination or the reference source can be identified or derived from other information, the identification information for specifying the data or the identification information indicating the reference relationship may be omitted.

[0167] The three-dimensional data encoding device stores encoded data in the payload of a NAL unit. The NAL unit header includes pcc_nal_unit_type, which is identification information of the encoded data. Figure 31 is a diagram showing an example of the semantics of pcc_nal_unit_type.

[0168] As shown in FIG. 31, when pcc_codec_type is codec 1 (Codec1: first encoding method), the values ​​0 to 10 of pcc_nal_unit_type correspond to the encoded position data (Geometry), encoded attribute X data (AttributeX), encoded attribute Y data (AttributeY), position PS (Geom.PS), attribute XPS (AttrX.PS), attribute YPS (AttrY.PS), position SPS (Geometry Sequence PS), attribute XSPS (AttributeX Sequence PS), attribute YSPS (AttributeY Sequence PS), AU header (AU Header), and GOF header (GOF Values ​​11 and above are assigned as spares for Codec 1.

[0169] A modified example of this embodiment will be described below. PS has levels, such as a frame-level PS, a sequence-level PS, and a PCC sequence-level PS, and if the PCC sequence level is the higher level and the frame level is the lower level, the following method may be used to store parameters.

[0170] The default PS value is indicated in the higher PS. Also, if the value of the lower PS differs from the value of the higher PS, the PS value is indicated in the lower PS. Alternatively, the PS value is not written in the higher PS, but written in the lower PS. Alternatively, information on whether the PS value is to be written in the lower PS, the higher PS, or both is written in either or both of the lower PS and the higher PS. Alternatively, the lower PS may be merged with the higher PS. Alternatively, if the lower PS and the higher PS overlap, the three-dimensional data encoding device may omit sending one of them.

[0171] The three-dimensional data encoding device may divide data into slices or tiles and transmit the divided data. The divided data includes information for identifying the divided data, and parameters used to decode the divided data are included in the parameter set. In this case, an identifier indicating that the data stores data or parameters related to tiles or slices is defined in pcc_nal_unit_type.

[0172] [Gaussian Data] Gaussian data (Gaussian Splatting 3D data) will be described below. First, the configuration of a system that encodes or decodes three-dimensional data generated using the Gaussian Splatting method will be described.

[0173] Fig. 32 is a diagram showing a part of the processing in the three-dimensional data generation system shown in Fig. 1. A sensor information input unit 301 and a three-dimensional data generation unit 302 shown in Fig. 32 correspond to, for example, the sensor information acquisition unit 117 and the three-dimensional data generation unit 118 shown in Fig. 1.

[0174] The sensor information input unit 301 acquires point cloud data (three-dimensional point cloud data) and outputs the acquired point cloud data to the three-dimensional data generation unit 302. The three-dimensional data generation unit 302 generates Gaussian data (Gaussian splatting data) from the point cloud data and outputs the generated Gaussian data.

[0175] Specifically, the three-dimensional data generation unit 302 first generates a grid according to the density of the input point cloud data. Each point of the point cloud data is projected onto a nearby grid. At this time, attribute information associated with the point, such as color, reflectance, or normal, is also simultaneously associated with the grid. The three-dimensional data generation unit 302 derives parameters such as the mean value and variance of the Gaussian function from the information projected onto the grid.

[0176] Note that the method for generating Gaussian data is not limited to the above. For example, Gaussian data may be generated by machine learning using point cloud data generated from multiple two-dimensional images using structure from motion (SfM) or the like.

[0177] Gaussian data is a collection of multiple 3D Gaussians (ellipsoids). Each 3D Gaussian includes data such as three-dimensional point coordinates, a 3x3 covariance matrix, color, and transparency. The above 3D Gaussian data can also be converted into components described in the PLY format and expressed.

[0178] Fig. 33 is a diagram showing the components of Gaussian data described in the PLY format. The Gaussian data shown in Fig. 33 includes position, rotation, scale, SH coefficient, and transparency. Note that transparency is also called transparency, transparency, or opacity.

[0179] Position is the three-dimensional coordinate of a three-dimensional point. Scale and Orientation are parameters of a three-dimensional covariance matrix. SH Coefficient is the coefficient of a spherical harmonic function when color information is expressed as a spherical harmonic function.

[0180] That is, the Gaussian data includes, for each 3D Gaussian, three-dimensional coordinates (position) that serve as a reference, and information associated with the three-dimensional coordinates (rotation, scale, SH coefficient, transmittance).

[0181] The configuration of the Gaussian data is not limited to the above. For example, the Gaussian data does not need to include a covariance matrix. For example, the Gaussian data includes a position (Position) indicating the three-dimensional coordinates of a three-dimensional point, a scale (Scale) indicating the scale along each axis, a color (Color) indicating, for example, RGB, and a transparency (Transparency).

[0182] Furthermore, the Gaussian data may include a plurality of cubes (voxels) that divide a 3D space instead of a plurality of ellipsoids. Each voxel has attribute information. For example, each voxel may include a position (Position) indicating the three-dimensional coordinates of the center point of the voxel, attribute information such as color, reflectance, or material, and a scale (Scale) indicating the size of the voxel.

[0183] Furthermore, the position (Position) may be represented by two-dimensional point coordinates (x, y) instead of three-dimensional coordinates. In this case, the scale (Scale) may indicate the size or shape of the ellipsoid on a two-dimensional plane. Specifically, the Gaussian data may include a position (Position) indicating a position on a two-dimensional plane, a scale vector (Scale Vector) indicating the scale (Su, Sv) along each axis, and rotation information (Rotation) indicating the orientation of the ellipsoid.

[0184] The format for writing Gaussian data is not limited to the PLY format, but may be, for example, the SPZ format.

[0185] [Gaussian Data Rendering Process] Next, the Gaussian data rendering process will be described. FIG. 34 is a diagram showing the Gaussian data rendering process. As shown in FIG. 34, the rendering unit 303 generates a 3D model (three-dimensional model) by rendering the Gaussian data. This makes it possible to present the generated 3D model in an application such as a 3D display device. The 3D display device is, for example, a device that displays a stereoscopic image in space like a hologram, or a naked-eye 3D display device.

[0186] 35 is a diagram showing an example of the number of SH coefficients of a spherical harmonic function. The number of elements (the number of SH coefficients) of a spherical harmonic function is determined according to the level indicating the resolution. For example, a higher level (a larger numerical value of the level) indicates a higher resolution. For example, the higher the level, the more information of high frequency components is added.

[0187] For example, when expressing level 2, the spherical harmonic function has a total of nine elements (SH coefficients) from levels 0 to 2. For example, when expressing level 3, the spherical harmonic function has a total of 16 elements (SH coefficients) from levels 0 to 3.

[0188] For example, the example shown in Figure 33 shows a spherical harmonic function for a level 3 color, where each of the three color elements (R, G, B) is expressed by 16 SH coefficients. Therefore, the spherical harmonic function has 16 x 3 = 48 elements. Note that levels 4 and above can also be defined.

[0189] By expressing color using spherical harmonic functions, it is possible to express three-dimensional color information. Fig. 36 is a diagram showing an example of input and output of a spherical harmonic function. For example, as shown in Fig. 36, by inputting viewpoint information into a color spherical harmonic function 304, color information (R, G, B) seen from that viewpoint is output.

[0190] 37 is a diagram showing the rendering process. As shown in FIG. 37, the rendering unit 305 generates a 3D model or a 2D image viewed from a specified viewpoint based on Gaussian data and the specified viewpoint. The generated 3D model can be presented in an application such as a VR display. The generated 2D image can also be presented in an application such as a 2D image display device.

[0191] Note that, in addition to color, other attribute information (such as reflectance or infrared information) may be expressed using spherical harmonic functions. In this case, the Gaussian data includes SH coefficients of the spherical harmonic functions for each attribute information. Furthermore, the level may be changed depending on the attribute information or resolution.

[0192] [Encoding and Decoding of Gaussian Data] Next, encoding and decoding of Gaussian data will be described. Fig. 38 is a block diagram of an encoding device (encoding system) according to this embodiment. This encoding device encodes and multiplexes Gaussian data. Fig. 39 is a block diagram of a decoding device (decoding system) according to this embodiment. This decoding device decodes the Gaussian data and presents a 3D model or a 2D image in an application. The encoding device and decoding device shown in Figs. 38 and 39 are included in, for example, the three-dimensional data generation system shown in Fig. 1.

[0193] 38 includes an encoding unit 311 and a multiplexing unit 312. The encoding unit 311 generates encoded data (bit stream) by encoding input Gaussian data using a predetermined encoding method, and outputs the generated encoded data to the multiplexing unit 312.

[0194] The multiplexing unit 312 multiplexes the input coded data using a predetermined multiplexing method to generate multiplexed data, and outputs the generated multiplexed data. This multiplexed data is stored or transmitted.

[0195] 39 includes a demultiplexing unit 321, a decoding unit 322, and an application unit 323. The demultiplexing unit 321 demultiplexes input multiplexed data using a predetermined multiplexing method to generate coded data, and outputs the generated coded data to the decoding unit 322.

[0196] The decoding unit 322 generates Gaussian data by decoding the input coded data using a predetermined coding method (decoding method), and outputs the generated Gaussian data to the application unit 323 .

[0197] The application unit 323 includes an input interface unit 324, a rendering unit 325, and a presentation unit 326. The input interface unit 324 acquires a user operation. For example, the input interface unit 324 generates viewpoint information indicating a viewpoint input by the user based on the user operation.

[0198] The rendering unit 325 generates a three-dimensional model or a two-dimensional image by rendering the input Gaussian data. For example, the rendering unit 325 generates a 3D model or a 2D image viewed from a viewpoint input by the user, which viewpoint information indicates. Note that the rendering unit 325 may also generate a 3D model or a 2D image viewed from a predetermined viewpoint.

[0199] The presentation unit 326 presents (displays) a three-dimensional model or a two-dimensional image generated by the rendering unit 325. Whether to present a 3D model or a 2D image may be determined, for example, by the following method. For example, the rendering unit 325 first generates a three-dimensional model and generates point coordinate information, as in the process shown in FIG. 34 . After viewpoint information is input, the rendering unit 325 generates a 2D image viewed from a specified viewpoint. In this case, the rendering unit 325 may also receive information specifying an area along with the specified viewpoint. In this way, by first generating a 3D model and then generating a 2D image of the specified area, the amount of processing can be reduced compared to generating a 2D image of the entire area. This allows for increased rendering speed. Alternatively, whether to present a 3D display (3D model) or a 2D image may be determined depending on the processing capabilities of the device performing the rendering. For example, in the case of a device with low processing capabilities, the device may present a 3D model without generating a 2D image.

[0200] (First Aspect) [Encoding of Gaussian Data Using G-PCC] Fig. 40 is a block diagram showing an example configuration of an encoding device 330 (three-dimensional data encoding device) according to this embodiment. The encoding device 330 generates a bit stream (encoded data) by encoding first Gaussian data.

[0201] The encoding device 330 includes a first pre-processing unit 331 , a second pre-processing unit 332 , and a G-PCC encoding unit 333 .

[0202] The first Gaussian data includes, for example, a covariance matrix. The first pre-processing unit 331 converts the first Gaussian data into second Gaussian data composed of elements in a ply format. For example, the elements of a ply file include three-dimensional coordinates, scale, rotation, SH coefficients of spherical harmonics, and transmittance, as shown in FIG.

[0203] Note that, although an example will be described here in which first Gaussian data is input to the encoding device 330 and the encoding device 330 converts the first Gaussian data into second Gaussian data, the second Gaussian data may also be input to the encoding device 330. In this case, the encoding device 330 does not need to include the first pre-processing unit 331.

[0204] The second pre-processing unit 332 converts the multiple components contained in the second Gaussian data into a format that can be encoded using an encoding method corresponding to each component, and outputs the converted multiple components to the G-PCC encoding unit 333.

[0205] The G-PCC encoding unit 333 encodes three-dimensional coordinates using a geometry (position information) encoding method (such as an octet tree encoding method, a predictive tree encoding method, or a TriSoup method) in the G-PCC encoding method (Geometry based PCC). The G-PCC encoding unit 333 encodes information associated with three-dimensional coordinates, such as rotation, scale, SH coefficients, and transmittance, using an attribute encoding method in the G-PCC encoding method (such as an LoD-based encoding method (LoD attribute information encoding method) or a Transform-based encoding method (transform attribute information encoding method)).

[0206] The G-PCC encoding unit 333 includes a position information encoding unit 341, a rotation encoding unit 342, a scale encoding unit 343, an SH coefficient encoding unit 344, a transmittance encoding unit 345, a metadata encoding unit 346, and a multiplexing unit 347.

[0207] The position information encoding unit 341 encodes three-dimensional coordinates. The three-dimensional coordinates are reference or representative three-dimensional coordinates for the Gaussian data. For example, the three-dimensional coordinates may be the center coordinates of an ellipse that constitutes the Gaussian data, or may be other coordinates. For example, the three-dimensional coordinates may be the origin coordinates of a grid used to generate the Gaussian data.

[0208] The three-dimensional coordinates are not processed by the first pre-processing unit 331, but are input directly to the second pre-processing unit 332. The second pre-processing unit 332 converts the input three-dimensional coordinates into positive integer information (three-dimensional coordinates). The scale value and offset value used in the conversion are stored in metadata such as an SPS, and are notified to the decoding device.

[0209] The position information encoding unit 341 generates encoded data by encoding the three-dimensional coordinates output from the second pre-processing unit 332 using the G-PCC geometry encoding method. This encoded data is stored in a GDU (Geometry Data Unit) and output. In addition, metadata required for decoding the GDU is stored in a GPS (Geometry Parameter Set) and output.

[0210] Furthermore, when the three-dimensional coordinates of the multiple Gaussian data are sparse (the density is less than a predetermined threshold), the position information encoding unit 341 may use a predictive tree encoding method suitable for encoding sparse three-dimensional points. Furthermore, when the three-dimensional coordinates of the multiple Gaussian data are dense (the density is equal to or greater than a predetermined threshold), the position information encoding unit 341 may use an octree encoding method suitable for encoding dense three-dimensional points.

[0211] The first pre-processing unit 331 converts the covariance matrix included in the first Gaussian data into a rotation value and a scale value, converts color information included in the first Gaussian data into a spherical harmonic function, and outputs SH coefficients in the spherical harmonic function.

[0212] The second pre-processing unit 332 converts the data type of each piece of data included in the second Gaussian data. Specifically, the second pre-processing unit 332 converts each piece of data into positive integer data in order to encode it using the G-PCC encoding method. For example, if the data is float type, the second pre-processing unit 332 converts the data into positive integer data by performing scaling and offset processing on the data. At this time, the scale value and offset value for each piece of attribute information used in the conversion are stored in metadata such as an SPS and notified to the decoding device.

[0213] The second pre-processing unit 332 also performs format conversion for each data item and mapping processing for mapping each data item to an attribute component. For example, if the maximum number of elements (number of dimensions) per attribute component that can be coded using the G-PCC attribute coding method is three, the second pre-processing unit 332 converts the input configuration element into attribute components with three dimensions (three elements) per attribute component. For example, since scale (Scale) has three elements, it is output as a single attribute component. Furthermore, since rotation (Orientation) has four elements, it is converted into two two-dimensional attribute components. The number of elements for the SH coefficients changes for each level. For example, SH coefficients with levels greater than three are converted into multiple attribute components.

[0214] At this time, the second pre-processing unit 332 generates, as metadata, mapping information indicating to which attribute component each element is mapped for each piece of attribute information.

[0215] The rotation encoding unit 342, scale encoding unit 343, SH coefficient encoding unit 344, and transmittance encoding unit 345 generate a plurality of encoded data by encoding the rotation, scale, SH coefficient, and transmittance, respectively, using an attribute encoding method in the G-PCC encoding method. Each of the generated plurality of encoded data is stored in an ADU (Attribute Data Unit) and output. Metadata required for decoding the ADU is stored in an APS (Attribute Parameter Set) and output. In addition, the attribute information is encoded using three-dimensional coordinates.

[0216] The metadata encoding unit 346 stores transformation information and the like including the transformation parameters used in the first pre-processing unit 331 or the second pre-processing unit 332 in metadata such as an SPS or SEI. Note that the metadata encoding unit 346 may encode the transformation information and then store it in the metadata. Furthermore, the metadata may include configuration information of the Gaussian data or information indicating a correspondence relationship of how the Gaussian data is transmitted by G-PCC encoding.

[0217] The multiplexing unit 347 stores the coded data (ADU and GDU) and the parameter set (APS, GPS, and SPS) in a TLV unit and transmits it as a bit stream (coded data). The bit stream may be multiplexed using a predetermined multiplexing method. The bit stream may also be formatted.

[0218] [Decoding Gaussian Data Using G-PCC] Fig. 41 is a block diagram showing an example configuration of a decoding device 350 (three-dimensional data decoding device) according to this embodiment. The decoding device 350 generates first Gaussian data or second Gaussian data by decoding a bit stream (encoded data). For example, the decoding device 350 decodes the bit stream generated by the encoding device 330 shown in Fig. 40.

[0219] The decoding device 350 includes a G-PCC decoding unit 351 , a second post-processing unit 352 , and a first post-processing unit 353 .

[0220] The G-PCC decoding unit 351 generates three-dimensional coordinates, rotation, scale, SH coefficients, and transmittance by decoding the bitstream using the G-PCC encoding method. The G-PCC decoding unit 351 includes a demultiplexing unit 361, a position information decoding unit 362, a rotation decoding unit 363, a scale decoding unit 364, an SH coefficient decoding unit 365, a transmittance decoding unit 366, and a metadata decoding unit 367.

[0221] The demultiplexing unit 361 analyzes a plurality of TLV units included in the input bitstream and generates data units of coded data such as GDU, ADU, SPS, GPS, APS, etc. The position information decoding unit 362, rotation decoding unit 363, scale decoding unit 364, SH coefficient decoding unit 365, transmittance decoding unit 366, and metadata decoding unit 367 decode these data units using the respective coding methods for each attribute component using information in the parameter set.

[0222] The position information decoding unit 362 decodes three-dimensional coordinates from the GDU. The rotation decoding unit 363, scale decoding unit 364, SH coefficient decoding unit 365, and transmittance decoding unit 366 decode attribute information (rotation, scale, SH coefficient, and transmittance) from the ADU. The attribute information is decoded using the three-dimensional coordinates. The metadata decoding unit 367 acquires transformation information and the like from metadata such as SPS or SEI. The metadata decoding unit 367 may decode the transformation information from the metadata. The metadata may also include configuration information of the Gaussian data or information indicating a correspondence relationship between how the Gaussian data is transmitted using G-PCC encoding.

[0223] The second post-processing unit 352 generates second Gaussian data by inversely transforming the decoded data. Here, the inverse transform is a process opposite to the transform process performed by the second pre-processing unit 332, and is performed based on the transform information (transformation parameters) included in the metadata.

[0224] The first post-processing unit 353 generates first Gaussian data by inversely transforming the second Gaussian data. Here, the inverse transform is a process opposite to the transform process performed by the first pre-processing unit 331. The first post-processing unit 353 outputs at least one of the first Gaussian data and the second Gaussian data. Note that the decoding device 350 does not necessarily have to include the first post-processing unit 353.

[0225] [Mapping Process] The second pre-processing unit 332 maps the Gaussian data to the attribute components of the G-PCC. The mapping method and the mapping information generated at that time will be described.

[0226] In the following, an example of mapping rotation, scale, and SH coefficients will be shown, and a description of transmittance will be omitted. Note that similar processing can be applied when other attribute information is added.

[0227] 42 is a diagram showing an example of mapping multiple attribute components included in Gaussian data to attribute components of G-PCC, which is a component of the encoding method. In this example, the maximum number of dimensions (number of subcomponents) that can be encoded in encoding of attribute components in G-PCC is 3.

[0228] In the following description, the attribute component of Gaussian data may be referred to as a Gaussian component, and the attribute component of G-PCC may be referred to as a G-PCC component. Also, the attribute component may be simply referred to as a component.

[0229] For example, a Gaussian component of a scale has subcomponents (B1 to B3) with a dimensionality of 3, and is mapped to one G-PCC component having three-dimensional subcomponents. In this case, the mapping information indicates that the identifiers of the Gaussian component of the scale and the identifiers of the G-PCC component correspond to each other.

[0230] Furthermore, the Gaussian rotation component has subcomponents (A1 to A4) with a dimensionality of four, and cannot be mapped to a single G-PCC component with three-dimensional subcomponents. Therefore, the four rotation subcomponents (A1 to A4) are mapped to two G-PCC components with two-dimensional subcomponents. In this case, the mapping information indicates that the identifier of the Gaussian rotation component and the identifier indicating the subcomponent number included in the attribute component correspond to the identifier of the G-PCC component and the identifier indicating the subcomponent number included in the G-PCC component.

[0231] Since there are 16 SH coefficients per level for each three-dimensional (RGB) element, the SH coefficients have a total of 48 dimensional elements (C1 to C48), which are mapped to 16 G-PCC components with three-dimensional subcomponents.

[0232] 42, "ID" is an attribute component ID, which indicates an identification number for identifying an attribute component of encoded data. "ND" is the number of dimensions, which indicates the number of elements (number of subcomponents) per attribute component.

[0233] Furthermore, the division method and mapping method of Gaussian data are not limited to the above examples, and any combination of mapping method and number of dimensions may be used.

[0234] In the mapping process, the second pre-processing unit 332 generates mapping information indicating the G-PCC components corresponding to the Gaussian components, and stores the mapping information as metadata in the header or SEI.

[0235] The decoding device decodes the mapping information from the bitstream. The second post-processing unit 352 remaps the decoded data for each G-PCC component to Gaussian data based on the mapping information, thereby reconstructing the Gaussian data.

[0236] Fig. 43 is a diagram showing an example of mapping information. In Fig. 43, the mapping information is stored in SEI (Gaussian data attribute mapping information SEI). The mapping information includes the number of Gaussian data components. The number of Gaussian data components indicates the number of attribute information included in the Gaussian data. For example, in the example shown in Fig. 42, rotation, scale, and SH coefficients are present, and the number of Gaussian data components is 3.

[0237] The mapping information also includes, as information for each Gaussian component, a Gaussian data component ID, a Gaussian data type, and the number of Gaussian data dimensions.

[0238] The Gaussian component ID is identification information for uniquely identifying a Gaussian component. The Gaussian data type indicates the type of the Gaussian component. For example, a value of 0 indicates "rotation," a value of 1 indicates "scale," a value of 2 indicates "SH coefficient," and a value of 3 indicates "transmittance." Note that the combinations of values ​​and types here are merely examples, and the combinations are not limited to these.

[0239] The Gaussian data dimension indicates the dimension of a Gaussian component (the number of subcomponents). For example, in the example shown in Fig. 42, the Gaussian data dimension of rotation is 4, the Gaussian data dimension of scale is 3, and the Gaussian data dimension of SH coefficients is 48.

[0240] The mapping information also includes a G-PCC component ID, a G-PCC dimension ID, and conversion information as information for each dimension of the Gaussian component. The G-PCC subcomponent ID corresponds to "ID" shown in FIG. 42. The G-PCC dimension ID is an identifier indicating the subcomponent number. For example, C1 included in the SH coefficients shown in FIG. 42 has a G-PCC dimension ID of 0, C2 has a G-PCC dimension ID of 1, and C3 has a G-PCC dimension ID of 2.

[0241] In this way, the mapping information indicates the G-PCC component ID and dimension ID corresponding to each dimension of each component of the Gaussian data. For example, for subcomponent C46 shown in Figure 42, a G-PCC component ID of 18 and a G-PCC dimension ID of 2 are indicated.

[0242] Note that Figure 43 shows an example of mapping information indicating the subcomponents (dimensions) of the G-PCC corresponding to each subcomponent (dimension) of the Gaussian data, but mapping information indicating the subcomponents of the Gaussian data corresponding to each subcomponent (dimension) of the G-PCC may also be used.

[0243] 43 shows an example of mapping information indicating the correspondence between subcomponents of Gaussian data and subcomponents of G-PCC, but if there is no restriction on the number of subcomponents in a G-PCC component, mapping information indicating the correspondence between Gaussian components and G-PCC components may be used. In other words, it is not necessary to indicate the correspondence between dimensions.

[0244] The conversion information indicates the parameters (conversion parameters) used when converting the component data. Fig. 44 shows an example of the configuration of the conversion information. For example, if a scale and an offset are used in the conversion, the conversion information includes a scale value and an offset value.

[0245] For example, the conversion is performed using the scale value and offset value according to the following formula:

[0246] Post-conversion data = pre-conversion data x scale value + offset

[0247] The formula used for the conversion may be a predefined formula, or multiple formulas may be defined and the formula to be used may be switched according to predetermined conditions. Furthermore, information indicating the formula used may be stored in the bitstream.

[0248] 43, the transformation information is provided for each subcomponent (dimension) of the Gaussian data, but the transformation information may be provided for each component of the Gaussian data. In other words, the transformation method may be changed for each component or for each subcomponent.

[0249] Note that some of the syntax shown in Fig. 43 may be omitted. For example, if the order of description of the information on rotation, scale, and SH coefficients and the number of dimensions are predetermined, the Gaussian data component ID, Gaussian data type, the number of Gaussian data dimensions, etc. may be omitted.

[0250] Furthermore, each piece of attribute information is encoded based on the mapping information to the attribute component determined by the second pre-processing unit 332. Fig. 45 is a diagram showing an example of the structure of encoded data when Gaussian data is G-PCC encoded.

[0251] In the SPS, for each G-PCC component, a G-PCC attribute component ID (attr_id), a G-PCC attribute type (attribute_type), and the number of dimensions of the G-PCC attribute component (num_dimension) are described. Here, a new G-PCC attribute type may be defined that indicates that each element of Gaussian data is to be coded.

[0252] 46 is a diagram showing an example of G-PCC attribute types for Gaussian data. As shown in FIG. 46, for example, G-PCC attribute types may be defined that respectively indicate the rotation, scale, SH coefficient, and transmittance of Gaussian data.

[0253] A G-PCC attribute component ID (attr_id) is assigned to the header of each ADU. A decoding device can identify the G-PCC attribute type of each ADU by using the G-PCC attribute component ID included in the ADU header and the information included in the SPS.

[0254] Furthermore, the mapping information (Gaussian data attribute mapping information SEI) may be transmitted as SEI or may be included in the SPS. For example, the SPS may include mapping information indicating Gaussian components corresponding to G-PCC components.

[0255] 47 is a flowchart of processing by the decoding device according to the present embodiment. First, the decoding device acquires a plurality of TLV units from a bit stream, and acquires a plurality of data units (GDU, ADU) and a plurality of parameter sets (SPS, GPS, APS) from the plurality of TLV units (S101).

[0256] Next, the decoding device decodes the three-dimensional coordinates using the SPS, GPS, and GDU (S102). Next, the decoding device decodes the multiple attribute components described in the SPS using the APS, ADU, and three-dimensional coordinates (S103).

[0257] Next, the decoding device analyzes the Gaussian data attribute mapping information SEI to obtain mapping information (S104). Next, the decoding device reconstructs second Gaussian data from the decoded attribute components and the mapping information (S105). Next, the decoding device obtains transformation information and generates first Gaussian data by inversely transforming the second Gaussian data using the transformation information (S106).

[0258] [Others] In the above description, for simplicity, an example in which Gaussian data is one frame is shown, but this method can also be applied to cases in which Gaussian data is present in multiple frames. In this case, multi-frame coding in the G-PCC method or inter-prediction in the G-PCC method may be used.

[0259] The encoding device may also generate a plurality of pieces of divided data by dividing the Gaussian data into a plurality of regions and encode each of the plurality of pieces of divided data. The encoding device may also assign the pieces of divided data to tiles or slices in the G-PCC and encode them.

[0260] In the above explanation, an example was shown in which attribute information (scale, rotation, SH coefficient, transmittance, etc.) of the second Gaussian data was encoded, but by using the method of this embodiment, it is also possible to encode other attribute information.

[0261] Fig. 48 is a block diagram showing a system for encoding and decoding first Gaussian data. The system shown in Fig. 48 includes a second pre-processing unit 371, a G-PCC encoding unit 372, a G-PCC decoding unit 373, and a second post-processing unit 374. Fig. 49 is a block diagram showing an example configuration of the second pre-processing unit 371. The second pre-processing unit 371 includes a conversion unit 375 and a mapping unit 376.

[0262] For example, when encoding a covariance matrix associated with three-dimensional coordinates included in the first Gaussian data, the conversion unit 375 included in the second pre-processing unit 371 converts the format of the covariance matrix. The mapping unit 376 generates converted first Gaussian data by mapping the format-converted covariance matrix to G-PCC attribute information. The G-PCC encoding unit 372 generates encoded data by encoding the converted first Gaussian data, the conversion information, and the mapping information.

[0263] The G-PCC decoding unit 373 decodes the encoded data to generate transformed first Gaussian data, transformation information, and mapping information. The second post-processing unit 374 uses the mapping information and transformation information to inversely transform the transformed first Gaussian data, thereby restoring the first Gaussian data. In other words, the covariance matrix is ​​restored. In this way, the system can encode and decode the first Gaussian data.

[0264] Furthermore, in converting the covariance matrix, the conversion unit 375 may convert a 3x3 covariance matrix into a format that can be encoded by G-PCC. For example, the conversion unit 375 divides the 3x3 covariance matrix into three three-dimensional components. Alternatively, if the covariance matrix is ​​a symmetric matrix, the conversion unit 375 may divide the six-dimensional elements, excluding elements having the same value, into two three-dimensional components. FIG. 50 is a diagram illustrating this conversion process. In the example shown in FIG. 50 , of the nine elements of the covariance matrix, six elements ([Sxx, Sxy, Sxz, Syy, Syz, Szz]) are extracted, excluding elements (Syz, Szx, Szy) having the same value as other elements.

[0265] Furthermore, conversion information indicating the details of the conversion is generated. The conversion information is stored in the metadata. Note that if the conversion method is uniquely determined in advance, the conversion information does not need to be stored in the coded data (bitstream).

[0266] As described above, by using the processes and methods described in this embodiment, it is possible to encode Gaussian data using G-PCC.

[0267] Specifically, the encoding device encodes three-dimensional coordinates included in the Gaussian data using position information encoding in G-PCC, and also encodes attribute information (scale, rotation, SH coefficient, transparency, etc.) associated with the three-dimensional coordinates using attribute information encoding in G-PCC.

[0268] The encoding device also converts each element of the attribute information of the Gaussian data into a data format that can be encoded by G-PCC. The encoding device also stores conversion information indicating the content of the conversion in metadata. This allows the decoding device to reconstruct the Gaussian data from the decoded data.

[0269] The encoding device also associates components or dimensions of Gaussian data with attribute components or dimensions that are units that can be coded using G-PCC. This allows the encoding device to code the Gaussian data using the G-PCC coding method. Furthermore, by defining a G-PCC attribute type that corresponds to the Gaussian data, the encoding device can code the Gaussian data using the G-PCC coding method.

[0270] The encoding device also stores mapping information in the metadata that indicates the correspondence between the components and dimensions of the Gaussian data and the attribute components and dimensions, allowing the decoding device to reconstruct the Gaussian data from the decoded data.

[0271] (Second Aspect) [Gaussian Data Encoding Method] Hereinafter, a general-purpose data structure for encoding Gaussian data and a data structure that enables Gaussian data to be encoded by G-PCC or other encoding methods will be described.

[0272] The following data structure provides a general-purpose method for encoding Gaussian data using various 3D encoding methods or 3D codec standards, allowing Gaussian data to be encoded using various 3D codec methods, including standards created by various standards organizations, international standards, national standards, and standards established by any organization.

[0273] 51 is a diagram showing a protocol stack of an encoding standard for encoding Gaussian data. 3D Gaussian codec is an encoding standard for encoding Gaussian data. Specific encoding methods for the 3D Gaussian codec include the V3C standard (ISO / IEC 23090-5), which encodes 3D data using one or more video codecs, the G-PCC standard (ISO-IEC 23090-9), which is a point cloud compression standard, and Draco, which is used for 3D mesh data. When the V3C standard is used, a video codec standard such as HEVC or VVC (Versatile Video Coding) is used.

[0274] 52 is a block diagram showing an example configuration of an encoding device 400 (three-dimensional data encoding device) according to this embodiment. The encoding device 400 generates a bit stream (encoded data) by encoding second Gaussian data. This encoding device 400 includes a second pre-processing unit 401, a G-PCC encoding unit 402, a V3C encoding unit 403, a metadata encoding unit 404, and a multiplexing unit 405.

[0275] The second pre-processing unit 401 converts and maps the second Gaussian data into data that can be encoded by the 3D codec used, depending on the 3D codec used.

[0276] For example, the second preprocessing unit 401 converts and maps the second Gaussian data into data that can be coded using the G-PCC coding method. The G-PCC coding unit 402 codes the converted data using the G-PCC coding method to generate a G-PCC TLV unit.

[0277] The second preprocessing unit 401 converts and maps the second Gaussian data into data that can be coded using the V3C coding method. The V3C coding unit 403 codes the converted data using the V3C coding method to generate a V3C unit.

[0278] The metadata encoding unit 404 generates metadata by encoding the transformation information and mapping information related to the transformation and mapping performed by the second pre-processing unit 401. For example, the metadata encoding unit 404 stores the transformation information and mapping information in the metadata.

[0279] The multiplexing unit 405 generates a bitstream by unitizing and multiplexing the G-PCC TLV unit or V3C unit and the metadata. Specifically, the multiplexing unit 405 stores the G-PCC TLV unit or V3C unit and the metadata in a 3D data unit, which is a data unit of a 3D Gaussian Codec Layer.

[0280] Fig. 53 is a block diagram showing an example configuration of a decoding device 410 (three-dimensional data decoding device) according to this embodiment. The decoding device 410 generates second Gaussian data by decoding a bit stream. This bit stream is generated, for example, by the encoding device 400 shown in Fig. 52. The decoding device 410 includes a demultiplexing unit 411, a G-PCC decoding unit 412, a V3C decoding unit 413, a metadata decoding unit 414, and a second post-processing unit 415.

[0281] The demultiplexing unit 411 demultiplexes and demultiplexes the 3D data units included in the bitstream to generate G-PCC TLV units or V3C units and metadata.

[0282] The G-PCC decoder 412 generates decoded data by decoding the G-PCC TLV unit using the G-PCC decoding method. The V3C decoder 413 generates decoded data by decoding the V3C unit using the V3C decoding method.

[0283] The metadata decoding unit 414 generates the conversion information and the mapping information by decoding the metadata. For example, the metadata decoding unit 414 obtains the conversion information and the mapping information from the metadata.

[0284] The second post-processing unit 415 inversely transforms the decoded data into second Gaussian data according to the 3D codec used for decoding.

[0285] 54 is a flowchart of the encoding process performed by the encoding device 400. First, the encoding device 400 determines the codec type to be used for encoding (S201). For example, the codec type to be used for encoding may be specified by the user, or may be determined based on the contents of the Gaussian data.

[0286] If the codec type used for encoding is the G-PCC encoding method (G-PCC in S201), the encoding device 400 encodes the Gaussian data using the G-PCC encoding method and outputs a G-PCC TLV unit (S202).

[0287] On the other hand, if the codec type used for encoding is the V3C encoding method (V3C in S201), the encoding device 400 encodes the Gaussian data using the V3C encoding method and outputs a V3C unit (S203).

[0288] Next, the encoding device 400 stores information indicating the codec type used for encoding in common metadata (S204). Here, the common metadata is metadata that can be used in common across multiple encoding methods (G-PCC encoding method and V3C encoding method).

[0289] Next, the encoding device 400 stores the conversion information and mapping information in the metadata (S205). Next, the encoding device 400 multiplexes the metadata and the encoding unit (G-PCC TLV unit or V3C unit) into a 3D data unit that can be commonly used in the G-PCC encoding method and the V3C encoding method (S206).

[0290] 55 is a flowchart of the decoding process by the decoding device 410. First, the decoding device 410 obtains a common 3D data unit, and then obtains data units and metadata from the 3D data unit (S211). Next, the decoding device 410 analyzes the metadata and determines the codec type used for encoding (the codec type to be used for decoding) based on information indicating the codec type included in the metadata (S212).

[0291] If the codec type used for encoding is the G-PCC encoding method (G-PCC in S213), the decoding device 410 acquires the TLV unit and decodes it using the G-PCC decoding method to generate a component (S214). On the other hand, if the codec type used for encoding is V3C (V3C in S213), the decoding device 410 acquires the V3C unit and decodes it using the V3C decoding method to generate a component (S215).

[0292] Next, the decoding device 410 reconstructs Gaussian data using the decoded components and the mapping information (S216). Next, the decoding device 410 obtains transformation information and generates second Gaussian data by inversely transforming the Gaussian data using the transformation information (S217).

[0293] 56 is a diagram showing the basic data structure of the Gaussian codec. The 3D data unit in the 3D Gaussian codec layer is a common unit in the Gaussian codec.

[0294] Fig. 57 is a diagram showing an example of the configuration of a 3D data unit (3DDataUnit). The payload (3DDU_data) of the 3D data unit stores various data units that make up the Gaussian codec. The header (3DDU_header) of the 3D data unit includes a 3D data unit type (3DDataUnitType) that indicates the type of data stored in the payload. Examples of the 3D data unit type (3DDataUnitType) include (1) a V3C unit, (2) a G-PCC TLV unit, (3) a Draco unit, (4) a 3D parameter set (3DPS) that is a parameter set common to multiple codecs, (5) a Gaussian sequence parameter set (GSSPS) that is a parameter set including metadata common to a sequence, (6) a Gaussian frame parameter set (GSFPS) that is a parameter set including metadata common to a frame (a 3D model at a certain time or time interval), and (7) an SEI (Gaussian GSSEI).

[0295] Furthermore, when the 3D data unit type (3DDataUnitType) is a V3C unit, the payload of the V3C unit includes an ACL_NAL unit or a VCL_NAL unit. The VCL NAL unit belongs to the video codec layer and is defined in a two-dimensional video compression standard such as HEVC or VVC. The ACL_NAL unit belongs to the atlas codec layer. The ACL_NAL unit stores data for restoring a two-dimensional image to three-dimensional data.

[0296] The 3D data unit may store a Draco unit based on a standard such as Draco.

[0297] Fig. 58 is a diagram showing a configuration example of a 3DPS (3DParameterset), which is a common parameter set in Gaussian coding. The 3DPS includes information (profile_level) indicating a profile and level that indicates a combination of an encoding standard, encoding method, and toolset used in encoding Gaussian data, and Gaussian component information (3D_codec_type). This information includes information (profile) indicating a profile, information (level) indicating a level, and information (3D_codec_type) indicating the encoding standard and encoding method.

[0298] 3D_codec_type indicates the encoding standard used for encoding Gaussian data. For example, 3D_codec_type=0 indicates that the V3C encoding method was used for encoding, and 3D_codec_type=1 indicates that the G-PCC encoding method was used for encoding.

[0299] A decoding device can determine a decoding method by analyzing 3D_codec_type. The decoding device may switch the decoding method based on 3D_codec_type, or may decide not to perform decoding if it does not support the decoding function of the encoding method indicated by 3D_codec_type. Note that the profile and level may include information indicating the encoding standard.

[0300] Furthermore, restrictions may be imposed on the 3D data unit type (3DDataUnitType) depending on the encoding method. For example, if the encoding method is G-PCC, the use of V3C units may be prohibited, and if the encoding method is V3C, the use of G-PCC TLV units may be prohibited.

[0301] The Gaussian data component information includes information indicating the configuration of a Gaussian component. Fig. 59 is a diagram showing an example of the configuration of the Gaussian data component information. The Gaussian data component information includes the number of Gaussian data components indicating the number of Gaussian components.

[0302] The Gaussian data component information includes, for each Gaussian component, a Gaussian component ID, a Gaussian data type, and the number of Gaussian data dimensions. The Gaussian component ID is identification information for uniquely identifying a Gaussian component. The Gaussian data type indicates the type of Gaussian component (rotation, scale, SH coefficient, transmittance, etc.). The number of Gaussian data dimensions indicates the number of dimensions of the Gaussian component (the number of subcomponents).

[0303] The Gaussian data component information also includes mapping information that depends on the codec. For example, if the codec type (3D_codec_type) is G-PCC, the Gaussian data component information includes G-PCC mapping information (gpcc_mapping_info) that indicates the correspondence between the Gaussian data components and the G-PCC codec components.

[0304] 60 is a diagram showing an example of the configuration of G-PCC mapping information. The G-PCC mapping information includes, as information for each dimension of a Gaussian component, a G-PCC component ID, a G-PCC dimension ID, and conversion information.

[0305] The G-PCC subcomponent ID is an identifier that indicates the G-PCC component number. The G-PCC dimension ID is an identifier that indicates the subcomponent number of the G-PCC component. The conversion information indicates the parameters (conversion parameters) used when converting the data of the dimension.

[0306] Furthermore, when the codec type (3D_codec_type) is V3C, the Gaussian data component information includes V3C mapping information that indicates the correspondence between the components of the Gaussian data and the components of the V3C codec.

[0307] 61 is a diagram showing an example of the configuration of V3C mapping information. The V3C mapping information includes, as information for each dimension of a Gaussian component, a V3C component ID, a V3C dimension ID, and conversion information.

[0308] The V3C subcomponent ID is an identifier indicating the V3C component number. The V3C dimension ID is an identifier indicating the subcomponent number of the V3C component. The conversion information indicates the parameters (conversion parameters) used when converting the data of the dimension.

[0309] The Gaussian data component information may include geometry component information in addition to attribute component information.

[0310] Moreover, although an example of mapping information indicating G-PCC components corresponding to Gaussian components has been shown here, mapping information indicating Gaussian components corresponding to G-PCC components may also be used. Moreover, although an example of mapping information indicating the correspondence between subcomponents of Gaussian data and subcomponents of G-PCC has been shown here, mapping information indicating the correspondence between Gaussian components and G-PCC components may also be used. In other words, the correspondence between subcomponents does not have to be shown.

[0311] Furthermore, some of the syntax shown here may be omitted. For example, if the order of description of the information on rotation, scale, and SH coefficients and the number of dimensions are predetermined, the Gaussian data component ID, Gaussian data type, and number of Gaussian data dimensions may be omitted.

[0312] Another example of a data unit will be described below. Instead of storing Gaussian data in either the V3C unit or the G-PCC TLV unit, a data unit specific to Gaussian data (Gaussian data unit) may be defined.

[0313] 62 is a diagram showing an example of the configuration of a Gaussian data unit. For example, the header of the Gaussian data unit includes a Gaussian data unit type (Gaussian DataUnitType) corresponding to the above-mentioned 3D data unit type (3DDataUnitType). The Gaussian data unit type indicates the type of data stored in the payload of the Gaussian data unit. For example, the Gaussian data unit type includes a geometry data unit and an attribute data unit. The payloads of the geometry data unit and the attribute data unit store encoded geometry data and attribute data, respectively.

[0314] The header of the attribute data unit includes an attribute data unit type. The attribute data unit type indicates the type of data stored in the payload of the attribute data unit. The attribute data unit type indicates an element of attribute information, and includes, for example, a rotation data unit, a scale data unit, a color coefficient data unit, an alpha data unit, etc. The payloads of the rotation data unit, scale data unit, color coefficient data unit, and alpha data unit store encoded rotation, scale, SH coefficient, and transparency, respectively.

[0315] Although the above description has been given of an example of encoding 3D Gaussian data, a similar technique may be applied to data other than 3D Gaussian data. For example, a similar technique may be applied to a 3D model or 3D data using AI.

[0316] Furthermore, in the above description, examples have been described in which the encoding methods are mainly V3C and G-PCC, but encoding methods other than V3C and G-PCC may also be used. Furthermore, the encoding method may be other than a 3D codec. For example, an image codec, a neural network codec, an audio codec, or the like may also be used as the encoding method.

[0317] 63 is a diagram showing a protocol stack of a standard for encoding a 3D Gaussian. As a method for encoding a 3D Gaussian, a protocol stack or a data structure for encoding using at least one of a video base codec and a geometry base codec may be used.

[0318] A video-based codec is image encoding that uses a video codec, such as standards such as AVC, HEVC, or VVC standardized by MPEG, or standards such as VPCC (Video-based Point Cloud Compression), VCM (Video Coding for Machines), or VDMC (Video-based Dynamic Mesh Compression).

[0319] Geometry-based codecs include point cloud compression standards such as G-PCC or Draco, or mesh data compression standards, etc. Note that 3D Gaussian coding methods may be extended to include neural network-based codecs such as Neural Network Coding (NNC).

[0320] Note that instead of the 3D Gaussian, a standard for encoding a 3D model may be defined. Fig. 64 is a diagram showing a protocol stack of a standard for encoding a 3D model. A protocol stack and a data structure for encoding a 3D model using at least one of V3C and G-PCC may be used.

[0321] The 3D model may include (1) a point cloud, (2) a mesh, (3) a 3D model created using NeRF (Neural Radiance Fields), and (4) a 3D model generated using Gaussian Splatting.

[0322] Furthermore, the encoding device may encode some of the components of the 3D model data using video encoding and other components using geometry-based encoding. For example, the encoding device may encode the three-dimensional coordinates, rotation, and scale, which are information representing shape information included in the Gaussian data, using G-PCC, and encode the SH coefficients using a video-based encoding method. Here, geometry-based encoding may be suitable for sparse geometric structure shapes, while video-based encoding may be suitable for multidimensional SH coefficients. Therefore, in such cases, encoding efficiency can be improved.

[0323]

[0111] Figure 65 is a diagram showing an example of the configuration of Gaussian component information in this case. The Gaussian component information shown in Figure 65 includes, in addition to the Gaussian component information shown in Figure 59, a 3D codec type (3D_codec_type) for each Gaussian component. This allows the 3D codec type to be set for each component, thereby realizing the above function. Note that, when a standard for encoding a 3D model as shown in Figure 64 is used, for example, instead of the Gaussian component information, similar information is stored in 3D model component information indicating information about the components of the 3D model.

[0324] (Third Aspect) Hereinafter, a method and data structure for extending the V3C standard to enable the application of general-purpose 3D codecs such as Gaussian data will be described. Also, a method and data structure for extending the V3C standard to enable the application of other 3D codecs such as G-PCC will be described.

[0325] The following data structure allows Gaussian data to be encoded and decoded using various V3C-based standards. It also provides a general-purpose method for encoding Gaussian data using various 3D encoding methods or 3D codec standards. This allows Gaussian data to be encoded using various 3D codec methods, including standards created by various standards organizations, international standards, national standards, and standards established by any organization.

[0326] Figure 66 shows a protocol stack of an encoding standard for encoding Gaussian data. The V3C encoding method (V3C codec) is a structure that allows 3D data to be encoded using one or more video codec standards, such as HEVC or VVC.

[0327] Here, the V3C encoding method is extended to define a Gaussian codec layer. Specifically, atlas data is extended, and metadata necessary for reconstructing Gaussian data and encoding or decoding is stored in the atlas data. Here, the atlas data includes data for converting a two-dimensional image into three-dimensional data in the V3C encoding method. In addition, the V3C encoding method is extended so that geometry-based 3D encoding standards (hereinafter also referred to as GPC) such as G-PCC and Draco can be applied. In the V3C encoding method, the atlas data includes data for converting a decoded two-dimensional image or encoded three-dimensional data decoded using a 3D encoding standard into Gaussian data.

[0328] 67 is a block diagram showing an example configuration of a coding device 500 (three-dimensional data coding device) according to this embodiment. The coding device 500 generates a bit stream (coded data) by coding second Gaussian data. This coding device 500 includes a second pre-processing unit 501, a geometry-based coding unit 502, a video-based coding unit 503, a metadata coding unit 504, and a multiplexing unit 505.

[0329] The second pre-processing unit 501 converts and maps the second Gaussian data into data that can be encoded by the 3D codec used, depending on the 3D codec used.

[0330] For example, the second pre-processing unit 501 converts and maps the second Gaussian data into data that can be coded using a geometry-based coding method (such as G-PCC or Draco). The geometry-based coding unit 502 generates a GCL NAL unit by coding the converted data using the geometry-based coding method. Note that when G-PCC is used, the geometry-based coding unit 502 may output a TLV unit.

[0331] Furthermore, the second pre-processing unit 501 converts and maps the second Gaussian data into data that can be coded using a video-based coding method (such as AVC, HEVC, or VVC). The video-based coding unit 503 codes the converted data using the video-based coding method to generate VCL NAL units.

[0332] The metadata encoder 504 stores the transformation data and mapping information in metadata. Here, the metadata is metadata that is an extension of the V3C standard. For example, an extension area of ​​the V3C parameter set in the V3C standard may be used as this metadata. Alternatively, an extension of the Atlas sequence parameter set (ASPS) or the Atlas frame parameter set (AFPS) may be used as this metadata. The metadata encoder 504 encodes the metadata to generate a V3C parameter set or an Atlas data unit (ACL NAL unit).

[0333] The multiplexing unit 505 generates a bitstream by unitizing and multiplexing the GCL NAL unit or the VCL NAL unit and the V3C parameter set or the ACL NAL unit. Specifically, the multiplexing unit 505 generates a bitstream by storing the GCL NAL unit or the VCL NAL unit and the V3C parameter set or the ACL NAL unit in a 3V3C unit.

[0334] Fig. 68 is a block diagram showing an example configuration of a decoding device 510 (three-dimensional data decoding device) according to this embodiment. The decoding device 510 generates second Gaussian data by decoding a bitstream. This bitstream is generated by, for example, the encoding device 500 shown in Fig. 67. The decoding device 510 includes a demultiplexing unit 511, a geometry-based decoding unit 512, a video-based decoding unit 513, a metadata decoding unit 514, and a second post-processing unit 515.

[0335] The demultiplexer 511 demultiplexes and demultiplexes the V3C units included in the bitstream to generate GCL NAL units or VCL NAL units, and V3C parameter sets or ACL NAL units.

[0336] The geometry-based decoding unit 512 generates decoded data by decoding the GCL NAL units using a geometry-based decoding method (G-PCC, Draco, etc.). The video-based decoding unit 513 generates decoded data by decoding the VCL NAL units using a video-based decoding method (AVC, HEVC, VVC, etc.).

[0337] The metadata decoder 514 generates the transformation information and the mapping information by decoding the V3C parameter set or the ACL NAL unit. For example, the metadata decoder 514 obtains the transformation information and the mapping information from the V3C parameter set or the ACL NAL unit.

[0338] The second post-processing unit 515 inversely transforms the decoded data into second Gaussian data according to the 3D codec used for decoding.

[0339] 69 is a flowchart of the encoding process performed by the encoding device 500. First, the encoding device 500 determines the codec type (encoding method) to be used for encoding (S501). For example, the codec type to be used for encoding may be specified by the user, or may be determined based on the contents of the Gaussian data.

[0340] If the codec type used for encoding is a geometry-based encoding method (G-PCC or Draco) (geometry-based in S501), the encoding device 500 encodes the Gaussian data using the geometry-based encoding method (G-PCC or Draco) and outputs a GCL NAL unit (S502).

[0341] On the other hand, if the codec type used for encoding is a video-based encoding method (video-based in S501), the encoding device 500 encodes the Gaussian data using a video-based encoding method (video codec) and outputs a VCL NAL unit (S503).

[0342] Next, the encoding device 500 stores information indicating the codec type used for encoding in the metadata (S504).

[0343] Next, the encoding device 500 stores the conversion information and mapping information in the metadata (S505). Next, the encoding device 500 multiplexes the metadata and the encoding unit (GCL NAL unit or VCL NAL unit) into a V3C unit (S506).

[0344] 70 is a flowchart of the decoding process by the decoding device 510. First, the decoding device 510 acquires a plurality of V3C units, and acquires data units (GCL NAL units or VCL NAL units) and metadata from the plurality of V3C units (S511). Next, the decoding device 510 analyzes the metadata and determines the codec type used for encoding (the codec type to be used for decoding) based on information indicating the codec type included in the metadata (S512).

[0345] If the codec type used for encoding is a geometry-based encoding method (geometry-based in S513), the decoding device 510 acquires the GCL NAL unit and generates components by decoding the GCL NAL unit using a geometry-based decoding method (G-PCC or Draco) (S514). On the other hand, if the codec type used for encoding is a video-based encoding method (video-based in S513), the decoding device 510 acquires the VCL NAL unit and generates components by decoding the VCL NAL unit using a video-based decoding method (video codec) (S515).

[0346] Next, the decoding device 510 reconstructs Gaussian data using the decoded components and the mapping information (S516). Next, the decoding device 510 obtains transformation information and generates second Gaussian data by inversely transforming the Gaussian data using the transformation information (S517).

[0347] Fig. 71 is a diagram showing an example of the configuration of a V3C unit that has been extended to enable the application of a geometry-based 3D codec to the V3C standard. Fig. 72 is a diagram showing an example of the configuration of a V3C unit. The V3C unit includes a header (V3C unit header) and a payload (V3C data unit).

[0348] The payload of the V3C unit stores ACL NAL units belonging to the atlas codec layer or the video codec layer, and the payload of the extended V3C unit stores GCL NAL units that store data of a geometry-based 3D codec.

[0349] The header of a V3C unit contains a V3C unit type that indicates the type of data contained in the payload, for example, V3C unit type=10 indicates a geometry-based PCC unit (GCL NAL unit).

[0350] 73 is a diagram showing an example of the structure of a GCL NAL unit (GCL NAL unit). The GCL NAL unit includes a header (GCL unit header) and a payload (GCL data unit). The payload of the GCL NAL stores a G-PCC TLV unit or a Draco unit.

[0351] The header of the GCL NAL unit stores a GCL unit type indicating the type of geometry-based 3D codec, which is the type of data included in the payload. For example, GCL unit type = 0 indicates G-PCC point cloud compression (G-PCC TLV unit). GCL unit type = 1 indicates Draco point cloud compression (Draco unit).

[0352] Note that, although an example has been described here in which a G-PCC TLV unit or the like is stored in a GCL NAL unit, a G-PCC TLV unit or a unit defined by each 3D codec method may be directly stored in a V3C unit.

[0353] 74 is a diagram showing an example of the configuration of an extended V3C parameter set (V3CParameterset). The V3C parameter set includes information indicating a profile and a level (profile_level) that indicates a combination of an encoding standard, an encoding method, and a toolset used in encoding Gaussian data. The information includes information indicating a profile (profile), information indicating a level (level), and information indicating a codec used for encoding (is_video_codec and is_3d_codec).

[0354] The profile indicates a codec that uses an extension of V3C, such as VPCC or DMC. For example, the profile indicates an extension of the 3D Gaussian codec of V3C. For example, profile=0 indicates VPCC, profile=1 indicates DMC, and profile=2 indicates the 3D Gaussian codec.

[0355] is_video_codec is a flag indicating whether a video codec is used. If a video codec is used (is_video_codec=1), the V3C parameter set includes a video codec type (video_codec_type) indicating the video codec to be used.

[0356] is_3d_codec is a flag indicating whether a 3D codec is used. If a 3D codec is used (is_3d_codec=1), the V3C parameter set includes a 3D codec type (3d_codec_type) indicating the 3D codec to be used. For example, 3D codec type=0 indicates G-PCC, and 3D codec type=1 indicates Draco. Note that a predefined "4CC" code may be used for the 3D codec type.

[0357] Note that, when a 3D Gaussian codec is used and the values ​​of is_video_codec and is_3d_codec are exclusive (one is 1 and the other is 0), a constraint may be specified that the exclusiveness of the values ​​of is_video_codec and is_3d_codec is a compatibility condition for the bitstream. In other words, when the values ​​of is_video_codec and is_3d_codec are not exclusive, the decoding device may determine that the bitstream is inappropriate (contains an error).

[0358] When a 3D Gaussian codec is used (profile=3D gaussian codec), the V3C parameter set includes Gaussian component information (3d_codec_type) as extended information of the 3D Gaussian codec of V3C.

[0359] 75 is a diagram showing an example of the configuration of Gaussian data component information, which includes the number of Gaussian data components indicating the number of Gaussian components.

[0360] The Gaussian data component information includes, for each Gaussian component, a Gaussian component ID, a Gaussian data type, and the number of Gaussian data dimensions. The Gaussian component ID is identification information for uniquely identifying a Gaussian component. The Gaussian data type indicates the type of Gaussian component (rotation, scale, SH coefficient, transmittance, etc.). The number of Gaussian data dimensions indicates the number of dimensions of the Gaussian component (the number of subcomponents).

[0361] The Gaussian data component information also includes mapping information depending on the codec. For example, when a video codec is used (is_video_codec=1), the Gaussian data component information includes video mapping information (video_mapping_info) indicating a correspondence between Gaussian data components and video codec components. When a 3D codec is used (is_3d_codec=1), the Gaussian data component information includes 3D mapping information (3D_mapping_info) indicating a correspondence between Gaussian data components and 3D codec components.

[0362] 76 is a diagram showing an example of the configuration of video mapping information. The video mapping information includes, as information for each dimension of a Gaussian component, a video component ID, a video dimension ID, and transformation information.

[0363] The video subcomponent ID is an identifier indicating the video component number. The video dimension ID is an identifier indicating the subcomponent number of the video component. The conversion information indicates the parameters (conversion parameters) used when converting the data of the dimension.

[0364] 77 is a diagram showing an example of the configuration of 3D mapping information. When G-PCC is used, the 3D mapping information includes a G-PCC component ID, a G-PCC dimension ID, and conversion information as information for each dimension of the Gaussian component.

[0365] The G-PCC subcomponent ID is an identifier that indicates the G-PCC component number. The G-PCC dimension ID is an identifier that indicates the subcomponent number of the G-PCC component. The conversion information indicates the parameters (conversion parameters) used when converting the data of the dimension.

[0366] Furthermore, when Draco is used, the 3D mapping information includes information indicating the correspondence between the components of the Gaussian data and the components of the Draco codec.

[0367] The Gaussian data component information may include geometry component information in addition to attribute component information.

[0368] Furthermore, although an example of mapping information indicating G-PCC components corresponding to Gaussian components has been shown here, mapping information indicating Gaussian components corresponding to G-PCC components may also be used. Although an example of mapping information indicating the correspondence between subcomponents of Gaussian data and subcomponents of G-PCC has been shown here, mapping information indicating the correspondence between Gaussian components and G-PCC components may also be used. In other words, the correspondence between subcomponents does not have to be shown.

[0369] Furthermore, some of the syntax shown here may be omitted. For example, if the order of description of information and the number of dimensions of rotation, scale, and SH coefficients are predetermined, the Gaussian data component ID, Gaussian data type, and number of Gaussian data dimensions may be omitted.

[0370] In the above description, an example has been described in which the data to be encoded is 3D Gaussian data, but the data to be encoded is not limited to 3D Gaussian data. For example, the protocol stack or data structure described in this embodiment may be used for encoding a 3D model (3D model codec) generated by another method such as NeRF. For example, the data to be encoded may be a 3D model (point cloud, mesh, or other 3D model) using AI (Artificial Intelligence).

[0371] In addition, as the V3C encoding method, in addition to the video-based and geometry-based standards, a neural network codec trained by machine learning or the like may also be used.

[0372] Figure 78 is a diagram showing an extended V3C protocol stack. For example, as shown in Figure 78, the V3C standard may be extended to enable transmission of a neural network (NN). For example, a neural network-based coding method such as NNC (Neural Network Coding) is used for the neural network.

[0373] Alternatively, a codec that is an extension of the V3C standard may be defined, and one or more encoding methods may be used among video-based, geometry-based, and neural network-based encoding methods.

[0374] Furthermore, for example, when geometry information (position information) and attribute information are encoded in a point cloud compression method using AI, the geometry information may be encoded using learning. For example, a neural network-based encoding method or a geometry-based encoding method may be used for the geometry information, and a video-based encoding method may be used for the attribute information. This may improve compression performance (encoding efficiency).

[0375] Figure 79 is a diagram showing an extended V3C protocol stack. For example, the V3C standard is extended as shown in Figure 79. For example, the V3C parameter set, ASPS, or AFPS is extended to define a data unit that stores AI-PCC. This allows at least one of video-based, geometry-based, and neural network-based encoding methods to be used as a codec method, and enables encoding and decoding of point cloud compression using AI.

[0376] [Summary] As described above, the encoding device (three-dimensional data encoding device) according to the embodiment performs the processing shown in FIG. 80 . FIG. 80 is a flowchart of the encoding processing by the encoding device. The encoding device encodes three-dimensional data (e.g., Gaussian data, point clouds, meshes, etc.) including one or more first components. The encoding device selects one of a plurality of encoding methods including a first encoding method (e.g., V3C, AVC, HEVC, VVC, etc.) that is a video-based three-dimensional data encoding method, and a second encoding method different from the first encoding method (S601). If the first encoding method is selected (first encoding method in S602), the encoding device generates first encoded data (e.g., VCL NAL units) by encoding the one or more first components using the first encoding method (S603). If the second encoding method is selected (second encoding method in S602), the encoding device generates second encoded data (e.g., GCL NAL units) by encoding the one or more first components using the second encoding method (S604). NAL unit) is generated (S604), first information (is_video_codec, is_3d_codec, video_codec_type, or 3d_codec_type) indicating the selected encoding method is stored in first metadata (for example, V3C parameter set, ASPS, or AFPS) that is an extension of the metadata of the first encoding method (S605), and a bitstream including the first metadata and the first encoded data or the second encoded data is generated (S606).

[0377] This allows the encoding device to selectively select an encoding method to be used for encoding 3D data from multiple encoding methods, which may improve encoding efficiency.Furthermore, by extending the video-based 3D data encoding method, a system that can easily achieve the above processing can be constructed.

[0378] For example, the encoding device further converts one or more first components into one or more second components corresponding to the selected encoding method, and when the first encoding method is selected, generates first encoded data by encoding the one or more second components using the first encoding method, and when the second encoding method is selected, generates second encoded data by encoding the one or more second components using the second encoding method, and the first metadata further includes second information (e.g., mapping information) indicating a correspondence between the one or more first components and the one or more second components.

[0379] According to this, the encoding device can encode the 3D data using the selected encoding method by converting the data into a format corresponding to the selected encoding method, and the decoding device can convert the plurality of second components into the plurality of first components using the second information included in the bitstream.

[0380] For example, the second encoding method is a geometry-based three-dimensional data encoding method (such as G-PCC or Draco). For example, the second encoding method is a neural network-based three-dimensional data encoding method (such as NNC). For example, the second encoding method is a three-dimensional data encoding method using AI.

[0381] For example, if the three-dimensional data is Gaussian data, the encoding device can selectively select an encoding method to use for encoding the Gaussian data from among multiple encoding methods, which may improve encoding efficiency.

[0382] 81 is a block diagram of the encoding device 10. For example, the encoding device 10 includes a processor 11 and a memory 12, and the processor 11 performs the above-described processing using the memory 12.

[0383] Moreover, a decoding device (three-dimensional data decoding device) according to an embodiment performs the processing shown in FIG. 82 . FIG. 82 is a flowchart of the decoding processing by the decoding device. The decoding device decodes three-dimensional data (e.g., Gaussian data, point clouds, meshes, etc.) including one or more first components. The decoding device decodes first metadata (e.g., V3C parameter set, ASPS, or AFPS) that is an extension of metadata of a first encoding method (e.g., V3C, AVC, HEVC, or VVC, etc.) that is a video-based three-dimensional data encoding method, and encoded data (e.g., VCL NAL unit or GCL NAL units) is obtained (S611), and first information (is_video_codec, is_3d_codec, video_codec_type, or 3d_codec_type) indicating an encoding method used to encode one or more first components, among a plurality of encoding methods including a first encoding method and a second encoding method different from the first encoding method, is obtained from the first metadata (S612). If the first information indicates the first encoding method (first encoding method in S613), the encoded data is decoded using a first decoding method corresponding to the first encoding method to generate one or more first components (S614). If the first information indicates a second encoding method (second encoding method in S613), the encoded data is decoded using a second decoding method corresponding to the second encoding method to generate one or more first components (S615).

[0384] This allows the encoding device to selectively select an encoding method to be used for encoding three-dimensional data from among multiple encoding methods. This potentially improves encoding efficiency. Furthermore, the decoding device can appropriately decode encoded data generated by the encoding device. Furthermore, by extending the video-based three-dimensional data encoding method, a system capable of easily implementing the above processing can be constructed.

[0385] For example, the encoded data is generated by encoding one or more second components obtained by converting one or more first components. The decoding device further acquires, from the first metadata, second information (e.g., mapping information) indicating a correspondence between the one or more first components and the one or more second components, and, if the first information indicates a first encoding method, generates the one or more second components by decoding the encoded data using a first decoding method, and, if the first information indicates a second encoding method, generates the one or more second components by decoding the encoded data using a second decoding method, and converts the one or more second components into one or more first components using the correspondence indicated in the second information.

[0386] According to this, the encoding device can encode the 3D data using the selected encoding method by converting the data into a format corresponding to the selected encoding method, and the decoding device can convert the plurality of second components into the plurality of first components using the second information included in the bitstream.

[0387] For example, the second encoding method is a geometry-based three-dimensional data encoding method (such as G-PCC or Draco). For example, the second encoding method is a neural network-based three-dimensional data encoding method (such as NNC). For example, the second encoding method is a three-dimensional data encoding method using AI.

[0388] For example, if the three-dimensional data is Gaussian data, the encoding device can selectively select an encoding method to use for encoding the Gaussian data from among multiple encoding methods, which may improve encoding efficiency.

[0389] 83 is a block diagram of the decoding device 20. For example, the decoding device 20 includes a processor 21 and a memory 22, and the processor 21 uses the memory 22 to perform the above-described processing.

[0390] The above describes the encoding device (three-dimensional data encoding device) and decoding device (three-dimensional data decoding device) according to the embodiments and modifications of the present disclosure, but the present disclosure is not limited to these embodiments.

[0391] Furthermore, each processing unit included in the encoding device, decoding device, etc. according to the above-described embodiments is typically realized as an LSI, which is an integrated circuit. These may be individually implemented as single chips, or some or all of them may be integrated into a single chip.

[0392] Furthermore, the integrated circuit is not limited to an LSI, but may be realized by a dedicated circuit or a general-purpose processor. An FPGA (Field Programmable Gate Array) that can be programmed after the LSI is manufactured, or a reconfigurable processor that can reconfigure the connections and settings of circuit cells within the LSI may also be used.

[0393] In each of the above embodiments, each component may be configured with dedicated hardware, or may be realized by executing a software program suitable for that component. Each component may be realized by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0394] Furthermore, the present disclosure may be realized as an encoding method (three-dimensional data encoding method) or a decoding method (three-dimensional data decoding method) executed by an encoding device (three-dimensional data encoding device) and a decoding device (three-dimensional data decoding device), etc.

[0395] The present disclosure may also be realized as a program that causes a computer, processor, or device to execute the encoding method or decoding method. The present disclosure may also be realized as a bitstream generated by the encoding method. The present disclosure may also be realized as a recording medium on which the program or the bitstream is recorded. For example, the present disclosure may also be realized as a non-transitory computer-readable recording medium on which the program or the bitstream is recorded.

[0396] The division of functional blocks in the block diagram is an example, and multiple functional blocks may be realized as a single functional block, one functional block may be divided into multiple blocks, or some functions may be moved to another functional block.Furthermore, the functions of multiple functional blocks having similar functions may be processed in parallel or in time-sharing by a single piece of hardware or software.

[0397] The order in which the steps in the flowchart are executed is merely an example for specifically explaining the present disclosure, and other orders may be used. Also, some of the steps may be executed simultaneously (in parallel) with other steps.

[0398] While the encoding device and decoding device according to one or more aspects have been described based on the embodiments, the present disclosure is not limited to these embodiments. As long as they do not deviate from the spirit of the present disclosure, various modifications conceivable by those skilled in the art to the present embodiments and configurations constructed by combining components of different embodiments may also be included within the scope of one or more aspects.

[0399] The present disclosure is applicable to encoding devices and decoding devices.

[0400] 10 Encoding device 11, 21 Processor 12, 22 Memory 20 Decoding device 101 Three-dimensional data encoding system 102 Three-dimensional data decoding system 103 Sensor terminal 104 External connection unit 111 Three-dimensional data generation system 112 Presentation unit 113 Encoding unit 114 Multiplexing unit 115 Input / output unit 116 Control unit 117 Sensor information acquisition unit 118 Three-dimensional data generation unit 121 Sensor information acquisition unit 122 Input / output unit 123 Demultiplexing unit 124 Decoding unit 125 Presentation unit 126 User interface 127 Control unit 130 First encoding unit 131 Position information encoding unit 132 Attribute information encoding unit 133 Additional information encoding unit 134 Multiplexing unit 140 First decoding unit 141 Demultiplexing unit 142 Position information decoding unit 143 Attribute information decoding unit 144 Additional information decoding unit 150 Second encoding unit 151 Additional information generation unit 152 Position image generation unit 153 Attribute image generation unit 154 Video encoding unit 155 Additional information encoding unit 156 Multiplexing unit 160 Second decoding unit 161 Demultiplexing unit 162 Video decoding unit 163 Additional information decoding unit 164 Position information generation unit 165 Attribute information generation unit 171 Octree encoding unit 172 Prediction tree encoding unit 173 Octree decoding unit 174 Prediction tree decoding unit 181, 191 Octree generation unit 182, 192 Geometric information calculation unit 183, 193 Encoding table selection unit 184 Entropy encoding unit 194 Entropy decoding unit 201 LoD attribute information encoding unit 202 Transformed attribute information encoding unit 203 LoD attribute information decoding unit 204 Transformed attribute information decoding unit 211 Sorting unit 212 Haar transform unit 213 Quantization unit 214, 222 Inverse quantization unit 215, 223 Inverse Haar transform unit 216, 224 Memory 217 Arithmetic coding unit 221 Arithmetic decoding unit 231 Data division unit 232 Encoding unit 233 Decoding unit 234 Data combination unit 241 Encoding unit 242 TLV storage unit 301 Sensor information input unit 302 Three-dimensional data generation unit 303, 305 Rendering unit 304 Spherical harmonic function 311 Encoding unit312 Multiplexing unit 321 Demultiplexing unit 322 Decoding unit 323 Application unit 324 Input interface unit 325 Rendering unit 326 Presentation unit 330, 400 Encoding device 331 First pre-processing unit 332, 371, 401 Second pre-processing unit 333, 372, 402 G-PCC encoding unit 341 Position information encoding unit 342 Rotation encoding unit 343 Scale encoding unit 344 SH coefficient encoding unit 345 Transmittance encoding unit 346 Metadata encoding unit 347, 405 Multiplexing unit 350, 410 Decoding device 351, 373, 412 G-PCC decoding unit 352, 374, 415 Second post-processing unit 353 First post-processing unit 361, 411 Demultiplexing unit 362 Position information decoding unit 363 Rotation decoding unit 364 Scale decoding unit 365 SH coefficient decoding unit 366 Transmittance decoding unit 367 Metadata decoding unit 375 Transformation unit 376 Mapping unit 403 V3C encoding unit 404 Metadata encoding unit 413 V3C decoding unit 414 Metadata decoding unit 500 Encoding device 501 Second pre-processing unit 502 Geometry-based encoding unit 503 Video-based encoding unit 504 Metadata encoding unit 505 Multiplexing unit 510 Decoding device 511 Demultiplexing unit 512 Geometry-based decoding unit 513 Video-based decoding unit 514 Metadata decoding unit 515 Second post-processing unit

Claims

1. A three-dimensional data encoding method for encoding three-dimensional data including one or more first components, comprising: selecting one of a plurality of encoding methods including a first encoding method that is a video-based three-dimensional data encoding method and a second encoding method different from the first encoding method; if the first encoding method is selected, generating first encoded data by encoding the one or more first components using the first encoding method; if the second encoding method is selected, generating second encoded data by encoding the one or more first components using the second encoding method; storing first information indicating the selected encoding method in first metadata that is an extension of the metadata of the first encoding method; and generating a bitstream that includes the first metadata and the first encoded data or the second encoded data.

2. The three-dimensional data encoding method according to claim 1, further comprising: converting the one or more first components into one or more second components corresponding to the selected encoding method; if the first encoding method is selected, encoding the one or more second components using the first encoding method to generate the first encoded data; if the second encoding method is selected, encoding the one or more second components using the second encoding method to generate the second encoded data; and the first metadata further includes second information indicating a correspondence between the one or more first components and the one or more second components.

3. The three-dimensional data encoding method according to claim 1, wherein the second encoding method is a geometry-based three-dimensional data encoding method.

4. The three-dimensional data encoding method according to claim 1, wherein the second encoding method is a neural network-based three-dimensional data encoding method.

5. The three-dimensional data encoding method according to claim 1, wherein the second encoding method is a three-dimensional data encoding method using AI.

6. The three-dimensional data encoding method according to claim 1, wherein the three-dimensional data is Gaussian data.

7. A three-dimensional data decoding method for decoding three-dimensional data including one or more first components, comprising: obtaining a bitstream including first metadata, which is an extension of the metadata of a first encoding method that is a video-based three-dimensional data encoding method, and encoded data; obtaining, from the first metadata, first information indicating an encoding method used to encode the one or more first components from a plurality of encoding methods including the first encoding method and a second encoding method different from the first encoding method; if the first information indicates the first encoding method, generating the one or more first components by decoding the encoded data using a first decoding method corresponding to the first encoding method; and if the first information indicates the second encoding method, generating the one or more first components by decoding the encoded data using a second decoding method corresponding to the second encoding method.

8. The three-dimensional data decoding method according to claim 7, wherein the encoded data is generated by encoding one or more second components into which the one or more first components are converted, and the three-dimensional data decoding method further comprises: obtaining second information from the first metadata, the second information indicating a correspondence between the one or more first components and the one or more second components; if the first information indicates the first encoding method, generating the one or more second components by decoding the encoded data using the first decoding method; if the first information indicates the second encoding method, generating the one or more second components by decoding the encoded data using the second decoding method; and converting the one or more second components into the one or more first components using the correspondence indicated by the second information.

9. The three-dimensional data decoding method according to claim 7, wherein the second encoding method is a geometry-based three-dimensional data encoding method.

10. The three-dimensional data decoding method according to claim 7, wherein the second encoding method is a neural network-based three-dimensional data encoding method.

11. The three-dimensional data decoding method according to claim 7, wherein the second encoding method is a three-dimensional data encoding method using AI.

12. The three-dimensional data decoding method according to claim 7, wherein the three-dimensional data is Gaussian data.

13. A three-dimensional data encoding device that encodes three-dimensional data including one or more first components, comprising: a processor; and a memory, wherein the processor uses the memory to select one of a plurality of encoding methods including a first encoding method that is a video-based three-dimensional data encoding method and a second encoding method different from the first encoding method; when the first encoding method is selected, generate first encoded data by encoding the one or more first components using the first encoding method; when the second encoding method is selected, generate second encoded data by encoding the one or more first components using the second encoding method; store first information indicating the selected encoding method in first metadata that is an extension of the metadata of the first encoding method; and generate a bitstream that includes the first metadata and the first encoded data or the second encoded data.

14. A three-dimensional data decoding device that decodes three-dimensional data including one or more first components, comprising: a processor; and a memory, wherein the processor uses the memory to: obtain a bitstream including first metadata, which is an extension of the metadata of a first encoding method that is a video-based three-dimensional data encoding method, and encoded data; obtain from the first metadata first information indicating an encoding method used to encode the one or more first components, from a plurality of encoding methods including the first encoding method and a second encoding method; if the first information indicates the first encoding method, generate the one or more first components by decoding the encoded data using a first decoding method corresponding to the first encoding method; and if the first information indicates the second encoding method, generate the one or more first components by decoding the encoded data using a second decoding method corresponding to the second encoding method.

Images