Three-dimensional data encoding method, three-dimensional data decoding method, three-dimensional data encoding device, and three-dimensional data decoding device

Abstract

The three-dimensional data encoding method selects any one of multiple encoding methods, including a first encoding method and a second encoding method. The first encoding method is a video-based three-dimensional data encoding method, and the second encoding method is different from the first encoding method (S601). When the first encoding method is selected (the first encoding method in S602), the first encoding method is used to encode one or more first components to generate first encoded data (S603). When the second encoding method is selected (the second encoding method in S602), the second encoding method is used to encode one or more first components to generate second encoded data (S604). The first information representing the selected encoding method is stored in the first metadata obtained by expanding the metadata of the first encoding method (S605), and a bit stream is generated. The bit stream contains the first metadata and the first encoded data or the second encoded data (S606).

Description

Technical Field

[0001] This disclosure relates to a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, and a three-dimensional data decoding device. Background Technology

[0002] Devices and services utilizing 3D data are expected to become increasingly common in a wide range of fields, including computer vision, mapping, surveillance, infrastructure inspection, and image distribution, which enable autonomous movement of vehicles or robots. 3D data is acquired through various methods, such as distance sensors like rangefinders, stereo cameras, or combinations of multiple monocular cameras.

[0003] As one method of representing 3D data, there is the so-called point cloud method, which uses a group of points in 3D space to represent the shape of a 3D structure. In a point cloud, the position and color of the point group are stored. It is envisioned that point clouds will become the mainstream method of representing 3D data, but the data volume of point groups is extremely large. Therefore, in the accumulation or transmission of 3D data, similar to 2D moving images (for example, MPEG-4 AVC (Advanced Video Coding) or HEVC (High Efficiency Video Coding) standardized by MPEG), data compression based on coding is necessary.

[0004] In addition, point cloud compression is partly supported by publicly available libraries (such as Point CloudLibrary) that perform point cloud association processing.

[0005] In addition, there are known technologies that use three-dimensional map data to retrieve and display facilities located around a vehicle (for example, see Patent Document 1).

[0006] Existing technical documents Patent documents Patent Document 1: International Publication No. 2014 / 020663 Summary of the Invention

[0007] The problem that the invention aims to solve In the encoding and decoding of such three-dimensional data, it is hoped that the encoding efficiency can be improved.

[0008] The purpose of this disclosure is to provide a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, or a three-dimensional data decoding device that can improve encoding efficiency.

[0009] Methods for solving problems One aspect of this disclosure is a 3D data encoding method for encoding 3D data containing one or more first components. The method selects any one of multiple encoding methods, including a first encoding method and a second encoding method. The first encoding method is a video-based 3D data encoding method. The second encoding method differs from the first encoding method. When the first encoding method is selected, it is used to encode the one or more first components to generate first encoded data. When the second encoding method is selected, it is used to encode the one or more first components to generate second encoded data. First metadata, obtained by extending the metadata of the first encoding method, stores first information representing the selected encoding method. A bitstream is generated, containing the first metadata and either the first encoded data or the second encoded data.

[0010] One aspect of the three-dimensional data decoding method disclosed herein is a three-dimensional data decoding method for decoding three-dimensional data containing one or more first components. It obtains a bitstream containing first metadata and encoded data. The first metadata is obtained by extending metadata of a first encoding method, which is a video-based three-dimensional data encoding method. First information is obtained from the first metadata. This first information represents the encoding method used in encoding the one or more first components, among multiple encoding methods including the first encoding method and a second encoding method different from the first encoding method. When the first information represents the first encoding method, the encoded data is decoded using a first decoding method corresponding to the first encoding method, thereby generating the one or more first components. When the first information represents the second encoding method, the encoded data is decoded using a second decoding method corresponding to the second encoding method, thereby generating the one or more first components.

[0011] Invention Effects This disclosure provides a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, or a three-dimensional data decoding device that can improve encoding efficiency. Attached Figure Description

[0012] Figure 1 This is a diagram illustrating an example of the configuration of a three-dimensional data encoding and decoding system according to an implementation method.

[0013] Figure 2 This is a diagram illustrating an example of the configuration of point group data in an implementation method.

[0014] Figure 3This is a diagram illustrating an example of the structure of a data file containing group data information, as described in the implementation method.

[0015] Figure 4 This is a diagram illustrating an example of the structure of grid data in an implementation method.

[0016] Figure 5 This is a diagram illustrating an example of the structure of a data file containing information about grid data in an implementation method.

[0017] Figure 6 It is a diagram representing the types of three-dimensional data for the implementation method.

[0018] Figure 7 This is a diagram showing the configuration of the first coding section in the implementation method.

[0019] Figure 8 This is a block diagram of the first coding section of the implementation method.

[0020] Figure 9 This is a diagram showing the configuration of the first decoding unit in the embodiment.

[0021] Figure 10 This is a block diagram of the first decoding unit in the implementation method.

[0022] Figure 11 This is a diagram showing the configuration of the second coding section in the implementation method.

[0023] Figure 12 This is a block diagram of the second coding section of the implementation method.

[0024] Figure 13 This is a diagram showing the configuration of the second decoding unit in the embodiment.

[0025] Figure 14 This is a block diagram of the second decoding unit in the implementation method.

[0026] Figure 15 This is a block diagram of the location information encoding unit in the implementation method.

[0027] Figure 16 This is a block diagram of the location information decoding unit in the implementation method.

[0028] Figure 17 This is a block diagram of the octree coding section of the implementation method.

[0029] Figure 18 This is a diagram illustrating an example of location information in an implementation method.

[0030] Figure 19 This is a diagram illustrating an example of an octree representation of the location information in an implementation method.

[0031] Figure 20This is a block diagram of the octree decoding unit in the implementation method.

[0032] Figure 21 This is a block diagram of the attribute information encoding unit in the implementation method.

[0033] Figure 22 This is a block diagram of the attribute information decoding unit in the implementation method.

[0034] Figure 23 This is a block diagram of the attribute information encoding unit in the implementation method.

[0035] Figure 24 This is a block diagram of the attribute information decoding unit in the implementation method.

[0036] Figure 25 This is a block diagram of a three-dimensional data encoding device according to an implementation method.

[0037] Figure 26 This is a block diagram of a three-dimensional data decoding device according to an implementation method.

[0038] Figure 27 This is a diagram showing the relationship between the tiles and slices in the implementation method.

[0039] Figure 28 This is a diagram illustrating an example of the configuration of a bitstream in an implementation method.

[0040] Figure 29 This is a block diagram illustrating an example configuration of a three-dimensional data encoding device according to an embodiment.

[0041] Figure 30 This is a diagram illustrating the configuration example of the encoded data and NAL unit in the implementation method.

[0042] Figure 31 This is a diagram representing a semantic example of pcc_nal_unit_type in an implementation.

[0043] Figure 32 This is a diagram illustrating a portion of the processing in a three-dimensional data generation system according to an implementation method.

[0044] Figure 33 This is a diagram representing the constituent elements of Gaussian data in the implementation method.

[0045] Figure 34 This is a diagram representing the rendering process of Gaussian data in the implementation method.

[0046] Figure 35 This is a graph showing an example of the number of SH coefficients of the spherical harmonic function in the implementation method.

[0047] Figure 36 This is a diagram illustrating an example of the input and output of a spherical harmonic function in an implementation.

[0048] Figure 37 This is a diagram illustrating the rendering process of the implementation method.

[0049] Figure 38 This is a block diagram of the encoding device in the implementation method.

[0050] Figure 39 This is a block diagram of the decoding device according to the implementation method.

[0051] Figure 40 This is a block diagram of the encoding device in the implementation method.

[0052] Figure 41 This is a block diagram of the decoding device according to the implementation method.

[0053] Figure 42 This is a diagram illustrating an example of a Gaussian data mapping in an implementation.

[0054] Figure 43 This is a diagram showing an example of mapping information for an implementation method.

[0055] Figure 44 This is a diagram illustrating a configuration example of the transformation information in the implementation method.

[0056] Figure 45 This is a diagram illustrating an example of the structure of encoded data in an implementation method.

[0057] Figure 46 This is a diagram illustrating an example of the G-PCC attribute type of Gaussian data in an implementation.

[0058] Figure 47 This is a flowchart of the decoding device's processing in the implementation method.

[0059] Figure 48 This is a block diagram illustrating a system for encoding and decoding the first Gaussian data according to an implementation method.

[0060] Figure 49 This is a block diagram of the second preprocessing unit in the embodiment.

[0061] Figure 50 This is a diagram illustrating the transformation process of the implementation method.

[0062] Figure 51 This is a diagram illustrating the protocol stack of the encoding standard used to encode Gaussian data in an implementation.

[0063] Figure 52 This is a block diagram of the encoding device in the implementation method.

[0064] Figure 53 This is a block diagram of the decoding device according to the implementation method.

[0065] Figure 54This is a flowchart of the encoding process in the implementation method.

[0066] Figure 55 This is a flowchart of the decoding process in the implementation method.

[0067] Figure 56 This is a diagram representing the basic data structure of the Gaussian codec implementation.

[0068] Figure 57 This is a diagram illustrating an example of the configuration of a 3D data unit in an implementation method.

[0069] Figure 58 This is a diagram illustrating an example of the configuration of a 3DPS implementation.

[0070] Figure 59 This is a diagram illustrating an example of the composition of Gaussian data component information in an implementation method.

[0071] Figure 60 This is a diagram illustrating an example of the configuration of G-PCC mapping information in an implementation method.

[0072] Figure 61 This is a diagram illustrating an example of the configuration of V3C mapping information in an implementation method.

[0073] Figure 62 This is a diagram illustrating an example of the configuration of a Gaussian data unit in an implementation method.

[0074] Figure 63 This is a diagram illustrating the standard protocol stack for encoding 3D Gaussians in an implementation.

[0075] Figure 64 This is a diagram illustrating the standard protocol stack for encoding 3D models in an implementation method.

[0076] Figure 65 This is a diagram illustrating an example of the composition of Gaussian component information in an implementation method.

[0077] Figure 66 This is a diagram illustrating the protocol stack of the encoding standard used to encode Gaussian data in an implementation.

[0078] Figure 67 This is a block diagram of the encoding device in the implementation method.

[0079] Figure 68 This is a block diagram of the decoding device according to the implementation method.

[0080] Figure 69 This is a flowchart of the encoding process in the implementation method.

[0081] Figure 70 This is a flowchart of the decoding process in the implementation method.

[0082] Figure 71 This is a diagram illustrating an example of the configuration of an extended V3C unit in an implementation method.

[0083] Figure 72 This is a diagram illustrating an example of the configuration of the V3C unit in an implementation method.

[0084] Figure 73 This is a diagram illustrating an example of the configuration of a GCL NAL unit in an implementation method.

[0085] Figure 74 This is a diagram illustrating an example of the configuration of the extended V3C parameter set in the implementation method.

[0086] Figure 75 This is a diagram illustrating an example of the composition of Gaussian data component information in an implementation method.

[0087] Figure 76 This is a diagram illustrating an example of the configuration of video mapping information in an implementation method.

[0088] Figure 77 This is a diagram illustrating an example of the configuration of 3D mapping information in an implementation method.

[0089] Figure 78 This is a diagram showing the extended V3C protocol stack of the implementation method.

[0090] Figure 79 This is a diagram showing the extended V3C protocol stack of the implementation method.

[0091] Figure 80 This is a flowchart of the encoding process of the encoding device in the embodiment.

[0092] Figure 81 This is a block diagram of the encoding device in the implementation method.

[0093] Figure 82 This is a flowchart of the decoding process of the decoding device in the embodiment.

[0094] Figure 83 This is a block diagram of the decoding device according to the implementation method. Detailed Implementation

[0095] One aspect of this disclosure is a 3D data encoding method for encoding 3D data containing one or more first components. The method selects any one of multiple encoding methods, including a first encoding method and a second encoding method. The first encoding method is a video-based 3D data encoding method. The second encoding method differs from the first encoding method. When the first encoding method is selected, it is used to encode the one or more first components to generate first encoded data. When the second encoding method is selected, it is used to encode the one or more first components to generate second encoded data. First metadata, obtained by extending the metadata of the first encoding method, stores first information representing the selected encoding method. A bitstream is generated, containing the first metadata and either the first encoded data or the second encoded data.

[0096] Therefore, the 3D data encoding method can selectively choose the encoding scheme for 3D data from multiple encoding schemes. This could potentially improve encoding efficiency. Furthermore, by extending video-based 3D data encoding methods, it is possible to construct a system that can easily implement the above-mentioned processing.

[0097] For example, in the three-dimensional data encoding method, the one or more first components can be further transformed into one or more second components corresponding to the selected encoding method. When the first encoding method is selected, the one or more second components are encoded using the first encoding method to generate the first encoded data. When the second encoding method is selected, the one or more second components are encoded using the second encoding method to generate the second encoded data. The first metadata also includes second information, which represents the correspondence between the one or more first components and the one or more second components.

[0098] Therefore, the three-dimensional data encoding method can encode three-dimensional data using the selected encoding method by changing the data into a form corresponding to the selected encoding method. Furthermore, the decoding device can use the second information contained in the bitstream to transform multiple second components into multiple first components.

[0099] For example, the second encoding method can also be a geometry-based three-dimensional data encoding method. For example, the second encoding method can also be a neural network-based three-dimensional data encoding method. For example, the second encoding method can also be a three-dimensional data encoding method using AI. For example, the three-dimensional data can also be Gaussian data.

[0100] Therefore, the 3D data encoding method can selectively choose the encoding scheme for Gaussian data from multiple encoding schemes. This could potentially improve encoding efficiency.

[0101] One aspect of the three-dimensional data decoding method disclosed herein is a method for decoding three-dimensional data containing one or more first components. The method involves obtaining a bitstream containing first metadata and encoded data. The first metadata is obtained by extending metadata of a first encoding scheme, which is a video-based three-dimensional data encoding scheme. First information is obtained from the first metadata. This first information represents an encoding scheme used in the encoding of the one or more first components, among multiple encoding schemes including the first encoding scheme and a second encoding scheme different from the first encoding scheme. If the first information represents the first encoding scheme, the encoded data is decoded using a first decoding scheme corresponding to the first encoding scheme to generate the one or more first components. If the first information represents the second encoding scheme, the encoded data is decoded using a second decoding scheme corresponding to the second encoding scheme to generate the one or more first components.

[0102] Therefore, the encoding device can selectively select the encoding method for encoding 3D data from multiple encoding methods. This could potentially improve encoding efficiency. Furthermore, the 3D data decoding method can appropriately decode the encoded data generated by the encoding device. Moreover, by extending video-based 3D data encoding methods, a system capable of easily implementing the above-mentioned processes can be constructed.

[0103] For example, the encoded data may be generated by encoding one or more second components after the transformation of one or more first components. In the three-dimensional data decoding method, second information is obtained from the first metadata, which represents the correspondence between the one or more first components and the one or more second components. When the first information represents the first encoding method, the encoded data is decoded using the first decoding method to generate the one or more second components. When the first information represents the second encoding method, the encoded data is decoded using the second decoding method to generate the one or more second components. And the one or more second components are transformed into the one or more first components using the correspondence represented by the second information.

[0104] Therefore, the encoding device can encode 3D data using the selected encoding method by changing the data into a form corresponding to the selected encoding method. Furthermore, the 3D data decoding device can use the second information contained in the bitstream to transform multiple second components into multiple first components.

[0105] For example, the second encoding method can also be a geometry-based three-dimensional data encoding method. For example, the second encoding method can also be a neural network-based three-dimensional data encoding method. For example, the second encoding method can also be a three-dimensional data encoding method using AI.

[0106] For example, the three-dimensional data can also be Gaussian data.

[0107] Therefore, the encoding device can selectively select the encoding method for Gaussian data from multiple encoding methods. This could potentially improve encoding efficiency.

[0108] One aspect of this disclosure is a three-dimensional data encoding apparatus for encoding three-dimensional data containing one or more first components. It includes: a processor; and a memory. The processor uses the memory to select any one of a plurality of encoding methods, including a first encoding method and a second encoding method. The first encoding method is a video-based three-dimensional data encoding method. The second encoding method is different from the first encoding method. When the first encoding method is selected, it is used to encode the one or more first components to generate first encoded data. When the second encoding method is selected, it is used to encode the one or more first components to generate second encoded data. First information representing the selected encoding method is stored in first metadata obtained by extending the metadata of the first encoding method. A bitstream is generated, which includes the first metadata and either the first encoded data or the second encoded data.

[0109] Therefore, the 3D data encoding device can selectively select the encoding method for 3D data from multiple encoding methods. This could potentially improve encoding efficiency. Furthermore, by extending video-based 3D data encoding methods, a system capable of easily implementing the above-mentioned processes can be constructed.

[0110] One aspect of the present disclosure is a 3D data decoding apparatus for decoding 3D data containing one or more first components, comprising: a processor; and a memory, wherein the processor uses the memory to acquire a bitstream containing first metadata and encoded data, the first metadata being obtained by extending metadata of a first encoding method, the first encoding method being a video-based 3D data encoding method; first information is obtained from the first metadata, the first information representing an encoding method used in encoding the one or more first components from among a plurality of encoding methods including the first encoding method and a second encoding method; when the first information represents the first encoding method, the encoded data is decoded using a first decoding method corresponding to the first encoding method, thereby generating the one or more first components; when the first information represents the second encoding method, the encoded data is decoded using a second decoding method corresponding to the second encoding method, thereby generating the one or more first components.

[0111] Therefore, the encoding device can selectively select the encoding method for encoding 3D data from multiple encoding methods. This could potentially improve encoding efficiency. Furthermore, the 3D data decoding device can appropriately decode the encoded data generated by the encoding device. Moreover, by extending video-based 3D data encoding methods, a system capable of easily implementing the above-mentioned processes can be constructed.

[0112] Furthermore, these general or specific methods can be implemented either through systems, methods, integrated circuits, computer programs, or recording media such as computer-readable CD-ROMs, or through any combination of systems, methods, integrated circuits, computer programs, and recording media.

[0113] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Furthermore, the embodiments described below represent specific examples of this disclosure. The numerical values, shapes, materials, constituent elements, arrangement positions of constituent elements, connection methods, steps, and order of steps shown in the following embodiments are examples and are not intended to limit this disclosure. Additionally, constituent elements in the following embodiments not described in the independent claims are described as optional constituent elements.

[0114] (Implementation Method) [3D Data Encoding and Decoding System] First, the structure of the three-dimensional data encoding and decoding system of this embodiment will be explained. Figure 1 This is a diagram illustrating an example of the configuration of the three-dimensional data encoding and decoding system of this embodiment. For example... Figure 1As shown, the three-dimensional data encoding and decoding system includes a three-dimensional data encoding system 101, a three-dimensional data decoding system 102, a sensor terminal 103, and an external connection unit 104.

[0115] The 3D data encoding system 101 generates encoded data or reused data by encoding 3D point group data or 3D mesh data, which are 3D data. Furthermore, the 3D data encoding system 101 can be a 3D data encoding device implemented by a single device, or it can be a system implemented by multiple devices. Additionally, the 3D data encoding device may include a portion of the multiple processing units contained in the 3D data encoding system 101.

[0116] The 3D data encoding system 101 includes a 3D data generation system 111, a prompting unit 112, an encoding unit 113, a multiplexing unit 114, an input / output unit 115, and a control unit 116. The 3D data generation system 111 includes a sensor information acquisition unit 117 and a 3D data generation unit 118.

[0117] The sensor information acquisition unit 117 acquires sensor information (sensor signal) from the sensor terminal 103 and outputs the sensor information to the three-dimensional data generation unit 118. The three-dimensional data generation unit 118 generates three-dimensional data based on the sensor information and outputs the three-dimensional data to the encoding unit 113.

[0118] The prompting unit 112 prompts the user with sensor information or 3D data. For example, the prompting unit 112 displays information or images based on sensor information or 3D data.

[0119] The encoding unit 113 encodes (compresses) the three-dimensional data and outputs the resulting encoded data, control information obtained during the encoding process, and other additional information to the multiplexing unit 114. The additional information may include, for example, sensor information.

[0120] The multiplexing unit 114 generates multiplexed data by multiplexing the encoded data, control information, and additional information input from the encoding unit 113. The format of the multiplexed data is, for example, a file format for storage or a packet format for transmission.

[0121] The input / output unit 115 (e.g., a communication unit or interface) outputs multiplexed data to the outside. Alternatively, the multiplexed data is stored in an internal memory or other storage unit. The control unit 116 (or application execution unit) controls each processing unit. That is, the control unit 116 performs encoding and multiplexing control.

[0122] Alternatively, sensor information can also be input to the encoding unit 113 or the multiplexing unit 114. Furthermore, the input / output unit 115 can directly output 3D data or encoded data to the outside.

[0123] The transmission signal (multiplexed data) output from the three-dimensional data encoding system 101 is input to the three-dimensional data decoding system 102 via the external connection unit 104.

[0124] The 3D data decoding system 102 generates 3D point group data or 3D mesh data, etc., as 3D data by decoding encoded data or multiplexed data. Furthermore, the 3D data decoding system 102 can be a single device or a system implemented by multiple devices. Additionally, the 3D data decoding device may include a portion of the multiple processing units contained in the 3D data decoding system 102.

[0125] The 3D data decoding system 102 includes a sensor information acquisition unit 121, an input / output unit 122, a demultiplexing unit 123, a decoding unit 124, a prompting unit 125, a user interface 126, and a control unit 127.

[0126] The sensor information acquisition unit 121 acquires sensor information (sensor signal) from the sensor terminal 103.

[0127] The input / output unit 122 acquires the transmission signal, decodes the multiplexed data (file format or packet) from the transmission signal, and outputs the multiplexed data to the demultiplexing unit 123.

[0128] The demultiplexing unit 123 obtains encoded data, control information, and additional information from the multiplexed data, and outputs the encoded data, control information, and additional information to the decoding unit 124.

[0129] The decoding unit 124 reconstructs the three-dimensional data by decoding the encoded data.

[0130] The prompting unit 125 prompts the user with 3D data. For example, the prompting unit 125 displays information or images based on the 3D data. The user interface 126 receives instructions based on the user's operation. The control unit 127 (or application execution unit) controls each processing unit. That is, the control unit 127 performs demultiplexing, decoding, and prompting control, etc.

[0131] Furthermore, the input / output unit 122 can also directly acquire 3D data or encoded data from external sources. Additionally, the prompting unit 125 can acquire additional information such as sensor information and provide prompts based on this additional information. Furthermore, the prompting unit 125 can also provide prompts based on user instructions obtained from the user interface 126.

[0132] Sensor terminal 103 generates sensor information, which is the information obtained by the sensor. Sensor terminal 103 is a terminal equipped with a sensor or camera, such as a moving object like a car, a flying object like an airplane, a portable terminal, or a camera.

[0133] Sensor information that can be obtained by sensor terminal 103 includes, for example, (1) the distance between sensor terminal 103 and an object or the reflectivity of an object obtained from a LIDAR, millimeter-wave radar, or infrared sensor; and (2) the distance between a camera and an object or the reflectivity of an object obtained from multiple monocular camera images or stereo camera images. Additionally, sensor information may also include sensor posture, orientation, gyroscope (angular velocity), position (GPS information or altitude), velocity, or acceleration. Furthermore, sensor information may also include temperature, air pressure, humidity, or magnetism.

[0134] The external connection unit 104 is realized through communication or broadcasting with integrated circuits (LSI or IC), external storage units, cloud servers via the Internet, etc.

[0135] Next, the three-dimensional point cluster data (hereinafter also referred to as point cluster data) will be explained. Figure 2 It is a graph representing the composition of point group data. Figure 3 This is a diagram illustrating the structure of a data file that describes information about a group of data.

[0136] Point cluster data contains data from multiple points. Each point's data includes location information (3D coordinates) and attribute information related to that location. A group of these points is called a point cluster. For example, a point cluster represents the 3D shape of an object.

[0137] Position information, such as three-dimensional coordinates, is sometimes referred to as geometric information. Additionally, the data for each point can also contain attribute information across multiple attribute categories. Attribute categories could include, for example, color or reflectivity.

[0138] A location information can correspond to a single attribute information, or it can correspond to attribute information with multiple different attribute categories. Furthermore, a location information can also correspond to multiple attribute information of the same attribute category.

[0139] Figure 3 The example data file shown illustrates a one-to-one correspondence between location information and attribute information, illustrating the location and attribute information of N points constituting the point group data.

[0140] Positional information includes, for example, information along the x, y, and z axes. Attribute information includes, for example, RGB color information. Representative data files include .ply files, etc.

[0141] Next, the three-dimensional mesh data (hereinafter also referred to as mesh data) will be explained. Figure 4 This is a diagram illustrating an example of the structure of grid data. Figure 5This is a diagram illustrating an example of the structure of a data file that records information containing grid data.

[0142] Mesh data is a data format used in Computer Graphics (CG). Mesh data represents the three-dimensional shape of an object through a set of face information. Face information consists of polygons such as triangles or quadrilaterals, and is also known as polygons or polygon meshes.

[0143] The components of grid data are, in addition to a group of three-dimensional points (a set of points with three-dimensional position information and corresponding attribute information), three-dimensional points as vertices, an edge connecting two vertices, and a set of faces enclosed by the edges.

[0144] Vertices (also represented as vertex or position) sometimes possess attributes such as color, reflectivity, or normal vector. Information representing the relationships between vertices that make up an edge or face is also called connectivity. The orientation of the normal vector to a point can represent the front and back faces of a face. Additionally, mesh data can also contain face-specific attribute information.

[0145] As a form of mesh data file, there is, for example, an object file. The data file represents the position information G(1)~G(N) of the N vertices that constitute the mesh, and the attribute information A(1)~A(N) of the vertices. Alternatively, the data file may not contain attribute information. Furthermore, the attribute information may not correspond one-to-one with a vertex. Additionally, Figure 5 Example A2 representing a data file with 1 to M attribute information.

[0146] Face information is represented by a combination of vertex indices. n[1, 3, 4] indicates that the face is a triangle formed by vertices n=1, n=3, and n=4. Additionally, m[2, 4, 6] indicates that the attribute information for m=1, m=4, and m=6 corresponds to the three vertices respectively.

[0147] Alternatively, attribute information can sometimes be recorded in a file different from the data file, with the data file representing its pointer information. For example, attribute information can also be stored in a two-dimensional attribute map file, with the filename of the attribute map and the two-dimensional coordinates in the attribute map recorded in attribute information A2 of the data file. Regardless of the method used, attribute information can be specified for a specific point.

[0148] Next, the types of 3D data (point cluster data or grid data) will be explained. Figure 6 It is a graph representing the types of three-dimensional data. For example... Figure 6 As shown, the 3D data includes static and dynamic objects.

[0149] Static objects are 3D data at any given time (a specific moment). Dynamic objects are 3D data that changes over time. Hereinafter, the 3D point cluster data at a specific moment will be referred to as a PCC (Point Cloud Compression) frame or frame. Additionally, the 3D mesh data at a specific moment will be referred to as a mesh frame or frame.

[0150] The number of points constituting an object is sometimes limited, such as the area, pixels, or points, as in typical image data. Conversely, the number of points constituting an object is sometimes unrestricted, as in map information.

[0151] In addition, there are point clusters or grid data of various densities, including sparse point clusters or grid data, as well as dense point clusters or grid data.

[0152] The details of each processing unit are described below. Sensor information is acquired through various methods such as distance sensors like LIDAR or rangefinders, stereo cameras, or combinations of multiple monocular cameras. The 3D data generation unit 118 generates 3D data based on the sensor information obtained by the sensor information acquisition unit 117. The 3D data generation unit 118 generates position information and adds attribute information specific to that position information to the position information to obtain 3D data.

[0153] The 3D data generation unit 118 can also process the 3D data when generating position information or additional attribute information. For example, the 3D data generation unit 118 can reduce the amount of data by deleting point groups with duplicate positions. Furthermore, the 3D data generation unit 118 can also transform the position information (position shift, rotation, or normalization, etc.). Additionally, the 3D data generation unit 118 can generate mesh data from point group data. Furthermore, the 3D data generation unit 118 can also render attribute information.

[0154] In addition, Figure 1 In this system, the three-dimensional data generation system 111 is included in the three-dimensional data encoding system 101, but it can also be set independently outside the three-dimensional data encoding system 101.

[0155] The encoding unit 113 generates encoded data by encoding the three-dimensional data. The encoding methods include the following: The first is an encoding method using location information, hereinafter referred to as the first encoding method. The second is an encoding method using video encoding / decoding, hereinafter referred to as the second encoding method.

[0156] The decoding unit 124 decodes the point group data by decoding the encoded data. The multiplexing unit 114 generates multiplexed data by multiplexing the encoded data using existing multiplexing methods. The generated multiplexed data is transmitted or stored. In addition to PCC encoded data, the multiplexing unit 114 also multiplexes other media such as images, audio, subtitles, applications, and files, or reference time information. Furthermore, the multiplexing unit 114 can also multiplex sensor information or attribute information associated with 3D data.

[0157] As multiplexing methods or file formats, there are ISOBMFF, ISOBMFF-based transmission methods such as MPEG-DASH, MMT, MPEG-2 TS Systems, and RTP.

[0158] The demultiplexing unit 123 extracts encoded data, other media, and time information from the multiplexed data.

[0159] The input / output unit 115 transmits multiplexed data using a method such as broadcasting or communication that matches the medium to be transmitted or stored. The input / output unit 115 can communicate with other devices via the Internet or with storage units such as cloud servers.

[0160] As a communication protocol, it can use HTTP, FTP, TCP, or UDP, etc. It can also use either a PULL or PUSH communication method.

[0161] Either wired or wireless transmission can be used. For wired transmission, Ethernet (registered trademark), USB, RS-232C, HDMI (registered trademark), or coaxial cable can be used. For wireless transmission, wireless LAN, Wi-Fi (registered trademark), Bluetooth (registered trademark), or millimeter wave can be used.

[0162] In addition, broadcast methods can be used, such as DVB-T2, DVB-S2, DVB-C2, ATSC3.0, or ISDB-S3.

[0163] [First Encoding Method] The following describes the encoding and decoding methods for 3D point groups or 3D meshes. When the apparatus, processing, or syntax in this disclosure involves encoding or decoding point group data, it can also be applied to encoding or decoding vertices in a 3D mesh. Furthermore, the disclosure regarding encoding or decoding vertices in a 3D mesh can also be applied to encoding or decoding point group data. Additionally, the disclosure regarding encoding or decoding attribute information of point group data can also be applied to encoding or decoding face information, or attribute information for faces or vertices, in a 3D mesh. Furthermore, the processing can be commonalized in point group encoding and mesh encoding. Through commonalization, it is possible to reduce the size of the circuit or software.

[0164] Figure 7 This is a diagram showing the configuration of the first coding section 130, which is an example of the coding section 113 performing the first coding method. Figure 8 This is a block diagram of the first encoding unit 130. The first encoding unit 130 generates encoded data (encoded stream) by encoding point group data using a first encoding method. The first encoding unit 130 includes a location information encoding unit 131, an attribute information encoding unit 132, an additional information encoding unit 133, and a multiplexing unit 134.

[0165] The first encoding unit 130 is characterized by being aware of the three-dimensional structure for encoding. Furthermore, the first encoding unit 130 is characterized by the attribute information encoding unit 132 encoding information obtained from the position information encoding unit 131. This first encoding method is also known as G-PCC (Geometry-based PCC).

[0166] The point cluster data is PCC point cluster data such as a PLY file or PCC point cluster data generated based on sensor information, which includes position information, attribute information, and other additional information (MetaData). The position information is input to the position information encoding unit 131, the attribute information is input to the attribute information encoding unit 132, and the additional information is input to the additional information encoding unit 133.

[0167] In addition, when encoding mesh data, the position information of vertex information is input to the position information encoding unit 131, and the face information, or the attribute information for faces and vertices, is input to the attribute information encoding unit 132.

[0168] The location information encoding unit 131 generates compressed geometry as encoded data by encoding the location information. For example, the location information encoding unit 131 uses an N-ary tree structure such as an octree to encode the location information. Specifically, in the octree, the object space is divided into 8 nodes (subspaces), and 8-bit information (occupancy code) indicating whether each node contains a point group is generated. In addition, nodes containing point groups are further divided into 8 nodes, and 8-bit information indicating whether each of the 8 nodes contains a point group is generated. This process is repeated until a predetermined level or a threshold for the number of point groups contained in a node is reached.

[0169] The attribute information encoding unit 132 generates encoded attribute information (Compressed Attribute) as encoded data by encoding the composition information generated by the position information encoding unit 131. For example, the attribute information encoding unit 132 determines the reference point (reference node) to be referenced in the encoding of the object point (object node) of the processed object based on the octree structure generated by the position information encoding unit 131. For example, the attribute information encoding unit 132 refers to a node in the octree whose parent node is the same as the object node among the surrounding nodes or adjacent nodes. However, the method for determining the reference relationship is not limited to this.

[0170] Alternatively, the encoding process for attribute information may include at least one of quantization, prediction, and arithmetic encoding. In this case, a reference refers to the use of a reference node in the calculation of the predicted value of the attribute information, or the use of the state of the reference node (e.g., indicating whether the reference node contains occupancy information of a point group) in the determination of the encoding parameters. For example, the encoding parameters may be quantization parameters in quantization or context in arithmetic encoding.

[0171] The additional information encoding unit 133 generates encoded additional information (Compressed MetaData) as encoded data by encoding the compressible data in the additional information.

[0172] The multiplexing unit 134 generates a compressed stream as encoded data by multiplexing encoded position information, encoded attribute information, encoded additional information, and other additional information. The generated compressed stream is output to the processing unit of the system layer (not shown).

[0173] Next, the first decoding unit 140 will be described as an example of the decoding unit 124 that performs the first encoding method. Figure 9 This is a diagram showing the configuration of the first decoding unit 140. Figure 10This is a block diagram of the first decoding unit 140. The first decoding unit 140 generates point group data by decoding the encoded data (encoded stream) obtained by encoding using the first encoding method. The first decoding unit 140 includes a demultiplexing unit 141, a position information decoding unit 142, an attribute information decoding unit 143, and an additional information decoding unit 144.

[0174] The compressed stream of encoded data is input to the first decoding unit 140 from the processing unit of the system layer not shown in the figure.

[0175] The demultiplexing unit 141 separates the encoded location information (Compressed Geometry), encoded attribute information (Compressed Attribute), encoded additional information (Compressed MetaData), and other additional information from the encoded data.

[0176] The location information decoding unit 142 generates location information by decoding the encoded location information. For example, the location information decoding unit 142 reconstructs the location information of a group of points represented by three-dimensional coordinates based on the encoded location information represented by an N-ary tree structure such as an octree.

[0177] The attribute information decoding unit 143 decodes the encoded attribute information based on the composition information generated by the position information decoding unit 142. For example, based on the octree structure obtained by the position information decoding unit 142, the attribute information decoding unit 143 determines the reference point (reference node) to be referenced in the decoding of the object point (object node) of the processed object. For example, the attribute information decoding unit 143 refers to a node in the octree whose parent node is the same as the object node among the surrounding or adjacent nodes. However, the method for determining the reference relationship is not limited to this.

[0178] Alternatively, the decoding process for attribute information may include at least one of inverse quantization, prediction, and arithmetic decoding. In this case, a reference refers to the use of a reference node in the calculation of the predicted value of the attribute information, or the use of the state of the reference node (e.g., indicating whether the reference node contains occupancy information of a point group) in the determination of the decoding parameters. For example, the decoding parameters may be the quantization parameters in inverse quantization or the context in arithmetic decoding.

[0179] The additional information decoding unit 144 generates additional information by decoding the encoded additional information. In addition, the first decoding unit 140 uses the decoding processing of position information and attribute information to generate the additional information required for the application during decoding, and outputs the additional information required for the application to the outside.

[0180] [Second Encoding Method] Next, the second coding section 150 will be described as an example of the coding section 113 that performs the second coding method. Figure 11 This is a diagram showing the structure of the second encoding section 150. Figure 12 This is a block diagram of the second coding section 150.

[0181] The second encoding unit 150 generates encoded data (encoded stream) by encoding the point group data using a second encoding method. The second encoding unit 150 includes an additional information generation unit 151, a location image generation unit 152, an attribute image generation unit 153, an image encoding unit 154, an additional information encoding unit 155, and a multiplexing unit 156.

[0182] The second encoding unit 150 has the following characteristics: it generates a position image and an attribute image by projecting a three-dimensional structure onto a two-dimensional image, and encodes the generated position image and attribute image using an existing image encoding method. The second encoding method is also known as VPCC (Video based PCC).

[0183] Point cluster data is PCC point cluster data such as PLY files or PCC point cluster data generated based on sensor information, which includes position information, attribute information and other additional information (Metadata).

[0184] The additional information generation unit 151 generates mapping information of multiple two-dimensional images by projecting a three-dimensional structure onto a two-dimensional image.

[0185] The position image generation unit 152 generates a position image (Geometry Image) based on position information and mapping information generated by the additional information generation unit 151. This position image may be, for example, a distance image showing distance as pixel values. Alternatively, the distance image may be an image obtained by observing multiple point groups from a single viewpoint (an image obtained by projecting multiple point groups onto a two-dimensional plane), or it may be multiple images obtained by observing multiple point groups from multiple viewpoints, or it may be a single image obtained by integrating these multiple images.

[0186] The attribute image generation unit 153 generates an attribute image based on attribute information and mapping information generated by the additional information generation unit 151. This attribute image may be, for example, an image that displays attribute information (e.g., color (RGB)) as pixel values. Alternatively, the image may be an image obtained by observing multiple point groups from a single viewpoint (an image obtained by projecting multiple point groups onto a two-dimensional plane), or it may be multiple images obtained by observing multiple point groups from multiple viewpoints, or it may be a single image obtained by integrating these multiple images.

[0187] Furthermore, when encoding the mesh data, the position information of the vertices is input to the position image generation unit 152, and the face information, or the attribute information for the faces and vertices, is input to the attribute image generation unit 153.

[0188] The image coding unit 154 encodes the location image and the attribute image using an image coding method, thereby generating a compressed geometry image and a compressed attribute image as coding data. Furthermore, any known coding method can be used as the image coding method. For example, the image coding method could be AVC or HEVC.

[0189] The additional information encoding unit 155 generates compressed metadata by encoding additional information and mapping information contained in the point group data.

[0190] The multiplexing unit 156 generates a compressed stream as encoded data by multiplexing the encoded position image, encoded attribute image, encoded additional information, and other additional information. The generated compressed stream is output to the processing unit of the system layer (not shown).

[0191] Next, we will describe the second decoding unit 160 as an example of the decoding unit 124 that performs the second encoding method. Figure 13 This is a diagram showing the configuration of the second decoding unit 160. Figure 14 This is a block diagram of the second decoding unit 160. The second decoding unit 160 generates dot group data by decoding the encoded data (encoded stream) obtained by encoding using the second encoding method. The second decoding unit 160 includes a demultiplexing unit 161, an image decoding unit 162, an additional information decoding unit 163, a position information generation unit 164, and an attribute information generation unit 165.

[0192] The compressed stream of encoded data is input to the second decoding unit 160 from the processing unit of the system layer not shown in the figure.

[0193] The demultiplexing unit 161 separates the compressed geometry image, compressed attribute image, compressed metadata, and other additional information from the encoded data.

[0194] The image decoding unit 162 decodes the encoded position image and the encoded attribute image using an image encoding method to generate a position image and an attribute image. Furthermore, any known encoding method can be used as the image encoding method. For example, the image encoding method could be AVC or HEVC.

[0195] The additional information decoding unit 163 generates additional information, including mapping information, by decoding the encoded additional information.

[0196] The location information generation unit 164 generates location information using a location image and mapping information. The attribute information generation unit 165 generates attribute information using an attribute image and mapping information.

[0197] The second decoding unit 160 uses the additional information required for decoding during decoding and outputs the additional information required for the application to the outside.

[0198] [Location information encoding in the first encoding method] Figure 15 This is a block diagram illustrating an example of the configuration of the location information encoding unit 131. The location information encoding unit 131 includes an octree encoding unit 171 and a prediction tree encoding unit 172. The octree encoding unit 171 encodes the location information using an octree encoding method (octree coding) to generate encoded location information and metadata. The prediction tree encoding unit 172 encodes the location information using a prediction tree encoding method (prediction tree coding) to generate encoded location information and metadata.

[0199] The location information encoding unit 131 encodes the location information using either or both of octree coding and prediction tree coding. The location information encoding unit 131 can switch between these two methods, use encoding methods other than these two, or include the location information as raw data in the bitstream without encoding. Information indicating the encoding method of the encoded data is stored in metadata and is communicated to the decoding device.

[0200] Figure 16 This is a block diagram illustrating an example configuration of the location information decoding unit 142. The location information decoding unit 142 includes an octree decoding unit 173 and a prediction tree decoding unit 174. The octree decoding unit 173 generates location information by decoding the encoded location information using an octree-based decoding method (octree decoding). The prediction tree decoding unit 174 generates location information by decoding the encoded location information using a prediction tree-based decoding method (prediction tree decoding). Furthermore, the location information decoding unit 142 performs decoding using an encoding method that is notified via metadata.

[0201] Next, an example of the configuration of the location information encoding unit will be explained. Figure 17This is a block diagram of the octree coding unit 171 of this embodiment. The octree coding unit 171 includes an octree generation unit 181, a geometric information calculation unit 182, a coding table selection unit 183, and an entropy coding unit 184.

[0202] The octree generation unit 181 generates an octree based on the input position information, such as the octet code of each node in the octree. The geometry information calculation unit 182 obtains information indicating whether the neighboring nodes of the object node are occupied nodes. For example, the geometry information calculation unit 182 calculates the occupancy information (indicating whether the neighboring node is occupied) of the neighboring nodes based on the occupancy code of the parent node to which the object node belongs. Alternatively, the geometry information calculation unit 182 can store the encoded nodes in a list and search for neighboring nodes from this list. Furthermore, the geometry information calculation unit 182 can also switch neighboring nodes based on the position of the object node within its parent node.

[0203] The encoding table selection unit 183 uses the occupancy information of neighboring nodes calculated by the geometric information calculation unit 182 to select an encoding table for entropy encoding of the object node. For example, the encoding table selection unit 183 may also use the occupancy information of neighboring nodes to generate a bit string and select an encoding table with an index number generated based on that bit string.

[0204] The entropy encoding unit 184 uses the encoding table of the selected index number to entropy encode the occupancy code of the object node, thereby generating encoded position information and metadata. The entropy encoding unit 184 may also append information representing the selected encoding table to the encoded position information.

[0205] The following explains the scanning order of the octree representation and location information. Location information (location data) is encoded after being transformed into an octree structure (octreeification). The octree structure consists of nodes and leaf nodes. Each node has 8 nodes or leaf nodes, and each leaf node contains voxel (VXL) information. Figure 18 This is a diagram representing a configuration example containing positional information of multiple voxels. Figure 19 It means to Figure 18 The diagram shows an example of how location information is transformed into an octree structure. Here, Figure 19 Among the leaf nodes shown, leaf nodes 1, 2, and 3 respectively represent Figure 18 The voxels VXL1, VXL2, and VXL3 shown represent VXL containing point groups (hereinafter, effective VXL).

[0206] Specifically, node 1 corresponds to containing Figure 18The overall space of location information. The overall space corresponding to node 1 is divided into 8 nodes. The nodes among the 8 nodes that contain valid VXL are further divided into 8 nodes or leaf nodes. This process is repeated in a manner equivalent to the hierarchy of the tree structure. Here, each node corresponds to a subspace, and the node information contains information (occupancy code) indicating which position of the next node or leaf node exists after the division. In addition, the lowest-level block is set as a leaf node, and the leaf node information maintains the number of point groups contained within the leaf node, etc.

[0207] Next, an example of the configuration of the location information decoding unit will be explained. Figure 20 This is a block diagram of the octree decoding unit 173 according to this embodiment. The octree decoding unit 173 includes an octree generation unit 191, a geometric information calculation unit 192, an encoding table selection unit 193, and an entropy decoding unit 194.

[0208] The octree generation unit 191 uses header information or metadata of the bitstream to generate an octree for a certain space (nodes). For example, the octree generation unit 191 uses the size of the space along the x, y, and z axes appended to the header information to generate a large space (root node), and generates eight smaller spaces A (nodes A0 to A7) by dividing this space into two along the x, y, and z axes respectively, thereby generating an octree. Furthermore, nodes A0 to A7 are sequentially assigned as target nodes.

[0209] The geometric information calculation unit 192 obtains occupancy information indicating whether the adjacent nodes of the object node are occupied nodes. For example, the geometric information calculation unit 192 calculates the occupancy information of adjacent nodes based on the occupancy code of the parent node to which the object node belongs. Alternatively, the geometric information calculation unit 192 can store the decoded nodes in a list and search for adjacent nodes from this list. Furthermore, the geometric information calculation unit 192 can switch adjacent nodes based on the position of the object node within its parent node.

[0210] The encoding table selection unit 193 uses the occupancy information of neighboring nodes calculated by the geometric information calculation unit 192 to select an encoding table (decoding table) for entropy decoding of the target node. For example, the encoding table selection unit 193 may also use the occupancy information of neighboring nodes to generate a bit string and select an encoding table with an index number generated based on that bit string.

[0211] The entropy decoding unit 194 performs entropy decoding on the occupancy code of the target node using the selected encoding table, thereby generating position information. Alternatively, the entropy decoding unit 194 can also obtain information about the selected encoding table from the bitstream decoding, and use the encoding table represented by this information to perform entropy decoding on the occupancy code of the target node.

[0212] [Attribute information encoding in the first encoding method] The following describes the structure of the attribute information encoding unit and the attribute information decoding unit. Figure 21 This is a block diagram illustrating an example configuration of the attribute information encoding unit 132. The attribute information encoding unit may also include multiple encoding units that perform different encoding methods. For example, the attribute information encoding unit may switch between two modes depending on the usage scenario.

[0213] The attribute information encoding unit 132 includes a LoD attribute information encoding unit 201 and a transform attribute information encoding unit 202. The LoD attribute information encoding unit 201 uses the position information of the 3D points to classify each 3D point into multiple levels, predicts the attribute information of the 3D points belonging to each level, and encodes the prediction residuals. Here, each level after classification is referred to as LoD (Level of Detail).

[0214] The transformation attribute information encoding unit 202 uses RAHT (Region Adaptive Hierarchical Transform) to encode the attribute information. Specifically, the transformation attribute information encoding unit 202 applies RAHT or Haar transform to each attribute information based on the position information of the three-dimensional points, thereby generating high-frequency and low-frequency components at each level, and encodes these values ​​using quantization and entropy coding.

[0215] Figure 22 This is a block diagram illustrating an example of the configuration of the attribute information decoding unit 143. The attribute information decoding unit may also include multiple decoding units that perform different decoding methods. For example, the attribute information decoding unit may switch between the following two decoding methods based on information contained in the header or metadata.

[0216] The attribute information decoding unit 143 includes a LoD attribute information decoding unit 203 and a transform attribute information decoding unit 204. The LoD attribute information decoding unit 203 uses the position information of the three-dimensional points to classify each three-dimensional point into multiple levels, and decodes the attribute values ​​while predicting the attribute information of the three-dimensional points belonging to each level.

[0217] The attribute information decoding unit 204 uses RAHT (Region Adaptive Hierarchical Transform) to decode the attribute information. Specifically, the attribute information decoding unit 204 applies inverse RAHT or inverse Haar transform to the high-frequency and low-frequency components of each attribute value based on the position information of the three-dimensional points, thereby decoding the attribute value.

[0218] Figure 23This is a block diagram of a transformation attribute information encoding unit 202, which is an example of such a unit. The transformation attribute information encoding unit 202 includes a sorting unit 211, a Haar transformation unit 212, a quantization unit 213, an inverse quantization unit 214, an inverse Haar transformation unit 215, a memory 216, and an arithmetic encoding unit 217.

[0219] The sorting unit 211 generates Morton codes using the position information of the three-dimensional points and sorts the multiple three-dimensional points according to the Morton code order. The Haar transform unit 212 generates coding coefficients by applying the Haar transform to the attribute information. The quantization unit 213 quantizes the coding coefficients of the attribute information.

[0220] The inverse quantization unit 214 performs inverse quantization on the quantized coding coefficients. The inverse Haar transform unit 215 applies the inverse Haar transform to the coding coefficients. The memory 216 stores the values ​​of the attribute information of the decoded multiple 3D points. For example, the attribute information of the decoded 3D points stored in the memory 216 can also be used for the prediction of uncoded 3D points.

[0221] The arithmetic coding unit 217 calculates ZeroCnt based on the quantized coding coefficients and performs arithmetic coding on ZeroCnt. Additionally, the arithmetic coding unit 217 performs arithmetic coding on the non-zero quantized coding coefficients. The arithmetic coding unit 217 may also binarize the coding coefficients before arithmetic coding. Furthermore, the arithmetic coding unit 217 may generate and encode various header information.

[0222] Figure 24 This is a block diagram of a transformation attribute information decoding unit 204, which is one example of such a unit. The transformation attribute information decoding unit 204 includes an arithmetic decoding unit 221, an inverse quantization unit 222, an inverse Haar transform unit 223, and a memory 224.

[0223] The arithmetic decoding unit 221 performs arithmetic decoding on the ZeroCnt and coding coefficients contained in the bitstream. Additionally, the arithmetic decoding unit 221 can also decode various header information.

[0224] The inverse quantization unit 222 performs inverse quantization on the encoded coefficients after arithmetic decoding. The inverse Haar transform unit 223 applies the inverse Haar transform to the inverse quantized encoded coefficients. The memory 224 stores the values ​​of the attribute information of the decoded multiple 3D points. For example, the attribute information of the decoded 3D points stored in the memory 224 can also be used for the prediction of 3D points that have not been decoded.

[0225] [Slicing and segmentation] Encoding devices can also segment 3D data into more than one 3D data set for encoding. A group of 3D points segmented into multiple sets is called a slice. A slice is a collection of points with both geometry and attribute information.

[0226] Figure 25 This is a block diagram illustrating an example of the configuration of a three-dimensional data encoding device under this condition. The three-dimensional data encoding device includes a data segmentation unit 231 and an encoding unit 232.

[0227] The data segmentation unit 231 generates multiple segmented 3D data by segmenting the 3D data. Each segmented 3D data corresponds to a slice. The encoding unit 232 generates encoded data by encoding the position information and attribute information of each of the multiple segmented 3D data (slices).

[0228] Figure 26 This is a block diagram illustrating an example configuration of a 3D data decoding apparatus in this case. The 3D data decoding apparatus includes a decoding unit 233 and a data combining unit 234. The decoding unit 233 generates multiple segmented 3D data by decoding encoded data. Each segmented 3D data includes slice position information and attribute information. The data combining unit 234 reconstructs 3D data by combining multiple segmented 3D data.

[0229] Furthermore, during encoding, slices may or may not have dependencies. When there are no dependencies, slices can be encoded or decoded independently. Therefore, processing time can be reduced by processing multiple slices in parallel. Additionally, partial decoding, which decodes only a subset of slices, can further reduce the amount of processing required.

[0230] In cases of dependencies, identifiers representing the dependencies are stored in the bitstream and encoded or decoded sequentially starting from the data being depended upon (referencing the data at the destination).

[0231] The number of segments or the segmentation method can be arbitrary. The 3D data encoding device can also determine the shape of an object and segment it into point groups for each object, or it can segment based on the number of points contained in a tile. Alternatively, the 3D data encoding device can use map information or location information to segment based on whether point groups are contained in 3D space (tile information).

[0232] Figure 27 This is a diagram showing the relationship between tiles and slices. For example... Figure 27 As shown, tiles correspond to three-dimensional space, and slices correspond to segmented three-dimensional point groups. Furthermore, multiple tiles can overlap. This enables parallel processing in adaptive encoding or decoding corresponding to content or objects, improving the flexibility of point group encoding or decoding systems.

[0233] [The structure of encoded data] The location and attribute information of each slice are encoded. At least a portion of the resulting encoded data is stored in the payload of the data unit. Additionally, a header is appended to the payload.

[0234] When multiple attribute information exists for each point, the multiple attribute information is stored in the data unit respectively. Figure 28 This is a diagram illustrating an example of the structure of a bitstream (encoded data). Figure 28 This represents the encoded data of a group of points in two frames, each possessing two types of attribute information. The bitstream consists of a Geometry Data Unit (Geom) and two Attribute Data Units (Attr(0), Attr(1)) for each slice. For example, in the case where each point has both color and reflectance attributes, the encoded color data is stored in Attr(0), and the encoded reflectance data is stored in Attr(1). The slice identifier (slice_id) is stored in the header of the Geometry Data Unit. The slice_id of the corresponding (to be referenced) Geometry Data Unit is shown in the header of the Attribute Data Unit. Furthermore, sometimes the data unit is referred to as a slice, and the data unit header is called the slice header.

[0235] Additionally, metadata related to location information encoding is stored in the Geometry Parameter Set (GPS). Metadata related to attribute information encoding is stored in the Attribute Parameter Set (APS). Metadata common to multiple PCC frames (PCC sequences) is stored in the Sequence Parameter Set (SPS).

[0236] Each data unit and parameter set is transformed into a NAL (Network Access Layer) unit or a TLV (Type Length Value) unit, and the data string (stream) of the output unit is generated.

[0237] Figure 29 This is a block diagram illustrating an example configuration of a three-dimensional data encoding apparatus that outputs a stream of TLV units. The three-dimensional data encoding apparatus includes an encoding unit 241 and a TLV storage unit 242. The encoding unit 241 generates encoded data by encoding point group data. The TLV storage unit 242 generates a TLV stream (a single stream of TLV units) by storing (encapsulating) the encoded data in multiple TLV units.

[0238] A unit includes a Type representing the data category, a Length representing the length information, and a Value storing the data represented by the Type. Alternatively, a unit can also be in other forms, such as one that does not include length information.

[0239] Figure 28 This represents a string of cells that store data units and parameter sets. Figure 28 The arrows shown indicate dependencies related to the decoding of encoded data. The starting point of the arrow depends on the data at the tip of the arrow, and the identifier of the data at the tip of the arrow (referencing the destination) is shown in the header of the source data. The 3D data decoding device decodes the data at the tip of the arrow and uses this decoded data to decode the data at the starting point of the arrow. For example, the ID of the GPS and the ID of the SPS corresponding to the geometric data unit (Geom) are shown in the geometric data unit. The ID of the APS corresponding to the attribute data unit (Attr) is shown in the attribute data unit.

[0240] GPS and APS can also be set for each frame. Alternatively, if the encoding method is changed for each slice, GPS and APS can also be set for each slice. Furthermore, GPS and APS can be shared across multiple frames (sequences).

[0241] Additionally, APS can be shared by multiple attribute information sets. The parameter set of the reference destination is sent out before that of the reference source.

[0242] In cases where dependencies exist within data units across frames, such as when using inter-frame prediction, a Group of Frames (GOF) containing multiple frames can be constructed. A GOF is a random access unit; the initial slice of a GOF has no dependencies and is the slice at the start of decoding. The stream can also have delimiters representing frame boundaries or GOF boundaries, or TLV units representing boundaries.

[0243] Next, the composition of the encoded data and the method of storing the encoded data in the NAL unit will be explained.

[0244] For example, the data format is specified according to each type of encoded data. Figure 30 This is a diagram showing examples of encoded data and NAL units.

[0245] For example, such as Figure 30 As shown, the encoded data includes a header and a payload. Additionally, the encoded data may also include length information indicating the length (data volume) of the encoded data, header, or payload. The encoded data can also be in TLV format, including length information. Alternatively, the encoded data may not include a header.

[0246] The header may contain identification information used to determine the data. This identification information may indicate, for example, the data category or frame number.

[0247] The header, for example, contains identification information indicating a reference relationship. This identification information is stored in the header, for example, when there is a dependency between data, and is used to refer from the reference source to the reference destination. For example, the header of the reference destination contains identification information used to identify the data. The header of the reference source contains identification information indicating the reference destination.

[0248] Furthermore, if the reference destination or reference source can be identified or derived from other information, the identification information used to determine the data or the identification information indicating the reference relationship can be omitted.

[0249] The 3D data encoding device stores the encoded data in the payload of the NAL unit. The NAL unit header contains pcc_nal_unit_type, which serves as identification information for the encoded data. Figure 31 This is a diagram illustrating an example of the semantics of pcc_nal_unit_type.

[0250] like Figure 31 As shown, when pcc_codec_type is codec1 (Codec1: the first encoding method), the values ​​of pcc_nal_unit_type from 0 to 10 are assigned to the following data in codec1: Geometry, AttributeX, AttributeY, Geometry PS, AttrX PS, AttrY PS, Geometry Sequence PS, AttributeX Sequence PS, AttributeY Sequence PS, AU Header, and GOF Header. Additionally, values ​​from 11 onwards are assigned as preparation for codec1.

[0251] Hereinafter, a variation of this embodiment will be described. As with frame-level PS, sequence-level PS, and PCC sequence-level PS, PS has levels. If the PCC sequence level is set as the higher level and the frame level is set as the lower level, the parameter storage method can also be the method described below.

[0252] The default value of PS is represented by the higher-level PS. Additionally, if the value of the lower-level PS differs from the value of the higher-level PS, the lower-level PS represents the value of PS. Alternatively, the value of PS may not be recorded in the higher-level PS, but may be recorded in the lower-level PS. Alternatively, information may be provided in either or both of the lower-level and higher-level PS indicating whether the value of PS is represented by the lower-level PS, the higher-level PS, or both. Alternatively, the lower-level PS and the higher-level PS may be combined. Alternatively, if the lower-level PS and the higher-level PS are duplicated, the 3D data encoding device may omit the transmission of either one.

[0253] Alternatively, a 3D data encoding device can segment data into slices or tiles, and output the segmented data. The segmented data contains information for identifying the segmented data, and the parameter set contains parameters to be used in decoding the segmented data. In this case, an identifier is defined in `pcc_nal_unit_type` to represent data or parameters associated with tiles or slices.

[0254] [Gaussian Data] The following section explains Gaussian Splatting 3D data. First, it describes the structure of a system for encoding or decoding 3D data generated using Gaussian Splatting.

[0255] Figure 32 It means Figure 1 A diagram showing a portion of the processing in the 3D data generation system. Figure 32 The sensor information input unit 301 and the three-dimensional data generation unit 302 shown correspond, for example, to... Figure 1 The sensor information acquisition unit 117 and the three-dimensional data generation unit 118 shown are illustrated.

[0256] The sensor information input unit 301 acquires point group data (three-dimensional point group data) and outputs the acquired point group data to the three-dimensional data generation unit 302. The three-dimensional data generation unit 302 generates Gaussian splatting data based on the point group data and outputs the generated Gaussian splatting data.

[0257] Specifically, the 3D data generation unit 302 first generates a raster based on the density of the input point group data. Each point in the point group data is projected onto a nearby raster. At this time, attribute information such as color, reflectivity, or normal of the points is also associated with the raster. Based on the information projected onto the raster, the 3D data generation unit 302 derives parameters such as the mean and variance of the Gaussian function.

[0258] Furthermore, the methods for generating Gaussian data are not limited to those described above. For example, Gaussian data can also be generated using machine learning from point clusters generated from multiple two-dimensional images using techniques such as Structure from Motion (SfM).

[0259] Gaussian data is a collection of multiple 3D Gaussians (ellipsoids). Each 3D Gaussian includes data such as three-dimensional point coordinates, a 3×3 covariance matrix, color, and transmissivity. Furthermore, the above data of 3D Gaussians can be transformed into the constituent elements recorded in PLY format for representation.

[0260] Figure 33 It is a graph representing the constituent elements of Gaussian data recorded in PLY format. Figure 33 The Gaussian data shown includes position, orientation, scale, SH coefficient, and transparent. Transparent is also referred to as transmittance, transparency, or opacity.

[0261] Position refers to the three-dimensional coordinates of a point. Scale and Orientation are parameters of the three-dimensional covariance matrix. Additionally, the SH coefficients are the coefficients of the spherical harmonic function when color information is represented using spherical harmonics.

[0262] In other words, Gaussian data contains the reference three-dimensional coordinates (position) of each 3D Gaussian and the information associated with the three-dimensional coordinates (rotation, scale, SH coefficient, transmittance).

[0263] Furthermore, the composition of Gaussian data is not limited to the above. For example, Gaussian data may not contain a covariance matrix. For example, Gaussian data may contain the position representing the three-dimensional coordinates of a three-dimensional point, the scale representing the scale along each axis, such as the color representing RGB, and the transparency.

[0264] Alternatively, Gaussian data can also consist of multiple cubes (voxels) representing the division of 3D space, instead of multiple ellipsoids. Each voxel has attribute information. For example, each voxel may also contain: position representing the three-dimensional coordinates of the voxel's center point; attribute information such as color, reflectivity, or material; and scale representing the size of the voxel.

[0265] Alternatively, position may not be represented by three-dimensional coordinates, but by two-dimensional point coordinates (x, y). In this case, scale can also represent the size or shape of an ellipse on a two-dimensional plane. Specifically, Gaussian data can also include position representing the location on a two-dimensional plane, scale vectors representing the scale (Su, Sv) along each axis, and rotation information representing the orientation of the ellipse.

[0266] Furthermore, the format for recording Gaussian data is not limited to PLY format. For example, Gaussian data can also be recorded in SPZ format, etc.

[0267] [Rendering and processing of Gaussian data] Next, the rendering process of Gaussian data will be explained. Figure 34 This is a graph representing the rendering process of Gaussian data. For example... Figure 34 As shown, the rendering unit 303 generates a 3D model (three-dimensional model) by rendering Gaussian data. Therefore, the generated 3D model can be displayed using a 3D display device or similar application. The 3D display device could be, for example, a device that displays stereoscopic images in space like a hologram, or a glasses-free 3D display device.

[0268] Figure 35 This is a graph representing an example of the number of SH coefficients in a spherical harmonic function. The number of features (the number of SH coefficients) in a spherical harmonic function is determined by the level representing the resolution. For example, a higher level (a larger level value) indicates a higher resolution. For instance, a higher level adds more information about high-frequency components.

[0269] For example, at performance level 2, the spherical harmonic function has a total of 9 elements (SH coefficients) from level 0 to 2. For example, at performance level 3, the spherical harmonic function has a total of 16 elements (SH coefficients) from level 0 to 3.

[0270] For example, Figure 33 The example shown represents the spherical harmonic function for level 3 color, where the three color elements (R, G, B) are each represented by 16 SH coefficients. Therefore, the spherical harmonic function has 16 × 3 = 48 elements. Furthermore, levels 4 and above can also be determined.

[0271] By using spherical harmonic functions to represent color, three-dimensional color information can be represented. Figure 36 This is a graph representing an example of the input and output of a spherical harmonic function. For example, as shown... Figure 36 As shown, by inputting the viewpoint information into the spherical harmonic function 304 of color, the color information (R, G, B) observed from that viewpoint is output.

[0272] Figure 37 This is a diagram representing the rendering process. For example... Figure 37 As shown, the rendering unit 305 generates a 3D model or 2D image viewed from a specified viewpoint based on Gaussian data and that viewpoint. The generated 3D model can be displayed in applications such as VR displays. Additionally, the generated 2D image can be displayed in applications such as 2D image display devices.

[0273] Furthermore, beyond color, spherical harmonic functions can also be used to represent other attribute information (reflectivity or infrared information, etc.). In this case, the Gaussian data includes the SH coefficients of the spherical harmonic function for each attribute. Additionally, the level can be changed based on the attribute information or resolution.

[0274] Encoding and Decoding of Gaussian Data Next, the encoding and decoding processes for Gaussian data will be explained. Figure 38 This is a block diagram of the encoding apparatus (encoding system) of this embodiment. The encoding apparatus encodes and multiplexes Gaussian data. Figure 39 This is a block diagram of the decoding apparatus (decoding system) of this embodiment. This decoding apparatus decodes Gaussian data and uses it to suggest 3D models or 2D images. Furthermore, Figure 38 and Figure 39 The encoding and decoding devices shown are, for example, included in Figure 1 The three-dimensional data generation system shown.

[0275] Figure 38 The encoding apparatus shown includes an encoding unit 311 and a multiplexing unit 312. The encoding unit 311 generates encoded data (bit stream) by encoding the input Gaussian data in a prescribed encoding method, and outputs the generated encoded data to the multiplexing unit 312.

[0276] The multiplexing unit 312 generates multiplexed data by multiplexing the input encoded data using a prescribed multiplexing method, and outputs the generated multiplexed data. The multiplexed data is stored or transmitted.

[0277] Figure 39 The decoding device shown includes a demultiplexing unit 321, a decoding unit 322, and an application unit 323. The demultiplexing unit 321 generates encoded data by demultiplexing the input multiplexed data using a predetermined multiplexing method, and outputs the generated encoded data to the decoding unit 322.

[0278] The decoding unit 322 generates Gaussian data by decoding the input encoded data in a prescribed encoding method (decoding method), and outputs the generated Gaussian data to the application unit 323.

[0279] The application unit 323 includes an input interface unit 324, a rendering unit 325, and a prompting unit 326. The input interface unit 324 acquires user operations. For example, based on the user operations, the input interface unit 324 generates viewpoint information representing the viewpoint input by the user.

[0280] The rendering unit 325 generates a 3D model or a 2D image by rendering the input Gaussian data. For example, the rendering unit 325 generates a 3D model or 2D image viewed from a viewpoint input by the user, represented by viewpoint information. Alternatively, the rendering unit 325 can also generate a 3D model or 2D image viewed from a predetermined viewpoint.

[0281] The prompting unit 326 prompts (displays) a 3D model or a 2D image generated by the rendering unit 325. For example, it can also be determined whether to prompt a 3D model or a 2D image using the following methods. For example, ... Figure 34 As shown in the process, the rendering unit 325 first generates a 3D model and generates the coordinate information of the points. After receiving the input viewpoint information, the rendering unit 325 generates a 2D image viewed from the specified viewpoint. At this time, the rendering unit 325 can also receive information on specifying a region along with the specified viewpoint. In this way, by first generating a 3D model and then generating a 2D image of the specified region, the processing load can be reduced compared to generating a 2D image of the entire region. Therefore, the rendering speed can be improved. Alternatively, it can be determined whether to prompt for a 3D display (3D model) or a 2D image based on the processing power of the device performing the rendering. For example, in the case of a device with low processing power, the device may prompt for a 3D model without generating a 2D image.

[0282] (Method 1) [Encoding of Gaussian data using G-PCC] Figure 40 This is a block diagram illustrating a configuration example of the encoding apparatus 330 (three-dimensional data encoding apparatus) of this embodiment. The encoding apparatus 330 generates a bit stream (encoded data) by encoding the first Gaussian data.

[0283] The encoding device 330 includes a first preprocessing unit 331, a second preprocessing unit 332, and a G-PCC encoding unit 333.

[0284] The first Gaussian data includes, for example, a covariance matrix. The first preprocessing unit 331 transforms the first Gaussian data into second Gaussian data composed of elements in PLY format. For example, ... Figure 33 As shown, the components of a PLY file include three-dimensional coordinates, scale, rotation, the SH coefficient of the spherical harmonic function, and transmittance.

[0285] Furthermore, this example describes the input of first Gaussian data to encoding device 330, where encoding device 330 transforms the first Gaussian data into second Gaussian data. However, it is also possible that second Gaussian data is input to encoding device 330. In this case, encoding device 330 may not have a first preprocessing unit 331.

[0286] The second preprocessing unit 332 transforms the multiple constituent elements contained in the second Gaussian data into a format that can be encoded in an encoding method corresponding to each constituent element, and outputs the transformed multiple constituent elements to the G-PCC encoding unit 333.

[0287] The G-PCC encoding unit 333 uses geometric (position information) encoding methods (octree encoding, prediction tree encoding, TriSoup, etc.) in the G-PCC encoding method (Geometry based PCC) to encode the three-dimensional coordinates. The G-PCC encoding unit 333 uses attribute encoding methods (LoD-based encoding method (LoD attribute information encoding method) or Transform-based encoding method (transform attribute information encoding method) in the G-PCC encoding method to encode information related to the three-dimensional coordinates, such as rotation, scale, SH coefficient, and transmittance.

[0288] The G-PCC encoding unit 333 includes a location information encoding unit 341, a rotation encoding unit 342, a scale encoding unit 343, an SH coefficient encoding unit 344, a transmittance encoding unit 345, a metadata encoding unit 346, and a multiplexing unit 347.

[0289] The location information encoding unit 341 encodes the three-dimensional coordinates. These three-dimensional coordinates are the reference or representative coordinates of the Gaussian data. For example, the three-dimensional coordinates could be the center coordinates of the ellipse constituting the Gaussian data, or other coordinates. For example, the three-dimensional coordinates could also be the origin coordinates of the raster used to generate the Gaussian data.

[0290] Furthermore, the three-dimensional coordinates are not processed by the first preprocessing unit 331, but are directly input to the second preprocessing unit 332. The second preprocessing unit 332 transforms the input three-dimensional coordinates into positive integer information (three-dimensional coordinates). The scale value and offset value used during the transformation are stored in metadata such as SPS and are notified to the decoding device.

[0291] The location information encoding unit 341 generates encoded data by encoding the three-dimensional coordinates output from the second preprocessing unit 332 using the G-PCC geometric encoding method. This encoded data is stored in a GDU (Geometry Data Unit) and output. Furthermore, the metadata required for decoding the GDU is stored in a GPS (Geometry Parameter Set) and output.

[0292] Furthermore, when the multiple three-dimensional coordinates of multiple Gaussian data are sparse (density less than a predetermined threshold), the location information coding unit 341 can also use a prediction tree coding method suitable for encoding sparse three-dimensional points. Alternatively, when the multiple three-dimensional coordinates of multiple Gaussian data are dense (density greater than a predetermined threshold), the location information coding unit 341 can use an octree coding method suitable for encoding dense three-dimensional points.

[0293] The first preprocessing unit 331 transforms the covariance matrix contained in the first Gaussian data into rotation and scaling values. The first preprocessing unit 331 transforms the color information contained in the first Gaussian data into a spherical harmonic function and outputs the SH coefficients in the spherical harmonic function.

[0294] The second preprocessing unit 332 transforms the data type of each data element contained in the second Gaussian data. Specifically, the second preprocessing unit 332 transforms the data into positive integer data in order to encode each data element using the G-PCC encoding method. For example, if the data is of type float, the second preprocessing unit 332 transforms the data into positive integer data by scaling and offsetting the data. At this time, the scale value and offset value of each attribute information used in the transformation are stored in metadata such as SPS and are notified to the decoding device.

[0295] Furthermore, the second preprocessing unit 332 performs format transformation on each data item and mapping processing to map each data item to attribute components. For example, when the maximum number of features (dimensions) for each attribute component that can be encoded using the G-PCC attribute encoding method is 3, the second preprocessing unit 332 transforms the input constituent elements into attribute components with three dimensions (3 features) for each attribute component. For example, Scale has 3 features, so it is directly output as 1 attribute component. In addition, Orientation has four features, so it is transformed into two two-dimensional attribute components. The number of features for SH coefficients varies according to each level. For example, SH coefficients with more than 3 levels are transformed into multiple attribute components.

[0296] At this time, the second preprocessing unit 332 generates mapping information as metadata for each attribute information, which indicates which attribute component each feature is mapped to.

[0297] Rotation encoding unit 342, scale encoding unit 343, SH coefficient encoding unit 344, and transmittance encoding unit 345 respectively encode rotation, scale, SH coefficient, and transmittance using the attribute encoding method in the G-PCC encoding scheme, thereby generating multiple encoded data. The generated encoded data are stored in ADUs (Attribute Data Units) and output. The metadata required for decoding the ADUs is stored in APS (Attribute Parameter Set) and output. Additionally, attribute information is encoded using three-dimensional coordinates.

[0298] The metadata encoding unit 346 stores transformation information, including the transformation parameters used in the first preprocessing unit 331 or the second preprocessing unit 332, in metadata such as SPS or SEI. Alternatively, the metadata encoding unit 346 may encode the transformation information and then store it in the metadata. Furthermore, the metadata may also include information about the composition of the Gaussian data, or information indicating how the Gaussian data is transmitted via G-PCC encoding.

[0299] The multiplexing unit 347 saves the encoded data (ADU and GDU) and parameter sets (APS, GPS, and SPS) into the TLV unit and transmits them as a bit stream (encoded data). Furthermore, the bit stream can also be multiplexed using a specified multiplexing method. Additionally, the bit stream can be formatted.

[0300] [Decoding Gaussian data using G-PCC] Figure 41 This is a block diagram illustrating an example configuration of the decoding apparatus 350 (three-dimensional data decoding apparatus) of this embodiment. The decoding apparatus 350 generates first Gaussian data or second Gaussian data by decoding the bitstream (encoded data). For example, the decoding apparatus 350 decodes data from... Figure 40 The bit stream generated by the encoding device 330 shown is decoded.

[0301] The decoding device 350 includes a G-PCC decoding unit 351, a second post-processing unit 352, and a first post-processing unit 353.

[0302] The G-PCC decoding unit 351 decodes the bitstream using the G-PCC encoding method to generate three-dimensional coordinates, rotation, scale, SH coefficients, and transmittance. The G-PCC decoding unit 351 includes a demultiplexing unit 361, a position information decoding unit 362, a rotation decoding unit 363, a scale decoding unit 364, an SH coefficient decoding unit 365, a transmittance decoding unit 366, and a metadata decoding unit 367.

[0303] The demultiplexing unit 361 parses the multiple TLV units contained in the input bitstream to generate data units of encoded data such as GDU, ADU, SPS, GPS, and APS. The location information decoding unit 362, rotation decoding unit 363, scale decoding unit 364, SH coefficient decoding unit 365, transmittance decoding unit 366, and metadata decoding unit 367 use the information from the parameter set to decode these data units according to their respective encoding methods for each attribute component.

[0304] The position information decoding unit 362 decodes three-dimensional coordinates from the GDU. The rotation decoding unit 363, scale decoding unit 364, SH coefficient decoding unit 365, and transmittance decoding unit 366 decode attribute information (rotation, scale, SH coefficient, and transmittance) from the ADU. Additionally, the attribute information is decoded using three-dimensional coordinates. The metadata decoding unit 367 obtains transformation information from metadata such as SPS or SEI. Furthermore, the metadata decoding unit 367 can also decode this transformation information from metadata. Moreover, the metadata may also contain compositional information of the Gaussian data, or information indicating how the Gaussian data is transmitted via G-PCC encoding.

[0305] The second post-processing unit 352 generates the second Gaussian data by performing an inverse transform on the decoded data. Here, the inverse transform is the opposite of the transform process performed by the second pre-processing unit 332, and is performed based on the transform information (transformation parameters) contained in the metadata.

[0306] The first post-processing unit 353 generates the first Gaussian data by performing an inverse transform on the second Gaussian data. Here, the inverse transform is the opposite of the transform process performed by the first pre-processing unit 331. Furthermore, at least one of the first Gaussian data and the second Gaussian data is output. Alternatively, the decoding device 350 may not include the first post-processing unit 353.

[0307] [Mapping Processing] The second preprocessing unit 332 maps Gaussian data to the attribute components of G-PCC. The method of this mapping and the mapping information generated at this time are explained.

[0308] The following example illustrates the mapping of rotation, scale, and SH coefficients, omitting an explanation of transmittance. Furthermore, the same process can be applied when additional attribute information is added.

[0309] Figure 42 This is a diagram illustrating an example of mapping multiple attribute components contained in Gaussian data to attribute components in G-PCC, which is used as an encoding method. Here, an example is shown where the maximum dimension (number of sub-components) that can be encoded in the attribute component encoding in G-PCC is 3.

[0310] In addition, the attribute components of Gaussian data will sometimes be referred to as Gaussian components, and the attribute components of G-PCC will be referred to as G-PCC components. Furthermore, attribute components may sometimes be abbreviated as components.

[0311] For example, a Gaussian component of scale has subcomponents (B1~B3) with dimension = 3, which is mapped to a G-PCC component with three-dimensional subcomponents. In this case, the identifier of the Gaussian component of scale is shown in the mapping information to correspond to the identifier of the G-PCC component.

[0312] Furthermore, the rotated Gaussian component has subcomponents (A1~A4) with dimension 4, which cannot be mapped to a single G-PCC component with three-dimensional subcomponents. Therefore, the four rotated subcomponents (A1~A4) are mapped to two G-PCC components with two-dimensional subcomponents. In this case, the mapping information shows that the identifier of the rotated Gaussian component and the identifier indicating the number of the subcomponent contained in that attribute component correspond to the identifier of the G-PCC component and the identifier indicating the number of the subcomponent contained in that G-PCC component.

[0313] The 16 SH coefficients at each level exist for each feature in three dimensions (RGB), thus the SH coefficients have a total of 48 features (C1~C48). These 48 features are mapped to 16 G-PCC components with three-dimensional subcomponents.

[0314] also, Figure 42 The “ID” shown is the attribute component ID, which represents the identification number used to identify the attribute component used to encode the data. “ND” is the number of dimensions, which represents the number of features (sub-components) for each attribute component.

[0315] Furthermore, the methods for partitioning and mapping Gaussian data are not limited to the examples mentioned above; any combination of mapping methods and dimensions can be used.

[0316] The second preprocessing unit 332 generates mapping information representing the G-PCC components corresponding to the Gaussian components during the mapping process. The mapping information is stored as metadata in the header or SEI.

[0317] The decoding device decodes the mapping information from the bitstream. Based on the mapping information, the second post-processing unit 352 remaps the decoded data of each G-PCC component back to Gaussian data, thereby reconstructing the Gaussian data.

[0318] Figure 43 This is a diagram representing an example of mapping information. Furthermore, in... Figure 43 In this context, mapping information is stored in the SEI (Gaussian Data Attribute Mapping Information). The mapping information includes the number of Gaussian data components. The number of Gaussian data components represents the number of attribute information contained in the Gaussian data. For example, in... Figure 42 In the example shown, there is rotation, scaling, and SH coefficients, and the number of Gaussian data components is 3.

[0319] In addition, the mapping information includes the Gaussian data component ID, Gaussian data type, and Gaussian data dimension as information for each Gaussian component.

[0320] The Gaussian component ID is identification information used to uniquely identify a Gaussian component. The Gaussian data type indicates the type of the Gaussian component. For example, a value of 0 represents "rotation," a value of 1 represents "scale," a value of 2 represents "SH coefficient," and a value of 3 represents "transmittance." Furthermore, this is just one example of a combination of value and type; combinations are not limited to this.

[0321] The dimension of Gaussian data refers to the dimension of the Gaussian components (the number of subcomponents). For example, in... Figure 42 In the example shown, the dimension of the rotated Gaussian data is 4, the dimension of the scaled Gaussian data is 3, and the dimension of the SH coefficients Gaussian data is 48.

[0322] Additionally, the mapping information includes G-PCC component IDs, G-PCC dimension IDs, and transformation information for each dimension of the Gaussian component. The G-PCC sub-component IDs correspond to... Figure 42 The "ID" shown. The G-PCC dimension ID is an identifier representing the sub-component number. For example, Figure 42 The SH coefficients shown include C1, which is a G-PCC dimension ID=0; C2, which is a G-PCC dimension ID=1; and C3, which is a G-PCC dimension ID=2.

[0323] Thus, the mapping information represents the ID of the G-PCC component and the dimension ID corresponding to each dimension of the Gaussian data. For example, for Figure 42 The subcomponent C46 shown illustrates G-PCC component ID=18 and G-PCC dimension ID=2.

[0324] In addition, Figure 43The example shown is a mapping information of the sub-components (dimensions) of G-PCC corresponding to each sub-component (dimension) of Gaussian data, but mapping information of the sub-components of Gaussian data corresponding to each sub-component of G-PCC can also be used.

[0325] In addition, Figure 43 The example shown illustrates the mapping information representing the correspondence between subcomponents of Gaussian data and subcomponents of G-PCC. However, when there is no limit to the number of subcomponents in a G-PCC component, mapping information representing the correspondence between Gaussian components and G-PCC components can also be used. That is, the correspondence between dimensions may not be shown.

[0326] Transformation information represents the parameters (transformation parameters) used when transforming the data of the components. Figure 44 This is a diagram representing an example of transformation information. For instance, in the case where scale and offset are used in the transformation, the transformation information includes scale values ​​and offset values.

[0327] For example, the transformation can be performed using scale and offset values ​​using the following formula.

[0328] Transformed data = Original data × Scale value + Offset Furthermore, the formula used in the transformation can be a predefined formula, or multiple transformation formulas can be defined, and the transformation formula used can be switched according to specified conditions. Additionally, information representing the transformation formula used can be stored in a bitstream.

[0329] In addition, Figure 43 In this context, the transformation information is set for each sub-component (dimension) of the Gaussian data, but the transformation information can also be set for each component of the Gaussian data. That is, the transformation method can be changed on a component-by-component basis or on a sub-component-by-sub-component basis.

[0330] Alternatively, it can be omitted. Figure 43 This is part of the syntax shown. For example, if the order and dimension of information such as rotation, scale, and SH coefficients are predetermined, the Gaussian data component ID, Gaussian data type, and Gaussian data dimension can be omitted.

[0331] In addition, based on the mapping information to the attribute components determined by the second preprocessing unit 332, each attribute information is encoded. Figure 45 This is a diagram illustrating an example of the structure of encoded data when Gaussian data is encoded using G-PCC.

[0332] In SPS, each G-PCC component is described with its G-PCC attribute component ID (attr_id), G-PCC attribute type (attribute_type), and G-PCC attribute component dimension (num_dimension). Alternatively, a new G-PCC attribute type can be defined to encode each feature of the Gaussian data.

[0333] Figure 46 This is a diagram representing an example of the G-PCC attribute type for Gaussian data. For example... Figure 46 As shown, for example, G-PCC attribute types can also be defined to represent the rotation, scale, SH coefficient, and transmittance of Gaussian data, respectively.

[0334] The header of the ADU is assigned a G-PCC attribute component ID (attr_id). The decoding device can also use the G-PCC attribute component ID contained in the header of the ADU, as well as the information contained in the SPS, to identify which G-PCC attribute type each ADU belongs to.

[0335] Furthermore, the mapping information (Gaussian data attribute mapping information SEI) can be transmitted as the SEI or included in the SPS. For example, the SPS can also contain mapping information representing the Gaussian components corresponding to the G-PCC components.

[0336] Figure 47 This is a flowchart of the processing of the decoding device in this embodiment. First, the decoding device obtains multiple TLV units from the bit stream, and obtains multiple data units (GDU, ADU) and multiple parameter sets (SPS, GPS, APS) from the multiple TLV units (S101).

[0337] Next, the decoding device decodes the three-dimensional coordinates using SPS, GPS, and GDU (S102). Next, the decoding device decodes the multiple attribute components recorded in SPS using APS, ADU, and three-dimensional coordinates (S103).

[0338] Next, the decoding device obtains the mapping information by parsing the Gaussian data attribute mapping information SEI (S104). Then, the decoding device reconstructs the second Gaussian data based on the decoded attribute components and the mapping information (S105). Next, the decoding device obtains the transformation information and uses the transformation information to perform an inverse transformation on the second Gaussian data, thereby generating the first Gaussian data (S106).

[0339] [other] Furthermore, for simplicity, the above explanation illustrates an example with a single frame of Gaussian data, but this method can also be applied when there are multiple frames of Gaussian data. In this case, either multi-frame coding in the G-PCC method or inter-frame prediction in the G-PCC method can be used.

[0340] Alternatively, the encoding device can generate multiple segmented data by dividing the Gaussian data into multiple regions, and encode each segmented data separately. Alternatively, the encoding device can assign the segmented data to tiles or slices in the G-PCC for encoding.

[0341] Furthermore, the above description shows an example of encoding the attribute information (scale, rotation, SH coefficient, transmittance, etc.) of the second Gaussian data, but other attribute information can also be encoded by using the method of this embodiment.

[0342] Figure 48 This is a block diagram representing a system for encoding and decoding the first Gaussian data. Figure 48 The system shown includes a second preprocessing unit 371, a G-PCC encoding unit 372, a G-PCC decoding unit 373, and a second postprocessing unit 374. Figure 49 This is a block diagram showing an example of the configuration of the second preprocessing unit 371. The second preprocessing unit 371 includes a transformation unit 375 and a mapping unit 376.

[0343] For example, when encoding the covariance matrix associated with three-dimensional coordinates contained in the first Gaussian data, the transformation unit 375 included in the second preprocessing unit 371 performs a format transformation on the covariance matrix. The mapping unit 376 generates the transformed first Gaussian data by mapping the format-transformed covariance matrix to G-PCC attribute information. The G-PCC encoding unit 372 generates encoded data by encoding the transformed first Gaussian data, the transformation information, and the mapping information.

[0344] The G-PCC decoding unit 373 decodes the encoded data to generate transformed first Gaussian data, transformation information, and mapping information. The second post-processing unit 374 uses the mapping information and transformation information to perform an inverse transformation on the transformed first Gaussian data to restore the first Gaussian data. That is, it restores the covariance matrix. In this way, the system can encode and decode first Gaussian data.

[0345] Furthermore, the transformation unit 375 can also transform the 3×3 covariance matrix into a format that can be encoded using G-PCC during the transformation of the covariance matrix. For example, the transformation unit 375 divides the 3×3 covariance matrix into three three-dimensional components. Alternatively, if the covariance matrix is ​​a symmetric matrix, the transformation unit 375 can also divide the 6-dimensional features, excluding features with the same value, into two three-dimensional components. Figure 50 This is a diagram representing the transformation process. Figure 50 In the example shown, six of the nine elements of the covariance matrix are extracted, excluding the elements with the same values ​​as the other elements (Syz, Szx, Szy) ([Sxx, Sxy, Sxz, Syy, Syz, Szz]).

[0346] Additionally, transformation information representing the transformed content is generated. This transformation information is stored in metadata. Alternatively, if the transformation method is uniquely determined beforehand, the transformation information may not be stored in the encoded data (bitstream).

[0347] As described above, by using the processing and methods described in this embodiment, Gaussian data can be encoded using G-PCC.

[0348] Specifically, the encoding device uses position information encoding in G-PCC to encode the three-dimensional coordinates contained in the Gaussian data. In addition, the encoding device uses attribute information encoding in G-PCC to encode the attribute information (scale, rotation, SH coefficient, transmittance, etc.) associated with the three-dimensional coordinates.

[0349] Furthermore, the encoding device transforms the attribute information of the Gaussian data into a data format that can be encoded using G-PCC. Additionally, the encoding device stores transformation information representing the transformation in metadata. Therefore, the decoding device can reconstruct the Gaussian data based on the decoded data.

[0350] Furthermore, the encoding device establishes a correspondence between the components or dimensions of the Gaussian data and the attribute components or dimensions that can be encoded using G-PCC. Thus, the encoding device can encode the Gaussian data using the G-PCC encoding method. Additionally, by defining G-PCC attribute types corresponding to the Gaussian data, the encoding device can encode the Gaussian data using the G-PCC encoding method.

[0351] In addition, the encoding device stores the mapping information representing the correspondence between the components and dimensions of the Gaussian data and the attribute components and dimensions in the metadata. Therefore, the decoding device can reconstruct the Gaussian data based on the decoded data.

[0352] (Method 2) Encoding method for Gaussian data The following describes a general data structure for encoding Gaussian data, as well as a data structure that can encode Gaussian data using G-PCC or other encoding methods.

[0353] By using the data structure described below, a universal method can be achieved to encode Gaussian data using various 3D encoding methods or 3D encoding / decoding standards. Therefore, Gaussian data can be encoded using various 3D encoding / decoding methods, including standards developed by various standards organizations, international standards, national standards, or standards developed by any organization.

[0354] Figure 51 This is a diagram representing the protocol stack of encoding standards used to encode Gaussian data. 3D Gaussian Codec is an encoding standard used to encode Gaussian data. Specific encoding methods for 3D Gaussian Codec include the V3C standard (ISO / IEC 23090-5), which uses more than one video codec to encode 3D data; the G-PCC standard (ISO-IEC 23090-9), which is a point group compression standard; and Draco, which is used for 3D mesh data. When using the V3C standard, video codec standards such as HEVC or VVC (Versatile Video Coding) are used.

[0355] Figure 52 This is a block diagram illustrating a configuration example of the encoding apparatus 400 (three-dimensional data encoding apparatus) according to this embodiment. The encoding apparatus 400 generates a bitstream (encoded data) by encoding second Gaussian data. The encoding apparatus 400 includes a second preprocessing unit 401, a G-PCC encoding unit 402, a V3C encoding unit 403, a metadata encoding unit 404, and a multiplexing unit 405.

[0356] The second preprocessing unit 401 transforms and maps the second Gaussian data into data that can be encoded by the 3D codec used, according to the 3D codec used.

[0357] For example, the second preprocessing unit 401 transforms and maps the second Gaussian data into data that can be encoded using the G-PCC encoding method. The G-PCC encoding unit 402 encodes the transformed data using the G-PCC encoding method to generate a G-PCC TLV unit.

[0358] Furthermore, the second preprocessing unit 401 transforms and maps the second Gaussian data into data that can be encoded using the V3C encoding method. The V3C encoding unit 403 encodes the transformed data using the V3C encoding method to generate V3C units.

[0359] The metadata encoding unit 404 generates metadata by encoding transformation information and mapping information related to the transformation and mapping of the second preprocessing unit 401. For example, the metadata encoding unit 404 stores the transformation information and mapping information in the metadata.

[0360] The multiplexing unit 405 generates a bitstream by unitizing and multiplexing G-PCC TLV units or V3C units and metadata. Specifically, the multiplexing unit 405 stores G-PCC TLV units or V3C units and metadata in 3D data units, which are data units of the 3D Gaussian Codec Layer.

[0361] Figure 53 This is a block diagram illustrating a configuration example of the decoding apparatus 410 (three-dimensional data decoding apparatus) of this embodiment. The decoding apparatus 410 generates second Gaussian data by decoding a bitstream. This bitstream is, for example, generated by... Figure 52 The encoding device 400 shown is generated. The decoding device 410 includes a demultiplexing unit 411, a G-PCC decoding unit 412, a V3C decoding unit 413, a metadata decoding unit 414, and a second post-processing unit 415.

[0362] The demultiplexing unit 411 generates G-PCC TLV units or V3C units and metadata by inverse cellization and demultiplexing the 3D data units contained in the bitstream.

[0363] The G-PCC decoding unit 412 generates decoded data by decoding the G-PCC TLV unit using the G-PCC decoding method. The V3C decoding unit 413 generates decoded data by decoding the V3C unit using the V3C decoding method.

[0364] The metadata decoding unit 414 generates transformation information and mapping information by decoding the metadata. For example, the metadata decoding unit 414 obtains transformation information and mapping information from the metadata.

[0365] The second post-processing unit 415 inversely transforms the decoded data into second Gaussian data based on the 3D encoding and decoding used for decoding.

[0366] Figure 54 This is a flowchart of the encoding process of the encoding device 400. First, the encoding device 400 determines the encoding / decoding type to be used in the encoding (S201). For example, the encoding / decoding type used in the encoding can be specified by the user or determined based on the content of the Gaussian data.

[0367] When the encoding and decoding type used in the encoding is G-PCC encoding (G-PCC in S201), the encoding device 400 uses G-PCC encoding to encode the Gaussian data and outputs a G-PCC TLV unit (S202).

[0368] On the other hand, when the encoding and decoding type used in the encoding is V3C encoding (V3C in S201), the encoding device 400 uses V3C encoding to encode the Gaussian data and outputs a V3C unit (S203).

[0369] Next, the encoding device 400 stores information indicating the encoding / decoding type used in the encoding in common metadata (S204). Here, common metadata refers to metadata that can be used in multiple encoding methods (G-PCC encoding method and V3C encoding method).

[0370] Next, the encoding device 400 stores the transformation information and mapping information in the metadata (S205). Then, the encoding device 400 multiplexes the metadata and encoding unit (G-PCC TLV unit or V3C unit) into a 3D data unit that can be used in both G-PCC encoding and V3C encoding (S206).

[0371] Figure 55 This is a flowchart of the decoding process of the decoding device 410. First, the decoding device 410 obtains a common 3D data unit, and obtains data units and metadata from the 3D data unit (S211). Next, the decoding device 410 parses the metadata and determines the encoding / decoding type used in encoding (the encoding / decoding type to be used in decoding) based on the information representing the encoding / decoding type contained in the metadata (S212).

[0372] When the encoding / decoding type used in the encoding is G-PCC (G-PCC in S213), the decoding device 410 obtains the TLV unit, decodes the TLV unit using the G-PCC decoding method, and generates a component (S214). On the other hand, when the encoding / decoding type used in the encoding is V3C (V3C in S213), the decoding device 410 obtains the V3C unit, decodes the V3C unit using the V3C decoding method, and generates a component (S215).

[0373] Next, the decoding device 410 uses the decoded components and mapping information to reconstruct the Gaussian data (S216). Then, the decoding device 410 obtains the transformation information and uses the transformation information to perform an inverse transformation on the Gaussian data, thereby generating the second Gaussian data (S217).

[0374] Figure 56This is a diagram representing the basic data structure of Gaussian encoding and decoding. The 3D data units in the 3D Gaussian codec layer are common units in Gaussian encoding and decoding.

[0375] Figure 57 This is a diagram illustrating the structure of a 3D Data Unit. The payload (3DDU_data) of a 3D Data Unit stores various data units that constitute the Gaussian codec. The header (3DDU_header) of a 3D Data Unit contains the 3D Data Unit Type (3DDataUnitType), which indicates the type of data stored in the payload. Examples of 3D Data Unit Types (3DDataUnitType) include: (1) V3C Unit, (2) G-PCC TLV Unit, (3) Draco Unit, (4) a common parameter set in multiple codecs, namely the 3D Parameter Set (3DPS), (5) a parameter set containing common metadata in a sequence, namely the Gaussian Sequence Parameter Set (GSSPS), (6) a parameter set containing common metadata in a frame (a 3D model in a certain time or time interval), namely the Gaussian Frame Parameter Set (GSFPS), and (7) SEI (Gaussian GSSEI), etc.

[0376] Furthermore, when the 3D Data Unit Type (3DDataUnitType) is V3C, the payload of the V3C unit includes either an ACL_NAL unit or a VCL_NAL unit. The VCL_NAL unit belongs to the Video Codec Layer, defined by 2D motion picture compression standards such as HEVC or VVC. The ACL_NAL unit belongs to the Atlas Codec Layer. The ACL_NAL unit stores data used to reconstruct 3D data from 2D images.

[0377] In addition, Draco cells based on standards such as Draco can also be stored in 3D data cells.

[0378] Figure 58This diagram illustrates the structure of the common parameter set (3DPS) in Gaussian coding. The 3DPS includes: profile and level information (profile_level), which represents the combination of coding standards, coding methods, and toolsets used in encoding Gaussian data; and Gaussian component information (3D_codec_type). This information includes profile information, level information, and coding standard and coding method information (3D_codec_type).

[0379] 3D_codec_type indicates the encoding standard used for encoding Gaussian data. For example, 3D_codec_type=0 indicates that V3C encoding is used, and 3D_codec_type=1 indicates that G-PCC encoding is used.

[0380] The decoding device can determine the decoding method by parsing the 3D_codec_type. The decoding device can also switch decoding methods based on the 3D_codec_type, or decide not to decode if it does not support decoding of the encoding method represented by the 3D_codec_type. Additionally, the grade and level can also contain information indicating the encoding standard.

[0381] In addition, restrictions can be imposed on 3D data unit types (3DDataUnitType) based on the encoding method. For example, the use of V3C units can be prohibited when the encoding method is G-PCC, and the use of G-PCC TLV units can be prohibited when the encoding method is V3C.

[0382] Gaussian data component information contains information representing the composition of Gaussian components. Figure 59 This is a diagram illustrating an example of the composition of Gaussian data component information. The Gaussian data component information includes the number of Gaussian data components, representing the quantity of Gaussian components.

[0383] The Gaussian data component information includes the Gaussian component ID, Gaussian data type, and Gaussian data dimension for each Gaussian component. The Gaussian component ID is the unique identifier for each Gaussian component. The Gaussian data type indicates the type of Gaussian component (rotation, scaling, SH coefficients, and transmittance, etc.). The Gaussian data dimension indicates the number of dimensions (number of subcomponents) of the Gaussian component.

[0384] In addition, the Gaussian data component information includes mapping information that depends on the encoding / decoding. For example, when the encoding / decoding type (3D_codec_type) is G-PCC, the Gaussian data component information includes G-PCC mapping information (gpcc_mapping_info) that represents the correspondence between the Gaussian data components and the G-PCC encoded / decoded components.

[0385] Figure 60 This is a diagram illustrating an example of the structure of G-PCC mapping information. The G-PCC mapping information includes the G-PCC component ID, the G-PCC dimension ID, and transformation information for each dimension of the Gaussian component.

[0386] G-PCC subcomponent ID is an identifier representing the G-PCC component number. G-PCC dimension ID is an identifier representing the subcomponent number of a G-PCC component. Transformation information indicates the parameters (transformation parameters) used when transforming the data in that dimension.

[0387] Additionally, when the codec type (3D_codec_type) is V3C, the Gaussian data component information includes V3C mapping information that represents the correspondence between the Gaussian data components and the V3C codec components.

[0388] Figure 61 This is a diagram illustrating an example of the structure of V3C mapping information. V3C mapping information includes V3C component IDs, V3C dimension IDs, and transformation information for each dimension of the Gaussian component.

[0389] V3C subcomponent ID is an identifier representing the V3C component number. V3C dimension ID is an identifier representing the subcomponent number of the V3C component. Transformation information indicates the parameters (transformation parameters) used when transforming the data in this dimension.

[0390] In addition to the information of attribute components, Gaussian data component information can also include information of geometric components.

[0391] Furthermore, an example is shown here representing mapping information for the G-PCC components corresponding to Gaussian components, but mapping information representing the Gaussian components corresponding to G-PCC components can also be used. Additionally, an example is shown here representing mapping information showing the correspondence between sub-components of Gaussian data and sub-components of G-PCC, but mapping information showing the correspondence between Gaussian components and G-PCC components can also be used. That is, the correspondence between sub-components may not be shown.

[0392] Alternatively, a portion of the syntax shown here can be omitted. For example, if the order and dimension of the rotation, scale, and SH coefficient information are predetermined, the Gaussian data component ID, Gaussian data type, and Gaussian data dimension can also be omitted.

[0393] The following describes another example of a data unit. Alternatively, instead of storing Gaussian data in either the V3C unit or the G-PCC TLV unit, a data unit specific to Gaussian data (Gaussian data unit) can be defined.

[0394] Figure 62 This is a diagram illustrating an example of the structure of a Gaussian data unit. For instance, the header of a Gaussian data unit contains a Gaussian DataUnitType, equivalent to the 3D DataUnitType described above. The Gaussian DataUnitType indicates the type of data stored in the payload of the Gaussian DataUnit. For example, the Gaussian DataUnitType contains geometry data units and attribute data units. The payloads of the geometry data units and attribute data units respectively store the encoded geometry data and attribute data.

[0395] The header of an attribute data unit contains the attribute data unit type. The attribute data unit type indicates the type of data stored in the payload of the attribute data unit. The attribute data unit type represents the elements of attribute information, such as rotation data units, scale data units, color coefficient data units, and alpha data units. The payloads of rotation data units, scale data units, color coefficient data units, and alpha data units respectively store the encoded rotation, scale, SH coefficients, and transmittance.

[0396] Furthermore, while the above explanation illustrates an example of encoding 3D Gaussian data, the same method can be applied to other types of data. For instance, the same method can be used for 3D models or 3D data that utilizes AI.

[0397] Furthermore, while the above explanation primarily focused on examples using V3C and G-PCC encoding methods, other encoding methods besides V3C and G-PCC can also be used. Additionally, encoding methods can be anything other than 3D codecs. For example, image codecs, neural network codecs, or audio codecs can also be used.

[0398] Figure 63 This is a diagram representing a standard protocol stack for encoding 3D Gaussians. Alternatively, a protocol stack or data structure can be used for encoding 3D Gaussians via at least one of video-based codecs and geometry-based codecs.

[0399] Video-based encoding and decoding is image encoding that uses video encoding and decoding, such as standards standardized by MPEG such as AVC, HEVC, or VVC, or standards such as VPCC (Video based Point Cloud Compression), VCM (Video Coding for Machines), or VDMC (Video based Dynamic Mesh Compression).

[0400] Geometry-based encoding and decoding include point group compression standards such as G-PCC or Draco, or grid data compression standards. Furthermore, the encoding of 3D Gaussian data can be extended to include neural network-based encoding and decoding methods such as NNC (Neural Network Coding).

[0401] Alternatively, it can replace 3D Gaussian to define a standard for encoding 3D models. Figure 64 This is a diagram representing a standard protocol stack for encoding 3D models. Alternatively, a protocol stack and data structures for encoding 3D models using at least one of V3C and G-PCC can be used.

[0402] 3D models can also include (1) point groups, (2) meshes, (3) 3D models made by NeRF (Neural Radiance Fields), and (4) 3D models generated by Gaussian Splatting, etc.

[0403] Alternatively, the encoding device can use video coding to encode one component of the 3D model data and geometry-based coding to encode another. For example, the encoding device can use G-PCC to encode the 3D coordinates, rotations, and scale—information representing shape—from the Gaussian data, and use video-based coding to encode the SH coefficients. Here, sparse geometric shapes are suitable for geometry-based coding, while multidimensional SH coefficients are sometimes suitable for video-based coding. Therefore, in this case, coding efficiency can be improved.

[0404] Figure 65 This is a diagram illustrating an example of the composition of Gaussian component information in this case. Figure 65 The Gaussian component information shown includes Figure 59 In addition to the Gaussian component information shown, the 3D codec type (3D_codec_type) for each Gaussian component is also included. Therefore, the 3D codec type can be set for each component, thus enabling the aforementioned functionality. Furthermore, as... Figure 64 As shown, when using a standard for encoding 3D models, for example, instead of Gaussian component information, the same information is stored in the 3D model component information that represents the components of the 3D model.

[0405] (Method 3) The following describes the methods and data structures used to extend the V3C standard and enable the application of general 3D encoding and decoding methods such as Gaussian data. Furthermore, other 3D encoding and decoding methods and data structures used to extend the V3C standard and enable the application of G-PCC and other methods are also described.

[0406] By using the data structure described below, Gaussian data can be encoded and decoded using various V3C-based standards. Furthermore, a universal method for encoding Gaussian data using various 3D encoding methods or 3D encoding / decoding standards can be implemented. Therefore, Gaussian data can be encoded using various 3D encoding / decoding methods, including standards developed by various standards organizations, international standards, national standards, or standards defined by any organization.

[0407] Figure 66 This is a diagram representing the protocol stack of the encoding standard used to encode Gaussian data. V3C encoding (V3C codec) is a structure that can encode 3D data using more than one video codec standard such as HEVC or VVC.

[0408] Here, the V3C encoding method is extended to define a Gaussian codec layer. Specifically, the atlas data is extended to store metadata required for Gaussian data reconstruction, encoding, and decoding. The atlas data contains data used to recover 3D data from 2D images using the V3C encoding method. Furthermore, the V3C encoding method is extended to enable the application of geometry-based 3D encoding standards such as G-PCC and Draco (hereinafter also referred to as GPC). The atlas data contains data used to recover Gaussian data from decoded 2D images or encoded 3D data decoded using 3D encoding standards using the V3C encoding method.

[0409] Figure 67This is a block diagram illustrating a configuration example of the encoding apparatus 500 (three-dimensional data encoding apparatus) according to this embodiment. The encoding apparatus 500 generates a bitstream (encoded data) by encoding second Gaussian data. The encoding apparatus 500 includes a second preprocessing unit 501, a geometry-based encoding unit 502, a video-based encoding unit 503, a metadata encoding unit 504, and a multiplexing unit 505.

[0410] The second preprocessing unit 501 transforms and maps the second Gaussian data into data that can be encoded by the 3D codec used, according to the 3D codec used.

[0411] For example, the second preprocessing unit 501 transforms and maps the second Gaussian data into data that can be encoded using a geometry-based encoding method (G-PCC or Draco, etc.). The geometry-based encoding unit 502 generates GCL NAL cells by encoding the transformed data using a geometry-based encoding method. Furthermore, when using G-PCC, the geometry-based encoding unit 502 can also output TLV cells.

[0412] Furthermore, the second preprocessing unit 501 transforms and maps the second Gaussian data into data that can be encoded using a video-based encoding method (AVC, HEVC, or VVC, etc.). The video-based encoding unit 503 generates VCL NAL units by encoding the transformed data using a video-based encoding method.

[0413] The metadata encoding unit 504 stores the transformation data and mapping information in the metadata. Here, the metadata is metadata obtained by extending the V3C standard. For example, the extended area of ​​the V3C parameter set in the V3C standard can be used as this metadata. Alternatively, the extended Atlas sequence parameter set (ASPS) or Atlas frame parameter set (AFPS) can also be used as this metadata. The metadata encoding unit 504 generates V3C parameter sets or Atlas data units (ACL NAL units) by encoding the metadata.

[0414] The multiplexing unit 505 generates a bitstream by cellifying and multiplexing GCL NAL units or VCL NAL units, as well as V3C parameter sets or ACL NAL units. Specifically, the multiplexing unit 505 generates a bitstream by storing GCL NAL units or VCL NAL units, as well as V3C parameter sets or ACL NAL units, in 3V3C units.

[0415] Figure 68This is a block diagram illustrating a configuration example of the decoding apparatus 510 (three-dimensional data decoding apparatus) of this embodiment. The decoding apparatus 510 generates second Gaussian data by decoding a bitstream. This bitstream is, for example, generated by... Figure 67 The encoding device 500 shown is generated. The decoding device 510 includes a demultiplexing unit 511, a geometry-based decoding unit 512, a video-based decoding unit 513, a metadata decoding unit 514, and a second post-processing unit 515.

[0416] The demultiplexing unit 511 generates GCLNAL units or VCL NAL units, as well as V3C parameter sets or ACL NAL units, by performing inverse cellization and demultiplexing on the V3C units contained in the bit stream.

[0417] The geometry-based decoding unit 512 generates decoded data by decoding the GCL NAL unit using a geometry-based decoding method (G-PCC, Draco, etc.). The video-based decoding unit 513 generates decoded data by decoding the VCL NAL unit using a video-based decoding method (AVC, HEVC, or VVC, etc.).

[0418] The metadata decoding unit 514 generates transformation information and mapping information by decoding the V3C parameter set or ACL NAL unit. For example, the metadata decoding unit 514 obtains transformation information and mapping information from the V3C parameter set or ACL NAL unit.

[0419] The second post-processing unit 515 inversely transforms the decoded data into second Gaussian data according to the 3D encoding and decoding to be used in decoding.

[0420] Figure 69 This is a flowchart of the encoding process of the encoding device 500. First, the encoding device 500 determines the encoding / decoding type (encoding method) to be used in the encoding (S501). For example, the encoding / decoding type to be used in the encoding can be specified by the user or determined based on the content of the Gaussian data.

[0421] When the encoding and decoding type to be used in the encoding is a geometry-based encoding method (G-PCC or Draco) (geometry-based in S501), the encoding device 500 uses the geometry-based encoding method (G-PCC or Draco) to encode the Gaussian data and outputs GCL NAL units (S502).

[0422] On the other hand, when the encoding and decoding type to be used in the encoding is a video-based encoding method (video-based in S501), the encoding device 500 uses the video-based encoding method (video encoding and decoding) to encode the Gaussian data and outputs the VCL NAL unit (S503).

[0423] Next, the encoding device 500 stores information indicating the encoding / decoding type used in the encoding in the metadata (S504).

[0424] Next, the encoding device 500 stores the transformation information and mapping information in the metadata (S505). Then, the encoding device 500 multiplexes the metadata and the encoding unit (GCL NAL unit or VCL NAL unit) into the V3C unit (S506).

[0425] Figure 70 This is a flowchart of the decoding process of the decoding device 510. First, the decoding device 510 obtains multiple V3C units and acquires data units (GCL NAL units or VCL NAL units) and metadata from the multiple V3C units (S511). Next, the decoding device 510 parses the metadata and determines the encoding / decoding type used in encoding (the encoding / decoding type to be used in decoding) based on the information representing the encoding / decoding type contained in the metadata (S512).

[0426] When the encoding / decoding type used is geometry-based (geometry-based in S513), the decoding device 510 acquires the GCL NAL unit and generates a component by decoding the GCL NAL unit using a geometry-based decoding method (G-PCC or Draco) (S514). On the other hand, when the encoding / decoding type used is video-based (video-based in S513), the decoding device 510 acquires the VCL NAL unit and generates a component by decoding the VCL NAL unit using a video-based decoding method (video encoding / decoding) (S515).

[0427] Next, the decoding device 510 uses the decoded components and mapping information to reconstruct the Gaussian data (S516). Then, the decoding device 510 obtains the transformation information and uses the transformation information to perform an inverse transformation on the Gaussian data to generate the second Gaussian data (S517).

[0428] Figure 71 This is a diagram illustrating an example of the structure of a V3C unit extended to enable the application of geometry-based 3D encoding and decoding to the V3C standard. Figure 72 This is a diagram illustrating an example of the structure of a V3C unit. A V3C unit consists of a header (V3C unit header) and a payload (V3C data unit).

[0429] The payload of the V3C unit stores ACL NAL units belonging to the atlas encoding / decoding layer or the video encoding / decoding layer. Furthermore, the payload of the extended V3C unit stores GCL NAL units containing geometry-based 3D encoding / decoding data.

[0430] The header of a V3C unit contains the V3C unit type, which indicates the type of data contained in the payload. For example, V3C unit type = 10 indicates a geometry-based PCC unit (GCL NAL unit).

[0431] Figure 73 This is a diagram illustrating an example of the structure of a GCL NAL unit. A GCL NAL unit includes a header and a data unit. The data unit of the GCL NAL unit stores either G-PCC TLV units or Draco units.

[0432] The header of a GCL NAL unit stores the GCL unit type, which indicates the type of data contained in the payload, i.e., the type of geometry-based 3D encoding / decoding. For example, GCL unit type = 0 indicates G-PCC point group compression (G-PCC TLV unit). GCL unit type = 1 indicates Draco point group compression (Draco unit).

[0433] Furthermore, examples of storing G-PCC TLV units, etc., in GCL NAL units are described here, but G-PCC TLV units or units determined by various 3D encoding and decoding methods can also be stored directly in V3C units.

[0434] Figure 74 This diagram illustrates the structure of the extended V3C Parameterset. The V3C Parameterset contains information representing the profile and level, which indicate the combination of encoding standards, encoding methods, and toolsets used in Gaussian data encoding. This information includes profile information, level information, and information on the codecs to be used in encoding (is_video_codec and is_3d_codec).

[0435] A profile indicates an extension of V3C, such as VPCC or DMC, for use with the codec. For example, profile indicates an extension of V3C's 3D Gaussian codec. For instance, profile=0 indicates VPCC, profile=1 indicates DMC, and profile=2 indicates 3D Gaussian codec.

[0436] `is_video_codec` is a flag indicating whether a video codec is used. When a video codec is used (`is_video_codec=1`), the V3C parameter set contains the video codec type (`video_codec_type`) indicating the video codec to be used.

[0437] `is_3d_codec` is a flag indicating whether 3D codec is used. When 3D codec is used (`is_3d_codec=1`), the V3C parameter set contains the 3D codec type (`3d_codec_type`) indicating the 3D codec to be used. For example, `3D_codec_type=0` indicates G-PCC, and `3D_codec_type=1` indicates Draco. Alternatively, the predefined "4CC" code in the 3D codec type can also be used.

[0438] Furthermore, when using 3D Gaussian encoding and decoding, and the values ​​of `is_video_codec` and `is_3d_codec` are mutually exclusive (one is 1, the other is 0), it is also possible to stipulate that the mutual exclusivity of `is_video_codec` and `is_3d_codec` is a compliance condition for the bitstream. That is, the decoding device can also determine that the bitstream is inappropriate (containing errors) even if the values ​​of `is_video_codec` and `is_3d_codec` are not mutually exclusive.

[0439] When using 3D Gaussian codec (profile=3D gaussian codec), the V3C parameter set includes Gaussian component information (3d_codec_type) as extended information for V3C's 3D Gaussian codec.

[0440] Figure 75 This is a diagram illustrating an example of the composition of Gaussian data component information. The Gaussian data component information includes the number of Gaussian data components, representing the quantity of Gaussian components.

[0441] The Gaussian data component information includes the Gaussian component ID, Gaussian data type, and Gaussian data dimension for each Gaussian component. The Gaussian component ID is the unique identifier for each Gaussian component. The Gaussian data type indicates the type of Gaussian component (rotation, scaling, SH coefficients, and transmittance, etc.). The Gaussian data dimension indicates the number of dimensions (number of subcomponents) of the Gaussian component.

[0442] Additionally, the Gaussian data component information includes mapping information that depends on the codec. For example, when using a video codec (is_video_codec=1), the Gaussian data component information includes video mapping information (video_mapping_info) representing the correspondence between the Gaussian data components and the video codec components. When using a 3D codec (is_3d_codec=1), the Gaussian data component information includes 3D mapping information (3D_mapping_info) representing the correspondence between the Gaussian data components and the 3D codec components.

[0443] Figure 76 This is a diagram illustrating an example of the structure of video mapping information. Video mapping information includes video component IDs, video dimension IDs, and transform information for each dimension of the Gaussian components.

[0444] The video sub-component ID is an identifier representing the video component number. The video dimension ID is an identifier representing the sub-component number of the video component. Transform information indicates the parameters (transform parameters) used when transforming the data in that dimension.

[0445] Figure 77 This is a diagram illustrating an example of the composition of 3D mapping information. When using G-PCC, the 3D mapping information includes the G-PCC component ID, the G-PCC dimension ID, and transformation information as information for each dimension of the Gaussian components.

[0446] G-PCC subcomponent ID is an identifier representing the G-PCC component number. G-PCC dimension ID is an identifier representing the subcomponent number of a G-PCC component. Transformation information indicates the parameters (transformation parameters) used when transforming the data in that dimension.

[0447] Furthermore, when using Draco, the 3D mapping information contains information representing the correspondence between the components of Gaussian data and the components encoded and decoded by Draco.

[0448] In addition to the information of attribute components, Gaussian data component information can also include information of geometric components.

[0449] Furthermore, an example of mapping information representing the G-PCC component corresponding to the Gaussian component is shown here, but mapping information representing the Gaussian component corresponding to the G-PCC component can also be used. An example of mapping information representing the correspondence between sub-components of Gaussian data and sub-components of G-PCC is shown here, but mapping information representing the correspondence between Gaussian components and G-PCC components can also be used. That is, the correspondence between sub-components may not be shown.

[0450] Alternatively, a portion of the syntax shown here can be omitted. For example, if the order and dimension of information recording for rotation, scaling, and SH coefficients are predetermined, the Gaussian data component ID, Gaussian data type, and Gaussian data dimension can also be omitted.

[0451] Furthermore, while the above description illustrates an example where the data of the encoded object is 3D Gaussian data, the data of the encoded object is not limited to 3D Gaussian data. For example, the protocol stack or data structure described in this embodiment can also be used in the encoding of 3D models generated by other methods such as NeRF. For example, the data of the encoded object can also be a 3D model (point group, mesh, or other 3D model) that uses AI (Artificial Intelligence).

[0452] In addition to video-based and geometry-based standards, V3C encoding methods can also use neural network encoding and decoding methods that have been learned through machine learning.

[0453] Figure 78 This is a diagram representing the extended V3C protocol stack. For example, as shown... Figure 78 As shown, the V3C standard can also be extended to enable the transmission of neural networks (NNs). For example, for neural networks, neural network-based coding methods such as NNC (Neural Network Coding) can be used.

[0454] Alternatively, codecs that extend the V3C standard can be defined, using one or more of the following encoding methods: video-based, geometry-based, and neural network-based.

[0455] Furthermore, for example, in point-group compression methods using AI to encode geometric information (location information) and attribute information, learned encoding can be applied to the geometric information. For instance, neural network-based or geometry-based encoding methods can be used for the geometric information, while video-based encoding methods can be used for the attribute information. This could potentially improve compression performance (encoding efficiency).

[0456] Figure 79 This is a diagram representing the extended V3C protocol stack. For example, as shown... Figure 79 As shown, the V3C standard is extended. For example, the V3C parameter set, ASPS, or AFPS are extended to define data units that store AI-PCC. Therefore, as an encoding / decoding method, at least one of video-based, geometry-based, and neural network-based encoding methods can be used, enabling encoding and decoding of point-group compression using AI.

[0457] [Summarize] As described above, the encoding device (three-dimensional data encoding device) of the embodiment performs... Figure 80 The processing is shown. Figure 80 This is a flowchart of the encoding process of the encoding device. The encoding device encodes three-dimensional data (e.g., Gaussian data, point groups, or meshes) containing one or more first components. The encoding device selects any one of several encoding methods, including a first encoding method (e.g., V3C, AVC, HEVC, or VVC) and a second encoding method different from the first encoding method. The first encoding method is a video-based three-dimensional data encoding method (S601). When the first encoding method is selected (the first encoding method in S602), first encoded data (e.g., VCL NAL units) is generated by encoding one or more first components using the first encoding method (S603). When the second encoding method is selected (the second encoding method in S602), second encoded data (e.g., GCL) is generated by encoding one or more first components using the second encoding method. The NAL unit (S604) stores the first information (is_video_codec, is_3d_codec, video_codec_type or 3d_codec_type) representing the selected encoding method in the first metadata (e.g., V3C parameter set, ASPS or AFPS) obtained by extending the metadata of the first encoding method (S605), and generates a bitstream containing the first metadata and the first encoded data or the second encoded data (S606).

[0458] Therefore, the encoding device can selectively select the encoding method for 3D data from multiple encoding methods. This could potentially improve encoding efficiency. Furthermore, by extending video-based 3D data encoding methods, a system capable of easily implementing the above-mentioned processes can be constructed.

[0459] For example, the encoding device may also transform one or more first components into one or more second components corresponding to the selected encoding method. When the first encoding method is selected, first encoded data is generated by encoding one or more second components using the first encoding method. When the second encoding method is selected, second encoded data is generated by encoding one or more second components using the second encoding method. The first metadata may also contain second information (e.g., mapping information) indicating the correspondence between one or more first components and one or more second components.

[0460] Therefore, the encoding device can encode three-dimensional data using the selected encoding method by changing the data into a form corresponding to the selected encoding method. Furthermore, the decoding device can use the second information contained in the bitstream to transform multiple second components into multiple first components.

[0461] For example, the second encoding method is a geometry-based 3D data encoding method (e.g., G-PCC or Draco). For example, the second encoding method is a neural network-based 3D data encoding method (e.g., NNC). For example, the second encoding method is a 3D data encoding method that utilizes AI.

[0462] For example, three-dimensional data is Gaussian data. Therefore, the encoding device can selectively select the encoding method for Gaussian data from multiple encoding methods. This could potentially improve encoding efficiency.

[0463] Figure 81 This is a block diagram of the encoding device 10. For example, the encoding device 10 includes a processor 11 and a memory 12, and the processor 11 uses the memory 12 to perform the above-described processing.

[0464] In addition, the decoding device (three-dimensional data decoding device) of the implementation method performs... Figure 82 The processing is shown. Figure 82 This is a flowchart of the decoding process of the decoding device. The decoding device decodes 3D data (e.g., Gaussian data, point groups, or meshes) containing more than one first component. The decoding device obtains first-component metadata (e.g., V3C parameter sets, ASPS, or AFPS) and encoded data (e.g., VCL NAL cells or GCL). The bitstream of the NAL unit, the first metadata is obtained by extending the metadata of the first encoding method (e.g., V3C, AVC, HEVC or VVC, etc.), the first encoding method is a three-dimensional data encoding method based on video (S611), the first information (is_video_codec, is_3d_codec, video_codec_type or 3d_codec_type) is obtained from the first metadata, the first information represents the encoding method used in the encoding of more than one first component, including the first encoding method and the second encoding method different from the first encoding method (S612), when the first information represents the first encoding method (the first encoding method in S613), the encoded data is decoded by using the first decoding method corresponding to the first encoding method to generate more than one first component (S614), when the first information represents the second encoding method (the second encoding method in S613), the encoded data is decoded by using the second decoding method corresponding to the second encoding method to generate more than one first component (S615).

[0465] Therefore, the encoding device can selectively select the encoding method for encoding 3D data from multiple encoding methods. This could potentially improve encoding efficiency. Furthermore, the decoding device can appropriately decode the encoded data generated by the encoding device. Moreover, by extending video-based 3D data encoding methods, a system capable of easily implementing the above-mentioned processes can be constructed.

[0466] For example, encoded data is generated by encoding one or more second components after one or more first components have been transformed. The decoding device also obtains second information (e.g., mapping information) representing the correspondence between one or more first components and one or more second components from the first metadata. When the first information represents a first encoding method, one or more second components are generated by decoding the encoded data using the first decoding method. When the first information represents a second encoding method, one or more second components are generated by decoding the encoded data using the second decoding method. Using the correspondence represented by the second information, one or more second components are transformed into one or more first components.

[0467] Therefore, the encoding device can encode three-dimensional data using the selected encoding method by changing the data into a form corresponding to the selected encoding method. Furthermore, the decoding device can use the second information contained in the bitstream to transform multiple second components into multiple first components.

[0468] For example, the second encoding method is a geometry-based 3D data encoding method (e.g., G-PCC or Draco). For example, the second encoding method is a neural network-based 3D data encoding method (e.g., NNC). For example, the second encoding method is a 3D data encoding method that utilizes AI.

[0469] For example, three-dimensional data is Gaussian data. Therefore, the encoding device can selectively select the encoding method for Gaussian data from multiple encoding methods. This could potentially improve encoding efficiency.

[0470] Figure 83 This is a block diagram of the decoding device 20. For example, the decoding device 20 includes a processor 21 and a memory 22, and the processor 21 uses the memory 22 to perform the above-described processing.

[0471] The above describes the encoding device (three-dimensional data encoding device) and decoding device (three-dimensional data decoding device) of the embodiments and variations of this disclosure, but this disclosure is not limited to these embodiments.

[0472] Furthermore, the processing units included in the encoding and decoding devices of the above embodiments are typically implemented as LSIs (Liquid Crystal Sensors) of integrated circuits. They can be implemented individually on a single chip, or in a manner that includes some or all of them on a single chip.

[0473] Furthermore, integrated circuitry is not limited to LSIs; it can also be achieved through dedicated circuits or general-purpose processors. Alternatively, FPGAs (Field Programmable Gate Arrays) that can be programmed after LSI manufacturing, or reconfigurable processors that can reconfigure the connections and settings of the circuit cells within the LSI, can be used.

[0474] Furthermore, in the above embodiments, each component may be constructed using dedicated hardware, or implemented by executing software programs suitable for each component. Each component may also be implemented by a program execution unit such as a CPU or processor reading and executing software programs recorded on a recording medium such as a hard disk or semiconductor memory.

[0475] In addition, this disclosure can also be implemented as an encoding method (three-dimensional data encoding method) or a decoding method (three-dimensional data decoding method) executed by an encoding device (three-dimensional data encoding device) and a decoding device (three-dimensional data decoding device).

[0476] Furthermore, this disclosure can also be implemented as a program that causes a computer, processor, or device to execute the above-described encoding or decoding methods. Additionally, this disclosure can also be implemented as a bitstream generated by the above-described encoding methods. Furthermore, this disclosure can also be implemented as a recording medium on which the program or the bitstream is recorded. For example, this disclosure can also be implemented as a non-transitory, computer-readable recording medium on which the program or the bitstream is recorded.

[0477] Furthermore, the segmentation of functional blocks in the block diagram is one example. Multiple functional blocks can also be implemented as a single functional block, or a single functional block can be divided into multiple functional blocks, or some functionality can be transferred to other functional blocks. Additionally, the functionality of multiple functional blocks with similar functions can be processed in parallel or time-sharing by a single piece of hardware or software.

[0478] Furthermore, the order of the steps in the execution flowchart is illustrative for the purpose of explaining this disclosure, and may be in a different order than described above. Additionally, some of the steps described above may be executed simultaneously (in parallel) with other steps.

[0479] The above description describes one or more encoding and decoding devices based on embodiments, but this disclosure is not limited to these embodiments. Various modifications conceived by those skilled in the art to these embodiments, and configurations constructed by combining constituent elements from different embodiments, can also be included within the scope of these one or more embodiments, as long as they do not depart from the spirit of this disclosure.

[0480] Industrial applicability This disclosure can be applied to encoding devices and decoding devices.

[0481] Explanation of reference numerals in the attached figures 10 Encoding Device 11, 21 processors 12, 22 memory 20 Decoding devices 101 Three-Dimensional Data Encoding System 102 Three-Dimensional Data Decoding System 103 Sensor Terminal 104 External Connection Part 111 Three-dimensional data generation system 112 Reminder Department 113 Coding Department 114 Reuse Department 115 Input / Output Section 116 Control Department 117 Sensor Information Acquisition Department 118 Three-Dimensional Data Generation Department 121 Sensor Information Acquisition Department 122 Input / Output Section 123 Demultiplexing Department 124 Decoding Department 125 Reminder Department 126 User Interface 127 Control Department 130 First Coding Section 131 Location Information Coding Department 132 Attribute Information Encoding Department 133 Additional Information Coding Department 134 Reuse Department 140 Decoding Department 1 141 Demultiplexing section 142 Location Information Decoding Unit 143 Attribute Information Decoding Department 144 Additional Information Decoding Department 150 Second Coding Section 151 Additional Information Generation Department 152 Position Image Generation Unit 153 Attribute Image Generation Unit 154 Image Encoding Department 155 Additional Information Coding Department 156 Reuse Department 160 Second Decoding Section 161 Demultiplexing Department 162 Image Decoding Department 163 Additional Information Decoding Department 164 Location Information Generation Department 165 Attribute Information Generation Department 171 Octree Coding Department 172 Prediction Tree Coding Department 173 Octree Decoding Department 174 Prediction Tree Decoding Department Octree Spanning Parts 181 and 191 Geometric Information Computing Department 182, 192 183, 193 Encoding Table Selection Section 184 Entropy Coding Unit 194 Entropy Decoding Department 201 LoD Attribute Information Encoding Department 202 Transformation Attribute Information Encoding Unit 203 LoD Attribute Information Decoding Department 204 Transformation Attribute Information Decoding Unit 211 Sorting Department 212 Haar Transformer Quantitative Department 213 214, 222 Inverse Quantization Section 215, 223 Inverse Haar Transform Unit 216, 224 memory 217 Arithmetic Coding Department 221 Arithmetic Decoding Department 231 Data Segmentation Department 232 Coding Department 233 Decoding Department 234 Data Integration Section 241 Coding Department 242 TLV Preservation Department 301 Sensor Information Input Section 302 Three-Dimensional Data Generation Department Rendering Departments 303 and 305 304 Spherical Harmonic Function 311 Coding Department 312 Reuse Department 321 Demultiplexing section 322 Decoding Unit 323 Application Department 324 Input Interface Section Rendering Department 325 326 Reminder Department 330, 400 encoding devices 331 First Pre-processing Department 332, 371, 401 Second Pre-processing Section 333, 372, 402 G-PCC Encoding Section 341 Location Information Coding Department 342 Rotary Encoding Unit 343 Scale Coding Department 344 SH Coefficient Encoding Section 345 Transmittance Encoding Section 346 Metadata Encoding Department 347, 405 Reuse Section 350, 410 decoding devices 351, 373, 412 G-PCC Decoding Unit 352, 374, 415 Second Post-Processing Department 353 Post-processing Department 1 361, 411 Demultiplexing Department 362 Location Information Decoding Unit 363 Rotating Decoding Unit 364-Scale Decoding Unit 365 SH Coefficient Decoding Department 366 Transmittance Decoding Unit 367 Metadata Decoding Department 375 Transformer 376 Mapping Department 403 V3C Coding Department 404 Metadata Encoding Department 413 V3C Decoding Section 414 Metadata Decoding Department 500 encoding device 501 Pre-processing Section 2 502 Geometry-based coding unit 503 Video-based Encoding Department 504 Metadata Encoding Department 505 Reuse Department 510 Decoding Device 511 Demultiplexing Department 512 Geometry-based Decoding Unit 513 Video-based decoding unit 514 Metadata Decoding Department 515 Post-processing Section 2

Claims

1. A three-dimensional data encoding method for encoding three-dimensional data containing one or more first components, wherein, Choose any one of several encoding methods, including a first encoding method and a second encoding method, wherein the first encoding method is a video-based 3D data encoding method, and the second encoding method is different from the first encoding method. When the first encoding method is selected, the first encoding method is used to encode one or more first components, thereby generating first encoded data. When the second encoding method is selected, the second encoding method is used to encode one or more first components, thereby generating second-encoded data. The first metadata, obtained by expanding the metadata of the first encoding method, stores the first information representing the selected encoding method. Generate a bitstream containing the first metadata and either the first encoded data or the second encoded data.

2. The three-dimensional data encoding method according to claim 1, wherein, In the aforementioned three-dimensional data encoding method, further, The one or more first components are transformed into one or more second components corresponding to the selected encoding method. When the first encoding method is selected, the first encoding method is used to encode one or more second components, thereby generating the first encoded data. When the second encoding method is selected, the second encoding method is used to encode one or more second components, thereby generating the second encoded data. The first metadata also includes second information, which represents the correspondence between the one or more first components and the one or more second components.

3. The three-dimensional data encoding method according to claim 1, wherein, The second encoding method is a geometry-based three-dimensional data encoding method.

4. The three-dimensional data encoding method according to claim 1, wherein, The second encoding method is a three-dimensional data encoding method based on neural networks.

5. The three-dimensional data encoding method according to claim 1, wherein, The second encoding method is a three-dimensional data encoding method that uses AI.

6. The three-dimensional data encoding method according to claim 1, wherein, The three-dimensional data is Gaussian data.

7. A three-dimensional data decoding method for decoding three-dimensional data containing one or more first components, wherein, Obtain a bitstream containing first metadata and encoded data. The first metadata is obtained by extending the metadata of a first encoding method, which is a video-based 3D data encoding method. First information is obtained from the first metadata, which represents the encoding method used in the encoding of the more than one first component, including the first encoding method and a second encoding method different from the first encoding method. When the first information represents the first encoding method, the encoded data is decoded using the first decoding method corresponding to the first encoding method, thereby generating the one or more first components. When the first information represents the second encoding method, the encoded data is decoded using the second decoding method corresponding to the second encoding method, thereby generating the one or more first components.

8. The three-dimensional data decoding method according to claim 7, wherein, The encoded data is generated by encoding one or more second components after the transformation of one or more first components. In the aforementioned three-dimensional data decoding method, further, The second information is obtained from the first metadata, and the second information represents the correspondence between the one or more first components and the one or more second components. When the first information represents the first encoding method, the encoded data is decoded using the first decoding method to generate the one or more second components. When the first information represents the second encoding method, the encoded data is decoded using the second decoding method to generate the one or more second components; as well as Using the correspondence represented by the second information, the one or more second components are transformed into the one or more first components.

9. The three-dimensional data decoding method according to claim 7, wherein, The second encoding method is a geometry-based three-dimensional data encoding method.

10. The three-dimensional data decoding method according to claim 7, wherein, The second encoding method is a three-dimensional data encoding method based on neural networks.

11. The three-dimensional data decoding method according to claim 7, wherein, The second encoding method is a three-dimensional data encoding method that uses AI.

12. The three-dimensional data decoding method according to claim 7, wherein, The three-dimensional data is Gaussian data.

13. A three-dimensional data encoding device for encoding three-dimensional data containing one or more first components, wherein, have: Processor; and memory, The processor uses the memory. Choose any one of several encoding methods, including a first encoding method and a second encoding method, wherein the first encoding method is a video-based 3D data encoding method, and the second encoding method is different from the first encoding method. When the first encoding method is selected, the first encoding method is used to encode one or more first components, thereby generating first encoded data. When the second encoding method is selected, the second encoding method is used to encode one or more first components, thereby generating second-encoded data. The first metadata, obtained by expanding the metadata of the first encoding method, stores the first information representing the selected encoding method. Generate a bitstream containing the first metadata and either the first encoded data or the second encoded data.

14. A three-dimensional data decoding apparatus for decoding three-dimensional data containing one or more first components, wherein, have: Processor; and memory, The processor uses the memory. Obtain a bitstream containing first metadata and encoded data. The first metadata is obtained by extending the metadata of a first encoding method, which is a video-based 3D data encoding method. First information is obtained from the first metadata, which represents the encoding method used in the encoding of one or more first components, including the first encoding method and the second encoding method. When the first information represents the first encoding method, the encoded data is decoded using the first decoding method corresponding to the first encoding method, thereby generating the one or more first components. When the first information represents the second encoding method, the encoded data is decoded using the second decoding method corresponding to the second encoding method, thereby generating the one or more first components.

Images