A point cloud data transmission device and a method performed with this transmission device, and a point cloud data receiving device and a method performed with this receiving device.

Abstract

A transmitting device for point cloud data, a method performed by the transmitting device, a receiving device, and a method performed by the receiving device are provided. The method performed by the receiving device for point cloud data according to the present disclosure may include the steps of obtaining a geometry-based point cloud compression (G-PCC) file including the point cloud data, obtaining temporal level number information indicating the number of temporal levels of tracks in the G-PCC file from the G-PCC file, and obtaining sample group information for samples in the track from the G-PCC file based on the temporal level number information, where the sample group information may be obtained based on the temporal level number information indicating that the track has a plurality of temporal levels.

Description

Technical Field

[0001] The present disclosure relates to a method and apparatus for processing point cloud content.

Background Art

[0002] Point cloud content is content represented by a point cloud, which is a set of points (points) belonging to a coordinate system representing a three-dimensional space. Point cloud content can represent three-dimensional media and is used to provide various services such as VR (virtual reality), AR (augmented reality), MR (mixed reality), and autonomous driving services. To represent point cloud content, tens of thousands to hundreds of thousands of point data are required, so a method for efficiently processing a huge amount of point data is required.

Summary of the Invention

Problems to be Solved by the Invention

[0003] The present disclosure provides an apparatus and method for efficiently processing point cloud data. The present disclosure provides a point cloud data processing method and apparatus for solving latency and encoding / decoding complexity.

[0004] Also, the present disclosure provides an apparatus and method for assisting "tele" sample grouping according to the number of temporal levels in a Geometry track.

[0005] Also, the present disclosure proposes an apparatus and method for processing a file storage technique that enables efficient access to a stored G-PCC bitstream.

[0006] The technical problems that this disclosure seeks to solve are not limited to those described above, and other technical problems not mentioned above will be clearly understood by a person with ordinary skill in the art to which this disclosure pertains from the following description. [Means for solving the problem]

[0007] A method performed by a point cloud data receiving device according to one embodiment of the present disclosure may include the steps of: acquiring a G-PCC (Geometry-based point cloud compression) file containing the point cloud data; acquiring temporal level count information from the G-PCC file indicating the number of temporal levels of tracks in the G-PCC file; and acquiring sample group information for samples in the tracks from the G-PCC file based on the temporal level count information, wherein the sample group information is acquired based on the temporal level count information indicating that there are multiple temporal levels in the tracks.

[0008] A method used in a point cloud data transmission device according to another embodiment of the present disclosure may include the steps of: generating temporal level count information indicating the number of temporal levels in a G-PCC (geometry-based point cloud compression) file; generating sample group information which is information that groups samples in the track based on the temporal level count; and generating the G-PCC file containing the temporal level count information, wherein the sample group information is generated based on the fact that there are multiple temporal levels.

[0009] A point cloud data receiving device according to another embodiment of the present disclosure may include a memory and at least one processor, the at least one processor which acquires a G-PCC (geometry-based point cloud compression) file containing the point cloud data, acquires temporal level count information from the G-PCC file indicating the number of temporal levels of tracks in the G-PCC file, and acquires sample group information for samples in the tracks from the G-PCC file based on the temporal level count information, wherein the sample group information is acquired based on the temporal level count information indicating that there are multiple temporal levels in the tracks.

[0010] A transmission device for point cloud data according to another embodiment of the present disclosure includes a memory and at least one processor, the at least one processor which generates temporal level count information indicating the number of temporal levels of tracks in a G-PCC (geometry-based point cloud compression) file, generates sample group information which is information that groups samples in the tracks based on the temporal level count information, and generates the G-PCC file containing the temporal level count information, wherein the sample group information is generated based on the fact that there are multiple temporal levels. [Effects of the Invention]

[0011] The apparatus and method according to the embodiments of this disclosure can process point cloud data with high efficiency.

[0012] The apparatus and method according to the embodiments of this disclosure can provide a high-quality point cloud service.

[0013] The apparatus and method according to the embodiments of this disclosure can provide point cloud content for providing general-purpose services such as VR services and autonomous driving services.

[0014] The apparatus and method according to the embodiments of this disclosure can provide time scalability that enables effective access to desired components among the G-PCC components.

[0015] The apparatus and method according to the embodiments of this disclosure can save bits by performing "tele" sample grouping when there are tracks with multiple temporal levels.

[0016] The apparatus and method according to the embodiments of this disclosure can save bits by performing “tele” sample grouping when there are geometry tracks having multiple time levels. [Brief explanation of the drawing]

[0017] [Figure 1] This is a block diagram illustrating an example of a point cloud content provision system according to an embodiment of the present disclosure. [Figure 2] A block diagram illustrating the process of providing point cloud content according to the embodiments of this disclosure. [Figure 3] An example of a point cloud coding device according to an embodiment of this disclosure is shown. [Figure 4] This is a block diagram illustrating an example of a point cloud decryption device according to an embodiment of the present disclosure. [Figure 5] This block diagram shows other examples of point cloud decryption devices according to embodiments of the present disclosure. [Figure 6] This disclosure provides an example of a structure that can be linked with a point cloud data transmission / reception method / device according to an embodiment of this disclosure. [Figure 7] This is a block diagram illustrating another example of a transmission device according to an embodiment of the present disclosure. [Figure 8] This block diagram shows another example of a receiving device according to an embodiment of the present disclosure. [Figure 9] Examples of TLV encapsulation structures according to the embodiments of this disclosure are shown. [Figure 10] Illustrations of the TLV encapsulation syntax structure and payload types according to embodiments of the present disclosure are shown. [Figure 11] Illustrations of files including a single track according to embodiments of the present disclosure are shown. [Figure 12] Illustrations of files including multiple tracks according to embodiments of the present disclosure are shown. [Figure 13] Illustrations of sample grouping within a track according to embodiments of the present disclosure are shown. [Figure 14] Illustrations of sample grouping within a track according to embodiments of the present disclosure are shown. [Figure 15] Illustrations of sample grouping within a track according to embodiments of the present disclosure are shown. [Figure 16] Flowcharts for "tele" sample grouping according to embodiments of the present disclosure are shown. [Figure 17] Flowcharts for "tele" sample grouping according to embodiments of the present disclosure are shown. [Figure 18] Illustrations of the structure of a G-PCC file for "tele" sample grouping according to embodiments of the present disclosure are shown. [Figure 19] Flowcharts for data transmission based on how sample grouping is performed according to embodiments of the present disclosure are shown. [Figure 20] Flowcharts for data reception based on how sample grouping is performed according to embodiments of the present disclosure are shown.

Modes for Carrying Out the Invention

[0018] Hereinafter, with reference to the accompanying drawings, embodiments of the present disclosure will be described in detail so that those having ordinary knowledge in the technical field to which the present disclosure pertains can easily implement them. However, the present disclosure can be realized in various different forms and is not limited to the embodiments described herein.

[0019] In describing embodiments of this disclosure, if it is determined that a specific description of a known configuration or function would obscure the gist of this disclosure, such detailed description will be omitted. In the drawings, parts unrelated to the description of this disclosure will be omitted, and similar parts will be denoted by the same reference numerals.

[0020] In this disclosure, when one component is described as being “connected,” “joined,” or “linked” to another component, this can include not only direct connections but also indirect connections where another component exists between them. Furthermore, when one component is described as “containing” or “having” another component, this means, unless otherwise stated to the contrary, that it may include another component rather than excluding it.

[0021] In this disclosure, terms such as "first," "second," etc., are used solely for the purpose of distinguishing one component from another, and do not limit the order or importance of the components unless otherwise specified. Therefore, within the scope of this disclosure, a first component in one embodiment may be called a second component in another embodiment, and similarly, a second component in one embodiment may be called a first component in another embodiment.

[0022] In this disclosure, components that are distinguished from each other are used to clearly describe their respective characteristics and do not necessarily mean that the components are separate. In other words, multiple components may be integrated to constitute a single hardware or software unit, or a single component may be distributed to constitute multiple hardware or software units. Therefore, such integrated or distributed embodiments are also included in the scope of this disclosure, without needing to be specifically mentioned.

[0023] In this disclosure, the components described in various embodiments are not necessarily essential components, and some may be optional components. Therefore, embodiments consisting of a subset of the components described in one embodiment are also included in the scope of this disclosure. Furthermore, embodiments that include additional components in addition to the components described in various embodiments are also included in the scope of this disclosure.

[0024] This disclosure relates to the encoding and decoding of point cloud-related data, and the terms used in this disclosure may have the ordinary meanings common in the art to which this disclosure pertains, unless newly defined in this disclosure.

[0025] In this disclosure, " / " and "," may be interpreted as "and / or." For example, "A / B" and "A, B" may be interpreted as "A and / or B." Also, "A / B / C" and "A, B, C" may mean "at least one of A, B and / or C."

[0026] In this disclosure, “or” may be interpreted as “and / or.” For example, “A or B” may mean 1) “A” only, 2) “B” only, or 3) “A and B.” Alternatively, in this disclosure, “or” may mean “additionally or alternatively.”

[0027] This disclosure relates to the compression of point cloud-related data. Various methods or embodiments of this disclosure can be applied to MPEG (Moving Picture Experts Group) PCC (point cloud compression or point cloud coding) standards (e.g., G-PCC or V-PCC standards) or next-generation video / image coding standards.

[0028] In this disclosure, “point cloud” can mean a collection of points located in three-dimensional space. Furthermore, in this disclosure, “point cloud content” can mean content represented by a point cloud, and can mean “point cloud video / image.” Hereinafter, “point cloud video / image” will be referred to as “point cloud video.” A point cloud video may contain one or more frames, each of which may be a still image or a picture. Therefore, a point cloud video may contain point cloud images / frames / pictures, and may be referred to as any one of “point cloud images,” “point cloud frames,” or “point cloud pictures.”

[0029] In this disclosure, “point cloud data” may mean data or information related to each point within a point cloud. Point cloud data may include geometry and / or attributes. Point cloud data may also include meta data. Point cloud data may be referred to as “point cloud content data” or “point cloud video data,” etc. Point cloud data may also be referred to as “point cloud content,” “point cloud video,” “G-PCC data,” etc.

[0030] In this disclosure, a point cloud object, which corresponds to point cloud data, can be represented in the form of a box based on a coordinate system, and this coordinate system-based box form can be called a bounding box. That is, the bounding box may be a rectangular cuboid capable of containing all the points of the point cloud, or it may be a rectangular cuboid that contains the source point cloud frame.

[0031] In this disclosure, geometry includes the position (or position information) of each point, and this position can be represented by parameters (e.g., x-axis values, y-axis values, and z-axis values) that represent a three-dimensional coordinate system (e.g., a coordinate system consisting of x, y, and z axes). Geometry may be referred to as "geometry information".

[0032] In this disclosure, an attribute may include the attributes of each point, which may include one or more of the following: texture information, hue (RGB or YCbCr), reflectance (r), transparency, etc. Attributes may be referred to as "attribute information." Metadata may include various data related to the acquisition process described later.

[0033] Overview of the Point Cloud Content Provisioning System

[0034] Figure 1 shows an example of a system for providing point cloud content according to an embodiment of this disclosure (hereinafter referred to as the "point cloud content provision system"). Figure 2 shows an example of the process by which the point cloud content provision system provides point cloud content.

[0035] As shown in Figure 1, the point cloud content provision system may include a transmission device 10 and a reception device 20. The point cloud content provision system can perform the acquisition process (S20), encoding process (S21), transmission process (S22), decoding process (S23), rendering process (S24), and / or feedback process (S25) shown in Figure 2 through the operation of the transmission device 10 and the reception device 20.

[0036] The transmission device 10 can acquire point cloud data and output a bitstream after a series of processes (e.g., encoding process) on the acquired point cloud data (original point cloud data) in order to provide point cloud content. Here, the point cloud data can be output in bitstream format after the encoding process. Depending on the embodiment, the transmission device 10 can transmit the output bitstream to the receiving device 20 via a digital storage medium or network in file or streaming (streaming segment) format. The digital storage medium can include a variety of storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The receiving device 20 can process the received data (e.g., encoded point cloud data) back into the original point cloud data (e.g., decode or restore) and render it. Through these processes, point cloud content can be provided to the user, and this disclosure can provide a variety of embodiments necessary to effectively carry out these processes.

[0037] As shown in Figure 1, the transmission device 10 may include an acquisition unit 11, an encoding unit 12, an encapsulation processing unit 13, and a transmission unit 14, and the receiving device 20 may include a receiving unit 21, a decapsulation processing unit 22, a decoding unit 23, and a rendering unit 24.

[0038] The acquisition unit 11 can perform the process of acquiring point cloud video (S20) through capture, synthesis, or generation processes. Therefore, the acquisition unit 11 is also referred to as the "point cloud video acquisition unit."

[0039] The acquisition process (S20) can generate point cloud data (geometry and / or attributes, etc.) for a large number of points. The acquisition process (S20) can also generate metadata related to the acquisition of point cloud video. Furthermore, the acquisition process (S20) can generate mesh data (e.g., triangular shape data) indicating connectivity information between point clouds.

[0040] Metadata can include initial viewing orientation metadata. Initial viewing orientation metadata can indicate whether the point cloud data is forward-facing or backward-facing. Metadata is sometimes referred to as "auxiliary data," which is metadata for a point cloud.

[0041] The acquired point cloud video may contain PLY (polygon file format or the Stanford triangle format) files. Since a point cloud video has one or more frames, the acquired point cloud video may contain one or more PLY files. A PLY file can contain point cloud data for each point.

[0042] To acquire point cloud video (or point cloud data), the acquisition unit 11 can be configured as a combination of a camera capable of acquiring depth (depth information) and an RGB camera capable of extracting hue information corresponding to the depth information. Here, the camera capable of acquiring depth information may be a combination of an infrared pattern projector and an infrared camera. Alternatively, the acquisition unit 11 may be configured as a LiDAR, but a radar system can be used that measures the position coordinates of a reflector by irradiating it with a LiDAR laser pulse and measuring the time it takes for the reflected pulse to return.

[0043] The acquisition unit 110 can extract the shape of the geometry consisting of points in three-dimensional space from depth information, and extract attributes that represent the hue and reflection of each point from RGB information.

[0044] Possible methods for extracting (or capturing, acquiring, etc.) point cloud video (or point cloud data) include inward-facing, which captures the central object, and outward-facing, which captures the external environment.

[0045] The encoding unit 12 can perform an encoding process (S21) that encodes the data generated from the acquisition unit 11 (such as geometry, attributes and / or metadata and / or mesh data) into one or more bitstreams. Therefore, the encoding unit 12 is also referred to as a "point cloud video encoder". The encoding unit 12 can encode the data generated from the acquisition unit 11 serially or in parallel.

[0046] The encoding process (S21) performed by the encoding unit 12 may be geometry-based point cloud compression (G-PCC). The encoding unit 12 can perform a series of procedures such as prediction, transformation, quantization, and entropy coding for compression and coding efficiency.

[0047] The encoded point cloud data can be output in bitstream format. According to the G-PCC procedure, the encoding unit 12 can encode the point cloud data by separating it into geometry and attributes, as described below. In this case, the output bitstream may include a geometry bitstream containing the encoded geometry and an attribute bitstream containing the encoded attributes. Furthermore, the output bitstream may also include one or more of the following: a metadata bitstream containing metadata, an auxiliary bitstream containing auxiliary data, and a mesh data bitstream containing mesh data. The encoding process (S21) will be described in more detail below. The bitstream containing the encoded point cloud data is sometimes called a "point cloud bitstream" or "point cloud video bitstream."

[0048] The encapsulation processing unit 13 can encapsulate one or more bitstreams output from the decoding unit 12 into a file or segment. Therefore, the encapsulation processing unit 13 is sometimes called a "file / segment encapsulation module." Although the diagram shows an example where the encapsulation processing unit 13 consists of a separate component / module in relation to the transmission unit 14, depending on the embodiment, the encapsulation processing unit 13 can also be included in the transmission unit 14.

[0049] The encapsulation processing unit 13 can encapsulate the data in a file format such as ISOBMFF (ISO Base Media File Format), or process it in other forms such as DASH segments. Depending on the embodiment, the encapsulation processing unit 13 can include metadata in the file format. The metadata can be included, for example, in various levels of boxes in the ISOBMFF file format, or as data in separate tracks within the file. Depending on the embodiment, the encapsulation processing unit 130 can encapsulate the metadata itself in the file. The metadata processed by the encapsulation processing unit 13 may be transmitted from a metadata processing unit or the like, which is not shown in the drawings. The metadata processing unit may be included in the encoding unit 12, or it may be composed of a separate component / module.

[0050] The transmission unit 14 can perform a transmission process (S22) in which it applies a file format (transmission processing) to the "encapsulated point cloud bitstream". The transmission unit 14 can transmit the bitstream or a file / segment containing the bitstream to the receiving unit 21 of the receiving device 20 via a digital storage medium or network. Therefore, the transmission unit 14 is also called a "transmitter" or "communication module".

[0051] The transmission unit 14 can process point cloud data according to any transmission protocol. Here, "processing point cloud data according to any transmission protocol" can be "processing for transmission." Processing for transmission may include processing for transmission via a broadcast network or processing for transmission via broadband. Depending on the embodiment, the transmission unit 14 can receive metadata from the metadata processing unit in addition to point cloud data, and apply processing for transmission to the transmitted metadata. Depending on the embodiment, the processing for transmission may also be performed by the transmission processing unit, which may be included in the transmission unit 14 or constitute a separate component / module from the transmission unit 14.

[0052] The receiving unit 21 can receive a bitstream transmitted by the transmission device 10, or a file / segment containing said bitstream. Depending on the transmission channel, the receiving unit 21 can receive the bitstream or a file / segment containing said bitstream via a broadcast network, or it can receive the bitstream or a file / segment containing said bitstream via broadband. Alternatively, the receiving unit 21 can receive the bitstream or a file / segment containing said bitstream via a digital storage medium.

[0053] The receiving unit 21 can process the received bitstream, or the file / segment containing the bitstream, according to the transmission protocol. The receiving unit 21 can perform the reverse process of the transmission process (processing for transmission) corresponding to the processing performed for transmission by the transmission device 10. The receiving unit 21 can transmit encoded point cloud data from the received data to the decapsulation processing unit 22 and metadata to the metadata parsing unit. The metadata may be in the form of a signaling table. Depending on the embodiment, the reverse process of the transmission process can be performed by the receiving unit. The receiving unit, the decapsulation processing unit 22, and the metadata parsing unit can each be included in the receiving unit 21 or consist of components / modules separate from the receiving unit 21.

[0054] The decapsulation processing unit 22 can decapsulate point cloud data in file format (i.e., a bitstream in file format) transmitted from the receiving unit 21 or the receiving processing unit. Therefore, the decapsulation processing unit 22 is also called the "file / segment decapsulation module".

[0055] The decapsulation processing unit 22 can obtain a point cloud bitstream or metadata bitstream by decapsulating the file using ISOBMFF or the like. Depending on the embodiment, the metadata (metadata bitstream) may be included in the point cloud bitstream. The obtained point cloud bitstream can be transmitted to the decoding unit 23, and the obtained metadata bitstream can be transmitted to the metadata processing unit. The metadata processing unit may be included in the decoding unit 23, or it may be composed of a separate component / module. The metadata obtained by the decapsulation processing unit 23 may be in the form of boxes or tracks within the file format. The decapsulation processing unit 23 can also receive metadata necessary for decapsulation from the metadata processing unit if necessary. The metadata can be transmitted to the decoding unit 23 and used in the decoding process (S23), or it can be transmitted to the rendering unit 24 and used in the rendering process (S24).

[0056] The decoding unit 23 receives the bitstream input and performs operations corresponding to the operations of the encoding unit 12, thereby enabling a decoding process (S23) to decode the point cloud bitstream (encoded point cloud data). Therefore, the decoding unit 23 is also called a "point cloud video decoder".

[0057] The decoding unit 23 can decode point cloud data by separating it into geometry and attributes. For example, the decoding unit 23 can reconstruct (decode) geometry from the geometry bitstream included in the point cloud bitstream, and can reconstruct (decode) attributes based on the attribute bitstream included in the point cloud bitstream and the reconstructed geometry. A three-dimensional point cloud video / image can be reconstructed based on the position information from the reconstructed geometry and the attributes (such as color or texture) from the decoded attributes. The decoding process (S23) will be described in more detail below.

[0058] The rendering unit 24 can perform a rendering process (S24) to render the restored point cloud video. Therefore, the rendering unit 24 is also called a "renderer".

[0059] The rendering process (S24) can mean the process of rendering and displaying point cloud content in 3D space. The rendering process (S24) can be performed using a desired rendering method based on the point position information and attribute information decoded via the decoding process.

[0060] The feedback process (S25) may include a process of transmitting various feedback information obtained during the rendering process (S24) or the display process to the transmission device 10 or to other components within the receiving device 20. The feedback process (S25) may be performed by one or more of the components included in the receiving device 20 in Figure 1, or by one or more of the components shown in Figures 9 and 10. Depending on the embodiment, the feedback process (S25) may be performed by a "feedback unit" or a "sensing / tracking unit".

[0061] Overview of Point Cloud Coding Device

[0062] Figure 3 shows an example of a point cloud coding device 300 according to an embodiment of the present disclosure. The point cloud coding device 300 in Figure 3 can correspond to the coding unit 12 in Figure 1 in terms of configuration and function.

[0063] As shown in Figure 3, the point cloud coding device 300 may include a coordinate system transformation unit 305, a geometry quantization unit 310, an octree analysis unit 315, an approximation unit 320, a geometry encoding unit 325, a reconstruction unit 330, an attribute transformation unit 340, an RAHT transformation unit 345, an LOD generation unit 350, a lift unit 355, an attribute quantization unit 360, an attribute encoding unit 365, and / or a color transformation unit 335.

[0064] The point cloud data acquired by the acquisition unit 11 can undergo a process to adjust the quality of the point cloud content (e.g., lossless, lossy, near-lossless) depending on the network conditions or application. While it is also possible to transmit each point of the acquired point cloud content without loss, in such cases, the large size of the point cloud content may make real-time streaming impossible. Therefore, a process is necessary to reconfigure the point cloud content to match the maximum target bitrate for smooth delivery of the point cloud content.

[0065] The process for adjusting the quality of point cloud content may involve reconstructing and encoding the point's location information (position information included in geometry information) or hue information (color information included in attribute information). The process of reconstructing and encoding the point's position information is sometimes called geometry coding, and the process of reconstructing and encoding the attribute information associated with each point is sometimes called attribute coding.

[0066] Geometry coding may include a geometry quantization process, a voxelization process, an octree analysis process, an approximation process, a geometry encoding process, and / or a coordinate system transformation process. Geometry coding may further include a geometry reconstruction process. Attribute coding may include a hue transformation process, an attribute transformation process, a predictive transformation process, a lift transformation process, a RAHT transformation process, an attribute quantization process, an attribute encoding process, and the like.

[0067] Geometry coding

[0068] The coordinate system transformation process can refer to the process of transforming the coordinate system for the position of a point. Therefore, the coordinate system transformation process is sometimes referred to as "transform coordinates". The coordinate system transformation process can be performed by the coordinate system transformation unit 305. For example, the coordinate system transformation unit 305 can transform the position of a point from a global spatial coordinate system to position information in three-dimensional space (for example, three-dimensional space represented by an X-axis, Y-axis, and Z-axis coordinate system). The position information in three-dimensional space according to the embodiment is sometimes referred to as "geometry information".

[0069] The geometry quantization process can be described as the process of quantizing the position information of points, and can also be performed by the geometry quantization unit 310. For example, the geometry quantization unit 310 can search for the position information with the smallest (x, y, z) value among the point position information and subtract the position information with the smallest (x, y, z) value from the position information of each point. Furthermore, the geometry quantization unit 310 can perform the quantization process by multiplying the subtracted value by a preset quantum scale value and then adjusting the result to the nearest integer value (by lowering or raising it).

[0070] The voxelization process can be described as the process of matching quantized geometric information obtained through the quantization process to specific voxels existing in three-dimensional space. The voxelization process can also be performed by the geometry quantization unit 310. The geometry quantization unit 310 can perform octree-based voxelization based on the position information of the points in order to reconstruct each point to which the quantization process has been applied.

[0071] The geometry encoding process can correspond to the process of performing entropy coding on the occupied code. The geometry encoding process can be performed by the geometry encoding unit 325. The geometry encoding unit 325 can perform entropy coding on the occupied code. The generated occupied code may be encoded immediately, or it may be encoded via intra and intercoding processes to improve compression efficiency. The receiving device 20 can reconstruct the octree via the occupied code.

[0072] On the other hand, in the case of specific regions with no points or very few points, voxelizing the entire region may be inefficient. In other words, since there are almost no points in a particular region, it may not be necessary to construct an octree of the whole. For such cases, an early termination strategy may be necessary.

[0073] The point cloud coding device 300, for a specific region (a specific region that does not correspond to a leaf node), can either directly transmit the position of points only to that specific region, instead of dividing the node corresponding to that specific region (specific node) into eight subnodes (child nodes), or it can reconstruct the position of points within the specific region based on voxels using a surface model.

[0074] A mode in which the location of each point is transmitted directly to a specific node may be called direct mode. The point cloud coding device 300 can check whether the conditions for enabling direct mode are met.

[0075] The conditions for enabling direct mode may include: 1) the direct mode option must be activated; 2) the specific node must not be a leaf node; 3) there must be points below a threshold within the specific node; and 4) the total number of points to be transmitted directly must not exceed the threshold.

[0076] The point cloud encoding device 300 can transmit the position values ​​of points directly to the specific node via the geometry encoding unit 325 using entropy encoding if all of these conditions are met.

[0077] The mode in which the positions of points within a specific region are reconstructed based on voxels using a surface model may be the trisoup mode. The trisoup mode may be performed by the approximation unit 320. The approximation unit 320 defines a specific level of the octree, and from the defined specific level, it can reconstruct the positions of points within the node region based on voxels using a surface model.

[0078] The point cloud coding device 300 can also selectively apply the trisoup mode. Specifically, when using the trisoup mode, the point cloud coding device 300 can specify the level (specific level) to which the trisoup mode should be applied. For example, if the specified specific level is the same as the depth (d) of the octree, the trisoup mode does not need to be applied. In other words, the specified specific level must be smaller than the depth value of the octree.

[0079] A three-dimensional cubic region of a node at a specified level is called a block, and a single block can contain one or more voxels. A block or voxel can also correspond to a brick. Each block may have 12 edges, and the approximation unit 320 can check whether each edge is adjacent to an occupied voxel that has a point. Each edge can be adjacent to multiple occupied voxels. A specific position of an edge adjacent to a voxel is called a vertex, and if multiple occupied voxels are adjacent to a single edge, the approximation unit 320 can determine the average position of those positions as the vertex.

[0080] The point cloud encoding device 300 can entropy encode the starting point (x, y, z) of an edge, the direction vector of the edge (△x, △y, △z), and the position value of the vertex (relative position value within the edge) via the geometry encoding unit 325, if a vertex exists.

[0081] The geometry reconstruction process can correspond to the process of reconstructing the octree and / or approximated octree to generate the reconstructed geometry. The geometry reconstruction process can be performed by the reconstruction unit 330. The reconstruction unit 330 can perform the geometry reconstruction process through processes such as triangle reconstruction, upsampling, and voxelization.

[0082] When the trisoup mode is applied to the approximation unit 320, the reconstruction unit 330 can reconstruct the triangle based on the starting point of the edge, the direction vector of the edge, and the position values ​​of the vertices.

[0083] The reconstruction unit 330 can perform an upsampling process to add intermediate points along the edges of the triangle and voxelize them. The reconstruction unit 330 can generate additional points based on the upsampling factor and the width of the block. These points are sometimes called refined vertices. The reconstruction unit 330 can voxelize the refined vertices, and the point cloud coding device 300 can perform attribute coding based on the voxelized position values.

[0084] In some embodiments, the geometry encoding unit 325 can improve compression efficiency by applying context-adaptive arithmetic coding. The geometry encoding unit 325 can immediately entropy code the occupied code using arithmetic code. In some embodiments, the geometry encoding unit 325 can perform adaptive encoding based on the occupation of surrounding neighboring nodes (intra-coding), or adaptive encoding based on the occupied code of a previous frame (inter-coding). Here, a frame can mean a set of point cloud data generated at the same time. Intra-coding and inter-coding are optional processes and are therefore optional.

[0085] Attribute coding

[0086] Attribute coding can be described as the process of coding attribute information based on the restored (reconstructed) geometry and the geometry before coordinate system transformation (original geometry). Since attributes can be dependent on geometry, the restored geometry can be utilized in attribute coding.

[0087] As mentioned earlier, attributes can include hue, reflectance, and other properties. The same attribute coding method can be applied to the information or parameters contained within an attribute. Hue has three elements, reflectance has one element, and each element can be processed independently.

[0088] Attribute coding may include processes such as hue conversion, attribute conversion, predictive conversion, lift conversion, RAHT conversion, attribute quantization, and attribute encoding. Predictive conversion, lift conversion, and RAHT conversion may be used selectively, or one or more combinations may be used.

[0089] The hue conversion process can correspond to the process of converting the hue format within an attribute to another format. The hue conversion process can be performed by the color conversion unit 335. That is, the color conversion unit 335 can convert the hue within an attribute. For example, the color conversion unit 335 can perform coding work to convert the hue within an attribute from RGB to YCbCr. In this embodiment, the operation of the color conversion unit 335, i.e., the hue conversion process, can be applied selectively (optionally) depending on the hue value contained in the attribute.

[0090] As mentioned above, when a voxel contains one or more points, the position value of the points within the voxel can be set to the voxel's center point in order to integrate and represent them as a single point for that voxel. This may necessitate a process of converting the attribute values ​​associated with those points. Furthermore, this attribute conversion process can also be performed when trisoup mode is activated.

[0091] The attribute conversion process can correspond to a process of converting attributes based on positions where geometry coding has not been performed and / or reconstructed geometry. For example, the attribute conversion process can correspond to a process of converting the attributes of a point at a given position based on the position of that point contained in a voxel. The attribute conversion process can be performed by the attribute conversion unit 340.

[0092] The attribute conversion unit 340 can calculate the average value of the central position value of a voxel and the attribute values ​​of adjacent points (adjacent points) within a specific radius. Alternatively, the attribute conversion unit 340 can apply a weight based on the distance from the central position to the attribute value and calculate the average value of the weighted attribute values. In this case, each voxel has a position and a calculated attribute value.

[0093] The predictive transformation process can be described as a process that predicts the attribute value of the current point (the point that is the target of the prediction) based on the attribute values ​​of one or more points adjacent to the current point (adjacent points). The predictive transformation process can be performed by the LOD (level of detail) generation unit 350.

[0094] The predictive transformation is a method to which the LOD transformation technique is applied, and the LOD generation unit 350 can calculate and set the LOD value of each point based on the LOD distance value of each point.

[0095] The LOD generation unit 350 can generate a predictor for each point for predictive transformation. Therefore, if there are N points, N predictors can be generated. The predictors can be set by calculating a weight value (=1 / distance) based on the LOD value for each point, indexing information for adjacent points, and the distance value to adjacent points. Here, adjacent points may be points that are within a distance set separately for each LOD from the current point.

[0096] Furthermore, the predictor can multiply the attribute values ​​of adjacent points by a "set weight," and set the average of these weighted attribute values ​​as the predicted attribute value for the current point. The attribute quantization process can then be performed on the residual attribute value obtained by subtracting the predicted attribute value for the current point from the attribute value of the current point.

[0097] The lift transformation process, like the predictive transformation process, can correspond to a process of reconstructing points into a set of detail levels via the LOD generation process. The lift transformation process can be performed by the lift unit 355. The lift transformation process may also include a process of generating a predictor for each point, a process of setting the calculated LOD in the predictor, a process of registering adjacent points, and a process of setting weights based on the distance between the current point and adjacent points.

[0098] The RAHT transformation process can be described as a method for predicting the attribute information of higher-level nodes using attribute information associated with lower-level nodes in an octvine. In other words, the RAHT transformation process can be described as an attribute information intracoding method using an octvine backward scan. The RAHT transformation process can be performed by the RAHT transformation unit 345.

[0099] The RAHT conversion unit 345 can perform the RAHT conversion process up to the root node by scanning the entire area with voxels and merging (combining) the voxels into larger blocks at each step. Since the RAHT conversion unit 345 performs the RAHT conversion process only for occupied nodes, in the case of an empty node that is not occupied, the RAHT conversion process can be performed on the node directly above it at a higher level.

[0100] The attribute quantization process can correspond to the process of quantizing the attributes output from the RAHT conversion unit 345, the LOD generation unit 350, and / or the lift unit 355. The attribute quantization process can be performed by the attribute quantization unit 360. The attribute encoding process can correspond to the process of encoding the quantized attributes and outputting an attribute bitstream. The attribute encoding process can be performed by the attribute encoding unit 365.

[0101] Overview of Point Cloud Decryption Device

[0102] Figure 4 shows an example of a point cloud decoding device 400 according to one embodiment of the present disclosure. The point cloud decoding device 400 in Figure 4 can correspond to the decoding unit 23 in Figure 1 in terms of configuration and function.

[0103] The point cloud decoding device 400 can perform a decoding process based on the data (bitstream) transmitted from the transmission device 10. The decoding process may include a process of restoring (decoding) the point cloud video by performing operations corresponding to the encoding operations described above on the bitstream.

[0104] As shown in Figure 4, the decoding process can include a geometry decoding process and an attribute decoding process. The geometry decoding process can be performed by the geometry decoding unit 410, and the attribute decoding process can be performed by the attribute decoding unit 420. That is, the point cloud decoding device 400 can include a geometry decoding unit 410 and an attribute decoding unit 420.

[0105] The geometry decoding unit 410 can reconstruct geometry from the geometry bitstream, and the attribute decoding unit 420 can reconstruct attributes based on the reconstructed geometry and attribute bitstream. Furthermore, the point cloud decoding device 400 can reconstruct a three-dimensional point cloud video (point cloud data) based on the position information from the reconstructed geometry and the attribute information from the reconstructed attributes.

[0106] Figure 5 shows a specific example of a point cloud decoding device 500 according to another embodiment of the present disclosure. As shown in Figure 5, the point cloud decoding device 500 may include a geometry decoding unit 505, an octree synthesis unit 510, an approximation synthesis unit 515, a geometry reconstruction unit 520, a coordinate system inverse transformation unit 525, an attribute decoding unit 530, an attribute inverse quantization unit 535, a RATH transformation unit 550, an LOD generation unit 540, an inverse lift unit 545 and / or a color inverse transformation unit 555.

[0107] The geometry decoding unit 505, the octree synthesis unit 510, the approximation synthesis unit 515, the geometry reconstruction unit 520, and the coordinate system inverse transformation unit 550 can perform geometry decoding. Geometry decoding can be performed in the reverse process of geometry coding as described in Figures 1 to 3. Geometry decoding can include direct coding and trisoup geometry decoding. Direct coding and trisoup geometry decoding can be applied selectively.

[0108] The geometry decoding unit 505 can decode the received geometry bitstream based on arithmetic coding. The operation of the geometry decoding unit 505 can correspond to the reverse process of the operation performed by the geometry encoding unit 325.

[0109] The octree synthesis unit 510 can generate an octree by obtaining an octree code from the decoded geometry bitstream (or from the geometry information secured as a result of decoding). The operation of the octree synthesis unit 510 can correspond to the reverse process of the operation performed by the octree analysis unit 315.

[0110] When trisou geometry encoding is applied, the approximation synthesis unit 515 can synthesize surfaces based on the decoded geometry and / or the generated octree.

[0111] The geometry restoration unit 520 can restore geometry based on the surface and the decoded geometry. When direct coding is applied, the geometry restoration unit 520 can directly retrieve and add the position information of the points to which direct coding has been applied. Also, when trisoup geometry encoding is applied, the geometry restoration unit 520 can restore geometry by performing reconstruction operations, such as triangle reconstruction, upsampling, and voxelization. The restored geometry may include a point cloud picture or frame that does not contain attributes.

[0112] The coordinate system inverse transformation unit 550 can obtain the position of a point by transforming the coordinate system based on the restored geometry. For example, the coordinate system inverse transformation unit 550 can inversely transform the position of a point from three-dimensional space (e.g., three-dimensional space represented by an X-axis, Y-axis, and Z-axis coordinate system) to position information in a global spatial coordinate system.

[0113] The attribute decoding unit 530, attribute inverse quantization unit 535, LOD generation unit 540, and / or inverse lift unit 545 can perform attribute decoding. Attribute decoding can include RAHT transform decoding, predictive transform decoding, and lift transform decoding. The three types of decoding described above may be used selectively, or a combination of one or more decodings may be used.

[0114] The attribute decoding unit 530 can decode the attribute bitstream based on arithmetic coding. For example, if there are no adjacent points in the predictor for each point, and the attribute value of the current point is directly entropy encoded, the attribute decoding unit 530 can decode the attribute value of the current point (the unquantized attribute value). As another example, if there are adjacent points in the predictor for the current point and the quantized residual attribute value is entropy encoded, the attribute decoding unit 530 can decode the quantized residual attribute value.

[0115] The attribute dequantization unit 535 can dequantize the decoded attribute bitstream or the information about the attribute secured as a result of decoding, and output the dequantized attribute (or attribute value). For example, if the attribute decoding unit 530 outputs the quantized residual attribute value, the attribute dequantization unit 535 can dequantize the quantized residual attribute value and output the residual attribute value. The dequantization process can be selectively applied based on the attribute encoding of the point cloud coding device 300. That is, if the attribute value of the current point is directly encoded because there are no adjacent points in the predictor for each point, the attribute decoding unit 530 can output the unquantized attribute value of the current point, and the attribute encoding process can be skipped.

[0116] The RATH transformer 550, the LOD generation unit 540, and / or the inverse lift unit 545 can process the reconstructed geometry and the inversely quantized attributes. The RATH transformer 550, the LOD generation unit 540, and / or the inverse lift unit 545 can selectively perform decoding operations corresponding to the encoding operations of the point cloud encoding device 300.

[0117] The color inverse conversion unit 555 can perform inverse conversion coding to inversely convert the color values ​​(or textures) contained in the decoded attributes. The operation of the color inverse conversion unit 555 can be selectively performed based on the operation of the color conversion unit 335.

[0118] Figure 6 shows an example of a structure that can be linked with a point cloud data transmission / reception method / device according to an embodiment of this disclosure.

[0119] The structure in Figure 6 shows a configuration in which at least one of the following is connected to a cloud network: a server (AI Server), a robot, a self-driving vehicle, an XR device, a smartphone, a home appliance, and / or an HMD. The robot, self-driving vehicle, XR device, smartphone, or home appliance may be referred to as a device. The XR device may correspond to or be linked with a point cloud data device (PCC) according to the embodiment.

[0120] A cloud network can refer to a network that constitutes part of a cloud computing infrastructure or a network that exists within a cloud computing infrastructure. Here, a cloud network can be constructed using 3G networks, 4G or LTE (Long Term Evolution) networks, or 5G networks, etc.

[0121] The server is connected to at least one of the following via a cloud network: a robot, an autonomous vehicle, an XR device, a smartphone, a home appliance, and / or an HMD, and can assist in at least part of the processing of the connected device.

[0122] The HMD can indicate one of the types that can be realized by the XR device and / or PCC device according to the embodiments. The device of the HMD type according to the embodiments can include a communication unit, a control unit, a memory unit, an I / O unit, a sensor unit, a power supply unit, etc.

[0123] <PCC+XR>

[0124] The XR / PCC device may be realized in an HMD, a HUD equipped in a vehicle, a TV, a mobile phone, a smartphone, a computer, a wearable device, a household appliance, a digital signage, a vehicle, a stationary robot, a mobile robot, etc. by applying PCC and / or XR technology.

[0125] The XR / PCC device can obtain information about the surrounding space or real objects by analyzing three-dimensional point cloud data or image data acquired via various sensors or from an external device to generate position (geometry) data and attribute data for three-dimensional points, and render and output an XR object that outputs the information. For example, the XR / PCC device can output an XR object including additional information about the recognized object corresponding to the recognized object.

[0126] <PCC+XR+mobile phone> [[ID=1८]]

[0127] The XR / PCC device can be realized in a mobile phone by applying PCC technology. The mobile phone can decode and display point cloud content based on PCC technology.

[0128] <PCC+autonomous driving+XR>

[0129] Autonomous vehicles can be realized as mobile robots, vehicles, and unmanned aerial vehicles by applying PCC technology and XR technology. An autonomous vehicle to which XR / PCC technology is applied can mean an autonomous vehicle equipped with means for providing XR images, or an autonomous vehicle that is the target of control / interaction within XR images. In particular, an autonomous vehicle that is the target of control / interaction within XR images is distinguished from an XR device and can work in conjunction with it.

[0130] An autonomous vehicle equipped with means for providing XR / PCC images can acquire sensor information from sensors, including cameras, and output XR / PCC images generated based on the acquired sensor information. For example, an autonomous vehicle equipped with a HUD can provide the occupant with XR / PCC objects corresponding to real-world objects or objects on the screen by outputting XR / PCC images.

[0131] In this case, when XR / PCC objects are output to the HUD, at least a portion of the XR / PCC objects can be output so as to overlap with the actual objects that the occupant's gaze is directed towards. Conversely, when XR / PCC objects are output to a display inside the autonomous vehicle, at least a portion of the XR / PCC objects can be output so as to overlap with objects on the screen. For example, an autonomous vehicle can output XR / PCC objects that correspond to objects such as roadways, other vehicles, traffic lights, traffic signs, motorcycles, pedestrians, and buildings.

[0132] The VR, AR, MR, and / or PCC technologies described in this example are applicable to a variety of devices. Specifically, VR technology is a display technology that provides real-world objects and backgrounds solely as computer graphics (CG) images. In contrast, AR technology refers to a technology that displays virtually created CG images alongside images of real objects. Furthermore, MR technology is similar to AR technology in that it mixes and combines virtual objects with the real world. However, while AR technology clearly distinguishes between real objects and virtual objects created from CG images, and uses virtual objects to complement real objects, MR technology distinguishes it from AR technology in that virtual objects are considered to have the same characteristics as real objects. More specifically, for example, a hologram service is an application of the MR technology described above. VR, AR, and MR technologies are sometimes combined and referred to as XR technology.

[0133] space division

[0134] Point cloud data (i.e., G-PCC data) can represent the volumetric encoding of a point cloud consisting of a sequence of frames (point cloud frames). Each point cloud frame can include the number of points, the position of the points, and the attributes of the points. The number of points, the position of the points, and the attributes of the points may differ from frame to frame. Each point cloud frame can represent a set of three-dimensional points specified by their Cartesian coordinates (x, y, z) and zero or more attributes in a particular time instance. Here, the Cartesian coordinates (x, y, z) of the three-dimensional points can be the position or the geometry.

[0135] Depending on the embodiment, the disclosure may further perform a spatial partitioning process in which point cloud data is divided into one or more three-dimensional blocks before encoding the point cloud data. A three-dimensional block may mean all or part of the three-dimensional space occupied by the point cloud data. A three-dimensional block may mean one or more of the following: tile group, tile, slice, coding unit (CU), prediction unit (PU), or transform unit (TU).

[0136] A tile corresponding to a three-dimensional block can represent all or part of the three-dimensional space occupied by point cloud data. Similarly, a slice corresponding to a three-dimensional block can also represent all or part of the three-dimensional space occupied by point cloud data. A tile can be divided into one or more slices based on the number of points contained in a single tile. A tile may be a group of slices having bounding box information. The bounding box information for each tile can be explicitly stated in the tile inventory (or tile parameter set, TPS). Tiles can overlap with other tiles within their bounding boxes. A slice may be a unit of data that is encoded independently, or a unit of data that is decoded independently. In other words, a slice may be a set of points that can be encoded or decoded independently. By embodiment, a slice may be a series of syntax elements that represent part or all of a coded point cloud frame. Each slice may include an index to identify the tile to which the slice belongs.

[0137] The spatially partitioned three-dimensional blocks can be processed independently or non-independently. For example, the spatially partitioned three-dimensional blocks may be encoded or decoded independently or non-independently, and transmitted or received independently or non-independently. Furthermore, the spatially partitioned three-dimensional blocks may be quantized or dequantized independently or non-independently, and transformed or inversely transformed independently or non-independently. Also, the spatially partitioned three-dimensional blocks may be rendered independently or non-independently. For example, encoding or decoding may be performed on a slice-by-slice or tile-by-tile basis. Furthermore, quantization or dequantization may be performed differently on a tile-by-tile or slice-by-slice basis, and transformed or inversely transformed may be performed differently on a tile-by-tile or slice-by-slice basis.

[0138] In this way, by spatially dividing point cloud data into one or more three-dimensional blocks and processing the spatially divided three-dimensional blocks independently or non-independently, the process of processing the three-dimensional blocks can be performed in real time and with low latency. Furthermore, random access to the three-dimensional space occupied by the point cloud data and parallel encoding or decoding become possible, and errors that accumulate during the encoding or decoding process can be prevented.

[0139] Figure 7 is a block diagram illustrating an example of a transmission device 700 performing a spatial division process according to an embodiment of the present disclosure. As shown in Figure 7, the transmission device 700 may include a spatial division unit 705 for performing the spatial division process, a signaling unit 710, a geometry encoder 715, an attribute encoder 720, an encapsulation unit 725, and / or a transmission unit 730.

[0140] The spatial division unit 705 can perform a spatial division process that divides point cloud data into one or more three-dimensional blocks based on bounding boxes and / or subbounding boxes. Through the spatial division process, point cloud data can be divided into one or more tiles and / or one or more slices. In some embodiments, the spatial division process can divide point cloud data into one or more tiles, and each divided tile can be further divided into one or more slices.

[0141] The signaling processing unit 710 can generate and / or process (e.g., entropy encoding) signaling information and output it in bitstream format. Hereinafter, the bitstream output from the signaling processing unit (in which the signaling information is encoded) will be referred to as the "signaling bitstream". The signaling information may include information for or related to spatial partitioning. That is, the signaling information may include information related to the spatial partitioning process performed by the spatial partitioning unit 705.

[0142] When point cloud data is divided into one or more three-dimensional blocks, information may be needed to decode some of the point cloud data corresponding to a specific tile or slice. Furthermore, information related to the three-dimensional spatial region may be needed to support spatial access (or partial access) to the point cloud data. Here, spatial access can mean extracting only the necessary portion of the point cloud data from the file. Signaling information can include information for decoding some of the point cloud data, information related to the three-dimensional spatial region to support spatial access, and so on. For example, signaling information may include three-dimensional bounding box information, three-dimensional spatial region information, tile information, and / or tile inventory information.

[0143] Signaling information can be provided by the spatial division unit 705, the geometry encoder 715, the attribute encoder 720, the transmission processing unit 725, and / or the encapsulation processing unit 730. In addition, the signaling processing unit 710 can provide feedback information fed back from the receiving device 800 in Figure 8 to the spatial division unit 705, the geometry encoder 715, the attribute encoder 720, the transmission processing unit 725, and / or the encapsulation processing unit 730.

[0144] Signaling information can be stored and signaled in samples, sample entries, sample groups, track groups, or separate metadata tracks within a track. Depending on the embodiment, signaling information can be signaled in units such as sequence parameter sets (SPS) for sequence-level signaling, geometry parameter sets (GPS) for signaling geometry coding information, attribute parameter sets (APS) for signaling attribute coding information, and tile parameter sets (TPS) (or tile inventory) for tile-level signaling. Signaling information can also be signaled in units of coding units such as slices or tiles.

[0145] On the other hand, the position (position information) of the three-dimensional block can be output to the geometry encoder 715, and the attributes (attribute information) of the three-dimensional block can be output to the attribute encoder 720.

[0146] The geometry encoder 715 can construct an octree based on position information, encode the constructed octree, and output a geometry bitstream. The geometry encoder 715 can also reconstruct (restore) the octree and / or the approximated octree and output it to the attribute encoder 720. The restored octree may be the restored geometry. The geometry encoder 715 can perform all or part of the operations performed by the coordinate system transformation unit 305, geometry quantization unit 310, octree analysis unit 315, approximation unit 320, geometry encoding unit 325, and / or restoration unit 330 shown in Figure 3.

[0147] The attribute encoder 720 can output an attribute bitstream by encoding the attributes based on the restored geometry. The attribute encoder 720 can perform all or part of the operations performed by the attribute conversion unit 340, RAHT conversion unit 345, LOD generation unit 350, lift unit 355, attribute quantization unit 360, attribute encoding unit 365 and / or color conversion unit 335 shown in Figure 3.

[0148] The encapsulation processing unit 725 can encapsulate one or more input bitstreams into a file or segment. For example, the encapsulation processing unit 725 can encapsulate a geometry bitstream, an attribute bitstream, and a signaling bitstream individually, or it can multiplex and encapsulate a geometry bitstream, an attribute bitstream, and a signaling bitstream. In some embodiments, the encapsulation processing unit 725 can encapsulate a bitstream (G-PCC bitstream) composed of a sequence of TLV (type-length-value) structures into a file. The TLV (or TLV encapsulation) structure constituting the G-PCC bitstream can include a geometry bitstream, an attribute bitstream, a signaling bitstream, etc. In some embodiments, the G-PCC bitstream may be generated by the encapsulation processing unit 725 or by the transmission processing unit 730. The TLV structure or TLV encapsulation structure will be described in detail later. In some embodiments, the encapsulation processing unit 725 can perform all or part of the operations performed by the encapsulation processing unit 13 in Figure 1.

[0149] The transmission processing unit 730 can process encapsulated bitstreams or files / segments according to any transmission protocol. The transmission processing unit 730 can perform all or part of the operations performed by the transmission unit 14 and the transmission processing unit as described with reference to Figure 1.

[0150] Figure 8 is a block diagram illustrating an example of a receiving device 800 according to an embodiment of the present disclosure. The receiving device 800 can perform operations corresponding to the operation of the transmission device 700 that performs spatial partitioning. As shown in Figure 8, the receiving device 800 may include a receiving processing unit 805, a decapsulation processing unit 810, a signaling processing unit 815, a geometry decoder 820, an attribute encoder 825 and / or a post-processing unit 830.

[0151] The receiving processing unit 805 can receive a file / segment containing an encapsulated G-PCC bitstream, a G-PCC bitstream, or a bitstream, and perform processing on these according to the transmission protocol. The receiving processing unit 805 can perform all or part of the operations performed by the receiving unit 21 and the receiving processing unit as described with reference to Figure 1.

[0152] The decapsulation processing unit 810 can obtain a G-PCC bitstream by performing the reverse process of the operations performed by the encapsulation processing unit 725. The decapsulation processing unit 810 can decapsulate a file / segment to obtain a G-PCC bitstream. For example, the decapsulation processing unit 810 can obtain a signaling bitstream and output it to the signaling processing unit 815, obtain a geometry bitstream and output it to the geometry decoder 820, and obtain an attribute bitstream and output it to the attribute decoder 825. The decapsulation processing unit 810 can perform all or part of the operations performed by the decapsulation processing unit 22 in Figure 1.

[0153] The signaling processing unit 815 can parse and decode signaling information by performing the reverse process of the operation performed by the signaling processing unit 710. The signaling processing unit 815 can parse and decode signaling information from the signaling bitstream. The signaling processing unit 815 can provide the decoded signaling information to the geometry decoder 820, the attribute decoder 825, and / or the post-processing unit 830.

[0154] The geometry decoder 820 can reconstruct the geometry from the geometry bitstream by performing the reverse operation of the geometry encoder 715. The geometry decoder 820 can reconstruct the geometry based on signaling information (parameters related to the geometry). The reconstructed geometry can be provided to the attribute decoder 825.

[0155] The attribute decoder 825 can reconstruct attributes from an attribute bitstream by performing the reverse operation of the attribute encoder 720. The attribute decoder 825 can reconstruct attributes based on signaling information (parameters associated with the attribute) and the reconstructed geometry.

[0156] The post-processing unit 830 can reconstruct point cloud data based on the reconstructed geometry and reconstructed attributes. Reconstruction of point cloud data can be performed by matching the reconstructed geometry and reconstructed attributes with each other. In some embodiments, when the reconstructed point cloud data is in tile and / or slice units, the post-processing unit 830 can reconstruct the bounding box of the point cloud data by performing the reverse process of the spatial division process of the transmission device 700 based on signaling information. In some embodiments, when the bounding box has been divided into a plurality of tiles and / or a plurality of slices via the spatial division process, the post-processing unit 830 can also reconstruct a portion of the bounding box by joining some slices and / or some tiles based on signaling information. Here, some slices and / or some tiles used to reconstruct the bounding box may be slices and / or some tiles related to a three-dimensional spatial region for which spatial proximity is desired.

[0157] TLV structure

[0158] As mentioned above, a G-PCC bitstream can refer to a bitstream of point cloud data consisting of a sequence of TLV structures. The TLV structure is also called a "TLV encapsulation structure," a "G-PCC TLV encapsulation structure," or a "G-PCC TLV structure."

[0159] An example of a TLV encapsulation structure is shown in Figure 9, an example of a TLV encapsulation syntax structure is shown in Figure 10a, and an example of a TLV encapsulation structure payload type is shown in Figure 10b. Each TLV encapsulation structure can consist of a TLV type, a TLV length, and / or a TLV payload. The TLV type may be type information of the TLV payload, the TLV length may be length information of the TLV payload, and the TLV payload may be the payload (or payload bytes). Looking at the TLV encapsulation syntax structure (tlv_encapsulation()) exemplified in Figure 10a, tlv_type can indicate the type information of the TLV payload, tlv_num_payload_bytes can indicate the length information of the TLV payload, and tlv_payload_byte[i] can indicate the TLV payload. tlv_payload_byte[i] can be signaled only by the value of tlv_num_payload_bytes, and i can be incremented by 1 from 0 until it becomes (tlv_num_payload_bytes-1).

[0160] A TLV payload can include an SPS, GPS, one or more APS, a tile inventory, a geometry slice, one or more attribute slices, and one or more metadata slices. Depending on the embodiment, the TLV payload of each TLV encapsulation structure may also include one of the SPS, GPS, one or more APS, a tile inventory, a geometry slice, one or more attribute slices, and one or more metadata slices, depending on the type information of the TLV payload. The data contained in the TLV payload can be distinguished via the type information of the TLV payload. For example, as shown in Figure 10b, a value of tlv_type of 0 indicates that the data contained in the TLV payload is an SPS, a value of tlv_type of 1 indicates that the data contained in the TLV payload is a GPS, a value of tlv_type of 2 indicates that the data contained in the TLV payload is a geometry slice, and a value of tlv_type of 3 indicates that the data contained in the TLV payload is an APS. A value of tlv_type of 4 indicates that the data contained in the TLV payload is an attribute slice, a value of tlv_type of 5 indicates that the data contained in the TLV payload is a tile inventory (or tile parameter set), a value of tlv_type of 6 indicates that the data contained in the TLV payload is a frame boundary marker, and a value of tlv_type of 7 indicates that the data contained in the TLV payload is a metadata slice. The payload of the TLV encapsulation structure can conform to the HEVC (High Efficiency Video Coding) NAL (Network Abstraction Layer) unit format.

[0161] Encapsulation / Decapsulation

[0162] Such TLV encapsulation structures can be generated by the transmission unit, transmission processing unit, and encapsulation unit mentioned herein. The G-PCC bitstream composed of the TLV encapsulation structure can be transmitted to the receiving device as is, or it can be transmitted to the receiving device after being encapsulated. For example, the encapsulation processing unit 725 can encapsulate the G-PCC bitstream consisting of the TLV encapsulation structure into a file / segment format and transmit it. The decapsulation processing unit 810 can decapsulate the encapsulated file / segment to obtain the G-PCC bitstream.

[0163] In some embodiments, a G-PCC bitstream can be encapsulated in an ISOBMFF-based file format. In this case, the G-PCC bitstream can be stored in a single track or multiple tracks within an ISOBMFF file. Here, a single track or multiple tracks within a file is also referred to as a "track" or "G-PCC track." An ISOBMFF-based file is also referred to as a container, container file, media file, G-PCC file, etc. Specifically, a file can consist of boxes and / or information that can be referred to as ftyp, moov, mdat, etc.

[0164] The ftyp box (file type box) can provide file type or file compatibility-related information for the file. The receiving device can categorize the file by referring to the ftyp box. The mdat box, also called the media data box, can contain actual media data. Depending on the embodiment, geometry slices (or coded geometry bitstreams) and zero or more attribute slices (or coded attribute bitstreams) may be included in the samples of the mdat box in the file. Here, the samples are sometimes referred to as G-PCC samples. The moov box, also called the movie box, can contain metadata for the media data of the file. For example, the moov box may contain information necessary for decoding and playing the media data, and may contain information about the tracks and samples of the file. The moov box can act as a container for all metadata. The moov box may be the top-level box among the metadata-related boxes.

[0165] Depending on the embodiment, a moov box may include a track (trak) box that provides information related to a file track, and the trak box may include a media (mdia) box that provides media information for the track, and a track reference container (tref) box for linking (reference) the track to a sample of the file corresponding to the track. The media box (MediaBox) may include a media information container (minf) box that provides information about the media data, and a handler (hdlr) box that indicates the type of stream. The minf box may include a sample table (stbl) box that provides metadata related to the sample in the mdat box. The stbl box may include a sample description (stsd) box that provides information about the coding type used and the initialization information required for that coding type. Depending on the embodiment, the sample description (stsd) box may include a sample entry for the track. Depending on the example, signaling information (or metadata) such as SPS, GPS, APS, and tile inventory can be included in the sample entries of the moov box or the mdat box within the file.

[0166] A G-PCC track can be defined as a geometry slice (or coded geometry bitstream) or an attribute slice (or coded attribute bitstream), or a volumetric visual track that carries both a geometry slice and an attribute slice. By example, a volumetric visual track can be identified by a volume visual media handler type 'volv' in a MediaBox's HandlerBox and / or a volumetric visual media header (vvhd) in a MediaBox's minf box. A minf box is also called a media information container or media information box. A minf box can be contained within a MediaBox, a MediaBox can be contained within a trackbox, and a trackbox can be contained within a file's moov box. A single volumetric visual track or multiple volumetric visual tracks can exist in a file.

[0167] Volumetric Visual Media Header Box

[0168] Volumetric visual tracks can use volumetric visual samples for the transmission of actual data. Volumetric visual sample entries may be called sample entries or G-PCC sample entries, and volumetric visual samples may be called samples or G-PCC samples. Single volumetric visual tracks may be called single tracks or G-PCC single tracks, and multiple volumetric visual tracks may be called multiple tracks or multiple G-PCC tracks. Signaling information related to sample grouping, track grouping, single-track encapsulation of G-PCC bitstreams, or multiple-track encapsulation of G-PCC bitstreams, or signaling information to support spatial access, can be added to sample entries in the form of boxes or full boxes. The signaling information may include at least one of the following: GPCCEntryInfoBox, GPCCComponentTypeBox, CubicRegionInfoBox, 3DBoundingBoxInfoBox, or TileInventoryBox.

[0169] GPCC entry information structure

[0170] The syntax structure of the G-PCC entry information box (GPCCEntryInfoBox) can be defined as follows:

[0171] class GPCCEntryInfoBox extends Box('gpsb') {

[0172] GPCCEntryInfoStruct();

[0173] }

[0174] In the syntax structure described above, a GPCCEntryInfoBox with the sample entry type 'gpsb' can contain a GPCCEntryInfoStruct(). The syntax for GPCCEntryInfoStruct() can be defined as follows:

[0175] aligned(8) class GPCCEntryInfoStruct {

[0176] unsigned int(1) main_entry_flag;

[0177] unsigned int(1) dependent_on;

[0178] if(dependent_on){ / / non-entry

[0179] unsigned int(16) dependency_id;

[0180] }

[0181] }

[0182] GPCCEntryInfoStruct() can include main_entry_flag and dependent_on. main_entry_flag can indicate whether it is an entry point for decoding a G-PCC bitstream. dependent_on indicates whether its decoding is dependent on others. If dependent_on is present in a sample entry, it can indicate that the decoding of the sample in the track is dependent on other tracks. If the value of dependent_on is 1, GPCCEntryInfoStruct() may further include dependency_id. dependency_id can indicate the identifier of the track to decode the associated data. If dependency_id is present in a sample entry, dependency_id can represent the identifier of the track carrying the G-PCC sub-bitstream to which the decoding of the sample in the track is dependent. If a dependency_id exists in the sample group, the dependency_id can represent the identifier of the sample that carries the G-PCC sub-bitstream to which the decoding of the associated sample depends.

[0183] Structure of G-PCC component information

[0184] The syntax structure for a G-PCC component type box (GPCCComponentTypeBox) can be defined as follows:

[0185] aligned(8) class GPCCComponentTypeBox extends FullBox('gtyp', version=0,0){

[0186] GPCCComponentTypeStruct();

[0187] }

[0188] A GPCCComponentTypeBox with the sample entry type 'gtyp' can contain a GPCCComponentTypeStruct(). The syntax for GPCCComponentTypeStruct() can be defined as follows:

[0189] aligned(8) class GPCCComponentTypeStruct {

[0190] unsigned int(8) numOfComponents;

[0191] for(i=0;i <numOfComponents;i++) {

[0192] unsigned int(8) gpcc_type;

[0193] if(gpcc_type==4)

[0194] unsigned int(8)AttrIdx;

[0195] }

[0196] / / additional fields

[0197] }

[0198] numOfComponents can indicate the number of G-PCC components signaled to the GPCCComponentTypeStruct. gpcc_type can be included in the GPCCComponentTypeStruct by a loop that repeats as many times as the value of numOfComponents. This loop can repeat, incrementing i by 1 from 0 until it becomes (numOfComponents-1). gpcc_type can indicate the type of G-PCC component. For example, a value of gpcc_type of 2 indicates a geometry component, and a value of 4 indicates an attribute component. If the value of gpcc_type is 4, i.e., an attribute component, the loop may further include AttrIdx. AttrIdx can indicate the identifier of the attribute signaled by SPS(). A G-PCC component type box (GPCCComponentTypeBox) can be included in the sample entry for multiple tracks. If a G-PCC component type box (GPCCComponentTypeBox) exists in a sample entry for a track that carries some or all of the G-PCC bitstream, then a GPCCComponentTypeStruct() can indicate one or more G-PCC component types carried by each track. A GPCCComponentTypeBox or GPCCComponentTypeStruct() containing a GPCCComponentTypeStruct() is sometimes referred to as G-PCC component information.

[0199] Sample group

[0200] The encapsulation processing unit referred to in this disclosure can group one or more samples to generate a sample group. The encapsulation processing unit, metadata processing unit, or signaling processing unit referred to in this disclosure can signal signaling information associated with the sample group to a sample, sample group, or sample entry. That is, sample group information associated with the sample group can be added to a sample, sample group, or sample entry. The sample group information may be 3D bounding box sample group information, 3D region sample group information, 3D tile sample group information, 3D tile inventory sample group information, etc.

[0201] Truck Group

[0202] The encapsulation processing unit referred to in this disclosure can group one or more tracks to generate a track group. The encapsulation processing unit, metadata processing unit, or signaling processing unit referred to in this disclosure can signal signaling information associated with a track group to a sample, track group, or sample entry. That is, track group information associated with a track group can be added to a sample, track group, or sample entry. The track group information may include 3D bounding box track group information, point cloud composition track group information, spatial region track group information, 3D tile track group information, 3D tile inventory track group information, etc.

[0203] Sample entry

[0204] Figure 11 is a diagram illustrating an ISOBMFF base file containing a single track. Figure 11(a) shows an example of the layout of an ISOBMFF base file containing a single track, and Figure 11(b) shows an example of a sample mdat box structure when a G-PCC bitstream is stored in a single track of the file. Figure 12 is a diagram illustrating an ISOBMFF base file containing multiple tracks. Figure 12(a) shows an example of the layout of an ISOBMFF base file containing multiple tracks, and Figure 12(b) shows an example of a sample mdat box structure when a G-PCC bitstream is stored in a single track of the file.

[0205] The stsd box (SampleDescriptionBox) contained within the file's moov box can contain sample entries for a single track that stores a G-PCC bitstream. SPS, GPS, APS, and tile inventory can be included in the sample entries of the moov box or the mdat box within the file. Geometry slices and zero or more attribute slices can also be included in the mdat box within the file. When a G-PCC bitstream is stored in a single track of a file, each sample can contain multiple G-PCC components. That is, each sample can consist of an encapsulation structure of one or more TLVs. A single-track sample entry can be defined as follows:

[0206] Sample Entry Type: 'gpe1','gpeg'

[0207] Container: SampleDescriptionBox

[0208] Mandatory: A'gpe1' or 'gpeg' sample entry is mandatory

[0209] Quantity: One or more sample entries may be present

[0210] The sample entry type 'gpel' or 'gpeg' is mandatory, and there can be one or more sample entries. A G-PCC track can use VolumetricVisualSampleEntry with a sample entry type of 'gpel' or 'gpeg'. Sample entries in a G-PCC track can contain a G-PCC decoder configuration box (GPCCConfigurationBox), and a G-PCC decoder configuration box can contain a G-PCC decoder configuration record (GPCCDecoderConfigurationRecord()). A GPCCDecoderConfigurationRecord() can contain at least one of the following: configurationVersion, profile_idc, profile_compatibility_flags, level_idc, numOfSetupUnitArrays, SetupUnitType, completeness, numOfSepupUnit, or setupUnit. The setupUnit array field contained in a GPCCDecoderConfigurationRecord() can contain a TLV encapsulation structure containing one SPS.

[0211] If the sample entry type is 'gpe1', all parameter sets, such as SPS, GPS, APS, and tile inventory, can be included in the setupUnits array. If the sample entry type is 'gpeg', the above parameter sets can be included in the setupUnits array (i.e., the sample entry) or in the stream (i.e., the sample). An example of the syntax for a G-PCC sample entry (GPCCSampleEntry) with a sample entry type of 'gpe1' is as follows:

[0212] aligned(8) class GPCCSampleEntry()

[0213] extends VolumetricVisualSampleEntry('gpe1'){

[0214] GPCCConfigurationBox config; / / mandatory

[0215] 3DBoundingBoxInfoBox();

[0216] CubicRegionInfoBox();

[0217] TileInventoryBox();

[0218] }

[0219] A G-PCC sample entry (GPCCSampleEntry) with sample entry type 'gpe1' may contain GPCCConfigurationBox, 3DBoundingBoxInfoBox(), CubicRegionInfoBox(), and TileInventoryBox(). 3DBoundingBoxInfoBox() can indicate 3D bounding box information for point cloud data associated with the sample carried in the track. CubicRegionInfoBox() can indicate information for one or more spatial regions of point cloud data carried by the sample in the track. TileInventoryBox() can indicate 3D tile inventory information for point cloud data carried by the sample in the track.

[0220] As shown in Figure 11(b), a sample may include a TLV encapsulation structure containing a geometry slice. A sample may also include a TLV encapsulation structure containing one or more parameter sets. Furthermore, a sample may include a TLV encapsulation structure containing one or more attribute slices.

[0221] As shown in Figure 12(a), when a G-PCC bitstream is carried by multiple tracks in an ISOBMFF-based file, each geometry slice or attribute slice can be mapped to an individual track. For example, a geometry slice can be mapped to track 1, and an attribute slice can be mapped to track 2. The track carrying the geometry slice (track 1) may be called a geometry track or G-PCC geometry track, and the track carrying the attribute slice (track 2) may be called an attribute track or G-PCC attribute track. The geometry track can be defined as a volumetric visual track that carries the geometry slice, and the attribute track can be defined as a volumetric visual track that carries the attribute slice.

[0222] A track that carries a portion of a G-PCC bitstream containing both geometry slices and attribute slices is also called a multiplexed track. When geometry slices and attribute slices are stored on separate tracks, each sample within a track can contain at least one TLV encapsulation structure that carries data for a single G-PCC component. In this case, each sample may not contain both geometry and attributes, nor may it contain multiple attributes. Multitrack encapsulation of a G-PCC bitstream can enable a G-PCC player to effectively access one of the G-PCC components. When a G-PCC bitstream is carried on multiple tracks, the following conditions must be met for a G-PCC player to effectively access one of the G-PCC components:

[0223] a) When a G-PCC bitstream consisting of a TLV encapsulated structure is carried by multiple tracks, the track carrying the geometry bitstream (or geometry slice) becomes the entry point.

[0224] b) In the sample entry, a new box is added to indicate the role of the streams included in the track. The new box may be the aforementioned G-PCC component type box (GPCCComponentTypeBox). That is, a GPCCComponentTypeBox may be included in the sample entry for multiple tracks.

[0225] c) In a track that carries only the G-PCC geometry bitstream, a track reference is introduced to carry the G-PCC attribute bitstream.

[0226] A GPCCComponentTypeBox can contain a GPCCComponentTypeStruct(). If a GPCCComponentTypeBox exists in a sample entry for a track that carries part or all of the G-PCC bitstream, the GPCCComponentTypeStruct() can indicate the type of one or more G-PCC components (e.g., geometry, attribute) carried by each track. For example, a value of 2 in the gpcc_type field contained in the GPCCComponentTypeStruct() indicates a geometry component, and a value of 4 indicates an attribute component. Furthermore, if the value of the gpcc_type field is 4, i.e., an attribute component, it can further contain an AttrIdx field that indicates the identifier of the attribute signaled to SPS().

[0227] When a G-PCC bitstream is carried across multiple tracks, the syntax for a sample entry can be defined as follows:

[0228] Sample Entry Type:'gpe1', 'gpeg', 'gpc1' or 'gpcg'

[0229] Container:SampleDescriptionBox

[0230] Mandatory:'gpc1','gpcg' sample entry is mandatory

[0231] Quantity:One or more sample entries may be present

[0232] Sample entry types 'gpc1', 'gpcg', 'gpc1', or 'gpcg' are required, and one or more sample entries may exist. Multiple tracks (e.g., geometry or attribute tracks) can use VolumetricVisualSampleEntry with sample entry types 'gpc1', 'gpcg', 'gpc1', or 'gpcg'. In a 'gpe1' sample entry, all parameter sets may reside in the setupUnit array. In a 'gpeg' sample entry, parameter sets may reside in the array or stream. In a 'gpe1' or 'gpeg' sample entry, a GPCCComponentTypeBox may be required. In a 'gpc1' sample entry, SPS, GPS, and tile inventory may reside in the SetupUnit array of the track transmitting the G-PCC geometry bitstream. All associated APS may reside in the SetupUnit array of the track transmitting the G-PCC attribute bitstream. In the 'gpcg' sample entry, an SPS, GPS, APS, or tile inventory may be present in the array or stream. In the 'gpc1' or 'gpcg' sample array, a GPCCComponentTypeBox may be present.

[0233] An example of the syntax for a sample entry in G-PCC is as follows:

[0234] aligned(8) class GPCCSampleEntry()

[0235] extends VolumetricVisualSampleEntry(codingname) {

[0236] GPCCConfigurationBox config; / / mandatory

[0237] GPCCComponentTypeBox type; / / optional

[0238] }

[0239] The `compressorname`, or `codingname`, of the base class `VolumetricVisualSampleEntry` can indicate the name of the compressor to be used with the recommended `\013GPCC coding` value. In `\013GPCC coding`, the first byte (13 in octal or 11 in decimal, represented by `\013`) is the number of remaining bytes, which can indicate the number of bytes in the remaining string. `congif` can contain G-PCC decoder configuration information. `info` can represent G-PCC component information carried on each track. `info` can indicate the component tile carried on the track, and can also indicate the attribute name, index, and attribute type of the G-PCC component carried on the G-PCC attribute track.

[0240] Sample format

[0241] When a G-PCC bitstream is saved to a single track, the syntax for the sample format is as follows:

[0242] (8) class GPCCSample

[0243] {

[0244] unsigned int GPCCLength=sample_size; / / Size of Sample

[0245] for(i=0;i <GPCCLength;) / / to end of the sample

[0246] {

[0247] tlv_encapsulation gpcc_unit;

[0248] i+=(1+4)+gpcc_unit.tlv_num_payload_bytes;

[0249] }

[0250] }

[0251] In the syntax described above, each sample (GPCCSample) corresponds to a single point cloud frame and can consist of one or more TLV encapsulation structures belonging to the same presentation time. Each TLV encapsulation structure can contain a single type of TLV payload. In addition, a single sample may be independent (e.g., a sink sample). GPCCLength indicates the length of the sample, and gpcc_unit can contain an instance of a TLV encapsulation structure containing a single G-PCC component (e.g., a geometry slice).

[0252] When a G-PCC bitstream is stored on multiple tracks, each sample may correspond to a single point cloud frame, and samples contributing to the same point cloud frame on different tracks may have the same presentation time. Each sample must consist of one or more G-PCC units of the G-PCC component displayed in the sample entry's GPCCComponentInfoBox, and zero or more G-PCC units carrying either a parameter set or a tile inventory. If a sample contains a G-PCC unit with a parameter set or tile inventory, that F-PCC sample must appear before the G-PCC unit of the G-PCC component. Each sample may contain one or more G-PCC units with attribute data units and zero or more G-PCC units carrying a parameter set. When a G-PCC bitstream is stored on multiple tracks, the syntax and semantics for the sample format may be the same as those for when the G-PCC bitstream is stored on a single track, as described above.

[0253] subsample

[0254] In a receiving device, the geometry slice must be decoded first, and the attribute slice must be decoded based on the decoded geometry. Therefore, if each sample consists of multiple TLV encapsulation structures, each TLV encapsulation structure must be accessed within that sample. Furthermore, if a single sample consists of multiple TLV encapsulation structures, each of the multiple TLV encapsulation structures can be stored as a subsample. Subsamples are sometimes called G-PCC subsamples. For example, if a single sample contains a parameter set TLV encapsulation structure containing a parameter set, a geometry TLV encapsulation structure containing a geometry slice, and an attribute TLV encapsulation structure containing an attribute slice, then the parameter set TLV encapsulation structure, the geometry TLV encapsulation structure, and the attribute TLV encapsulation structure can each be stored as a subsample. In this case, the type of TLV encapsulation structure carried by the subsample may be required to enable access to each G-PCC component within that sample.

[0255] When a G-PCC bitstream is stored on a single track, a G-PCC subsample can contain only one TLV encapsulation structure. One SubSampleInformationBox may exist in the SampleTableBox (stbl) of the moov box, or in the TrackFragmentBox (traf) of each MovieFragmentBox (moof). If a SubSampleInformationBox exists, the 8-bit type value of the TLV encapsulation structure can be included in the 32-bit codec_specific_parameters field of the subsample entry within the SubSampleInformationBox. If the TLV encapsulation structure contains an attribute payload, the 6-bit value of the attribute index can be included in the 32-bit codec_specific_parameters field of the subsample entry within the SubSampleInformationBox. Depending on the implementation, the type of each subsample can be included in the codec_specific_parameters field of the subsample entry within the SubSampleInformationBox. In the example, the type of each subsample can be identified by parsing the codec_specific_parameters field of the subsample entry in SubSampleInformationBox. The codec_specific_parameters of SubSampleInformationBox can be defined as follows:

[0256] if(flags==0) ​​{

[0257] unsigned int(8) PayloadType;

[0258] if(PayloadType==4){ / / attribute payload

[0259] unsigned int(6) AttrIdx;

[0260] bit(18) reserved=0;

[0261] }

[0262] else

[0263] bit(24) reserved=0;

[0264] } else if(flags==1){

[0265] unsigned int(1) tile_data;

[0266] bit(7) reserved=0;

[0267] if (tile_data)

[0268] unsigned int(24) tile_id;

[0269] else

[0270] bit(24) reserved=0;

[0271] }

[0272] In the subsample syntax described above, payloadType can indicate the tlv_type of the TLV encapsulation structure within the subsample. For example, a value of payloadType of 4 indicates an attribute slice (i.e., an attribute slice). attrIdx can indicate the identifier of the attribute information of the TLV encapsulation structure containing the attribute payload within the subsample. attrIdx may be the same as ash_attr_sps_attr_idx of the TLV encapsulation structure containing the attribute payload within the subsample. tile_data can indicate whether the subsample contains one tile or other tiles. A value of tile_data of 1 indicates that the subsample contains a TLV encapsulation structure containing a geometry data unit or attribute data unit corresponding to one G-PCC tile. A value of tile_data of 0 indicates that the subsample contains a TLV encapsulation structure containing each parameter set, tile inventory, or frame boundary marker. The tile_id can indicate the index of the G-PCC version to which the subsample relates within the tile inventory.

[0273] When a G-PCC bitstream is stored on multiple tracks (in the case of multiple-track encapsulation of G-PCC data in ISOBMFF), if subsamples exist, only SubSampleInformationBoxes with a flag of 1 should exist in the SampleTableBox or the TrackFragmentBox of each MovieFragmentBox. When a G-PCC bitstream is stored on multiple tracks, the syntax elements and semantics may be the same as when flag==1 in the case of a G-PCC bitstream stored on a single track.

[0274] References between tracks

[0275] When the G-PCC bitstream is carried on multiple tracks (i.e., when the G-PCC geometry bitstream and the attribute bitstream are carried on different (separated) tracks from each other), a track reference tool can be used to concatenate between tracks. One TrackReferenceTypeBox can be added to the TrackReferenceBox within the TrackBox of the G-PCC track. The TrackReferenceTypeBox can include an array of track_IDs that specify the tracks referenced by the G-PCC track.

[0276] According to an embodiment, the present disclosure can provide an apparatus and method for assisting temporal scalability in the carriage of G-PCC data (hereinafter sometimes referred to as a G-PCC bitstream, an encapsulated G-PCC bitstream, or a G-PCC file). Further, the present disclosure can propose an apparatus and method for point cloud content service provision that efficiently stores a G-PCC bitstream in a single track within a file or divides and stores it in multiple tracks and provides signaling therefor. Further, the present disclosure can propose an apparatus and method for processing a file storage technique to enable efficient access to the stored G-PCC bitstream.

[0277] Temporal scalability

[0278] Temporal scalability can mean the ability to extract one or more subsets of independently coded frames. Alternatively, temporal scalability can mean the ability to divide G-PCC data into multiple distinct temporal levels and process each G-PCC frame belonging to a distinct temporal level independently of the others. With temporal scalability supported, a G-PCC player (or the transmission and / or receiving equipment of this disclosure) can effectively access a desired component (target component) of the G-PCC components. Furthermore, with temporal scalability supported, G-PCC frames are processed independently of each other, allowing for more flexible temporal sub-layering at the system level. Additionally, with temporal scalability supported, the system processing the G-PCC data (point cloud content delivery system) can manipulate the data at a high level to match its network capabilities and decoder capabilities, thereby improving the performance of the point cloud content delivery system.

[0279] Sample grouping

[0280] Two methods can be used to support temporal scalability: sample grouping and track grouping. Sample grouping groups samples within a G-PCC file by temporal level, while track grouping groups tracks within a G-PCC file by temporal level.

[0281] Sample groups can be used to associate samples with the designated temporal levels associated with them. That is, a sample group can indicate which samples belong to which temporal level. Alternatively, a sample group can be information about the results of grouping one or more samples into one or more temporal levels. Sample groups are also called 'tele' sample groups or temporal level sample groups 'tele'.

[0282] Information for the sample group

[0283] Information about a sample group can include information about the results of the sample grouping. Therefore, information about a sample group can be used to relate samples to the temporal levels assigned to them. That is, information about a sample group can indicate which samples belong to which temporal level, and can be information about the results of grouping one or more samples into one or more temporal levels.

[0284] Information about sample groups can reside in the track containing the geometry data unit. When G-PCC data is carried in multiple tracks, information about sample groups can reside only in the geometry track to group each sample in the track to a specified temporal level. Samples in an attribute track can be inferred based on their relationship to the associated geometry track. For example, samples in an attribute track may belong to the same temporal level as samples in the associated geometry track.

[0285] If information for a sample group exists in a G-PCC tile track referenced by a G-PCC tile base track, then information for the sample group may also need to exist in the rest tile tracks referenced by the G-PCC tile base track. Here, a G-PCC tile track can be a volumetric visual track that carries all G-PCC components or a single G-PCC component per G-PCC tile (or one or more G-PCC tiles). A G-PCC tile base track can also be a volumetric visual track that carries all parameter sets and tile inventories corresponding to the G-PCC tile track.

[0286] Information at the temporal level

[0287] Information at the temporal level can be signaled to describe the temporal scalability supported by the G-PCC file. This temporal level information can be found in the sample entries of a track that contains a sample group (or information about a sample group). For example, temporal level information can be found in GPCCDecoderConfigurationRecord() or in the G-PCC ScalabilityInfoBox (GCCScalabilityInfoBox) that signals scalability information for a G-PCC track.

[0288] Temporal level sample grouping

[0289] Temporal-level sample grouping (hereinafter referred to as "tele sample grouping") can mean grouping samples within a track according to their temporal level. Such "tele" sample groups can exist within a track containing geometry data. In this case, "tele" sample grouping can be information about codec-independent sample grouping. Specifically, codec-independent can mean that a sample at one temporal level has no coding dependency on samples at other temporal levels. That is, if a sample at one temporal level has no coding dependency on samples at other temporal levels, a "tele" sample group can exist within a track. Furthermore, a "tele" sample group can contain information about the results of grouping samples within a track (and / or potential trace fragment) according to their temporal level, and / or information about temporal-level identifier values.

[0290] When G-PCC data is carried using multiple temporal level tracks, the "tele" sample group can only exist within the track carrying the geometry data. Samples within the track carrying attribute data can be inferred based on their relationship to the associated geometry track. For example, samples within an attribute track may belong to the same temporal level as samples within their associated geometry track.

[0291] Problems with conventional technology

[0292] The "tele" sample group is only required for tracks having multiple temporal levels. However, the prior art groups and processes samples or signals sample group information even for tracks having only one temporal level. Thus, the prior art may reduce coding efficiency or bit efficiency by performing unnecessary sample grouping or sample group information signaling.

[0293] Examples

[0294] According to an embodiment of the present disclosure, the "tele" sample group can exist in a track having multiple temporal levels. Also, the track having multiple temporal levels can be a track that carries geometry data.

[0295] According to an embodiment, information regarding the number of temporal levels can be defined by syntax elements in boxes that differ from each other depending on the sample entry type. For example, information regarding the number of temporal levels can be defined by syntax elements in a GPCC decoder configuration record (GPCCDecoderConfigrationRecord) and / or syntax elements in a GPCC temporal scalability information box (GPCCScalablityInfoBox).

[0296] For a track including the same sample entry type as "gpe1", "gpeg", "gpc1", or "gpcg", the number of temporal levels in the track can be defined by the num_temporal_levels syntax element of the GPCC decoder configuration record. The GPCC decoder configuration record can provide G-PCC decoder configuration information for geometry-based point cloud content. The syntax structure of the GPCC decoder configuration record can be defined as shown in Table 1 below.

[0297] [Table 1]

[0298] In the syntax structure described above, a GPCC decoder configuration record may include at least one of the following: configurationVersion, profile_idc, profile_compatibility_flags, level_idc, numOfSetupUnitArrays, SetupUnitType, completeness, numOfSepupUnit, or setupUnit. The num_temporal_levels syntax element can represent the maximum number of temporal levels to which G-PCC bitstream frames are grouped.

[0299] For tracks containing the same sample entry type as "gpcb" or "gpeb," the number of temporal levels within the track can be defined by the num_temporal_levels syntax element in the GPCC Time Scalability Information Box. The GPCC Time Scalability Information Box can contain scalability information for G-PCC tracks. The syntax structure of the GPCC Time Scalability Information Box can be defined as shown in Table 2 below.

[0300] [Table 2]

[0301] In the syntax structure described above, `num_temporal_levels` can represent the maximum number of temporal levels to which G-PCC bitstream frames are grouped. `temporal_level_id` can represent the temporal level identifier information for G-PCC samples.

[0302] According to other embodiments of this disclosure, for tracks containing the same sample entry type as “gpe1”, “gpeg”, “gpc1”, or “gpcg”, the number of temporal levels in the track may also be contained in a box other than the GPCC decoder configuration record. For example, the other box may be the GPCC time scalability information box or any box containing G-PCC related time scalability information.

[0303] Alternatively, for tracks containing the same sample entry type as "gpcb" or "gpeb," the number of temporal levels within the track may be contained in a box other than the GPCC temporal scalability information box. For example, the other box could be any box containing G-PCC-related temporal scalability information.

[0304] According to other embodiments of the present disclosure, the number of temporal levels within a track can be contained in a predetermined box, regardless of the sample entry type within the track. For example, the predetermined box may be a GPCC decoder configuration record, a GPCC time scalability information box, or any box containing G-PCC related time scalability information.

[0305] The box containing G-PCC-related time scalability information can be named in various ways. For example, the G-PCC-related time scalability information box is sometimes called GPCCScalabilityInfoBox or GPCCTileScalabilityInfoBox.

[0306] If a "tele" sample group exists within a G-PCC track, samples belonging to the first temporal level identifier can be mapped to the same sample group description index obtained by adding 1 to the first temporal level identifier.

[0307] The sample group description index can mean the index of sample entries describing the samples contained in the sample description (stsd) box. The sample description (stsd) box can contain sample description signals for all temporal levels signaled in the decoder configuration record. Additionally, the sample description (stsd) box can contain sample entries for tracks.

[0308] The temporal level of a sample in an attribute track can be the same as the temporal level of its corresponding sample in the referenced geometry track, i.e., a sample with the same composition timestamp. Therefore, a "tele" sample group can exist in a track that carries only geometry data.

[0309] Figures 13 to 15 illustrate an example of sample grouping within a track according to one embodiment of the present disclosure.

[0310] Referring to Figure 13, each track (Track 1 and Track 2) can contain at least one of the following: GPCC time scalability information ("gsci") box, sample group description ("sgpd") box, or sample group ("sbgp") box.

[0311] The "gsci" box can contain information about the number of temporal levels (num_temporal_levels). The "sgpd" box can contain information about the common characteristics of the samples included in the sample group. The "sbgp" box can contain information about the sample group in which the sample was included.

[0312] Referring to track 1 in Figure 13, the number of temporal levels present in the "gsci" box can be 2. In this case, the two temporal levels can be distinguished by temporal_level_id. The number of entries contained in the "sgpd" box can be 2 (entry_count: 2). The entries present in the "sgpd" box can be represented by SampleGroupDescripitionEntry("tele"), and SampleGroupDescriptionEntry("tele") can mean the "tele" sample group description entry. The "sbgp" box in track 1 can contain the "tele" sample group, and the number of entries can be 12.

[0313] Referring to track 2 in Figure 13, the number of temporal levels present in the "gsci" box of track 2 can be 2. In this case, the two temporal levels can be distinguished by temporal_level_id. The number of entries contained in the "sgpd" box of track 2 can be 4. The entries present in the "sgpd" box can be represented by SampleGroupDescripitionEntry("tele"), and SampleGroupDescriptionEntry("tele") can mean the "tele" sample group description entry. The "sbgp" box of track 2 can contain the "tele" sample group, and the number of entries can be 12.

[0314] The "mdat" box in Figure 13 can contain media data. Specifically, the "mdat" box can contain image frames. Image frames contained within the "mdat" box can have unique identifiers (TemporalId, Tid). For example, the "mdat" box can contain four TemporalIDs (Tid0, Tid1, Tid2, Tid3).

[0315] Referring to Figure 14, which shows an example of sample grouping within a temporal level tile track, track 1 in Figure 14 can include a "gsci" box, and track 2 can include "sgpd" and "sbgp" boxes. Since track 2 is a temporal level tile track that contains samples at all temporal levels, it does not need to include a GPCCTileScalabilityInfoBox ("gtsi" box).

[0316] Referring to Figure 15, the GPCC file structure in Figure 15 can include a tile-based track (Track 1) and two temporal-level tile tracks (Tracks 2 and 3). In each temporal-level tile track, the number of "tele" sample group description entries in the "sgpd" box may be equal to the maximum value of temporal_level_id plus 1 in the "gtsi" box.

[0317] Figures 16 and 17 show flowcharts for grouping "tele" samples according to embodiments of the present disclosure.

[0318] Referring to Figure 16, the transmission device 10 can generate information on the number of temporal levels in a track within the G-PCC file (S1610). This information may indicate the number of temporal levels present in a track. This information can be represented by num_temporal_levels.

[0319] The transmission device 10 can configure at least one sample group based on the number of time levels (or time level number information) (S1620). For example, the transmission device 10 can configure at least one sample group when there are multiple time levels in a track. As another example, the transmission device 10 can configure at least one sample group when there are multiple time levels in a track and the track is a geometry track. The sample group may be a "tele" sample group.

[0320] The transmission device 10 can generate sample group information (S1630). The sample group information may be information about the sample group configured in step S1620. The sample group information may include temporal level identifier information for G-PCC samples. The temporal level identifier information can be represented by temporal_level_id.

[0321] The transmission device 10 can generate a G-PCC file (S1640). The G-PCC file may include at least one of temporal level count information and / or sample group information.

[0322] According to this disclosure, if sample group information exists within a track, samples belonging to a temporal level identifier can be mapped to a sample group description index. In this case, the sample group description index may be the temporal level identifier value plus 1.

[0323] Referring to Figure 17, the receiving device 20 can acquire a G-PCC file (S1710). The G-PCC file may include point cloud data, time-level track count information, and sample group information.

[0324] The receiving device 20 can obtain information on the number of temporal levels of tracks in the G-PCC file from the G-PCC file (S1720). The information on the number of temporal levels may be num_temporal_levels.

[0325] The receiving device 20 can obtain sample group information for samples in a track from the G-PCC file based on the number of temporal levels (S1730). For example, the receiving device 20 can configure at least one sample group if there are multiple temporal levels in a track. As another example, the receiving device 20 can configure at least one sample group if there are multiple temporal levels in a track and the track is a geometry track. The sample group information may include temporal level identifier information for G-PCC samples. Temporal level identifier information can be represented by temporal_level_id.

[0326] Figure 18 shows an example of the structure of a G-PCC file for grouping "tele" samples according to an embodiment of the present disclosure.

[0327] Referring to Figure 18, the moov box 1805 within the G-PCC file 1800 may be a multi-track encapsulation file containing two or more tracks. Here, the two or more tracks present in the moov box 1805 may include geometry tracks and / or attribute tracks.

[0328] According to one embodiment of this disclosure, a "tele" sample group can only exist within a geometry track. Furthermore, a "tele" sample group can only exist within a track if there are multiple temporal levels within the geometry track. In Figure 18, geometry track #1 (1810) has multiple temporal levels, 0, 1, and 2. Therefore, a "tele" sample group can exist within geometry track #1 (1801). However, geometry track #2 (1815) has only one temporal level, 3. Therefore, a "tele" sample group cannot exist within geometry track #2 (1815).

[0329] Figures 19 and 20 are flowcharts for data transmission / reception based on how samples are grouped according to the embodiments of this disclosure.

[0330] Referring to Figure 19, the transmission device 10 can determine whether to group the samples based on the number of temporal levels in the track (S1910). If the number of temporal levels in the track is two or more, the transmission device 10 can group the samples in the track. Conversely, if the number of temporal levels in the track is one, the transmission device 10 does not need to group the samples in the track. If the samples are grouped based on the number of temporal levels in the track being two or more, the transmission device 10 can generate the sample group information (S1920). The sample group information may include temporal level index information.

[0331] The transmission device 10 can encapsulate samples based on the sample group information generated in step S1920 (S1930). In contrast, if the samples in a track are not grouped, the transmission device 10 can encapsulate samples without generating sample group information (S1930).

[0332] Referring to Figure 20, the receiving device 20 can determine whether the number of temporal levels is 2 or more (S2010). The information on the number of temporal levels may be information about the number of temporal levels present in a track. The receiving device 20 can determine that the samples in a track are grouped if the information on the number of temporal levels in a track is 2 or more. Conversely, the receiving device 20 can determine that the samples in a track are not grouped if the information on the number of temporal levels in a track is 1.

[0333] The receiving device 20 can acquire sample group information (S2020). The sample group information may include temporal level index information within the track. Temporal level index information may be temporal_level_id. The receiving device 20 can acquire sample group information based on the number of temporal levels confirmed in step S2010. Specifically, the receiving device 20 can acquire sample group information if the number of temporal levels confirmed in step S2010 is 2 or more.

[0334] The receiving device 20 can decapsulate the sample based on the sample group information acquired in step S2020 (S2030). Conversely, if the number of temporal levels in a track is not 2 or more, the receiving device 20 can decapsulate the sample without checking the sample group information (S2030).

[0335] According to the disclosure described above, when there is only one temporal level in a track, unnecessary "tele" sample groups can be removed by not performing sample grouping. Also, when there is only one temporal level in a track, unnecessary information does not need to be signaled by not signaling sample group information. This can improve the encapsulation / decapsulation efficiency and / or bit efficiency of G-PCC file data.

[0336] The scope of this disclosure includes software or machine-executable instructions (e.g., operating systems, applications, firmware, programs, etc.) that enable operation by various embodiments to be performed on a device or computer, and non-transitory computer-readable medium on which such software or instructions etc. are stored and executable on a device or computer. [Industrial applicability]

[0337] The embodiments described herein can be used for providing point cloud content. Furthermore, the embodiments described herein can be used for encoding / decoding point cloud data.

Claims

1. A method performed by a point cloud data receiving device, The steps include obtaining a G-PCC (geometry-based point cloud compression) file containing the aforementioned point cloud data, A step of obtaining time level count information from the G-PCC file, which indicates the number of time levels in a track within the G-PCC file, wherein the time level is a subset of frames within the G-PCC file. The step of obtaining sample group information of the samples in the track from the G-PCC file based on the aforementioned time-level number information, The aforementioned sample group information is obtained based on the fact that the temporal level count information indicates that there are multiple temporal levels within the track. The method by which the sample group information is obtained is not based on the temporal level count information indicating that there is only one temporal level in the track.

2. The method according to claim 1, wherein the sample group information includes the temporal level identifier information.

3. The method according to claim 1, wherein the track includes geometric information of the point cloud data.

4. The method according to claim 1, wherein, based on the track having a first sample entry type, the number at a temporal level is included in a G-PCC decoder configuration record (GPCCDecoderConfigurationRecord).

5. The method according to claim 4, wherein the first sample entry type is at least one of gpe1 sample entry type, gpeg sample entry type, gpc1 sample entry type, or gpcg sample entry type.

6. The method according to claim 1, wherein, based on the track having a second sample entry type, the number of items at a temporal level is included in the G-PCC time scalability information box (GPCCScalabilityInfoBox).

7. The method according to claim 6, wherein the second sample entry type is at least one of a gpcb sample entry type or a gpeb sample entry type.

8. Based on the presence of the sample group information within the aforementioned track, the samples belonging to the temporal level identifier are mapped to the sample group description index. The method according to claim 1, wherein the sample group description index is the same as the time level identifier plus 1.

9. The method according to claim 4, wherein the sample description for the time level signaled in the G-PCC decoder configuration record is included in a SampleDescriptionBox.

10. A method performed by a point cloud data transmission device, A step of generating temporal level count information indicating the number of temporal levels in a track within a G-PCC (geometry-based point cloud compression) file, wherein the temporal level is a subset of frames within the G-PCC file; A step of generating sample group information, which is information regarding the grouping of samples within the track, based on the aforementioned number at a temporal level, The step of generating the G-PCC file containing the aforementioned time-level number information includes, The aforementioned sample group information is generated based on the existence of multiple temporal levels. The aforementioned sample group information is not generated based on the existence of only one temporal level.

11. A point cloud data receiving device, Memory and Equipped with at least one processor, The at least one processor is Obtain a G-PCC (geometry-based point cloud compression) file containing the aforementioned point cloud data, Time level count information indicating the number of time levels in a track within the G-PCC file is obtained from the G-PCC file, and the time level is a subset of frames within the G-PCC file. Based on the aforementioned temporal-level number information, the system is configured to obtain sample group information of the samples within the track from the G-PCC file. The aforementioned sample group information is obtained based on the fact that the temporal level count information indicates that there are multiple temporal levels within the track. The receiving device does not acquire the sample group information based on the fact that the temporal level count information indicates that there is only one temporal level in the track.

12. A point cloud data transmission device, Memory and Equipped with at least one processor, The at least one processor is It generates temporal level count information indicating the number of temporal levels in a track within a G-PCC (geometry-based point cloud compression) file, where the temporal level is a subset of frames within the G-PCC file. Based on the aforementioned number at a temporal level, sample group information, which is information regarding the grouping of samples within the track, is generated. It is configured to generate the G-PCC file containing the aforementioned temporal-level number information, The aforementioned sample group information is generated based on the existence of multiple temporal levels. The aforementioned sample group information is not generated based on the existence of only one temporal level in the transmission device.

Images