Method performed by point cloud data reception device and method performed by point cloud data transmission device

Abstract

Provided are a method performed by a point cloud data reception device and a method performed by a reception device. The method performed by a point cloud data reception device according to the present disclosure comprises the steps of: identifying one or more tracks from the point cloud data; extracting one or more samples from the tracks; and decoding the samples on the basis of a parameter set, wherein the parameter set may be acquired on the basis of information included in a geometry-based point cloud compression (G-PCC) parameter set sample group entry.

Description

A method performed in a receiving device of point cloud data and a method performed in a transmitting device of point cloud data

[0001] The present disclosure relates to a method and apparatus for processing point cloud content.

[0002] Point cloud content is represented as a point cloud, a collection of points within a coordinate system representing three-dimensional space. Point cloud content can represent three-dimensional media and is used to provide various services such as virtual reality (VR), augmented reality (AR), mixed reality (MR), and autonomous driving services. Since representing point cloud content requires tens to hundreds of thousands of point data, a method for efficiently processing massive amounts of point data is required.

[0003] The present disclosure aims to provide a method and device for efficiently processing point cloud data.

[0004] In addition, the present disclosure aims to provide a method and device for providing a point cloud content service that efficiently stores a G-PCC bitstream in a single track within a file or stores it by dividing it into multiple tracks and provides signaling therefor.

[0005] In addition, the present disclosure aims to provide a method and device for processing a file storage technique to support efficient access to a stored G-PCC bitstream.

[0006] In addition, the present disclosure aims to provide a method and device for dividing and storing a G-PCC bitstream into one or more multiple tracks within a file in addition to the above file storage technique.

[0007] The technical problems to be achieved in the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present disclosure belongs from the description below.

[0008] A method performed in a receiving device of point cloud data according to one embodiment of the present disclosure includes the steps of identifying one or more tracks from the point cloud data, extracting one or more samples from the tracks, and decoding the samples based on a parameter set, wherein the parameter set can be obtained based on information included in a geometry-based point cloud compression (G-PCC) parameter set sample group entry.

[0009] A method performed in a transmission device of point cloud data according to another embodiment of the present disclosure comprises the steps of storing a bitstream including the point cloud data in one or more tracks, and generating a geometry-based point cloud compression (G-PCC) file based on the tracks, wherein the tracks include one or more samples, at least one of the samples includes a parameter set, and the parameter set can be encoded based on information included in a G-PCC parameter set sample group entry.

[0010] According to the present disclosure, a method and device for efficiently processing point cloud data can be provided.

[0011] In addition, according to the present disclosure, a method and device for providing a point cloud content service can be provided that efficiently stores a G-PCC bitstream in a single track within a file or stores it by dividing it into multiple tracks and provides signaling therefor.

[0012] Additionally, according to the present disclosure, a method and device for processing a file storage technique to support efficient access to a stored G-PCC bitstream may be provided.

[0013] In addition, according to the present disclosure, in addition to the above file storage technique, a method and device for dividing and storing a G-PCC bitstream into one or more multiple tracks within a file can be provided.

[0014] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned will be clearly understood by a person having ordinary skill in the art to which the present disclosure pertains from the description below.

[0015] FIG. 1 is a block diagram illustrating an example of a point cloud content provision system according to embodiments of the present disclosure.

[0016] FIG. 2 is a block diagram illustrating an example of a point cloud content provision process according to embodiments of the present disclosure.

[0017] FIG. 3 illustrates an example of a point cloud encoding device according to embodiments of the present disclosure.

[0018] FIG. 4 is a block diagram illustrating an example of a point cloud decoding device according to embodiments of the present disclosure.

[0019] FIG. 5 is a block diagram illustrating another example of a point cloud decoding device according to embodiments of the present disclosure.

[0020] FIG. 6 is a block diagram illustrating another example of a transmission device according to embodiments of the present disclosure.

[0021] FIG. 7 is a block diagram illustrating another example of a receiving device according to embodiments of the present disclosure.

[0022] FIG. 8 illustrates an example of a structure that can be linked with a point cloud data transmission / reception method / device according to embodiments of the present disclosure.

[0023] FIG. 9 is a block diagram illustrating another example of a transmission device according to embodiments of the present disclosure.

[0024] FIG. 10 illustrates an example of spatially dividing a bounding box into three-dimensional blocks according to embodiments of the present disclosure.

[0025] FIG. 11 is a block diagram illustrating another example of a receiving device according to embodiments of the present disclosure.

[0026] FIG. 12 illustrates an example of a file containing a single track according to embodiments of the present disclosure.

[0027] FIG. 13 illustrates an example of a file containing multiple tracks according to embodiments of the present disclosure.

[0028] FIG. 14 is a drawing for explaining a method performed in a receiving device for point cloud data according to the present disclosure.

[0029] FIG. 15 is a drawing for explaining a method performed in a transmission device of point cloud data according to the present disclosure.

[0030] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present disclosure. The present disclosure may be implemented in various different forms and is not limited to the embodiments described herein.

[0031] In describing embodiments of the present disclosure, detailed descriptions of known configurations or functions will be omitted if they are deemed to obscure the gist of the present disclosure. Furthermore, portions unrelated to the description of the present disclosure in the drawings have been omitted, and similar portions have been designated with similar reference numerals.

[0032] In the present disclosure, when a component is said to be "connected," "coupled," or "connected" to another component, this may include not only a direct connection, but also an indirect connection in which another component exists in between. Furthermore, when a component is said to "include" or "have" another component, unless otherwise specifically stated, this does not exclude the other component, but rather implies that the other component may be included.

[0033] In this disclosure, terms such as first, second, etc. are used solely to distinguish one component from another, and do not limit the order or importance of components unless specifically stated otherwise. Accordingly, within the scope of this disclosure, a first component in one embodiment may be referred to as a second component in another embodiment, and similarly, a second component in one embodiment may be referred to as a first component in another embodiment.

[0034] In this disclosure, distinct components are used to clearly illustrate their respective characteristics, and do not necessarily imply that the components are separated. That is, multiple components may be integrated into a single hardware or software unit, or a single component may be distributed into multiple hardware or software units. Therefore, even if not specifically mentioned, such integrated or distributed embodiments are also included within the scope of this disclosure.

[0035] In the present disclosure, the components described in various embodiments are not necessarily essential components, and some may be optional components. Therefore, embodiments comprising a subset of the components described in one embodiment are also within the scope of the present disclosure. Furthermore, embodiments including other components in addition to the components described in various embodiments are also within the scope of the present disclosure.

[0036] The present disclosure relates to encoding and decoding of point cloud-related data, and terms used in the present disclosure may have their usual meanings commonly used in the technical field to which the present disclosure belongs, unless newly defined in the present disclosure.

[0037] In this disclosure, “ / ” and “,” can be interpreted as “and / or.” For example, “A / B” and “A, B” can be interpreted as “A and / or B.” Additionally, “A / B / C” and “A, B, C” can mean “at least one of A, B, and / or C.”

[0038] In this disclosure, “or” may be interpreted as “and / or.” For example, “A or B” may mean 1) “A” only, 2) “B” only, or 3) “A and B.” Alternatively, “or” in this disclosure may mean “additionally or alternatively.”

[0039] The present disclosure relates to compression of point cloud-related data. Various methods or embodiments of the present disclosure can be applied to the MPEG (Moving Picture Experts Group) PCC (point cloud compression or point cloud coding) standard (e.g., G-PCC or V-PCC standard) or the next-generation video / image coding standard.

[0040] In the present disclosure, “point cloud” may refer to a collection of points located in a three-dimensional space. In addition, in the present disclosure, “point cloud content” may refer to content expressed as a point cloud, and may refer to “point cloud video / image.” Hereinafter, “point cloud video / image” will be referred to as “point cloud video.” A point cloud video may include one or more frames, and a frame may be a still image or a picture. Accordingly, a point cloud video may include a point cloud video / frame / picture, and may be referred to as any one of a “point cloud video,” a “point cloud frame,” and a “point cloud picture.”

[0041] In the present disclosure, “point cloud data” may refer to data or information related to each point within a point cloud. Point cloud data may include geometry and / or attributes. In addition, point cloud data may further include meta data. Point cloud data may be referred to as “point cloud content data” or “point cloud video data.” In addition, point cloud data may be referred to as “point cloud content,” “point cloud video,” “G-PCC data,” etc.

[0042] In the present disclosure, a point cloud object corresponding to point cloud data can be represented in the form of a box based on a coordinate system, and the box shape based on this coordinate system can be called a bounding box. That is, the bounding box can be a rectangular cuboid that can contain all points of the point cloud, and can be a rectangular cuboid that includes the original (source) point cloud frame.

[0043] In the present disclosure, geometry includes the position (or position information) of each point, and this position can be expressed as parameters (e.g., x-axis value, y-axis value, and z-axis value) representing a three-dimensional coordinate system (e.g., a coordinate system composed of an x-axis, a y-axis, and a z-axis). Geometry can be referred to as “geometry information.”

[0044] In the present disclosure, attributes may include properties of each point, and these properties may include one or more of texture information, color (RGB or YCbCr), reflectance (r), transparency, etc. of each point. The attributes may be referred to as “attribute information.” Metadata may include various data related to acquisition in the acquisition process described below.

[0045] Overview of the Point Cloud Content Delivery System

[0046] FIG. 1 illustrates an example of a system for providing point cloud content according to embodiments of the present disclosure (hereinafter referred to as a "point cloud content providing system"). FIG. 2 illustrates an example of a process by which the point cloud content providing system provides point cloud content.

[0047] As illustrated in FIG. 1, a point cloud content providing system may include a transmission device (10) and a reception device (20). The point cloud content providing system may perform the acquisition process (S20), encoding process (S21), transmission process (S22), decoding process (S23), rendering process (S24), and / or feedback process (S25) illustrated in FIG. 2 by the operations of the transmission device (10) and the reception device (20).

[0048] The transmission device (10) may acquire point cloud data to provide point cloud content, and output a bitstream through a series of processes (e.g., encoding processes) on the acquired point cloud data (original point cloud data). Here, the point cloud data may be output in the form of a bitstream through an encoding process. According to embodiments, the transmission device (10) may transmit the output bitstream to the reception device (20) through a digital storage medium or a network in the form of a file or streaming (streaming segment). The digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The reception device (20) may process (e.g., decode or restore) the received data (e.g., encoded point cloud data) back into the original point cloud data and render it. Through these processes, point cloud content may be provided to the user, and the present disclosure may provide various embodiments necessary to effectively perform these series of processes.

[0049] As illustrated in FIG. 1, the transmission device (10) may include an acquisition unit (11), an encoding unit (12), an encapsulation processing unit (13), and a transmission unit (14), and the reception device (20) may include a reception unit (21), a decapsulation processing unit (22), a decoding unit (23), and a rendering unit (24).

[0050] The acquisition unit (11) can perform a process (S20) of acquiring a point cloud video through a capture, synthesis, or generation process. Therefore, the acquisition unit (11) can be referred to as a 'point cloud video acquisition unit.'

[0051] Through the acquisition process (S20), point cloud data (geometry and / or attributes, etc.) for a plurality of points can be generated. In addition, through the acquisition process (S20), metadata related to the acquisition of point cloud video can be generated. In addition, through the acquisition process (S20), mesh data (e.g., data in the shape of a triangle) representing connection information between point clouds can also be generated.

[0052] Metadata may include initial viewing orientation metadata. Initial viewing orientation metadata may indicate whether the point cloud data represents the front or back. This metadata may be referred to as "auxiliary data," which is metadata about the point cloud.

[0053] The acquired point cloud video may include a PLY (polygon file format or the Stanford triangle format) file. Since a point cloud video has one or more frames, the acquired point cloud video may include one or more PLY files. A PLY file may contain point cloud data for each point.

[0054] In order to acquire point cloud video (or, point cloud data), the acquisition unit (11) may be configured with a combination of camera equipment capable of acquiring depth (depth information) and RGB cameras capable of extracting color information corresponding to the depth information. Here, the camera equipment capable of acquiring depth information may be a combination of an infrared pattern projector and an infrared camera. In addition, the acquisition unit (11) may be configured with LiDAR, which may utilize a radar system that measures the position coordinates of a reflector by emitting a laser pulse and measuring the time it takes for it to be reflected and return.

[0055] The acquisition unit (110) can extract the shape of a geometry composed of points in a three-dimensional space from depth information, and extract attributes expressing the color or reflection of each point from RGB information.

[0056] There are two ways to extract (or capture, acquire, etc.) point cloud video (or point cloud data): an inward-facing method that captures a central object and an outward-facing method that captures the external environment.

[0057] Meanwhile, when providing point cloud video of a computer-generated virtual space, capture using an actual camera may not be performed. In this case, post-processing may be required to improve the quality of the captured point cloud content. For example, during the acquisition process (S20), the maximum / minimum depth values ​​can be adjusted within the range provided by the camera equipment, but post-processing may also be performed to remove unwanted areas (e.g., background) or point data in unwanted areas, or post-processing may be performed to recognize connected spaces and fill in spatial holes. As another example, post-processing may be performed to integrate point cloud data extracted from cameras that share a spatial coordinate system into a single content by converting each point into a global coordinate system based on the positional coordinates of each camera. This can generate a single, wide-ranging point cloud content, or obtain point cloud content with a high point density.

[0058] The encoding unit (12) can perform an encoding process (S21) of encoding data (geometry, attributes, and / or metadata, and / or mesh data, etc.) generated from the acquisition unit (11) into one or more bitstreams. Accordingly, the encoding unit (12) can be referred to as a 'point cloud video encoder'. The encoding unit (12) can encode data generated from the acquisition unit (11) serially or in parallel.

[0059] The encoding process (S21) performed by the encoding unit (12) may be geometry-based point cloud compression (G-PCC). The encoding unit (12) may perform a series of procedures such as prediction, transformation, quantization, and entropy coding to improve compression and coding efficiency.

[0060] The encoded point cloud data can be output in the form of a bitstream. When based on the G-PCC procedure, the encoding unit (12) can encode the point cloud data by dividing it into geometry and attributes as described below. In this case, the output bitstream can include a geometry bitstream including the encoded geometry and an attribute bitstream including the encoded attributes. In addition, the output bitstream can further include one or more of a metadata bitstream including metadata, an auxiliary bitstream including auxiliary data, and a mesh data bitstream including mesh data. The encoding process (S21) will be described in more detail below. The bitstream including the encoded point cloud data can be referred to as a 'point cloud bitstream' or a 'point cloud video bitstream'.

[0061] The encapsulation processing unit (13) can perform a process of encapsulating one or more bitstreams output from the decryption unit (12) in the form of a file or segment. Therefore, the encapsulation processing unit (13) can be referred to as a 'file / segment encapsulation module'. Although the drawing shows an example in which the encapsulation processing unit (13) is configured as a separate component / module in relation to the transmission unit (14), according to embodiments, the encapsulation processing unit (13) may be included in the transmission unit (14).

[0062] The encapsulation processing unit (13) can encapsulate the corresponding data into a file format such as ISOBMFF (ISO Base Media File Format) or process it in the form of other DASH segments. According to embodiments, the encapsulation processing unit (13) can include metadata in the file format. The metadata can be included, for example, in boxes of various levels in the ISOBMFF file format or as data in a separate track within the file. According to embodiments, the encapsulation processing unit (130) can encapsulate the metadata itself into a file. The metadata processed by the encapsulation processing unit (13) may be received from a metadata processing unit, etc., which is not shown in the drawing. The metadata processing unit may be included in the encoding unit (12) or may be configured as a separate component / module.

[0063] The transmission unit (14) can perform a transmission process (S22) of applying processing (processing for transmission) according to the file format to the 'encapsulated point cloud bitstream'. The transmission unit (140) can transmit the bitstream or a file / segment including the bitstream to the reception unit (21) of the reception device (20) via a digital storage medium or a network. Accordingly, the transmission unit (14) can be referred to as a 'transmitter' or a 'communication module'.

[0064] The transmission unit (14) can process point cloud data according to any transmission protocol. Here, ‘processing point cloud data according to any transmission protocol’ may be ‘processing for transmission.’ Processing for transmission may include processing for transmission via a broadcast network, processing for transmission via broadband, etc. According to an embodiment, the transmission unit (14) may receive not only point cloud data but also metadata from the metadata processing unit and process the transmitted metadata for transmission. According to an embodiment, the processing for transmission may be performed in the transmission processing unit, and the transmission processing unit may be included in the transmission unit (14) or configured as a separate component / module from the transmission unit (14).

[0065] The receiving unit (21) can receive the bitstream transmitted by the transmitting device (10) or a file / segment including the bitstream. Depending on the channel through which the bitstream is transmitted, the receiving unit (21) can receive the bitstream or the file / segment including the bitstream through a broadcasting network, or can receive the bitstream or the file / segment including the bitstream through broadband. Alternatively, the receiving unit (21) can receive the bitstream or the file / segment including the bitstream through a digital storage medium.

[0066] The receiving unit (21) can perform processing according to a transmission protocol on the received bitstream or a file / segment including the bitstream. The receiving unit (21) can perform a reverse process of the transmission processing (processing for transmission) so as to correspond to the processing for transmission performed in the transmission device (10). The receiving unit (21) can transfer encoded point cloud data among the received data to the decapsulation processing unit (22) and transfer metadata to the metadata parsing unit. The metadata may be in the form of a signaling table. According to embodiments, the reverse process of the processing for transmission may be performed in the receiving unit. Each of the receiving unit, the decapsulation processing unit (22), and the metadata parsing unit may be included in the receiving unit (21) or configured as a separate component / module from the receiving unit (21).

[0067] The decapsulation processing unit (22) can decapsulate point cloud data in file form (i.e., bitstream in file form) received from the receiving unit (21) or the receiving processing unit. Therefore, the decapsulation processing unit (22) can be referred to as a 'file / segment decapsulation module'.

[0068] The decapsulation processing unit (22) can obtain a point cloud bitstream or metadata bitstream by decapsulating files according to ISOBMFF, etc. According to embodiments, metadata (metadata bitstream) may be included in the point cloud bitstream. The obtained point cloud bitstream may be transmitted to the decoding unit (23), and the obtained metadata bitstream may be transmitted to the metadata processing unit. The metadata processing unit may be included in the decoding unit (23), or may be configured as a separate component / module. The metadata obtained by the decapsulation processing unit (23) may be in the form of a box or track within a file format. The decapsulation processing unit (23) may also receive metadata required for decapsulation from the metadata processing unit, if necessary. The metadata may be transmitted to the decoding unit (23) and used in the decoding process (S23), or may be transmitted to the rendering unit (24) and used in the rendering process (S24).

[0069] The decoding unit (23) can perform a decoding process (S23) for decoding a point cloud bitstream (encoded point cloud data) by receiving a bitstream and performing an operation corresponding to the operation of the encoding unit (12). Therefore, the decoding unit (23) can be referred to as a 'point cloud video decoder'.

[0070] The decoding unit (23) can decode point cloud data by dividing it into geometry and attributes. For example, the decoding unit (23) can restore (decode) geometry from a geometry bitstream included in a point cloud bitstream, and can restore (decode) attributes based on an attribute bitstream included in the point cloud bitstream and the restored geometry. A 3D point cloud video / image can be restored based on position information according to the restored geometry and attributes (such as color or texture) according to the decoded attributes. The decoding process (S23) will be described in more detail below.

[0071] The rendering unit (24) can perform a rendering process (S24) for rendering the restored point cloud video. Therefore, the rendering unit (24) can be referred to as a 'renderer'.

[0072] The rendering process (S24) may refer to a process of rendering and displaying point cloud content in 3D space. The rendering process (S24) may render according to a desired rendering method based on position information and attribute information of points decoded through the decoding process.

[0073] Points in the point cloud content may be rendered as vertices with a certain thickness, cubes with a certain minimum size centered at the vertex location, or circles centered at the vertex location. A user can view all or part of the rendered result through a VR / AR display or a standard display. The rendered video may be displayed through a display unit. A user can view all or part of the rendered result through a VR / AR display or a standard display.

[0074] The feedback process (S25) may include a process of transmitting various feedback information that may be acquired during the rendering process (S24) or the display process to the transmission device (10) or to other components within the reception device (20). The feedback process (S25) may be performed by one or more of the components included in the reception device (20) of FIG. 1, or by one or more of the components represented in FIGS. 10 and 11. According to embodiments, the feedback process (S25) may also be performed by a 'feedback unit' or a 'sensing / tracking unit'.

[0075] Interactivity for point cloud content consumption may be provided through a feedback process (S25). According to embodiments, head orientation information, viewport information indicating the area currently being viewed by the user, etc. may be fed back in the feedback process (S25). According to embodiments, the user may interact with objects implemented in a VR / AR / MR / autonomous driving environment, in which case information related to the interaction may be transmitted to the transmission device (10) or the service provider in the feedback process (S25). According to embodiments, the feedback process (S25) may not be performed.

[0076] Head orientation information can refer to information about the user's head position, angle, and movement. Based on this information, information about the area the user is currently viewing within the point cloud video, i.e., viewport information, can be calculated.

[0077] Viewport information can be information about the area the user is currently viewing in a point cloud video. The viewpoint refers to the point the user is viewing in the point cloud video, and can refer to the exact center of the viewport area. In other words, the viewport is an area centered around the viewpoint, and the size and shape of that area can be determined by the field of view (FOV). Gaze analysis using viewport information can be used to determine how the user consumes the point cloud video, which area of ​​the point cloud video they are gazing at, and for how long. Gaze analysis can be performed on the receiving side (receiving device) and transmitted to the transmitting side (transmitting device) through a feedback channel. Devices such as VR / AR / MR displays can extract the viewport area based on the user's head position / orientation, the vertical or horizontal FOV supported by the device, etc.

[0078] According to embodiments, feedback information may not only be transmitted to the transmitter (transmitting device), but may also be consumed by the receiver (receiving device). That is, decoding processes, rendering processes, etc., may be performed on the receiver (receiving device) using the feedback information.

[0079] For example, the receiving device (20) may preferentially decode and render only the point cloud video for the area currently being viewed by the user using head orientation information and / or viewport information. In addition, the receiving unit (21) may receive all point cloud data, or may receive point cloud data indicated by the orientation information and / or viewport information based on the orientation information and / or viewport information. In addition, the decapsulation processing unit (22) may decapsulate all point cloud data, or may decapsulate point cloud data indicated by the orientation information and / or viewport information based on the orientation information and / or viewport information. In addition, the decoding unit (23) may decode all point cloud data, or may decode point cloud data indicated by the orientation information and / or viewport information based on the orientation information and / or viewport information.

[0080] Overview of the Point Cloud Encoding Device

[0081] Fig. 3 illustrates an example of a point cloud encoding device (300) according to embodiments of the present disclosure. The point cloud encoding device (300) of Fig. 3 may correspond to the encoding unit (12) of Fig. 1 in terms of configuration and function.

[0082] As illustrated in FIG. 3, the point cloud encoding device (300) may include a coordinate system transformation unit (305), a geometry quantization unit (310), an octree analysis unit (315), an approximation unit (320), a geometry encoding unit (325), a restoration unit (330), an attribute transformation unit (340), a RAHT transformation unit (345), an LOD generation unit (350), a lifting unit (355), an attribute quantization unit (360), an attribute encoding unit (365), and / or a color transformation unit (335).

[0083] The point cloud data acquired by the acquisition unit (11) may undergo processes to adjust the quality of the point cloud content (e.g., lossless, lossy, near-lossless) depending on the network conditions or applications. In addition, each point of the acquired point cloud content may be transmitted without loss, but in that case, real-time streaming may not be possible due to the large size of the point cloud content. Therefore, in order to smoothly provide the point cloud content, a process of reconstructing the point cloud content to match the maximum target bitrate is necessary.

[0084] Processes for controlling the quality of point cloud content may include reconstructing and encoding point location information (position information included in geometry information) or color information (color information included in attribute information). The process of reconstructing and encoding point position information may be referred to as geometry coding, and the process of reconstructing and encoding attribute information associated with each point may be referred to as attribute coding.

[0085] Geometry coding may include a geometry quantization process, a voxelization process, an octree analysis process, an approximation process, a geometry encoding process, and / or a coordinate system transformation process. Furthermore, geometry coding may further include a geometry restoration process. Attribute coding may include a color transformation process, an attribute transformation process, a prediction transformation process, a lifting transformation process, a RAHT transformation process, an attribute quantization process, an attribute encoding process, and the like.

[0086] Geometry coding

[0087] The coordinate system transformation process may correspond to a process of transforming the coordinate system for the positions of points. Therefore, the coordinate system transformation process may be referred to as 'transform coordinates'. The coordinate system transformation process may be performed by the coordinate system transformation unit (305). For example, the coordinate system transformation unit (305) may transform the positions of points from a global space coordinate system into position information of a three-dimensional space (e.g., a three-dimensional space expressed by an X-axis, Y-axis, and Z-axis coordinate system, etc.). The position information of the three-dimensional space according to embodiments may be referred to as 'geometry information'.

[0088] The geometry quantization process may correspond to a process of quantizing position information of points, and may be performed by a geometry quantization unit (310). For example, the geometry quantization unit (310) may find position information having the minimum (x, y, z) value among position information of points, and subtract the position information having the minimum (x, y, z) value from the position information of each point. In addition, the geometry quantization unit (310) may perform a quantization process by multiplying the subtracted value by a preset quantization scale value, and then adjusting (lowering or raising) the result to a nearby integer value.

[0089] The voxelization process may correspond to a process of matching quantized geometric information through a quantization process to a specific voxel existing in a three-dimensional space. The voxelization process may also be performed by a geometry quantization unit (310). The geometry quantization unit (310) may perform octree-based voxelization based on position information of points to reconstruct each point to which the quantization process is applied.

[0090] A voxel can refer to a space for storing information about three-dimensional points, similar to a pixel, the smallest unit containing information in a two-dimensional image or video. Voxel is a portmanteau of volume and pixel.

[0091] A voxel may not have only one point (matching). That is, information related to multiple points may exist in a voxel. Alternatively, information related to multiple points within a voxel may be integrated into a single point. This adjustment can be performed selectively. When integrated into a single point information for a voxel, the position value of the center point of the voxel may be set based on the position values ​​of the points within the voxel, and an attribute transformation process related to this may need to be performed. For example, the attribute transformation process may be adjusted based on the average value of the color or reflectance of the points within the voxel or the position value of the center point of the voxel and the neighboring points within a specific radius.

[0092] The octree analysis unit (315) can use an octree to efficiently manage the area / position of voxels. An octree can be expressed as an occupancy code. For example, the octree analysis unit (315) can express the occupancy code of each node as 1 if a point is contained within the node, and can express the occupancy code of the node as 0 if a point is not contained within the node.

[0093] The geometry encoding process may correspond to a process of performing entropy coding on the occupancy code. The geometry encoding process may be performed by the geometry encoding unit (325). The geometry encoding unit (325) may perform entropy coding on the occupancy code. The generated occupancy code may be encoded directly, or may be encoded through an intra / inter coding process to increase compression efficiency. The receiving device (20) may reconstruct an octree through the occupancy code.

[0094] On the other hand, for certain regions with few or no points, voxelizing the entire region may be inefficient. That is, because there are few points in a given region, constructing a full octree may not be necessary. In such cases, an early termination scheme may be necessary.

[0095] Instead of dividing a node (a specific node) corresponding to a specific area (a specific area that does not correspond to a leaf node) into eight sub-nodes (child nodes), the point cloud encoding device (300) can directly transmit the positions of points only for the specific area, or can reconstruct the positions of points within the specific area on a voxel basis using a surface model.

[0096] A mode that directly transmits the location of each point to a specific node may be a direct mode. The point cloud encoding device (300) can check whether conditions for enabling the direct mode are satisfied.

[0097] Conditions for enabling direct mode may include 1) the option to use direct mode must be enabled, 2) the specific node is not a leaf node, 3) there must be a threshold or less points within the specific node, and 4) the total number of points to be directly transferred does not exceed the threshold.

[0098] When all of the above conditions are satisfied, the point cloud encoding device (300) can directly transmit the position value of the point for the specific node by entropy encoding through the geometry encoding unit (325).

[0099] A mode for reconstructing the locations of points within a specific region on a voxel basis using a surface model may be a trisoup mode. The trisoup mode may be performed by an approximation unit (320). The approximation unit (320) may determine a specific level of the octree, and from the determined specific level, may reconstruct the locations of points within a node region on a voxel basis using a surface model.

[0100] The point cloud encoding device (300) may also selectively apply the TriShop mode. Specifically, when using the TriShop mode, the point cloud encoding device (300) may designate a level (a specific level) to which the TriShop mode is to be applied. For example, if the designated specific level is equal to the depth (d) of the octree, the TriShop mode may not be applied. In other words, the designated specific level must be smaller than the depth value of the octree.

[0101] A three-dimensional cubic area of ​​nodes at a specified level is called a block, and one block may include one or more voxels. A block or voxel may correspond to a brick. Each block may have 12 edges, and the approximation unit (320) may check whether each edge is adjacent to a voxel that has a point (occupied voxel). Each edge may be adjacent to multiple occupied voxels. A specific position of an edge adjacent to a voxel is called a vertex, and when multiple occupied voxels are adjacent to one edge, the approximation unit (320) may determine the average position of the corresponding positions as the vertex.

[0102] The point cloud encoding device (300) can entropy code the starting point (x, y, z) of the edge, the direction vector (△x, △y, △z) of the edge, and the position value (relative position value within the edge) of the vertex through the geometry encoding unit (325) when a vertex exists.

[0103] The geometry restoration process may correspond to a process of generating restored geometry by reconstructing an octree and / or an approximated octree. The geometry restoration process may be performed by a restoration unit (330). The restoration unit (330) may perform the geometry restoration process through triangle reconstruction, up-sampling, voxelization, etc.

[0104] When the tri-sub mode is applied in the approximation unit (320), the restoration unit (330) can reconstruct a triangle based on the starting point of the edge, the direction vector of the edge, and the position value of the vertex. The restoration unit (330) can perform an upsampling process to voxelize by adding points in the middle along the edge of the triangle. The restoration unit (330) can generate additional points based on the upsampling factor and the width of the block. These points can be referred to as refined vertices. The restoration unit (330) can voxelize the refined vertices, and the point cloud encoding device (300) can perform attribute coding based on the voxelized position value.

[0105] According to embodiments, the geometry encoding unit (325) may apply context adaptive arithmetic coding to increase compression efficiency. The geometry encoding unit (325) may directly entropy code the occupancy code using the arithmetic code. According to embodiments, the geometry encoding unit (325) may adaptively perform encoding based on the occupancy of surrounding neighboring nodes (intra coding) or may adaptively perform encoding based on the occupancy code of the previous frame (inter coding). Here, a frame may mean a set of point cloud data generated at the same time. Since intra coding and inter coding are optional processes, they may be omitted.

[0106] Compression efficiency can vary depending on how many neighboring nodes are referenced. While larger bits complicate the encoding process, they can also be biased to improve compression efficiency. For example, a 3-bit context might require coding in 23 = 8 different ways. This coding can impact implementation complexity, so it's important to balance compression efficiency with complexity.

[0107] Attribute Coding

[0108] Attribute coding can be the process of encoding attribute information based on restored (reconstructed) geometry and the geometry before coordinate system transformation (original geometry). Since attributes can be geometry-dependent, restored geometry can be utilized for attribute coding.

[0109] As previously explained, attributes can include color, reflectance, and more. The same attribute coding method can be applied to the information or parameters contained in an attribute. Color has three elements, and reflectance has one element, and each element can be processed independently.

[0110] Attribute coding may include a color transformation process, an attribute transformation process, a prediction transformation process, a lifting transformation process, a RAHT transformation process, an attribute quantization process, an attribute encoding process, etc. The prediction transformation process, the lifting transformation process, and the RAHT transformation process may be used selectively, or a combination of one or more may be used.

[0111] The color conversion process may correspond to a process of converting the format of a color within an attribute into another format. The color conversion process may be performed by a color conversion unit (335). That is, the color conversion unit (335) may convert a color within an attribute. For example, the color conversion unit (335) may perform a coding task of converting a color within an attribute from RGB to YCbCr. According to embodiments, the operation of the color conversion unit (335), i.e., the color conversion process, may be optionally applied depending on the color value included in the attribute.

[0112] As previously explained, when one or more points exist in a voxel, the position values ​​for the points within the voxel can be set to the center point of the voxel to integrate and represent them as a single point for that voxel. Accordingly, a process of converting the values ​​of the attributes associated with those points may be required. Furthermore, the attribute conversion process may also be performed when the TriShop mode is used.

[0113] The attribute transformation process may correspond to a process of transforming attributes based on positions for which geometry coding has not been performed and / or reconstructed geometry. For example, the attribute transformation process may correspond to a process of transforming attributes of points at positions within a voxel based on the positions of those points. The attribute transformation process may be performed by the attribute transformation unit (340).

[0114] The attribute conversion unit (340) can calculate the average value of the attribute values ​​of neighboring points (neighboring points) within a specific radius from the central position value of a voxel. Alternatively, the attribute conversion unit (340) can apply a weight according to the distance from the central position to the attribute values ​​and calculate the average value of the attribute values ​​to which the weights are applied. In this case, each voxel has a position and a calculated attribute value.

[0115] The prediction transformation process may correspond to a process of predicting an attribute value of a current point (a point corresponding to the target of prediction) based on the attribute values ​​of one or more points (neighboring points) neighboring the current point. The prediction transformation process may be performed by a LOD (level of detail) generation unit (350).

[0116] Prediction conversion is a method that applies the LOD conversion technique, and the LOD generation unit (350) can calculate and set the LOD value of each point based on the LOD distance value of each point. Points with the lowest LOD may be sparsely distributed, and points with the highest LOD may be densely distributed. In other words, as the LOD increases, the interval (or distance) between points may become shorter.

[0117] Each point within a point cloud can be separated by LOD, and the composition of points by LOD can include points belonging to a LOD lower than the LOD value. For example, the composition of points with LOD level 2 can include all points belonging to LOD level 1 and LOD level 2. Points can be reordered by LOD, and points belonging to a higher LOD can include points belonging to a lower LOD.

[0118] The LOD generation unit (350) can generate a predictor for each point for prediction transformation. Therefore, when there are N points, N predictors can be generated. The predictor can calculate and set a weight value (= 1 / distance) based on the LOD value for each point, indexing information for neighboring points, and distance values ​​to neighboring points. Here, neighboring points can be points that exist within a distance set for each LOD from the current point.

[0119] Additionally, the predictor can multiply the attribute values ​​of neighboring points by a "set weight value" and set the average of the attribute values ​​multiplied by the weight value as the predicted attribute value of the current point. An attribute quantization process can be performed on the residual attribute value obtained by subtracting the predicted attribute value of the current point from the attribute value of the current point.

[0120] The lifting transformation process, similar to the prediction transformation process, may correspond to a process of reconstructing points into a set of detail levels through a LOD generation process. The lifting transformation process may be performed by the lifting unit (355). The lifting transformation process may also include a process of generating a predictor for each point, a process of setting the calculated LOD to the predictor, a process of registering neighboring points, and a process of setting weights according to the distance between the current point and neighboring points.

[0121] The RAHT transformation process may correspond to a method of predicting attribute information of nodes at a higher level using attribute information associated with nodes at a lower level of an octree. In other words, the RAHT transformation process may correspond to an attribute information intra-coding method through an octree backward scan. The RAHT transformation process may be performed by a RAHT transformation unit (345).

[0122] The RAHT transformation unit (345) can scan from voxels to the entire area and perform the RAHT transformation process up to the root node while combining (merging) voxels into larger blocks at each step. Since the RAHT transformation unit (345) performs the RAHT transformation process only on occupied nodes, in the case of empty nodes that are not occupied, the RAHT transformation process can be performed on the node at the immediately upper level above.

[0123] The attribute quantization process may correspond to a process of quantizing attributes output from the RAHT transform unit (345), the LOD generation unit (350), and / or the lifting unit (355). The attribute quantization process may be performed by the attribute quantization unit (360). The attribute encoding process may correspond to a process of encoding the quantized attributes and outputting an attribute bitstream. The attribute encoding process may be performed by the attribute encoding unit (365).

[0124] For example, if the predicted attribute value of the current point is calculated from the LOD generation unit (350), the attribute quantization unit (360) can quantize the residual attribute value obtained by subtracting the predicted attribute value of the current point from the attribute value of the current point.

[0125] If there are no neighboring points in the predictor of each point, the attribute encoding unit (365) can directly entropy encode the attribute value (non-quantized attribute value) of the current point. Conversely, if there are neighboring points in the predictor of the current points, the attribute encoding unit (365) can entropy encode the quantized residual attribute value.

[0126] As another example, when a value obtained by multiplying an attribute value updated through a lift update process by a weight (stored in QW) updated through a lift prediction process is output from a lifting unit (360), the attribute quantization unit (360) can quantize the result (the multiplied value), and the attribute encoding unit (365) can entropy encode the quantized value.

[0127] Overview of the Point Cloud Decoding Device

[0128] Fig. 4 illustrates an example of a point cloud decoding device (400) according to one embodiment of the present disclosure. The point cloud decoding device (400) of Fig. 4 may correspond to the decoding unit (23) of Fig. 1 in terms of configuration and function.

[0129] The point cloud decoding device (400) can perform a decoding process based on data (bitstream) transmitted from the transmission device (10). The decoding process can include a process of restoring (decoding) a point cloud video by performing an operation corresponding to the encoding operation described above on the bitstream.

[0130] As illustrated in FIG. 4, the decoding process may include a geometry decoding process and an attribute decoding process. The geometry decoding process may be performed by a geometry decoding unit (410), and the attribute decoding process may be performed by an attribute decoding unit (420). That is, the point cloud decoding device (400) may include a geometry decoding unit (410) and an attribute decoding unit (420).

[0131] The geometry decoding unit (410) can restore geometry from a geometry bitstream, and the attribute decoding unit (420) can restore attributes based on the restored geometry and the attribute bitstream. In addition, the point cloud decoding device (400) can restore a 3D point cloud video (point cloud data) based on position information according to the restored geometry and attribute information according to the restored attribute.

[0132] FIG. 5 illustrates a specific example of a point cloud decoding device (500) according to another embodiment of the present disclosure. As illustrated in FIG. 5, the point cloud decoding device (500) may include a geometry decoding unit (505), an octree synthesis unit (510), an approximation synthesis unit (515), a geometry restoration unit (520), a coordinate system inverse transformation unit (525), an attribute decoding unit (530), an attribute inverse quantization unit (535), a RATH transformation unit (550), a LOD generation unit (540), an inverse lifting unit (545), and / or a color inverse transformation unit (555).

[0133] The geometry decoding unit (505), the octree synthesis unit (510), the approximation synthesis unit (515), the geometry restoration unit (520), and the coordinate system inversion unit (550) can perform geometry decoding. The geometry decoding can be performed by the reverse process of the geometry coding described in FIGS. 1 to 3. The geometry decoding can include direct coding and trisoup geometry decoding. Direct coding and trisoup geometry decoding can be selectively applied.

[0134] The geometry decoding unit (505) can decode the received geometry bitstream based on arithmetic coding. The operation of the geometry decoding unit (505) can correspond to the reverse process of the operation performed by the geometry encoding unit (325).

[0135] The octree synthesis unit (510) can generate an octree by obtaining an occupancy code from a decoded geometry bitstream (or information about the geometry obtained as a result of decoding). The operation of the octree synthesis unit (510) can correspond to the reverse process of the operation performed by the octree analysis unit (515).

[0136] The approximation synthesis unit (515) can synthesize a surface based on the decoded geometry and / or the generated octree when the tri-subject geometry encoding is applied.

[0137] The geometry restoration unit (520) can restore the geometry based on the surface and decoded geometry. When direct coding is applied, the geometry restoration unit (520) can directly obtain and add position information of points to which direct coding is applied. In addition, when try-subtract geometry encoding is applied, the geometry restoration unit (520) can restore the geometry by performing a reconstruction operation, such as triangle reconstruction, up-sampling, voxelization, etc. The restored geometry can include a point cloud picture or frame that does not include attributes.

[0138] The coordinate system inversion unit (550) can obtain the positions of points by converting the coordinate system based on the restored geometry. For example, the coordinate system inversion unit (550) can inversely convert the positions of points from a three-dimensional space (e.g., a three-dimensional space expressed by an X-axis, Y-axis, and Z-axis coordinate system, etc.) into position information of a global space coordinate system.

[0139] The attribute decoding unit (530), the attribute inverse quantization unit (535), the RATH transform unit (530), the LOD generation unit (540), the inverse lifting unit (545), and / or the color inverse transform unit (550) can perform attribute decoding. The attribute decoding can include RAHT transform decoding, prediction transform decoding, and lifting transform decoding. The three decodings described above can be used selectively, or a combination of one or more decodings can be used.

[0140] The attribute decoding unit (530) can decode the attribute bitstream based on arithmetic coding. For example, if there are no neighboring points in the predictor of each point and the attribute value of the current point is directly entropy encoded, the attribute decoding unit (530) can decode the attribute value (non-quantized attribute value) of the current point. As another example, if there are neighboring points in the predictor of the current points and the quantized residual attribute value is entropy encoded, the attribute decoding unit (530) can decode the quantized residual attribute value.

[0141] The attribute dequantization unit (535) can dequantize information about the decoded attribute bitstream or the attributes obtained as a result of the decoding, and output the dequantized attributes (or attribute values). For example, if a quantized residual attribute value is output from the attribute decoding unit (530), the attribute dequantization unit (535) can dequantize the quantized residual attribute value and output the residual attribute value. The dequantization process can be selectively applied based on whether the point cloud encoding device (300) performs attribute encoding. That is, if there are no neighboring points in the predictor of each point and the attribute value of the current point is directly encoded, the attribute decoding unit (530) can output the non-quantized attribute value of the current point, and the attribute encoding process can be skipped.

[0142] The RATH transformation unit (550), the LOD generation unit (540), and / or the inverse lifting unit (545) can process the reconstructed geometry and inverse quantized attributes. The RATH transformation unit (550), the LOD generation unit (540), and / or the inverse lifting unit (545) can selectively perform a decoding operation corresponding to the encoding operation of the point cloud encoding device (300).

[0143] The color inverse conversion unit (555) can perform inverse conversion coding to inversely convert the color values ​​(or textures) included in the decoded attributes. The operation of the color inverse conversion unit (555) can be selectively performed based on whether the color conversion unit (335) is operating.

[0144] Fig. 6 illustrates another example of a transmission device according to embodiments of the present disclosure. As illustrated in Fig. 6, the transmission device may include a data input unit (605), a quantization processing unit (610), a voxelization processing unit (615), an octree occupancy code generation unit (620), a surface model processing unit (625), an intra / inter coding processing unit (630), an arithmetic coder (635), a metadata processing unit (640), a color conversion processing unit (645), an attribute conversion processing unit (650), a prediction / lifting / RAHT conversion processing unit (655), an arithmetic coder (660), and a transmission processing unit (665).

[0145] The function of the data input unit (605) may correspond to the acquisition process performed by the acquisition unit (11) of FIG. 1. That is, the data input unit (605) may acquire a point cloud video and generate point cloud data for a plurality of points. Geometry information (position information) in the point cloud data may be generated in the form of a geometry bitstream through a quantization processing unit (610), a voxelization processing unit (615), an octree occupancy code generation unit (620), a surface model processing unit (625), an intra / inter coding processing unit (630), and an arithmetic coder (635). Attribute information in the point cloud data may be generated in the form of an attribute bitstream through a color conversion processing unit (645), an attribute conversion processing unit (650), a prediction / lifting / RAHT conversion processing unit (655), and an arithmetic coder (660). The geometry bitstream, attribute bitstream, and / or metadata bitstream can be transmitted to the receiving device through processing by the transmission processing unit (665).

[0146] Specifically, the function of the quantization processing unit (610) may correspond to the quantization process performed by the geometry quantization unit (310) of FIG. 3 and / or the function of the coordinate system transformation unit (305). The function of the voxelization processing unit (615) may correspond to the voxelization process performed by the geometry quantization unit (310) of FIG. 3, and the function of the octree occupancy code generation unit (620) may correspond to the function performed by the octree analysis unit (315) of FIG. 3. The function of the surface model processing unit (625) may correspond to the function performed by the approximation unit (320) of FIG. 3, and the functions of the intra / inter coding processing unit (630) and the arithmetic coder (635) may correspond to the function performed by the geometry encoding unit (325) of FIG. 3. The function of the metadata processing unit (640) may correspond to the function of the metadata processing unit described in FIG. 1.

[0147] In addition, the function of the color conversion processing unit (645) may correspond to the function performed by the color conversion unit (335) of FIG. 3, and the function of the attribute conversion processing unit (650) may correspond to the function performed by the attribute conversion unit (340) of FIG. 3. The function of the prediction / lifting / RAHT conversion processing unit (655) may correspond to the functions performed by the RAHT conversion unit (345), LOD generation unit (350), and lifting unit (355) of FIG. 3, and the function of the arithmetic coder (660) may correspond to the function of the attribute encoding unit (365) of FIG. 3. The function of the transmission processing unit (665) may correspond to the functions performed by the transmission unit (14) and / or the encapsulation processing unit (13) of FIG. 1.

[0148] Fig. 7 illustrates another example of a receiving device according to embodiments of the present disclosure. As illustrated in Fig. 7, the receiving device may include a receiving unit (705), a receiving processing unit (710), an arithmetic decoder (715), a metadata parser (735), an occupancy code-based octree reconstruction processing unit (720), a surface model processing unit (725), an inverse quantization processing unit (730), an arithmetic decoder (740), an inverse quantization processing unit (745), a prediction / lifting / RAHT inverse transform processing unit (750), a color inverse transform processing unit (755), and a renderer (760).

[0149] The function of the receiving unit (705) may correspond to the function performed by the receiving unit (21) of FIG. 1, and the function of the receiving processing unit (710) may correspond to the function performed by the decapsulation processing unit (22) of FIG. 1. That is, the receiving unit (705) receives a bitstream from the transmitting processing unit (765), and the receiving processing unit (710) may extract a geometry bitstream, an attribute bitstream, and / or a metadata bitstream through decapsulation processing. The geometry bitstream may be generated as a reconstructed (restored) position value (position information) through an arithmetic decoder (715), an occupancy code-based octree reconstruction processing unit (720), a surface model processing unit (725), and an inverse quantization processing unit (730). The attribute bitstream can be generated as restored attribute values ​​through an arithmetic decoder (740), an inverse quantization processor (745), a prediction / lifting / RAHT inverse transform processor (750), and a color inverse transform processor (755). The metadata bitstream can be generated as restored metadata (or metadata information) through a metadata parser (735). Position values, attribute values, and / or metadata can be rendered in a renderer (760) to provide users with experiences such as VR / AR / MR / autonomous driving.

[0150] Specifically, the function of the arithmetic decoder (715) may correspond to the function performed by the geometry decoding unit (505) of FIG. 5, and the function of the occupancy code-based octree reconstruction processing unit (720) may correspond to the function performed by the octree synthesis unit (510) of FIG. 5. The function of the surface model processing unit (725) may correspond to the function performed by the approximation synthesis unit of FIG. 5, and the function of the inverse quantization processing unit (730) may correspond to the function performed by the geometry restoration unit (520) and / or the coordinate system inversion unit (525) of FIG. 5. The function of the metadata parser (735) may correspond to the function performed by the metadata parsing unit described in FIG. 1.

[0151] In addition, the function of the arithmetic decoder (740) may correspond to the function performed by the attribute decoding unit (530) of FIG. 5, and the function of the inverse quantization processing unit (745) may correspond to the function of the attribute inverse quantization unit (535) of FIG. 5. The function of the prediction / lifting / RAHT inverse transform processing unit (750) may correspond to the functions performed by the RAHT transform unit (550), LOD generation unit (540), and inverse lifting unit (545) of FIG. 5, and the function of the color inverse transform processing unit (755) may correspond to the function performed by the color inverse transform unit (555) of FIG. 5.

[0152] FIG. 8 illustrates an example of a structure that can be linked with a point cloud data transmission / reception method / device according to embodiments of the present disclosure.

[0153] The structure of FIG. 8 represents a configuration in which at least one of a server (AI Server), a robot, a self-driving vehicle (Self-Driving Vehicle), an XR device (XR device), a smartphone, a home appliance (Home Appliance), and / or an HMD) is connected to a cloud network (Network). The robot, self-driving vehicle, XR device, smartphone, or home appliance may be referred to as a device. In addition, the XR device may correspond to a point cloud data device (PCC) according to embodiments or may be linked with a PCC device.

[0154] A cloud network can be a network that forms part of a cloud computing infrastructure or exists within a cloud computing infrastructure. Here, a cloud network can be configured using 3G networks, 4G or LTE (Long Term Evolution) networks, or 5G networks.

[0155] The server may be connected to at least one of a robot, an autonomous vehicle, an XR device, a smartphone, an appliance, and / or an HMD via a cloud network and may assist in at least some of the processing of the connected devices.

[0156] The HMD may represent one of the types in which an XR device and / or a PCC device according to embodiments may be implemented. A device of the HMD type according to embodiments may include a communication unit, a control unit, a memory unit, an I / O unit, a sensor unit, and a power supply unit.

[0157] <PCC+XR>

[0158] XR / PCC devices may be implemented as HMDs, HUDs installed in vehicles, televisions, mobile phones, smart phones, computers, wearable devices, home appliances, digital signage, vehicles, stationary robots or mobile robots, etc., with PCC and / or XR technologies applied.

[0159] An XR / PCC device can acquire information about surrounding space or real-world objects by analyzing 3D point cloud data or image data acquired through various sensors or from external devices to generate positional (geometry) data and attribute data for 3D points, and can render and output an XR object to be output. For example, an XR / PCC device can output an XR object containing additional information about a recognized object in correspondence with the recognized object.

[0160] <PCC+XR+모바일폰>

[0161] XR / PCC devices can be implemented in mobile phones and other devices using PCC technology. Mobile phones can decode and display point cloud content based on PCC technology.

[0162] <PCC+자율주행+XR>

[0163] Autonomous vehicles can be implemented as mobile robots, vehicles, and unmanned aerial vehicles by applying PCC and XR technologies. Autonomous vehicles utilizing XR / PCC technologies can refer to autonomous vehicles equipped with means to provide XR images, or autonomous vehicles that are the subject of control / interaction within XR images. Specifically, autonomous vehicles that are the subject of control / interaction within XR images are distinct from XR devices and can be interconnected.

[0164] An autonomous vehicle equipped with a means for providing XR / PCC images can acquire sensor information from sensors including cameras and output XR / PCC images generated based on the acquired sensor information. For example, the autonomous vehicle can be equipped with a head-up display (HUD) to output XR / PCC images, thereby providing passengers with XR / PCC objects corresponding to real-world objects or objects on the screen.

[0165] At this time, when the XR / PCC object is output to the HUD, at least a portion of the XR / PCC object may be output so as to overlap with an actual object toward which the passenger's gaze is directed. On the other hand, when the XR / PCC object is output to a display provided inside an autonomous vehicle, at least a portion of the XR / PCC object may be output so as to overlap with an object on the screen. For example, an autonomous vehicle may output XR / PCC objects corresponding to objects such as a lane, another vehicle, a traffic light, a traffic sign, a two-wheeled vehicle, a pedestrian, a building, etc.

[0166] The VR technology, AR technology, MR technology, and / or PCC technology according to the embodiments can be applied to various devices. In other words, VR technology is a display technology that provides only CG images of objects or backgrounds in the real world. On the other hand, AR technology refers to a technology that shows a virtually created CG image on top of an image of an actual object. Furthermore, MR technology is similar to the aforementioned AR technology in that it mixes and combines virtual objects in the real world. However, in AR technology, the distinction between real objects and virtual objects created with CG images is clear, and virtual objects are used in a form that complements real objects, whereas in MR technology, virtual objects are considered to have the same characteristics as real objects. More specifically, a hologram service is an example of an application of the aforementioned MR technology. VR, AR, and MR technologies can be referred to as XR technology by integrating them.

[0167] space division

[0168] Point cloud data (i.e., G-PCC data) can represent a volumetric encoding of a point cloud consisting of a sequence of frames (point cloud frames). Each point cloud frame can include the number of points, positions of the points, and attributes of the points. The number of points, positions of the points, and attributes of the points can vary from frame to frame. Each point cloud frame can mean a set of 3D points specified by Cartesian coordinates (x, y, z) of 3D points and zero or more attributes at a particular time instance. Here, the Cartesian coordinates (x, y, z) of the 3D points can be positions or geometries.

[0169] According to embodiments, the present disclosure may further perform a spatial partitioning process of partitioning point cloud data into one or more three-dimensional blocks before encoding (encoding) the point cloud data. The three-dimensional block may refer to all or part of a three-dimensional space occupied by the point cloud data. The three-dimensional block may refer to one or more of a tile group, a tile, a slice, a coding unit (CU), a prediction unit (PU), or a transform unit (TU).

[0170] A tile corresponding to a 3D block may mean all or part of a 3D space occupied by point cloud data. In addition, a slice corresponding to a 3D block may also mean all or part of a 3D space occupied by point cloud data. A tile may be divided into one or more slices based on the number of points included in one tile. A tile may be a group of slices having bounding box information. The bounding box information of each tile may be specified in a tile inventory (or tile parameter set (TPS)). A tile may overlap with another tile within a bounding box. A slice may be a unit of data for which encoding is performed independently, and may be a unit of data for which decoding is performed independently. In other words, a slice may be a set of points that can be encoded or decoded independently. According to embodiments, a slice may be a series of syntax elements representing part or all of a coded point cloud frame. Each slice may include an index for identifying a tile to which the slice belongs.

[0171] The spatially partitioned 3D blocks can be processed independently or non-independently. For example, the spatially partitioned 3D blocks can be encoded or decoded independently or non-independently, and transmitted or received independently or non-independently. Furthermore, the spatially partitioned 3D blocks can be quantized or dequantized independently or non-independently, and transformed or inverted independently or non-independently. Furthermore, the spatially partitioned 3D blocks can be rendered independently or non-independently. For example, encoding or decoding can be performed on a slice-by-slice basis or a tile-by-tile basis. Furthermore, quantization or inverse quantization can be performed differently on a tile-by-tile basis or a slice-by-slice basis, and can be performed differently on a tile-by-tile or slice-by-slice basis when transformed or inversely transformed.

[0172] In this way, by spatially dividing point cloud data into one or more 3D blocks and processing these spatially divided 3D blocks independently or non-independently, the 3D block processing process can be performed in real time with low latency. Furthermore, random access and parallel encoding or decoding of the 3D space occupied by the point cloud data can be enabled, and errors accumulated during the encoding or decoding process can be prevented.

[0173] FIG. 9 is a block diagram illustrating an example of a transmission device (900) that performs a spatial division process according to embodiments of the present disclosure. As illustrated in FIG. 9, the transmission device (900) may include a spatial division unit (905) that performs a spatial division process, a signaling processing unit (910), a geometry encoder (915), an attribute encoder (920), an encapsulation processing unit (925), and / or a transmission processing unit (930).

[0174] The spatial partitioning unit (905) may perform a spatial partitioning process to partition point cloud data into one or more three-dimensional blocks based on a bounding box and / or a sub-bounding box. Through the spatial partitioning process, the point cloud data may be partitioned into one or more tiles and / or one or more slices. According to embodiments, through the spatial partitioning process, the point cloud data may be partitioned into one or more tiles, and each of the partitioned tiles may be further partitioned into one or more slices.

[0175] Figure 10 illustrates an example of spatially dividing a bounding box (i.e., point cloud data) into one or more 3D blocks. As illustrated in Figure 10, the overall bounding box of the point cloud data can be divided into three tiles, namely tile #0, tile #1, and tile #2. Furthermore, tile #0 can be further divided into two slices, namely slice #0 and slice #1. Furthermore, tile #1 can be further divided into two slices, namely slice #2 and slice #3. Furthermore, tile #2 can be further divided into slice #4.

[0176] The signaling processing unit (910) can generate and / or process (e.g., entropy-encode) signaling information and output it in the form of a bitstream. Hereinafter, the bitstream output from the signaling processing unit (in which signaling information is encoded) is referred to as a "signaling bitstream." The signaling information may include information for space division or information about space division. In other words, the signaling information may include information related to the space division process performed by the space division unit (905).

[0177] When point cloud data is divided into one or more 3D blocks, information may be required to decode a portion of the point cloud data corresponding to a specific tile or a specific slice among the point cloud data. Furthermore, information related to 3D spatial regions may be required to support spatial access (or partial access) to the point cloud data. Here, spatial access may mean extracting only a portion of the necessary point cloud data from the file. The signaling information may include information for decoding a portion of the point cloud data, information related to 3D spatial regions to support spatial access, and the like. For example, the signaling information may include 3D bounding box information, 3D spatial region information, tile information, and / or tile inventory information.

[0178] The signaling information may be provided from the space partitioning unit (905), the geometry encoder (915), the attribute encoder (920), the transmission processing unit (925), and / or the encapsulation processing unit (930). In addition, the signaling processing unit (910) may provide feedback information fed back from the receiving device (900) of FIG. 13 to the space partitioning unit (905), the geometry encoder (915), the attribute encoder (920), the transmission processing unit (925), and / or the encapsulation processing unit (930).

[0179] The signaling information may be stored and signaled in a sample, a sample entry, a sample group, a track group, or a separate metadata track within a track. According to embodiments, the signaling information may be signaled in units of a sequence parameter set (SPS) for signaling at a sequence level, a geometry parameter set (GPS) for signaling geometry coding information, an attribute parameter set (APS) for signaling attribute coding information, a tile parameter set (TPS) (or tile inventory) for signaling at a tile level, etc. In addition, the signaling information may also be signaled in units of coding units such as slices or tiles.

[0180] Meanwhile, the positions (position information) of the 3D blocks can be output to the geometry encoder (915), and the attributes (attribute information) of the 3D blocks can be output to the attribute encoder (920).

[0181] The geometry encoder (915) can construct an octree based on position information and output a geometry bitstream by encoding the constructed octree. In addition, the geometry encoder (915) can reconstruct (restore) the octree and / or the approximated octree and output the reconstructed octree to the attribute encoder (920). The reconstructed octree may be a reconstructed geometry. The geometry encoder (915) can perform all or part of the operations performed by the coordinate system transformation unit (305), the geometry quantization unit (310), the octree analysis unit (315), the approximation unit (320), the geometry encoding unit (325), and / or the restoration unit (330) of FIG. 3. According to embodiments, the geometry encoder (915) may perform all or part of the operations performed by the quantization processing unit (610), the voxelization processing unit (615), the octree occupancy code generation unit (620), the surface model processing unit (625), the intra / inter coding processing unit (630), and / or the arithmetic coder (635) of FIG. 6.

[0182] The attribute encoder (920) can output an attribute bitstream by encoding attributes based on the restored geometry. The attribute encoder (920) can perform all or part of the operations performed by the attribute transformation unit (340), the RAHT transformation unit (345), the LOD generation unit (350), the lifting unit (355), the attribute quantization unit (360), the attribute encoding unit (365), and / or the color transformation unit (335) of FIG. 3. According to embodiments, the attribute encoder (920) can perform all or part of the operations performed by the attribute transformation processing unit (650), the prediction / lifting / RAHT transformation processing unit (655), the arithmetic coder (660), and / or the color transformation processing unit (645) of FIG. 6.

[0183] The encapsulation processing unit (925) can encapsulate one or more input bitstreams into a file or segment, etc. For example, the encapsulation processing unit (925) can encapsulate a geometry bitstream, an attribute bitstream, and a signaling bitstream, respectively, or can multiplex and encapsulate a geometry bitstream, an attribute bitstream, and a signaling bitstream. According to embodiments, the encapsulation processing unit (925) can encapsulate a bitstream (G-PCC bitstream) composed of a sequence of type-length-value (TLV) structures into a file. The TLV (or TLV encapsulation) structures constituting the G-PCC bitstream can include a geometry bitstream, an attribute bitstream, a signaling bitstream, etc. According to embodiments, the G-PCC bitstream can be generated by the encapsulation processing unit (925) or can be generated by the transmission processing unit (930). The TLV structure or TLV encapsulation structure will be described in detail later. According to embodiments, the encapsulation processing unit (925) may perform all or part of the operations performed by the encapsulation processing unit (13) of FIG. 1.

[0184] The transmission processing unit (930) can process encapsulated bitstreams or files / segments, etc. according to any transmission protocol. The transmission processing unit (930) can perform all or part of the operations performed by the transmission unit (14) and the transmission processing unit described through FIG. 1 or the transmission processing unit (665) of FIG. 6.

[0185] FIG. 11 is a block diagram illustrating an example of a receiving device (1100) according to embodiments of the present disclosure. The receiving device (1100) may perform operations corresponding to those of the transmitting device (900) performing spatial segmentation. As illustrated in FIG. 11 , the receiving device (1100) may include a receiving processing unit (1105), a decapsulation processing unit (1110), a signaling processing unit (1115), a geometry decoder (1120), an attribute encoder (1125), and / or a post-processing unit (1130).

[0186] The receiving processing unit (1105) can receive a file / segment, a G-PCC bitstream or a bitstream encapsulated with a G-PCC bitstream, and perform processing on them according to a transmission protocol. The receiving processing unit (1105) can perform all or part of the operations performed by the receiving unit (21) and the receiving processing unit described through Fig. 1, or the receiving unit (705) or the receiving processing unit (710) of Fig. 7.

[0187] The decapsulation processing unit (1110) can obtain a G-PCC bitstream by performing the reverse process of the operations performed by the encapsulation processing unit (925). The decapsulation processing unit (1110) can decapsulate a file / segment to obtain a G-PCC bitstream. For example, the decapsulation processing unit (1110) can obtain a signaling bitstream and output it to the signaling processing unit (1115), obtain a geometry bitstream and output it to the geometry decoder (1120), and obtain an attribute bitstream and output it to the attribute decoder (1125). The decapsulation processing unit (1110) can perform all or part of the operations performed by the decapsulation processing unit (22) of FIG. 1 or the reception processing unit (710) of FIG. 7.

[0188] The signaling processing unit (1115) can parse and decode signaling information by performing the reverse process of the operations performed by the signaling processing unit (910). The signaling processing unit (1115) can parse and decode signaling information from a signaling bitstream. The signaling processing unit (1115) can provide the decoded signaling information to the geometry decoder (1120), the attribute decoder (1125), and / or the post-processing unit (1130).

[0189] The geometry decoder (1120) can reconstruct geometry from a geometry bitstream by performing the reverse process of the operations performed by the geometry encoder (915). The geometry decoder (1120) can reconstruct geometry based on signaling information (parameters related to the geometry). The reconstructed geometry can be provided to the attribute decoder (1125).

[0190] The attribute decoder (1125) can restore attributes from the attribute bitstream by performing the reverse process of the operations performed by the attribute encoder (920). The attribute decoder (1125) can restore attributes based on signaling information (parameters related to the attribute) and restored geometry.

[0191] The post-processing unit (1130) can restore point cloud data based on the restored geometry and restored attributes. Restoration of the point cloud data can be performed through a process of matching the restored geometry and restored attributes with each other. According to embodiments, when the restored point cloud data is in tile and / or slice units, the post-processing unit (1130) can restore the bounding box of the point cloud data by performing the reverse process of the spatial division process of the transmission device (900) based on signaling information. According to embodiments, when the bounding box is divided into a plurality of tiles and / or a plurality of slices through the spatial division process, the post-processing unit (1130) can also restore a portion of the bounding box by combining some slices and / or some tiles based on signaling information. Here, some slices and / or some tiles used for restoring the bounding box may be slices and / or some tiles related to a three-dimensional space region to which spatial access is desired.

[0192] Encapsulation / decapsulation

[0193] The TLV encapsulation structure can be generated in the transmission unit, transmission processing unit, and encapsulation unit mentioned in this specification. The G-PCC bitstream composed of the TLV encapsulation structures can be transmitted to the receiving device as is, or can be transmitted to the receiving device after being encapsulated. For example, the encapsulation processing unit (1125) can encapsulate the G-PCC bitstream composed of the TLV encapsulation structures in the form of a file / segment and transmit it. The decapsulation processing unit (1110) can decapsulate the encapsulated file / segment to obtain the G-PCC bitstream.

[0194] According to embodiments, a G-PCC bitstream may be encapsulated in an ISOBMFF-based file format. In this case, the G-PCC bitstream may be stored in a single track or multiple tracks within an ISOBMFF file. Here, a single track or multiple tracks within the file may be referred to as a “track” or a “G-PCC track.” An ISOBMFF-based file may be referred to as a container, a container file, a media file, a G-PCC file, etc. Specifically, the file may be composed of boxes and / or information, which may be referred to as ftyp, moov, mdat, etc.

[0195] The ftyp box (file type box) can provide file type or file compatibility information for the corresponding file. The receiving device can distinguish the corresponding file by referring to the ftyp box. The mdat box, also called a media data box, can contain actual media data. According to embodiments, a geometry slice (or a coded geometry bitstream) and zero or more attribute slices (or a coded attribute bitstream) can be included in a sample of the mdat box in the file. Here, the sample can be referred to as a G-PCC sample. The moov box, also called a movie box, can contain metadata about the media data of the corresponding file. For example, the moov box can contain information necessary for decoding and playing the corresponding media data, and can contain information about the tracks and samples of the corresponding file. The moov box can act as a container for all metadata. The moov box can be a top-layer box among the metadata-related boxes.

[0196] According to embodiments, a moov box may include a track (trak) box that provides information related to a track of a file, and the trak box may include a media (mdia) box (MediaBox) that provides media information of the track, and a track reference container (tref) box for linking (referencing) the track and a sample of a file corresponding to the track. The media box (MediaBox) may include a media information container (minf) box that provides information about the media data, and a handler (hdlr) box (HandlerBox) that indicates the type of the stream. The minf box may include a sample table (stbl) box that provides metadata related to the sample of the mdat box. The stbl box may include a sample description (stsd) box that provides information about the coding type used and initialization information necessary for the coding type. According to embodiments, the sample description (stsd) box may include a sample entry for the track. Depending on the embodiments, signaling information (or metadata) such as SPS, GPS, APS, tile inventory, etc. may be included in sample entries of the moov box or samples of the mdat box within the file.

[0197] A G-PCC track may be defined as a volumetric visual track carrying a geometry slice (or a coded geometry bitstream), an attribute slice (or a coded attribute bitstream), or both a geometry slice and an attribute slice. In some embodiments, a volumetric visual track may be identified by a volumetric visual media handler type 'volv' in a HandlerBox of a MediaBox and / or a volumetric visual media header (vvhd) in a minf box of the MediaBox. The minf box may be referred to as a media information container or a media information box. The minf box may be contained in a MediaBox, the MediaBox may be contained in a Track box, and the Track box may be contained in a moov box of a file. A single volumetric visual track or multiple volumetric visual tracks can exist in a file.

[0198] sample group

[0199] The encapsulation processing unit mentioned in the present disclosure can group one or more samples to create a sample group. The encapsulation processing unit, metadata processing unit, or signaling processing unit mentioned in the present disclosure can signal signaling information associated with the sample group to a sample, a sample group, or a sample entry. That is, the sample group information associated with the sample group can be added to the sample, the sample group, or the sample entry. The sample group information can be 3D bounding box sample group information, 3D area sample group information, 3D tile sample group information, 3D tile inventory sample group information, etc.

[0200] Sample Entry

[0201] Fig. 12 is a diagram for explaining an ISOBMFF-based file including a single track. Fig. 12 (a) shows an example of the layout of an ISOBMFF-based file including a single track, and Fig. 12 (b) shows an example of a sample structure of an mdat box when a G-PCC bitstream is stored in a single track of the file. Fig. 13 is a diagram for explaining an ISOBMFF-based file including multiple tracks. Fig. 13 (a) shows an example of the layout of an ISOBMFF-based file including multiple tracks, and Fig. 13 (b) shows an example of a sample structure of an mdat box when a G-PCC bitstream is stored in a single track of the file.

[0202] The stsd box (SampleDescriptionBox) included in the moov box of a file may contain sample entries for a single track storing a G-PCC bitstream. SPS, GPS, APS, and tile inventory may be included in the sample entries of the moov box in the file or in the samples of the mdat box. Additionally, geometry slices and zero or more attribute slices may be included in the samples of the mdat box in the file. When a G-PCC bitstream is stored in a single track of a file, each sample may contain multiple G-PCC components. That is, each sample may be composed of one or more TLV encapsulation structures.

[0203] As illustrated in (b) of FIG. 13, the sample may include TLV encapsulation structures that include geometry slices. Furthermore, the sample may include TLV encapsulation structures that include one or more parameter sets. Furthermore, the sample may include TLV encapsulation structures that include one or more attribute slices.

[0204] As illustrated in (a) of FIG. 13, when a G-PCC bitstream is carried as multiple tracks of an ISOBMFF-based file, each geometry slice or attribute slice can be mapped to an individual track. For example, a geometry slice can be mapped to track 1, and an attribute slice can be mapped to track 2. A track (track 1) carrying a geometry slice can be referred to as a geometry track or a G-PCC geometry track, and a track (track 2) carrying an attribute slice can be referred to as an attribute track or a G-PCC attribute track. In addition, a geometry track can be defined as a volumetric visual track carrying a geometry slice, and an attribute track can be defined as a volumetric visual track carrying an attribute slice.

[0205] A track carrying a portion of a G-PCC bitstream that includes both a geometry slice and an attribute slice may be referred to as a multiplexed track. When the geometry slice and the attribute slice are stored in separate tracks, each sample within the track may include at least one TLV encapsulation structure carrying data of a single G-PCC component. In this case, each sample may not include both a geometry slice and an attribute slice, and may also not include multiple attributes. Multi-track encapsulation of a G-PCC bitstream may enable a G-PCC player to effectively access any one of the G-PCC components.

[0206] G-PCC data may include a group of G-PCC parameter set samples to provide further description of the parameter sets when carried in samples that support random access features. The design of the sample group may be as follows.

[0207] 1. To describe a parameter set of a specific type, a parameter set sample group entry 'gpsg' may be used. In this way, at most, a sample group description box of type 'gpsg' may only have as many entries as the number of different parameter set types that can be carried (i.e., an entry for each of the SPS, GPS, and APS types). Parameter sets may still be carried in sample entries and / or samples. A sample group may describe parameter sets carried in samples.

[0208] 2. There can be multiple sample-to-group boxes 'sgbp' of type 'gpsg'. Each 'sgbp' of grouping type 'gpsg' must be distinguished by its group_type_parameter to identify which type of parameter set it maps to.

[0209] 3. 'sgbp' of grouping type 'gpsg' can map samples to specific parameter sets. At this time, only samples containing the specific type of parameter can be restricted to have a group_description_index value other than 0. Samples that do not contain a parameter set or samples containing a parameter set of another type can be restricted to have a group_description_index value of 0.

[0210] 4. For normal playback operations, e.g., playback without random access, parameter set sample group boxes (e.g., sample group description boxes and sample-to-group boxes of grouping type 'gpsg') can be ignored.

[0211] 5. For random access from a specific sample SampleX, if there are parameter set sample group boxes, the file parser / player may be restricted to identify samples (either SampleX itself or samples preceding SampleX, which is a random access starting point) containing each of the above parameter set types, and extracting such parameter sets. The extracted parameter sets may be included in SampleX before transmitting it for decryption.

[0212] 6. When parameter set sample group boxes exist, it can be restricted that there will be no nested parameter set referencing such that after the previous parameter set PS_A is updated by a new occurrence of parameter set PS_B in sample X, there will be no samples subsequent to sample X in decoding order and referencing PS_A.

[0213] 7. When parameter set sample group boxes exist, for a parameter set of a particular parameter set type (i.e., SPS, GPS and / or APS) transmitted within a sampleX and referenced by multiple samples, the sampleX may be restricted to be mapped in a sample-to-group box of group_type_parameter of said parameter set type and grouping type 'gpsg'.

[0214] The above design has the following problems and can be improved in two aspects.

[0215] The restrictions discussed in section 6 above are intended to facilitate random access. That is, when a random access occurs, the decoder can easily find the samples containing the required parameter set by finding the closest samples preceding the random access point sample in decoding order from the sample-to-entry box. However, in some situations, this may be too restrictive. In some situations, the decoder may have no problem finding the required parameter set up to the first sync sample preceding the random access point sample in decoding / bitstream order.

[0216] It would be advantageous to have another mode capable of locating samples containing the required parameter set. The present disclosure proposes a method for reconciling conventional designs with the newly proposed method.

[0217] To address the above issues, the present disclosure proposes configurations 1 to 4 below. Each configuration may be applied individually, or two or more may be combined.

[0218] 1) A new syntax element may be signaled to specify the mode in which the parameter set sample group entry is used. This syntax element may be referred to as “mode” or “mode information” in the present disclosure. In the present disclosure, the mode may have a value of 0 or 1. More generally, in the present disclosure, 0 may be represented as the first value, and 1 may be represented as the second value.

[0219] When mode has a value of 0, it may mean that the sample mapped to the entry (i.e., mapped in a sample-to-group box with grouping type 'gpsg') contains a set of parameters of the same type as the grouping_type_parameter of the sample-to-group box.

[0220] When the mode has a value of 1, it may mean that the sample mapped to the entry (i.e., mapped in a sample-to-group box with grouping type 'gpsg') refers to a parameter set contained within a sample corresponding to the G-PCC parameter set sample group entry.

[0221] Configuration 2) When the above mode is 1, the following information can be signaled.

[0222] Offset of the corresponding sample relative to the sync sample

[0223] The number of parameter sets that the corresponding sample contains

[0224] In terms of the G-PCC unit, a flag indicating whether the type of each parameter set and the offset of the parameter set exist within the sample.

[0225] The type of parameter set contained within the corresponding sample

[0226] In terms of the G-PCC unit, the offset of the parameter set within the sample

[0227] Composition 3) According to the present disclosure, at least one of the following limitations may be applied.

[0228] All G-PCC parameter set sample group entries within a same sample group description box may be constrained to have modes with the same value.

[0229] For a G-PCC parameter set sample group entry with mode 1, the number of sample-to-group boxes with grouping type 'gpsg' may be restricted to be no greater than the number of unique values ​​of parameter_set_type. In this case, a sample-to-group box with grouping type 'gpsg' may contain samples mapped to the sample group entry.

[0230] For a G-PCC parameter set sample group entry with mode 1, the value of the syntax element specifying the number of parameter sets in the sample may be constrained to be less than or equal to the number of parameter sets in the sample.

[0231] For a G-PCC parameter set sample group entry with mode 1, it may be restricted that no two parameter set offset values ​​have the same value.

[0232] Configuration 4) Alternatively, the signaling of the mode may not be used. Instead, the signaling of Configuration 2 may be applied by default.

[0233] Hereinafter, various embodiments of the present disclosure applying at least one of the above configurations 1 to 4 will be described.

[0234] Example 1)

[0235] This embodiment can be implemented by a combination of the above configurations 1 and 2.

[0236] A sample group of G-PCC parameter sets according to this embodiment can be defined as shown in Table 1.

[0237] Group Types: 'gpsg'Container: Sample Group Description Box ('sgpd')Mandatory: NoQuantity: Zero or more

[0238] A G-PCC parameter set sample group entry can define parameter set information for all samples referencing shared G-PCC parameter sets. If there are multiple instances of a sample-to-group box SampleToGroupBox with grouping_type equal to 'gpsg', the version of all sample-to-group boxes must be set to 1. A G-PCC parameter set sample group entry can provide information that can help a decoder find the parameter set needed to decode a sample. A G-PCC parameter set sample group entry with mode 0 can mean that the sample mapped to the entry (i.e., mapped in a sample-to-group box with grouping type 'gpsg') contains a parameter set of the same type as the grouping_type_parameter of the sample-to-group box. A G-PCC parameter set sample group entry with mode 1 may mean that the samples mapped to said entry (i.e., mapped in a sample-to-group box with grouping type 'gpsg') refer to a parameter set contained within the samples corresponding to the G-PCC parameter set sample group entry.

[0239] If mode is 0 and a sample group description with grouping_type equal to 'gpsg' exists, the following constraints may apply:

[0240] - If parameter set PS_A is updated by a new occurrence of parameter set PS_B in sample X, there must be no sample that references parameter set PS_A and follows sample X in decoding order.

[0241] - For a parameter set having a specific parameter set type (e.g., SPS, GPS or APS) carried in a sample X and referenced by multiple samples, sample X shall be mapped within a sample-to-group having a grouping type of 'gpsg' and a unique group_type_parameter, based on the type of parameter set present in that sample.

[0242] - Samples that do not contain a parameter set or contain a different type of parameter set must have a group_description_index value of 0.

[0243] When a sample group description with grouping_type equal to 'gpsg' and mode equal to 0 exists, for random access from a particular sample sampleX, the file parser / player shall identify the samples preceding sampleX as the random access starting point (which may be sampleX itself or samples containing each of the parameter set types) and extract such parameter sets so that they can be included within sampleX before transmitting it for decoding.

[0244] When a sample group description with grouping_type of 'gpsg' and mode of 1 exists, for random access from a particular sample SampleX, for each parameter type required to decode SampleX, the file parser / player shall identify the sample group entry to which SampleX is mapped, and if sample offset information and / or parameter set offset information within said sample group entry exist, shall locate the sample containing the required parameter set using them, and shall extract said parameter set so that it can be included within SampleX before transmitting it for decoding.

[0245] Additionally, according to the present embodiment, at least one of the following limitations may be applied.

[0246] All G-PCC parameter set sample group entries within a same sample group description box must have the same value of mode.

[0247] For a G-PCC parameter set sample group entry with mode 1, the number of sample-to-group boxes with grouping type 'gpsg' must not be greater than the number of unique values ​​of parameter_set_type. In this case, a sample-to-group box with grouping type 'gpsg' may contain samples mapped to the sample group entry.

[0248] For example, if the value of num_parameter_sets_minus1 of the samples corresponding to an entry is 2 (i.e., 3 parameter sets), one of the parameter_set_types is 1 (i.e., SPS), and the other two parameter_set_types are 3 (i.e., APS), then since there are only two different parameter types, this entry can only be mapped to samples from at most two sample-to-group boxes whose grouping type is 'gpsg'.

[0249] For a G-PCC parameter set sample group entry with mode 1, the value of the syntax element num_parameter_sets_minus1 plus 1, which specifies the number of parameter sets in the sample, must be less than or equal to the number of parameter sets in that sample.

[0250] For a G-PCC parameter set sample group entry with mode 1, it may be restricted that no two parameter set offset (gpcc_unit_offset) values ​​have the same value.

[0251] The syntax for a G-PCC parameter set sample group entry according to the present embodiment may be as follows.

[0252]

[0253] In the above, the mode can indicate the information contained in the corresponding sample group entry. A value of 0 can specify that the corresponding entry represents a specific parameter set type. A value of 1 can specify that the corresponding entry represents a sample including one or more G-PCC parameter sets.

[0254] If parameter_set_info_present is 1 in the above, it can be specified that the corresponding parameter set type and its offset information exist. If parameter_set_info_present is 0, it can be specified that the corresponding parameter set type and its offset information do not exist.

[0255] The value obtained by adding 1 to num_parameter_sets_minus1 above can specify the number of G-PCC parameter sets included in the sample corresponding to the entry.

[0256] In the above, sample_offset can specify a sample corresponding to the entry. The (sample_offset+1)th sample after the sync sample can be the sample corresponding to the entry. The value of sample_offset must be 0 or greater. If sample_offset is 0, the first sample after the sync sample can be the sample corresponding to the entry.

[0257] In the above, parameter_set_type can specify the type of the i-th parameter set according to Table 2 below.

[0258] Parameterset typeDescription0Reserved1Sequence Parameter set2Geometry parameter set3Attribute parameter set4 to 255Reserved

[0259] In the above, parameter_set_offset can specify that the i-th parameter set is the (parameter_set_offset+1)-th G-PCC unit in the sample corresponding to the entry. The value of parameter_set_offset must be 0 or greater. When parameter_set_offset is 0, the parameter set may be the first G-PCC unit in the sample corresponding to the entry. Hereinafter, the operation of the player according to the present embodiment during random access will be described.

[0260] When randomly accessing a specific sample SampleX, the file player can perform the following actions:

[0261] Step1) If the sample entry of the track is 'gpe1' or 'gpc1', the file player can start decoding and playing directly from sample X. The parameter set required to decode sample X and the samples that follow it in the decoding order are available in the sample entry of the track.

[0262] Step2) Otherwise, if the SyncSampleBox exists and SampleX is marked as a Sync Sample, the file player can start decoding and playing directly from SampleX. The parameter set required to decode SampleX and the samples that follow it in the decoding order must be available either in the sample entry of the track or in SampleX.

[0263] Step3) Otherwise, if parameter set sample group boxes exist and the mode of the sample group entry in said box is 0, the file player can locate samples containing parameter sets (i.e., SPS, GPS and / or APS) required for decoding starting from sample X from the information provided by said parameter set sample group boxes. The samples containing the required parameter sets may be sample X itself or samples preceding sample X. Once such samples are located, the file parser can extract all parameter sets in such samples and make them available for decoding / playback starting from sample X.

[0264] Step4) Otherwise, if parameter set sample group boxes exist and the mode of the sample group entry in said box is 1, the file player can locate samples containing parameter sets (i.e., SPS, GPS and / or APS) required for decoding starting from sample X from the information provided by said parameter set sample group boxes. Samples containing the required parameter sets must be present in samples corresponding to parameter set sample group entries mapped to sample X. Once such samples are located, the file parser can extract all parameter sets within such samples and make them available for decoding / playback starting from sample X.

[0265] Step5) Otherwise, i.e., if the track is not 'gpe1' or 'gpc1', there is no sync sample box, and there are no parameter set sample group boxes, then random access from sampleX is not specified.

[0266] Steps 1 to 5 described above may be performed sequentially. However, this is not limited to this, and some steps may be omitted or the order may be changed.

[0267] According to this embodiment, it is possible to more easily find the location of a sample containing a parameter set required for random access.

[0268] Example 2)

[0269] This embodiment can be implemented by the above configuration 3.

[0270] A sample group of G-PCC parameter sets according to this embodiment can be defined as in Table 3.

[0271] Group Types: 'gpsg'Container: Sample Group Description Box ('sgpd')Mandatory: NoQuantity: Zero or more

[0272] A G-PCC parameter set sample group entry may define parameter set information for all samples referencing shared G-PCC parameter sets. If there are multiple instances of the sample-to-group box SampleToGroupBox with a grouping_type equal to 'gpsg', the version of all sample-to-group boxes must be set to 1. A G-PCC parameter set sample group entry may provide information that may help a decoder find the parameter set needed to decode a sample. Each G-PCC parameter set sample group entry may correspond to a sample containing the parameter set(s) needed to decode the sample mapped to the entry.

[0273] For random access from a particular sample SampleX, for each parameter type required to decode SampleX, the file parser / player must identify the sample group entry to which said SampleX is mapped, use sample offset information and / or parameter set offset information, if present in said sample group entry, to locate the sample containing the required parameter set, and extract said parameter set so that it can be included in SampleX before transmitting it for decoding.

[0274] Additionally, according to the present embodiment, at least one of the following limitations may be applied.

[0275] The number of sample-to-group boxes with grouping type 'gpsg' must not be greater than the number of unique values ​​of parameter_set_type. In this case, a sample-to-group box with grouping type 'gpsg' may contain samples mapped to sample group entries.

[0276] For example, if the value of num_parameter_sets_minus1 of the samples corresponding to an entry is 2 (i.e., 3 parameter sets), one of the parameter_set_types is 1 (i.e., SPS), and the other two parameter_set_types are 3 (i.e., APS), then since there are only two different parameter types, this entry can only be mapped to samples from at most two sample-to-group boxes whose grouping type is 'gpsg'.

[0277] The value of the syntax element num_parameter_sets_minus1 plus 1, which specifies the number of parameter sets in the sample, must be less than or equal to the number of parameter sets in the sample.

[0278] It can be restricted that no two parameter set offset (gpcc_unit_offset) values ​​have the same value.

[0279] The syntax for a G-PCC parameter set sample group entry according to the present embodiment may be as follows.

[0280]

[0281] If parameter_set_info_present is 1 in the above, it can be specified that the corresponding parameter set type and its offset information exist. If parameter_set_info_present is 0, it can be specified that the corresponding parameter set type and its offset information do not exist.

[0282] The value obtained by adding 1 to num_parameter_sets_minus1 above can specify the number of G-PCC parameter sets included in the sample corresponding to the entry.

[0283] In the above, sample_offset can specify a sample corresponding to the entry. The (sample_offset+1)th sample after the sync sample can be the sample corresponding to the entry. The value of sample_offset must be 0 or greater. If sample_offset is 0, the first sample after the sync sample can be the sample corresponding to the entry.

[0284] In the above, parameter_set_type can specify the type of the i-th parameter set according to Table 2 above.

[0285] In the above, parameter_set_offset can specify that the i-th parameter set is the (parameter_set_offset+1)-th G-PCC unit in the sample corresponding to the entry. The value of parameter_set_offset must be greater than or equal to 0. When parameter_set_offset is 0, the parameter set may be the first G-PCC unit in the sample corresponding to the entry.

[0286] Below, the player's operation according to this embodiment during random access is described.

[0287] When randomly accessing a specific sample SampleX, the file player can perform the following actions:

[0288] Step1) If the sample entry of the track is 'gpe1' or 'gpc1', the file player can start decoding and playing directly from sample X. The parameter set required to decode sample X and the samples that follow it in the decoding order are available in the sample entry of the track.

[0289] Step2) Otherwise, if the SyncSampleBox exists and SampleX is marked as a Sync Sample, the file player can start decoding and playing directly from SampleX. The parameter set required to decode SampleX and the samples that follow it in the decoding order must be available either in the sample entry of the track or in SampleX.

[0290] Step3) Otherwise, if parameter set sample group boxes exist, the file player can locate samples containing parameter sets (i.e., SPS, GPS and / or APS) required for decoding starting from sample X from the information provided by said parameter set sample group boxes. Samples containing the required parameter sets must be present in the samples corresponding to the parameter set sample group entries mapped to sample X. Once such samples are located, the file parser can extract all parameter sets within such samples and make them available for decoding / playback starting from sample X.

[0291] Step4) Otherwise, i.e., if the track is not 'gpe1' or 'gpc1', there is no sync sample box, and there are no parameter set sample group boxes, then random access from sampleX is not specified.

[0292] Steps 1 to 4 described above may be performed sequentially. However, this is not limited to this, and some steps may be omitted or the order may be changed.

[0293] According to this embodiment, it is possible to more easily find the location of a sample containing a parameter set required for random access without signaling separate mode information.

[0294] Hereinafter, with reference to FIGS. 14 and 15, a method performed in a receiving device for point cloud data according to the present disclosure and a method performed in a transmitting device for point cloud data will be described.

[0295] A method performed in a receiving device of point cloud data according to the present disclosure may include a step of identifying one or more tracks from point cloud data (S1401), a step of extracting one or more samples from the tracks (S1402), and a step of decoding the samples based on a parameter set (S1403).

[0296] The parameter set according to the present disclosure can be obtained based on information included in a geometry-based point cloud compression (G-PCC) parameter set sample group entry.

[0297] In a method performed in a receiving device of point cloud data according to the present disclosure, information included in the G-PCC parameter set sample group entry may include mode information indicating a mode of the G-PCC parameter set sample group entry.

[0298] In a method performed in a receiving device of point cloud data according to the present disclosure, based on the mode information having a first value (e.g., “0”), a sample mapped within a sample-to-group box of grouping type ‘gpsg’ may include a parameter set of the same type as the grouping_type_parameter of the sample-to-group box.

[0299] In a method performed in a receiving device of point cloud data according to the present disclosure, based on the mode information having a second value (e.g., “1”), a sample mapped within a sample-to-group box of grouping type ‘gpsg’ can refer to a parameter set included in a sample corresponding to the G-PCC parameter set sample group entry.

[0300] A method performed in a receiving device of point cloud data according to the present disclosure may further include a step of identifying samples including each of parameter set types as samples preceding a sample X, which is a random access starting point, based on the presence of a sample group description in which the grouping type is 'gpsg' and the mode information is a first value, a step of extracting the parameter sets, and a step of including the extracted parameter sets in the sample X.

[0301] A method performed in a receiving device of point cloud data according to the present disclosure may further include a step of identifying a sample group entry to which a sample X, which is a random access starting point, is mapped based on the presence of a sample group description in which a grouping type is 'gpsg' and the mode information is a second value, a step of determining a position of a sample including a parameter set using at least one of sample offset information or parameter set offset information within the sample group entry, a step of extracting the parameter set, and a step of including the extracted parameter set in the sample X.

[0302] In a method performed in a receiving device of point cloud data according to the present disclosure, all G-PCC parameter set sample group entries within one sample group description box may be restricted to have the same value of the above mode information.

[0303] In a method performed in a receiving device of point cloud data according to the present disclosure, for the G-PCC parameter set sample group entry having the mode information having the second value, the grouping type is 'gpsg', and the number of sample-to-group boxes including samples mapped to the sample group entry may be limited to be no greater than the number of unique values ​​of parameter_set_type.

[0304] In a method performed in a receiving device of point cloud data according to the present disclosure, for the G-PCC parameter set sample group entry in which the mode information has a second value, a value of a syntax element specifying the number of parameter sets in the sample may be restricted to be equal to or less than the number of parameter sets in the sample.

[0305] In a method performed in a receiving device of point cloud data according to the present disclosure, for the G-PCC parameter set sample group entry having the second value of the mode information, it may be restricted that two parameter set offset values ​​of the same value do not exist.

[0306] A method performed in a transmission device of point cloud data according to the present disclosure may include a step (S1501) of storing a bitstream including the point cloud data in one or more tracks, and a step (S1502) of generating a G-PCC (geometry-based point cloud compression) file based on the tracks.

[0307] In a method performed in a transmission device of point cloud data according to the present disclosure, the tracks include one or more samples, at least one of the samples includes a parameter set, and the parameter set can be encoded based on information included in a G-PCC parameter set sample group entry.

[0308] In a method performed in a transmission device of point cloud data according to the present disclosure, information included in the G-PCC parameter set sample group entry may include mode information indicating a mode of the G-PCC parameter set sample group entry.

[0309] As described above, the second embodiment of the present disclosure does not signal mode information. In this case, the method according to the second embodiment of the present disclosure can operate in the same manner as the method in the first embodiment when the mode information is the second value. The description of the method performed in the device for receiving point cloud data according to the present disclosure can be applied in the same manner or in a corresponding manner to the method performed in the device for transmitting point cloud data according to the present disclosure.

[0310] While the exemplary methods of this disclosure are presented as a series of operations for clarity of description, this is not intended to limit the order in which the steps are performed, and individual steps may be performed simultaneously or in different orders, if desired. To implement a method according to this disclosure, additional steps may be included in addition to the steps illustrated, some steps may be excluded and the remaining steps included, or some steps may be excluded and additional steps included.

[0311] In the present disclosure, a transmitting device or receiving device performing a predetermined operation (step) may perform an operation (step) that verifies the conditions or circumstances under which the operation (step) is performed. For example, if it is described that a predetermined operation is performed when a predetermined condition is satisfied, a video encoding device or video decoding device may perform an operation to verify whether the predetermined condition is satisfied and then perform the predetermined operation.

[0312] The various embodiments of the present disclosure are not intended to list all possible combinations but rather to illustrate representative aspects of the present disclosure, and the matters described in the various embodiments may be applied independently or in combination of two or more.

[0313] Additionally, various embodiments of the present disclosure may be implemented by hardware, firmware, software, or a combination thereof. In the case of hardware implementation, the embodiments may be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), general processors, controllers, microcontrollers, microprocessors, etc.

[0314] The scope of the present disclosure includes software or machine-executable instructions (e.g., operating systems, applications, firmware, programs, etc.) that cause operations according to the methods of various embodiments to be executed on a device or a computer, and a non-transitory computer-readable medium having such software or instructions stored thereon and executable on the device or computer.

[0315] Embodiments according to the present disclosure can be used to provide point cloud content. Furthermore, embodiments according to the present disclosure can be used to encode / decode point cloud data.

Claims

1. A method performed in a receiving device of point cloud data, A step of identifying one or more tracks from the above point cloud data; extracting one or more samples from said tracks; and comprising a step of decoding the samples based on a set of parameters, A method wherein the above parameter set is obtained based on information included in a G-PCC (geometry-based point cloud compression) parameter set sample group entry.

2. In paragraph 1, A method wherein the information included in the above G-PCC parameter set sample group entry includes mode information indicating the mode of the above G-PCC parameter set sample group entry.

3. In paragraph 2, A method wherein a sample mapped within a sample-to-group box of grouping type 'gpsg' includes a parameter set of the same type as the grouping_type_parameter of the sample-to-group box, based on the above mode information having a first value.

4. In paragraph 2, A method in which a sample mapped within a sample-to-group box of grouping type 'gpsg' refers to a parameter set included in a sample corresponding to the G-PCC parameter set sample group entry, based on the above mode information having a second value.

5. In paragraph 3, A step of identifying samples including each of the parameter set types as samples preceding sample X, which is a random access starting point, based on the existence of a sample group description in which the grouping type is 'gpsg' and the mode information is a first value; a step of extracting the above parameter sets; and A method further comprising the step of including the extracted parameter sets in the sample X.

6. In paragraph 4, A step of identifying a sample group entry to which a sample X, which is a random access starting point, is mapped based on the existence of a sample group description whose grouping type is 'gpsg' and whose mode information is a second value; A step of determining the position of a sample including a parameter set using at least one of sample offset information or parameter set offset information within the sample group entry; a step of extracting the above parameter set; and A method further comprising the step of including the extracted parameter set within the sample X.

7. In paragraph 2, A method wherein all G-PCC parameter set sample group entries within a sample group description box are restricted to have the same value of the above mode information.

8. In paragraph 2, A method wherein, for the G-PCC parameter set sample group entry having the above mode information having the second value, the grouping type is 'gpsg', and the number of sample-to-group boxes containing samples mapped to the sample group entry is restricted to be no greater than the number of unique values ​​of parameter_set_type.

9. In paragraph 2, A method wherein, for the G-PCC parameter set sample group entry having the second value of the mode information, the value of the syntax element specifying the number of parameter sets in the sample is restricted to be equal to or less than the number of parameter sets in the sample.

10. In paragraph 2, A method wherein, for the G-PCC parameter set sample group entry having the second value of the above mode information, there are no two parameter set offset values ​​having the same value.

11. A method performed in a transmission device of point cloud data, A step of storing a bitstream including the above point cloud data in one or more tracks; and Including a step of generating a G-PCC (geometry-based point cloud compression) file based on the above tracks, The above tracks comprise one or more samples, at least one of the samples comprising a parameter set, A method wherein the above parameter set is encoded based on information included in a G-PCC parameter set sample group entry.

Images