Point cloud data transmitting device, method executed by transmitting device, point cloud data receiving device, and method executed by receiving device

Abstract

A transmitting apparatus, a method performed by the transmitting apparatus, a receiving apparatus, and a method performed by the receiving apparatus for point cloud data are provided. According to the present disclosure, a method performed by a receiving device for point cloud data comprises the steps of: obtaining a parameter set required for random access from point cloud data on the basis of a random access request from a first sample among samples included in the point cloud data; and decoding the first sample and one or more samples following the first sample according to an order of decoding the samples based on the parameter set, in which at least one of the presence of a sample frame and the presence of a sample group frame of the parameter set is synchronized based on a sample entry type of the track, a set of parameters may be obtained from a sample entry of a track including a first sample or a predetermined sample.

Description

Technical Field

[0001] The present disclosure relates to methods and apparatus for processing point cloud content. Background Art

[0002] Point cloud content is expressed as a point cloud that is a set of points belonging to a coordinate system representing a three-dimensional space. Point cloud content can represent three-dimensional media and is used to provide various services such as virtual reality (VR), augmented reality (AR), mixed reality (MR), and self-driving services. Since tens of thousands to hundreds of thousands of point data are required to express point cloud content, a method for efficiently processing a large amount of point data is needed. Summary of the invention

[0003] Technical issues

[0004] The purpose of the present disclosure is to provide a method and apparatus for efficiently processing point cloud data.

[0005] In addition, an object of the present disclosure is to provide a method and apparatus for supporting a correct random access function.

[0006] In addition, an object of the present disclosure is to provide a method and apparatus for providing a random access function based on a sample entry type of a track to which the random access sample belongs.

[0007] In addition, an object of the present disclosure is to provide a method and apparatus for providing a random access function based on whether a synchronization sample box or a parameter set sample group box exists.

[0008] Additionally, it is an object of the present disclosure to provide methods and apparatus for handling file storage techniques to support efficient access to stored G-PCC bitstreams.

[0009] The technical problems solved by the present disclosure are not limited to the above-mentioned technical problems, and a person skilled in the art in the technical field to which the present disclosure belongs will clearly understand other technical problems not described herein according to the following description.

[0010] Technical Solution

[0011] According to one embodiment of the present disclosure, a method performed by a receiving device of point cloud data may include the following steps: based on a request for random access starting from a first sample among samples included in the point cloud data, obtaining a parameter set required for the random access from the point cloud data, and decoding the first sample and one or more samples after the first sample in a decoding order based on the parameter set. The parameter set may be obtained from a sample entry or a predetermined sample of a track including the first sample based on at least one of a sample entry type of the track, the presence of a synchronization sample box, or the presence of a parameter set sample group box.

[0012] According to another embodiment of the present disclosure, a receiving device of point cloud data may include a memory and at least one processor. The at least one processor may be configured to: based on a request for random access starting from a first sample among samples included in the point cloud data, obtain a parameter set required for the random access from the point cloud data, and decode the first sample and one or more samples after the first sample in a decoding order based on the parameter set. The parameter set may be obtained from a sample entry or a predetermined sample of a track including the first sample based on at least one of a sample entry type of the track, the presence of a synchronization sample box, or the presence of a parameter set sample group box.

[0013] According to another embodiment of the present disclosure, a method performed by a transmitting device of point cloud data may include the following steps: storing a bitstream including the point cloud data into one or more tracks, and generating a geometry-based point cloud compression (G-PCC) file based on the tracks. Based on at least one of a sample entry type of the track, the presence of a synchronization sample box, or the presence of a parameter set sample group box, a parameter set required for random access starting from a first sample within the point cloud data may be included in a sample entry or a predetermined sample of the track including the first sample.

[0014] According to another embodiment of the present disclosure, a device for transmitting point cloud data may include a memory and at least one processor. The at least one processor may be configured to store a bit stream including the point cloud data into one or more tracks, and generate a geometry-based point cloud compression (G-PCC) file based on the tracks. Based on at least one of a sample entry type of the track, the presence of a synchronization sample box, or the presence of a parameter set sample group box, a parameter set required for random access starting from a first sample in the point cloud data may be included in a sample entry or a predetermined sample of the track including the first sample.

[0015] Beneficial Effects

[0016] According to the present disclosure, a method and apparatus for efficiently processing point cloud data can be provided.

[0017] In addition, according to the present disclosure, a method and an apparatus for supporting a correct random access function may be provided.

[0018] In addition, according to the present disclosure, a method and apparatus for providing a point cloud content service for processing random access based on a sample entry type of a track to which the random access sample belongs may be provided.

[0019] In addition, according to the present disclosure, a method and apparatus for providing a point cloud content service for processing random access based on whether a synchronization sample box or a parameter set sample group box exists may be provided.

[0020] In addition, according to the present disclosure, methods and apparatus for processing file storage technology to support efficient access to stored G-PCC bitstreams may be provided.

[0021] Effects that can be obtained from the present disclosure are not limited to the above-mentioned effects, and other effects not mentioned herein will be clearly understood by those skilled in the art through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a block diagram illustrating an example of a point cloud content providing system according to an embodiment of the present disclosure.

[0023] Figure 2 is a block diagram illustrating an example of a point cloud content providing process according to an embodiment of the present disclosure.

[0024] Figure 3 is a diagram illustrating an example of a point cloud encoding device according to an embodiment of the present disclosure.

[0025] Figure 4 is a block diagram illustrating an example of a point cloud decoding device according to an embodiment of the present disclosure.

[0026] Figure 5 is a block diagram illustrating another example of a point cloud decoding device according to an embodiment of the present disclosure.

[0027] Figure 6 is a block diagram illustrating another example of a transmitting device according to an embodiment of the present disclosure.

[0028] Figure 7 is a block diagram showing another example of a receiving device according to an embodiment of the present disclosure.

[0029] Figure 8 An example of a structure capable of interacting with a method / apparatus for transmitting and receiving point cloud data according to an embodiment of the present disclosure is illustrated.

[0030] Fig. 9is a block diagram showing still another example of the transmitting device according to an embodiment of the present disclosure.

[0031] Fig.10 An example of spatially segmenting a bounding box into three-dimensional blocks according to an embodiment of the present disclosure is illustrated.

[0032] Fig.11 is a block diagram showing another example of a receiving device according to an embodiment of the present disclosure.

[0033] Fig.12 An example of a tile including a single track according to an embodiment of the present disclosure is illustrated.

[0034] Fig.13 An example of a tile including a plurality of tracks according to an embodiment of the present disclosure is illustrated.

[0035] Fig.14 is a flowchart illustrating a method for processing random access according to an embodiment of the present disclosure.

[0036] Fig.15 Detailed description is a flowchart illustrating a method executed in a receiving device for point cloud data according to an embodiment of the present disclosure.

[0037] Fig.16 Detailed description is a flowchart illustrating a method executed in a point cloud data transmitting device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0038] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art to which the present disclosure belongs can easily implement them. The present disclosure can be implemented in many different forms and is not limited to the embodiments described herein.

[0039] In describing the present disclosure, detailed descriptions of known functions and configurations may be omitted when they may obscure the gist of the present disclosure. In addition, in the drawings, parts irrelevant to the description of the present disclosure are omitted, and like reference numerals refer to like parts.

[0040] In the present disclosure, when one component is "connected", "coupled" or "linked" to another component, it may include not only a direct connection relationship but also an indirect connection relationship with another component therebetween. In addition, when a component "includes" or "has" other components, unless the context clearly indicates otherwise, this means that other components may be further included rather than excluded.

[0041] In the present disclosure, terms such as first and second are used only for the purpose of distinguishing one component from other components, and unless otherwise specified, do not limit the order or importance of the components. Therefore, within the scope of the present disclosure, a first component in one embodiment may be referred to as a second component in another embodiment, and similarly, a second component in one embodiment may be referred to as a first component in another embodiment.

[0042] In the present disclosure, components that are distinguished from each other are intended to clearly describe each of their features, and do not necessarily mean that these components are separate from each other. That is, multiple components can be integrated into one hardware or software unit, or one component can be distributed to be configured according to multiple hardware or software units. Therefore, even if not specifically mentioned, such integrated or distributed implementations are also included in the scope of the present disclosure.

[0043] In the present disclosure, the components described in various embodiments are not necessarily essential components, and some of them may be optional components. Therefore, an embodiment consisting of a subset of the components described in an embodiment is also included in the scope of the present disclosure. In addition, an embodiment including other components in addition to the components described in various embodiments is also included in the scope of the present disclosure.

[0044] The present disclosure relates to encoding and decoding of point cloud related data, and unless the terms used in the present disclosure are redefined in the present disclosure, the terms may have general meanings commonly used in the technical field to which the present disclosure belongs.

[0045] In the present disclosure, the terms " / " and "," should be interpreted as indicating "and / or". For example, "A / B" and "A, B" may be interpreted as indicating "A and / or B". In addition, "A / B / C" and "A, B, C" may mean "at least one of A, B and / or C".

[0046] In the present disclosure, the term "or" should be interpreted as indicating "and / or". For example, the expression "A or B" may include 1) only "A", 2) only "B", and / or 3) both "A and B". In other words, in the present disclosure, the term "or" should be interpreted as indicating "additionally or alternatively".

[0047] The present disclosure relates to compression of point cloud related data. Various methods or embodiments of the present invention can be applied to point cloud compression or point cloud coding (PCC) standards of the Moving Picture Experts Group (MPEG) (e.g., G-PCC or V-PCC standards) or next generation video / image coding standards.

[0048] In the present disclosure, "point cloud" may refer to a set of points located in a three-dimensional space. Moreover, in the present disclosure, "point cloud content" is expressed as a point cloud, and may refer to a "point cloud video / image". Hereinafter, a "point cloud video / image" is referred to as a "point cloud video". A point cloud video may include one or more frames, and a frame may be a still image or a picture. Therefore, a point cloud video may include a point cloud image / frame / picture, and may be referred to as any one of a "point cloud image", a "point cloud frame", and a "point cloud picture".

[0049] In the present disclosure, "point cloud data" may refer to data or information related to each point in a point cloud. Point cloud data may include geometry and / or attributes. In addition, point cloud data may also include metadata. Point cloud data may be referred to as "point cloud content data" or "point cloud video data", etc. In addition, point cloud data may be referred to as "point cloud content", "point cloud video", "G-PCC data", etc.

[0050] In the present disclosure, a point cloud object corresponding to point cloud data may be represented by a frame shape based on a coordinate system, and the frame shape based on a coordinate system may be referred to as a bounding box. That is, the bounding box may be a cuboid that can accommodate all points of the point cloud, and may be a cuboid that includes a source point cloud frame.

[0051] In the present disclosure, geometry includes the position (or position information) of each point, and the position can be represented by parameters (e.g., x-axis value, y-axis value, and z-axis value) representing a three-dimensional coordinate system (e.g., a coordinate system composed of an x-axis, a y-axis, and a z-axis). Geometry can be referred to as "geometric information".

[0052] In the present disclosure, the attribute may include the attribute of each point, and the attribute may include one or more of texture information, color (RGB or YCbCr), reflectivity (r), transparency, etc. of each point. The attribute may be referred to as "attribute information". Metadata may include various data related to acquisition in an acquisition process to be described later.

[0053] Overview of Point Cloud Content Provision System

[0054] Figure 1 An example of a system for providing point cloud content (hereinafter, referred to as a “point cloud content providing system”) according to an embodiment of the present disclosure is illustrated. Figure 2 An example of a process in which a point cloud content providing system provides point cloud content is illustrated.

[0055] like Figure 1 As shown, the point cloud content providing system may include a sending device 10 and a receiving device 20. The point cloud content providing system may be performed by the operations of the sending device 10 and the receiving device 20. Figure 2The obtained process S20, the encoded process S21, the sent process S22, the decoded process S23, the rendered process S24 and / or the feedback process S25 are shown.

[0056] The sending device 10 obtains point cloud data, and outputs a bit stream through a series of processes (e.g., encoding process) for the obtained point cloud data (source point cloud data) to provide point cloud content. Here, the point cloud data can be output in the form of a bit stream through the encoding process. In some embodiments, the sending device 10 can send the output bit stream to the receiving device 20 in the form of a file or streaming (streaming segment) through a digital storage medium or a network. The digital storage medium may include a variety of storage media, such as USB, SD, CD, DVD, Blu-ray disc, HDD, and SSD. The receiving device 20 can process (e.g., decode or reconstruct) the received data (e.g., encoded point cloud data) into source point cloud data and render it. Point cloud content can be provided to users through these processes, and the present disclosure can provide various embodiments required to efficiently perform a series of these processes.

[0057] like Figure 1 As illustrated, the sending device 10 may include: an acquisition unit 11, an encoding unit 12, an encapsulation processing unit 13 and a sending unit 14, and the receiving device 20 may include: a receiving unit 21, a decapsulation processing unit 22, a decoding unit 23, and a rendering unit 24.

[0058] The acquisition unit 11 may perform a process S20 of acquiring a point cloud video through a capturing, composing or generating process. Therefore, the acquisition unit 11 may be referred to as a 'point cloud video acquisition unit'.

[0059] Point cloud data (geometry and / or attributes, etc.) of a plurality of points may be generated through the acquisition process (S20). Furthermore, metadata related to the acquisition of the point cloud video may be generated through the acquisition process (S20). Furthermore, mesh data (e.g., triangle data) indicating connection information between point clouds may be generated through the acquisition process (S20).

[0060] The metadata may include initial viewing orientation metadata. The initial viewing orientation metadata may indicate whether the point cloud data represents front-side or back-side data. The metadata may be referred to as "auxiliary data," which is metadata of the point cloud.

[0061] The obtained point cloud video may include a polygon file format or a Stanford triangle format (PLY) file. Since the point cloud video has one or more frames, the obtained point cloud video may include one or more PLY files. The PLY file may include point cloud data of each point.

[0062] In order to obtain a point cloud video (or point cloud data), the acquisition unit 11 may be composed of a camera device capable of obtaining depth (depth information) and a combination of an RGB camera capable of extracting color information corresponding to the depth information. Here, the camera device capable of obtaining depth information may be a combination of an infrared pattern projector and an infrared camera. In addition, the acquisition unit 11 may be composed of a laser radar (LiDAR), and the laser radar may use a radar system for measuring the position coordinates of a reflector by measuring the time required for a laser pulse to be emitted and to return after being reflected.

[0063] The acquisition unit 110 may extract a shape of geometry composed of points in a three-dimensional space from the depth information, and may extract a property representing the color or reflection of each point from the RGB information.

[0064] As a method of extracting (or capturing, obtaining, etc.) a point cloud video (or point cloud data), there may be an inward-facing method of capturing a central object and an outward-facing method of capturing an external environment.

[0065] In addition, when a point cloud video is to be provided with respect to a virtual space generated by a computer, the capture should be performed by a real camera. In this case, post-processing may be required to enhance the quality of the captured point cloud content. For example, the acquisition process (S20) can adjust the maximum / minimum depth value within the range provided by the camera device, but post-processing can be performed to remove point data of unexpected areas (e.g., background) or unexpected areas, or post-processing can be performed to identify connected spaces and fill spatial holes. As another example, post-processing can be performed to integrate point cloud data extracted from cameras sharing a spatial coordinate system into one content by converting each point to a global coordinate system based on the position coordinates of each camera. Therefore, a point cloud content can be generated over a wide range, or point cloud content with a high density of points can be obtained.

[0066] The encoding unit 12 may perform an encoding process (S21) of encoding the data generated by the acquisition unit 11 (e.g., geometry, attributes and / or metadata, and / or mesh data, etc.) into one or more bitstreams. Therefore, the encoding unit 12 may be referred to as a "point cloud video encoder". The encoding unit 12 may encode the data generated by the acquisition unit 11 serially or in parallel.

[0067] The encoding process S21 performed by the encoding unit 12 may be geometry-based point cloud compression (G-PCC). The encoding unit 12 may perform a series of processes such as prediction, transform, quantization, and entropy encoding for compression and encoding efficiency.

[0068] The encoded point cloud data can be output in the form of a bitstream. Based on the G-PCC process, the encoding unit 12 can split the point cloud data into geometry and attributes and encode them as described below. In this case, the output bitstream may include a geometry bitstream containing the encoded geometry and an attribute bitstream containing the encoded attributes. In addition, the output bitstream may also include one or more of a metadata bitstream containing metadata, an auxiliary bitstream containing auxiliary data, and a mesh data bitstream containing mesh data. The encoding process (S21) will be described in more detail below. The bitstream including the encoded point cloud data may be referred to as a "point cloud bitstream" or a "point cloud video bitstream".

[0069] The encapsulation processing unit 13 may perform a process of encapsulating one or more bit streams output from the encoding unit 12 in the form of a file or segment. Therefore, the encapsulation processing unit 13 may be referred to as a "file / segment encapsulation module". Although the drawings show an example in which the encapsulation processing unit 13 is composed of a separate component / module related to the sending unit 14, in some embodiments, the encapsulation processing unit 13 may be included in the sending unit 14.

[0070] The encapsulation processing unit 13 may encapsulate data in a file format such as ISO Base Media File Format (ISOBMFF), or process data in the form of other DASH segments. In some embodiments, the encapsulation processing unit 13 may include metadata of the file format. The metadata may be included in boxes at various levels of, for example, the ISOBMFF file format, or as data in a separate track within a file. In some embodiments, the encapsulation processing unit 130 may encapsulate the metadata itself into a file. The metadata processed by the encapsulation processing unit 13 may be sent from a metadata processing unit not shown in the figure. The metadata processing unit may be included in the encoding unit 12, or may be configured as a separate component / module.

[0071] The sending unit 14 may perform a sending process (S22) of applying a process according to a file format (a process for sending) to the “encapsulated point cloud bitstream”. The sending unit 140 may send a bitstream or a file / segment including a bitstream to the receiving unit 21 of the receiving device 20 through a digital storage medium or a network. Therefore, the sending unit 14 may be referred to as a “sender” or a “communication module”.

[0072] The sending unit 14 may process the point cloud data according to any transmission protocol. Here, "processing the point cloud data according to any transmission protocol" may be "processing for sending". The processing for sending may include processing for sending via a broadcast network / processing for sending via broadband, etc. In some embodiments, the sending unit 14 may receive not only the point cloud data from the metadata processing unit, but also metadata from the metadata processing unit, and may perform processing for sending on the sent metadata. In some embodiments, the processing for sending may be performed by the sending processing unit, and the sending processing unit may be included in the sending unit 14, or configured as a component / module separate from the sending unit 14.

[0073] The receiving unit 21 can receive the bitstream or the file / segment including the bitstream sent by the sending device 10. According to the channel sent, the receiving unit 21 can receive the bitstream or the file / segment including the bitstream through a broadcast network, or can receive the bitstream or the file / segment including the bitstream through a broadband. Alternatively, the receiving unit 21 can receive the bitstream or the file / segment including the bitstream through a digital storage medium.

[0074] The receiving unit 21 may perform processing on the received bitstream or the file / segment including the bitstream according to the transmission protocol. The receiving unit 21 may perform the inverse process of the sending process (processing for sending) to correspond to the processing for sending performed by the sending device 10. The receiving unit 21 may send the encoded point cloud data among the received data to the decapsulation processing unit 22, and may send metadata to the metadata parsing unit. The metadata may be in the form of a signaling table. In some embodiments, the inverse process of the processing for sending may be performed in the receiving processing unit. Each of the receiving processing unit, the decapsulation processing unit 22, and the metadata parsing unit may be included in the receiving unit 21, or may be configured as a component / module separate from the receiving unit 21.

[0075] The decapsulation processing unit 22 can decapsulate the point cloud data in the file format (ie, the bit stream in the file format) received from the receiving unit 21 or the receiving processing unit. Therefore, the decapsulation processing unit 22 can be called a "file / segment decapsulation module".

[0076] The decapsulation processing unit 22 can obtain a point cloud bitstream or a metadata bitstream by decapsulating the file according to ISOBMFF, etc. In some embodiments, metadata (metadata bitstream) can be included in the point cloud bitstream. The obtained point cloud bitstream can be sent to the decoding unit 23, and the obtained metadata bitstream can be sent to the metadata processing unit. The metadata processing unit can be included in the decoding unit 23, or can be configured as a separate component / module. The metadata obtained by the decapsulation processing unit 22 can be in the form of a box or track in the file format. If necessary, the decapsulation processing unit 22 can receive the metadata required for decapsulation from the metadata processing unit. The metadata can be sent to the decoding unit 23 and used in the decoding process (S23), or can be sent to the rendering unit 24 and used in the rendering process (S24).

[0077] The decoding unit 23 may receive the bitstream and perform operations corresponding to those of the encoding unit 12, thereby performing a decoding process (S23) of decoding the point cloud bitstream (encoded point cloud data). Therefore, the decoding unit 23 may be referred to as a "point cloud video decoder".

[0078] The decoding unit 23 may segment the point cloud data into geometry and attributes and decode them. For example, the decoding unit 23 may reconstruct (decode) the geometry from the geometry bitstream included in the point cloud bitstream, and restore (decode) the attributes based on the reconstructed geometry and the attribute bitstream included in the point cloud bitstream. The three-dimensional point cloud video / image may be reconstructed based on the position information according to the reconstructed geometry and the attributes (such as color or texture) according to the decoded attributes. The decoding process (S23) will be described in more detail below.

[0079] The rendering unit 24 may perform a rendering process S24 of rendering the reconstructed point cloud video. Therefore, the rendering unit 24 may be referred to as a “renderer”.

[0080] The rendering process (S24) may refer to a process of rendering and displaying point cloud content in a 3D space. The rendering process S24 may perform rendering according to a desired rendering method based on the position information and attribute information of the points decoded by the decoding process.

[0081] The points of the point cloud content may be rendered as vertices with a predetermined thickness, a cube with a specific minimum size centered at the corresponding vertex position, or a circle centered at the vertex position. The user may see the entire or partial area of ​​the rendered result through a VR / AR display or a public display. The rendered video may be displayed through a display unit. The user may see the entire or partial area of ​​the rendered result through a VR / AR display or a public display.

[0082] The feedback process (S25) may include a process of sending various feedback information obtained during the rendering process (S24) or the display process to other components in the transmitting device 10 or the receiving device 20. The feedback process (S25) may be performed by Figure 1 The receiving device 20 may be executed by one or more components included in the receiving device 20, or may be executed by Fig.10 and Fig.11 In some embodiments, the feedback process (S25) may be performed by a "feedback unit" or a "sensing / tracking unit".

[0083] Interactivity for point cloud content consumption may be provided through a feedback process (S25). According to an embodiment, in the feedback process (S25), head orientation information and viewport information indicating the area currently viewed by the user may be given as feedback. According to an embodiment, the user may interact with things implemented in a VR / AR / MR / autonomous driving environment, and in this case, information related to such interaction may be passed to the sending device 10 or the service provider side during the feedback process (S25). According to an embodiment, the feedback process (S25) may not be performed.

[0084] Head orientation information may refer to information about the position, angle, and movement of the user's head. Based on this information, information about the area that the user is currently viewing in the point cloud video (i.e., viewport information) may be calculated.

[0085] The viewport information may be information about the area that the user is currently viewing in the point cloud video. The viewpoint is the point that the user is viewing in the point cloud video, and it may represent the center point of the viewport area. That is, the viewport is an area centered on the viewpoint, and the size and shape occupied by the area may be determined by the field of view (FOV). By using gaze analysis of the viewport information, it is possible to identify how the user consumes the point cloud video, which area of ​​the point cloud video the user is viewing, and how long the user is viewing the area. The gaze analysis may be performed on the receiving side (receiving device) and passed to the sending side (sending device) through a feedback channel. Devices such as VR / AR / MR displays may extract the viewport area based on the position / orientation of the user's head, the vertical or horizontal FOV supported by the device, and the like.

[0086] According to the embodiment, the feedback information can not only be transmitted to the transmitting side (transmitting device), but also be consumed at the receiving side (receiving device). That is, the decoding process and rendering process of the receiving side (receiving device) can be performed by using the feedback information.

[0087] For example, the receiving device 20 may first decode and render only the point cloud video for the area currently viewed by the user by using the head orientation information and / or the viewport information. In addition, the receiving unit 21 may receive all point cloud data or receive point cloud data represented by the orientation information and / or the viewport information based on the orientation information and / or the viewport information. In addition, the decapsulation processing unit 22 may decapsulate all point cloud data or decapsulate the point cloud data represented by the orientation information and / or the viewport information based on the orientation information and / or the viewport information. In addition, the decoding unit 23 may decode all point cloud data or decode the point cloud data represented by the orientation information and / or the viewport information based on the orientation information and / or the viewport information.

[0088] Overview of Point Cloud Coding Equipment

[0089] Figure 3 An example of a point cloud encoding device 300 according to an embodiment of the present disclosure is illustrated. Figure 3 The point cloud encoding device 300 may correspond to Figure 1 The encoding unit 12.

[0090] like Figure 3 As shown, the point cloud encoding device 300 may include: a coordinate system transformation unit 305, a geometry quantization unit 310, an octree analysis unit 315, an approximation unit 320, a geometry encoding unit 325, a reconstruction unit 330, and an attribute transformation unit 340, a RAHT transformation unit 345, an LOD generation unit 350, a lifting unit 355, an attribute quantization unit 360, an attribute encoding unit 365, and / or a color transformation unit 335.

[0091] The point cloud data obtained by the acquisition unit 11 may undergo a process of adjusting the quality of the point cloud content (e.g., lossless, lossy, near lossless) according to the network situation or application. In addition, each point of the obtained point cloud content may be sent losslessly, but in this case, because the size of the point cloud content is large, real-time streaming is not possible. Therefore, in order to smoothly provide the point cloud content, a process of reconstructing the point cloud content according to the maximum target bit rate is required.

[0092] The process of adjusting the quality of the point cloud content may be a process of reconstructing and encoding the position information (position information included in the geometric information) or color information (color information included in the attribute information) of the point. The process of reconstructing and encoding the position information of the point may be referred to as geometric encoding, and the process of reconstructing and encoding the attribute information associated with each point may be referred to as attribute encoding.

[0093] Geometry coding may include: a geometry quantization process, a voxelization process, an octree analysis process, an approximation process, a geometry coding process, and / or a coordinate system transformation process. Moreover, geometry coding may also include a geometry reconstruction process. Attribute coding may include: a color transformation process, an attribute transformation process, a prediction transformation process, a lifting transformation process, a RAHT transformation process, an attribute quantization process, an attribute coding process, etc.

[0094] Geometry Encoding

[0095] The coordinate system transformation process may correspond to a process of transforming the coordinate system of the point position. Therefore, the coordinate system transformation process may be referred to as "transformed coordinates". The coordinate system transformation process may be performed by the coordinate system transformation unit 305. For example, the coordinate system transformation unit 305 may transform the position of the point from the global space coordinate system into position information in a three-dimensional space (e.g., a three-dimensional space expressed in a coordinate system of an X-axis, a Y-axis, and a Z-axis). The position information in the 3D space according to an embodiment may be referred to as "geometric information".

[0096] The geometric quantization process may correspond to a process of quantizing the position information of the points, and may be performed by the geometric quantization unit 310. For example, the geometric quantization unit 310 may find the position information having the minimum (x, y, z) value among the position information of the points, and subtract the position information having the minimum (x, y, z) position from the position information of each point. In addition, the geometric quantization unit 310 may multiply the subtracted value by a preset quantization scale value, and then adjust (lower or raise) the result to a value close to an integer, thereby performing the quantization process.

[0097] The voxelization process may correspond to a process of matching the geometric information quantized by the quantization process with a specific voxel existing in the 3D space. The voxelization process may also be performed by the geometric quantization unit 310. The geometric quantization unit 310 may perform octree-based voxelization based on the position information of the point so as to reconstruct each point to which the quantization process is applied.

[0098] Like a pixel which is a minimum unit having information about a 2D image / video, a voxel may mean a space for storing information about a point existing in a 3D space. “Voxel” is a portmanteau of volume and pixel.

[0099] A voxel does not necessarily have a point (or does not necessarily match a point). That is, a voxel can have information related to multiple points. Alternatively, multiple pieces of information related to multiple points included in a voxel can be merged into one point information. This adjustment can be performed selectively. In the case where a voxel is represented by a merged point information, the position value of the center point of the voxel can be set based on the position value for the point in the voxel, and the relevant attribute transformation process needs to be performed. For example, the attribute transformation process can be adjusted by the average value of the color or reflectivity of the points included in the voxel or the neighboring points within a specific radius of the position value of the center point of the voxel.

[0100] The octree analysis unit 315 can use the octree to efficiently manage the region / position of the voxel. The octree can be represented by an occupancy code. For example, if each node includes a point, the octree analysis unit 315 can represent the occupancy code of the node as 1, and if the node does not include a point, the octree analysis unit 315 can represent the occupancy code of the node as 0.

[0101] The geometric coding process may correspond to a process of performing entropy coding on the occupancy code. The geometric coding process may be performed by the geometric coding unit 325. The geometric coding unit 325 may perform entropy coding on the occupancy code. The generated occupancy code may be encoded immediately or may be encoded by an intra / inter coding process to improve compression efficiency. The receiving device 20 may reconstruct the octet tree by the occupancy code.

[0102] On the other hand, in the case where there are no points or only a few points in a particular region, voxelizing all the regions may be inefficient. That is, since there are very few points in a particular region, it may not be necessary to construct the entire octree. For such cases, an early termination method may be required.

[0103] The point cloud encoding device 300 can directly send the position of the point only for a specific area, or use a surface model to reconfigure the position of the point within a specific area based on voxels, instead of splitting the node (specific node) corresponding to the specific area (a specific area not corresponding to a leaf node) into 8 sub-nodes (sub-nodes).

[0104] A mode for directly transmitting the positions of each point for a specific node may be a direct mode. The point cloud encoding device 300 may check whether a condition for enabling the direct mode is satisfied.

[0105] The conditions for enabling direct mode are: 1) the option to use direct mode should be enabled, 2) the specific node does not correspond to a leaf node, 3) there should be points below the threshold within the specific node, and 4) the total number of points to be sent directly does not exceed the limit value.

[0106] When all the above conditions are met, the point cloud encoding device 300 can directly entropy encode and send the position value of the point for a specific node through the geometric encoding unit 325.

[0107] The mode of reconstructing the position of the point in the specific area based on voxels using the surface model may be a trisoup mode. The trisoup mode may be performed by the approximation unit 320. The approximation unit 320 may determine a specific level of the octree, and reconstruct the position of the point in the node area based on voxels using the surface model from the determined specific level.

[0108] The point cloud encoding device 300 may selectively apply the triangle soup mode. Specifically, when the triangle soup mode is applied, the point cloud encoding device 300 may specify the level (specific level) at which the triangle soup mode is applied. For example, when the specified specific level is equal to the depth (d) of the octree, the triangle soup mode may not be applied. That is, the specified specific level should be less than the depth value of the octree.

[0109] The three-dimensional cube area of ​​the node of the specified specific level is called a block, and a block can include one or more voxels. The block or voxel can correspond to a brick. Each block can have 12 edges, and the approximation unit 320 can check whether each edge is adjacent to an occupied voxel with a point. Each edge can be adjacent to multiple occupied voxels. The specific position of the edge adjacent to a voxel is called a vertex, and when multiple occupied voxels are adjacent to an edge, the approximation unit 320 can determine the average position of these positions as a vertex.

[0110] When there are vertices, the point cloud encoding device 300 can perform entropy encoding on the starting point (x, y, z) of the edge, the direction vector (Δx, Δy, Δz) of the edge, and the position value of the vertex (relative position value within the edge) through the geometric encoding unit 325.

[0111] The geometry reconstruction process may correspond to a process of generating reconstructed geometry by reconstructing an octree and / or an approximate octree. The geometry reconstruction process may be performed by the reconstruction unit 330. The reconstruction unit 330 may perform the geometry reconstruction process by triangle reconstruction, upsampling, voxelization, and the like.

[0112] When the triangle soup mode is applied in the approximation unit 320, the reconstruction unit 330 can reconstruct the triangle based on the starting point of the edge, the direction vector of the edge, and the position value of the vertex. The reconstruction unit 330 can perform an upsampling process for voxelization by adding points in the middle along the edge of the triangle. The reconstruction unit 330 can generate additional points based on the upsampling factor and the width of the block. These points can be called refined vertices. The reconstruction unit 330 can voxelize the refined vertices, and the point cloud encoding device 300 can perform attribute encoding based on the voxelized position value.

[0113] In some embodiments, the geometry coding unit 325 can increase compression efficiency by applying context adaptive arithmetic coding. The geometry coding unit 325 can entropy encode the occupancy code directly using arithmetic codes. In some embodiments, the geometry coding unit 325 adaptively performs encoding based on the occupancy of neighboring nodes (intra-frame encoding), or adaptively performs encoding based on the occupancy code of the previous frame (inter-frame encoding). Here, a frame can refer to a set of point cloud data generated at the same time. Intra-frame encoding and inter-frame encoding are optional processes and can therefore be omitted.

[0114] Depending on how many neighbor nodes are referenced, the compression efficiency may be different, and as the number of bits increases, the encoding process becomes more complex, but it is also possible to improve the compression efficiency by skewing the bits to one side. For example, if a 3-bit context is given, 23 bits may have to be encoded in 8 parts. Because the divided encoding parts may affect the complexity of the implementation, it is necessary to balance the efficiency of compression and the appropriate complexity level.

[0115] Attribute Encoding

[0116] Attribute encoding may correspond to a process of encoding attribute information based on the reconstructed geometry and the geometry before the coordinate system transformation (source geometry). Since the attributes may depend on the geometry, the reconstructed geometry may be used for attribute encoding.

[0117] As mentioned above, attributes may include color, reflectivity, etc. The same attribute encoding method may be applied to the information or parameters included in the attributes. Color has three elements, reflectivity has one element, and each element may be processed independently.

[0118] Attribute coding may include: color transformation process, attribute transformation process, prediction transformation process, lifting transformation process, RAHT transformation process, attribute quantization process, attribute coding process, etc. The prediction transformation process, lifting transformation process, and RAHT transformation process may be selectively used, or a combination of one or more processes may be used.

[0119] The color conversion process may correspond to a process of converting the format of the color in the attribute to another format. The color conversion process may be performed by the color conversion unit 335. That is, the color conversion unit 335 may convert the color in the attribute. For example, the color conversion unit 335 may perform an encoding operation for converting the color in the attribute from RGB to YCbCr. In some embodiments, the operation of the color conversion unit 335 (i.e., the color conversion process) may be optionally applied according to the color value included in the attribute.

[0120] As described above, when one or more points exist in one voxel, the position values ​​of the points existing in the voxel are set to the center point of the voxel so that they can be displayed by integrating them into one point information of the voxel. Therefore, a process of transforming the attribute values ​​associated with the points may be required. Moreover, even if the triangulation mode is executed, the attribute transformation process may be performed.

[0121] The attribute transformation process may correspond to a process of transforming an attribute based on a position where geometry encoding is not performed and / or a reconstructed geometry. For example, the attribute transformation process may correspond to a process of transforming an attribute of a point having the position based on the position of the point included in the voxel. The attribute transformation process may be performed by the attribute transformation unit 340.

[0122] The attribute conversion unit 340 may calculate the center position value of the voxel and the average value of the attribute values ​​of the neighboring points within a certain radius. Alternatively, the attribute conversion unit 340 may apply weights to the attribute values ​​according to the distance from the center position and calculate the average value of the attribute values ​​to which the weights are applied. In this case, each voxel has a position and a calculated attribute value.

[0123] The prediction transformation process may correspond to a process of predicting the attribute value of the current point based on the attribute values ​​of one or more points (neighboring points) adjacent to the current point (corresponding to the prediction target point). The prediction transformation process may be performed by the level-of-detail (LOD) generation unit 350.

[0124] Prediction transformation is a method of applying LOD transformation technology, and the LOD generation unit 350 can calculate and set the LOD value of each point based on the LOD distance value of each point. Points with the lowest LOD can be sparsely distributed, and points with the highest LOD can be densely distributed. That is, the higher the LOD, the shorter the spacing (or distance) between points can be.

[0125] Each point present in the point cloud may be separated according to each LOD, and the points constituting each LOD may include points belonging to a lower LOD than the LOD. For example, the points constituting LOD 2 may include all points belonging to LOD 1 and LOD 2. The points may be rearranged according to each LOD, and a higher LOD may include points belonging to a lower LOD.

[0126] The LOD generation unit 350 may generate a predictor for each point of the prediction transformation. Therefore, when there are N points, N predictors may be generated. The predictor may calculate and set a weight value (=1 / distance) based on the LOD value of each point, the index information of the neighboring point, and the distance value from the neighboring point. Here, the neighboring point may be a point that exists within the distance from the current point set for each LOD.

[0127] In addition, the predictor may multiply the attribute value of the neighboring point by the "set weight value" and set the value obtained by averaging the attribute values ​​multiplied by the weight value as the predicted attribute value of the current point. The attribute quantization process may be performed on the residual attribute value obtained by subtracting the predicted attribute value of the current point from the attribute value of the current point.

[0128] The lifting transformation process may correspond to a process of reconstructing a point into a set of detail levels through an LOD generation process, just like the prediction transformation process. The lifting transformation process may be performed by the lifting unit 355. The lifting transformation process may also include a process of generating a predictor for each point, a process of setting the calculated LOD in the predictor, a process of registering neighbor points, and a process of setting weights according to the distance between the current point and the neighbor points.

[0129] The RAHT transformation process may correspond to a method of predicting attribute information of a higher-level node using attribute information associated with a lower-level node of the octree. That is, the RATH transformation process may correspond to an attribute information intra-frame encoding method performed by octree backward scanning. The RAHT transformation process may be performed by the RAHT transformation unit 345.

[0130] The RAHT transformation unit 345 scans the entire region in the voxel and can perform the RAHT transformation process up to the root node while summing (merging) the voxels into larger blocks at various steps. Since the RAHT transformation unit 345 only performs the RAHT transformation process on occupied nodes, in the case of an unoccupied empty node, the RAHT transformation process can be performed on the higher-level node directly above it.

[0131] The attribute quantization process may correspond to a process of quantizing the attribute output from the RAHT transform unit 345, the LOD generation unit 350, and / or the lifting unit 355. The attribute quantization process may be performed by the attribute quantization unit 360. The attribute encoding process may correspond to a process of encoding the quantized attribute and outputting the attribute bitstream. The attribute encoding process may be performed by the attribute encoding unit 365.

[0132] For example, when the predicted attribute value of the current point is calculated from the LOD generating unit 350 , the attribute quantizing unit 360 may quantize the residual attribute value obtained by subtracting the predicted attribute value of the current point from the attribute value of the current point.

[0133] If there are no neighboring points in the predictor of each point, the attribute encoding unit 365 itself can entropy encode the attribute value (non-quantized attribute value) of the current point. On the other hand, if there are neighboring points in the predictor of the current point, the attribute encoding unit 365 can entropy encode the quantized residual attribute value.

[0134] As another example, if the boost unit 355 outputs the product of an updated attribute value from the boost update process and an updated (QW stored) weight from the boost prediction process, the attribute quantization unit 360 may quantize the result (the product) and the attribute encoding unit 365 may entropy encode the quantized value.

[0135] Overview of Point Cloud Decoding Equipment

[0136] Figure 4 An example of a point cloud decoding device 400 according to an embodiment of the present disclosure is shown. Figure 4 The point cloud decoding device 400 may correspond in configuration and function to Figure 1 The decoding unit 23.

[0137] The point cloud decoding apparatus 400 may perform a decoding process based on the data (bit stream) transmitted from the transmitting device 10. The decoding process may include a process of reconstructing (decoding) a point cloud video by performing an operation corresponding to the above-described encoding operation on the bit stream.

[0138] like Figure 4 As illustrated, the decoding process may include a geometry decoding process and an attribute decoding process. The geometry decoding process may be performed by a geometry decoding unit 410, and the attribute decoding process may be performed by an attribute decoding unit 420. That is, the point cloud decoding device 400 may include a geometry decoding unit 410 and an attribute decoding unit 420.

[0139] The geometry decoding unit 410 may reconstruct the geometry from the geometry bitstream, and the attribute decoding unit 420 may reconstruct the attribute based on the reconstructed geometry and the attribute bitstream. Furthermore, the point cloud decoding apparatus 400 may reconstruct the three-dimensional point cloud video (point cloud data) based on the position information according to the reconstructed geometry and the attribute information according to the reconstructed attribute.

[0140] Figure 5 A specific example of a point cloud decoding device 500 according to another embodiment of the present disclosure is illustrated. Figure 5 As illustrated, the point cloud decoding device 500 includes: a geometry decoding unit 505, an octree synthesis unit 510, an approximate synthesis unit 515, a geometry reconstruction unit 520, a coordinate system inverse transformation unit 525, an attribute decoding unit 530, an attribute inverse quantization unit 535, a RATH transformation unit 550, an LOD generation unit 540, an inverse lifting unit 545, and / or a color inverse transformation unit 555.

[0141] The geometry decoding unit 505, the octree synthesis unit 510, the approximate synthesis unit 515, the geometry reconstruction unit 520, and the coordinate system inverse transformation unit 525 may perform geometry decoding. The geometry decoding may be used as a reference. Figures 1 to 3 The geometric decoding may include direct coding and triangular soup geometric decoding. Direct coding and triangular soup geometric decoding may be applied selectively.

[0142] The geometry decoding unit 505 may decode the received geometry bitstream based on arithmetic coding. The operation of the geometry decoding unit 505 may correspond to the inverse process of the operation performed by the geometry encoding unit 325.

[0143] The octree synthesis unit 510 may generate an octree by obtaining an occupancy code (or information about geometry obtained as a result of decoding) from the decoded geometry bitstream. The operation of the octree synthesis unit 510 may correspond to the inverse process of the operation performed by the octree analysis unit 315.

[0144] When triangle soup geometry encoding is applied, the approximate synthesis unit 515 can synthesize the surface based on the decoded geometry and / or the generated octree.

[0145] The geometry reconstruction unit 520 may reconstruct the geometry based on the surface and the decoded geometry. When direct coding is applied, the geometry reconstruction unit 520 may directly bring and add the position information of the points to which direct coding is applied. In addition, when triangulated geometry coding is applied, the geometry reconstruction unit 520 may reconstruct the geometry by performing reconstruction operations (e.g., triangle reconstruction, upsampling, voxelization operations, etc.). The reconstructed geometry may include a point cloud picture or frame that does not include attributes.

[0146] The coordinate system inverse transformation unit 525 can obtain the position of the point by transforming the coordinate system based on the reconstructed geometry. For example, the coordinate system inverse transformation unit 525 can inversely transform the position of the point from a three-dimensional space (e.g., a three-dimensional space expressed by a coordinate system of an x-axis, a y-axis, and a z-axis, etc.) into position information of a global space coordinate system.

[0147] The attribute decoding unit 530, the attribute inverse quantization unit 535, the RATH transform unit 550, the LOD generation unit 540, the inverse lifting unit 545 and / or the color inverse transform unit 555 may perform attribute decoding. Attribute decoding may include: RAHT transform decoding, prediction transform decoding, and lifting transform decoding. The above three types of decoding may be selectively used, or a combination of one or more types of decoding may be used.

[0148] The attribute decoding unit 530 may decode the attribute bitstream based on arithmetic coding. For example, when there is no neighbor point in the predictor of each point and thus the attribute value of the current point is directly entropy encoded, the attribute decoding unit 530 may decode the attribute value (unquantized attribute value) of the current point. As another example, when there is a neighbor point in the predictor of the current point and thus the quantized residual attribute value is entropy encoded, the attribute decoding unit 530 may decode the quantized residual attribute value.

[0149] The attribute inverse quantization unit 535 may inverse quantize the decoded attribute bitstream or information about the attribute obtained as a result of decoding, and output the inverse quantized attribute (or attribute value). For example, when the quantized residual attribute value is output from the attribute decoding unit 530, the attribute inverse quantization unit 535 may inverse quantize the quantized residual attribute value to output the residual attribute value. The inverse quantization process may be selectively applied based on whether the attribute is encoded in the point cloud encoding device 300. That is, when there is no neighbor point in the predictor of each point and therefore the attribute value of the current point is directly encoded, the attribute decoding unit 530 may output the attribute value of the current point that is not quantized, and the attribute encoding process may be skipped.

[0150] The RATH transform unit 550, the LOD generation unit 540, and / or the inverse lifting unit 545 may process the reconstructed geometry and the inverse quantized attributes. The RATH transform unit 550, the LOD generation unit 540, and / or the inverse lifting unit 545 may selectively perform a decoding operation corresponding to the encoding operation of the point cloud encoding apparatus 300.

[0151] The color inverse transform unit 555 may perform inverse transform encoding to inversely transform a color value (or texture) included in the decoded attribute. The operation of the color inverse transform unit 555 may be selectively performed based on whether the color transform unit 335 operates.

[0152] Figure 6 Another example of a transmitting device according to an embodiment of the present disclosure is illustrated. Figure 6 As shown, the sending device may include a data input unit 605, a quantization processing unit 610, a voxelization processing unit 615, an octree occupancy code generation unit 620, a surface model processing unit 625, an intra-frame / inter-frame coding processing unit 630, an arithmetic encoder 635, a metadata processing unit 640, a color transformation processing unit 645, an attribute transformation processing unit 650, a prediction / enhancement / RAHT transformation processing unit 655, an arithmetic encoder 660 and a sending processing unit 665.

[0153] The function of the data input unit 605 may correspond to Figure 1 The acquisition process performed by the acquisition unit 11. That is, the data input unit 605 can obtain the point cloud video and generate point cloud data for multiple points. The geometric information (position information) in the point cloud data can be generated in the form of a geometric bit stream by the quantization processing unit 610, the voxelization processing unit 615, the octree occupancy code generation unit 620, the surface model processing unit 625, the intra / inter coding processing unit 630 and the arithmetic encoder 635. The attribute information in the point cloud data can be generated in the form of an attribute bit stream by the color transformation processing unit 645, the attribute transformation processing unit 650, the prediction / lifting / RAHT transformation processing unit 655 and the arithmetic encoder 660. The geometric bit stream, the attribute bit stream and / or the metadata bit stream can be sent to the receiving device through the processing of the sending processing unit 665.

[0154] Specifically, the function of the quantization processing unit 610 may correspond to Figure 3 The function of the voxelization processing unit 615 may correspond to the quantization process performed by the geometric quantization unit 310 and / or the function of the coordinate system transformation unit 305. Figure 3 The voxelization process performed by the geometric quantization unit 310 of , and the function of the octree occupancy code generation unit 620 may correspond to the voxelization process performed by the geometric quantization unit 310 of , and the function of the octet occupancy code generation unit 620 may correspond to the function of the octet occupancy code generation unit 620 of . Figure 3 The functions of the surface model processing unit 625 may correspond to those of the octree analysis unit 315. Figure 3 The functions performed by the approximation unit 320 of the embodiment, and the functions of the intra-frame / inter-frame encoding processing unit 630 and the functions of the arithmetic encoder 635 may correspond to those performed by Figure 3 The functions of the metadata processing unit 640 may correspond to Figure 1 The functions of the metadata processing unit are described in .

[0155] In addition, the function of the color conversion processing unit 645 may correspond to Figure 3 The function of the color conversion unit 335 is performed by the color conversion unit 335, and the function of the attribute conversion processing unit 650 may correspond to the function of the attribute conversion processing unit 650. Figure 3 The function of the prediction / lifting / RAHT transformation processing unit 655 may correspond to the function of the attribute transformation unit 340 performed by Figure 3 The functions performed by the RAHT transform unit 345, the LOD generation unit 350, and the lifting unit 355 of the arithmetic encoder 660 may correspond to Figure 3 The function of the attribute encoding unit 365 may correspond to the function of the transmission processing unit 665. Figure 1 The functions performed by the sending unit 14 and / or the encapsulation processing unit 13.

[0156] Figure 7 Another example of a receiving device according to an embodiment of the present disclosure is illustrated. Figure 7 As shown, the receiving device may include a receiving unit 705, a receiving processing unit 710, an arithmetic decoder 715, a metadata parser 735, an octree reconstruction processing unit 720 based on an occupancy code, a surface model processing unit 725, an inverse quantization processing unit 730, an arithmetic decoder 740, an inverse quantization processing unit 745, a prediction / lifting / RAHT inverse transform processing unit 750, a color inverse transform processing unit 755 and a renderer 760.

[0157] The function of the receiving unit 705 may correspond to Figure 1 The functions of the receiving unit 21 are performed, and the functions of the receiving processing unit 710 may correspond to those of Figure 1 The function performed by the decapsulation processing unit 22 of the receiving unit 705. That is, the receiving unit 705 can receive the bitstream from the sending processing unit 765, and the receiving processing unit 710 can extract the geometry bitstream, the attribute bitstream and / or the metadata bitstream through the decapsulation process. The geometry bitstream can be generated as a reconstructed (recovered) position value (position information) d by the arithmetic decoder 715, the octtree reconstruction processing unit 720 based on the occupancy code, the surface model processing unit 725 and the inverse quantization processing unit 730, and the attribute bitstream can be generated as a reconstructed attribute value by the arithmetic decoder 740, the inverse quantization processing unit 745, the prediction / lifting / RAHT inverse transform processing unit 750 and the color inverse transform processing unit 755. The metadata bitstream can be generated as reconstructed metadata (or metadata information) by the metadata parser 735. The position value, attribute value and / or metadata can be rendered in the renderer 760 and provide the user with an experience such as VR / AR / MR / autonomous driving.

[0158] Specifically, the function of the arithmetic decoder 715 may correspond to Figure 5 The functions performed by the geometry decoding unit 506 and the functions of the occupancy code based octree reconstruction processing unit 720 may correspond to those performed by Figure 5 The functions of the surface model processing unit 725 may correspond to those of the octree synthesis unit 510. Figure 5 The functions of the approximate synthesis unit 730 may correspond to the functions performed by the geometric reconstruction unit 520 and / or the inverse coordinate system transform unit 525. The functions of the metadata parser 735 may correspond to Figure 1 The functionality of the metadata parsing unit described in .

[0159] In addition, the function of the arithmetic decoder 740 may correspond to Figure 5 The function of the attribute decoding unit 530 is performed, and the function of the inverse quantization processing unit 745 may correspond to Figure 5 The function of the prediction / lifting / RAHT inverse transform processing unit 750 may correspond to the function of the attribute inverse transform unit 535. Figure 5 The functions performed by the RAHT transform unit 550, the LOD generation unit 540, and the inverse lifting unit 545, and the function of the color inverse transform processing unit 755 may correspond to the functions performed by Figure 5 The color inverse conversion unit 555 performs the function.

[0160] Figure 8 An example of a structure capable of interworking with a method / apparatus for transmitting and receiving point cloud data according to an embodiment of the present disclosure is illustrated.

[0161] Figure 8 The structure of illustrates a configuration in which at least one of a server (AI server), a robot, an autonomous vehicle, an XR device, a smart phone, a home appliance, and / or an HMD is connected to a cloud network. A robot, an autonomous vehicle, an XR device, a smart phone, or a home appliance may be referred to as a device. In addition, the XR device may correspond to a point cloud data device (PCC) according to an embodiment, or may interact with a PCC device.

[0162] A cloud network may refer to a network that forms part of or exists within a cloud computing infrastructure. Here, a 3G network, a 4G or Long Term Evolution (LTE) network, or a 5G network may be used to configure the cloud network.

[0163] The server can be connected to at least one of a robot, an autonomous vehicle, an XR device, a smart phone, a household appliance, and / or an HMD through a cloud network, and can assist with at least a part of the processing of the connected devices.

[0164] The HMD can represent one of the types of XR devices and / or PCC devices in which the embodiments can be implemented. The HMD type device according to the embodiments can include: a communication unit, a control unit, a memory unit, an I / O unit, a sensor unit, and a power supply unit.

[0165] <PCC+XR>

[0166] The XR / PCC device can be implemented by an HMD, an HUD installed in a vehicle, a TV, a mobile phone, a smart phone, a computer, a wearable device, a household appliance, a digital sign, a vehicle, a stationary robot, or a mobile robot, etc. by applying PCC and / or XR technologies.

[0167] The XR / PCC device can obtain information about the surrounding space or real objects by analyzing 3D point cloud data or image data obtained through various sensors or from external devices, to generate position (geometry) data and attribute data of 3D points, and render and output the XR object to be output. For example, the XR / PCC device can output an XR object corresponding to the identified object, including additional information about the identified object.

[0168] <PCC+XR+mobile phone>

[0169] The XR / PCC device can be implemented by a mobile phone, etc. by applying PCC technology. The mobile phone can decode and display point cloud content based on PCC technology.

[0170] <PCC+autonomous driving+XR>

[0171] The autonomous vehicle can be implemented by a mobile robot, a vehicle, an unmanned aerial vehicle, etc. by applying PCC technology and XR technology. The autonomous vehicle applying XR / PCC technology can refer to an autonomous vehicle equipped with a unit for providing XR images, or an autonomous vehicle that undergoes control / interaction within the XR image. In particular, the autonomous vehicle that undergoes control / interaction within the XR image is distinguished from the XR device and can work interactively with each other.

[0172] An autonomous vehicle equipped with a unit for providing an XR / PCC image may obtain sensor information from a sensor including a camera, and output an XR / PCC image generated based on the obtained sensor information. For example, an autonomous vehicle has a HUD, and may provide passengers with an XR / PCC object corresponding to a real object or an object in a screen by outputting an XR / PCC image.

[0173] In this case, when the XR / PCC object is output to the HUD, at least a portion of the XR / PCC object may be output so as to overlap the actual object to which the passenger's gaze is directed. On the other hand, when the XR / PCC object is output to a display disposed inside the autonomous vehicle, at least a portion of the XR / PCC object may be output so as to overlap the object in the screen. For example, the autonomous vehicle may output XR / PCC objects corresponding to objects such as lanes, other vehicles, traffic lights, traffic signs, two-wheeled vehicles, pedestrians, and buildings.

[0174] VR technology, AR technology, MR technology and / or PCC technology according to the embodiment can be applied to various devices. That is, VR technology is a display technology that provides objects or backgrounds in the real world only as CG images. On the other hand, AR technology refers to a technology that displays a virtual CG image on top of a real object image. Moreover, MR technology is similar to the above-mentioned AR technology in that the mixing and combination of virtual objects in the real world are shown. However, in AR technology, the distinction between real objects and virtual objects composed of CG images is clear, and virtual objects are used in the form of supplementing real objects, while in MR technology, virtual objects are considered to be equivalent to real objects, which is different from AR technology. More specifically, for example, a hologram service is applied to the above-mentioned MR technology. VR, AR and MR technologies can be integrated and referred to as XR technology.

[0175] Space segmentation

[0176] Point cloud data (i.e., G-PCC data) may represent a volumetric encoding of a point cloud consisting of a sequence of frames (point cloud frames). Each point cloud frame may include: the number of points, the locations of the points, and the attributes of the points. The number of points, the locations of the points, and the attributes of the points may change from one frame to another. Each point cloud frame may refer to a set of three-dimensional points specified by zero or more attributes and the Cartesian coordinates (x, y, z) of the three-dimensional points at a particular time instance. Here, the Cartesian coordinates (x, y, z) of the three-dimensional points may be positions or geometries.

[0177] In some embodiments, the present disclosure may also perform a spatial segmentation process of segmenting the point cloud data into one or more 3D blocks before encoding the point cloud data. A 3D block may refer to all or part of a 3D space occupied by the point cloud data. A 3D block may be one or more of a tile group, a tile, a slice, a coding unit (CU), a prediction unit (PU), or a transform unit (TU).

[0178] A patch corresponding to a 3D block may refer to all or part of a 3D space occupied by point cloud data. Moreover, a slice corresponding to a 3D block may refer to all or part of a 3D space occupied by point cloud data. A patch may be divided into one or more slices based on the number of points included in a patch. A patch may be a group of slices with bounding box information. The bounding box information of each patch may be specified in a patch list (or a tile parameter set, TPS). A patch may overlap with another patch in a bounding box. A slice may be a data unit that is independently encoded or a data unit that is independently decoded. That is, a slice may be a collection of points that may be independently encoded or decoded. In some embodiments, a slice may be a series of syntax elements representing part or all of an encoded point cloud frame. Each slice may include an index for identifying the patch to which the slice belongs.

[0179] The spatially divided 3D blocks can be processed independently or non-independently. For example, the spatially divided 3D blocks can be encoded or decoded independently or non-independently, and can be sent or received independently or non-independently. In addition, the spatially divided 3D blocks can be quantized or inversely quantized independently or non-independently, and can be transformed or inversely transformed independently or non-independently. In addition, the spatially divided 3D blocks can be rendered independently or non-independently. For example, encoding or decoding can be performed in units of slices or in units of tiles. In addition, quantization or inverse quantization can be performed differently for each tile or slice, and quantization or inverse quantization can be performed differently for each transformed or inversely transformed tile or slice.

[0180] In this way, when point cloud data is spatially segmented into one or more 3D blocks and the spatially segmented 3D blocks are processed independently or non-independently, the process of processing the 3D blocks is performed in real time and with low latency. In addition, random access and parallel encoding or parallel decoding in the three-dimensional space occupied by the point cloud data can be enabled, and errors accumulated during the encoding or decoding process can be prevented.

[0181] Fig. 9 is a block diagram illustrating an example of a transmitting device 900 for performing a space division process according to an embodiment of the present disclosure. Fig. 9As illustrated, the sending device 900 may include: a space segmentation unit 905 for performing a space segmentation process, a signaling processing unit 910, a geometry encoder 915, an attribute encoder 920, and an encapsulation processing unit 925 and / or a sending processing unit 930.

[0182] The spatial segmentation unit 905 may perform a spatial segmentation process of segmenting the point cloud data into one or more 3D blocks based on the bounding box and / or the sub-bounding box. Through the spatial segmentation process, the point cloud data may be segmented into one or more tiles and / or one or more slices. In some embodiments, the point cloud data may be segmented into one or more tiles, and each segmented tile may be further segmented into one or more slices through the spatial segmentation process.

[0183] Fig.10 An example of spatially segmenting a bounding box (that is, point cloud data) into three-dimensional blocks is illustrated. Fig.10 As illustrated, the overall bounding box of the point cloud data can be segmented into three tiles (i.e., tile #0, tile #1, and tile #2). In addition, tile #0 can be segmented again into two slices (i.e., slice #0 and slice #1). In addition, tile #1 can be segmented again into two slices (i.e., slice #2 and slice #3). In addition, tile #2 can be segmented again into slice #4.

[0184] The signaling processing unit 910 can generate and / or process (e.g., entropy encoding) signaling information and output the signaling information in the form of a bit stream. Hereinafter, the bit stream output from the signaling processing unit (wherein the signaling information is encoded) is referred to as a "signaling bit stream". The signaling information may include information for or about spatial segmentation. That is, the signaling information may include information related to the spatial segmentation process performed by the spatial segmentation unit 905.

[0185] When point cloud data is segmented into one or more 3D blocks, information may be required for decoding some point cloud data corresponding to a specific patch or a specific slice in the point cloud data. In addition, in order to support spatial access (or partial access) to the point cloud data, information related to the 3D spatial region may be required. Here, spatial access may refer to extracting only the necessary part of the point cloud data from the entire point cloud data from the file. The signaling information may include: information for decoding some point cloud data, information related to the 3D spatial region for supporting spatial access, etc. For example, the signaling information may include: 3D bounding box information, 3D spatial region information, patch information, and / or patch list information.

[0186] The signaling information may be provided from the spatial segmentation unit 905, the geometry encoder 915, the attribute encoder 920, the transmission processing unit 925, and / or the encapsulation processing unit 930. In addition, the signaling processing unit 910 may provide the signaling information from Fig.13 The feedback information fed back by the receiving device 700 is provided to the space segmentation unit 905, the geometry encoder 915, the attribute encoder 920, the sending processing unit 925 and / or the encapsulation processing unit 930.

[0187] The signaling information may be stored and signaled in samples, sample entries, sample groups, track groups, or separate metadata tracks in a track. In some embodiments, the signaling information may be signaled in units of sequence parameter sets (SPS) for sequence-level signaling, geometry parameter sets (GPS) for signaling of geometry coding information, attribute parameter sets (APS) for signaling of attribute coding information, and tile parameter sets (TPS) (or tile lists) for signaling at the tile level. In addition, the signaling information may be signaled in coding units such as slices or tiles.

[0188] In addition, the position (position information) of the 3D block may be output to the geometry encoder 915 , and the attribute (attribute information) of the 3D block may be output to the attribute encoder 920 .

[0189] The geometry encoder 915 may construct an octree based on the position information, encode the constructed octree, and output a geometry bitstream. Furthermore, the geometry encoder 915 may reconstruct the octree and / or the approximate octree and output it to the attribute encoder 920. The reconstructed octree may be a reconstructed geometry. The geometry encoder 915 may perform the following steps: Figure 3 The coordinate system transformation unit 305, the geometric quantization unit 310, the octree analysis unit 315, the approximation unit 320, the geometric encoding unit 325 and / or the reconstruction unit 330 of the geometric encoder 915 may perform all or some operations performed by the coordinate system transformation unit 305, the geometric quantization unit 310, the octree analysis unit 315, the approximation unit 320, the geometric encoding unit 325 and / or the reconstruction unit 330. According to an embodiment, the geometric encoder 915 may perform Figure 6 All or some operations performed by the quantization processing unit 610, the voxelization processing unit 615, the octree occupancy code generation unit 620, the surface model processing unit 625, the intra / inter encoding processing unit 630 and / or the arithmetic encoder 635.

[0190] The attribute encoder 920 may output an attribute bitstream by encoding the attribute based on the reconstructed geometry. The attribute encoder 920 may perform Figure 3The attribute encoder 920 may perform all or some operations performed by the attribute transform unit 340, the RAHT transform unit 345, the LOD generation unit 350, the lifting unit 355, the attribute quantization unit 360, the attribute encoding unit 365 and / or the color transform unit 335. Figure 6 All or some operations performed by the attribute transform processing unit 650, the prediction / lifting / RAHT transform processing unit 655, the arithmetic encoder 660 and / or the color transform processing unit 645.

[0191] The encapsulation processing unit 925 can encapsulate one or more input bit streams into files or segments. For example, the encapsulation processing unit 925 can encapsulate each of the geometry bit stream, the attribute bit stream, and the signaling bit stream, or multiplex and encapsulate the geometry bit stream, the attribute bit stream, and the signaling bit stream. In some embodiments, the encapsulation processing unit 925 can encapsulate a bit stream (G-PCC bit stream) composed of a sequence of type-length-value (TLV) structures into a file. The TLV (or TLV encapsulation) structure that constitutes the G-PCC bit stream may include: geometry bit stream, attribute bit stream, signaling bit stream, etc. In some embodiments, the G-PCC bit stream can be generated by the encapsulation processing unit 925, or by the sending processing unit 930. The TLV structure or TLV encapsulation structure will be described in detail below. In some embodiments, the encapsulation processing unit 925 may be executed by Figure 1 The encapsulation processing unit 13 performs all or some operations.

[0192] The transmission processing unit 930 can process the encapsulated bit stream or file / segment according to any transmission protocol. The transmission processing unit 930 can perform the Figure 1 The sending unit 14 and the sending processing unit described or Figure 6 All or some operations performed by the sending processing unit 665.

[0193] Fig.11 1 is a block diagram illustrating an example of a receiving device 1100 according to an embodiment of the present disclosure. The receiving device 1100 may perform an operation corresponding to the operation of the transmitting device 900 for performing space division. Fig.11 As illustrated, the receiving device 1100 may include: a receiving processing unit 1105 , a decapsulation processing unit 1110 , a signaling processing unit 1115 , a geometry decoder 1120 , an attribute decoder 1125 and / or a post-processing unit 1130 .

[0194] The receiving processing unit 1105 may receive a file / segment in which a G-PCC bitstream is encapsulated, a G-PCC bitstream, or a bitstream, and process it according to the transport protocol. The receiving processing unit 1105 may perform the processing by referring to Figure 1 The receiving unit 21 and the receiving processing unit described or Figure 7 All or some operations performed by the receiving processing unit 710.

[0195] The decapsulation processing unit 1110 may obtain a G-PCC bitstream by performing the inverse process of the operation performed by the encapsulation processing unit 925. The decapsulation processing unit 1110 may obtain a G-PCC bitstream by decapsulating a file / segment. For example, the decapsulation processing unit 1110 may obtain a signaling bitstream and output the signaling bitstream to the signaling processing unit 1115, obtain a geometry bitstream and output the geometry bitstream to the geometry decoder 1120, and obtain an attribute bitstream and output the attribute bitstream to the attribute decoder 1125. The decapsulation processing unit 1110 may perform the operation performed by the decapsulation processing unit 925. Figure 1 The decapsulation processing unit 22 or Figure 7 All or some of the operations performed by the receiving processing unit.

[0196] The signaling processing unit 1115 may parse and decode the signaling information by performing the inverse process of the operation performed by the signaling processing unit 910. The signaling processing unit 1115 may parse and decode the signaling information from the signaling bitstream. The signaling processing unit 1115 may provide the decoded signaling information to the geometry decoder 1120, the attribute decoder 1125, and / or the post-processing unit 1130.

[0197] The geometry decoder 1120 may reconstruct the geometry from the geometry bitstream by performing the inverse process of the operation performed by the geometry encoder 915. The geometry decoder 1120 may reconstruct the geometry based on the signaling information (geometry-related parameters). The reconstructed geometry may be provided to the attribute decoder 1125.

[0198] The attribute decoder 1125 may reconstruct the attribute from the attribute bitstream by performing the inverse process of the operation performed by the attribute encoder 920. The attribute decoder 1125 may reconstruct the attribute based on the signaling information (attribute-related parameters) and the reconstructed geometry.

[0199] The post-processing unit 1130 may reconstruct the point cloud data based on the reconstructed geometry and the reconstructed attributes. The reconstruction of the point cloud data may be performed by a process of matching the reconstructed geometry with the reconstructed attributes. In some embodiments, when the reconstructed point cloud data is in units of tiles and / or slices, the post-processing unit 1130 may reconstruct the bounding box of the point cloud data by performing an inverse process of the spatial segmentation process of the sending device 900 based on the signaling information. In some embodiments, when the bounding box is divided into a plurality of tiles and / or a plurality of slices by the spatial segmentation process, the post-processing unit 1130 may reconstruct portions of the bounding box by combining some slices and / or some tiles based on the signaling information. Here, some slices and / or some tiles used to reconstruct the bounding box may be slices and / or some tiles associated with a 3D space region to which spatial access is desired.

[0200] Encapsulation / Decapsulation

[0201] The TLV encapsulation structure may be generated in the transmission unit, the transmission processing unit, and the encapsulation unit mentioned in this specification. The G-PCC bit stream composed of the TLV encapsulation structure may be sent to the receiving device as it is, or may be encapsulated and sent to the receiving device. For example, the encapsulation processing unit 925 may encapsulate the G-PCC bit stream composed of the TLV encapsulation structure in the form of a file / segment and send it. The decapsulation processing unit 1110 may obtain the G-PCC bit stream by decapsulating the encapsulated file / segment.

[0202] In some embodiments, the G-PCC bitstream may be encapsulated in a file format based on ISOBMFF. In this case, the G-PCC bitstream may be stored in a single track or multiple tracks in an ISOBMFF file. Here, the single track or multiple tracks in the file may be referred to as a "track" or a "G-PCC track". A file based on ISOBMFF may be referred to as a container, a container file, a media file, a G-PCC file, etc. Specifically, the file may consist of a box and / or information, which may be referred to as ftyp, moov, mdat, etc.

[0203] The ftyp box (ftyp box, file type box) can provide file type or file compatibility related information for a file. The receiving device can identify the file by referring to the ftyp box. The mdat box is also called the media data box and can include actual media data. In some embodiments, a geometric slice (or an encoded geometric bitstream) and zero or more attribute slices (or an encoded attribute bitstream) can be included in the sample of the mdat box in the file. Here, the sample can be referred to as a G-PCC sample. The moov box is also called a movie box and can include metadata of the media data of the file. For example, the moov box can include information required for decoding and playback of media data, and can include information about the track and sample of the file. The moov box can act as a container for all metadata. The moov box can be the topmost box among the metadata-related boxes.

[0204] In some embodiments, the moov box may include a track (trak) box that provides information related to the track of the file, and the trak box may include a media (mdia) box (MediaBox) that provides media information of the track, and a track reference container (tref) box for linking (referencing) the track and the sample of the file corresponding to the track. The media box MediaBox may include a media information container (minf) box that provides information about the media data, and a handler (hdlr) box that indicates the stream type. The minf box may include a sample table (stbl) box that provides metadata related to the sample of the mdat box. The stbl box may include a sample description (stsd) box that provides information about the encoding type used and initialization information required for the encoding type. In some embodiments, the sample description (stsd) box may include a sample entry of the track. In some embodiments, signaling information (or metadata) such as SPS, GPS, APS, and patch list may be included in the sample entry of the moov box or the sample of the mdat box in the file.

[0205] A G-PCC track may be defined as a volume visual track that carries a geometry slice (or a coded geometry bitstream) or an attribute slice (or a coded attribute bitstream), or both a geometry slice and an attribute slice. In some embodiments, a volume visual track may be identified by a volume visual media handler type "volv" in a handler box HandlerBox of a media box (MediaBox), and / or a volume visual media header vvhd in a minf box of a media box MediaBox. A minf box may be referred to as a media information container or a media information box. A minf box may be included in a media box MediaBox, a media box MediaBox may be included in a track box, and a track box may be included in a moov box of a file. A single volume visual track or multiple volume visual tracks may be present in a file.

[0206] Sample Group

[0207] The encapsulation processing unit mentioned in the present disclosure may generate a sample group by grouping one or more samples. The encapsulation processing unit, metadata processing unit, or signaling processing unit mentioned in the present disclosure may signal signaling information associated with the sample group in the sample, sample group, or sample entry. That is, the sample group information associated with the sample group may be added to the sample, sample group, or sample entry. The sample group information may be 3D bounding box sample group information, 3D region sample group information, 3D patch sample group information, 3D patch list sample group information, etc.

[0208] Sample entry

[0209] Fig.12 is a diagram for explaining an ISOBMFF-based file including a single track. Fig.12 (a) illustrates an example of a layout of an ISOBMFF-based file including a single track, and Fig.12 (b) illustrates an example of a sample structure of the mdat box when the G-PCC bitstream is stored in a single track of a file. Fig.13 is a diagram for explaining an ISOBMFF-based file including multiple tracks. Fig.13 (a) illustrates an example of a layout of an ISOBMFF-based file including a plurality of tracks, and Fig.13 (b) illustrates an example of a sample structure of the mdat box when the G-PCC bitstream is stored in a single track of a file.

[0210] The stsd box (SampleDescriptionBox) included in the moov box of the file can include sample entries for a single track storing a G-PCC bitstream. SPS, GPS, APS, and tile lists can be included in the sample entries in the moov box in the file or in the samples in the mdat box. Moreover, geometric slices and zero or more attribute slices can be included in the samples of the mdat box in the file. When the G-PCC bitstream is stored in a single track of the file, each sample can contain multiple G-PCC components. That is, each sample can be composed of one or more TLV encapsulation structures.

[0211] like Fig.13 As illustrated in (b), the sample may include a TLV encapsulation structure including a geometric slice. In addition, the sample may include a TLV encapsulation structure including one or more parameter sets. In addition, the sample may include a TLV encapsulation structure including one or more attribute slices.

[0212] like Fig.13 As illustrated in (a), when the G-PCC bitstream is carried by multiple tracks of an ISOBMFF-based file, each geometry slice or attribute slice may be mapped to a separate track. For example, the geometry slice may be mapped to track 1, and the attribute slice may be mapped to track 2. The track carrying the geometry slice (track 1) may be referred to as a geometry track or a G-PCC geometry track, and the track carrying the attribute slice (track 2) may be referred to as an attribute track or a G-PCC attribute track. In addition, the geometry track may be defined as a volume visual track carrying the geometry slice, and the attribute track may be defined as a volume visual track carrying the attribute slice.

[0213] A track that carries a portion of a G-PCC bitstream that includes both geometry slices and attribute slices may be referred to as a multiplexed track. Where geometry slices and attribute slices are stored on separate tracks, each sample in the track may include at least one TLV encapsulation structure that carries data for a single G-PCC component. In this case, each sample contains neither geometry nor attributes, and may not contain multiple attributes. Multi-track encapsulation of a G-PCC bitstream may enable a G-PCC player to efficiently access one of the G-PCC components.

[0214] Time Scalability

[0215] Temporal scalability may refer to a function that allows the possibility of extracting one or more subsets of independently encoded frames. In addition, temporal scalability may refer to a function that divides G-PCC data into multiple different temporal levels and independently processes each G-PCC frame belonging to a different temporal level. If temporal scalability is supported, a G-PCC player (or a transmitting device and / or receiving device of the present disclosure) can effectively access a desired component (target component) among the G-PCC components. In addition, if temporal scalability is supported, since the G-PCC frames are processed independently of each other, the temporal scalability support at the system level can be expressed as a more flexible temporal sub-layering. In addition, if temporal scalability is supported, a system (point cloud content providing system) that processes G-PCC data can manipulate the data at a high level to match the network capability or decoder capability, thereby improving the performance of the point cloud content providing system.

[0216] Problems with existing technology

[0217] The G-PCC bitstream according to the existing file format has only intra-coded frames. In this case, since all frames in the G-PCC bitstream are intra-coded, the random access function can be automatically activated, but in practice, depending on how the frames are transmitted / stored in the tracks of the G-PCC file, the random access function may not be available without additional information.

[0218] One of the issues regarding the random access functionality is how to store and transmit parameter sets in a track. Parameter sets can be transmitted only in a sample entry or in a sample, or in both. In case parameter sets are transmitted in a sample, even if the frame in the sample is an intra-coded frame, when a frame in a particular sample X refers to a parameter set transmitted not in sample X but in one of the previous samples of sample X, the decoding process cannot start from sample X. Therefore, for safe playback from a random sample, the file parser / player needs to solve the problem of finding all parameter sets required or referenced by the frames starting from the random access point.

[0219] According to the existing ISOBMFF file format, there is a SyncSampleBox ("stss") that is capable of transmitting information about which sample in a track is a synchronization sample. In addition, a sample grouping mechanism that provides useful information for supporting a random access function has been proposed. The SyncSampleBox and the sample group mechanism can operate together to provide a random access function. However, the relevant mechanism (that is, how they operate together to support the random access function) has not yet been embodied in detail. Therefore, the present disclosure may provide various implementations of SyncSampleBox properties and player operations for supporting correct random access functions.

[0220] The embodiments of the present disclosure may be provided based on the following aspects. According to the embodiment, the following aspects may be applied alone or as a combination of two or more thereof.

[0221] - (Aspect 1): In the case where parameter sets are not exclusively transmitted in sample entries of a track transmitting geometry components or in a tile-based track, a parameter set sample group box or a SyncSampleBox and a parameter set sample group box may be present in the track.

[0222] Alternatively, in case parameter sets are not exclusively transmitted in sample entries of tracks transmitting geometry components or in tile-based tracks, the Parameter Set Sample Group Box or the SyncSampleBox and the Parameter Set Sample Group Box may be constrained to be present in the track.

[0223] - (Aspect 2): In case parameter sets are not transmitted exclusively in sample entries of a track, the SyncSampleBox does not have to be present because all samples can be considered as synchronization samples (ie, samples included in the SyncSampleBox).

[0224] - (Aspect 3): In the presence of a SyncSampleBox, the box can be constrained to exist only in tracks containing geometry components or in patch-based tracks. Sync samples in tracks that only include attribute components can be inferred from the associated track that conveys the geometry component.

[0225] - (Aspect 4): When supporting temporal scalability together with multiple temporal level tracks, SyncSampleBox may exist in a track including only temporal level 0.

[0226] - (Aspect 5): In a random access event starting from a specific sample X, the file player has the following behavior.

[0227] i) In case the sample entry of the track is "gpe1" or "gpc1", the file player can decode and play back by itself starting from sample X. The parameter sets necessary to decode sample X and subsequent samples in decoding order may be available in the sample entry of the track.

[0228] ii) Alternatively, in the presence of SyncSampleBox and sample X is indicated as a sync sample, the file player can decode and play back itself starting from sample X. At this point, the parameter set necessary to decode sample X and subsequent samples in decoding order should be available in the sample entry of the track or in sample X.

[0229] iii) Alternatively, if the parameter set sample group box exists, the file player can find samples including parameter sets necessary for decoding from the information provided by the parameter set sample group box and start decoding from sample X.

[0230] iv) Alternatively (that is, the track is not "gpe1" or "gpc1" and the SyncSampleBox and Parameter Set Sample Group Box are not present), random access starting from sample X may not be specified.

[0231] Based on the above aspects, the embodiments of the present disclosure may be implemented as follows.

[0232] (1) SyncSampleBox

[0233] A sync sample may mean a sample in a track starting from an ISAU of a stream access point (SAP) of a predetermined type (e.g., type 1 or type 2) or a media sample that starts a sequence of new independent samples. In the case where the decoding process starts from a sync sample, the sample and subsequent samples in decoding order may be correctly decoded. In this context, the resulting set of decoded samples may form a correct presentation of the media starting from the decoded sample with the earliest composition time.

[0234] The Sync Sample box can provide a simple marker for sync samples in a stream. An example of the syntax structure of the Sync Sample box is shown in Table 1.

[0235] [Table 1]

[0236]

[0237] In Table 1, the syntax element entry_count may indicate the number of entries in the subsequent table. If entry_count has a value of 0, there are no synchronization samples in the stream and the subsequent table may be empty. The syntax element sample_number may indicate the number of synchronization samples in the stream.

[0238] The sync sample box can exist in the following tracks.

[0239] - Tracks with sample entry type 'gpeg'

[0240] - Tracks that include a geometry component and have sample entry type "gpcg"

[0241] - Tracks with sample entry type "gpeb" or "gpcb"

[0242] The sync sample box does not have to exist in tracks with sample entry type “gpe1” and tracks with sample entry type “gpc1”. This is because all samples have intra-coded frames and the parameter sets necessary to decode samples in the track are transmitted in the sample entry.

[0243] SyncSampleBox may be constrained to not include a geometry component and not be present in a track with sample entry type "gpcg". In the case where samples in a track including a geometry component and having sample entry type "gpcg" are indicated as sync samples (i.e., samples included in a SyncSampleBox), corresponding samples in the associated track (i.e., samples with the same decoding timestamp and / or presentation timestamp) may also be considered sync samples.

[0244] In the case where neither the SyncSampleBox nor the parameter sample set sample group box exists in a track with sample entry type "gpeg", a track with sample entry "gpcg" including a geometry component, or a track with sample entry "gpeb" or "gpcb", a method for performing random access playback may not be specified.

[0245] In case temporal scalability is supported and the bitstream is stored in multiple temporal level tracks, SyncSampleBox may only exist in the track including temporal level 0.

[0246] (2) Player Operation During Random Access

[0247] In case of random access starting from a specific sample X, the file player may perform the following process.

[0248] i) In case the sample entry of the track is "gpe1" or "gpc1", the file player can decode and play back by itself starting from sample X. In this context, the parameter sets necessary to decode sample X and subsequent samples in decoding order may be available in the sample entry of the track.

[0249] ii) Alternatively, in the presence of SyncSampleBox and sample X is indicated as a sync sample, the file player may decode and play back itself starting from sample X. In this context, the parameter sets necessary to decode sample X and subsequent samples in decoding order may be constrained to be available in the sample entry of the track or in sample X.

[0250] iii) Alternatively, if the parameter set sample group box exists, the file player can find a sample including a parameter set (e.g., SPS, GPS, APS, and FSAP) necessary for decoding from the information provided by the parameter set sample group box and start decoding from sample X. The sample including the parameter set may be sample X itself or a sample before sample X. In the case of finding the sample, the file parser can extract all parameter sets in the sample and make them available for decoding / playback starting from sample X.

[0251] iv) Alternatively (that is, the track is not "gpe1" or "gpc1" and the SyncSampleBox and no parameter set sample group box is present), random access starting from sample X may not be specified.

[0252] The above file player operation based on the synchronous sample frame property in the case of random access can be summarized as Fig.14 .

[0253] Reference Fig.14 , when a random access request occurs starting from a specific sample X, the file player may determine whether the track (including sample X) has a predetermined sample entry type "gpe1" or "gpc1" (S1410). As a result of the determination, if the track has the predetermined sample entry type (Yes at S1410), the file player may perform decoding for random access playback starting from sample X by referring to the parameter set in the sample entry of the track (S1440).

[0254] On the other hand, if the track does not have the predetermined sample entry type (No in S1410), the file player may determine whether a SyncSampleBox exists and whether sample X is indicated as a synchronization sample (S1420). As a result of the determination, if a SyncSampleBox exists and sample X is indicated as a synchronization sample (Yes in S1420), the file player may start decoding for random access playback from sample X by referring to a parameter set in a sample entry of the track or in sample X (S1440).

[0255] On the other hand, if the SyncSampleBox does not exist or the sample X is not indicated as a synchronization sample (No in S1420), the file player may determine whether a parameter set sample group box exists (S1430). As a result of the determination, if the parameter set sample group box exists (Yes in S1430), the file player may start decoding for random access playback from the sample X by referring to the parameter set obtained based on the parameter set sample group box (S1440).

[0256] Furthermore, if the parameter set sample group box does not exist (No of S1430 ), the file player may not perform decoding starting from sample X, thereby determining that random access playback is impossible.

[0257] Therefore, according to the present disclosure, a detailed mechanism of SyncSampleBox properties and player operations can be provided for random access. Therefore, the random access function can be correctly supported.

[0258] Hereinafter, a method performed in a receiving device and a transmitting device of point cloud data according to an embodiment of the present disclosure will be described in detail.

[0259] Fig.15 Detailed description is a flowchart illustrating a method executed in a receiving device for point cloud data according to an embodiment of the present disclosure.

[0260] Reference Fig.15 , the receiving device may obtain a parameter set required for random access from the point cloud data based on a request for random access starting from a first sample among samples included in the point cloud data ( S1510 ).

[0261] In addition, based on the parameter set, the receiving apparatus may decode the first sample and one or more samples subsequent to the first sample in the decoding order ( S1520 ).

[0262] In one embodiment, based on the sample entry type of the track including the first sample being "gpe1" or "gpc1", the parameter set may be obtained from the sample entry of the track.

[0263] In one embodiment, based on the track sample entry type of "gpe1" or "gpc1", the point cloud data may not include a synchronization sample box.

[0264] In one embodiment, when the sample entry type of the track including the first sample is not "gpe1" or "gpc1", a parameter set may be obtained from the sample entry of the track or the first sample based on the point cloud data including a synchronization sample box and the first sample being indicated as a synchronization sample.

[0265] In one embodiment, the synchronization sample box may exist in a predetermined track of the point cloud data. In this document, the predetermined track may be any of a track having a sample entry type "gpeg", a track including a geometry component and having a sample entry type "gpcg", or a track having a sample entry type "gpeb" or "gpcb".

[0266] In one embodiment, a sync sample box may be constrained to include no geometry component and not be present in tracks with sample entry type "gpcg".

[0267] In one embodiment, based on samples in a track including a geometry component and having a sample entry type of "gpcg" being indicated as synchronization samples, corresponding samples in associated tracks may be considered synchronization samples.

[0268] In one embodiment, when the point cloud data does not include a synchronization sample box or the first sample is not indicated as a synchronization sample, the parameter set may be obtained from the first sample or a sample preceding the first sample in the decoding order based on the point cloud data including a parameter set sample group box. On the other hand, when the point cloud data does not include a synchronization sample box, the above-mentioned acquisition (S1510) and decoding (S1520) of the parameter set may not be performed.

[0269] Fig.16 Detailed description is a flowchart illustrating a method executed in a point cloud data transmitting device according to an embodiment of the present disclosure.

[0270] Reference Fig.16 , the transmitting device may store a bit stream including point cloud data in one or more tracks (S1610).

[0271] In addition, the transmitting device may generate a G-PCC file based on the track (S1620).

[0272] In one embodiment, based on the sample entry type of the track including the first sample being "gpe1" or "gpc1", a parameter set required to perform random access from the first sample among samples included in the point cloud data may be included in the sample entry of the track.

[0273] In one embodiment, when the sample entry type of the track is not "gpe1" or "gpc1", based on the point cloud data including a synchronization sample box and the first sample being indicated as a synchronization sample, a parameter set may be included in the sample entry of the track or the first sample.

[0274] In one embodiment, when the point cloud data does not include a synchronization sample box or the first sample is not indicated as a synchronization sample, based on the point cloud data including a parameter set sample group box, the parameter set may be included in the first sample or a sample preceding the first sample in the decoding order. On the other hand, when the point cloud data does not include a parameter set sample group box, the parameter set may not be included in the point cloud data.

[0275] Although the exemplary methods of the present disclosure are represented as a series of operations for the sake of clarity of description, the order of the steps is not limited thereto, and the steps shown may be performed simultaneously or in a different order if necessary. To implement the methods according to the present disclosure, other steps may be added to the exemplary steps, some steps may be excluded and the remaining steps may be included, or some steps may be excluded and additional steps may be included.

[0276] In the present disclosure, a sending device or a receiving device that performs a predetermined operation (step) may perform an operation (step) for checking a condition or situation for performing the predetermined operation (step). For example, in the case where it is specified that the predetermined operation is not performed unless the predetermined condition is met, the image encoding device or the image decoding device may perform the predetermined operation after performing the operation for checking whether the predetermined condition is met.

[0277] The various embodiments of the present disclosure are not intended to list all possible combinations, but rather to illustrate representative aspects of the present disclosure. Matters described in the various embodiments may be applied independently or in combination of two or more.

[0278] In addition, various embodiments of the present disclosure may be implemented by hardware, firmware, software, or a combination thereof. With regard to hardware implementation, the embodiments may be implemented by using at least one selected from the group consisting of an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a general purpose processor, a controller, a microcontroller, a microprocessor, and the like.

[0279] The scope of the present disclosure includes software or machine-executable commands (e.g., operating systems, applications, firmware, programs, etc.) for enabling operations of methods according to various embodiments to be performed on a device or computer, and non-transitory computer-readable media having such software or commands stored thereon and executable on a device or computer.

[0280] Industrial Applicability

[0281] The embodiments of the present disclosure may be used to provide point cloud content. In addition, the embodiments of the present disclosure may be used to encode / decode point cloud data.

Claims

1. A method performed by a receiving device of point cloud data, the method comprising the following steps: Based on a request for random access starting from a first sample in the point cloud data, obtaining a parameter set required for the random access from the point cloud data; as well as decoding the first sample and one or more samples following the first sample in decoding order based on the obtained parameter set, The parameter set is obtained from a sample entry or a predetermined sample of a track including the first sample based on at least one of a sample entry type of the track, the presence of a synchronization sample box, or the presence of a parameter set sample group box.

2. The method according to claim 1, in, The parameter set is obtained from the sample entry of the track based on that the sample entry type of the track is "gpe1" or "gpc1".

3. The method according to claim 1, in, Based on the sample entry type of the track being "gpe1" or "gpc1", the point cloud data does not include the synchronization sample frame.

4. The method according to claim 1, in, based on the sample entry type of the track being not "gpe1" or "gpc1", The parameter set is obtained from the sample entry of the track or the first sample based on the point cloud data including the synchronization sample box and the first sample being indicated as a synchronization sample.

5. The method according to claim 4, in, The synchronization sample frame exists in a track with a sample entry type of "gpeg", a track including a geometry component and having a sample entry type of "gpcg", or a track with a sample entry type of "gpeb" or "gpcb".

6. The method according to claim 4, in, The synchronization sample box is constrained to not exist in tracks with sample entry type "gpcg" that do not include a geometry component.

7. The method according to claim 4, in, Based on the samples in a track with sample entry type "gpcg" including a geometry component being indicated as synchronization samples, the corresponding samples in the associated track are regarded as synchronization samples.

8. The method according to claim 4, in, Based on the fact that the point cloud data does not include the synchronization sample frame or the first sample is not indicated as the synchronization sample, Based on the point cloud data including the parameter set sample group box, the parameter set is obtained from the first sample or a sample preceding the first sample in a decoding order.

9. The method according to claim 8, in, Based on the fact that the point cloud data does not include the parameter set sample group box, the step of obtaining the parameter set and the step of decoding are not performed.

10. A point cloud data receiving device, the receiving device comprising: Memory; as well as at least one processor, the at least one processor being configured to: Based on a request for random access starting from a first sample among samples in the point cloud data, obtaining a parameter set required for the random access from the point cloud data, and decoding the first sample and one or more samples following the first sample in decoding order based on the obtained parameter set, and The parameter set is obtained from a sample entry or a predetermined sample of a track including the first sample based on at least one of a sample entry type of the track, the presence of a synchronization sample box, or the presence of a parameter set sample group box.

11. A method performed by a point cloud data transmitting device, the method comprising the following steps: storing a bitstream including the point cloud data into one or more tracks; as well as Generate a geometry-based point cloud compressed G-PCC file based on the track, Wherein, based on at least one of the sample entry type of the track, the presence of a synchronization sample box, or the presence of a parameter set sample group box, a parameter set required for random access starting from a first sample within the point cloud data is included in the sample entry or predetermined sample of the track including the first sample.

12. The method according to claim 11, in, Based on the sample entry type of the track including the first sample being "gpe1" or "gpc1", the parameter set is included in the sample entry of the track.

13. The method according to claim 11, in, The sample entry type based on the track is not "gpe1" or "gpc1", Based on the point cloud data including the synchronization sample box and the first sample being indicated as a synchronization sample, the parameter set is included in the sample entry of the track or the first sample.

14. The method according to claim 12, in, Based on the fact that the point cloud data does not include the synchronization sample frame or the first sample is not indicated as the synchronization sample, Based on the point cloud data including the parameter set sample group box, the parameter set is included in the first sample or a sample preceding the first sample in a decoding order.

15. A point cloud data sending device, the sending device comprising: Memory; as well as at least one processor, the at least one processor being configured to: storing a bitstream comprising the point cloud data into one or more tracks, and generating a geometry-based point cloud compression G-PCC file based on the track, and Wherein, based on at least one of the sample entry type of the track, the presence of a synchronization sample box, or the presence of a parameter set sample group box, a parameter set required for random access starting from a first sample within the point cloud data is included in the sample entry or predetermined sample of the track including the first sample.

Images