Point cloud data transmitting device, method executed by transmitting device, point cloud data receiving device, and method executed by receiving device
By using G-PCC time-scalable grouping tracks in the point cloud data receiving and transmitting device, the problems of latency and encoding complexity in point cloud data processing are solved, achieving efficient point cloud services and time-scalable access, and improving system performance.
Patent Information
- Application Number
- CN202380033904.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-04-14
- Filing Date
- 2023-04-14
- Publication Date
- 2024-11-26
AI Technical Summary
Existing technologies struggle to efficiently process point cloud data, particularly when dealing with large amounts of point data, as they suffer from latency and high encoding/decoding complexity. Furthermore, they cannot support G-PCC bitstreams with time scalability and efficient access to storage.
By using G-PCC temporal scalability group boxes in the point cloud data receiving and transmitting devices to combine tracks, time-scalable track groups are supported, time track mixing is prevented, and point cloud data is accessed efficiently through TLV encapsulation structure.
It achieves efficient processing of point cloud data, provides high-quality point cloud services, supports time scalability, reduces bitstream processing time, and improves the performance of the point cloud content delivery system.
Smart Images

Figure CN119032573A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to methods and apparatus for processing point cloud content. Background Art
[0002] Point cloud content is expressed as a point cloud that is a set of points belonging to a coordinate system representing a three-dimensional space. Point cloud content can represent three-dimensional media and is used to provide various services such as virtual reality (VR), augmented reality (AR), mixed reality (MR), and self-driving services. Since tens of thousands to hundreds of thousands of point data are required to express point cloud content, a method for efficiently processing a large amount of point data is needed. Summary of the invention
[0003] Technical issues
[0004] The purpose of the present disclosure is to provide a method and device for efficiently processing point cloud data. Another purpose of the present disclosure is to provide a method and device for processing point cloud data to solve the time delay and encoding / decoding complexity.
[0005] Another object of the present disclosure is to provide a method and apparatus for supporting temporal scalability track groups.
[0006] Another object of the present disclosure is to provide a method and apparatus for processing file storage technology to support efficient access to stored G-PCC bitstreams.
[0007] The technical problems solved by the present disclosure are not limited to the above-mentioned technical problems, and other technical problems not described herein will be apparent to those skilled in the art through the following description.
[0008] Technical Solution
[0009] According to an embodiment of the present disclosure, a method performed in a receiving device of point cloud data may include: obtaining a geometry-based point cloud compression (G-PCC) file including point cloud data; and obtaining a G-PCC temporal scalability group box GPCCTemporalScalabilityGroupBox from a track in the G-PCC file. A sample entry type of the track may be one of "gpel", "gpeg", "gpcl" or "gpcg".
[0010] According to one embodiment of the present disclosure, the track may include a geometric component.
[0011] According to one embodiment of the present disclosure, the sample entry type of a track may be one of “gpcl” or “gpcg”.
[0012] According to one embodiment of the present disclosure, based on the track including the attribute component and the sample entry type of the track is “gpcl” or “gpcg”, the G-PCC temporal scalability group box may not be obtained from the track.
[0013] According to one embodiment of the present disclosure, based on the first tracks included in the same time scalability group box being combined, the second track referenced by the first track may be combined, the first track may be a track including a geometry component, and the second track may be a track including an attribute component.
[0014] According to one embodiment of the present disclosure, the first track may be combined based on whether the sample entry type of the track is one of “gpcl” or “gpcg”.
[0015] According to an embodiment of the present disclosure, a method performed in a transmitting device of point cloud data may include: generating a track including a G-PCC temporal scalability group box GPCCTemporalScalabilityGroupBox; and generating a G-PCC file including the track. The sample entry type of the track may be one of "gpel", "gpeg", "gpcl" or "gpcg".
[0016] According to another embodiment of the present disclosure, a receiving device of point cloud data may include a memory and at least one processor. The at least one processor may obtain a geometry-based point cloud compression (G-PCC) file including point cloud data; and obtain a G-PCC temporal scalability group box GPCCTemporalScalabilityGroupBox from a track in the G-PCC file. The sample entry type of the track may be one of "gpel", "gpeg", "gpcl" or "gpcg".
[0017] According to another embodiment of the present disclosure, a device for transmitting point cloud data may include a memory and at least one processor. The at least one processor may generate a track including a G-PCC temporal scalability group box GPCCTemporalScalabilityGroupBox; and generate a G-PCC file including the track. The sample entry type of the track may be one of "gpel", "gpeg", "gpcl" or "gpcg".
[0018] Beneficial Effects
[0019] The apparatus and method according to the embodiments of the present disclosure can process point cloud data with high efficiency.
[0020] The apparatus and method according to the embodiments of the present disclosure can provide high-quality point cloud services.
[0021] The device and method according to the embodiments of the present disclosure may provide point cloud content for providing general services such as VR services and autonomous driving services.
[0022] The apparatus and method according to the embodiments of the present disclosure may provide temporal scalability capable of efficiently accessing a desired component among G-PCC components.
[0023] The apparatus and method according to an embodiment of the present disclosure may provide a time scalability track group, thereby preventing time tracks from being mixed.
[0024] The apparatus and method according to an embodiment of the present disclosure may perform temporal scalability grouping when a plurality of temporal level tracks exist, thereby reducing bits. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 is a block diagram illustrating an example of a point cloud content providing system according to an embodiment of the present disclosure.
[0026] Figure 2 is a block diagram illustrating an example of a point cloud content providing process according to an embodiment of the present disclosure.
[0027] Figure 3 is a diagram illustrating an example of a point cloud encoding device according to an embodiment of the present disclosure.
[0028] Figure 4 is a block diagram illustrating an example of a point cloud decoding device according to an embodiment of the present disclosure.
[0029] Figure 5 is a block diagram illustrating another example of a point cloud decoding device according to an embodiment of the present disclosure.
[0030] Figure 6 An example of a structure capable of interworking with a method / apparatus for transmitting and receiving point cloud data according to an embodiment of the present disclosure is illustrated.
[0031] Figure 7 is a block diagram illustrating another example of a transmitting device according to an embodiment of the present disclosure.
[0032] Figure 8 is a block diagram illustrating another example of a receiving device according to an embodiment of the present disclosure.
[0033] Fig. 9 An example of a TLV encapsulation structure according to an embodiment of the present disclosure is illustrated.
[0034] FIG. 10 illustrates an example of a TLV encapsulation syntax structure and payload types according to an embodiment of the present disclosure.
[0035] Fig.11 An example for a tile including a single track according to an embodiment of the present disclosure is illustrated.
[0036] Fig.12 An example for a tile including a plurality of tracks according to an embodiment of the present disclosure is illustrated.
[0037] Fig.13 is a flowchart of a method for obtaining a temporal scalability group box GPCCTemporalScalabilityGroupBox in a receiving device according to an embodiment of the present disclosure.
[0038] Fig.14 is a flowchart of a method for generating a G-PCC file in a transmitting device according to an embodiment of the present disclosure.
[0039] Figure 15 to Figure 16 is a flowchart of a method for generating a G-PCC file in a transmitting device according to an embodiment of the present disclosure.
[0040] Fig.17 is a flowchart of a method for obtaining a geometry / attribute component in a receiving device according to an embodiment of the present disclosure.
[0041] Fig.18 is a flowchart of a method for combining tracks in a transmitting device according to an embodiment of the present disclosure. DETAILED DESCRIPTION
[0042] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art to which the present disclosure belongs can easily implement them. The present disclosure can be implemented in many different forms and is not limited to the embodiments described herein.
[0043] In describing the present disclosure, a detailed description of known functions and configurations may be omitted when it may obscure the gist of the present disclosure. In the drawings, parts irrelevant to the description of the present disclosure are omitted, and like reference numerals are given to like parts.
[0044] In the present disclosure, when one component is "connected," "coupled," or "linked" to another component, it may include not only a direct connection relationship but also an indirect connection relationship with another component therebetween. In addition, when it is said that one component "includes" or "has" another component, this indicates that other components are not excluded but also included unless specifically described.
[0045] In the present disclosure, terms such as first, second, etc. are used only for the purpose of distinguishing one component from other components, and do not limit the order or importance of the components unless otherwise specified. Therefore, within the scope of the present disclosure, a first component in one embodiment may be referred to as a second component in another embodiment, and similarly, a second component in one embodiment may be referred to as a first component in another embodiment.
[0046] In the present disclosure, components that are distinguished from each other are used to clearly explain their features and do not necessarily mean that these components are separated. That is, multiple components can be integrated to form one hardware or software unit, or one component can be distributed to form multiple hardware or software units. Therefore, even if not specifically mentioned, such integrated or distributed implementations are also included in the scope of the present disclosure.
[0047] In the present disclosure, the components described in various embodiments are not necessarily essential components, and some of them may be optional components. Therefore, an embodiment consisting of a subset of the components described in an embodiment is also included in the scope of the present disclosure. In addition, an embodiment including other components in addition to the components described in various embodiments is also included in the scope of the present disclosure.
[0048] The present disclosure relates to encoding and decoding of point cloud related data, and unless the terms used in the present disclosure are redefined in the present disclosure, the terms may have general meanings commonly used in the technical field to which the present disclosure belongs.
[0049] In the present disclosure, the terms " / " and "," should be interpreted as indicating "and / or". For example, the expressions "A / B" and "A, B" may mean "A and / or B". In addition, "A / B / C" and "A / B / C" may mean "at least one of A, B, and / or C".
[0050] In the present disclosure, the term "or" should be interpreted as indicating "and / or". For example, the expression "A or B" may include 1) only "A", 2) only "B", and / or 3) both "A and B". In other words, in the present disclosure, the term "or" should be interpreted as indicating "additionally or alternatively".
[0051] The present disclosure relates to compression of point cloud related data. Various methods or embodiments of the present invention can be applied to point cloud compression or point cloud coding (PCC) standards of the Moving Picture Experts Group (MPEG) (e.g., G-PCC or V-PCC standards) or next generation video / image coding standards.
[0052] In the present disclosure, "point cloud" may refer to a set of points located in a three-dimensional space. Moreover, in the present disclosure, "point cloud content" is expressed as a point cloud, and may refer to a "point cloud video / image". Hereinafter, a "point cloud video / image" is referred to as a "point cloud video". A point cloud video may include one or more frames, and a frame may be a still image or a picture. Therefore, a point cloud video may include a point cloud image / frame / picture, and may be referred to as any one of a "point cloud image", a "point cloud frame", and a "point cloud picture".
[0053] In the present disclosure, "point cloud data" may refer to data or information related to each point in a point cloud. Point cloud data may include geometry and / or attributes. In addition, point cloud data may also include metadata. Point cloud data may be referred to as "point cloud content data" or "point cloud video data", etc. In addition, point cloud data may be referred to as "point cloud content", "point cloud video", "G-PCC data", etc.
[0054] In the present disclosure, a point cloud object corresponding to point cloud data may be represented by a frame shape based on a coordinate system, and the frame shape based on a coordinate system may be referred to as a bounding box. That is, the bounding box may be a cuboid that can accommodate all points of the point cloud, and may be a cuboid that includes a source point cloud frame.
[0055] In the present disclosure, geometry includes the position (or position information) of each point, and the position can be represented by parameters (e.g., x-axis value, y-axis value, and z-axis value) representing a three-dimensional coordinate system (e.g., a coordinate system composed of an x-axis, a y-axis, and a z-axis). Geometry can be referred to as "geometric information".
[0056] In the present disclosure, the attribute may include the attribute of each point, and the attribute may include one or more of texture information, color (RGB or YCbCr), reflectivity (r), transparency, etc. of each point. The attribute may be referred to as "attribute information". Metadata may include various data related to acquisition in an acquisition process to be described later.
[0057] Overview of Point Cloud Content Provision System
[0058] Figure 1 An example of a system for providing point cloud content (hereinafter, referred to as a “point cloud content providing system”) according to an embodiment of the present disclosure is illustrated. Figure 2 An example of a process in which a point cloud content providing system provides point cloud content is illustrated.
[0059] like Figure 1 As shown, the point cloud content providing system may include a sending device 10 and a receiving device 20. The point cloud content providing system may be performed by the operations of the sending device 10 and the receiving device 20. Figure 2The obtained process S20, the encoded process S21, the sent process S22, the decoded process S23, the rendered process S24 and / or the feedback process S25 are shown.
[0060] The sending device 10 acquires point cloud data, and outputs a bit stream through a series of processes (e.g., encoding process) for the acquired point cloud data (source point cloud data) to provide point cloud content. Here, the point cloud data can be output in the form of a bit stream through the encoding process. In some embodiments, the sending device 10 can send the output bit stream to the receiving device 20 in the form of a file or streaming (streaming segment) through a digital storage medium or a network. The digital storage medium may include a variety of storage media, such as USB, SD, CD, DVD, Blu-ray disc, HDD, and SSD. The receiving device 20 can process (e.g., decode or reconstruct) the received data (e.g., encoded point cloud data) into source point cloud data and render it. Point cloud content can be provided to users through these processes, and the present disclosure can provide various embodiments required to efficiently perform a series of these processes.
[0061] like Figure 1 As illustrated, the sending device 10 may include: an acquisition unit 11, an encoding unit 12, an encapsulation processing unit 13 and a sending unit 14, and the receiving device 20 may include: a receiving unit 21, a decapsulation processing unit 22, a decoding unit 23, and a rendering unit 24.
[0062] The acquisition unit 11 may perform a process S20 of acquiring a point cloud video by a capturing, composing or generating process. Therefore, the acquisition unit 11 may be referred to as a 'point cloud video acquisition unit'.
[0063] Point cloud data (geometry and / or attributes, etc.) of a plurality of points may be generated through the acquisition process (S20). Furthermore, metadata related to the acquisition of the point cloud video may be generated through the acquisition process (S20). Furthermore, mesh data (e.g., triangle data) indicating connection information between point clouds may be generated through the acquisition process (S20).
[0064] The metadata may include initial viewing orientation metadata. The initial viewing orientation metadata may indicate whether the point cloud data represents front-side or back-side data. The metadata may be referred to as "auxiliary data," which is metadata of the point cloud.
[0065] The acquired point cloud video may include a polygon file format or a Stanford triangle format (PLY) file. Since the point cloud video has one or more frames, the acquired point cloud video may include one or more PLY files. The PLY file may include point cloud data of each point.
[0066] In order to acquire point cloud video (or point cloud data), the acquisition unit 11 may be composed of a camera device capable of acquiring depth (depth information) and a combination of an RGB camera capable of extracting color information corresponding to the depth information. Here, the camera device capable of acquiring depth information may be a combination of an infrared pattern projector and an infrared camera. In addition, the acquisition unit 11 may be composed of a laser radar (LiDAR), and the laser radar may use a radar system for measuring the position coordinates of a reflector by measuring the time required for a laser pulse to be emitted and to return after being reflected.
[0067] The acquisition unit 110 may extract a shape of geometry composed of points in a three-dimensional space from the depth information, and may extract a property representing the color or reflection of each point from the RGB information.
[0068] As a method of extracting (or capturing, acquiring, etc.) a point cloud video (or point cloud data), there may be an inward-facing method of capturing a central object and an outward-facing method of capturing an external environment.
[0069] The encoding unit 12 may perform an encoding process (S21) of encoding the data generated by the acquisition unit 11 (e.g., geometry, attributes and / or metadata, and / or mesh data, etc.) into one or more bitstreams. Therefore, the encoding unit 12 may be referred to as a "point cloud video encoder". The encoding unit 12 may encode the data generated by the acquisition unit 11 serially or in parallel.
[0070] The encoding process S21 performed by the encoding unit 12 may be geometry-based point cloud compression (G-PCC). The encoding unit 12 may perform a series of processes such as prediction, transform, quantization, and entropy encoding for compression and encoding efficiency.
[0071] The encoded point cloud data can be output in the form of a bitstream. Based on the G-PCC process, the encoding unit 12 can split the point cloud data into geometry and attributes and encode them as described below. In this case, the output bitstream may include a geometry bitstream containing the encoded geometry and an attribute bitstream containing the encoded attributes. In addition, the output bitstream may also include one or more of a metadata bitstream containing metadata, an auxiliary bitstream containing auxiliary data, and a mesh data bitstream containing mesh data. The encoding process (S21) will be described in more detail below. The bitstream including the encoded point cloud data may be referred to as a "point cloud bitstream" or a "point cloud video bitstream".
[0072] The encapsulation processing unit 13 may perform a process of encapsulating one or more bit streams output from the encoding unit 12 in the form of a file or segment. Therefore, the encapsulation processing unit 13 may be referred to as a "file / segment encapsulation module". Although the drawings show an example in which the encapsulation processing unit 13 is composed of a separate component / module related to the sending unit 14, in some embodiments, the encapsulation processing unit 13 may be included in the sending unit 14.
[0073] The encapsulation processing unit 13 may encapsulate data in a file format such as ISO Base Media File Format (ISOBMFF), or process data in the form of other DASH segments. In some embodiments, the encapsulation processing unit 13 may include metadata of the file format. The metadata may be included in boxes at various levels of, for example, the ISOBMFF file format, or as data in a separate track within a file. In some embodiments, the encapsulation processing unit 130 may encapsulate the metadata itself into a file. The metadata processed by the encapsulation processing unit 13 may be sent from a metadata processing unit not shown in the figure. The metadata processing unit may be included in the encoding unit 12, or may be configured as a separate component / module.
[0074] The sending unit 14 may perform a sending process (S22) of applying a process according to a file format (a process for sending) to the “encapsulated point cloud bitstream”. The sending unit 140 may send a bitstream or a file / segment including a bitstream to the receiving unit 21 of the receiving device 20 through a digital storage medium or a network. Therefore, the sending unit 14 may be referred to as a “sender” or a “communication module”.
[0075] The sending unit 14 may process the point cloud data according to any transmission protocol. Here, "processing the point cloud data according to any transmission protocol" may be "processing for sending". The processing for sending may include processing for sending via a broadcast network / processing for sending via broadband, etc. In some embodiments, the sending unit 14 may receive not only the point cloud data from the metadata processing unit, but also metadata from the metadata processing unit, and may perform processing for sending on the sent metadata. In some embodiments, the processing for sending may be performed by the sending processing unit, and the sending processing unit may be included in the sending unit 14, or configured as a component / module separate from the sending unit 14.
[0076] The receiving unit 21 can receive the bitstream or the file / segment including the bitstream sent by the sending device 10. According to the channel sent, the receiving unit 21 can receive the bitstream or the file / segment including the bitstream through a broadcast network, or can receive the bitstream or the file / segment including the bitstream through a broadband. Alternatively, the receiving unit 21 can receive the bitstream or the file / segment including the bitstream through a digital storage medium.
[0077] The receiving unit 21 may perform processing on the received bitstream or the file / segment including the bitstream according to the transmission protocol. The receiving unit 21 may perform the inverse process of the sending process (processing for sending) to correspond to the processing for sending performed by the sending device 10. The receiving unit 21 may send the encoded point cloud data among the received data to the decapsulation processing unit 22, and may send metadata to the metadata parsing unit. The metadata may be in the form of a signaling table. In some embodiments, the inverse process of the processing for sending may be performed in the receiving processing unit. Each of the receiving processing unit, the decapsulation processing unit 22, and the metadata parsing unit may be included in the receiving unit 21, or may be configured as a component / module separate from the receiving unit 21.
[0078] The decapsulation processing unit 22 can decapsulate the point cloud data in the file format (ie, the bit stream in the file format) received from the receiving unit 21 or the receiving processing unit. Therefore, the decapsulation processing unit 22 can be called a "file / segment decapsulation module".
[0079] The decapsulation processing unit 22 can obtain a point cloud bitstream or a metadata bitstream by decapsulating the file according to ISOBMFF, etc. In some embodiments, metadata (metadata bitstream) can be included in the point cloud bitstream. The acquired point cloud bitstream can be sent to the decoding unit 23, and the acquired metadata bitstream can be sent to the metadata processing unit. The metadata processing unit can be included in the decoding unit 23, or can be configured as a separate component / module. The metadata obtained by the decapsulation processing unit 22 can be in the form of a box or track in the file format. If necessary, the decapsulation processing unit 22 can receive the metadata required for decapsulation from the metadata processing unit. The metadata can be sent to the decoding unit 23 and used in the decoding process (S23), or can be sent to the rendering unit 24 and used in the rendering process (S24).
[0080] The decoding unit 23 may receive the bitstream and perform operations corresponding to those of the encoding unit 12, thereby performing a decoding process (S23) of decoding the point cloud bitstream (encoded point cloud data). Therefore, the decoding unit 23 may be referred to as a "point cloud video decoder".
[0081] The decoding unit 23 may segment the point cloud data into geometry and attributes and decode them. For example, the decoding unit 23 may reconstruct (decode) the geometry from the geometry bitstream included in the point cloud bitstream, and restore (decode) the attributes based on the reconstructed geometry and the attribute bitstream included in the point cloud bitstream. The three-dimensional point cloud video / image may be reconstructed based on the position information according to the reconstructed geometry and the attributes (such as color or texture) according to the decoded attributes. The decoding process (S23) will be described in more detail below.
[0082] The rendering unit 24 may perform a rendering process S24 of rendering the reconstructed point cloud video. Therefore, the rendering unit 24 may be referred to as a “renderer”.
[0083] The rendering process S24 may refer to a process of rendering and displaying point cloud content in a 3D space. The rendering process S24 may perform rendering according to a desired rendering method based on the position information and attribute information of the points decoded by the decoding process.
[0084] The feedback process S25 may include a process of sending various feedback information obtained during the rendering process S24 or the display process to other components in the sending device 10 or the receiving device 20. The feedback process S25 may be performed by Figure 1 The receiving device 20 of FIG. 10 and FIG. 11 may be executed by one or more components included in the receiving device 20, or may be executed by FIG. Fig.11 In some implementations, the feedback process S25 may be performed by a "feedback unit" or a "sensing / tracking unit".
[0085] Overview of Point Cloud Coding Equipment
[0086] Figure 3 An example of a point cloud encoding device 300 according to an embodiment of the present disclosure is illustrated. Figure 3 The point cloud encoding device 300 may correspond to Figure 1 The encoding unit 12.
[0087] like Figure 3 As shown, the point cloud encoding device 300 may include: a coordinate system transformation unit 305, a geometry quantization unit 310, an octree analysis unit 315, an approximation unit 320, a geometry encoding unit 325, a reconstruction unit 330, and an attribute transformation unit 340, a RAHT transformation unit 345, an LOD generation unit 350, a lifting unit 355, an attribute quantization unit 360, an attribute encoding unit 365, and / or a color transformation unit 335.
[0088] The point cloud data acquired by the acquisition unit 11 may undergo a process of adjusting the quality of the point cloud content (e.g., lossless, lossy, near lossless) according to the network situation or application. In addition, each point of the acquired point cloud content may be sent losslessly, but in this case, because the size of the point cloud content is large, real-time streaming is not possible. Therefore, in order to smoothly provide the point cloud content, a process of reconstructing the point cloud content according to the maximum target bit rate is required.
[0089] The process of adjusting the quality of the point cloud content may be a process of reconstructing and encoding the position information (position information included in the geometric information) or color information (color information included in the attribute information) of the point. The process of reconstructing and encoding the position information of the point may be referred to as geometric encoding, and the process of reconstructing and encoding the attribute information associated with each point may be referred to as attribute encoding.
[0090] Geometry coding may include: a geometry quantization process, a voxelization process, an octree analysis process, an approximation process, a geometry coding process, and / or a coordinate system transformation process. Moreover, geometry coding may also include a geometry reconstruction process. Attribute coding may include: a color transformation process, an attribute transformation process, a prediction transformation process, a lifting transformation process, a RAHT transformation process, an attribute quantization process, an attribute coding process, etc.
[0091] Geometry Encoding
[0092] The coordinate system transformation process may correspond to a process of transforming the coordinate system of the point position. Therefore, the coordinate system transformation process may be referred to as "transformed coordinates". The coordinate system transformation process may be performed by the coordinate system transformation unit 305. For example, the coordinate system transformation unit 305 may transform the position of the point from the global space coordinate system into position information in a three-dimensional space (e.g., a three-dimensional space expressed in a coordinate system of an X-axis, a Y-axis, and a Z-axis). The position information in the 3D space according to an embodiment may be referred to as "geometric information".
[0093] The geometric quantization process may correspond to a process of quantizing the position information of the points, and may be performed by the geometric quantization unit 310. For example, the geometric quantization unit 310 may find the position information having the minimum (x, y, z) value among the position information of the points, and subtract the position information having the minimum (x, y, z) position from the position information of each point. In addition, the geometric quantization unit 310 may multiply the subtracted value by a preset quantization scale value, and then adjust (lower or raise) the result to a value close to an integer, thereby performing the quantization process.
[0094] The voxelization process may correspond to a process of matching the geometric information quantized by the quantization process with a specific voxel existing in the 3D space. The voxelization process may also be performed by the geometric quantization unit 310. The geometric quantization unit 310 may perform octree-based voxelization based on the position information of the point so as to reconstruct each point to which the quantization process is applied.
[0095] The geometric coding process may correspond to a process of performing entropy coding on the occupancy code. The geometric coding process may be performed by the geometric coding unit 325. The geometric coding unit 325 may perform entropy coding on the occupancy code. The generated occupancy code may be encoded immediately or may be encoded by an intra / inter coding process to improve compression efficiency. The receiving device 20 may reconstruct the octet tree by the occupancy code.
[0096] On the other hand, in the case where there are no points or only a few points in a particular region, voxelizing all the regions may be inefficient. That is, since there are very few points in a particular region, it may not be necessary to construct the entire octree. For such cases, an early termination method may be required.
[0097] The point cloud encoding device 300 can directly send the position of the point only for a specific area, or use a surface model to reconfigure the position of the point within a specific area based on voxels, instead of splitting the node (specific node) corresponding to the specific area (a specific area not corresponding to a leaf node) into 8 sub-nodes (sub-nodes).
[0098] A mode for directly transmitting the positions of each point for a specific node may be a direct mode. The point cloud encoding device 300 may check whether a condition for enabling the direct mode is satisfied.
[0099] The conditions for enabling direct mode are: 1) the option to use direct mode should be enabled, 2) the specific node does not correspond to a leaf node, 3) there should be points below the threshold within the specific node, and 4) the total number of points to be sent directly does not exceed the limit value.
[0100] When all the above conditions are met, the point cloud encoding device 300 can directly entropy encode and send the position value of the point for a specific node through the geometric encoding unit 325.
[0101] The mode of reconstructing the position of the point in the specific area based on voxels using the surface model may be a trisoup mode. The trisoup mode may be performed by the approximation unit 320. The approximation unit 320 may determine a specific level of the octree, and reconstruct the position of the point in the node area based on voxels using the surface model from the determined specific level.
[0102] The point cloud encoding device 300 may selectively apply the triangle soup mode. Specifically, when the triangle soup mode is applied, the point cloud encoding device 300 may specify the level (specific level) at which the triangle soup mode is applied. For example, when the specified specific level is equal to the depth (d) of the octree, the triangle soup mode may not be applied. That is, the specified specific level should be less than the depth value of the octree.
[0103] The three-dimensional cube area of the node of the specified specific level is called a block, and a block can include one or more voxels. The block or voxel can correspond to a brick. Each block can have 12 edges, and the approximation unit 320 can check whether each edge is adjacent to an occupied voxel with a point. Each edge can be adjacent to multiple occupied voxels. The specific position of the edge adjacent to a voxel is called a vertex, and when multiple occupied voxels are adjacent to an edge, the approximation unit 320 can determine the average position of these positions as a vertex.
[0104] When there are vertices, the point cloud encoding device 300 can perform entropy encoding on the starting point (x, y, z) of the edge, the direction vector (Δx, Δy, Δz) of the edge, and the position value of the vertex (relative position value within the edge) through the geometric encoding unit 325.
[0105] The geometry reconstruction process may correspond to a process of generating reconstructed geometry by reconstructing an octree and / or an approximate octree. The geometry reconstruction process may be performed by the reconstruction unit 330. The reconstruction unit 330 may perform the geometry reconstruction process by triangle reconstruction, upsampling, voxelization, and the like.
[0106] When the triangle soup mode is applied in the approximation unit 320, the reconstruction unit 330 may reconstruct a triangle based on a start point of an edge, a direction vector of the edge, and a position value of a vertex.
[0107] The reconstruction unit 330 may perform an upsampling process for voxelization by adding points in the middle along the sides of the triangle. The reconstruction unit 330 may generate additional points based on the upsampling factor and the width of the block. These points may be referred to as refinement vertices. The reconstruction unit 330 may voxelize the refinement vertices, and the point cloud encoding device 300 may perform attribute encoding based on the voxelized position values.
[0108] In some embodiments, the geometry coding unit 325 can increase compression efficiency by applying context adaptive arithmetic coding. The geometry coding unit 325 can entropy encode the occupancy code directly using arithmetic codes. In some embodiments, the geometry coding unit 325 adaptively performs encoding based on the occupancy of neighboring nodes (intra-frame encoding), or adaptively performs encoding based on the occupancy code of the previous frame (inter-frame encoding). Here, a frame can refer to a set of point cloud data generated at the same time. Intra-frame encoding and inter-frame encoding are optional processes and can therefore be omitted.
[0109] Attribute Encoding
[0110] Attribute encoding may correspond to a process of encoding attribute information based on the reconstructed geometry and the geometry before the coordinate system transformation (source geometry). Since the attributes may depend on the geometry, the reconstructed geometry may be used for attribute encoding.
[0111] As mentioned above, attributes may include color, reflectivity, etc. The same attribute encoding method may be applied to the information or parameters included in the attributes. Color has three elements, reflectivity has one element, and each element may be processed independently.
[0112] Attribute coding may include: color transformation process, attribute transformation process, prediction transformation process, lifting transformation process, RAHT transformation process, attribute quantization process, attribute coding process, etc. The prediction transformation process, lifting transformation process, and RAHT transformation process may be selectively used, or a combination of one or more processes may be used.
[0113] The color conversion process may correspond to a process of converting the format of the color in the attribute to another format. The color conversion process may be performed by the color conversion unit 335. That is, the color conversion unit 335 may convert the color in the attribute. For example, the color conversion unit 335 may perform an encoding operation for converting the color in the attribute from RGB to YCbCr. In some embodiments, the operation of the color conversion unit 335 (i.e., the color conversion process) may be optionally applied according to the color value included in the attribute.
[0114] As described above, when one or more points exist in one voxel, the position values of the points existing in the voxel are set to the center point of the voxel so that they can be displayed by integrating them into one point information of the voxel. Therefore, a process of transforming the attribute values associated with the points may be required. Moreover, even if the triangulation mode is executed, the attribute transformation process may be performed.
[0115] The attribute transformation process may correspond to a process of transforming an attribute based on a position where geometry encoding is not performed and / or a reconstructed geometry. For example, the attribute transformation process may correspond to a process of transforming an attribute of a point having the position based on the position of the point included in the voxel. The attribute transformation process may be performed by the attribute transformation unit 340.
[0116] The attribute conversion unit 340 may calculate the center position value of the voxel and the average value of the attribute values of the neighboring points within a certain radius. Alternatively, the attribute conversion unit 340 may apply weights to the attribute values according to the distance from the center position and calculate the average value of the attribute values to which the weights are applied. In this case, each voxel has a position and a calculated attribute value.
[0117] The prediction transformation process may correspond to a process of predicting the attribute value of the current point based on the attribute values of one or more points (neighboring points) adjacent to the current point (corresponding to the prediction target point). The prediction transformation process may be performed by the level-of-detail (LOD) generation unit 350.
[0118] The prediction transformation is a method of applying the LOD transformation technology, and the LOD generation unit 350 may calculate and set the LOD value of each point based on the LOD distance value of each point.
[0119] The LOD generation unit 350 may generate a predictor for each point of the prediction transformation. Therefore, when there are N points, N predictors may be generated. The predictor may calculate and set a weight value (=1 / distance) based on the LOD value of each point, the index information of the neighboring point, and the distance value from the neighboring point. Here, the neighboring point may be a point that exists within the distance from the current point set for each LOD.
[0120] In addition, the predictor may multiply the attribute value of the neighboring point by the "set weight value" and set the value obtained by averaging the attribute values multiplied by the weight value as the predicted attribute value of the current point. The attribute quantization process may be performed on the residual attribute value obtained by subtracting the predicted attribute value of the current point from the attribute value of the current point.
[0121] The lifting transformation process may correspond to a process of reconstructing a point into a set of detail levels through an LOD generation process, just like the prediction transformation process. The lifting transformation process may be performed by the lifting unit 355. The lifting transformation process may also include a process of generating a predictor for each point, a process of setting the calculated LOD in the predictor, a process of registering neighbor points, and a process of setting weights according to the distance between the current point and the neighbor points.
[0122] The RAHT transformation process may correspond to a method of predicting attribute information of a higher-level node using attribute information associated with a lower-level node of the octree. That is, the RATH transformation process may correspond to an attribute information intra-frame encoding method performed by octree backward scanning. The RAHT transformation process may be performed by the RAHT transformation unit 345.
[0123] The RAHT transformation unit 345 scans the entire region in the voxel and can perform the RAHT transformation process up to the root node while summing (merging) the voxels into larger blocks at various steps. Since the RAHT transformation unit 345 only performs the RAHT transformation process on occupied nodes, in the case of an unoccupied empty node, the RAHT transformation process can be performed on the higher-level node directly above it.
[0124] The attribute quantization process may correspond to a process of quantizing the attribute output from the RAHT transform unit 345, the LOD generation unit 350, and / or the lifting unit 355. The attribute quantization process may be performed by the attribute quantization unit 360. The attribute encoding process may correspond to a process of encoding the quantized attribute and outputting the attribute bitstream. The attribute encoding process may be performed by the attribute encoding unit 365.
[0125] Overview of Point Cloud Decoding Equipment
[0126] Figure 4 An example of a point cloud decoding device 400 according to an embodiment of the present disclosure is shown. Figure 4 The point cloud decoding device 400 may correspond in configuration and function to Figure 1 The decoding unit 23.
[0127] The point cloud decoding apparatus 400 may perform a decoding process based on the data (bit stream) transmitted from the transmitting device 10. The decoding process may include a process of reconstructing (decoding) a point cloud video by performing an operation corresponding to the above-described encoding operation on the bit stream.
[0128] like Figure 4 As illustrated, the decoding process may include a geometry decoding process and an attribute decoding process. The geometry decoding process may be performed by a geometry decoding unit 410, and the attribute decoding process may be performed by an attribute decoding unit 420. That is, the point cloud decoding device 400 may include a geometry decoding unit 410 and an attribute decoding unit 420.
[0129] The geometry decoding unit 410 may reconstruct the geometry from the geometry bitstream, and the attribute decoding unit 420 may reconstruct the attribute based on the reconstructed geometry and the attribute bitstream. Furthermore, the point cloud decoding apparatus 400 may reconstruct the three-dimensional point cloud video (point cloud data) based on the position information according to the reconstructed geometry and the attribute information according to the reconstructed attribute.
[0130] Figure 5 A specific example of a point cloud decoding device 500 according to another embodiment of the present disclosure is illustrated. Figure 5 As illustrated, the point cloud decoding device 500 includes: a geometry decoding unit 505, an octree synthesis unit 510, an approximate synthesis unit 515, a geometry reconstruction unit 520, a coordinate system inverse transformation unit 525, an attribute decoding unit 530, an attribute inverse quantization unit 535, a RATH transformation unit 550, an LOD generation unit 540, an inverse lifting unit 545, and / or a color inverse transformation unit 555.
[0131] The geometry decoding unit 505, the octree synthesis unit 510, the approximate synthesis unit 515, the geometry reconstruction unit 520, and the coordinate system inverse transformation unit 550 may perform geometry decoding. Figures 1 to 3 The geometric decoding may include direct coding and triangular geometric decoding. Direct coding and triangular geometric decoding may be applied selectively.
[0132] The geometry decoding unit 505 may decode the received geometry bitstream based on arithmetic coding. The operation of the geometry decoding unit 505 may correspond to the inverse process of the operation performed by the geometry encoding unit 325.
[0133] The octree synthesis unit 510 may generate an octree by obtaining an occupancy code (or information about geometry obtained as a result of decoding) from the decoded geometry bitstream. The operation of the octree synthesis unit 510 may correspond to the inverse process of the operation performed by the octree analysis unit 315.
[0134] When triangle soup geometry encoding is applied, the approximate synthesis unit 515 can synthesize the surface based on the decoded geometry and / or the generated octree.
[0135] The geometry reconstruction unit 520 may reconstruct the geometry based on the surface and the decoded geometry. When direct coding is applied, the geometry reconstruction unit 520 may directly bring and add the position information of the points to which direct coding is applied. In addition, when triangulated geometry coding is applied, the geometry reconstruction unit 520 may reconstruct the geometry by performing reconstruction operations (e.g., triangle reconstruction, upsampling, voxelization operations, etc.). The reconstructed geometry may include a point cloud picture or frame that does not include attributes.
[0136] The coordinate system inverse transformation unit 550 can obtain the position of the point by transforming the coordinate system based on the reconstructed geometry. For example, the coordinate system inverse transformation unit 550 can inversely transform the position of the point from a three-dimensional space (e.g., a three-dimensional space expressed by a coordinate system of an x-axis, a y-axis, and a z-axis, etc.) into position information of a global space coordinate system.
[0137] The attribute decoding unit 530, the attribute dequantization unit 535, the RATH transform unit 550, the LOD generator 540, the inverse lifting unit 545 and / or the color inverse transform unit 555 may perform attribute decoding. Attribute decoding may include: RAHT transform decoding, prediction transform decoding, and lifting transform decoding. The above three decodings may be selectively used, or a combination of one or more decodings may be used.
[0138] The attribute decoding unit 530 may decode the attribute bitstream based on arithmetic coding. For example, when there is no neighbor point in the predictor of each point and thus the attribute value of the current point is directly entropy encoded, the attribute decoding unit 530 may decode the attribute value (unquantized attribute value) of the current point. As another example, when there is a neighbor point in the predictor of the current point and thus the quantized residual attribute value is entropy encoded, the attribute decoding unit 530 may decode the quantized residual attribute value.
[0139] The attribute dequantization unit 535 may inversely quantize the decoded attribute bitstream or information about the attribute obtained as a result of decoding, and output the inversely quantized attribute (or attribute value). For example, when the quantized residual attribute value is output from the attribute decoding unit 530, the attribute inverse quantization unit 535 may inversely quantize the quantized residual attribute value to output the residual attribute value. The inverse quantization process may be selectively applied based on whether the attribute is encoded in the point cloud encoding device 300. That is, when there is no neighbor point in the predictor of each point and therefore the attribute value of the current point is directly encoded, the attribute decoding unit 530 may output the attribute value of the current point that is not quantized, and the attribute encoding process may be skipped.
[0140] The RATH transform unit 550, the LOD generation unit 540, and / or the inverse lifting unit 545 may process the reconstructed geometry and the inverse quantized attributes. The RATH transform unit 550, the LOD generation unit 540, and / or the inverse lifting unit 545 may selectively perform a decoding operation corresponding to the encoding operation of the point cloud encoding apparatus 300.
[0141] The color inverse transform unit 555 may perform inverse transform encoding to inversely transform a color value (or texture) included in the decoded attribute. The operation of the color inverse transform unit 555 may be selectively performed based on whether the color transform unit 335 operates.
[0142] Figure 6 An example of a structure capable of interworking with a method / apparatus for transmitting and receiving point cloud data according to an embodiment of the present disclosure is illustrated.
[0143] Figure 6 The structure of illustrates a configuration in which at least one of a server (AI server), a robot, an autonomous vehicle, an XR device, a smart phone, a home appliance, and / or an HMD is connected to a cloud network. A robot, an autonomous vehicle, an XR device, a smart phone, or a home appliance may be referred to as a device. In addition, the XR device may correspond to a point cloud data device (PCC) according to an embodiment, or may interact with a PCC device.
[0144] A cloud network may refer to a network that forms part of or exists within a cloud computing infrastructure. Here, a 3G network, a 4G or Long Term Evolution (LTE) network, or a 5G network may be used to configure the cloud network.
[0145] The server may be connected to at least one of a robot, an autonomous vehicle, an XR device, a smart phone, a home appliance, and / or an HMD via a cloud network and may assist at least a portion of the processing of the connected device.
[0146] HMD may mean one of the types in which an XR device and / or a PCC device according to an embodiment may be implemented. An HMD type device according to an embodiment may include a communication unit, a control unit, a memory unit, an I / O unit, a sensor unit, and a power supply unit.
[0147] <PCC+XR>
[0148] XR / PCC devices can be implemented by applying PCC and / or XR technologies, such as HMD, HUD set in a vehicle, TV, mobile phone, smart phone, computer, wearable device, home appliance, digital signage, vehicle, fixed robot or mobile robot.
[0149] The XR / PCC device can obtain information about the surrounding space or real objects by analyzing 3D point cloud data or image data obtained through various sensors or from external devices, generate position (geometry) data and attribute data of 3D points, and render and output the XR object to be output. For example, the XR / PCC device can output an XR object corresponding to the recognized object, including additional information about the recognized object.
[0150] <PCC + XR + Mobile Phone>
[0151] The XR / PCC device can be implemented by a mobile phone or the like by applying PCC technology. The mobile phone can decode and display point cloud content based on PCC technology.
[0152] <PCC + Autopilot + XR>
[0153] An autonomous vehicle can be implemented by a mobile robot, a vehicle, an unmanned aerial vehicle, etc. by applying PCC technology and XR technology. An autonomous vehicle applying XR / PCC technology can refer to an autonomous vehicle equipped with a unit for providing XR images, or an autonomous vehicle that undergoes control / interaction within an XR image. In particular, an autonomous vehicle that undergoes control / interaction within an XR image is distinguishable from an XR device and can work interactively with each other.
[0154] An autonomous vehicle equipped with a unit for providing XR / PCC images can obtain sensor information from sensors including cameras and output an XR / PCC image generated based on the obtained sensor information. For example, an autonomous vehicle has an HUD and can provide an XR / PCC object corresponding to a real object or an object on the screen to passengers by outputting an XR / PCC image.
[0155] In this case, when the XR / PCC object is output to the HUD, at least a part of the XR / PCC object can be output so as to overlap the actual object pointed at by the passenger's gaze. On the other hand, when the XR / PCC object is output to a display provided inside the autonomous vehicle, at least a part of the XR / PCC object can be output to overlap the object on the screen. For example, an autonomous vehicle can output an XR / PCC object corresponding to objects such as lanes, other vehicles, traffic lights, traffic signs, two-wheeled vehicles, pedestrians, and buildings.
[0156] VR technology, AR technology, MR technology and / or PCC technology according to the embodiment can be applied to various devices. That is, VR technology is a display technology that provides objects or backgrounds in the real world only as CG images. On the other hand, AR technology refers to a technology that displays a virtual CG image on top of a real object image. Moreover, MR technology is similar to the above-mentioned AR technology in that the mixing and combination of virtual objects in the real world are shown. However, in AR technology, the distinction between real objects and virtual objects composed of CG images is clear, and virtual objects are used in the form of supplementing real objects, while in MR technology, virtual objects are considered to be equivalent to real objects, which is different from AR technology. More specifically, for example, a hologram service is applied to the above-mentioned MR technology. VR, AR and MR technologies can be integrated and referred to as XR technology.
[0157] Space segmentation
[0158] Point cloud data (i.e., G-PCC data) may represent a volumetric encoding of a point cloud consisting of a sequence of frames (point cloud frames). Each point cloud frame may include: the number of points, the locations of the points, and the attributes of the points. The number of points, the locations of the points, and the attributes of the points may change from one frame to another. Each point cloud frame may refer to a set of three-dimensional points specified by zero or more attributes and the Cartesian coordinates (x, y, z) of the three-dimensional points at a particular time instance. Here, the Cartesian coordinates (x, y, z) of the three-dimensional points may be positions or geometries.
[0159] In some embodiments, the present disclosure may also perform a spatial segmentation process of segmenting the point cloud data into one or more 3D blocks before encoding the point cloud data. A 3D block may refer to all or part of a 3D space occupied by the point cloud data. A 3D block may be one or more of a tile group, a tile, a slice, a coding unit (CU), a prediction unit (PU), or a transform unit (TU).
[0160] A patch corresponding to a 3D block may refer to all or part of a 3D space occupied by point cloud data. Moreover, a slice corresponding to a 3D block may refer to all or part of a 3D space occupied by point cloud data. A patch may be divided into one or more slices based on the number of points included in a patch. A patch may be a group of slices with bounding box information. The bounding box information of each patch may be specified in a patch list (or a tile parameter set, TPS). A patch may overlap with another patch in a bounding box. A slice may be a data unit that is independently encoded or a data unit that is independently decoded. That is, a slice may be a collection of points that may be independently encoded or decoded. In some embodiments, a slice may be a series of syntax elements representing part or all of an encoded point cloud frame. Each slice may include an index for identifying the patch to which the slice belongs.
[0161] The spatially divided 3D blocks can be processed independently or non-independently. For example, the spatially divided 3D blocks can be encoded or decoded independently or non-independently, and can be sent or received independently or non-independently. In addition, the spatially divided 3D blocks can be quantized or inversely quantized independently or non-independently, and can be transformed or inversely transformed independently or non-independently. In addition, the spatially divided 3D blocks can be rendered independently or non-independently. For example, encoding or decoding can be performed in units of slices or in units of tiles. In addition, quantization or inverse quantization can be performed differently for each tile or slice, and quantization or inverse quantization can be performed differently for each transformed or inversely transformed tile or slice.
[0162] In this way, when point cloud data is spatially segmented into one or more 3D blocks and the spatially segmented 3D blocks are processed independently or non-independently, the process of processing the 3D blocks is performed in real time and with low latency. In addition, random access and parallel encoding or parallel decoding in the three-dimensional space occupied by the point cloud data can be enabled, and errors accumulated during the encoding or decoding process can be prevented.
[0163] Figure 7 is a block diagram illustrating an example of a transmitting device 700 for performing a space division process according to an embodiment of the present disclosure. Figure 7 As illustrated, the transmitting device 700 may include: a space segmentation unit 705 for performing a space segmentation process, a signaling processing unit 710, a geometry encoder 715, an attribute encoder 720, and an encapsulation processing unit 725 and / or a transmitting processing unit 730.
[0164] The spatial segmentation unit 705 may perform a spatial segmentation process of segmenting the point cloud data into one or more 3D blocks based on the bounding box and / or the sub-bounding box. Through the spatial segmentation process, the point cloud data may be segmented into one or more tiles and / or one or more slices. In some embodiments, the point cloud data may be segmented into one or more tiles, and each segmented tile may be further segmented into one or more slices through the spatial segmentation process.
[0165] The signaling processing unit 710 can generate and / or process (e.g., entropy encoding) signaling information and output the signaling information in the form of a bit stream. Hereinafter, the bit stream output from the signaling processing unit (wherein the signaling information is encoded) is referred to as a "signaling bit stream". The signaling information may include information for or about spatial segmentation. That is, the signaling information may include information related to the spatial segmentation process performed by the spatial segmentation unit 705.
[0166] When point cloud data is segmented into one or more 3D blocks, information may be required for decoding some point cloud data corresponding to a specific patch or a specific slice in the point cloud data. In addition, in order to support spatial access (or partial access) to the point cloud data, information related to the 3D spatial region may be required. Here, spatial access may refer to extracting only the necessary part of the point cloud data from the entire point cloud data from the file. The signaling information may include: information for decoding some point cloud data, information related to the 3D spatial region for supporting spatial access, etc. For example, the signaling information may include: 3D bounding box information, 3D spatial region information, patch information, and / or patch list information.
[0167] The signaling information may be provided from the spatial segmentation unit 705, the geometry encoder 715, the attribute encoder 720, the transmission processing unit 725, and / or the encapsulation processing unit 730. In addition, the signaling processing unit 710 may provide the signaling information from Figure 8 The feedback information fed back by the receiving device 800 is provided to the space segmentation unit 705, the geometry encoder 715, the attribute encoder 720, the sending processing unit 725 and / or the encapsulation processing unit 730.
[0168] The signaling information may be stored and signaled in samples, sample entries, sample groups, track groups, or separate metadata tracks in a track. In some embodiments, the signaling information may be signaled in units of sequence parameter sets (SPS) for sequence-level signaling, geometry parameter sets (GPS) for signaling of geometry coding information, attribute parameter sets (APS) for signaling of attribute coding information, and tile parameter sets (TPS) (or tile lists) for signaling at the tile level. In addition, the signaling information may be signaled in units of coding units such as slices or tiles.
[0169] At this time, the position (position information) of the 3D block may be output to the geometry encoder 715 , and the attribute (attribute information) of the 3D block may be output to the attribute encoder 720 .
[0170] The geometry encoder 715 may construct an octree based on the position information, encode the constructed octree, and output a geometry bitstream. Furthermore, the geometry encoder 715 may reconstruct the octree and / or the approximate octree and output it to the attribute encoder 720. The reconstructed octree may be a reconstructed geometry. The geometry encoder 715 may perform the following steps: Figure 3 All or some operations performed by the coordinate system transformation unit 305, the geometric quantization unit 310, the octree analysis unit 315, the approximation unit 320, the geometric encoding unit 325 and / or the reconstruction unit 330.
[0171] The attribute encoder 720 may output an attribute bitstream by encoding the attribute based on the reconstructed geometry. The attribute encoder 720 may perform Figure 3 All or some operations performed by the attribute transform unit 340, the RAHT transform unit 345, the LOD generation unit 350, the lifting unit 355, the attribute quantization unit 360, the attribute encoding unit 365 and / or the color transform unit 335.
[0172] The encapsulation processing unit 725 may encapsulate one or more input bitstreams into files or segments. For example, the encapsulation processing unit 725 may encapsulate each of the geometry bitstream, the attribute bitstream, and the signaling bitstream, or multiplex and encapsulate the geometry bitstream, the attribute bitstream, and the signaling bitstream. In some embodiments, the encapsulation processing unit 725 may encapsulate a bitstream (G-PCC bitstream) consisting of a sequence of type-length-value (TLV) structures into a file. The TLV (or TLV encapsulation) structure constituting the G-PCC bitstream may include: geometry bitstream, attribute bitstream, signaling bitstream, etc. In some embodiments, the G-PCC bitstream may be generated by the encapsulation processing unit 725, or generated by the sending processing unit 930. In some embodiments, the encapsulation processing unit 725 may be executed by Figure 1 The encapsulation processing unit 13 performs all or some operations.
[0173] The transmission processing unit 730 can process the encapsulated bit stream or file / segment according to any transmission protocol. The transmission processing unit 730 can perform the Figure 1 All or some of the operations performed by the described sending unit 14 and the sending processing unit.
[0174] Figure 8 800 according to an embodiment of the present disclosure. The receiving device 800 may perform operations corresponding to the operations of the transmitting device 700 for performing space division. Figure 8 As illustrated, the receiving device 800 may include: a receiving processing unit 805, a decapsulation processing unit 810, a signaling processing unit 815, a geometry decoder 820, an attribute decoder 825, and / or a post-processing unit 830.
[0175] The receiving processing unit 805 may receive a file / segment in which a G-PCC bitstream is encapsulated, a G-PCC bitstream or a bitstream, and process it according to the transport protocol. The receiving processing unit 805 may perform the processing by referring to Figure 1 All or some of the operations performed by the receiving unit 21 and the receiving processing unit described.
[0176] The decapsulation processing unit 810 may obtain a G-PCC bitstream by performing the inverse process of the operation performed by the encapsulation processing unit 925. The decapsulation processing unit 810 may obtain a G-PCC bitstream by decapsulating a file / segment. For example, the decapsulation processing unit 810 may obtain a signaling bitstream and output the signaling bitstream to the signaling processing unit 815, obtain a geometry bitstream and output the geometry bitstream to the geometry decoder 820, and obtain an attribute bitstream and output the attribute bitstream to the attribute decoder 825. The decapsulation processing unit 810 may perform the operation performed by the decapsulation processing unit 925. Figure 1 All or some operations performed by the decapsulation processing unit 22.
[0177] The signaling processing unit 815 may parse and decode the signaling information by performing the inverse process of the operation performed by the signaling processing unit 710. The signaling processing unit 815 may parse and decode the signaling information from the signaling bitstream. The signaling processing unit 815 may provide the decoded signaling information to the geometry decoder 820, the attribute decoder 825, and / or the post-processing unit 830.
[0178] The geometry decoder 820 may reconstruct the geometry from the geometry bitstream by performing the inverse process of the operation performed by the geometry encoder 715. The geometry decoder 820 may reconstruct the geometry based on the signaling information (geometry-related parameters).
[0179] The attribute decoder 825 may reconstruct the attribute from the attribute bitstream by performing the inverse process of the operation performed by the attribute encoder 720. The attribute decoder 825 may reconstruct the attribute based on the signaling information (attribute-related parameters) and the reconstructed geometry.
[0180] The post-processing unit 830 may reconstruct the point cloud data based on the reconstructed geometry and the reconstructed attributes. The reconstruction of the point cloud data may be performed by a process of matching the reconstructed geometry with the reconstructed attributes. In some embodiments, when the reconstructed point cloud data is in units of tiles and / or slices, the post-processing unit 830 may reconstruct the bounding box of the point cloud data by performing an inverse process of the spatial segmentation process of the sending device 700 based on the signaling information. In some embodiments, when the bounding box is divided into a plurality of tiles and / or a plurality of slices by the spatial segmentation process, the post-processing unit 830 may reconstruct portions of the bounding box by combining some slices and / or some tiles based on the signaling information. Here, some slices and / or some tiles used to reconstruct the bounding box may be slices and / or some tiles associated with a 3D space region to which spatial access is desired.
[0181] TLV Structure
[0182] As described above, the G-PCC bitstream may refer to a bitstream of point cloud data consisting of a sequence of TLV structures. The TLV structure may be referred to as a "TLV encapsulation structure", a "G-PCC TLV encapsulation structure", or a "G-PCC TLV structure".
[0183] Fig. 9 shows an example of a TLV encapsulation structure, Fig.10a shows an example of the syntax structure of a TLV encapsulation, and Fig.10b An example of a payload type of a TLV encapsulation structure is shown. Each TLV encapsulation structure may be composed of a TLV type (TLVTYPE), a TLV length (TLV LENGTH), and / or a TLV payload (TLV PAYLOAD). The TLV type may be type information of the TLV payload, the TLV length may be length information of the TLV payload, and the TLV payload may be a payload (or payload bytes). Fig.10a In the TLV encapsulation syntax structure (tlv_encapsulation()) illustrated in the example, tlv_type can indicate the type information of the TLV payload, and tlv_num_payload_bytes can indicate the length information of the TLV payload. In addition, tlv_payload_byte[i] can indicate the TLV payload. tlv_payload_byte[i] can be signaled with the same value as the value of tlv_num_payload_bytes, and i can be increased by 1 from 0 to (tlv_num_payload_bytes-1).
[0184] The TLV payload may include: SPS, GPS, one or more APS, tile list, geometry slice, one or more attribute slices, and one or more metadata slices. According to an embodiment, according to the type information of the TLV payload, the TLV payload of each TLV encapsulation structure may include: one of SPS, GPS, one or more APS, tile list, geometry slice, one or more attribute slices, and one or more metadata slices. The data included in the TLV payload can be distinguished by the type information of the TLV payload. For example, as in Fig.10bAs illustrated in , if the value of tlv_type is 0, it may indicate that the data included in the TLV payload is SPS, and if the value of tlv_type is 1, it may indicate that the data included in the TLV payload is GPS. If the value of tlv_type is 2, it may indicate that the data included in the TLV payload is a geometry slice, and if the value of tlv_type is 3, it may indicate that the data included in the TLV payload is an APS. If the value of tlv_type is 4, it may indicate that the data included in the TLV payload is an attribute slice, and if the value of tlv_type is 5, it may indicate that the data included in the TLV payload is a tile list (or tile parameter set). If the value of tlv_type is 6, it may indicate that the data included in the TLV payload is a frame boundary marker, and if the value of tlv_type is 7, it may indicate that the data included in the TLV payload is a metadata slice. The payload of the TLV encapsulation structure can observe the format of the High Efficiency Video Coding (HEVC) Network Abstraction Layer (NAL) unit.
[0185] Encapsulation / Decapsulation
[0186] The G-PCC bitstream composed of the TLV encapsulation structure may be sent to the receiving device without change, or may be encapsulated and sent to the receiving device. For example, the encapsulation processing unit 725 may encapsulate the G-PCC bitstream composed of the TLV encapsulation structure in the form of a file / segment and send it. The decapsulation processing unit 810 may obtain the G-PCC bitstream by decapsulating the encapsulated file / segment.
[0187] In some embodiments, the G-PCC bitstream may be encapsulated in a file format based on ISOBMFF. In this case, the G-PCC bitstream may be stored in a single track or multiple tracks in an ISOBMFF file. Here, the single track or multiple tracks in the file may be referred to as a "track" or a "G-PCC track". A file based on ISOBMFF may be referred to as a container, a container file, a media file, a G-PCC file, etc. Specifically, the file may consist of a box and / or information, which may be referred to as ftyp, moov, mdat, etc.
[0188] The ftyp box (ftyp box, file type box) can provide file type or file compatibility related information for a file. The receiving device can identify the file by referring to the ftyp box. The mdat box is also called the media data box and can include actual media data. In some embodiments, a geometric slice (or an encoded geometric bitstream) and zero or more attribute slices (or an encoded attribute bitstream) can be included in the sample of the mdat box in the file. Here, the sample can be referred to as a G-PCC sample. The moov box is also called a movie box and can include metadata of the media data of the file. For example, the moov box can include information required for decoding and playback of media data, and can include information about the track and sample of the file. The moov box can act as a container for all metadata. The moov box can be the topmost box among the metadata-related boxes.
[0189] In some embodiments, the moov box may include a track (trak) box that provides information related to the track of the file, and the trak box may include a media (mdia) box (MediaBox) that provides media information of the track, and a track reference container (tref) box for linking (referencing) the track and the sample of the file corresponding to the track. The media box MediaBox may include a media information container (minf) box that provides information about the media data, and a handler (hdlr) box that indicates the stream type. The minf box may include a sample table (stbl) box that provides metadata related to the sample of the mdat box. The stbl box may include a sample description (stsd) box that provides information about the encoding type used and initialization information required for the encoding type. In some embodiments, the sample description (stsd) box may include a sample entry of the track. In some embodiments, signaling information (or metadata) such as SPS, GPS, APS, and patch list may be included in the sample entry of the moov box or the sample of the mdat box in the file.
[0190] A G-PCC track may be defined as a volume visual track that carries a geometry slice (or a coded geometry bitstream) or an attribute slice (or a coded attribute bitstream), or both a geometry slice and an attribute slice. In some embodiments, a volume visual track may be identified by a volume visual media handler type "volv" in a handler box HandlerBox of a media box (MediaBox), and / or a volume visual media header vvhd in a minf box of a media box MediaBox. A minf box may be referred to as a media information container or a media information box. A minf box may be included in a media box MediaBox, a media box MediaBox may be included in a track box, and a track box may be included in a moov box of a file. A single volume visual track or multiple volume visual tracks may be present in a file.
[0191] Sample Group
[0192] The encapsulation processing unit mentioned in the present disclosure may generate a sample group by grouping one or more samples. The encapsulation processing unit, metadata processing unit, or signaling processing unit mentioned in the present disclosure may signal signaling information associated with the sample group in the sample, sample group, or sample entry. That is, the sample group information associated with the sample group may be added to the sample, sample group, or sample entry. The sample group information may be 3D bounding box sample group information, 3D region sample group information, 3D patch sample group information, 3D patch list sample group information, etc.
[0193] Track Group
[0194] The encapsulation processing unit mentioned in the present disclosure may generate a track group by grouping one or more tracks. The encapsulation processing unit, metadata processing unit, or signaling processing unit mentioned in the present disclosure may signal signaling information associated with the track group in a sample, a track group, or a sample entry. That is, track group information associated with the track group may be added to the sample, the track group, or the sample entry. The track group information may be 3D bounding box track group information, point cloud composition track group information, spatial region track group information, 3D tile track group information, 3D tile list track group information, etc.
[0195] Sample entry
[0196] Fig.11 is a diagram for explaining an ISOBMFF-based file including a single track. Fig.11 (a) illustrates an example of a layout of an ISOBMFF-based file including a single track, and Fig.11(b) illustrates an example of a sample structure of the mdat box when the G-PCC bitstream is stored in a single track of a file. Fig.12 is a diagram for explaining an ISOBMFF-based file including a plurality of tracks. Fig.12 (a) illustrates an example of a layout of an ISOBMFF-based file including a plurality of tracks, and Fig.12 (b) illustrates an example of a sample structure of the mdat box when the G-PCC bitstream is stored in a single track of a file.
[0197] The stsd box (SampleDescriptionBox) included in the moov box of the file can include sample entries for a single track storing a G-PCC bitstream. SPS, GPS, APS, and tile lists can be included in the sample entries in the moov box in the file or in the samples in the mdat box. Moreover, geometric slices and zero or more attribute slices can be included in the samples of the mdat box in the file. When the G-PCC bitstream is stored in a single track of the file, each sample can contain multiple G-PCC components. That is, each sample can be composed of one or more TLV encapsulation structures. The sample entries of a single track can be defined as follows.
[0198] Sample entry types: "gpe1", "gpeg"
[0199] Container: SampleDescriptionBox
[0200] Mandatory: "gpe1" or "gpeg" sample entries are mandatory Quantity: One or more sample entries may be present
[0201] The sample entry type "gpe1" or "gpeg" is required, and one or more sample entries may exist. The G-PCC track may use a VolumetricVisualSampleEntry with a sample entry type "gpe1" or "gpeg". The sample entry of the G-PCC track may include a G-PCC decoding configuration box (GPCCConfigurationBox), and the G-PCC decoding configuration box may include a G-PCC decoder configuration record (GPCCDecoderConfigurationRecord()). GPCCDecoderConfigurationRecord() may include at least one of the following items: configurationVersion, profile_idc, profile_compatibility_flags, level_idc, numOfSetupUnitArrays, SetupUnitType, completeness, numOfSepupUnit, and setupUnit. The setupUnit array field included in GPCCDecoderConfigurationRecord() may include a TLV encapsulation structure containing one SPS.
[0202] When the sample entry type is "gpel", all parameter sets (such as SPS, GPS, APS, and tile list) can be included in the array of setupUints. When the sample entry type is "gpeg", the above parameter sets can be included in the array of setupUints (i.e., sample entry) or included in the corresponding stream (i.e., sample). An example of the syntax of a G-PCC sample entry (GPCCSampleEntry) with a sample entry type "gpe1" can be described as follows.
[0203] aligned(8)class GPCCSampleEntry()
[0204] extends VolumetricVisualSampleEntry('gpe1'){
[0205] GPCCConfigurationBox config; / / Mandatory 3DBoundingBoxInfoBox();
[0206] CubicRegionInfoBox();
[0207] TileInventoryBox();
[0208] }
[0209] A G-PCC sample entry (GPCCSampleEntry) with a sample entry type of "gpe1" may include: GPCCConfigurationBox, 3DBoundingBoxInfoBox(), CubicRegionInfoBox(), and TileInventoryBox(). 3DBoundingBoxInfoBox() may indicate 3D bounding box information of point cloud data associated with a sample carried by the corresponding track. CubicRegionInfoBox() may include one or more pieces of spatial region information of point cloud data of a sample carried into the corresponding track. TileInventoryBox() may indicate 3D tile inventory information of point cloud data of a sample that has been carried into the corresponding track.
[0210] like Fig.12 As illustrated in (b), the sample may include a TLV encapsulation structure including a geometric slice. In addition, the sample may include a TLV encapsulation structure including one or more parameter sets. In addition, the sample may include a TLV encapsulation structure including one or more attribute slices.
[0211] like Fig.12 As illustrated in (a), when the G-PCC bitstream is carried by multiple tracks of an ISOBMFF-based file, each geometry slice or attribute slice may be mapped to a separate track. For example, the geometry slice may be mapped to track 1, and the attribute slice may be mapped to track 2. The track carrying the geometry slice (track 1) may be referred to as a geometry track or a G-PCC geometry track, and the track carrying the attribute slice (track 2) may be referred to as an attribute track or a G-PCC attribute track. In addition, the geometry track may be defined as a volume visual track carrying the geometry slice, and the attribute track may be defined as a volume visual track carrying the attribute slice.
[0212] A track that carries a portion of a G-PCC bitstream that includes both geometry slices and attribute slices may be referred to as a multiplexed track. In the case where geometry slices and attribute slices are stored on separate tracks, each sample in the track may include at least one TLV encapsulation structure that carries data for a single G-PCC component. In this case, each sample contains neither geometry nor attributes, and may not contain multiple attributes. Multi-track encapsulation of a G-PCC bitstream may enable a G-PCC player to effectively access one of the G-PCC components. When a G-PCC bitstream is carried by multiple tracks, the following conditions need to be met to effectively access one of the G-PCC components.
[0213] a) When the G-PCC bitstream consisting of a TLV encapsulation structure is carried by multiple tracks, the track carrying the geometry bitstream (or geometry slice) becomes the entry point.
[0214] b) In the sample entry, a new box is added to indicate the role of the stream included in the corresponding track. The new box may be the above-mentioned G-PCC component type box (GPCCComponentTypeBox). That is, the GPCCComponentTypeBox may be included in the sample entries of multiple tracks.
[0215] c) Introduce a track reference from the track carrying the G-PCC geometry bitstream alone to the track carrying the G-PCC attribute bitstream.
[0216] GPCCComponentTypeBox may include GPCCComponentTypeStruct(). When GPCCComponentTypeBox is present in a sample entry of a track carrying part or all of a G-PCC bitstream, GPCCComponentTypeStruct() may indicate one or more G-PCC component types (e.g., geometry, attributes) carried by each track. For example, when the value of the gpcc_type field included in GPCCComponentTypeStruct() is 2, it may indicate a geometry component, and when the value is 4, it may indicate an attribute component. In addition, when the value of the gpcc_type field indicates 4 (i.e., attribute component), it may also include an AttrIdx field indicating an identifier of an attribute signaled in SPS().
[0217] When the G-PCC bitstream is carried by multiple tracks, the syntax of the sample entry may be defined as follows.
[0218] Sample entry types: 'gpe1', 'gpeg', 'gpc1', or 'gpcg'
[0219] Container: SampleDescriptionBox
[0220] Mandatory: "gpc1" or "gpcg" sample entries are mandatory Quantity: One or more sample entries may be present
[0221] The sample entry type "gpc1", "gpcg", "gpc1", or "gpcg" is required, and one or more sample entries may be present. Multiple tracks (e.g., geometry or attribute tracks) may use a VolumetricVisualSampleEntry with the sample entry type "gpc1", "gpcg", "gpc1", or "gpcg". In the "gpel" sample entry, each parameter set may be present in the setupUnit array. In the "gpeg" sample entry, the parameter set may be present in the corresponding array or stream. In the "gpe1" or "gpeg" sample entry, the GPCCComponentTypeBox may not be present. In the "gpcl" sample entry, the SPS, GPS, and tile list may be present in the SetupUnit array of the track carrying the G-PCC geometry bitstream. Each associated APS may be present in the SetupUnit array of the track carrying the G-PCC attribute bitstream. In the "gpcg" sample entry, the SPS, GPS, APS, or tile list may be present in the corresponding array or stream. In the "gpc1" or "gpcg" sample arrays, GPCCComponentTypeBox may not be present.
[0222] An example of the syntax of a G-PCC sample entry is as follows.
[0223] aligned(8)class GPCCSampleEntry()
[0224] extends VolumetricVisualSampleEntry(codingname){
[0225] GPCCConfigurationBox config; / / Mandatory
[0226] GPCCComponentTypeBox type; / / optional}
[0227] The compressorname (i.e., codingname) of the base class VolumetricVisualSampleEntry can indicate the name of the compressor used with the recommended "W013GPCC encoding". In the W013GPCC encoding, the first byte (octal 13 or decimal 11 marked by W013) is the number of remaining bytes and can indicate the number of bytes of the remaining string. congif can include G-PCC decoder configuration information. info can indicate the G-PCC component information carried in each track. info can indicate the component patches carried in the track, and can also indicate the attribute name, index, and attribute type of the G-PCC component carried in the G-PCC attribute track.
[0228] References between tracks
[0229] When G-PCC bitstreams are carried in multiple tracks (i.e., when G-PCC geometry bitstream and attribute bitstream are carried in different (separate) tracks from each other), the track reference tool can be used to connect the tracks. A TrackReferenceTypeBoxes can be added to the TrackReferenceBox in the TrackBox of the G-PCC track. The TrackReferenceTypeBox can include an array of track_IDs that specify the tracks referenced by the G-PCC track.
[0230] According to an embodiment, the present disclosure may provide an apparatus and method for supporting the temporal scalability of the transport of G-PCC data (hereinafter, this may be referred to as a G-PCC bitstream, an encapsulated G-PCC bitstream, or a G-PCC file). In addition, the present disclosure may propose an apparatus and method for providing a point cloud content service, which efficiently stores a G-PCC bitstream in a single track in a file, or separates and stores the G-PCC bitstream in multiple tracks and provides signaling for them. In addition, the present disclosure proposes an apparatus and method for processing file storage technology to support efficient access to stored G-PCC bitstreams.
[0231] Time Scalability
[0232] Temporal scalability may refer to a function that allows the possibility of extracting one or more subsets of independently encoded frames. In addition, temporal scalability may refer to a function that divides G-PCC data into multiple different temporal levels and independently processes each G-PCC frame belonging to a different temporal level. If temporal scalability is supported, a G-PCC player (or a transmitting device and / or receiving device of the present disclosure) can effectively access a desired component (target component) among the G-PCC components. In addition, if temporal scalability is supported, since the G-PCC frames are processed independently of each other, the temporal scalability support at the system level can be expressed as a more flexible temporal sub-layering. In addition, if temporal scalability is supported, a system (point cloud content providing system) that processes G-PCC data can manipulate the data at a high level to match the network capability or decoder capability, thereby improving the performance of the point cloud content providing system.
[0233] Sample Grouping
[0234] For methods for supporting temporal scalability, there may be a sample grouping method and a track grouping method. The sample grouping method may be a method for grouping samples within a G-PCC file according to a temporal level, and the track grouping method may be a method for grouping tracks within a G-PCC file according to a temporal level.
[0235] The sample group may be used to connect a sample with a time level assigned to the sample. That is, the sample group may indicate which sample belongs to which time level. In addition, the sample group may be information about the result of grouping one or more samples into one or more time levels. The sample group may be referred to as a "tele" sample group and a time level sample group "tele".
[0236] Information about the sample group
[0237] The information about the sample group may include information about the result of the sample grouping. Therefore, the information about the sample group may be information to be used to connect the sample with the time level assigned to the sample. That is, the information about the sample group may indicate which sample belongs to which time level, and may be information about the result of grouping one or more samples into one or more time levels.
[0238] Information about sample groups may be present in tracks that include geometry data units. When G-PCC data is carried on multiple tracks, information about sample groups may be present only in the geometry track to facilitate grouping of samples in the track by temporal level. Samples in the attribute track may be inferred based on their relationship to the geometry track associated with the sample. For example, a sample in the attribute track may belong to the same temporal level as a sample in the geometry track associated with it.
[0239] When information about sample groups exists in a G-PCC patch track referenced by a G-PCC patch base track, information about sample groups may have to exist in the remaining patch tracks referenced by the G-PCC patch base track. Here, the G-PCC patch track may be a volumetric visual track that carries each G-PCC component or a single G-PCC component corresponding to one or more G-PCC patches. In addition, the G-PCC patch base track may be a volumetric visual track that carries each parameter set and patch manifest corresponding to the G-PCC patch track.
[0240] Information about time levels
[0241] In order to describe the temporal scalability supported by the G-PCC file, information about the temporal level may be signaled. Information about the temporal level may be present in the sample entry of the track including the sample group (or information about the sample group). For example, information about the temporal level may be present in GPCCDecoderConfigurationRecord(), or in a G-PCC scalability information box (GPCCScalabilityInfoBox) that signals scalability information for a G-PCC track.
[0242] Time Scalability Track Grouping
[0243] The temporal level tracks carrying the geometry component of the G-PCC bitstream may be grouped into G-PCC temporal scalability track groups.
[0244] When multiple temporal level tracks are used to encapsulate a G-PCC bitstream and there are one or more alternative tracks for these temporal level tracks, all temporal level tracks including geometric components of the same G-PCC bitstream may be grouped into the same G-PCC temporal scalability track group. In this document, temporal scalability track grouping may prevent a file parser from mixing temporal tracks in alternative tracks when combining two or more temporal level tracks.
[0245] Problems with existing technology
[0246] The GPCC temporal scalability group box (GPCCTemporalScalabilityGroupBox) may be a box including information for grouping temporal level tracks. The syntax structure of the GPCC temporal scalability group box may be as follows.
[0247] aligned(8)class GPCCTemporalScalabilityGroupBox
[0248] extends TrackGroupTypeBox('gtsg'){
[0249] / / track_group_id inherits from TrackGroupTypeBox;
[0250] }
[0251] In the above syntax structure, track_group_id may be an identifier indicating a group to which the track belongs.
[0252] The GPCC Temporal Scalability Group Box (GPCCTemporalScalabilityGroupBox) can be present in a track with sample type "gpcl" or "gpcg". However, it is unclear whether the GPCC Temporal Scalability Group Box can be present in a track with sample type "gpel" or "gpeg".
[0253] The GPCC Temporal Scalability Group Box can exist in a temporal level track including a geometry component, but it is unclear whether the GPCC Temporal Scalability Group Box can exist in a temporal level track including an attribute component. In addition, when the GPCC Temporal Scalability Group Box exists in a temporal level track including an attribute component, it is still unclear how to handle such information.
[0254] Implementation
[0255] In order to solve these problems, the present disclosure proposes that 1) the GPCC temporal scalability group box may exist in a track having a sample entry type of one of "gpel", "gpeg", "gpcl" or "gpcg". In addition, the present disclosure proposes that 2) when the sample entry type of a track is one of "gpcl" or "gpcg" and the track is a geometric track including a geometric component, the GPCC temporal scalability group box may exist in the track. In addition, the present disclosure proposes that when the sample entry type of a track is one of "gpcl" or "gpcg" and the track is an attribute track including an attribute component, the GPCC temporal scalability group box may not exist in the track. In addition, the present disclosure proposes that 3) attribute tracks may be combined based on a combination of geometric tracks included in the same temporal scalability group box.
[0256] Hereinafter, Proposal 1) to Proposal 3) are described through different embodiments.
[0257] Implementation Method 1
[0258] Embodiment 1 is an embodiment related to a sample entry type of a track that may include a temporal scalability group box. Fig.13 is a flowchart of a method performed in receiving device 20 according to embodiment 1, and Fig.14 It is a flowchart of the method executed in the transmitting device 10 according to the first embodiment.
[0259] Reference Fig.13 , the receiving device 20 may obtain a G-PCC file (S1310). The G-PCC file may include: point cloud data, track time level number information, time scalability track group information, etc.
[0260] The receiving device 20 may obtain a G-PCC temporal scalability group box from a track included in the G-PCC file (S1320). Herein, the sample entry type of the track including the G-PCC temporal scalability group box may be any one of "gpel", "gpeg", "gpcl", and "gpcg". That is, the temporal scalability group box may be included not only in a track having a sample entry type of "gpcl" or "gpcg", but also in a track having a sample entry type of "gpel" or "gpeg".
[0261] Reference Fig.14, the transmitting device 10 may generate a track including a G-PCC temporal scalability group box (GPCCTemporalScalabilityGroupBox) (S1410). Herein, the sample entry type of the track may be any one of "gpel", "gpeg", "gpcl" or "gpcg". That is, the temporal scalability group box may be included not only in a track having a sample entry type of "gpcl" or "gpcg", but also in a track having a sample entry type of "gpel" or "gpeg".
[0262] Next, the transmitting device 10 may generate a G-PCC file including the generated track (S1420).
[0263] Therefore, according to an embodiment of the present disclosure, the GPCC Time Scalability Group Box can be included not only in a track having a sample entry type of "gpcl" or "gpcg", but also in a track having a sample entry type of "gpel" or "gpeg". Therefore, according to an embodiment of the present disclosure, the problem of the prior art, that is, the ambiguity as to whether the GPCC Time Scalability Group Box can exist in a track when the sample entry type is "gpel" or "gpeg", can be solved.
[0264] Implementation Method 2
[0265] Embodiment 2 is an embodiment related to a sample entry type of a track that can include a temporal scalability group box and the type of the track. Fig.15 and Fig.16 is a flowchart of a method performed in the transmitting device 10 according to Embodiment 2, and Fig.17 It is a flowchart of the method executed in the receiving device 20 according to the second embodiment.
[0266] Reference Fig.15 , the sending device 10 may check whether the sample entry type of the track is one of "gpcl" or "gpcg" (S1510). If the sample entry type of the track is neither "gpcl" nor "gpcg" (No at step S1510), the sending device 10 may generate a track in which the GPCC time scalability group box is not included. Alternatively, if the sample entry type of the track is neither "gpcl" nor "gpcg" (No at step S1510), the process may be terminated. On the other hand, if the sample entry type of the track is one of "gpcl" and "gpcg" (Yes at step S1510), the sending device 10 may check whether the track includes a geometric component (S1520).
[0267] If the track does not include a geometric component (No at step S1520), the transmitting device 10 may generate a track in which the GPCC time scalability group box is not included. Alternatively, if the track does not include a geometric component (No at step S1520), the process may be terminated. On the other hand, if the track includes a geometric component (Yes at step S1520), the transmitting device 10 may generate a track including a G-PCC time scalability group box (S1530).
[0268] According to another embodiment of the present disclosure, the order of steps S1510 and S1520 may be changed. That is, the transmitting device 10 may first check whether a geometric component is included, and then identify the sample entry type. That is, if the sample entry type of the sample is one of "gpcl" or "gpcg" and includes a geometric component, a track including a GPCC time scalability box may be generated.
[0269] Reference Fig.16 , the sending device 10 may check whether the sample entry type of the track is one of "gpcl" or "gpcg" (S1610). If the sample entry type of the track is neither "gpcl" nor "gpcg" (No at step S1610), the sending device 10 may generate a track in which the GPCC time scalability group box is not included. Alternatively, if the sample entry type of the track is neither "gpcl" nor "gpcg" (No at step S1610), the process may terminate. On the other hand, if the sample entry type of the track is one of "gpcl" and "gpcg" (Yes at step S1610), the sending device 10 may check whether the track includes an attribute component (S1620).
[0270] If the track includes an attribute component (Yes at step S1620), the transmitting device 10 may generate a track in which the GPCC time scalability group box is not included. Alternatively, if the track includes an attribute component (Yes at step S1620), the process may be terminated. On the other hand, if the track does not include an attribute component (No at step S1620), the transmitting device 10 may generate a track including a G-PCC time scalability group box (S1630).
[0271] According to another embodiment of the present disclosure, the order of steps S1610 and S1620 may be changed. That is, the transmitting device 10 may first check whether the attribute component is included, and then identify the sample entry type, but the present disclosure is not limited thereto. That is, if the sample entry type of the sample is neither "gpcl" nor "gpcg" or the track includes the attribute component, the track including the GPCC time scalability box may not be generated.
[0272] Reference Fig.17 , the receiving device 20 may obtain a GPCC file (S1710). Here, the GPCC file may include: point cloud data, track time level number information, time scalability track group information, etc.
[0273] The receiving device 20 may check whether the track of the GPCC file includes a GPCC time scalability group box (S1720). If the track includes a GPCC time scalability group box (Yes at step S1720), the receiving device 20 may obtain a geometry component included in the track (S1730). On the other hand, if the track does not include a GPCC time scalability group box (No at step S1720), the receiving device 20 may obtain an attribute component included in the track (S1740).
[0274] Therefore, according to an embodiment of the present disclosure, whether the GPCC time scalability group box is included in a track may be different depending on the sample entry type of the track and whether the track includes a geometry component or an attribute component. Therefore, according to an embodiment of the present disclosure, the problem of the prior art, i.e., the ambiguity about whether the GPCC time scalability group box can be included in the track when the track includes an attribute component, can be solved.
[0275] Implementation 3
[0276] Embodiment 3 is an embodiment related to a method of combining tracks including attribute components based on a combination of tracks including geometric components. The "combination" mentioned in the present disclosure may refer to "grouping" and may be interpreted as "track grouping" according to the embodiment. Fig.18 It is a flowchart of the method executed in the transmitting device 10 according to the third embodiment.
[0277] Reference Fig.18 , the transmitting device 10 may check whether the first track is included in the same time scalability track group (S1820). In this article, the first track may be a track including a geometric component. That is, the first track may be a geometric track. The geometric track may refer to a track including a geometric component.
[0278] If the first track is not included in the same temporal scalability track frame (No at step S1820), the transmitting device 10 may terminate the process. On the other hand, if the first track is included in the same temporal scalability track frame (Yes at step S1820), the transmitting device 10 may combine the first track (S1830).
[0279] The transmitting device 10 may check whether the first track refers to the second track (S1840). Herein, the second track may be a track including an attribute component. That is, the second track may be an attribute track. The attribute track may refer to a track including an attribute component.
[0280] If the first track does not refer to the second track (No at step S1840), the transmitting device 10 may terminate the process. On the other hand, if the first track refers to the second track (Yes at step S1840), the transmitting device 10 may combine the second track (S1850). In this case, the combining method of the second track may be the same as the combining method of the first track. That is, according to the present disclosure, when combining the first track included in the same temporal scalability track group, the second track referenced by the first track may be combined.
[0281] According to the present disclosure, before performing steps S1820 to 1850, the transmitting device 10 may check whether the sample entry type of the first track is "gpcl" or "gpcg" (S1810). If the sample entry type of the first track is neither "gpcl" nor "gpcg" (No at step S1810), the transmitting device 10 may terminate the process. On the other hand, if the sample entry type of the first track is one of "gpcl" or "gpcg" (Yes at step S1810), the transmitting device 10 may check whether the first track is included in the same temporal scalability track group (S1820). Next, as described above, steps S1820 to S1850 may be performed.
[0282] That is, according to the present disclosure, when a sample entry type of a first track is either "gpcl" or "gpcg" and the first track is included in the same temporal scalability track group, a second track referenced by the first track may be combined based on the first track being combined.
[0283] According to one embodiment of the present disclosure, the receiving device 20 may identify a track group identifier (track_group_id). Herein, the track group identifier may be present in a GPCC file obtained from the transmitting device 10. Next, based on the track group identifier, the receiving device 20 may determine a combination of the first track. The receiving device 20 may determine a combination of the second track referenced by the first track according to the combination of the first track.
[0284] The scope of the present disclosure includes software or machine-executable commands (e.g., operating systems, applications, firmware, programs, etc.) for enabling operations of methods according to various embodiments to be performed on a device or computer, and non-transitory computer-readable media having such software or commands stored thereon and executable on a device or computer.
[0285] Industrial Applicability
[0286] The embodiments of the present disclosure may be used to provide point cloud content. In addition, the embodiments of the present disclosure may be used to encode / decode point cloud data.
Claims
1. A method performed in a receiving device for point cloud data, the method comprising the following steps: Obtaining a geometry-based point cloud compression G-PCC file including the point cloud data; as well as Get the G-PCC temporal scalability group box GPCCTemporalScalabilityGroupBox from the track in the G-PCC file, Wherein, the sample entry type of the track is one of "gpel", "gpeg", "gpcl" or "gpcg".
2. The method according to claim 1, wherein: The trajectory includes a geometric component.
3. The method according to claim 2, wherein: The sample entry type for the track is one of "gpcl" or "gpcg".
4. The method according to claim 1, wherein: Based on the track including an attribute component and the sample entry type of the track is “gpcl” or “gpcg”, the G-PCC temporal scalability group box is not obtained from the track.
5. The method according to claim 1, wherein: Based on the first track included in the same time scalability group box is combined, combining a second track referenced by the first track, the first track being a track including a geometric component, and The second track is a track including attribute components.
6. The method according to claim 5, wherein: The first track is combined based on the sample entry type of the track being one of "gpcl" or "gpcg".
7. A method executed in a point cloud data sending device, the method comprising the following steps: Generate a track including a G-PCC temporal scalability group box GPCCTemporalScalabilityGroupBox; as well as Generate a G-PCC file including the track, Wherein, the sample entry type of the track is one of "gpel", "gpeg", "gpcl" or "gpcg".
8. A device for receiving point cloud data, the device comprising: Memory; as well as at least one processor, Wherein, the at least one processor is configured to: Obtaining a geometry-based point cloud compression G-PCC file including the point cloud data; and obtain a G-PCC temporal scalability group box GPCCTemporalScalabilityGroupBox from a track in the G-PCC file, and Wherein, the sample entry type of the track is one of "gpel", "gpeg", "gpcl" or "gpcg".
9. A device for sending point cloud data, the device comprising: Memory; as well as at least one processor, Wherein, the at least one processor is configured to: generating a track including a G-PCC temporal scalability group box GPCCTemporalScalabilityGroupBox; and generating a G-PCC file including the track, and Wherein, the sample entry type of the track is one of "gpel", "gpeg", "gpcl" or "gpcg".
