Point cloud data transmission device and method executed in same transmission device, and point cloud data reception device and method executed in same reception device
Through geometry-based point cloud compression technology and the use of time scalability information, the problem of low efficiency and high complexity of point cloud data processing and storage is solved, and efficient point cloud services and performance improvements are achieved.
Patent Information
- Application Number
- CN202380033886.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-04-14
- Filing Date
- 2023-04-14
- Publication Date
- 2024-11-22
AI Technical Summary
The prior art is difficult to efficiently process and store large amounts of point cloud data, resulting in high waiting time and encoding/decoding complexity, and lack of time scalability support, affecting the performance of point cloud content services.
Through geometry-based point cloud compression (G-PCC) technology, time scalability information is used to reconstruct point cloud data, and G-PCC bitstreams are efficiently stored or decomposed in files, supporting time scalability and improving Coding efficiency and access performance.
It realizes efficient processing of point cloud data, provides high-quality point cloud services, improves the performance of point cloud content providing system, supports time scalability, and makes data access more flexible and efficient.
Smart Images

Figure CN119013988A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to methods and apparatus for processing point cloud content. Background Art
[0002] Point cloud content is expressed as a point cloud that is a set of points belonging to a coordinate system representing a three-dimensional space. Point cloud content can represent three-dimensional media and is used to provide various services such as virtual reality (VR), augmented reality (AR), mixed reality (MR), and self-driving services. Since tens of thousands to hundreds of thousands of point data are required to express point cloud content, a method for efficiently processing a large amount of point data is required. Summary of the invention
[0003] Technical issues
[0004] The present disclosure provides a device and method for efficiently processing point cloud data. The present disclosure provides a method and device for processing point cloud data for solving waiting time and encoding / decoding complexity.
[0005] In addition, the present disclosure provides an apparatus and method for supporting temporal scalability in the carriage of geometry-based point cloud compression data.
[0006] In addition, the present disclosure proposes a device and method for efficiently storing a G-PCC bit stream in a single track in a file, or storing the G-PCC bit stream separately in multiple tracks and providing a point cloud content service, which provides signaling of the G-PCC bit stream.
[0007] In addition, the present disclosure proposes an apparatus and method for processing file storage technology to support efficient access to stored G-PCC bitstreams.
[0008] In addition, the present disclosure proposes an apparatus and method for specifying a track capable of carrying temporal scalability information when temporal scalability is supported.
[0009] The technical problems solved by the present disclosure are not limited to the above-mentioned technical problems, and other technical problems not described herein will be apparent to those skilled in the art through the following description.
[0010] Technical Solution
[0011] According to an embodiment of the present disclosure, a method performed by a receiving device for point cloud data includes: obtaining a geometry-based point cloud compression (G-PCC) file including point cloud data, and reconstructing the point cloud based on temporal scalability information, wherein the temporal scalability information may include first information about a temporal level track of the G-PCC file, and the first information may specify constraints on the temporal level track based on a sample entry type.
[0012] Meanwhile, based on the sample entry type being the first type, the first value of the first information may indicate that no other time level tracks exist.
[0013] Meanwhile, based on the sample entry type being the first type, the second value of the first information may indicate that another time level track may exist.
[0014] Meanwhile, based on the fact that each sample belongs to a single time level, the second value of the first information may also indicate the presence of one or more tracks of another time level that are removed.
[0015] Meanwhile, based on the sample entry type being the second type, the first value of the first information may indicate that each tile track referenced by the tile base track includes samples at each temporal level.
[0016] Meanwhile, based on the sample entry type being the second type, the second value of the first information may indicate that there are one or more time tile tracks that do not include the sample of each time level.
[0017] Meanwhile, based on the fact that each sample belongs to a single temporal level, the second value of the first information may also indicate that there is another tile track whose temporal level is greater than 0 and is removed.
[0018] Meanwhile, the sample entry type may include a first type including 'gpe1', 'gpeg', 'gpc1', and 'gpcg', and a second type including 'gpcb' and 'gpeb'.
[0019] According to an embodiment of the present disclosure, a method performed by a point cloud data sending device includes: determining whether temporal scalability is applied to point cloud data in a three-dimensional space, and generating a G-PCC file by including temporal scalability information and point cloud data, wherein the temporal scalability information may include first information about a temporal level track of the file, and the first information may specify constraints on the temporal level track based on a sample entry type.
[0020] According to an embodiment of the present disclosure, a receiving device for point cloud data may include a memory and at least one processor, and the at least one processor acquires (i.e., obtains) a geometry-based point cloud compression (G-PCC) file including the point cloud data, and reconstructs the point cloud based on temporal scalability information, wherein the temporal scalability information may include first information about a temporal level track of the G-PCC file, and the first information may specify constraints on the temporal level track based on a sample entry type.
[0021] According to an embodiment of the present disclosure, a device for sending point cloud data may include a memory and at least one processor, and the at least one processor determines whether temporal scalability is applied to point cloud data in a three-dimensional space, and generates a G-PCC file by including temporal scalability information and point cloud data, wherein the temporal scalability information may include first information about a temporal level track of the file, and the first information may specify constraints on the temporal level track based on a sample entry type.
[0022] According to an embodiment of the present disclosure, a computer-readable medium for storing a G-PCC bitstream or file is disclosed. The G-PCC bitstream or file can be generated by a method executed by a transmitting device of point cloud data.
[0023] According to an embodiment of the present disclosure, a method for sending a G-PCC bitstream or file is disclosed. The G-PCC bitstream or file can be generated by a method executed by a point cloud data sending device.
[0024] Beneficial Effects
[0025] The apparatus and method according to the embodiments of the present disclosure can process point cloud data with high efficiency.
[0026] The apparatus and method according to the embodiments of the present disclosure can provide high-quality point cloud services.
[0027] The device and method according to the embodiments of the present disclosure may provide point cloud content for providing general services such as VR services and autonomous driving services.
[0028] The apparatus and method according to an embodiment of the present disclosure may provide time scalability for efficiently accessing a desired component among G-PCC components.
[0029] The apparatus and method according to an embodiment of the present disclosure may improve encoding efficiency by distinguishing information meant by temporal scalability information based on a sample entry type.
[0030] The apparatus and method according to the embodiments of the present disclosure may support temporal scalability, so that data may be manipulated at a high level consistent with a network function or a decoder function, and thereby the performance of a point cloud content providing system may be improved.
[0031] The apparatus and method according to an embodiment of the present disclosure may divide a G-PCC bitstream into one or more tracks in a file and store them.
[0032] The apparatus and method according to the embodiments of the present disclosure may enable smooth and progressive playback by reducing the increase in playback complexity. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Figure 1 is a block diagram illustrating an example of a system for providing point cloud content according to an embodiment of the present disclosure.
[0034] Figure 2 is a block diagram illustrating an example of a process of providing point cloud content according to an embodiment of the present disclosure.
[0035] Figure 3 An example of a point cloud encoding device according to an embodiment of the present disclosure is illustrated.
[0036] Figure 4 An example of voxel according to an embodiment of the present disclosure is illustrated.
[0037] Figure 5 An example of an octree and an occupancy code according to an embodiment of the present disclosure is illustrated.
[0038] Figure 6 An example of point configuration of each LOD according to an embodiment of the present disclosure is illustrated.
[0039] Figure 7 is a block diagram illustrating an example of a point cloud decoding device according to an embodiment of the present disclosure.
[0040] Figure 8 is a block diagram illustrating another example of a point cloud decoding device according to an embodiment of the present disclosure.
[0041] Fig. 9 is a block diagram illustrating another example of a transmitting device according to an embodiment of the present disclosure.
[0042] Fig.10 is a block diagram illustrating another example of a receiving device according to an embodiment of the present disclosure.
[0043] Fig.11An example of a structure capable of interworking with a method / apparatus for transmitting and receiving point cloud data according to an embodiment of the present disclosure is illustrated.
[0044] Fig.12 An example of a file including a single track according to an embodiment of the present disclosure is illustrated.
[0045] Fig.13 An example of a file including a plurality of tracks according to an embodiment of the present disclosure is illustrated.
[0046] Figure 14 to Figure 15 An example of a method performed in a point cloud receiving device or a point cloud sending device according to an embodiment of the present disclosure is illustrated. DETAILED DESCRIPTION
[0047] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art to which the present disclosure belongs can easily implement them. The present disclosure can be implemented in many different forms and is not limited to the embodiments described herein.
[0048] In describing the present disclosure, a detailed description of known functions and configurations may be omitted when it may obscure the gist of the present disclosure. In the drawings, parts irrelevant to the description of the present disclosure are omitted, and like reference numerals are given to like parts.
[0049] In the present disclosure, when one component is "connected," "coupled," or "linked" to another component, it may include not only a direct connection relationship but also an indirect connection relationship with another component therebetween. In addition, when it is said that one component "includes" or "has" another component, this indicates that other components are not excluded but also included unless specifically described.
[0050] In the present disclosure, terms such as first, second, etc. are used only for the purpose of distinguishing one component from other components, and do not limit the order or importance of the components unless otherwise specified. Therefore, within the scope of the present disclosure, a first component in one embodiment may be referred to as a second component in another embodiment, and similarly, a second component in one embodiment may be referred to as a first component in another embodiment.
[0051] In the present disclosure, components that are distinguished from each other are used to clearly explain their features and do not necessarily mean that these components are separated. That is, multiple components can be integrated to form one hardware or software unit, or one component can be distributed to form multiple hardware or software units. Therefore, even if not specifically mentioned, such integrated or distributed implementations are also included in the scope of the present disclosure.
[0052] In the present disclosure, the components described in various embodiments are not necessarily essential components, and some of them may be optional components. Therefore, an embodiment consisting of a subset of the components described in one embodiment is also included in the scope of the present disclosure. In addition, an embodiment including other components in addition to the components described in various embodiments is also included in the scope of the present disclosure.
[0053] The present disclosure relates to encoding and decoding of point cloud related data, and unless the terms used in the present disclosure are redefined in the present disclosure, the terms may have general meanings commonly used in the technical field to which the present disclosure belongs.
[0054] In the present disclosure, the terms " / " and "," should be interpreted as indicating "and / or". For example, the expressions "A / B" and "A, B" may mean "A and / or B". In addition, "A / B / C" and "A / B / C" may mean "at least one of A, B, and / or C".
[0055] In the present disclosure, the term "or" should be interpreted as indicating "and / or". For example, the expression "A or B" may include 1) only "A", 2) only "B", and / or 3) both "A and B". In other words, in the present disclosure, the term "or" should be interpreted as indicating "additionally or alternatively".
[0056] The present disclosure relates to compression of point cloud related data. Various methods or embodiments of the present disclosure may be applied to point cloud compression or point cloud coding (PCC) standards (e.g., G-PCC or V-PCC standards) of the Moving Picture Experts Group (MPEG) or next generation video / image coding standards.
[0057] In the present disclosure, "point cloud" may refer to a set of points located in a three-dimensional space. Moreover, in the present disclosure, "point cloud content" is expressed as a point cloud, and may refer to a "point cloud video / image". Hereinafter, a "point cloud video / image" is referred to as a "point cloud video". A point cloud video may include one or more frames, and a frame may be a still image or a picture. Therefore, a point cloud video may include a point cloud image / frame / picture, and may be referred to as any one of a "point cloud image", a "point cloud frame", and a "point cloud picture".
[0058] In the present disclosure, "point cloud data" may refer to data or information related to each point in a point cloud. Point cloud data may include geometry and / or attributes. In addition, point cloud data may also include metadata. Point cloud data may be referred to as "point cloud content data" or "point cloud video data", etc. In addition, point cloud data may be referred to as "point cloud content", "point cloud video", "G-PCC data", etc.
[0059] In the present disclosure, a point cloud object corresponding to point cloud data may be represented by a frame shape based on a coordinate system, and the frame shape based on a coordinate system may be referred to as a bounding box. That is, the bounding box may be a cuboid that can accommodate all points of the point cloud, and may be a cuboid that includes a source point cloud frame.
[0060] In the present disclosure, geometry includes the position (or position information) of each point, and the position can be represented by parameters (e.g., x-axis value, y-axis value, and z-axis value) representing a three-dimensional coordinate system (e.g., a coordinate system composed of an x-axis, a y-axis, and a z-axis). Geometry can be referred to as "geometric information".
[0061] In the present disclosure, the attribute may include the attribute of each point, and the attribute may include one or more of texture information, color (RGB or YCbCr), reflectivity (r), transparency, etc. of each point. The attribute may be referred to as "attribute information". Metadata may include various data related to acquisition in an acquisition process to be described later.
[0062] Overview of Point Cloud Content Provision System
[0063] Figure 1 An example of a system for providing point cloud content (hereinafter, referred to as 'point cloud content providing system') according to an embodiment of the present disclosure is illustrated. Figure 2 An example of a process in which a point cloud content providing system provides point cloud content is illustrated.
[0064] like Figure 1 As shown, the point cloud content providing system may include a sending device 10 and a receiving device 20. The point cloud content providing system may be performed by the operations of the sending device 10 and the receiving device 20. Figure 2 The obtained process S20, the encoded process S21, the sent process S22, the decoded process S23, the rendered process S24 and / or the feedback process S25 are shown.
[0065] The sending device 10 acquires (i.e., obtains) point cloud data, and outputs a bit stream through a series of processes (e.g., encoding process) for the acquired (obtained) point cloud data (source point cloud data) to provide point cloud content. Here, the point cloud data can be output in the form of a bit stream through the encoding process. In some embodiments, the sending device 10 can send the output bit stream to the receiving device 20 in the form of a file or streaming (streaming segment) through a digital storage medium or a network. The digital storage medium may include a variety of storage media, such as USB, SD, CD, DVD, Blu-ray disc, HDD, and SSD. The receiving device 20 can process (e.g., decode or reconstruct) the received data (e.g., encoded point cloud data) into source point cloud data and render it. Point cloud content can be provided to users through these processes, and the present disclosure can provide various embodiments required to efficiently perform a series of these processes.
[0066] like Figure 1 As illustrated, the sending device 10 may include: an acquisition unit 11, an encoding unit 12, an encapsulation processing unit 13 and a sending unit 14, and the receiving device 20 may include: a receiving unit 21, a decapsulation processing unit 22, a decoding unit 23, and a rendering unit 24.
[0067] The acquisition unit 11 may perform a process S20 of acquiring (ie, obtaining) a point cloud video by a capturing, synthesizing, or generating process. Therefore, the acquisition unit 11 may be referred to as a "point cloud video acquisition unit".
[0068] Point cloud data (geometry and / or attributes, etc.) of a plurality of points may be generated through the acquisition process (S20). Furthermore, metadata related to the acquisition of the point cloud video may be generated through the acquisition process (S20). Furthermore, mesh data (e.g., triangle data) indicating connection information between point clouds may be generated through the acquisition process (S20).
[0069] The metadata may include initial viewing orientation metadata. The initial viewing orientation metadata may indicate whether the point cloud data represents front-side or back-side data. The metadata may be referred to as "auxiliary data," which is metadata of the point cloud.
[0070] The acquired (ie, obtained) point cloud video may include a polygon file format or a Stanford triangle format (PLY) file. Since the point cloud video has one or more frames, the acquired point cloud video may include one or more PLY files. The PLY file may include point cloud data of each point.
[0071] In order to acquire (i.e., obtain) a point cloud video (or point cloud data), the acquisition unit 11 may be composed of a camera device capable of acquiring depth (depth information) and a combination of an RGB camera capable of extracting color information corresponding to the depth information. Here, the camera device capable of acquiring depth information may be a combination of an infrared pattern projector and an infrared camera. In addition, the acquisition unit 11 may be composed of a laser radar (LiDAR), and the laser radar may use a radar system for measuring the position coordinates of a reflector by measuring the time required for a laser pulse to be emitted and to return after being reflected.
[0072] The acquisition unit 110 may extract a shape of geometry composed of points in a three-dimensional space from the depth information, and may extract a property representing the color or reflection of each point from the RGB information.
[0073] As a method of extracting (or capturing, acquiring, etc.) a point cloud video (or point cloud data), there may be an inward-facing method of capturing a central object and an outward-facing method of capturing an external environment. Figure 3 Examples of an inward-facing approach and an outward-facing approach are shown. Figure 3 (a) shows an example of an inward-facing approach, and Figure 3 (b) shows an example of an outward-facing approach.
[0074] The encoding unit 12 may perform an encoding process (S21) of encoding the data generated by the acquisition unit 11 (e.g., geometry, attributes and / or metadata, and / or mesh data, etc.) into one or more bitstreams. Therefore, the encoding unit 12 may be referred to as a 'point cloud video encoder'. The encoding unit 12 may encode the data generated by the acquisition unit 11 serially or in parallel.
[0075] The encoding process S21 performed by the encoding unit 12 may be geometry-based point cloud compression (G-PCC). The encoding unit 12 may perform a series of processes such as prediction, transform, quantization, and entropy encoding for compression and encoding efficiency.
[0076] The encoded point cloud data can be output in the form of a bitstream. Based on the G-PCC process, the encoding unit 12 can split the point cloud data into geometry and attributes and encode them as described below. In this case, the output bitstream may include a geometry bitstream containing the encoded geometry and an attribute bitstream containing the encoded attributes. In addition, the output bitstream may also include one or more of a metadata bitstream containing metadata, an auxiliary bitstream containing auxiliary data, and a mesh data bitstream containing mesh data. The encoding process (S21) will be described in more detail below. The bitstream including the encoded point cloud data may be referred to as a 'point cloud bitstream' or a 'point cloud video bitstream'.
[0077] The encapsulation processing unit 13 may perform a process of encapsulating one or more bit streams output from the decoding unit 12 in the form of a file or segment. Therefore, the encapsulation processing unit 13 may be referred to as a 'file / segment encapsulation module'. Although the drawings show an example in which the encapsulation processing unit 13 is composed of a separate component / module associated with the sending unit 14, in some embodiments, the encapsulation processing unit 13 may be included in the sending unit 14.
[0078] The encapsulation processing unit 13 may encapsulate data in a file format such as ISO Base Media File Format (ISOBMFF), or process data in the form of other DASH segments. In some embodiments, the encapsulation processing unit 13 may include metadata of the file format. The metadata may be included in boxes at various levels of, for example, the ISOBMFF file format, or as data in a separate track within a file. In some embodiments, the encapsulation processing unit 130 may encapsulate the metadata itself into a file. The metadata processed by the encapsulation processing unit 13 may be sent from a metadata processing unit not shown in the figure. The metadata processing unit may be included in the encoding unit 12, or may be configured as a separate component / module.
[0079] The sending unit 14 may perform a sending process (S22) of applying a process according to a file format (a process for sending) to the 'encapsulated point cloud bitstream'. The sending unit 140 may send a bitstream or a file / segment including a bitstream to the receiving unit 21 of the receiving device 20 through a digital storage medium or a network. Therefore, the sending unit 14 may be referred to as a 'sender' or a 'communication module'.
[0080] The receiving unit 21 can receive the bitstream or the file / segment including the bitstream sent by the sending device 10. According to the channel sent, the receiving unit 21 can receive the bitstream or the file / segment including the bitstream through a broadcast network, or can receive the bitstream or the file / segment including the bitstream through a broadband. Alternatively, the receiving unit 21 can receive the bitstream or the file / segment including the bitstream through a digital storage medium.
[0081] The receiving unit 21 may perform processing on the received bitstream or the file / segment including the bitstream according to the transmission protocol. The receiving unit 21 may perform the inverse process of the sending process (processing for sending) to correspond to the processing for sending performed by the sending device 10. The receiving unit 21 may send the encoded point cloud data among the received data to the decapsulation processing unit 22, and may send the metadata to the metadata parsing unit. The metadata may be in the form of a signaling table. In some embodiments, the inverse process of the processing for sending may be performed in the receiving processing unit. Each of the receiving processing unit, the decapsulation processing unit 22, and the metadata parsing unit may be included in the receiving unit 21, or may be configured as a component / module separate from the receiving unit 21.
[0082] The decapsulation processing unit 22 can decapsulate the point cloud data in the file format (ie, the bit stream in the file format) received from the receiving unit 21 or the receiving processing unit. Therefore, the decapsulation processing unit 22 can be called a 'file / segment decapsulation module'.
[0083] The decapsulation processing unit 22 can obtain a point cloud bitstream or a metadata bitstream by decapsulating the file according to ISOBMFF, etc. In some embodiments, metadata (metadata bitstream) can be included in the point cloud bitstream. The acquired point cloud bitstream can be sent to the decoding unit 23, and the acquired metadata bitstream can be sent to the metadata processing unit. The metadata processing unit can be included in the decoding unit 23, or can be configured as a separate component / module. The metadata obtained by the decapsulation processing unit 23 can be in the form of a box or track in the file format. If necessary, the decapsulation processing unit 23 can receive the metadata required for decapsulation from the metadata processing unit. The metadata can be sent to the decoding unit 23 and used in the decoding process (S23), or can be sent to the rendering unit 24 and used in the rendering process (S24).
[0084] The decoding unit 23 may receive the bitstream and perform operations corresponding to those of the encoding unit 12, thereby performing a decoding process (S23) of decoding the point cloud bitstream (encoded point cloud data). Therefore, the decoding unit 23 may be referred to as a 'point cloud video decoder'.
[0085] The decoding unit 23 may segment the point cloud data into geometry and attributes and decode them. For example, the decoding unit 23 may reconstruct (decode) the geometry from the geometry bitstream included in the point cloud bitstream, and restore (decode) the attributes based on the reconstructed geometry and the attribute bitstream included in the point cloud bitstream. The three-dimensional point cloud video / image may be reconstructed based on the position information according to the reconstructed geometry and the attributes (such as color or texture) according to the decoded attributes. The decoding process (S23) will be described in more detail below.
[0086] The rendering unit 24 may perform a rendering process S24 of rendering the reconstructed point cloud video. Therefore, the rendering unit 24 may be referred to as a 'renderer'.
[0087] The rendering process S24 may refer to a process of rendering and displaying point cloud content in a 3D space. The rendering process S24 may perform rendering according to a desired rendering method based on the position information and attribute information of the points decoded by the decoding process.
[0088] The feedback process S25 may include a process of sending various feedback information obtained during the rendering process S24 or the display process to other components in the sending device 10 or the receiving device 20. The feedback process S25 may be performed by Figure 1 The receiving device 20 may be executed by one or more components included in the receiving device 20, or may be executed by Figure 7 and Figure 8 In some embodiments, the feedback process S25 may be performed by a 'feedback unit' or a 'sensing / tracking unit'.
[0089] Overview of Point Cloud Coding Equipment
[0090] Figure 3 An example of a point cloud encoding device 400 according to an embodiment of the present disclosure is illustrated. Figure 3 The point cloud encoding device 400 may correspond in configuration and function to Figure 1 The encoding unit 12.
[0091] like Figure 3 As shown, the point cloud encoding device 400 may include: a coordinate system transformation unit 405, a geometry quantization unit 410, an octree analysis unit 415, an approximation unit 420, a geometry encoding unit 425, a reconstruction unit 430, and an attribute transformation unit 440, a RAHT transformation unit 445, an LOD generation unit 450, a lifting unit 455, an attribute quantization unit 460, an attribute encoding unit 465, and / or a color transformation unit 435.
[0092] The point cloud data acquired by the acquisition unit 11 may undergo a process of adjusting the quality of the point cloud content (e.g., lossless, lossy, near lossless) according to the network situation or application. In addition, each point of the acquired point cloud content may be sent losslessly, but in this case, because the size of the point cloud content is large, real-time streaming is not possible. Therefore, in order to smoothly provide the point cloud content, a process of reconstructing the point cloud content according to the maximum target bit rate is required.
[0093] The process of adjusting the quality of the point cloud content may be a process of reconstructing and encoding the position information (position information included in the geometric information) or color information (color information included in the attribute information) of the point. The process of reconstructing and encoding the position information of the point may be referred to as geometric encoding, and the process of reconstructing and encoding the attribute information associated with each point may be referred to as attribute encoding.
[0094] Geometry coding may include: a geometry quantization process, a voxelization process, an octree analysis process, an approximation process, a geometry coding process, and / or a coordinate system transformation process. Moreover, geometry coding may also include a geometry reconstruction process. Attribute coding may include: a color transformation process, an attribute transformation process, a prediction transformation process, a lifting transformation process, a RAHT transformation process, an attribute quantization process, an attribute coding process, etc.
[0095] Geometry Encoding
[0096] The coordinate system transformation process may correspond to a process of transforming the coordinate system of the point position. Therefore, the coordinate system transformation process may be referred to as 'transformed coordinates'. The coordinate system transformation process may be performed by the coordinate system transformation unit 405. For example, the coordinate system transformation unit 405 may transform the position of the point from the global space coordinate system into position information in a three-dimensional space (e.g., a three-dimensional space expressed in a coordinate system of an X-axis, a Y-axis, and a Z-axis). The position information in the 3D space according to an embodiment may be referred to as 'geometric information'.
[0097] The geometric quantization process may correspond to a process of quantizing the position information of the points, and may be performed by the geometric quantization unit 410. For example, the geometric quantization unit 410 may find the position information having the minimum (x, y, z) value among the position information of the points, and subtract the position information having the minimum (x, y, z) position from the position information of each point. In addition, the geometric quantization unit 410 may multiply the subtracted value by a preset quantization scale value, and then adjust (lower or raise) the result to a value close to an integer, thereby performing the quantization process.
[0098] The voxelization process may correspond to a process of matching the geometric information quantized by the quantization process with a specific voxel existing in the 3D space. The voxelization process may also be performed by the geometric quantization unit 410. The geometric quantization unit 410 may perform octree-based voxelization based on the position information of the point so as to reconstruct each point to which the quantization process is applied.
[0099] Figure 4 An example of a voxel according to an embodiment of the present disclosure is shown. A voxel may refer to a space for storing information about a point existing in 3D, similar to a pixel which is the smallest unit having information about a 2D image / video. A voxel is a hybrid word obtained by combining volume and pixel. Figure 4 As illustrated, a voxel refers to a three-dimensional cubic space formed by dividing a three-dimensional space (2depth, 2depth, 2depth) into units (unit = 1.0) based on each axis (x-axis, y-axis, and z-axis). A voxel can estimate spatial coordinates based on a positional relationship with a voxel group, and can have color or reflectivity information similar to a pixel.
[0100] There may not be only one point (matched) in a voxel. That is, there may be information related to multiple points in a voxel. Alternatively, information related to multiple points included in a voxel can be integrated into one point information. This adjustment can be performed selectively. When a voxel is integrated and expressed as a point information, the position value of the center point of the voxel can be set based on the position value of the point existing in the voxel, and the attribute transformation process related thereto needs to be performed. For example, the attribute transformation process can be adjusted to the position value of the point included in the voxel or the center point of the voxel, and the average value of the color or reflectivity of the neighboring points within a specific radius.
[0101] The octree analysis unit 415 may use the octree to efficiently manage regions / positions of voxels. Figure 6 (a) shows an example of an octree according to an embodiment of the present disclosure. In order to efficiently manage the space of a two-dimensional image, if the entire space is divided based on the x-axis and the y-axis, four spaces are created, and when each of the four spaces is divided based on the x-axis and the y-axis, four spaces are created for each small space. The region can be divided until the leaf node becomes a pixel, and a quadtree can be used as a data structure for efficient management according to the size and position of the region.
[0102] Likewise, the present disclosure can apply the same method to efficiently manage 3D space according to the position and size of the space. Figure 6 As shown in the middle of (a), since the z-axis is added, when the three-dimensional space is divided based on the x-axis, y-axis, and z-axis, eight spaces can be created. Figure 6 As shown on the right side of (a), when each of the 8 spaces is divided again based on the x-axis, the y-axis, and the z-axis, 8 spaces can be created for each small space.
[0103] The octree analysis unit 415 may split the region until the leaf nodes become voxels, and may use an octree data structure capable of managing eight child node regions for efficient management according to the size and location of the region.
[0104] The octree can be expressed as an occupancy code, and Figure 5 (b) shows an example of an occupation code according to an embodiment of the present disclosure. The octree analysis unit 415 may express the occupation code of the node as 1 when one point is included in each node, and may express the occupation code of the node as 0 when the point is not included.
[0105] The geometric coding process may correspond to a process of performing entropy coding on the occupancy code. The geometric coding process may be performed by the geometric coding unit 425. The geometric coding unit 425 may perform entropy coding on the occupancy code. The generated occupancy code may be encoded immediately or may be encoded by an intra / inter coding process to improve compression efficiency. The receiving device 20 may reconstruct the octet tree by the occupancy code.
[0106] On the other hand, in the case where there are no points or only a few points in a particular region, voxelizing all the regions may be inefficient. That is, since there are very few points in a particular region, it may not be necessary to construct the entire octree. For such cases, an early termination method may be required.
[0107] The point cloud encoding device 400 can directly send the position of the point only for a specific area, or use a surface model to reconfigure the position of the point within a specific area based on voxels, instead of splitting the node (specific node) corresponding to the specific area (a specific area not corresponding to a leaf node) into 8 sub-nodes (sub-nodes).
[0108] A mode for directly transmitting the positions of each point for a specific node may be a direct mode. The point cloud encoding device 400 may check whether a condition for enabling the direct mode is satisfied.
[0109] The conditions for enabling direct mode are: 1) the option to use direct mode should be enabled, 2) the specific node does not correspond to a leaf node, 3) there should be points below the threshold within the specific node, and 4) the total number of points to be sent directly does not exceed the limit value.
[0110] When all the above conditions are met, the point cloud encoding device 400 can directly perform entropy encoding and transmission on the position value of the point for a specific node through the geometric encoding unit 425 .
[0111] The mode of reconstructing the position of the point in the specific area based on voxels using the surface model may be a trisoup mode. The trisoup mode may be performed by the approximation unit 420. The approximation unit 420 may determine a specific level of the octree, and reconstruct the position of the point in the node area based on voxels using the surface model from the determined specific level.
[0112] The point cloud encoding device 400 may selectively apply the triangle soup mode. Specifically, when the triangle soup mode is applied, the point cloud encoding device 400 may specify the level (specific level) at which the triangle soup mode is applied. For example, when the specified specific level is equal to the depth (d) of the octree, the triangle soup mode may not be applied. That is, the specified specific level should be less than the depth value of the octree.
[0113] The three-dimensional cube area of the node of the specified specific level is called a block, and a block can include one or more voxels. A block or voxel can correspond to a brick. Each block can have 12 edges, and the approximation unit 420 can check whether each edge is adjacent to an occupied voxel with a point. Each edge can be adjacent to multiple occupied voxels. The specific position of the edge adjacent to a voxel is called a vertex, and when multiple occupied voxels are adjacent to an edge, the approximation unit 420 can determine the average position of these positions as a vertex.
[0114] When there are vertices, the point cloud encoding device 400 can perform entropy encoding on the starting point (x, y, z) of the edge, the direction vector (Δx, Δy, Δz) of the edge, and the position value of the vertex (relative position value within the edge) through the geometric encoding unit 425.
[0115] The geometry reconstruction process may correspond to a process of generating reconstructed geometry by reconstructing an octree and / or an approximate octree. The geometry reconstruction process may be performed by the reconstruction unit 430. The reconstruction unit 430 may perform the geometry reconstruction process by triangle reconstruction, upsampling, voxelization, and the like.
[0116] When the triangle soup mode is applied in the approximation unit 420, the reconstruction unit 430 may reconstruct a triangle based on a start point of an edge, a direction vector of the edge, and a position value of a vertex.
[0117] The reconstruction unit 430 may perform an upsampling process for voxelization by adding points in the middle along the sides of the triangle. The reconstruction unit 430 may generate additional points based on the upsampling factor and the width of the block. These points may be referred to as refinement vertices. The reconstruction unit 430 may voxelize the refinement vertices, and the point cloud encoding device 400 may perform attribute encoding based on the voxelized position values.
[0118] In some embodiments, the geometry coding unit 425 can increase the compression efficiency by applying context adaptive arithmetic coding. The geometry coding unit 425 can entropy encode the occupancy code directly using arithmetic codes. In some embodiments, the geometry coding unit 425 adaptively performs encoding based on the occupancy of neighboring nodes (intra-frame encoding), or adaptively performs encoding based on the occupancy code of the previous frame (inter-frame encoding). Here, a frame can refer to a set of point cloud data generated at the same time. Intra-frame encoding and inter-frame encoding are optional processes and can therefore be omitted.
[0119] Attribute Encoding
[0120] Attribute encoding may correspond to a process of encoding attribute information based on the reconstructed geometry and the geometry before the coordinate system transformation (source geometry). Since the attributes may depend on the geometry, the reconstructed geometry may be used for attribute encoding.
[0121] As mentioned above, attributes may include color, reflectivity, etc. The same attribute encoding method may be applied to the information or parameters included in the attributes. Color has three elements, reflectivity has one element, and each element may be processed independently.
[0122] Attribute coding may include: color transformation process, attribute transformation process, prediction transformation process, lifting transformation process, RAHT transformation process, attribute quantization process, attribute coding process, etc. The prediction transformation process, lifting transformation process, and RAHT transformation process may be selectively used, or a combination of one or more processes may be used.
[0123] The color conversion process may correspond to a process of converting the format of the color in the attribute to another format. The color conversion process may be performed by the color conversion unit 435. That is, the color conversion unit 435 may convert the color in the attribute. For example, the color conversion unit 435 may perform an encoding operation for converting the color in the attribute from RGB to YCbCr. In some embodiments, the operation of the color conversion unit 435 (i.e., the color conversion process) may be optionally applied according to the color value included in the attribute.
[0124] As described above, when one or more points exist in one voxel, the position values of the points existing in the voxel are set to the center point of the voxel so that they can be displayed by integrating them into one point information of the voxel. Therefore, a process of transforming the attribute values associated with the points may be required. Moreover, even if the triangulation mode is executed, the attribute transformation process may be performed.
[0125] The attribute transformation process may correspond to a process of transforming attributes based on a position where geometry encoding is not performed and / or a reconstructed geometry. For example, the attribute transformation process may correspond to a process of transforming the attributes of a point having the position based on the position of the point included in the voxel. The attribute transformation process may be performed by the attribute transformation unit 440. The attribute transformation unit 440 may calculate the center position value of the voxel and the average value of the attribute values of neighboring points within a specific radius. Alternatively, the attribute transformation unit 440 may apply weights to the attribute values according to the distance from the center position and calculate the average value of the attribute values to which the weights are applied. In this case, each voxel has a position and a calculated attribute value.
[0126] The prediction transformation process may correspond to a process of predicting the attribute value of the current point based on the attribute values of one or more points (neighboring points) adjacent to the current point (corresponding to the prediction target point). The prediction transformation process may be performed by a level-of-detail (LOD) generation unit 450.
[0127] Prediction transformation is a method of applying LOD transformation technology, and the LOD generation unit 450 can calculate and set the LOD value of each point based on the LOD distance value of each point. The LOD generation unit 450 can generate a predictor for each point of the prediction transformation. Therefore, when there are N points, N predictors can be generated. The predictor can calculate and set the weight value (=1 / distance) based on the LOD value of each point, the index information of the neighboring point, and the distance value from the neighboring point. Here, the neighboring point can be a point that exists within the distance from the current point set for each LOD.
[0128] In addition, the predictor may multiply the attribute value of the neighboring point by a 'set weight value' and set the value obtained by averaging the attribute values multiplied by the weight value as the predicted attribute value of the current point. The attribute quantization process may be performed on the residual attribute value obtained by subtracting the predicted attribute value of the current point from the attribute value of the current point.
[0129] The lifting transformation process may correspond to a process of reconstructing a point into a set of detail levels through an LOD generation process, just like the prediction transformation process. The lifting transformation process may be performed by the lifting unit 455. The lifting transformation process may also include a process of generating a predictor for each point, a process of setting the calculated LOD in the predictor, a process of registering neighbor points, and a process of setting weights according to the distance between the current point and the neighbor points.
[0130] The RAHT transformation process may correspond to a method of predicting attribute information of a higher-level node using attribute information associated with a lower-level node of an octree. That is, the RATH transformation process may correspond to an attribute information intra-frame encoding method performed by backward scanning of the octree. The RAHT transformation process may be performed by a RAHT transformation unit 445. The RAHT transformation unit 445 scans the entire area in the voxel, and may perform a RAHT transformation process up to the root node while summing (merging) the voxels into larger blocks at various steps. Since the RAHT transformation unit 445 performs the RAHT transformation process only on occupied nodes, in the case of an unoccupied empty node, the RAHT transformation process may be performed on a higher-level node directly above it.
[0131] The attribute quantization process may correspond to a process of quantizing the attribute output from the RAHT transform unit 445, the LOD generation unit 450, and / or the lifting unit 455. The attribute quantization process may be performed by the attribute quantization unit 460. The attribute encoding process may correspond to a process of encoding the quantized attribute and outputting the attribute bitstream. The attribute encoding process may be performed by the attribute encoding unit 465.
[0132] Overview of Point Cloud Decoding Equipment
[0133] Figure 7 An example of a point cloud decoding device 1000 according to an embodiment of the present disclosure is shown. Figure 7 The point cloud decoding device 1000 may correspond to Figure 1 The decoding unit 23.
[0134] The point cloud decoding apparatus 1000 may perform a decoding process based on the data (bit stream) transmitted from the transmitting device 10. The decoding process may include a process of reconstructing (decoding) a point cloud video by performing an operation corresponding to the above-described encoding operation on the bit stream.
[0135] like Figure 7As illustrated, the decoding process may include a geometry decoding process and an attribute decoding process. The geometry decoding process may be performed by a geometry decoding unit 1010, and the attribute decoding process may be performed by an attribute decoding unit 1020. That is, the point cloud decoding device 1000 may include a geometry decoding unit 1010 and an attribute decoding unit 1020.
[0136] The geometry decoding unit 1010 may reconstruct the geometry from the geometry bitstream, and the attribute decoder 1020 may reconstruct the attribute based on the reconstructed geometry and the attribute bitstream. Furthermore, the point cloud decoding apparatus 1000 may reconstruct a three-dimensional point cloud video (point cloud data) based on position information according to the reconstructed geometry and attribute information according to the reconstructed attribute.
[0137] Figure 8 A specific example of a point cloud decoding device 1100 according to another embodiment of the present disclosure is illustrated. Figure 8 As illustrated, the point cloud decoding device 1100 includes: a geometry decoding unit 1105, an octree synthesis unit 1110, an approximate synthesis unit 1115, a geometry reconstruction unit 1120, a coordinate system inverse transformation unit 1125, an attribute decoding unit 1130, an attribute dequantization unit 1135, a RATH transformation unit 1150, an LOD generation unit 1140, an inverse lifting unit 1145, and / or a color inverse transformation unit 1155.
[0138] The geometry decoding unit 1105, the octree synthesis unit 1110, the approximate synthesis unit 1115, the geometry reconstruction unit 1120, and the coordinate system inverse transformation unit 1150 may perform geometry decoding. The geometry decoding may be used as a reference. Figures 1 to 6 The geometric decoding may include direct coding and triangular soup geometric decoding. Direct coding and triangular soup geometric decoding may be applied selectively.
[0139] The geometry decoding unit 1105 may decode the received geometry bitstream based on arithmetic coding. The operation of the geometry decoding unit 1105 may correspond to the inverse process of the operation performed by the geometry encoding unit 435.
[0140] The octree synthesis unit 1110 may generate an octree by obtaining an occupancy code (or information about geometry obtained as a result of decoding) from the decoded geometry bitstream. The operation of the octree synthesis unit 1110 may correspond to the inverse process of the operation performed by the octree analysis unit 415.
[0141] When triangle soup geometry encoding is applied, the approximate synthesis unit 1115 can synthesize the surface based on the decoded geometry and / or the generated octree.
[0142] The geometry reconstruction unit 1120 may reconstruct the geometry based on the surface and the decoded geometry. When direct coding is applied, the geometry reconstruction unit 1120 may directly bring and add the position information of the points to which the direct coding is applied. In addition, when triangulated geometry coding is applied, the geometry reconstruction unit 1120 may reconstruct the geometry by performing reconstruction operations (e.g., triangle reconstruction, upsampling, voxelization operations, etc.). The reconstructed geometry may include a point cloud picture or frame that does not include attributes.
[0143] The coordinate system inverse transformation unit 1150 can obtain the position of the point by transforming the coordinate system based on the reconstructed geometry. For example, the coordinate system inverse transformation unit 1150 can inversely transform the position of the point from a three-dimensional space (e.g., a three-dimensional space expressed by a coordinate system of an x-axis, a y-axis, and a z-axis, etc.) into position information of a global space coordinate system.
[0144] The attribute decoding unit 1130, the attribute dequantization unit 1135, the RATH transform unit 1230, the LOD generation unit 1140, the inverse lifting unit 1145, and / or the color inverse transform unit 1250 may perform attribute decoding. Attribute decoding may include: RAHT transform decoding, prediction transform decoding, and lifting transform decoding. The above three types of decoding may be selectively used, or a combination of one or more types of decoding may be used.
[0145] The attribute decoding unit 1130 may decode the attribute bitstream based on arithmetic coding. For example, when there is no neighbor point in the predictor of each point and thus the attribute value of the current point is directly entropy encoded, the attribute decoding unit 1130 may decode the attribute value (unquantized attribute value) of the current point. As another example, when there is a neighbor point in the predictor of the current point and thus the quantized residual attribute value is entropy encoded, the attribute decoding unit 1130 may decode the quantized residual attribute value.
[0146] The attribute dequantization unit 1135 may dequantize the decoded attribute bitstream or information about the attribute obtained as a result of decoding, and output the dequantized attribute (or attribute value). For example, when the quantized residual attribute value is output from the attribute decoding unit 1130, the attribute dequantization unit 1135 may dequantize the quantized residual attribute value to output the residual attribute value. The dequantization process may be selectively applied based on whether the attribute is encoded in the point cloud encoding device 400. That is, when there is no neighbor point in the predictor of each point and therefore the attribute value of the current point is directly encoded, the attribute decoding unit 1130 may output the attribute value of the current point that is not quantized, and the attribute encoding process may be skipped.
[0147] The RATH transform unit 1150, the LOD generation unit 1140, and / or the inverse lifting unit 1145 may process the reconstructed geometry and the dequantized attributes. The RATH transform unit 1150, the LOD generation unit 1140, and / or the inverse lifting unit 1145 may selectively perform a decoding operation corresponding to the encoding operation of the point cloud encoding device 400.
[0148] The color inverse transform unit 1155 may perform inverse transform encoding to inversely transform a color value (or texture) included in the decoded attribute. The operation of the color inverse transform unit 1155 may be selectively performed based on whether the color transform unit 435 operates.
[0149] Fig. 9 Another example of a transmitting device according to an embodiment of the present disclosure is shown. Fig. 9 As illustrated, the sending device may include: a data input unit 1205, a quantization processing unit 1210, a voxelization processing unit 1215, an octree occupancy code generation unit 1220, a surface model processing unit 1225, an intra-frame / inter-frame coding processing unit 1230, an arithmetic encoder 1235, a metadata processing unit 1240, a color transformation processing unit 1245, an attribute transformation processing unit 1250, a prediction / enhancement / RAHT transformation processing unit 1255, an arithmetic encoder 1260 and a sending processing unit 1265.
[0150] The function of the data input unit 1205 may correspond to Figure 1 The acquisition process is performed by the acquisition unit 11. That is, the data input unit 1205 can acquire the point cloud video and generate point cloud data of multiple points. The geometric information (position information) in the point cloud data can be generated in the form of a geometric bit stream through the quantization processing unit 1210, the voxelization processing unit 1215, the octree occupancy code generation unit 1220, the surface model processing unit 1225, the intra / inter-frame coding processing unit 1230 and the arithmetic encoder 1235. The attribute information in the point cloud data can be generated in the form of an attribute bit stream through the color transformation processing unit 1245, the attribute transformation processing unit 1250, the prediction / lifting / RAHT transformation processing unit 1255, and the arithmetic encoder 1260. The geometric bit stream, the attribute bit stream, and / or the metadata bit stream can be sent to the receiving device through the processing of the sending processing unit 1265.
[0151] Specifically, the function of the quantization processing unit 1210 may correspond to Figure 3 The function of the voxelization processing unit 1215 may correspond to the quantization process performed by the geometric quantization unit 410 and / or the function of the coordinate system transformation unit 405. Figure 3The voxelization process performed by the geometric quantization unit 410 of , and the function of the octet occupancy code generation unit 1220 may correspond to the ... function of the octet occupancy code generation unit 1220 of . Figure 3 The functions of the surface model processing unit 1225 may correspond to the functions performed by the octree analysis unit 415. Figure 3 The function of the approximation unit 420 of the present invention may correspond to the function performed by the geometry encoding unit 425. The function of the metadata processing unit 1240 may correspond to the function of the intra / inter encoding processing unit 1230 and the function of the arithmetic encoder 1235. Figure 1 Describes the functionality of the metadata processing unit.
[0152] In addition, the function of the color conversion processing unit 1245 may correspond to Figure 3 The function of the color conversion unit 435 is performed, and the function of the attribute conversion processing unit 1250 may correspond to the function of the color conversion unit 435 performed by Figure 3 The function of the prediction / lifting / RAHT transformation processing unit 1255 may correspond to the function of the attribute transformation unit 440 performed by Figure 3 The functions performed by the RAHT transform unit 4450, the LOD generation unit 450, and the lifting unit 455, and the function of the arithmetic encoder 1260 may correspond to Figure 3 The function of the attribute encoding unit 465 may correspond to the function of the attribute encoding unit 465. Figure 1 The functions performed by the sending unit 14 and / or the encapsulation processing unit 13.
[0153] Fig.10 Another example of a receiving device according to an embodiment of the present disclosure is shown. Fig.10 As illustrated, the receiving device includes: a receiving unit 1305, a receiving processing unit 1310, an arithmetic decoder 1315, a metadata parser 1335, an octree reconstruction processing unit 1320 based on an occupancy code, a surface model processing unit 1325, an inverse quantization processing unit 1330, an arithmetic decoder 1340, an inverse quantization processing unit 1345, a prediction / lifting / RAHT inverse transform processing unit 1350, a color inverse transform processing unit 1355 and a renderer 1360.
[0154] The function of the receiving unit 1305 may correspond to Figure 1 The functions of the receiving unit 21 and the receiving processing unit 1310 may correspond to the functions of Figure 1The function performed by the decapsulation processing unit 22 of the receiving unit 1305. That is, the receiving unit 1305 can receive the bitstream from the sending processing unit 1265, and the receiving processing unit 1310 can extract the geometry bitstream, the attribute bitstream, and / or the metadata bitstream through the decapsulation process. The geometry bitstream can be generated as a reconstructed position value (position information) by the arithmetic decoder 1315, the octtree reconstruction processing unit 1320 based on the occupancy code, the surface model processing unit 1325, and the inverse quantization processing unit 1330. The attribute bitstream can be generated as a reconstructed attribute value by the arithmetic decoder 1340, the inverse quantization processing unit 1345, the prediction / lifting / RAHT inverse transform processing unit 1350, and the color inverse transform processing unit 1355. The metadata bitstream can be generated as reconstructed metadata (or metadata information) by the metadata parser 1335. The position value, attribute value, and / or metadata can be rendered in the renderer 1360 to provide users with experiences such as VR / AR / MR / autonomous driving.
[0155] Specifically, the function of the arithmetic decoder 1315 may correspond to Figure 8 The function of the geometric decoding unit 1105 is performed, and the function of the occupancy code based octree reconstruction unit 1320 may correspond to the function of Figure 8 The functions of the surface model processing unit 1325 may correspond to the functions performed by the octree synthesis unit 1110. Figure 8 The function of the approximate synthesis unit 1300 may correspond to the function of the inverse quantization processing unit 1330 Figure 8 The functions of the metadata parser 1335 may correspond to the functions performed by the reference Figure 1 Describes the functions performed by a metadata parser.
[0156] In addition, the function of the arithmetic decoder 1340 may correspond to Figure 8 The function of the attribute decoding unit 1130 is performed, and the function of the inverse quantization processing unit 1345 may correspond to Figure 8 The function of the prediction / lifting / RAHT inverse transform processing unit 1350 may correspond to the function of the inverse quantization unit 1135 of the attribute. Figure 8 The functions of the RAHT transform unit 1150, the LOD generation unit 1140, and the inverse lifting unit 1145 are performed by the color inverse transform processing unit 1355, and the functions of the color inverse transform processing unit 1355 may correspond to the functions of the RAHT transform unit 1150, the LOD generation unit 1140, and the inverse lifting unit 1145. Figure 8 The function performed by the color inverse conversion unit 1155.
[0157] Fig.11An example of a structure that can interact with a method / apparatus for transmitting and receiving point cloud data according to an embodiment of the present disclosure is illustrated.
[0158] Fig.11 The structure of illustrates a configuration in which at least one of a server (AI server), a robot, an autonomous driving vehicle, an XR device, a smart phone, a home appliance, and / or an HMD is connected to a cloud network. A robot, an autonomous driving vehicle, an XR device, a smart phone, or a home appliance may be referred to as a device. Additionally, the XR device may correspond to a point cloud data device (PCC) according to an embodiment, or may interact with a PCC device.
[0159] The cloud network may refer to a network that forms part of or exists within a cloud computing infrastructure. Here, a 3G network, a 4G or Long Term Evolution (LTE) network, or a 5G network may be used to configure the cloud network.
[0160] The server may be connected to at least one of a robot, an autonomous driving vehicle, an XR device, a smart phone, a home appliance, and / or an HMD through the cloud network, and may assist with at least a part of the processing of the connected devices.
[0161] The HMD may represent one of the types of XR devices and / or PCC devices in which an embodiment may be implemented. The HMD type device according to an embodiment may include: a communication unit, a control unit, a memory unit, an I / O unit, a sensor unit, and a power supply unit.
[0162] <PCC+XR>
[0163] The XR / PCC device may be implemented by an HMD, a HUD provided in a vehicle, a TV, a mobile phone, a smart phone, a computer, a wearable device, a home appliance, a digital sign, a vehicle, a stationary robot, or a mobile robot by applying PCC and / or XR technologies.
[0164] The XR / PCC device may analyze 3D point cloud data or image data obtained through various sensors or from external devices to obtain information about the surrounding space or real objects, generate position (geometry) data and attribute data of 3D points, and render and output XR objects to be output. For example, the XR / PCC device may output an XR object corresponding to the identified object including additional information about the identified object.
[0165] <PCC+XR+mobile phone>
[0166] The XR / PCC device can be implemented by a mobile phone or the like through the application of PCC technology. The mobile phone can decode and display point cloud content based on PCC technology.
[0167] <PCC + Autopilot + XR>
[0168] An autonomous vehicle can be implemented by a mobile robot, a vehicle, an unmanned aerial vehicle, etc. through the application of PCC technology and XR technology. An autonomous vehicle applying XR / PCC technology may refer to an autonomous vehicle equipped with a unit for providing XR images, or an autonomous vehicle that undergoes control / interaction within an XR image. In particular, an autonomous vehicle that undergoes control / interaction within an XR image is distinguishable from an XR device and can work interactively with each other.
[0169] An autonomous vehicle equipped with a unit for providing XR / PCC images can acquire sensor information from sensors including cameras and output XR / PCC images generated based on the acquired sensor information. For example, an autonomous vehicle has an HUD and can provide an XR / PCC object corresponding to a real object or an object on the screen to passengers by outputting XR / PCC images.
[0170] In this case, when an XR / PCC object is output to the HUD, at least a part of the XR / PCC object can be output so as to overlap an actual object pointed at by a passenger's gaze. On the other hand, when an XR / PCC object is output to a display provided inside the autonomous vehicle, at least a part of the XR / PCC object can be output to overlap an object on the screen. For example, an autonomous vehicle can output XR / PCC objects corresponding to objects such as lanes, other vehicles, traffic lights, traffic signs, two-wheeled vehicles, pedestrians, and buildings.
[0171] VR technology, AR technology, MR technology, and / or PCC technology according to an embodiment can be applied to various devices. That is, VR technology is a display technology that provides only CG images of objects or backgrounds in the real world. On the other hand, AR technology refers to a technology of displaying virtual CG images on top of real object images. Moreover, MR technology is similar to the above AR technology in that it shows the mixing and combination of virtual objects in the real world. However, in AR technology, the distinction between real objects and virtual objects composed of CG images is clear, and virtual objects are used in a form that complements real objects, while in MR technology, virtual objects are considered equivalent to real objects, which is different from AR technology. More specifically, for example, hologram services apply the above MR technology. VR, AR, and MR technologies can be integrated and referred to as XR technology.
[0172] Space segmentation
[0173] Point cloud data (i.e., G-PCC data) may represent a volumetric encoding of a point cloud consisting of a sequence of frames (point cloud frames). Each point cloud frame may include: the number of points, the locations of the points, and the attributes of the points. The number of points, the locations of the points, and the attributes of the points may change from one frame to another. Each point cloud frame may refer to a set of three-dimensional points specified by zero or more attributes and the Cartesian coordinates (x, y, z) of the three-dimensional points at a particular time instance. Here, the Cartesian coordinates (x, y, z) of the three-dimensional points may be positions or geometries.
[0174] In some embodiments, the present disclosure may also perform a spatial segmentation process of segmenting the point cloud data into one or more 3D blocks before encoding the point cloud data. A 3D block may refer to all or part of a 3D space occupied by the point cloud data. A 3D block may be one or more of a tile group, a tile, a slice, a coding unit (CU), a prediction unit (PU), or a transform unit (TU).
[0175] A tile corresponding to a 3D block may refer to all or part of a 3D space occupied by point cloud data. Moreover, a slice corresponding to a 3D block may refer to all or part of a 3D space occupied by point cloud data. A tile may be divided into one or more slices based on the number of points included in a tile. A tile may be a group of slices with bounding box information. The bounding box information of each tile may be specified in a tile list (or tile parameter set, TPS). A tile may overlap with another tile in a bounding box. A slice may be a data unit that performs encoding independently, or a data unit that performs decoding independently. That is, a slice may be a collection of points that may be encoded or decoded independently. In some embodiments, a slice may be a series of syntax elements representing part or all of an encoded point cloud frame. Each slice may include an index for identifying the tile to which the slice belongs.
[0176] The spatially divided 3D blocks can be processed independently or non-independently. For example, the spatially divided 3D blocks can be encoded or decoded independently or non-independently, and can be sent or received independently or non-independently. In addition, the spatially divided 3D blocks can be quantized or dequantized independently or non-independently, and can be transformed or inversely transformed independently or non-independently. In addition, the spatially divided 3D blocks can be rendered independently or non-independently. For example, encoding or decoding can be performed in units of slices or in units of tiles. In addition, quantization or dequantization can be performed differently for each tile or slice, and quantization or dequantization can be performed differently for each transformed or inversely transformed tile or slice.
[0177] In this way, when the point cloud data is spatially segmented into one or more 3D blocks and the spatially segmented 3D blocks are processed independently or non-independently, the process of processing the 3D blocks is performed in real time and with low latency. In addition, random access and parallel encoding or parallel decoding in the three-dimensional space occupied by the point cloud data can be enabled, and errors accumulated during the encoding or decoding process can be prevented.
[0178] When the point cloud data is segmented into one or more 3D blocks, information may be needed to decode some point cloud data corresponding to a specific tile or a specific slice in the point cloud data. In addition, in order to support spatial access (or partial access) to the point cloud data, information related to the 3D spatial area may be required. Here, spatial access may refer to extracting only the necessary part of the point cloud data from the entire point cloud data from the file. The signaling information may include information for decoding some point cloud data, information related to the 3D spatial area used to support spatial access, etc. For example, the signaling information may include 3D bounding box information, 3D spatial area information, tile information and / or tile list information.
[0179] The signaling information may be stored and signaled in a sample in a track, a sample entry, a sample group, a track group, or a separate metadata track. In some embodiments, the signaling information may be signaled in units of a sequence parameter set (SPS) for signaling a sequence level, a geometry parameter set (GPS) for signaling geometry coding information, and an attribute parameter set (APS) for signaling attribute coding information, a tile parameter set (TPS) (or tile list) for signaling a tile level, etc. In addition, the signaling information may be signaled in units of coding units such as slices or tiles.
[0180] Sample Group
[0181] The encapsulation processing unit mentioned in the present disclosure may generate a sample group by grouping one or more samples. The encapsulation processing unit, metadata processing unit, or signaling processing unit mentioned in the present disclosure may signal signaling information associated with the sample group in the sample, sample group, or sample entry. That is, the sample group information associated with the sample group may be added to the sample, sample group, or sample entry. The sample group information may be 3D bounding box sample group information, 3D region sample group information, 3D tile sample group information, 3D tile list sample group information, etc.
[0182] Track Group
[0183] The encapsulation processing unit mentioned in the present disclosure may generate a track group by grouping one or more tracks. The encapsulation processing unit, metadata processing unit, or signaling processing unit mentioned in the present disclosure may signal signaling information associated with the track group in a sample, a track group, or a sample entry. That is, track group information associated with the track group may be added to the sample, the track group, or the sample entry. The track group information may be 3D bounding box track group information, point cloud composition track group information, spatial region track group information, 3D tile track group information, 3D tile list track group information, etc.
[0184] Sample entry
[0185] Fig.12 is a diagram for explaining an ISOBMFF-based file including a single track. Fig.12 (a) illustrates an example of a layout of an ISOBMFF-based file including a single track, and Fig.12 (b) illustrates an example of a sample structure of the mdat box when the G-PCC bitstream is stored in a single track of a file. Fig.13 is a diagram for explaining an ISOBMFF-based file including a plurality of tracks. Fig.13 (a) illustrates an example of a layout of an ISOBMFF-based file including a plurality of tracks, and Fig.13 (b) illustrates an example of a sample structure of the mdat box when the G-PCC bitstream is stored in a single track of a file.
[0186] The stsd box (SampleDescriptionBox) included in the moov box of the file can include sample entries for a single track for storing a G-PCC bitstream. SPS, GPS, APS, and tile lists can be included in the sample entries in the moov box in the file or in the samples in the mdat box. Moreover, geometric slices and zero or more attribute slices can be included in the samples of the mdat box in the file. When the G-PCC bitstream is stored in a single track of the file, each sample can contain multiple G-PCC components. That is, each sample can be composed of one or more TLV encapsulation structures. The sample entries of a single track can be defined as follows.
[0187] Sample entry types: 'gpe1', 'gpeg'
[0188] Container: SampleDescriptionBox
[0189] Mandatory: 'gpe1' or 'gpeg' Sample entries are mandatory Quantity: One or more sample entries may be present
[0190] The sample entry type 'gpe1' or 'gpeg' is mandatory, and one or more sample entries may exist. The G-PCC track may use VolumetricVisualSampleEntry with sample entry type gpe1' or 'gpeg'. The sample entry of the G-PCC track may include a G-PCC decoder configuration box GPCCConfigurationBox, and the G-PCC decoder configuration box may include a G-PCC decoder configuration record (GPCCDecoderConfigurationRecord()). GPCCDecoderConfigurationRecord() may include at least one of the following items: configurationVersion, profile_idc, profile_compatibility_flags, level_idc, numOfSetupUnitArrays, SetupUnitType, completeness, numOfSepupUnit, or setupUnit. The setupUnit array field included in GPCCDecoderConfigurationRecord() may include a TLV encapsulation structure containing one SPS.
[0191] If the sample entry type is 'gpe1', all parameter sets (e.g., SPS, GPS, APS, tile list) can be included in the array of setupUints. If the sample entry type is 'gpeg', the above parameter sets can be included in the array of setupUint (i.e., sample entry) or included in the stream (i.e., sample). An example of the syntax of a G-PCC sample entry (GPCCSampleEntry) with sample entry type 'gpe1' is as follows.
[0192] aligned(8)class GPCCSampleEntry()
[0193] extends VolumetricVisualSampleEntry('gpe1'){
[0194] GPCCConfigurationBox config; / / Mandatory
[0195] 3DboundingBoxInfoBox();
[0196] CubicRegionInfoBox();
[0197] TileInventoryBox();
[0198] }
[0199] A G-PCC sample entry (GPCCSampleEntry) with sample entry type 'gpe1' may include: GPCCConfigurationBox, 3DBoundingBoxInfoBox(), CubicRegionInfoBox(), and TileInventoryBox(). 3DBoundingBoxInfoBox() may indicate 3D bounding box information of point cloud data associated with samples carried by the track. CubicRegionInfoBox() may indicate information about one or more spatial regions of point cloud data carried by samples in the track. TileInventoryBox() may indicate 3D tile inventory information of point cloud data carried by samples in the track.
[0200] like Fig.12 As illustrated in (b), the sample may include a TLV encapsulation structure including a geometric slice. In addition, the sample may include a TLV encapsulation structure including one or more parameter sets. In addition, the sample may include a TLV encapsulation structure including one or more attribute slices.
[0201] like Fig.13 As illustrated in (a), when the G-PCC bitstream is carried by multiple tracks of an ISOBMFF-based file, each geometry slice or attribute slice can be mapped to a separate track. For example, a geometry slice can be mapped to track 1, and an attribute slice can be mapped to track 2. The track carrying the geometry slice (track 1) can be referred to as a geometry track or a G-PCC geometry track, and the track carrying the attribute slice (track 2) can be referred to as an attribute track or a G-PCC attribute track. In addition, a geometry track can be defined as a volume visual track carrying a geometry slice, and an attribute track can be defined as a volume visual track carrying an attribute slice.
[0202] A track that carries a portion of a G-PCC bitstream that includes both geometry slices and attribute slices may be referred to as a multiplexed track. In the case where geometry slices and attribute slices are stored on separate tracks, each sample in the track may include at least one TLV encapsulation structure that carries data for a single G-PCC component. In this case, each sample contains neither geometry nor attributes, and may not contain multiple attributes. Multi-track encapsulation of a G-PCC bitstream may enable a G-PCC player to effectively access one of the G-PCC components. When a G-PCC bitstream is carried by multiple tracks, the following conditions need to be met in order for a G-PCC player to effectively access one of the G-PCC components.
[0203] When a G-PCC bitstream consisting of a TLV encapsulation structure is carried by multiple tracks, the track carrying the geometry bitstream (or geometry slice) becomes the entry point.
[0204] In the sample entry, a new box is added to indicate the role of the stream included in the track. The new box may be the aforementioned G-PCC component type box (GPCCComponentTypeBox). That is, the GPCCComponentTypeBox may be included in the sample entries of multiple tracks.
[0205] A track reference is introduced from a track carrying only the G-PCC geometry bitstream to a track carrying the G-PCC attribute bitstream.
[0206] GPCCComponentTypeBox may include GPCCComponentTypeStruct(). If GPCCComponentTypeBox is present in a sample entry of a track carrying part or all of a G-PCC bitstream, GPCCComponentTypeStruct() may specify the type (e.g., geometry, attribute) of one or more G-PCC components carried by each track. For example, if the value of the gpcc_type field included in GPCCComponentTypeStruct() is 2, it may indicate a geometry component, and if it is 4, it may indicate an attribute component. In addition, when the value of the gpcc_type field indicates 4 (i.e., attribute component), an AttrIdx field indicating an attribute identifier signaled to SPS() may be further included.
[0207] In the case where the G-PCC bitstream is carried by multiple tracks, the syntax of the sample entry can be defined as follows.
[0208] Sample entry types: 'gpe1', 'gpeg', 'gpc1', or 'gpcg'
[0209] Container: SampleDescriptionBox
[0210] Mandatory: 'gpc1' or 'gpcg' sample entries are mandatory Quantity: One or more sample entries may be present
[0211] The sample entry types 'gpc1', 'gpcg', 'gpc1' or 'gpcg' are mandatory and one or more sample entries may be present. Multiple tracks (e.g., geometry or attribute tracks) may use a VolumetricVisualSampleEntry with the sample entry type 'gpc1', 'gpcg', 'gpc1' or 'gpcg'. In a 'gpe1' sample entry, all parameter sets may be present in the setupUnit array. In a 'gpeg' sample entry, parameter sets are present in an array or stream. In a 'gpe1' or 'gpeg' sample entry, GPCCComponentTypeBox shall not be present. In a 'gpc1' sample entry, SPS, GPS, and tile lists may be present in the SetupUnit array for tracks carrying G-PCC geometry bitstreams. All relevant APSs may be present in the SetupUnit array for tracks carrying G-PCC attribute bitstreams. In a 'gpcg' sample entry, SPS, GPS, APS, or tile lists may be present in an array or stream. In the 'gpc1' or 'gpcg' sample array, the GPCCComponentTypeBox should be present.
[0212] An example of the syntax of a G-PCC sample entry is as follows.
[0213] aligned(8)class GPCCSampleEntry()
[0214] extends VolumetricVisualSampleEntry(codingname){
[0215] GPCCConfigurationBox config; / / Mandatory
[0216] GPCCComponentTypeBoxtype; / / optional
[0217] }
[0218] The compressor name (i.e., coding name) of the base class VolumetricVisualSampleEntry can indicate the name of the compression program to be used with the recommended "W013GPCC coding" value. In "W013GPCC coding", the first byte (octal number 13 or decimal number 11 represented by \013) is the number of remaining bytes and can indicate the number of bytes of the remaining string. Congif can include G-PCC decoder configuration information. Info can indicate the G-PCC component information carried in each track. Info can indicate the component tiles carried in the track and can also indicate the attribute name, index, and attribute type of the G-PCC components carried in the G-PCC attribute track.
[0219] Sample Format
[0220] When the G-PCC bitstream is stored in a single track, the syntax of the sample format is as follows.
[0221] aligned(8) class GPCCSample
[0222] { unsigned int GPCCLength = sample_size; / / The size of the sample
[0223] for(i = 0; i < GPCCLength; ) / / To the end of the sample
[0224] {
[0225] tlv_encapsulation gpcc_unit;
[0226] i += (1 + 4) + gpcc_unit.tlv_num_payload_bytes;
[0227] }
[0228] }
[0229] In the above syntax, each sample (GPCCSample) corresponds to a single point cloud frame and can be composed of one or more TLV encapsulation structures belonging to the same presentation time. Each TLV encapsulation structure can include a single type of TLV payload. Additionally, a sample can be independent (e.g., a synchronous sample). GPCCLength indicates the length of the sample, and gpcc_unit can include an instance of a TLV encapsulation structure containing a single G-PCC component (e.g., a geometry slice).
[0230] When a G-PCC bitstream is stored in multiple tracks, each sample can correspond to a single point cloud frame, and samples contributing to the same point cloud frame in different tracks may have to have the same presentation time. Each sample can be composed of one or more G-PCC units of the G-PCC component indicated in the GPCCComponentInfoBox of the sample entry and zero or more G-PCC units carrying one of the parameter sets or tile lists. When a G-PCC unit including a parameter set or tile list is present in a sample, the F-PCC sample may need to appear before the G-PCC unit of the G-PCC component. Each sample can include one or more G-PCC units containing attribute data units, and zero or more G-PCC units carrying parameter sets. In the case where the G-PCC bitstream is stored in multiple tracks, the syntax and semantics of the sample format can be the same as the syntax and semantics of the above-mentioned case where the G-PCC bitstream is stored in a single track.
[0231] Subsample
[0232] In the receiving device, since the geometric slice is decoded first, and the attribute slice needs to be decoded based on the decoded geometry, when each sample is composed of multiple TLV encapsulation structures, it is necessary to access each TLV encapsulation structure in the sample. In addition, if a sample is composed of multiple TLV encapsulation structures, each TLV encapsulation structure in the multiple TLV encapsulation structures can be stored as a sub-sample. The sub-sample can be called a G-PCC sub-sample. For example, if a sample includes a parameter set TLV encapsulation structure containing a parameter set, a geometry TLV encapsulation structure containing a geometric slice, and an attribute TLV encapsulation structure containing an attribute slice, the parameter set TLV encapsulation structure, the geometry TLV encapsulation structure, and the attribute TLV encapsulation structure can be stored as sub-samples respectively. In this case, in order to be able to access each G-PCC component in the sample, the type of TLV encapsulation structure carried by the sub-sample may be required.
[0233] When the G-PCC bitstream is stored in a single track, the G-PCC subsample may include only one TLV encapsulation structure. One SubSampleInformationBox may exist in the sample table box (SampleTableBox, stbl) of the moov box, or may exist in the track fragment box (TrackFragmentBox, traf) of each movie fragment box (MovieFragmentBox, moof). If a SubSampleInformationBox exists, the 8-bit type value of the TLV encapsulation structure may be included in the 32-bit codec_specific_parameters field of the subsample entry in the SubSampleInformationBox. If the TLV encapsulation structure includes an attribute payload, the 6-bit value of the attribute index may be included in the 32-bit codec_specific_parameters field of the subsample entry in the SubSampleInformationBox. In some embodiments, the type of each subsample may be identified by parsing the codec_specific_parameters field of the subsample entry in the SubSampleInformationBox. The codec_specific_parameters of the SubSampleInformationBox may be defined as follows.
[0234] if (flags == 0) {
[0235] unsigned int(8)PayloadType;
[0236] if(PayloadType==4){ / / property payload
[0237] unsigned int(6)AttrIdx;
[0238] bit(18)reserved=0;
[0239] }
[0240] else
[0241] bit(24)reserved=0;
[0242] }else if(flags == 1){
[0243] unsigned int(1)tile_data;
[0244] bit(7)reserved=0;
[0245] if(tile_data)
[0246] unsigned int(24)tile_id;
[0247] else
[0248] bit(24)reserved=0;
[0249] }
[0250] In the above subsample syntax, payloadType may indicate the tlv_type of the TLV encapsulation structure in the subsample. For example, if the value of payloadType is 4, an attribute slice (i.e., attribute slice) may be indicated. attrIdx may indicate an identifier of the attribute information of the TLV encapsulation structure including the attribute payload in the subsample. attrIdx may be the same as the ash_attr_sps_attr_idx of the TLV encapsulation structure including the attribute payload in the subsample. tile_data may indicate whether the subsample includes one tile or another tile. When the value of tile_data is 1, it may indicate that the subsample includes a TLV encapsulation structure containing a geometry data unit or an attribute data unit corresponding to a G-PCC tile. When the value of tile_data is 0, it may indicate that the subsample includes a TLV encapsulation structure containing various parameter sets, tile lists, or frame boundary markers. tile_id may indicate the index of the G-PCC tile associated with the subsample in the tile list.
[0251] When the G-PCC bitstream is stored in multiple tracks (in the case of multi-track encapsulation of G-PCC data in ISOBMFF), if there are subsamples, only the SubSampleInformationBox with a flag of 1 may need to be present in the SampleTableBox or TrackFragmentBox of each moviefragmentbox. In the case where the G-PCC bitstream is stored in multiple tracks, the syntax elements and semantics may be the same as the case where the flag in the syntax elements and semantics is == 1 when the G-PCC bitstream is stored in a single track.
[0252] References between tracks
[0253] When G-PCC bitstreams are carried in multiple tracks (i.e., when G-PCC geometry bitstream and attribute bitstream are carried in different (separate) tracks), in order to connect between tracks, a track reference tool can be used. A TrackReferenceTypeBoxes can be added to the TrackReferenceBox in the TrackBox of the G-PCC track. The TrackReferenceTypeBox can contain an array of track_IDs that specify the tracks that the G-PCC track references.
[0254] In some embodiments, the present disclosure may provide an apparatus and method for supporting temporal scalability in carrying G-PCC data (hereinafter, may be referred to as a G-PCC bitstream, an encapsulated G-PCC bitstream, or a G-PCC file). In addition, the present disclosure may propose an apparatus and method for providing a point cloud content service, which efficiently stores the G-PCC bitstream in a single track in a file, or stores it separately in multiple tracks, and provides signaling for it. In addition, the present disclosure proposes an apparatus and method for processing file storage technology to support efficient access to stored G-PCC bitstreams.
[0255] Time Scalability
[0256] Temporal scalability may refer to a function that allows the possibility of extracting one or more subsets of independently encoded frames. Moreover, temporal scalability may refer to a function that divides G-PCC data into multiple different temporal levels and independently processes each G-PCC frame belonging to a different temporal level. If temporal scalability is supported, a G-PCC player (or a transmitting device and / or receiving device of the present disclosure) can effectively access a desired component (target component) among the G-PCC components. In addition, if temporal scalability is supported, since the G-PCC frames are processed independently of each other, the temporal scalability support at the system level can be expressed as a more flexible temporal sub-layering. In addition, if temporal scalability is supported, a system (point cloud content providing system) that processes G-PCC data can manipulate the data at a high level to match the network capability or decoder capability, thereby improving the performance of the point cloud content providing system.
[0257] Implementation
[0258] When temporal scalability is supported, G-PCC content can be carried in multiple tracks. In other words, to support temporal scalability of a G-PCC file, a G-PCC bitstream can be stored in one or more temporal level tracks. When temporal scalability is available or used, a G-PCC scalability information box can be present in the sample entry of the track, and information about temporal scalability can be signaled.
[0259] As an example, a box present in a track or a tile base track and a box present in a tile track, that is, a box for temporal scalability information (hereinafter referred to as a "temporal scalability information box" or "scalability information box"), may be used to carry temporal scalability information. The box carrying temporal scalability information present in a GPCC track or a tile base track may be a GPCCScalabilityInfoBox, and the box present in a tile track may be a GPCCTileScalabilityInfoBox. The GPCCTileScalabilityInfoBox may be present in each tile track associated with the tile base track in which the GPCCScalabilityInfoBox is present.
[0260] Currently, when each frame is signaled at a single temporal level, the G-PCC scalability information box is specified to not be present in tracks with sample entries 'gpe1', 'gpeg', 'gpc1', 'gpcg', 'gpcb', and 'gpeb'. Such a constraint is intended to explain the presence of a box indicating whether temporal scalability is available or used, and at this time, when one or more level tracks are removed and only one track in which each sample belongs to a single temporal level is left in the file, additional effort may be required to remove the box in the file. In addition, multiple_temporal_level_tracks_flag (which is a type of information about temporal scalability) may be included in the G-PCC scalability information box. Here, if the flag has a first value (e.g., 1), it indicates that there are multiple temporal level tracks for the G-PCC bitstream, and if the flag has a second value (e.g., 0), it indicates that there is only one temporal level track for the G-PCC bitstream. Also in this case, when one or more level tracks are removed and only one track is left in the file where every sample belongs to a single temporal level, additional effort may be required to remove boxes in the file.
[0261] In order to solve the above-mentioned problem, the present disclosure proposes the meaning of a value of information about multiple temporal level tracks (eg, multiple_temporal_level_tracks_flag) based on a sample entry type, which may be included in the temporal scalability information.
[0262] The embodiments of the present disclosure may be applied individually or in combination thereof.
[0263] Meanwhile, the meanings of terms used to describe the embodiments of the present disclosure are as follows.
[0264] Point cloud frame: A set of 3D points specified by Cartesian coordinates (x, y, z), and optionally a fixed set of corresponding attributes at a specific time instance
[0265] Bounding box: A rectangular cube that includes the source point cloud frame
[0266] Geometry: A set of Cartesian coordinates associated with the point cloud frame
[0267] Attributes: scalar or vector attributes that are selectively associated with each point in the point cloud, such as color, reflectivity, frame, and frame index
[0268] APS: Attribute Parameter Set
[0269] ASH: Attribute Slice Header
[0270] GSH: Geometry Slicing Head
[0271] GPS: Geometry parameter set
[0272] LSB: Least Significant Bit.
[0273] RAHT: Region Adaptive Hierarchical Transform
[0274] SPS: Sequence Parameter Set
[0275] TPS: Tile Parameter Set, same as Tile Manifest Slice: A sequence of syntax elements indicating part or all of a coded point cloud frame 3D tile: A rectangular cube within a bounding box
[0276] The technology proposed in the present disclosure is described in detail below in conjunction with the implementation modes.
[0277] According to an embodiment of the present disclosure, the G-PCC scalability information box may be configured as follows.
[0278] [Table 1]
[0279]
[0280]
[0281] The meaning of the syntax element multiple_temporal_level_tracks_flag (e.g., first information or a first syntax element) may be determined based on the sample entry type, and the syntax element may be included in the information about multiple temporal level tracks. In addition, the value of the first information may be limited to a predetermined value or may be determined as a predetermined value based on a specific condition. As an example, if a track carrying temporal scalability information (e.g., GPCCScalabilityInfoBox) capable of having information about multiple temporal level tracks has a first type of sample entry (e.g., 'gpe1', 'gpeg', 'gpc1', or 'gpcg') and the value of the first information (e.g., multiple_temporal_level_tracks_flag) indicating whether there are multiple temporal level tracks in the file is a first value (e.g., 0), it may mean that there are no other temporal level tracks for the G-PCC bitstream. As another example, if a track carrying temporal scalability information (e.g., GPCCScalabilityInfoBox) capable of having information about multiple temporal level tracks has a first type of sample entry (e.g., 'gpe1', 'gpeg', 'gpc1', or 'gpcg') and a value of first information (e.g., multiple_temporal_level_tracks_flag) indicating whether there are multiple temporal level tracks in the file is a second value (e.g., 1), it may mean that another temporal level track for the G-PCC bitstream may exist. In an embodiment, if a G-PCC scalability information box exists in a track having a first type of sample entry, each sample belongs to a single temporal level, and a value of the first information (e.g., multiple_temporal_level_tracks_flag) is a second value (e.g., 1), it may be possible to additionally describe / specify that there was originally another(s) temporal level track(s) being removed / discarded.
[0282] Meanwhile, as another example, if a track carrying temporal scalability information (e.g., GPCCScalabilityInfoBox) capable of having information about multiple temporal level tracks has a second type of sample entry (e.g., 'gpcb' or 'gpeb') and a value of the first information (e.g., multiple_temporal_level_tracks_flag) is a first value (e.g., 0), it may indicate that each tile track referenced by the tile base track includes samples of each temporal level included in the G-PCC bitstream. As another example, if a track carrying temporal scalability information (e.g., GPCCScalabilityInfoBox) capable of having information about multiple temporal level tracks has a second type of sample entry and a value of the first information (e.g., multiple_temporal_level_tracks_flag) is a second value (e.g., 1), it may indicate that there may be one or more temporal tile tracks that do not include samples of each temporal level included in the G-PCC bitstream. In an embodiment, if the G-PCC scalability information box is present in a track having sample entries of the second type, each sample belonging to a single temporal level, and the value of the first information (e.g., multiple_temporal_level_tracks_flag) is a second value (e.g., 1), it may be additionally described / specified that there was originally another tile track having a temporal level greater than 0 that was removed / discarded.
[0283] The syntax element frame_rate_present_flag may indicate whether average frame rate information is present. A first value (eg, 1) of frame_rate_present_flag may indicate the presence of average frame rate information, and a second value (eg, 0) of frame_rate_present_flag may indicate the absence of average frame rate information.
[0284] The syntax element num_temporal_levels may indicate the number of temporal levels present in the sample of each track. In case the sample entry type is 'gpcb' or 'gpeb', num_temporal_levels may indicate the maximum number of temporal levels into which the G-PCC frame is to be grouped, and the minimum value may be 1.
[0285] The syntax element temporal_level_id may indicate the temporal level identifier of the G-PCC samples of the individual track. The value of the syntax element shall be incremented by 1, and if there is a temporal level with temporal identifier (id) x, the temporal id of the next temporal level shall be x+1. Additionally, if track B (trackB) is the next temporal level track of another track A (trackA), track B shall include samples with a temporal id equal to the value obtained by adding 1 to the highest temporal id of track A.
[0286] The syntax element frame_rate may indicate an average frame rate at a temporal level in frame units (frames / 256 seconds). If the value of frame_rate is 0, it may indicate an unspecified average frame rate.
[0287] According to the above-mentioned embodiments of the present disclosure, information about multiple time level tracks (which can be included in the temporal scalability information that can be carried by the temporal scalability information box) can be specified according to the sample entry type of the track (for example, a track carrying information about multiple time level tracks), so that the signaling overhead can be reduced and the image encoding / decoding efficiency can be improved.
[0288] Encoding and decoding process
[0289] Fig.14 is an example of a method performed by a receiving device of point cloud data, and Fig.15 is an example of a method performed by a transmitting device of point cloud data. As an example, the receiving device or the transmitting device may include the contents described in the present disclosure with reference to the accompanying drawings, and may be the same as the receiving device or the transmitting device assumed for describing the above-mentioned embodiments. That is, performing Fig.14 The receiving device and execution Fig.15 The sending device can clearly execute the other embodiments mentioned above.
[0290] As an example, refer to Fig.14 , the receiving device may acquire (i.e., obtain) temporal scalability information of a point cloud in a 3D space based on a G-PCC file (S1401). The G-PCC file may be acquired (i.e., obtained) by being sent from a transmitting device. Next, the receiving device may reconstruct a 3D point cloud based on the temporal scalability information (S1402), and may acquire (i.e., obtain) the temporal scalability information from one or more tracks, and the one or more tracks may include a first track, ..., an nth track.
[0291] As another example, refer to Fig.15, the transmitting device may determine whether temporal scalability is applied to point cloud data of a 3D space (S1501), and generate a G-PCC file by including temporal scalability information and point cloud data (S1502). In this article, the temporal scalability information may be carried by one or more tracks, and as described above, the one or more tracks may include a first track, ..., an nth track.
[0292] As an example, the temporal scalability information may include first information about a temporal level track of a G-PCC file, and the first information may specify constraints on the temporal level track based on a sample entry type. As an example, the first information may include the above-mentioned multiple_temporal_level_tracks_flag. In addition, when the sample entry type is of the first type, a first value (e.g., 0) of the first information may indicate that another temporal level track does not exist. When the sample entry type is of the first type, a second value (e.g., 1) of the first information may indicate that another temporal level track may exist. As an example, the second value of the first information may also indicate that there are one or more other temporal level tracks that are removed based on each sample belonging to a single temporal level. As another example, when the sample entry type is of the second type, a first value (e.g., 0) of the first information may indicate that each tile track referenced by the tile base track includes samples of each temporal level. In addition, when the sample entry type is of the second type, a second value (e.g., 1) of the first information may indicate that there may be one or more temporal tile tracks that do not include samples of each temporal level. As an example, the second value of the first information may further indicate, based on each sample belonging to a single temporal level, that there are one or more other tile tracks that are removed and whose temporal level is greater than 0. As an example, the sample entry type may include a first type and a second type, the first type including 'gpe1', 'gpeg', 'gpc1', and 'gpcg', and the second type including 'gpcb' and 'gpeb'.
[0293] Although not shown, a receiving device for point cloud data may include a memory and at least one processor, and the at least one processor may acquire (i.e., obtain) a geometry-based point cloud compression (G-PCC) file including the point cloud data, and reconstruct the point cloud based on temporal scalability information, wherein the temporal scalability information may include first information about a temporal level track of the G-PCC file, and the first information may specify constraints on the temporal level track based on a sample entry type.
[0294] Although not shown, the point cloud data transmission device may include a memory
[0295] and at least one processor, and the at least one processor can determine whether temporal scalability is applied to point cloud data in 3D space, and generate a G-PCC file by including temporal scalability information and the point cloud data, wherein the temporal scalability information can include first information about multiple temporal level tracks of the file, and the first information can specify constraints on the temporal level tracks based on the sample entry type.
[0296] According to the embodiments of the present disclosure, since the temporal scalability information that can be signaled through a bitstream will be information having various different meanings according to the sample entry type, image encoding / decoding efficiency can be improved.
[0297] The scope of the present disclosure includes software or machine-executable instructions (e.g., operating systems, applications, firmware, programs, etc.) that enable operations according to the methods of various embodiments to be performed on a device or computer, as well as non-transitory computer-readable media in which such software, instructions, etc. are stored and can be executed on a device or computer.
[0298] Industrial Applicability
[0299] The embodiments according to the present disclosure may be used to provide point cloud content. Moreover, the embodiments according to the present disclosure may be used to encode / decode point cloud data.
Claims
1. A method performed by a receiving device for point cloud data, the method comprising: Obtaining a geometry-based point cloud compression G-PCC file including the point cloud data; as well as Reconstructing point clouds based on temporal scalability information, wherein the temporal scalability information comprises first information about a temporal level track of the G-PCC file, and The first information specifies constraints on the time level track based on a sample entry type.
2. The method according to claim 1, wherein: Based on the sample entry type being the first type, the first value of the first information indicates that no other time level tracks exist.
3. The method according to claim 1, wherein: Based on the sample entry type being the first type, the second value of the first information indicates that another time level track can exist.
4. The method according to claim 3, wherein: Further based on that each sample belongs to a single temporal level, the second value of the first information further indicates that there is another temporal level track being removed.
5. The method according to claim 1, wherein: Based on the sample entry type being the second type, the first value of the first information indicates that each tile track referenced by the tile base track includes samples at each temporal level.
6. The method according to claim 1, wherein: Based on the sample entry type being the second type, the second value of the first information indicates that there can be one or more temporal tile tracks that do not include samples of each temporal level.
7. The method according to claim 6, wherein: Further based on that each sample belongs to a single temporal level, the second value of the first information further indicates that there is another tile track whose temporal level is greater than 0 and is removed.
8. The method according to claim 1, wherein: The sample entry types include a first type including 'gpe1', 'gpeg', 'gpc1', and 'gpcg', and a second type including 'gpcb' and 'gpeb'.
9. A method performed by a point cloud data transmitting device, the method comprising: Determining whether temporal scalability is applied to point cloud data in three-dimensional 3D space; as well as generating a G-PCC file by including the temporal scalability information and the point cloud data, wherein the temporal scalability information comprises first information about a temporal level track of the file, and The first information specifies constraints on the time level track based on a sample entry type.
10. A device for receiving point cloud data, the device comprising: Memory; as well as at least one processor, Wherein, the at least one processor is configured to: Obtaining a geometry-based point cloud compression G-PCC file including the point cloud data; and Reconstructing point clouds based on temporal scalability information, wherein the temporal scalability information comprises first information about a temporal level track of the G-PCC file, and The first information specifies constraints on the time level track based on a sample entry type.
11. A device for sending point cloud data, the device comprising: Memory; as well as at least one processor, Wherein, the at least one processor is configured to: determining whether temporal scalability is applied to point cloud data in three-dimensional 3D space; and generating a G-PCC file by including the temporal scalability information and the point cloud data, Wherein the temporal scalability information comprises first information about a temporal level track of the file, and wherein the first information specifies constraints on the temporal level track based on a sample entry type.
