Point cloud data transmitting device, point cloud data transmitting method, point cloud data receiving device, and point cloud data receiving method

The method and apparatus address the inefficiencies in processing point cloud data by employing geometry-based and video-based compression techniques, optimizing encoding and decoding for VR and autonomous driving services through user-interaction-based data processing.

JP2024540714APending Publication Date: 2024-11-01LG ELECTRONICS INC

Patent Information

Application Number
JP2024525096
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2021-10-27
Filing Date
2022-10-26
Publication Date
2024-11-01

AI Technical Summary

Technical Problem

Existing technologies face challenges in efficiently processing large volumes of point cloud data required for VR, AR, MR, and autonomous driving services due to high latency and encoding/decoding complexity.

Method used

A method and apparatus for encoding and decoding point cloud data using geometry-based and video-based compression techniques, including geometry and attribute bitstreams, with multiplexing and decoding based on feedback information to optimize data processing.

Benefits of technology

The solution enables high-efficiency processing of point cloud data, providing high-quality services for VR and autonomous driving by efficiently encoding and decoding only necessary data based on user interaction, reducing latency and complexity.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024540714000001_ABST
    Figure 2024540714000001_ABST
Patent Text Reader

Abstract

A method for transmitting point cloud data according to the present invention includes the steps of: encoding point cloud data; and transmitting a bitstream including the point cloud data. A method for receiving point cloud data according to an embodiment includes the steps of receiving a bitstream including the point cloud data; and decoding the point cloud data.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The embodiments relate to a method and apparatus for processing Point Cloud Content. [Background technology]

[0002] Point cloud content is content represented by a point cloud, which is a collection of points belonging to a coordinate system that represents a three-dimensional space. Point cloud content can represent three-dimensional media and is used to provide various services such as VR (Virtual Reality), AR (Augmented Reality), MR (Mixed Reality) and autonomous driving services. However, to represent point cloud content, tens of thousands to hundreds of thousands of point data are required. Therefore, a method for efficiently processing a huge amount of point data is required. Summary of the Invention [Problem to be solved by the invention]

[0003] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The embodiments provide an apparatus and method for efficiently processing point cloud data.The embodiments provide a method and apparatus for processing point cloud data to address latency and encoding / decoding complexity.

[0004] However, the scope of the invention is not limited to the above-mentioned technical problems, and may be extended to other technical problems that a person skilled in the art can derive based on all the contents described herein. [Means for solving the problem]

[0005] A method for transmitting point cloud data according to an embodiment includes (comprises; configures; constructs; sets; includes; contains; contains) a step of encoding the point cloud data; and a step of transmitting a bitstream including the point cloud data. A method for receiving point cloud data according to an embodiment includes (comprises; configures; constructs; sets; includes; contains; contains) a step of receiving a bitstream including the point cloud data; and a step of decoding the point cloud data. Effect of the Invention

[0006] The apparatus and method according to the embodiment can process point cloud data with high efficiency.

[0007] The apparatus and method according to the embodiment can provide a high-quality point cloud service.

[0008] The apparatus and method according to the embodiment can provide point cloud content for providing general-purpose services such as VR services and autonomous driving services. [Brief description of the drawings]

[0009] The accompanying drawings are provided to aid in the understanding of the embodiments and, together with the associated description of the embodiments, illustrate the embodiments. For a better understanding of the various embodiments described below, reference should be made to the following description of the embodiments in conjunction with the following drawings, in which like reference numerals correspond to corresponding parts, and in which: [Figure 1] 1 illustrates an example of a point cloud content providing system according to an embodiment. [Diagram 2] FIG. 2 is a block diagram illustrating a point cloud content providing operation according to an embodiment. [Diagram 3] 1 illustrates an example of a point cloud video capture process according to an embodiment. [Figure 4] 1 illustrates an example of a Point Cloud Encoder according to an embodiment. [Diagram 5] 1 illustrates an example of a voxel according to an embodiment. [Figure 6] 1 illustrates an example of an octree and occupancy code according to an embodiment. [Figure 7] 1 illustrates an example of an adjacent node pattern according to an embodiment. [Figure 8] 13 illustrates an example of a point configuration for each LOD according to an embodiment. [Figure 9] 13 illustrates an example of a point configuration for each LOD according to an embodiment. [Figure 10] 1 illustrates an example of a Point Cloud Decoder according to an embodiment. [Figure 11] 1 illustrates an example of a Point Cloud Decoder according to an embodiment. [Figure 12] 1 illustrates an example of a point cloud data transmission device according to an embodiment. [Figure 13] 1 illustrates an example of a point cloud data receiving device according to an embodiment. [Figure 14] 1 illustrates an example of a structure that can be linked to a point cloud data transmission / reception method / apparatus according to an embodiment. [Figure 15] 1 illustrates integrated prediction tree generation according to an embodiment. [Figure 16] 1 illustrates a point cloud data transmission device according to an embodiment. [Figure 17] 1 shows a point cloud data receiving device according to an embodiment. [Figure 18] 1 illustrates a bitstream including point cloud data according to an embodiment. [Figure 19] 4 illustrates a sequence parameter set according to an embodiment. [Figure 20] 4 illustrates a set of geometry parameters according to an embodiment. [Figure 21] 4 illustrates a tile parameter set according to an embodiment. [Figure 22] 1 illustrates a geometry slice header according to an embodiment. [Figure 23] 1 illustrates geometry prediction tree data according to an embodiment. [Figure 24] 1 illustrates a method for transmitting point cloud data according to an embodiment. [Diagram 25] 1 illustrates a method for receiving point cloud data according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Preferred embodiments will be described in detail with reference to the accompanying drawings. The following detailed description with reference to the accompanying drawings is intended to illustrate preferred embodiments rather than to illustrate only embodiments that may be implemented by the embodiments. The following detailed description includes details to provide a thorough understanding of the embodiments. However, it will be apparent to those skilled in the art that the embodiments may be practiced without such details.

[0011] Most of the terms used in the examples are common and widely used in the relevant field, but some are arbitrarily selected by the applicant, and their meanings will be explained in detail below as necessary. Therefore, the examples should be understood based on the intended meaning of the terms, rather than the simple names and meanings of the terms.

[0012] FIG. 1 is a diagram illustrating an example of a point cloud content providing system according to an embodiment.

[0013] The point cloud content providing system shown in Fig. 1 includes a transmission device 10000 and a reception device 10004. The transmission device 10000 and the reception device 10004 are capable of wired or wireless communication to transmit and receive point cloud data.

[0014] The transmitting device 10000 according to the embodiment secures, processes, and transmits a point cloud video (or point cloud content). In the embodiment, the transmitting device 10000 includes a fixed station, a base transceiver system (BTS), a network, an AI (Artificial Intelligence) device and / or system, a robot, an AR / VR / XR device and / or server, etc. In the embodiment, the transmitting device 10000 includes a device that communicates with a base station and / or other wireless devices using a wireless connection technology (e.g., 5G NR (New RAT), LTE (Long Term Evolution)), a robot, a vehicle, an AR / VR / XR device, a mobile device, a home appliance, an IoT (Internet of Things) device, an AI device / server, etc.

[0015] The transmitting device 10000 according to the embodiment includes a Point Cloud Video Acquisition unit 10001, a Point Cloud Video Encoder 10002, and / or a Transmitter (or a communication module) 10003.

[0016] The point cloud video acquisition unit 10001 according to the embodiment acquires a point cloud video through a process such as capture, synthesis, or generation. The point cloud video is a point cloud content represented by a point cloud, which is a collection of points located in a three-dimensional space, and is also called point cloud video data. The point cloud video according to the embodiment includes one or more frames. One frame represents a still image / picture. Therefore, the point cloud video includes a point cloud image / frame / picture, and is called any of a point cloud image, a frame, and a picture.

[0017] The point cloud video encoder 10002 according to the embodiment encodes the acquired point cloud video data. The point cloud video encoder 10002 encodes the point cloud video data based on point cloud compression coding. The point cloud compression coding according to the embodiment includes Geometry-based Point Cloud Compression (G-PCC) coding and / or Video based Point Cloud Compression (V-PCC) coding or next generation coding. It should be noted that the point cloud compression coding according to the embodiment is not limited to the above-mentioned embodiment. The point cloud video encoder 10002 outputs a bitstream including encoded point cloud video data. The bitstream includes not only the encoded point cloud video data but also signaling information related to the encoding of the point cloud video data.

[0018] The transmitter 10003 according to the embodiment transmits a bitstream including encoded point cloud video data. The bitstream according to the embodiment is encapsulated into a file or a segment (e.g., a streaming segment) and transmitted via various networks such as a broadcast network and / or a broadband network. Although not shown, the transmitter 10000 includes an encapsulation unit (or an encapsulation module) that performs an encapsulation operation. In the embodiment, the encapsulation unit is included in the transmitter 10003. In the embodiment, the file or segment is transmitted to the receiving device 10004 via a network or stored in a digital storage medium (e.g., USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc.). The transmitter 10003 according to the embodiment can perform wired or wireless communication with the receiving device 10004 (or a receiver 10005) via a network such as 4G, 5G, or 6G. The transmitter 10003 also performs a data processing operation required by a network system (e.g., a communication network system such as 4G, 5G, or 6G). The transmitting device 10000 can also transmit encapsulated data by an on-demand method.

[0019] The receiving device 10004 according to the embodiment includes a receiver 10005, a point cloud video decoder 10006, and / or a renderer 10007. In the embodiment, the receiving device 10004 includes a device, a robot, a vehicle, an AR / VR / XR device, a mobile device, a home appliance, an IoT (Internet of Things) device, an AI device / server, etc., that communicates with a base station and / or other wireless devices using a wireless connection technology (e.g., 5G NR (New RAT), LTE (Long Term Evolution)).

[0020] The receiver 10005 according to the embodiment receives a bitstream including point cloud video data or a file / segment in which the bitstream is encapsulated from a network or storage medium. The receiver 10005 performs a data processing operation required by a network system (e.g., a communication network system such as 4G, 5G, 6G, etc.). The receiver 10005 according to the embodiment decapsulates the received file / segment and outputs a bitstream. In the embodiment, the receiver 10005 also includes a decapsulation unit (or a decapsulation module) for performing the decapsulation operation. The decapsulation unit is also embodied as an element (or component) separate from the receiver 10005.

[0021] The point cloud video decoder 10006 decodes a bitstream including point cloud video data. The point cloud video decoder 10006 can decode the point cloud video data in the manner in which it was encoded (e.g., the reverse process of the operation of the point cloud video encoder 10002). Thus, the point cloud video decoder 10006 can decode the point cloud video data by performing point cloud reconstruction coding, which is the reverse process of point cloud compression. Point cloud reconstruction coding includes G-PCC coding.

[0022] The renderer 10007 renders the decoded point cloud video data. The renderer 10007 renders not only the point cloud video data but also audio data to output point cloud content. In an embodiment, the renderer 10007 includes a display for displaying the point cloud content. In an embodiment, the display is not included in the renderer 10007 and is embodied by a separate device or component.

[0023] In the drawing, the dotted arrow indicates a transmission path of feedback information obtained by the receiving device 10004. The feedback information is information for reflecting interaction with a user consuming point cloud content, and includes user information (e.g., head orientation information, viewport information, etc.). In particular, when the point cloud content is for a service requiring interaction with a user (e.g., an autonomous driving service, etc.), the feedback information may be transmitted to a content transmitting side (e.g., the transmitting device 10000) and / or a service provider. In an embodiment, the feedback information may be used not only by the transmitting device 10000 but also by the receiving device 10004, or may not be provided.

[0024] According to an embodiment, the head orientation information is information regarding the position, direction, angle, movement, etc. of the user's head. According to an embodiment, the receiving device 10004 calculates viewport information based on the head orientation information. The viewport information is information regarding the area of ​​the point cloud video that the user is looking at. The viewpoint is the point at which the user is looking at the point cloud video, and means the center of the viewport area. That is, the viewport is an area centered on the viewpoint, and the size, shape, etc. of the area are determined by the FOV (Field Of View). Therefore, the receiving device 10004 can extract viewport information based on the vertical or horizontal FOV supported by the device in addition to the head orientation information. In addition, the receiving device 10004 performs gaze analysis, etc. to confirm the user's point cloud consumption manner, the point cloud video area that the user is gazing at, the gaze time, etc. In an embodiment, the receiving device 10004 transmits feedback information including the result of the gaze analysis to the transmitting device 10000. According to an embodiment, the feedback information is obtained during the rendering and / or display process. According to an embodiment, the feedback information is obtained by one or more sensors included in the receiving device 10004. Also, in an embodiment, the feedback information is obtained by the renderer 10007 or another external element (or device, component, etc.). The dotted lines shown in FIG. 1 indicate the process of transmitting the feedback information obtained by the renderer 10007. The point cloud content providing system processes (encodes / decodes) point cloud data based on the feedback information. Thus, the point cloud video data decoder 10006 can perform a decoding operation based on the feedback information. Also, the receiving device 10004 can transmit the feedback information to the transmitting device 10000. The transmitting device 10000 (or the point cloud video data encoder 10002) performs an encoding operation based on the feedback information.Therefore, the point cloud content providing system does not process (encode / decode) all point cloud data, but efficiently processes necessary data (e.g., point cloud data corresponding to the user's head position) based on feedback information, and can provide point cloud content to the user.

[0025] In an embodiment, the sending device 10000 may be referred to as an encoder, a sending device, a transmitter, etc., and the receiving device 10004 may be referred to as a decoder, a receiving device, a receiver, etc.

[0026] The point cloud data processed (processed in a series of steps of acquisition / encoding / transmission / decoding / rendering) in the point cloud content providing system of Fig. 1 according to the embodiment is also called point cloud content data or point cloud video data. In the embodiment, the point cloud content data is used as a concept including metadata or signaling information related to the point cloud data.

[0027] The elements of the point cloud content providing system illustrated in FIG. 1 may be implemented in hardware, software, a processor, and / or a combination thereof.

[0028] FIG. 2 is a block diagram illustrating operations for providing point cloud content according to an embodiment.

[0029] Figure 2 is a block diagram showing the operation of the point cloud content providing system described in Figure 1. As described above, the point cloud content providing system processes point cloud data based on point cloud compression coding (eg, G-PCC).

[0030] In a point cloud content providing system (e.g., a point cloud transmitting device 10000 or a point cloud video acquiring unit 10001) according to an embodiment, a point cloud video is acquired (20000). The point cloud video is represented by a point cloud belonging to a coordinate system representing a three-dimensional space. The point cloud video according to an embodiment includes a Ply (Polygon File format or the Stanford Triangle format) file. When the point cloud video has one or more frames, the acquired point cloud video includes one or more Ply files. A Ply file includes point cloud data such as the geometry and / or attributes of a point. The geometry includes the position of the point. The position of each point is represented by parameters (e.g., values ​​of each of the X-axis, Y-axis, and Z-axis) indicating a three-dimensional coordinate system (e.g., a coordinate system consisting of XYZ axes). The attributes include the attributes of the point (e.g., texture information of each point, hue (YCbCr or RGB), reflectance (r), transparency, etc.). One point has one or more attributes (or attributes). For example, one point may have one attribute of hue, or two attributes of hue and reflectance. In the embodiment, geometry may be referred to as position, geometry information, geometry data, etc., and attributes may be referred to as attributes, attribute information, attribute data, etc. In addition, the point cloud content providing system (e.g., the point cloud transmitting device 10000 or the point cloud video acquiring unit 10001) may obtain point cloud data from information related to the point cloud video acquiring process (e.g., depth information, hue information, etc.).

[0031] A point cloud content providing system (e.g., a transmitting device 10000 or a point cloud video encoder 10002) according to an embodiment encodes point cloud data (20001). The point cloud content providing system encodes point cloud data based on point cloud compression coding. As described above, point cloud data includes geometry and attributes of points. Thus, the point cloud content providing system performs geometry encoding to encode the geometry and outputs a geometry bitstream. The point cloud content providing system performs attribute encoding to encode the attributes and outputs an attribute bitstream. In an embodiment, the point cloud content providing system performs attribute encoding based on the geometry encoding. The geometry bitstream and the attribute bitstream according to the embodiment are multiplexed and output as one bitstream. The bitstream according to the embodiment further includes signaling information related to the geometry encoding and the attribute encoding.

[0032] A point cloud content providing system (e.g., transmitting device 10000 or transmitter 10003) according to an embodiment transmits encoded point cloud data (20002). As described in FIG. 1, the encoded point cloud data is represented by a geometry bitstream and a feature bitstream. The encoded point cloud data is transmitted in the form of a bitstream together with signaling information related to the encoding of the point cloud data (e.g., signaling information related to geometry encoding and feature encoding). The point cloud content providing system also encapsulates the bitstream for transmitting the encoded point cloud data and transmits it in the form of a file or segment.

[0033] A point cloud content providing system (e.g., receiving device 10004 or receiver 10005) according to an embodiment receives a bitstream including encoded point cloud data, and the point cloud content providing system (e.g., receiving device 10004 or receiver 10005) demultiplexes the bitstream.

[0034] The point cloud content providing system (e.g., the receiving device 10004 or the point cloud video decoder 10005) decodes the encoded point cloud data (e.g., geometry bitstream, attribute bitstream) transmitted in a bitstream. The point cloud content providing system (e.g., the receiving device 10004 or the point cloud video decoder 10005) decodes the point cloud video data based on signaling information related to the encoding of the point cloud video data included in the bitstream. The point cloud content providing system (e.g., the receiving device 10004 or the point cloud video decoder 10005) decodes the geometry bitstream to restore the position (geometry) of the point. The point cloud content providing system decodes the attribute bitstream based on the restored geometry to restore the attribute of the point. The point cloud content providing system (e.g., the receiving device 10004 or the point cloud video decoder 10005) restores the point cloud video based on the position according to the restored geometry and the decoded attribute.

[0035] A point cloud content providing system (e.g., receiving device 10004 or renderer 10007) according to an embodiment renders the decoded point cloud data (20004). The point cloud content providing system (e.g., receiving device 10004 or renderer 10007) renders the geometry and characteristics decoded during the decoding process using various rendering methods. Points of the point cloud content are rendered as a fixed point with a certain thickness, a cube with a predetermined minimum size centered on the position of the fixed point, or a circle centered on the position of the fixed point. All or a part of the region of the rendered point cloud content is provided to a user via a display (e.g., a VR / AR display, a general display, etc.).

[0036] The point cloud content providing system according to the embodiment (e.g., receiving device 10004) can obtain feedback information (20005). The point cloud content providing system encodes and / or decodes point cloud data based on the feedback information. The feedback information and the operation of the point cloud content providing system according to the embodiment are the same as the feedback information and the operation described in FIG. 1, so a detailed description will be omitted.

[0037] FIG. 3 illustrates an example of a point cloud video capture process according to an embodiment.

[0038] FIG. 3 illustrates an example of a point cloud video capture process of the point cloud content providing system described in FIG. 1 and FIG.

[0039] Point cloud content includes point cloud videos (images and / or videos) showing objects and / or environments located in various 3D spaces (e.g., a 3D space showing a real environment, a 3D space showing a virtual environment, etc.). Thus, the point cloud content providing system according to the embodiment captures point cloud videos using one or more cameras (e.g., an infrared camera capable of obtaining depth information, an RGB camera capable of extracting color information corresponding to the depth information, etc.), projectors (e.g., an infrared pattern projector for obtaining depth information, etc.), LiDAR, etc. to generate point cloud content. The point cloud content providing system according to the embodiment extracts a geometric form composed of points in a 3D space from the depth information, and extracts characteristics of each point from the color information to obtain point cloud data. The images and / or videos according to the embodiment are captured based on either an inward-facing approach or an outward-facing approach.

[0040] The inward-looking method is shown on the left side of FIG. 3. The inward-looking method is a method in which one or more cameras (or camera sensors) positioned around a central object capture the central object. The inward-looking method is used to generate point cloud content that provides the user with a 360° image of a core object (e.g., VR / AR content that provides the user with a 360° image of an object (e.g., a core object such as a character, player, item, actor, etc.)).

[0041] The outward-looking approach is shown on the right side of Figure 3. In the outward-looking approach, one or more cameras (or camera sensors) positioned around a central object capture the environment of the central object that is not the central object. The outward-looking approach is used to generate point cloud content to provide the surrounding environment from a user's perspective (e.g., content showing the external environment provided to a user of an autonomous vehicle).

[0042] As shown in the figure, the point cloud content is generated based on the capture operation of one or more cameras. In this case, since the coordinate systems of the respective cameras are different, the point cloud content providing system performs calibration of one or more cameras to set a global coordinate system before the capture operation. The point cloud content providing system generates the point cloud content by combining the image and / or video captured by the above-mentioned capture method with an arbitrary image and / or video. When generating point cloud content representing a virtual space, the point cloud content providing system does not perform the capture operation described in FIG. 3. The point cloud content providing system according to the embodiment can also perform post-processing on the captured image and / or video. That is, the point cloud content providing system can perform an operation of removing an undesired area (e.g., background) or recognizing a space where the captured images and / or videos are connected and filling a spatial hole if there is one.

[0043] The point cloud content providing system can also generate one point cloud content by performing coordinate system transformation on the points of the point cloud video acquired from each camera. The point cloud content providing system performs coordinate system transformation of the points based on the position coordinates of each camera. In this way, the point cloud content providing system can generate content showing a wide range or generate point cloud content with a high point density.

[0044] FIG. 4 is a diagram illustrating an example of a point cloud encoder according to an embodiment.

[0045] FIG. 4 shows an example of the point cloud video encoder 10002 of FIG. 1. The point cloud encoder performs encoding by reconstructing point cloud data (e.g., point positions and / or characteristics) to adjust the quality of the point cloud content (e.g., lossless, lossy, near-lossless) according to the network condition or application. If the overall size of the point cloud content is large (e.g., point cloud content of 60 Gbps for 30 fps), the point cloud content providing system cannot stream the corresponding content in real time. Therefore, the point cloud content providing system can reconstruct the point cloud content based on the maximum target bit rate to provide it according to the network environment.

[0046] As shown in Figures 1 and 2, a point cloud encoder can perform geometry encoding and feature encoding, where the geometry encoding is performed before the feature encoding.

[0047] The point cloud encoder according to the embodiment includes a coordinate system transformation unit (Transformation Coordinates) 40000, a quantization unit (Quantize and Remove Points (Voxelize)) 40001, an octree analysis unit (Analyze Octree) 40002, a surface approximation analysis unit (Analyze Surface Approximation) 40003, an arithmetic encoder (Arithmetic Encode) 40004, a geometry reconstruction unit (Reconstruct Geometry) 40005, a color transformation unit (Transform Colors) 40006, a characteristic transformation unit (Transfer Attributes) 40007, a RAHT transformation unit 40008, an LOD generation unit (Generated LOD) 40009, a lift transformation unit (Lifting) 40010, a coefficient quantization unit (Quantize Coefficients) 40011 and / or an arithmetic encoder (Arithmetic Encode) 40012.

[0048] The coordinate system conversion unit 40000, the quantization unit 40001, the octree analysis unit 40002, the surface approximation analysis unit 40003, the arithmetic encoder 40004, and the geometry reconstruction unit 40005 perform geometry encoding. The geometry encoding according to the embodiment includes octree geometry coding, direct coding, trisoup geometry encoding, and entropy coding. The direct coding and trisoup geometry encoding are applied selectively or in combination. Note that the geometry encoding is not limited to the above examples.

[0049] As shown in the figure, the coordinate system conversion unit 40000 according to the embodiment receives a position and converts it into a coordinate system. For example, the position is converted into position information in a three-dimensional space (e.g., a three-dimensional space expressed in an XYZ coordinate system). The position information in the three-dimensional space according to the embodiment is also referred to as geometry information.

[0050] The quantizer 40001 according to the embodiment quantizes the geometry. For example, the quantizer 40001 quantizes the points based on the minimum position value of all points (for example, the minimum value on each axis for the X-axis, Y-axis, and Z-axis). The quantizer 40001 performs a quantization operation by multiplying the difference between the minimum position value and the position value of each point by a predetermined quantization scale value, and then rounding down or up to find the closest integer value. Therefore, one or more points can have the same quantized position (or position value). The quantizer 40001 according to the embodiment performs voxelization based on the quantized position to reconstruct the quantized points. The smallest unit including 2D image / video information is a pixel, and the points of the point cloud content (or 3D point cloud video) according to the embodiment are included in one or more voxels. Voxel is a combination of volume and pixel, and refers to a 3D cubic space that is generated when a 3D space is divided into units (unit=1.0) based on the axes (e.g., X-axis, Y-axis, Z-axis) that represent the 3D space. The quantization unit 40001 matches a group of points in the 3D space with voxels. In an embodiment, one voxel includes only one point. In an embodiment, one voxel includes one or more points. In addition, to represent one voxel with one point, the position of the center of the voxel can be set based on the positions of one or more points included in the voxel. In this case, the characteristics of all positions included in one voxel are combined and assigned to the voxel.

[0051] The octree analysis unit 40002 according to the embodiment performs octree geometry coding (or octree coding) to represent the voxels in an octree structure, which represents points matched to the voxels based on an octet structure.

[0052] The surface approximation analysis unit 40003 according to the embodiment analyzes and approximates the octree. The octree analysis and approximation according to the embodiment is a process of analyzing a region containing a large number of points to voxelize it in order to efficiently provide an octree and voxelization.

[0053] According to the embodiment, the arithmetic encoder 40004 entropy encodes the octree and / or the approximated octree. For example, the encoding method includes an arithmetic encoding method. As a result of the encoding, a geometry bitstream is generated.

[0054] The color transform unit 40006, the attribute transform unit 40007, the RAHT transform unit 40008, the LOD generating unit 40009, the lift transform unit 40010, the coefficient quantization unit 40011 and / or the arithmetic encoder 40012 perform attribute coding. As described above, one point has one or more attributes. Attribute coding according to the embodiment is equally applied to attributes of one point. However, when one attribute (e.g., hue) includes one or more elements, independent attribute coding is applied to each element. Attribute coding according to the embodiment includes color transform coding, attribute transform coding, RAHT (Region Adaptive Hierarchial Transform) coding, Interpolarization-based hierarchical nearest-neighbour prediction-Prediction Transform coding, and Lift transform (interpolation-based hierarchical nearest-neighbour prediction with an update / lifting step (Lifting Transform)) coding. Depending on the point cloud content, the above-mentioned RAHT coding, predictive transform coding, and lift transform coding may be selectively used, or a combination of one or more coding may be used. Furthermore, the feature coding according to the embodiment is not limited to the above examples.

[0055] The color converter 40006 according to the embodiment performs color conversion coding to convert the color values ​​(or textures) included in the characteristics. For example, the color converter 40006 converts the format of the hue information (e.g., converts from RGB to YCbCr). The operation of the color converter 40006 according to the embodiment is optionally applied depending on the color values ​​included in the characteristics.

[0056] The geometry reconstruction unit 40005 according to the embodiment reconstructs (restores) the octree and / or the approximated octree. The geometry reconstruction unit 40005 reconstructs the octree / voxel based on the result of analyzing the distribution of points. The reconstructed octree / voxel is also called the reconstructed geometry (or restored geometry).

[0057] The feature converter 40007 according to the embodiment performs feature conversion to convert features based on a position where geometry encoding has not been performed and / or a reconstructed geometry. As described above, features depend on geometry, so the feature converter 40007 can convert features based on reconstructed geometry information. For example, the feature converter 40007 converts features of a point included in a voxel based on the position value of the point. As described above, when the position of the center point of a voxel is set based on the positions of one or more points included in a voxel, the feature converter 40007 converts features of one or more points. When trisoup geometry encoding is performed, the feature converter 40007 converts features based on the trisoup geometry encoding.

[0058] The attribute conversion unit 40007 performs attribute conversion by calculating the average value of the attributes or attribute values ​​(e.g., the hue or reflectance of each point) of adjacent points within a certain position / radius from the position (or position value) of the center point of each voxel. When calculating the average value, the attribute conversion unit 40007 applies a weighting value according to the distance from the center point to each point. Thus, each voxel has a position and a calculated attribute (or attribute value).

[0059] The feature conversion unit 40007 searches for adjacent points within a specific position / radius from the position of the center point of each voxel based on the KD tree or Moulton code. The KD tree supports a data structure that manages points based on their position so that a Nearest Neighbor Search (NNS) can be performed quickly using a binary search tree. The Moulton code is generated by mixing bits and expressing the coordinate values ​​(e.g., (x, y, z)) indicating the three-dimensional position of all points. For example, if the coordinate values ​​indicating the position of a point are (5, 9, 1), the bit values ​​of the coordinate values ​​are (0101, 1001, 0001). If the bit values ​​are mixed according to the bit index in the order of z, y, and x, the result is 010001000111. This value is expressed as 1095 in decimal. In other words, the Moulton code value of a point with coordinate values ​​(5, 9, 1) is 1095. The feature converter 40007 aligns points based on the Moulton code value and performs nearest neighbor search (NNS) through a depth-first traversal process. If nearest neighbor search (NNS) is required in other conversion processes for feature coding after the feature conversion operation, a KD tree or Moulton code is used.

[0060] As shown, the transformed attributes are input to a RAHT transformer 40008 and / or an LOD generator 40009 .

[0061] The RAHT converter 40008 according to the embodiment performs RAHT coding to predict feature information based on the reconstructed geometry information. For example, the RAHT converter 40008 can predict feature information of a node at a higher level of the octree based on feature information associated with a node at a lower level of the octree.

[0062] The LOD generating unit 40009 according to the embodiment generates LOD (Level of Detail) to perform predictive conversion coding. The LOD according to the embodiment indicates the degree of detail of the point cloud content, and the smaller the LOD value, the lower the detail of the point cloud content, and the larger the LOD value, the higher the detail of the point cloud content. Points can be classified according to the LOD.

[0063] The lift transform unit 40010 according to the embodiment performs lift transform coding, which transforms the characteristics of the point cloud based on weights. As described above, the lift transform coding is selectively applied.

[0064] The coefficient quantization unit 40011 according to the embodiment quantizes the encoded feature based on the coefficients.

[0065] The arithmetic encoder 40012 according to the embodiment encodes the quantized characteristic based on arithmetic coding.

[0066] The elements of the point cloud encoder of FIG. 4 may be implemented in hardware, software, firmware, or a combination thereof, including one or more processors or integrated circuits (not shown) configured to communicate with one or more memories included in the point cloud providing apparatus. The one or more processors may perform any one of the operations and / or functions of the elements of the point cloud encoder of FIG. 4 described above. The one or more processors may also operate or execute a software program and / or set of instructions to perform the operations and / or functions of the elements of the point cloud encoder of FIG. 4. According to an embodiment, the one or more memories may include high speed random access memory or may include non-volatile memory (e.g., one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state memory devices, etc.).

[0067] FIG. 5 is a diagram showing an example of a voxel according to the embodiment.

[0068] FIG. 5 shows a voxel located in a 3D space represented by a coordinate system consisting of three axes, X-axis, Y-axis, and Z-axis. As shown in FIG. 4, a point cloud encoder (e.g., quantizer 40001, etc.) performs voxelization. A voxel means a 3D cubic space generated when a 3D space is divided into units (unit=1.0) based on the axes (e.g., X-axis, Y-axis, Z-axis) that represent the 3D space. FIG. 5 shows an example of a voxel generated by an octree structure that recursively subdivides a cubical axis-aligned bounding box defined by two extreme points (0,0,0) and (2d,2d,2d). One voxel includes at least one point. The spatial coordinates of a voxel can be estimated from its positional relationship with a voxel group. As described above, a voxel has a characteristic (such as a hue or reflectance) like a pixel in a 2D image / video. A detailed description of the voxels is omitted since it has been described with reference to FIG.

[0069] FIG. 6 is a diagram illustrating an example of an octree and occupancy code according to an embodiment.

[0070] As shown in Figures 1 to 4, a point cloud content providing system (point cloud video encoder 10002) or a point cloud encoder (e.g., octree analysis unit 40002) performs octree geometry coding (or octree coding) based on an octree structure to efficiently manage the area and / or position of voxels.

[0071] The upper part of FIG. 6 shows an octree structure. The three-dimensional space of the point cloud content according to the embodiment is represented by the axes of a coordinate system (e.g., X-axis, Y-axis, and Z-axis). The octree structure is generated by recursively subdividing a cubical axis-aligned bounding box defined by two extreme points (0,0,0) and (2d,2d,2d). 2d is set to a value that constitutes the smallest bounding box that encloses all the points of the point cloud content (or point cloud video). d indicates the depth of the octree. The d value is determined by the following formula: In the following formula, (xintn, yintn, zintn) indicates the quantized point position (or position value).

[0072]

number

[0073] As shown in the upper center of Figure 6, the entire 3D space is divided into eight spaces through division. Each divided space is represented as a cube with six sides. As shown in the upper right of Figure 6, each of the eight spaces is again divided by the axes of the coordinate system (e.g., X-axis, Y-axis, Z-axis). Therefore, each space is again divided into eight smaller spaces. The divided smaller spaces are also represented as cubes with six sides. This division method is applied until the leaf nodes of the octree become voxels.

[0074] The lower part of FIG. 6 shows an occupancy code of an octree. The occupancy code of an octree is generated to indicate whether each of the eight divided spaces generated by dividing one space contains at least one point. Therefore, one occupancy code is expressed by eight child nodes. Each child node indicates the occupancy of the divided space, and has a 1-bit value. Therefore, the occupancy code is expressed by an 8-bit code. That is, if the space corresponding to a child node contains at least one point, the corresponding node has a value of 1. If the space corresponding to a node does not contain a point (empty), the corresponding node has a value of 0. The occupancy code shown in FIG. 6 is 00100001, which indicates that the spaces corresponding to the third and eighth child nodes of the eight child nodes each contain at least one point. As shown in the figure, the third and eighth child nodes each have eight child nodes, and each child node is expressed by an 8-bit occupancy code. In the figure, the occupied code of the third child node is 10000111, and the occupied code of the eighth child node is 01001111. A point cloud encoder (e.g., arithmetic encoder 40004) according to the embodiment entropy codes the occupied code. To improve compression efficiency, the point cloud encoder also intra / inter codes the occupied code. A receiving device (e.g., receiving device 10004 or point cloud video decoder 10006) according to the embodiment reconstructs an octree based on the occupied code.

[0075] A point cloud encoder according to an embodiment (e.g., the point cloud encoder or octree analyzer 40002 in FIG. 4) performs voxelization and octree coding to store the positions of points. However, since points in a 3D space are not always uniformly distributed, there may be certain areas where there are not many points. Therefore, it is inefficient to perform voxelization on the entire 3D space. For example, if there are almost no points in a certain area, there is no need to perform voxelization on that area.

[0076] Therefore, the point cloud encoder according to the embodiment does not perform voxelization for the above-mentioned specific region (or nodes other than the leaf nodes of the octree), but performs direct coding, which directly codes the positions of points included in the specific region. The coordinates of the direct coding points according to the embodiment are called a direct coding mode (DCM). The point cloud encoder according to the embodiment can also perform trisoup geometry encoding, which reconstructs the positions of points in a specific region (or node) based on voxels based on a surface model. Trisoup geometry encoding is a geometry encoding that represents an object as a series of triangle meshes. Therefore, the point cloud decoder can generate a point cloud from the mesh surface. The direct coding and trisoup geometry encoding according to the embodiment are selectively performed. The direct coding and trisoup geometry encoding according to the embodiment can also be combined with octree geometry coding (or octree coding).

[0077] To perform direct coding, an option to use direct mode for applying direct coding must be activated, and the node to which direct coding is applied must not be a leaf node, and points below a threshold must exist within a specific node. Also, the total number of points to be subject to direct coding must not exceed a certain threshold. If the above conditions are met, the point cloud encoder (or the computation encoder 40004) according to the embodiment can entropy code the positions (or position values) of the points.

[0078] The point cloud encoder (e.g., the surface approximation analysis unit 40003) according to the embodiment can determine a specific level of the octree (if the level is smaller than the depth d of the octree), and from that level, can perform trisoup geometry encoding, which uses a surface model to reconstruct the positions of points in the node area based on voxels (trisoup mode). The point cloud encoder according to the embodiment can specify the level to which the trisoup geometry encoding is applied. For example, if the specified level is equal to the depth of the octree, the point cloud encoder does not operate in trisoup mode. That is, the point cloud encoder according to the embodiment can operate in trisoup mode only when the specified level is smaller than the depth value of the octree. The 3D cubic area of ​​the node at a specified level according to the embodiment is called a block. One block includes one or more voxels. A block or a voxel can also correspond to a brick. Within each block, the geometry is represented as a surface. The surface according to the embodiment can intersect with each edge of the block at most once.

[0079] Since a block has 12 edges, there are at least 12 intersections within a block. Each intersection is called a vertex. A vertex along an edge is detected if there is at least one occupied voxel adjacent to the edge among all blocks that share the edge. In this embodiment, an occupied voxel refers to a voxel that contains a point. The position of a vertex detected along an edge is the average position along the edge of all voxels adjacent to the edge among all blocks that share the edge.

[0080] When a vertex is detected, the point cloud encoder according to the embodiment entropy codes the edge start point (x, y, z), the edge direction vector (Δx, Δy, Δz), and the vertex position value (relative position value within the edge). When trisoup geometry encoding is applied, the point cloud encoder according to the embodiment (e.g., the geometry reconstruction unit 40005) performs triangle reconstruction, up-sampling, and voxelization processes to generate a restored geometry (reconstructed geometry).

[0081] Vertices located on the edges of blocks determine the surface that passes through the blocks. In this embodiment, the surface is a non-planar polygon. In the triangulation process, the surface represented by triangles is reconstructed based on the start points of the edges, the edge direction vectors, and the vertex position values. The triangulation process is as follows: 1. Calculate the centroid value of each vertex, 2. Subtract the centroid value from the value of each vertex, 3. Square the result, and add all the values ​​together to obtain a value.

[0082]

number

[0083] The minimum of the added values ​​is found, and a projection process is performed along the axis where the minimum value is. For example, if the x element is the minimum, each vertex is projected onto the x-axis based on the center of the block, and then onto the (y, z) plane. If the value obtained by projecting onto the (y, z) plane is (ai, bi), the θ value is found using atan2(bi, ai), and the vertices are aligned based on the θ value. The following table shows the combination of vertices to generate a triangle depending on the number of vertices. The vertices are aligned in order from 1 to n. The following table shows that for four vertices, two triangles are formed by combining the vertices. The first triangle is formed from the 1st, 2nd, and 3rd vertices of the aligned vertices, and the second triangle is formed from the 3rd, 4th, and 1st vertices of the aligned vertices.

[0084] Table 2-1.Triangles formed from vertices ordered 1,…,n

[0085] n triangles

[0086] 3(1,2,3)

[0087] 4(1,2,3), (3,4,1)

[0088] 5(1,2,3), (3,4,5), (5,1,3)

[0089] 6(1,2,3), (3,4,5), (5,6,1), (1,3,5)

[0090] 7(1,2,3), (3,4,5), (5,6,7), (7,1,3), (3,5,7)

[0091] 8(1,2,3), (3,4,5), (5,6,7), (7,8,1), (1,3,5), (5,7,1)

[0092] 9(1,2,3), (3,4,5), (5,6,7), (7,8,9), (9,1,3), (3,5,7), (7,9,3)

[0093] 10(1,2,3), (3,4,5), (5,6,7), (7,8,9), (9,10,1), (1,3,5), (5,7,9), (9,1,5)

[0094] 11(1,2,3), (3,4,5), (5,6,7), (7,8,9), (9,10,11), (11,1,3), (3,5,7), (7,9,11), (11,3,7)

[0095] 12(1,2,3), (3,4,5), (5,6,7), (7,8,9), (9,10,11), (11,12,1), (1,3,5), (5,7,9), (9,11,1), (1,5,9)

[0096] The upsampling process is performed to add intermediate points along the edges of triangles for voxelization. The additional points are generated based on the upsampling factor and the block width. The additional points are called refined vertices. The point cloud encoder according to the embodiment can voxelize the refined vertices. The point cloud encoder can also perform feature encoding based on the voxelized positions (or position values).

[0097] FIG. 7 is a diagram illustrating an example of an adjacent node pattern according to the embodiment.

[0098] To increase the compression efficiency of the point cloud video, the point cloud encoder according to the embodiment performs entropy coding based on context adaptive arithmetic coding.

[0099] As described in FIG. 1 to FIG. 6, a point cloud content providing system or a point cloud encoder (e.g., a point cloud video encoder 10002, a point cloud encoder or an arithmetic encoder 40004 in FIG. 4) immediately performs entropy coding on the occupancy code. In addition, the point cloud content providing system or the point cloud encoder performs entropy coding (intra coding) based on the occupancy code of the current node and the occupancy rate of the neighboring node, or performs entropy coding (inter coding) based on the occupancy code of the previous frame. A frame according to the embodiment means a collection of point cloud videos generated at the same time. The compression efficiency of intra coding / inter coding according to the embodiment varies depending on the number of neighboring nodes to be referenced. Although the complexity increases as the number of bits increases, the compression efficiency can be improved by leaning to one side. For example, if there is a 3-bit context, coding is performed in 8 ways, which is 2 to the power of 3. The part that is coded separately affects the complexity of the implementation. Therefore, it is necessary to match the compression efficiency and the appropriate level of the complexity.

[0100] FIG. 7 illustrates a process of determining an occupancy pattern based on the occupancy of adjacent nodes. The point cloud encoder according to the embodiment obtains a neighbor pattern value by determining the occupancy of adjacent nodes of each node in an occupancy tree. The neighbor pattern is used to infer the occupancy pattern of the corresponding node. The left side of FIG. 7 illustrates a cube corresponding to the node (the cube located in the middle) and six cubes (neighboring nodes) that share at least one side with the corresponding cube. The illustrated nodes are nodes at the same depth. The illustrated numbers indicate the weights (1, 2, 4, 8, 16, 32, etc.) associated with each of the six nodes. Each weight is assigned in order according to the position of the neighboring node.

[0101] The right side of FIG. 7 shows the adjacent node pattern value. The adjacent node pattern value is the sum of values ​​multiplied by the weight values ​​of occupied adjacent nodes (adjacent nodes having points). Therefore, the adjacent node pattern value has a value from 0 to 63. An adjacent node pattern value of 0 means that there is no node (occupied node) that has points among the adjacent nodes of the corresponding node. An adjacent node pattern value of 63 means that all adjacent nodes are occupied nodes. As shown in the figure, adjacent nodes to which weight values ​​1, 2, 4, and 8 are assigned are occupied nodes, so the adjacent node pattern value is 15, which is a value obtained by adding 1, 2, 4, and 8. The point cloud encoder can perform coding according to the adjacent node pattern value (e.g., if the adjacent node pattern value is 63, 64 coding is performed). In an embodiment, the point cloud encoder can reduce the coding complexity by changing the adjacent node pattern value (e.g., based on a table that changes 64 to 10 or 6).

[0102] FIG. 8 is a diagram illustrating an example of a point configuration for each LOD according to the embodiment.

[0103] As described in Figures 1 to 7, before feature coding is performed, the coded geometry is reconstructed (restored). When direct coding is applied, the operation of geometry reconstruction includes changing the placement of the direct coded points (e.g., placing the direct coded points in front of the point cloud data). When trisoup geometry coding is applied, the process of geometry reconstruction includes the processes of triangulation, upsampling, and voxelization. Since features are dependent on geometry, feature coding is performed based on the reconstructed geometry.

[0104] A point cloud encoder (e.g., LOD generator 40009) reorganizes points according to LOD. The diagram shows point cloud content corresponding to LOD. The left side of the diagram shows the original point cloud content. The second from the left in the diagram shows the distribution of points with the lowest LOD, and the rightmost side shows the distribution of points with the highest LOD. That is, the points with the lowest LOD have a sparse distribution, and the points with the highest LOD have a fine distribution. That is, as the LOD increases along the direction of the arrow shown at the bottom of the diagram, the spacing (or distance) between points becomes shorter.

[0105] FIG. 9 is a diagram illustrating an example of a point configuration for each LOD according to the embodiment.

[0106] As described in FIG. 1 to FIG. 8, a point cloud content providing system or a point cloud encoder (e.g., a point cloud video encoder 10002, a point cloud encoder or LOD generator 40009 in FIG. 4) generates LOD. The LOD is generated by realigning points with a set of refinement levels according to a set LOD distance value (or a set of Euclidean distances). The LOD generation process is performed not only in the point cloud encoder but also in the point cloud decoder.

[0107] The upper part of Figure 9 shows an example of points (P0 to P9) of point cloud content distributed in 3D space. The original order in Figure 9 shows the order of points P0 to P9 before LOD generation. The LOD based order in Figure 9 shows the order of points after LOD generation. Points are re-ordered for each LOD. Also, higher LODs include points that belong to lower LODs. As shown in Figure 9, LOD0 includes P0, P5, P4, and P2. LOD1 includes points of LOD0, P1, P6, and P3. LOD2 includes points of LOD0, points of LOD1, and P9, P8, and P7.

[0108] As described in FIG. 4, the point cloud encoder according to the embodiment can selectively or in combination perform predictive transform coding, lift transform coding, and RAHT transform coding.

[0109] The point cloud encoder according to the embodiment generates a predictor for each point and performs predictive conversion coding to set a predicted characteristic (or predicted characteristic value) for each point. That is, N predictors are generated for N points. The predictor according to the embodiment can calculate a weight (=1 / distance) based on the LOD value of each point, index information for adjacent points within a distance set for each LOD, and a distance value to the adjacent point.

[0110] The predicted feature (or feature value) according to the embodiment is set as an average value of the feature (or feature value, e.g., hue, reflectance, etc.) of the neighboring points set in the predictor of each point multiplied by the weight (or weight value) calculated based on the distance to each neighboring point. The point cloud encoder (e.g., the coefficient quantization unit 40011) according to the embodiment may quantize and inverse quantize the residuals (also called residual features, residual feature values, feature predicted residual values, etc.) obtained by subtracting the predicted feature (feature value) from the feature (feature value) of each point. The quantization process is as shown in the following table.

[0111] Table.Attribute prediction residuals quantization pseudo code

[0112] int PCCQuantization(int value, int quantStep) {

[0113] if( value >= 0 ) {

[0114] return floor(value / quantStep + 1.0 / 3.0);

[0115] } else {

[0116] return -floor(-value / quantStep + 1.0 / 3.0);

[0117] }

[0118] }

[0119] Attribute prediction residuals inverse quantization pseudo code

[0120] int PCCInverseQuantization (int value, int quantStep) {

[0121] if(quantStep == 0) {

[0122] return value;

[0123] } else {

[0124] return value*quantStep;

[0125] }

[0126] }

[0127] The point cloud encoder (e.g., the arithmetic encoder 40012) according to the embodiment performs entropy coding on the quantized and dequantized residual values ​​as described above if there are adjacent points in the predictor of each point. If there are no adjacent points in the predictor of each point, the point cloud encoder (e.g., the arithmetic encoder 40012) according to the embodiment does not perform the above process and performs entropy coding on the characteristics of the corresponding point.

[0128] The point cloud encoder (e.g., lift transform unit 40010) according to the embodiment generates a predictor for each point, sets the calculated LOD in the predictor, registers adjacent points, and sets weights according to the distance to the adjacent points to perform lift transform coding. The lift transform coding according to the embodiment is similar to the above-mentioned point transform coding, but differs in that weights are cumulatively applied to feature values. The process of cumulatively applying weights to feature values ​​according to the embodiment is as follows.

[0129] 1) Create an array QW (QuantizationWeight) to store the weight value of each point. The initial value of all elements of QW is 1.0. The QW value of the predictor index of the adjacent node registered in the predictor is multiplied by the predictor weight value of the current point and added.

[0130] 2) Lift prediction process: To calculate the predicted attribute value, the attribute value of the point is multiplied by the weighting value and subtracted from the existing attribute value.

[0131] 3) Create temporary arrays called updateweight and update, and initialize the temporary arrays to 0.

[0132] 4) The calculated weights for all predictors are multiplied by the weights stored in the QW corresponding to the predictor index, and the calculated weights are accumulated and summed in the update weight array as the index of the adjacent node. The update array is accumulated and summed by multiplying the attribute values ​​of the index of the adjacent node by the calculated weights.

[0133] 5) Lift update process: For every predictor, divide the feature value in the update array by the weight value in the update weight array for the predictor index, and then add the divided value back to the existing feature value.

[0134] 6) For all predictors, the feature values ​​updated in the lift update process are further multiplied by the weights updated in the lift prediction process (stored in the QW) to calculate predicted feature values. A point cloud encoder (e.g., coefficient quantizer 40011) according to the embodiment quantizes the predicted feature values. In addition, a point cloud encoder (e.g., arithmetic encoder 40012) entropy codes the quantized feature values.

[0135] The point cloud encoder according to the embodiment (e.g., the RAHT converter 40008) performs RAHT transform coding, which predicts the features of higher level nodes using features associated with lower level nodes of the octree. RAHT transform coding is an example of feature intra-coding using octree backward scan. The point cloud encoder according to the embodiment scans the entire area from the voxel, and repeats the merging process up to the root node while combining the voxels into larger blocks at each step. The merging process according to the embodiment is performed only for occupied nodes. The merging process is not performed for empty nodes, but is performed for the nodes immediately above the empty nodes.

[0136] [Table 1]

[0137] FIG. 10 is a diagram illustrating an example of a point cloud decoder according to an embodiment.

[0138] The point cloud decoder illustrated in FIG. 10 is an example of the point cloud video decoder 10006 illustrated in FIG. 1, and performs the same or similar operations as the point cloud video decoder 10006 described in FIG. 1. As illustrated, the point cloud decoder receives a geometry bitstream and an attribute bitstream included in one or more bitstreams. The point cloud decoder includes a geometry decoder and an attribute decoder. The geometry decoder performs geometry decoding on the geometry bitstream and outputs a decoded geometry. The attribute decoder performs attribute decoding based on the decoded geometry and attribute bitstream and outputs decoded attributes. The decoded geometry and decoded attributes are used to restore the point cloud content (decoded point cloud).

[0139] FIG. 11 is a diagram illustrating an example of a point cloud decoder according to an embodiment.

[0140] The point cloud decoder shown in FIG. 11 is an example of the point cloud decoder described in FIG. 10, and performs a decoding operation that is the reverse process of the encoding operation of the point cloud encoder described in FIGS. 1 to 9.

[0141] As explained in Figure 1 and Figure 10, the point cloud decoder performs geometry decoding and feature decoding. Geometry decoding is performed before feature decoding.

[0142] The point cloud decoder according to the embodiment includes an arithmetic decoder 11000, an octree synthesis unit 11001, a surface approximation synthesis unit 11002, a reconstruct geometry unit 11003, an inverse transform coordinates unit 11004, an arithmetic decoder 11005, an inverse quantize unit 11006, a RAHT transform unit 11007, a generate LOD unit 11008, an inverse lifting unit 11009 and / or an inverse transform colors unit 11010.

[0143] The arithmetic decoder 11000, the octree synthesis unit 11001, the surface approximation synthesis unit 11002, the geometry reconstruction unit 11003, and the coordinate system inverse transformation unit 11004 perform geometry decoding. The geometry decoding according to the embodiment includes direct coding and trisoup geometry decoding. The direct coding and trisoup geometry decoding are selectively applied. Furthermore, the geometry decoding is not limited to the above examples, and is performed in the reverse process of the geometry encoding described in FIG. 1 to FIG. 9.

[0144] The arithmetic decoder 11000 according to the embodiment decodes the received geometry bitstream based on arithmetic coding. The operation of the arithmetic decoder 11000 corresponds to the reverse process of the arithmetic encoder 40004.

[0145] The octree synthesis unit 11001 according to the embodiment obtains exclusive codes from the decoded geometry bitstream (or from the decoded result, information on the secured geometry) to generate an octree. The exclusive codes are described in detail in FIG. 1 to FIG. 9.

[0146] The surface approximation synthesis unit 11002 according to the embodiment synthesizes a surface based on the decoded geometry and / or the generated octree if trisoup geometry encoding is applied.

[0147] The geometry reconstruction unit 11003 according to the embodiment regenerates geometry based on the surface and / or the decoded geometry. As described in FIG. 1 to FIG. 9, direct coding and trisoup geometry coding are selectively applied. Therefore, the geometry reconstruction unit 11003 directly brings and adds position information of the point to which the direct coding is applied. Also, when the trisoup geometry coding is applied, the geometry reconstruction unit 11003 restores the geometry by performing the reconstruction operation of the geometry reconstruction unit 40005, for example, triangulation, upsampling, and voxelization operations. The detailed contents are omitted since they are the same as those described in FIG. 6. The restored geometry includes a point cloud picture or frame that does not include features.

[0148] A coordinate system inverse transformation unit 11004 according to the embodiment transforms the coordinate system based on the reconstructed geometry to obtain the position of the point.

[0149] The arithmetic decoder 11005, the inverse quantization unit 11006, the RAHT transform unit 11007, the LOD generating unit 11008, the inverse lift unit 11009 and / or the color inverse transform unit 11010 perform the feature decoding described in FIG. 10. The feature decoding according to the embodiment includes RAHT (Region Adaptive Hierarchial Transform) decoding, Interpolarization-based hierarchical nearest-neighbour prediction-Prediction Transform (Interpolarization-based hierarchical nearest-neighbour prediction with an update / lifting step (Lifting Transform)) decoding. The above three decodings are selectively used, or one or more combinations of decodings are used. In addition, the feature decoding according to the embodiment is not limited to the above examples.

[0150] An arithmetic decoder 11005 according to an embodiment decodes the attribute bitstream into arithmetic coding.

[0151] The inverse quantization unit 11006 according to the embodiment inverse quantizes the decoded feature bitstream or the feature information obtained as a result of decoding, and outputs the inverse quantized feature (or feature value). The inverse quantization is selectively applied based on the feature coding of the point cloud encoder.

[0152] In an embodiment, the RAHT transform unit 11007, the LOD generator 11008 and / or the inverse lift unit 11009 process the reconstructed geometry and dequantized features. As described above, the RAHT transform unit 11007, the LOD generator 11008 and / or the inverse lift unit 11009 selectively perform decoding operations corresponding to the encoding of the point cloud encoder.

[0153] The color inverse converter 11010 according to the embodiment performs inverse coding to inversely convert color values ​​(or textures) included in the decoded features. The operation of the color inverse converter 11010 is selectively performed based on the operation of the color converter 40006 of the point cloud encoder.

[0154] The elements of the point cloud decoder of Figure 11 may be implemented in hardware, software, firmware, or a combination thereof, including one or more processors or integrated circuits (not shown) configured to communicate with one or more memories included in the point cloud providing device. The one or more processors perform any of the operations and / or functions of the elements of the point cloud decoder of Figure 11 described above. Additionally, the one or more processors operate or execute a software program and / or set of instructions to perform the operations and / or functions of the elements of the point cloud decoder of Figure 11.

[0155] FIG. 12 illustrates an example of a transmitting device according to an embodiment.

[0156] The transmitting device shown in FIG. 12 is an example of the transmitting device 10000 in FIG. 1 (or the point cloud encoder in FIG. 4). The transmitting device shown in FIG. 12 performs any of the same or similar operations and methods as the operations and encoding methods of the point cloud encoder described in FIG. 1 to FIG. 9. The transmitting device according to the embodiment includes a data input unit 12000, a quantization processing unit 12001, a voxelization processing unit 12002, an octree occupation code generating unit 12003, a surface model processing unit 12004, an intra / inter coding processing unit 12005, an arithmetic coder 12006, a metadata processing unit 12007, a hue conversion processing unit 12008, a characteristic conversion processing unit (or attribute conversion processing unit) 12009, a prediction / lift / RAHT conversion processing unit 12010, an arithmetic coder 12011 and / or a transmission processing unit 12012.

[0157] According to the embodiment, the data input unit 12000 receives or acquires point cloud data. The data input unit 12000 performs an operation and / or an acquisition method that is the same as or similar to the operation and / or acquisition method of the point cloud video acquisition unit 10001 (or the acquisition process 20000 shown in FIG. 2).

[0158] Geometry encoding is performed by a data input unit 12000, a quantization processing unit 12001, a voxelization processing unit 12002, an octree occupation code generation unit 12003, a surface model processing unit 12004, an intra / inter coding processing unit 12005, and an arithmetic coder 12006. The geometry encoding according to the embodiment is the same as or similar to the geometry encoding described with reference to Figs. 1 to 9, and therefore a detailed description thereof will be omitted.

[0159] The quantization unit 12001 according to the embodiment quantizes geometry (e.g., position values ​​of points). The operation and / or quantization of the quantization unit 12001 is the same as or similar to the operation and / or quantization of the quantization unit 40001 shown in Fig. 4. The specific description is as described in Figs. 1 to 9.

[0160] The voxelization processing unit 12002 according to the embodiment voxels the position values ​​of the quantized points. The voxelization processing unit 120002 performs operations and / or processes that are the same as or similar to the operations and / or voxelization processes of the quantization unit 40001 shown in Fig. 4. The details are as described in Figs. 1 to 9.

[0161] The octree occupation code generator 12003 according to the embodiment performs octree coding on the positions of the voxelized points based on the octree structure. The octree occupation code generator 12003 generates an occupation code. The octree occupation code generator 12003 performs operations and / or methods that are the same as or similar to the operations and / or methods of the point cloud encoder (or the octree analysis unit 40002) described in Figures 4 and 6. The details are as described in Figures 1 to 9.

[0162] The surface model processing unit 12004 according to the embodiment performs trisoup geometry encoding to reconstruct the positions of points within a specific region (or node) on a voxel basis based on a surface model. The surface model processing unit 12004 performs operations and / or methods that are the same as or similar to those of the point cloud encoder (e.g., the surface approximation analysis unit 40003) shown in Fig. 4. The details are as described in Figs. 1 to 9.

[0163] The intra / inter coding unit 12005 according to the embodiment performs intra / inter coding of the point cloud data. The intra / inter coding unit 12005 performs coding that is the same as or similar to the intra / inter coding described in FIG. 7. The details are as described in FIG. 7. In the embodiment, the intra / inter coding unit 12005 is included in the arithmetic coder 12006.

[0164] According to an embodiment, the arithmetic coder 12006 entropy codes the octree and / or approximated octree of the point cloud data. For example, the encoding method includes an arithmetic encoding method. The arithmetic coder 12006 performs operations and / or methods that are the same as or similar to those of the arithmetic encoder 40004.

[0165] The metadata processing unit 12007 according to the embodiment processes metadata related to point cloud data, such as setting values, and provides the metadata to necessary processing such as geometry encoding and / or feature encoding. The metadata processing unit 12007 according to the embodiment also generates and / or processes signaling information related to geometry encoding and / or feature encoding. The signaling information according to the embodiment is encoded and processed separately from the geometry encoding and / or feature encoding. The signaling information according to the embodiment may also be interleaved.

[0166] The hue conversion processor 12008, the feature conversion processor 12009, the prediction / lift / RAHT conversion processor 12010, and the arithmetic coder 12011 perform feature coding. The feature coding according to the embodiment is the same as or similar to the feature coding described in Figs. 1 to 9, so a detailed description thereof will be omitted.

[0167] The hue conversion processing unit 12008 according to the embodiment performs hue conversion coding to convert a hue value included in a feature. The hue conversion processing unit 12008 performs hue conversion coding based on the reconstructed geometry. The reconstructed geometry has been described with reference to FIGS. 1 to 9. The hue conversion processing unit 12008 also performs operations and / or methods that are the same as or similar to those of the color conversion unit 40006 described with reference to FIG. 4. Detailed description thereof will be omitted.

[0168] The feature conversion processing unit 12009 according to the embodiment performs feature conversion to convert features based on positions where geometry coding has not been performed and / or reconstructed geometry. The feature conversion processing unit 12009 performs the same or similar operation and / or method as the feature conversion unit 40007 described in FIG. 4. A detailed description will be omitted. The prediction / lift / RAHT conversion processing unit 12010 according to the embodiment codes the converted features by one or a combination of RAHT coding, predictive conversion coding, and lift conversion coding. The prediction / lift / RAHT conversion processing unit 12010 performs one of the same or similar operations as the RAHT conversion unit 40008, the LOD generating unit 40009, and the lift conversion unit 40010 described in FIG. 4. In addition, the predictive conversion coding, the lift conversion coding, and the RAHT conversion coding are the same as those described in FIG. 1 to FIG. 9, and a detailed description will be omitted.

[0169] The arithmetic coder 12011 according to the embodiment encodes the encoded attribute based on arithmetic coding. The arithmetic coder 12011 performs operations and / or methods that are the same as or similar to the operations and / or methods of the arithmetic encoder 400012.

[0170] The transmission processing unit 12012 according to the embodiment transmits each bitstream including the coded geometry and / or coded attribute and metadata information, or transmits the coded geometry and / or coded attribute and metadata information in one bitstream. When the coded geometry and / or coded attribute and metadata information according to the embodiment is configured in one bitstream, the bitstream includes one or more sub-bitstreams. The bitstream according to the embodiment includes signaling information including a Sequence Parameter Set (SPS) for sequence level signaling, a Geometry Parameter Set (GPS) for signaling geometry information coding, an Attribute Parameter Set (APS) for signaling attribute information coding, and a Tile Parameter Set (TPS) for tile level signaling, and slice data. The slice data includes information about one or more slices. One slice according to the embodiment includes one geometry bitstream (Geom00) and one or more attribute bitstreams (Attr00, Attr10).

[0171] A slice refers to a series of syntax elements that represents all or a portion of an encoded point cloud frame.

[0172] The TPS according to the embodiment includes information on each tile (e.g., bounding box coordinate information and height / size information, etc.) for one or more tiles. The geometry bitstream includes a header and a payload. The header of the geometry bitstream according to the embodiment includes identification information of a parameter set included in the GPS (geom_parameter_set_id), a tile identifier (geom_tile_id), a slice identifier (geom_slice_id), and information on data included in the payload. As described above, the metadata processing unit 12007 according to the embodiment can generate and / or process signaling information and transmit it to the transmission processing unit 12012. In the embodiment, the element performing geometry coding and the element performing attribute coding can share data / information with each other as shown by the dotted line. The transmission processing unit 12012 according to the embodiment performs an operation and / or a transmission method that is the same as or similar to the operation and / or transmission method of the transmitter 10003. A detailed description is omitted as it is the same as that described in FIG. 1 and FIG. 2.

[0173] FIG. 13 illustrates an example of a receiving device according to the embodiment.

[0174] The receiving device shown in Fig. 13 is an example of the receiving device 10004 in Fig. 1 (or the point cloud decoder in Figs. 10 and 11). The receiving device shown in Fig. 13 performs any of the same or similar operations and methods as the operations and decoding methods of the point cloud decoder described in Figs. 1 to 11.

[0175] The receiving device according to the embodiment includes a receiving unit 13000, a receiving processing unit 13001, an arithmetic decoder 13002, an occupancy code based octree reconstruction processing unit 13003, a surface model processing unit (triangle reconstruction, upsampling, voxelization) 13004, an inverse quantization processing unit 13005, a metadata parser 13006, an arithmetic decoder 13007, an inverse quantization processing unit 13008, a prediction / lift / RAHT inverse transform processing unit 13009, a hue inverse transform processing unit 13010, and / or a renderer 13011. Each component of the decoding according to the embodiment performs the inverse process of the component of the encoding according to the embodiment.

[0176] The receiving unit 13000 according to the embodiment receives point cloud data. The receiving unit 13000 performs an operation and / or a receiving method that is the same as or similar to the operation and / or the receiving method of the receiver 10005 of Fig. 1. A detailed description will be omitted.

[0177] The receiving processor 13001 according to the embodiment obtains a geometry bit stream and / or an attribute bit stream from the received data. The receiving processor 13001 is included in the receiving unit 13000.

[0178] Geometry decoding is performed by an arithmetic decoder 13002, a proprietary code based octree reconstruction processor 13003, a surface model processor 13004, and an inverse quantization processor 13005. The geometry decoding according to the embodiment is the same as or similar to the geometry decoding described in Figs. 1 to 10, so a detailed description thereof will be omitted.

[0179] The arithmetic decoder 13002 according to the embodiment decodes the geometry bitstream based on arithmetic coding. The arithmetic decoder 13002 performs operations and / or coding that are the same as or similar to those of the arithmetic decoder 11000.

[0180] The exclusive code based octree reconstruction processor 13003 according to the embodiment obtains an exclusive code from the decoded geometry bitstream (or from the decoded result, information on the secured geometry) and reconstructs an octree. The exclusive code based octree reconstruction processor 13003 performs an operation and / or method identical or similar to the operation of the octree synthesis unit 11001 and / or an octree generation method. The surface model processor 13004 according to the embodiment performs trisoup geometry decoding and associated geometry reconstruction (e.g., triangle reconstruction, upsampling, voxelization) based on a surface model scheme when trisoup geometry encoding is applied. The surface model processor 13004 performs an operation identical or similar to the operation of the surface approximation synthesis unit 11002 and / or the geometry reconstruction unit 11003.

[0181] The inverse quantization unit 13005 according to the embodiment inverse quantizes the decoded geometry.

[0182] According to an embodiment, the metadata parser 13006 analyzes metadata, such as setting values, included in the received point cloud data. The metadata parser 13006 transmits the metadata to the geometry decoder and / or the feature decoder. A detailed description of the metadata is omitted here as it has been described with reference to FIG.

[0183] The arithmetic decoder 13007, the inverse quantization processor 13008, the prediction / lift / RAHT inverse transform processor 13009, and the hue inverse transform processor 13010 perform feature decoding. The feature decoding is the same as or similar to the feature decoding described in FIG. 1 to FIG. 10, so a detailed description thereof will be omitted.

[0184] The operation decoder 13007 according to the embodiment decodes the attribute bitstream into operation coding. The operation decoder 13007 performs decoding of the attribute bitstream based on the reconstructed geometry. The operation decoder 13007 performs operations and / or coding that are the same as or similar to the operations and / or coding of the operation decoder 11005.

[0185] The inverse quantization unit 13008 according to the embodiment inversely quantizes the decoded characteristic bit stream. The inverse quantization unit 13008 performs the same or similar operation and / or method as the inverse quantization unit 11006.

[0186] The prediction / lift / RAHT inverse transform processing unit 13009 according to the embodiment processes the reconstructed geometry and the inverse quantized features. The prediction / lift / RAHT inverse transform processing unit 13009 performs any of the same or similar operations and / or decoding as those of the RAHT transform unit 11007, the LOD generating unit 11008, and / or the inverse lift unit 11009. The hue inverse transform processing unit 13010 according to the embodiment performs inverse transform coding for inversely transforming color values ​​(or textures) included in the decoded features. The hue inverse transform processing unit 13010 performs the same or similar operations and / or inverse transform coding as those of the color inverse transform unit 11010. The renderer 13011 according to the embodiment renders the point cloud data.

[0187] FIG. 14 is a diagram illustrating an example of a structure that can be linked to a method / apparatus for transmitting and receiving point cloud data according to an embodiment.

[0188] 14 illustrates a configuration in which any of a server 1460, a robot 1410, an autonomous vehicle 1420, an XR device 1430, a smartphone 1440, a home appliance 1450, and / or an HMD 1470 are connected to a cloud network 1410. The robot 1410, the autonomous vehicle 1420, the XR device 1430, the smartphone 1440, or the home appliance 1450 may also be referred to as an apparatus. The XR device 1430 may correspond to a point cloud data (PCC) device or may be linked to a PCC device according to an embodiment.

[0189] Cloud network 1400 refers to a network that constitutes a part of a cloud computing infrastructure or exists within a cloud computing infrastructure, where cloud network 1400 is configured using a 3G network, a 4G or LTE network, a 5G network, or the like.

[0190] The server 1460 may be coupled to any of the robot 1410, autonomous vehicle 1420, XR device 1430, smartphone 1440, home appliance 1450, and / or HMD 1470 via the cloud network 1400 and may assist with at least some of the processing of the coupled devices 1410-1470.

[0191] The HMD (Head-Mount Display) 1470 represents any of the types of XR devices and / or PCC devices according to the embodiment. The HMD type device according to the embodiment includes a communications unit, a control unit, a memory unit, an I / O unit, a sensor unit, and a power supply unit.

[0192] Various embodiments of the devices 1410 to 1450 to which the above-mentioned technology is applied will be described below. Here, the devices 1410 to 1450 shown in Fig. 14 can be linked / coupled to the point cloud data transmitting / receiving device according to the above-mentioned embodiment.

[0193] <PCC+XR>

[0194] The XR / PCC device 1430 may be embodied in a Head-Mount Display (HMD), a Head-Up Display (HUD) installed in a vehicle, a TV, a mobile phone, a smartphone, a computer, a wearable device, a home appliance, a digital sign, a vehicle, a fixed robot, a mobile robot, etc. by applying PCC and / or XR (AR+VR) technology.

[0195] The XR / PCC device 1430 can obtain information about the surrounding space or real objects by analyzing 3D point cloud data or image data acquired by various sensors or from external devices to generate position data and characteristic data for 3D points, and can render and output an XR object for output. For example, the XR / PCC device 1430 can output an XR object including additional information about the recognized object corresponding to the recognized object.

[0196] <PCC+XR+Mobile Phone>

[0197] The XR / PCC device 1430 is implemented in a mobile phone 1440 or the like by applying PCC technology.

[0198] The mobile phone 1440 decrypts and displays point cloud content based on PCC technology.

[0199] <PCC+Autonomous Driving+XR>

[0200] The autonomous driving vehicle 1420 is implemented in a mobile robot, vehicle, unmanned aerial vehicle, etc. by applying PCC technology and XR technology.

[0201] The autonomous driving vehicle 1420 to which XR / PCC technology is applied means an autonomous driving vehicle equipped with means for providing an XR video, an autonomous driving vehicle that is a target for control / interaction within the XR video, etc. In particular, the autonomous driving vehicle 1420 that is a target for control / interaction within the XR video is distinguished from the XR device 1430 and is interlocked with each other.

[0202] The autonomous driving vehicle 1420 equipped with means for providing an XR / PCC video obtains sensor information from sensors including a camera and outputs an XR / PCC video generated based on the obtained sensor information. For example, the autonomous driving vehicle 1420 can provide an XR / PCC object corresponding to a real object or an object within the screen to the passenger by outputting an XR / PCC video equipped with an HUD.

[0203] At this time, when the XR / PCC object is output to the HUD, at least a part of the XR / PCC object is output to overlap with an actual object toward which the passenger's gaze is directed. On the other hand, when the XR / PCC object is output to a display provided in the autonomous vehicle, at least a part of the XR / PCC object is output to overlap with an object in the screen. For example, the autonomous vehicle 1220 may output XR / PCC objects corresponding to objects such as a road, another vehicle, a traffic light, a traffic sign, a motorcycle, a pedestrian, a building, etc.

[0204] The VR (Virtual Reality) technology, the AR (Augmented Reality) technology, the MR (Mixed Reality) technology, and / or the PCC (Point Cloud Compression) technology according to the embodiment can be applied to various devices.

[0205] That is, VR technology is a display technology that provides real objects and backgrounds only in CG images. On the other hand, AR technology is a technology that displays virtual CG images on top of images of real things. Also, MR technology is similar to the AR technology in that it mixes virtual objects into the real world. However, AR technology clearly distinguishes between real objects and virtual objects made of CG images, and uses virtual objects to complement real objects, while MR technology is different from AR technology in that virtual objects and real objects are considered to have the same characteristics. More specifically, for example, the MR technology is applied to hologram services.

[0206] However, recently, rather than clearly distinguishing between VR, AR, and MR technologies, they are also referred to as XR (extended reality) technology. Therefore, the embodiments of the present invention can be applied to any of VR, AR, MR, and XR technologies. These technologies are applied with encoding / decoding based on PCC, V-PCC, and G-PCC technologies.

[0207] The PCC method / apparatus according to the embodiment can be applied to vehicles providing autonomous driving services.

[0208] Vehicles providing autonomous driving services are connected to the PCC device via wired or wireless communication.

[0209] When the point cloud data (PCC) transceiver according to the embodiment is connected to a vehicle for wired or wireless communication, it can receive / process AR / VR / PCC service related content data that can be provided along with an autonomous driving service and transmit it to the vehicle. Also, when the point cloud data transceiver is installed in a vehicle, the point cloud transceiver can receive / process AR / VR / PCC service related content data according to a user input signal inputted to a user interface device and provide it to a user. The vehicle or user interface device according to the embodiment receives a user input signal. The user input signal according to the embodiment includes a signal instructing an autonomous driving service.

[0210] The point cloud data transmission method / apparatus according to the embodiments may be interpreted as terms referring to the transmitting device 10000 of FIG. 1, the point cloud video encoder 10002, the transmitter 10003, the capture-encode-transmit 20000-20001-20002 of FIG. 2, the encoder of FIG. 4, the transmitting device of FIG. 12, the device of FIG. 14, the encoder of FIG. 16, the transmitting method of FIG. 24, etc.

[0211] The point cloud data receiving method / apparatus according to the embodiments may be interpreted as a term referring to the receiving device 10004, receiver 10005, point cloud video decoder 10006 of FIG. 1, the transmit-decode-render 20002-20003-20004 of FIG. 2, the decoder of FIGS. 10 to 11, the receiving device of FIG. 13, the device of FIG. 14, the decoder of FIG. 17, the receiving method of FIG. 25, etc.

[0212] In addition, the point cloud data transmitting / receiving method / apparatus according to the embodiment may be abbreviated as the method / apparatus according to the embodiment.

[0213] Depending on the embodiment, the geometry data, geometry information, position information, etc. constituting the point cloud data may be interpreted as meaning the same thing. The characteristic data, characteristic information, attribute information, etc. constituting the point cloud data may be interpreted as meaning the same thing.

[0214] The method / apparatus according to the embodiment provides a predictive tree building method for inter-prediction.

[0215] For example, when point cloud data is captured and transmitted in real time from a LiDAR, or when 3D map data is transmitted and processed in real time, the method / apparatus according to the embodiment can efficiently compress and restore the point cloud data.

[0216] The embodiment includes a method for constructing a predictive tree for inter-prediction compression to improve the efficiency of prediction tree-based geometry information coding compression applicable to geometry compression of Geometry-based Point Cloud Compression (G-PCC) of cloud content requiring low-latency coding. For example, the embodiment includes the following methods: a prediction tree construction method for inter-prediction, and / or a signaling method. Each operation will be described with reference to each drawing.

[0217] An embodiment relates to a method for improving the compression efficiency of geometry-based point cloud compression (G-PCC) for compressing three-dimensional point cloud data.

[0218] Hereinafter, an encoder and a coding device will be referred to as a coding device, and a decoder and a decoding device will be referred to as a decoding device.

[0219] A point cloud is composed of a set of points, and each point has geometry information and attribute information. The geometry information is three-dimensional position (XYZ) information, and the attribute information is color (RGB, YUV, etc.) and / or reflectance value.

[0220] The G-PCC encoding process divides the point cloud into tiles according to the area, divides each tile into slices for parallel processing, compresses the geometry for each slice, and compresses attribute information based on the reconstructed geometry with the position information changed by compression.

[0221] The G-PCC decoding process involves transmitting a coded slice-based geometry bitstream and attribute bitstream, decoding the geometry, and decoding the attribute information based on the geometry reconstructed by the decoding process.

[0222] For geometry information compression, octree-based, predictive tree-based, or trisoup-based compression methods are used.

[0223] Embodiments include a prediction tree based geometry inter compression method to support low latency geometry compression.

[0224] Typical examples of point cloud services that require low latency include real-time navigation using 3D map point clouds, and capturing point clouds in real time using LiDAR and compressing and transmitting them.

[0225] To improve the encoder and decoder methods for low-latency services, a function is required to start compression on a portion of the point cloud data first. The octree-based geometry coding method scans and codes points in a breadth-first manner, but the prediction tree-based geometry compression method, which targets low-latency geometry compression, reduces stepwise point scanning as much as possible in a depth-first manner, generates a predicted value based on the geometry information between parent and child nodes of the tree, and entropy codes the residual value to create a geometry bitstream. The depth-first method has the advantage that it does not require stepwise scanning of all points, so there is no need to wait for all data to be captured, and progressively captured point cloud data can be geometry coded.

[0226] However, prediction tree-based geometry compression methods only include intra-prediction and therefore require extension to inter-prediction, which can reduce the size of the bitstream due to the correlation between frames.

[0227] The embodiment aims to support inter-prediction compression technology using reference frames for point clouds captured by LiDAR and having multiple frames. For inter-prediction compression, a prediction tree generation / restoration method is proposed to improve the compression rate by using reference frames.

[0228] Modifications and combinations are possible between the embodiments. The terms used in this specification can be understood based on the intended meaning of the terms within the scope of their widespread use in the art. Prediction tree generation is performed in a PCC encoder and is restored in a PCC decoder.

[0229] The method / apparatus according to the embodiment is performed including global and / or local motion estimation and compensation operations.

[0230] For inter-prediction, global motion between frames is predicted, and content divided into LPUs (Largest Prediction units) is divided again into PUs. PU division is possible to apply global motion, and if it cannot be matched with the current frame, a process of dividing into PUs and finding local motion is performed to check whether local motion exists and find local motion vectors. Global motion compensation may or may not be performed by predicting RDO for each LPU. RDO is predicted for each PU, and it is determined whether global motion compensation or local motion compensation is appropriate and applied.

[0231] FIG. 15 illustrates integrated prediction tree generation according to an embodiment.

[0232] The point cloud data transmission method / apparatus according to the embodiment (transmitting apparatus 10000 of FIG. 1, point cloud video encoder 10002, transmitter 10003, capture-encode-transmit 20000-20001-20002 of FIG. 2, encoder of FIG. 4, transmitting apparatus of FIG. 12, device of FIG. 14, encoder of FIG. 16, transmitting method of FIG. 24) generates a prediction tree as shown in FIG. 15 and compresses point cloud data based on the prediction tree.

[0233] The point cloud data receiving method / apparatus according to the embodiment (receiving device 10004, receiver 10005, point cloud video decoder 10006 of FIG. 1, transmit-decode-render 20002-20003-20004 of FIG. 2, decoder of FIGS. 10-11, receiving device of FIG. 13, device of FIG. 14, decoder of FIG. 17, and receiving method of FIG. 25) generates a prediction tree such as that of FIG. 15 and restores point cloud data based on the prediction tree.

[0234] The method / apparatus according to the embodiment is performed including a prediction tree generation operation.

[0235] The method / apparatus according to the embodiment performs global / local motion compensation on a reference frame to generate a prediction tree. The method of generating a prediction tree according to a current frame is extended to generate an inter-prediction tree according to a reference frame and a current frame. The integrated prediction tree according to the embodiment may be abbreviated to a prediction tree.

[0236] The procedure for generating an integrated prediction tree according to the embodiment is as follows.

[0237] In order to explain the generation procedure, the following conditions are set: These conditions are merely an example, and the configuration can be set in various ways depending on the embodiment.

[0238] For example, the maximum number of child nodes of an inter prediction tree is the maximum number (MaxNumChildrenForInter) of inter predictor nodes (lower nodes). Each node has parent, child, mode, and inter reference flag (inter_reference_flag) information. That is, for each node (one or more points), the parent, child, mode, and inter reference flag are set to be equal or different. The inter reference flag (inter_reference_flag) indicates whether a prediction tree of a reference frame is used for inter prediction. A parent is also referred to as an upper node, and a child is also referred to as a lower node, etc.

[0239] At S1500, the method / apparatus according to the embodiment registers the first point as a start node (=a) according to the order sorted (or input) in the current frame.

[0240] S1501: A parent is found from the prediction tree generated as the RDO reference in the reference frame and registered.

[0241] S1501-1: Three nodes close to node a are found, and the closest node that is smaller than the maximum number of child nodes (MaxNumChildrenForInter) is registered as the parent.

[0242] In step S1501-2, the inter reference flag (inter_reference_flag) is set to 1 in node a.

[0243] S1501-3: Node a is added to the prediction tree based on the found parent.

[0244] S1502: The next point is brought in, and a parent is found from the currently integrated prediction tree and registered.

[0245] S1502-1: Three nodes close to the node are found, and the closest node that is smaller than the maximum number of child nodes (MaxNumChildrenForInter) is registered as the parent.

[0246] S1502-2: If the node set as the parent exists in the reference frame, the inter_reference_flag is set to 1.

[0247] S1502-3: Repeat S1502 until all points in the current frame are registered in the prediction tree.

[0248] S1503, when the integrated prediction tree is completed, a mode is selected according to the RDO for the node registered at the point belonging to the current frame.

[0249] S1504: The first start node is coded as the root, and the inter_reference_flag of each node is signaled.

[0250] Through the above process, a prediction tree for inter prediction is generated from the current frame and the reference frame.

[0251] Referring to FIG. 15, an example of generating a (integrated) prediction tree based on a prediction tree for a point (node) included in a reference frame and a prediction tree for a point (node) included in a current frame is shown. In order to predict a start node a (next node is b) for encoding / decoding the current frame, a neighboring node closest to node a is searched around node a. The neighboring node closest to node a is searched from the reference frame and / or the current frame. Among the candidate neighboring nodes included in the reference frame, a candidate neighboring node with a smaller number of child nodes is set as the parent node of node a. A prediction tree for the next node b is then generated. In order to encode / decode node a, a residual value, which is a difference between the parent node and the current node, which is a predicted value of node a, is encoded and transmitted, and node a is restored by adding up the residual values.

[0252] The method / apparatus according to the embodiment includes a generation method using a reference prediction tree, and the reference prediction tree and the current prediction tree are used together.

[0253] The current prediction tree is set according to the prediction tree generation method, and matching nodes are found in the reference prediction tree in tree level and traverse order, and virtual links are connected.

[0254] The RDO is calculated based on the matched points in the prediction tree generated from the current frame and the prediction tree generated from the reference frame, and the mode is set. If the RDO of the node in the reference prediction tree shows a better value, the inter_reference_flag is set to true.

[0255] The method / apparatus according to the embodiment includes a prediction tree restoration scheme.

[0256] The suitability of global / local motion is signaled to the reference frame, and motion compensation is performed according to the suitability, after which the prediction tree is restored.

[0257] The method of restoring a prediction tree based on a current frame is extended to restore an inter-prediction tree based on a reference frame and a current frame.

[0258] The method / apparatus according to the embodiment includes a unified prediction tree reconstruction method.

[0259] When generated by a joint prediction tree, the procedure for tree reconstruction is as follows.

[0260] In order to explain the generation procedure, the following conditions are set: These conditions are merely an example, and the structure can be set in various ways depending on the embodiment.

[0261] For example, the maximum number of child nodes in an inter prediction tree is MaxNumChildrenForInter. Each node has parent, child, mode, and inter reference flag information. The inter reference flag indicates whether or not a prediction tree of a reference frame is used for inter prediction. A parent is also referred to as an upper node, and a child is also referred to as a lower node.

[0262] S15000, a method / apparatus according to an embodiment decodes the first node information received.

[0263] S15000-1 decrypts the first node as if it were the root node.

[0264] S15001, however, if the transmitted inter_reference_flag = true, the subsequent child nodes search for their parent from the prediction tree generated in the reference frame.

[0265] S15001-1, if the reference flag (reference_flag)=1, a parent is found from the prediction tree decoded / reconstructed as the RDO reference in the reference frame and registered.

[0266] S15001-2: If the reference flag (reference_flag)=0, proceed based on the parent of the prediction tree constructed by the current decoding.

[0267] S15002, the point values ​​are restored by the reconstructed prediction tree.

[0268] S15003: Repeat S15001 to S15002 until all points are decrypted.

[0269] The method / apparatus according to the embodiment includes a restoration method using a reference prediction tree.

[0270] The reference prediction tree and the current prediction tree are used together.

[0271] The current prediction tree is restored according to the prediction tree generation method, and matching nodes are found in the reference prediction tree in tree level and traverse order, and virtual links are connected.

[0272] If the inter_reference_flag is true, a point value is restored according to the mode based on a node matched to a virtual link in a prediction tree generated in a reference frame.

[0273] FIG. 16 illustrates a point cloud data transmitting device according to an embodiment.

[0274] The point cloud data transmission method according to the embodiment is performed by the transmitting device of Fig. 16. Each component of the transmitting device corresponds to hardware, software, a processor, and / or a combination thereof.

[0275] The point cloud data transmitting apparatus according to the embodiment corresponds to the transmitting apparatus 10000, the point cloud video encoder 10002, the transmitter 10003, the capture-encode-transmit 20000-20001-20002 of FIG. 2, the encoder of FIG. 4, the transmitting apparatus of FIG. 12, the device of FIG. 14, and the transmitting method of FIG. 24, etc.

[0276] The operation of the PCC data encoder will be described with reference to Fig. 16. PCC data is input to the encoder, which then encodes it, outputting a geometry information bit stream and an attribute information bit stream.

[0277] The geometry information encoding unit according to the embodiment is as follows: Operations related to the tree according to the embodiment are performed by a prediction tree generator (or processor) of an encoder.

[0278] Meanwhile, the geometry information encoding unit according to the embodiment corresponds to a component for point cloud video encoding of a point cloud video encoder, encoding, point cloud encoder, and / or transmitter.

[0279] When the geometry coding type is prediction-based coding, the geometry information encoder generates a prediction tree by a prediction tree generator, performs an RDO process based on the prediction tree generated by a prediction determination unit, selects an optimal prediction mode, and generates a geometry prediction value according to the optimal prediction mode.

[0280] The geometry information entropy coding unit (or geometry information entropy encoder) entropy codes the residual value between the predicted value and the geometry information bitstream to generate the geometry information bitstream.

[0281] A prediction tree generation method is input to the prediction tree generation unit (or prediction tree generator), and the prediction tree is generated according to the input method. The prediction tree generation method includes an integrated prediction tree generation method and a generation method using a reference prediction tree. The method is selected according to the content characteristics and the type of service. The applied prediction tree generation method is signaled to the decoder. A prediction tree is constructed according to the prediction tree generation method.

[0282] 16, a data input unit receives geometry (variously referred to as geometry data, geometry information, position information, etc.), attributes (variously referred to as attribute data, attribute information, attribute information, etc.), and / or parameter information related to the geometry and attributes. The geometry / attributes are acquired from a frame / sequence, etc., including point cloud data, by an acquisition unit or a camera / sensor, etc., and the parameters are set according to an encoder method.

[0283] The coordinate system transformation unit converts the coordinate set of the data into a coordinate system for encoding, since geometry and / or characteristics are represented based on positions (coordinates).

[0284] The geometry information conversion and quantization processing unit quantizes the geometry information based on the quantization parameter.

[0285] The space divider divides the space for encoding as the geometry and / or features are located in the space.

[0286] The geometry information encoding unit voxels the geometry through the voxel processing unit. The geometry information encoding unit checks the geometry coding type. The coding type according to the embodiment includes an octree-based coding method, a trisoup-based coding method, and / or a prediction tree method. In the case of the octree-based coding, an octree including the geometry is generated by an octree generating unit, the geometry position is reconstructed, and the restored geometry information is transmitted to the attribute information encoding unit, and the geometry information is encoded based on the entropy method to generate a bitstream including the geometry information. In the case of the trisoup-based coding, a trisoup including the geometry is generated, the geometry is similarly restored, the restored geometry information is transmitted to the attribute information encoding unit, and the geometry information is encoded based on the entropy method to generate a bitstream including the geometry information. In the case of the prediction tree-based coding, a prediction tree for encoding the geometry node is generated. The prediction tree generation method according to the embodiment is set based on a parameter indicating the prediction tree generation method. A parameter indicating the applied prediction tree generation method is generated, and is included in the bitstream and transmitted to the decoder. The parameters will be described again with reference to FIG. 18, etc. The attribute information encoding unit encodes the attributes based on the input data (geometry, attributes, and / or parameters) and / or the restored geometry information. The attribute information encoding unit converts the attributes, for example, hue information, into a hue system for encoding. For the attribute information encoding method, refer to the description of attribute information encoding described in FIG. 1, etc. The attribute information is encoded based on an entropy method to generate a bitstream including the attribute information.

[0287] For a detailed description of each operation in FIG. 16, please refer to the description of the encoder in this specification.

[0288] FIG. 17 illustrates a point cloud data receiving device according to an embodiment.

[0289] The point cloud data receiving method according to the embodiment is performed by the receiving device of Fig. 17. Each component of the transmitting device corresponds to hardware, software, a processor, and / or a combination thereof.

[0290] The point cloud data receiving device according to the embodiment corresponds to the receiving device 10004, receiver 10005, point cloud video decoder 10006 in Figure 1, the transmit-decode-render 20002-20003-20004 in Figure 2, the decoder in Figures 10 to 11, the receiving device in Figure 13, the device in Figure 14, and the receiving method in Figure 25.

[0291] The PCC data decoder will be described with reference to Fig. 17. The encoded geometry information bit stream and attribute information bit stream are input to the decoder, which decodes and outputs reconstructed PCC data.

[0292] The geometry information decoding unit according to the embodiment is as follows: Operations on the tree according to the embodiment are performed by the prediction tree reconstruction unit (or processor).

[0293] Meanwhile, the geometry information decoding unit according to the embodiment corresponds to a component for decoding point cloud video of a point cloud video decoder, a point cloud decoder, and a receiver.

[0294] The prediction tree reconstructor (or prediction tree reconstructor) is used to receive and restore the prediction tree generation method from the transmitting side (encoder side), thereby reconstructing the prediction tree and decoding the geometry prediction value.

[0295] As shown in FIG. 17, in the receiving method / apparatus (point cloud data receiving method / apparatus) according to the embodiment, the point cloud data decoder performs the decoding process of FIG.

[0296] The restoration process of the prediction tree reconstruction unit according to the embodiment corresponds to the inverse process of the prediction tree generation unit. The operations of the point cloud data transmitting device and the receiving device according to the embodiment correspond to each other and follow the inverse process.

[0297] For a detailed description of each operation in FIG. 17, please refer to the description of the decoder in this specification.

[0298] FIG. 18 illustrates a bitstream containing point cloud data according to an embodiment.

[0299] A point cloud data transmitting device according to an embodiment (transmitting device 10000 of FIG. 1, point cloud video encoder 10002, transmitter 10003, acquisition-encoding-transmission 20000-20001-20002 of FIG. 2, encoder of FIG. 4, transmitting device of FIG. 12, device of FIG. 14, transmitting method of FIG. 24, etc.) encodes point cloud data, generates parameter information, and generates and transmits a bitstream such as that shown in FIG. 18.

[0300] A point cloud data receiving device according to an embodiment (receiving device 10004, receiver 10005, point cloud video decoder 10006 of FIG. 1, transmit-decode-render 20002-20003-20004 of FIG. 2, decoder of FIGS. 10 to 11, receiving device of FIG. 13, device of FIG. 14, receiving method of FIG. 25, etc.) receives a bitstream such as that of FIG. 18 and decodes point cloud data based on parameter information.

[0301] In order to add / execute the embodiment, relevant information is signaled. The signaling information according to the embodiment is used at the transmitting end or the receiving end, etc.

[0302] The structure of the encoded point cloud is as follows: A point cloud data encoder performing geometry encoding and / or feature encoding generates an encoded point cloud (or a bit stream including a point cloud) as follows: In addition, signaling information related to the point cloud data is generated and processed by a metadata processing unit of the point cloud data transmitting device, and is included in the point cloud as follows:

[0303] Each abbreviation has the following meaning. Each abbreviation is also referred to as other terms within the scope of equivalent meaning. SPS: Sequence Parameter Set, GPS: Geometry Parameter Set, APS: Attribute Parameter Set, TPS: Tile Parameter Set, Geom: Geometry bitstream = geometry slice header + [geometry PU header + Geometry PU data] | geometry slice data, Attr: Attribute bitstream = attribute data unit header + [attribute PU header + attribute PU data] | attribute data unit data.

[0304] A slice according to the embodiment may also be referred to as a data unit or the like.

[0305] Optional information related to prediction tree generation according to the embodiment is added to the SPS or GPS for signaling. Optional information related to prediction tree generation according to the embodiment is added to the TPS or a geometry header for each slice for signaling. Tiles or slices are provided to process the point cloud by dividing it into regions.

[0306] When dividing by region, different neighbor point set generation options can be set for each region, allowing you to select a method with low complexity but slightly lower reliability of the results, or conversely, a method with high complexity but high reliability. Different settings are made depending on the processing capacity of the receiver.

[0307] Therefore, if a point cloud is divided into tiles, different options can be applied to each tile. If a point cloud is divided into slices, different options can be applied to each slice. Whether or not to use the reference prediction tree for each point for reconstruction can be added to the geometry prediction tree data.

[0308] The syntax of the parameters included in the bitstream in FIG. 18 will be described below with reference to each drawing.

[0309] FIG. 19 shows a sequence parameter set according to an embodiment.

[0310] According to the embodiment, the prediction tree generation related option information is added to a sequence parameter set and signaled. The prediction tree generation process can be efficiently signaled. The names of the signaling information can be understood within the meaning and function of the signaling information.

[0311] The profile_idc indicates the profile to which the bitstream adheres. A bitstream is not required to contain profile_idc values ​​other than those specified. Other values ​​of profile_idc are reserved for future use by ISO / IEC.

[0312] Profile compatibility flags (profile_compatibility_flags) equal to 1 indicates that the bitstream adheres to the profile indicated by profile_idc equal to j.

[0313] The number of attribute sets (sps_num_attribute_sets) indicates the number of attributes coded in the bitstream. The value of sps_num_attribute_sets ranges from 0 to 63.

[0314] The attribute dimension (attribute_dimension[ i ]) indicates the number of components of the i-th attribute.

[0315] The attribute instance ID (attribute_instance_id[ i ]) indicates the instance ID for the i-th attribute.

[0316] Predicted Geometry Tree Build Method (pred_geom_tree_build_method): Indicates the predicted geometry tree build method in the sequence. For example, 0 = no reference frame prediction tree build method, 1 = integrated prediction tree build method, 2 = build method using reference prediction tree. Information by each integer value can be changed.

[0317] The SPS according to the embodiment may further include the following parameters:

[0318] simple_profile_compliant indicates whether the bitstream complies with the simple profile (when it is 1) or not (when it is 0).

[0319] Dense_profile_compliant indicates whether the bitstream complies with the Dense profile (when it is 1) or not (when it is 0).

[0320] predictive_profile_compliant indicates whether the bitstream conforms to the predictive profile (when it is 1) or not (when it is 0).

[0321] main_profile_compliant indicates whether the bitstream complies with the basic profile (when it is 1) or not (when it is 0).

[0322] reserved_profile_18bits is equal to 0 in bitstreams conforming to this version of this document. Other values ​​for reserved_profile_18bits are reserved for future use by ISO / IEC.

[0323] slice_reordering_constraint indicates whether the bitstream is sensitive to reordering or removal of slices within the encoded point cloud frame (when it is 1) or not (when it is 0). When slice_reordering_constraint is 1, the bitstream may not be fully decoded if slices are reordered or removed.

[0324] When unique_point_positions_constraint is 1, it indicates that every point should have a unique position in each encoded point cloud frame. When unique_point_positions_constraint is 0, it indicates that two or more points may have the same position in the encoded point cloud frame.

[0325] Even though points in each slice have unique positions, points in other slices in the same frame may match, in which case the unique_point_positions_constraint is set to 0.

[0326] If unique_point_positions_constraint is 1, then points with the same position in the same frame are prohibited, even if the frame index / number attribute values ​​are different.

[0327] level_idc indicates the level to which the bitstream adheres, as specified in Annex A. A bitstream shall not contain level_idc values ​​other than those specified in Annex A. Other values ​​of level_idc are reserved for future use by ISO / IEC.

[0328] The sps_seq_parameter_set_id identifies the SPS for reference by other DUs. sps_seq_parameter_set_id is 0 in bitstreams conforming to this version of this specification. Other values ​​of sps_seq_parameter_set_id are reserved for future use in ISO / IEC.

[0329] frame_ctr_lsb_bits indicates the length of the syntax element frame_ctr_lsb in bits.

[0330] slice_tag_bits specifies the length of the syntax element slice_tag in bits.

[0331] bypass_stream_enabled specifies whether bypass bins for arithmetically coded syntax elements are conveyed in a separate data stream. If it is 1, the two data streams are multiplexed using a fixed length chunk sequence (11.3). If it is 0, the bypass bins form part of the arithmetically coded bit stream.

[0332] entropy_continuation_enabled indicates whether (when it is 1) or not (when it is 0) the entropy parsing of a DU can depend on the final entropy parsing state of the DU in the previous slice. When slice_reordering_constraint is 0, it is a bitstream conformance requirement that entropy_continuation_enabled be 0.

[0333] sps_extension_present indicates whether (when it is 1) or not (when it is 0) the sps_extension_data syntax element is present in the SPS syntax structure. sps_extension_present is 0 in bitstreams conforming to this version of this specification. A value of 1 for sps_extension_present is reserved for future use in ISO / IEC.

[0334] sps_extension_data may have any value. Its presence and value do not affect decoder conformance to the profile specified in this version of this specification.

[0335] seq_origin_bits indicates the length in bits of each seq_origin_xyz syntax element excluding the sign bit.

[0336] seq_origin_xyz[ ] and seq_origin_log2_scale indicate the XYZ origin of the sequence and coding coordinate systems in sequence coordinate system units at the origin of a coordinate system specific to the application program. When seq_origin_bits is 0, seq_origin_xyz[ ] and seq_origin_log2_scale are inferred to be 0. The th XYZ component of the origin is indicated as SeqOrigin[ ].

[0337] seq_bbox_size_bits indicates the length of each seq_bbox_size_minus1_xyz syntax element in bits.

[0338] seq_bbox_size_minus1_xyz[ ]+1 indicates the th XYZ component of the encoded volume dimensions in the sequence coordinate system. If seq_bbox_size_bits is 0, the encoded volume dimensions are not specified.

[0339] seq_unit_numerator_minus1, seq_unit_denominator_minus1 and seq_unit_is_metres are the lengths of the unit vectors in the sequence coordinate system.

[0340] If seq_unit_is_metres is 1, it indicates the length of the sequence unit vector.

[0341] If seq_unit_is_metres is 0, it indicates that the sequence unit vector has a length relative to the application specific coordinate system unit vector length AppUnit as follows:

[0342] seq_coded_scale_exponent, seq_coded_scale_mantissa_bits and seq_coded_scale_mantissa indicate the scale factors for transforming the coding coordinate system to the sequence coordinate system. The scale factors are indicated by the syntax element as normalized binary floating-point values ​​greater than or equal to 1. seq_coded_scale_mantisssa_bits indicates the length in bits of the syntax element seq_coded_scale_mantissa. The scale factors are indicated as SeqCodedScale.

[0343] geom_axis_order indicates the correspondence between the XYZ axes and the STV axes of the encoded point cloud.

[0344] num_attributes indicates the number of attributes listed in the SPS attribute list.

[0345] AttrDim, AttrBitDepth, and AttrMaxVal indicate the number of components, the bit depth, and the maximum value, respectively, of the attribute identified by the AttrIdx variable.

[0346] AttrDim:=attr_components_minus1[AttrIdx]+1

[0347] AttrBitDepth:=attr_bitdepth_minus1[AttrIdx] + 1

[0348] AttrMaxVal:=Exp2(AttrBitDepth)-1

[0349] _components_minus1[attrIdx]+1 specifies the number of components of the identified attribute.

[0350] Attributes with more than two components can only be coded with primitive attribute data (attr_coding_type=3).

[0351] The attr_instance_id value is used to distinguish between the same attributes in the attribute label. For example, a point cloud may have multiple hue attributes sampled at different perspectives. In this case, an application can use attr_instance_id to distinguish between the perspectives.

[0352] attr_bitdepth_minus1[ attrIdx ]+1 specifies the bit depth of all components of the identified attribute.

[0353] attr_label_known[attrIdx], attr_label[attrIdx], and attr_label_oid[attrIdx] identify the data type conveyed by the identified attribute. attr_label_known[attrIdx] specifies whether the attribute is an attribute specified in this specification by the value of attr_label[attrIdx] (when it is 0) or an externally specified attribute identified by the object identifier attr_label_oid[attrIdx].

[0354] The attribute type identified by attr_label is specified in Table 9. There is a bitstream conformance requirement that the attribute identified by attr_label should have a valid number of components. Unspecified attr_label values ​​are reserved for future use in ISO / IEC. Decoders shall decode attributes with reserved values ​​of attr_label.

[0355] The attribute type identified by attr_label_oid is not specified in this specification.

[0356] attr_property_cnt indicates the number of attribute_property syntax structures in the SPS for the attribute.

[0357] FIG. 20 shows a set of geometry parameters according to an embodiment.

[0358] In the process of encoding / decoding geometry information, relevant option information for a prediction tree generation function is added to a geometry parameter set and signaled. It is efficiently signaled to support inter-prediction of geometry. The name of the signaling information can be understood within the meaning and function of the signaling information.

[0359] Geometry Parameter Set ID (gps_geom_parameter_set_id): Indicates an identifier for the GPS to reference in other syntax elements. The value of gps_seq_parameter_set_id is in the range of 0 to 15.

[0360] Sequence Parameter Set ID (gps_seq_parameter_set_id): Indicates the sps_seq_parameter_set_id value for the active SPS. The value of gps_seq_parameter_set_id is in the range of 0 to 15.

[0361] Geometry Tree Type (geom_tree_type): When geom_tree_type is 0, it indicates that the position information is coded using an Octree. When geom_tree_type is 1, it indicates that the position information is coded using a predictive tree.

[0362] Predicted geometry generation method (pred_geom_tree_build_method): Indicates the predicted geometry tree generation method for the frame. For example, it is signaled as 0 = prediction tree generation method without reference frame, 1 = integrated prediction tree generation method, 2 = generation method using reference prediction tree. Information by each integer value can be changed.

[0363] The GPS according to the embodiment may further include the following parameters:

[0364] The gps_geom_parameter_set_id identifies the GPS for reference by other DUs.

[0365] gps_seq_parameter_set_id identifies the active SPS with sps_seq_parameter_set_id.

[0366] slice_geom_origin_scale_present indicates whether slice_geom_origin_log2_scale is present in the GDU header (if it is 1) or not (if it is 0). If slice_geom_origin_scale_present is 0, it indicates that the slice origin scale is specified by gps_geom_origin_log2_scale.

[0367] gps_geom_origin_log2_scale indicates the scale factor used to derive the slice origin in slice_geom_origin_xyz when slice_geom_origin_scale_present is 0.

[0368] geom_dup_point_counts_enabled indicates whether duplicate points are signaled in the GDU by a duplicate count per point (when it is 1) or not (when it is 0).

[0369] If geom_dup_point_counts_enabled is 0, then there is no prohibition on coding the same point position multiple times within a single slice by means other than the direct_dup_point_cnt, occ_dup_point_cnt, or ptn_dup_point_cnt syntax elements.

[0370] geom_tree_type equal to 0 indicates that the slice geometry is encoded using occupation trees (7.3.3.4). geom_tree_type equal to 1 indicates that the slice geometry is encoded using predictive trees (7.3.3.8).

[0371] gps_extension_present indicates whether the gps_extension_data syntax element is present (when it is 1) or not (when it is 0) in the GPS syntax structure. gps_extension_present is 0 in bitstreams following this version of this specification. A value of 1 for gps_extension_present is reserved for future use in ISO / IEC.

[0372] gps_extension_data may have any value. Its presence and value do not affect decoder conformance to the profile specified in this version of this specification.

[0373] geom_angular_enabled indicates whether (when it is 1) or not (when it is 0) the slice geometry is encoded with information for a set of beams that lie and rotate along the V axis of the angular origin. If activated, point positions are assumed to have been sampled along the rays cast by the beams.

[0374] slice_angular_origin_present indicates whether the slice-related angular origin is signaled in the GDU header (when it is 1) or not (when it is 0). If slice_angular_origin_present is 0, it indicates that the angular origin is gps_angular_origin_xyz. If slice_angular_origin_present is absent, it is inferred to be 0.

[0375] gps_angular_origin_bits_minus1+1 indicates the length in bits of each gps_angular_origin_xyz syntax element.

[0376] gps_angular_origin_xyz[] indicates the th XYZ coordinates of the angular origin in the coding coordinate system.

[0377] num_beams_minus1+1 indicates the number of beams enumerated by the GPS.

[0378] beam_elevation_init and beam_elevation_diff[] indicate the beam elevation as a slope on the ST plane. The elevation slope of the th beam is given as BeamElev[]. It is a binary fixed-point value with 18 fractional bits.

[0379] beam_voffset_init and beam_voffset_diff[] indicate the V-axis offset of the beam enumerated at the angle origin. The offset is indicated in units of the coding coordinate system. The offset of the th beam is indicated as BeamOffsetV[].

[0380] beam_steps_per_rotation_init_minus1 and beam_steps_per_rotation_diff[] indicate the number of steps taken per rotation by the rotating beam. The value for the th beam is shown as BeamStepsPerRev[].

[0381] ptree_ang_azimuth_pi_bits_minus11+11 is the number of bits representing half a rotation of the beam around the V axis. A half rotation is * radians.

[0382] ptree_ang_radius_scale_log2 indicates the factor used to scale the radial angle coordinates of points during conversion to Cartesian coordinates.

[0383] ptree_ang_azimuth_step_minus1+1 indicates the expected change from the azimuth angle of the beam rotating between the coded points. The azimuth angle prediction residuals used for angle predictive tree coding are coded as multiples of ptree_ang_azimuth_step_minus1+1 plus the remainder.

[0384] occtree_point_cnt_list_present indicates whether the GDU footer lists the number of points at each occupancy tree level (when it is 1) or not (when it is 0). If occtree_point_cnt_list_present is absent, 0 is inferred.

[0385] occtree_direct_coding_mode greater than 0 indicates that the point position can be coded by a matching direct node in the occupancy tree. occtree_direct_coding_mode equal to 0 indicates that no direct node should be present in the occupancy tree.

[0386] In general, the higher the value of occtree_direct_coding_mode, the higher the direct node suitability.

[0387] occtree_direct_joint_coding_enabled indicates whether (when it is 1) or not (when it is 0) direct nodes that encode two points should jointly encode their positions according to a predefined ordering of the points.

[0388] occtree_coded_axis_list_present equal to 1 indicates that the GDU header contains an occtree_coded_axis syntax element that is used to derive node sizes for each occtree level. occtree_coded_axis_list_present equal to 0 indicates that the occtree_coded_axis syntax element is not present in the GDU syntax and that the occtree represents a cubic volume specified by the tree depth.

[0389] occtree_neigh_window_log2_minus1+1 indicates the number of occupied tree node positions that form each availability window within a tree level. Nodes outside the window are not available to processes related to nodes within the window. If occtree_neigh_window_log2_minus1 is 0, it indicates that only sibling nodes should be considered available to the current node.

[0390] occtree_adjacent_child_enabled indicates whether adjacent children of adjacent occupancy tree nodes are used for bit occupancy status (when it is 1) or not (when it is 0). If occtree_adjacent_child_enabled is absent, 0 is inferred.

[0391] occtree_intra_pred_max_nodesize_log2-1 indicates the maximum size of an occupancy tree node that fits into an occupancy prediction within a slice. If occtree_intra_pred_max_nodesize_log2 is absent, it is inferred to be 0.

[0392] occtree_bitwise_coding indicates that the node occupancy bitmap is coded using the occupancy_bit syntax element (when it is 1) or the pre-coded syntax element occupancy_byte (when it is 0).

[0393] occtree_planar_enabled indicates whether the coding of the node occupancy bitmap is done by signaling partially occupied and unoccupied planes (when it is 1) or not (when it is 0). If occtree_planar_enabled is absent, it is inferred to be 0.

[0394] occtree_planar_threshold[] indicates the thresholds used in part to determine per-axis suitability for planar occupancy coding. The thresholds are shown from the most likely planar axis (=0) to the least (=2). Each threshold indicates the minimum likelihood for a suitable axis for which occ_single_plane is expected to be 1. The range [8, 120] for occtree_planar_threshold corresponds to the likelihood interval [0, 1].

[0395] occtree_direct_node_rate_minus1, if present, indicates that of all 32 matching nodes, only occtree_direct_node_rate_minus1+1 will be coded at the direct node.

[0396] occtree_planar_buffer_disabled indicates whether the contextualization of per-node occupied planar positions using the previously encoded planar positions of the nodes is deactivated (when it is 1) or not (when it is 0). If occtree_planar_buffer_disabled is absent, it is inferred to be 0.

[0397] geom_scaling_enabled indicates whether the size of the encoded geometry should be adjusted (when it is 1) or not (when it is 0) during the geometry decoding process.

[0398] geom_qp denotes the geometry QP before adding slice and per-node offsets.

[0399] geom_qp_mul_log2 indicates the scale factor to apply to the geometry QP. For every doubling of the size of a scaling step there are Exp2(3-geom_qp_mul_log2) QP values.

[0400] ptree_qp_period_log2 indicates the period of the node in which the prediction tree node QP offsets are signaled. This period is one for all Exp2(ptree_qp_period_log2) nodes.

[0401] occtree_direct_node_qp_offset indicates the offset relative to the slice geometry QP to scale the position of the direct node encoded point.

[0402] FIG. 21 shows a tile parameter set according to an embodiment.

[0403] Prediction tree generation related option information is added to a tile parameter set and signaled. Inter-prediction can be efficiently signaled. The names of the signaling information can be understood within the meaning and function of the signaling information.

[0404] The number of tiles (num_tiles) indicates the number of tiles signaled for the bitstream. If not present, num_tiles is inferred as 0.

[0405] The tile bounding box offset x (tile_bounding_box_offset_x[i]) indicates the x offset of the i-th tile in Cartesian coordinates. If not present, the value of tile_bounding_box_offset_x[0] is inferred to be sps_bounding_box_offset_x.

[0406] The tile bounding box offset y (tile_bounding_box_offset_y[i]) indicates the y offset of the i-th tile in Cartesian coordinates. If it is not present, the value of tile_bounding_box_offset_y[0] is inferred as sps_bounding_box_offset_y.

[0407] Predicted Geometry Tree Build Method (pred_geom_tree_build_method): Indicates the method for generating a predicted geometry tree for a tile. For example, 0 = prediction tree generation method without reference frame, 1 = integrated prediction tree generation method, 2 = generation method using reference prediction tree, etc. The information indicated by each integer value can be changed.

[0408] The tile parameter set according to the embodiment may further include the following parameters: The tile parameter set according to the embodiment is called a tile inventory.

[0409] The sequence parameter set ID (ti_seq_parameter_set_id) indicates the active SPS with sps_seq_parameter_set_id.

[0410] The number of frame lsb bits (ti_frame_ctr_lsb_bits) indicates the length of the syntax element ti_frame_ctr_lsb in bits. ti_frame_ctr_lsb_bits is the same as the frame_ctr_lsb_bits of the active SPS.

[0411] The frame number lsb (ti_frame_ctr_lsb) is the ti_frame_ctr_lsb_bits LSB of the FrameCtr for the next encoded point cloud frame.

[0412] The number of tiles (tile_cnt) indicates the number of tiles enumerated by the tile inventory.

[0413] The tile ID (tile_id_bits) indicates the length in bits of each tile_id syntax element. When tile_id_bits is 0, it indicates that the tile is identified by the index tileIdx.

[0414] The tile origin bits (tile_origin_bits_minus1)+1 indicate the length in bits of each tile origin xyz (tile_origin_xyz) syntax element excluding the sign bit.

[0415] The tile size bits (tile_size_bits_minus1)+1 indicates the length in bits of each tile size (tile_size_minus1_xyz) syntax element.

[0416] Tile ID (tile_id[tileIdx]) indicates the identifier of the tileIdx-th tile in the tile inventory. When tile_id_bits is 0, the value of tile_id[tileIdx] is inferred to be tileIdx. Every value of tile_id is unique in the tile inventory.

[0417] The tile origin xyz and tile size xyz (tile_origin_xyz[tileId][] and tile_size_minus1_xyz[tileId][]) indicate the bounding box in the sequence coordinate system that contains the slice identified by the same slice_tag as tileId.

[0418] Tile origin xyz (tile_origin_xyz[tileId][]) gives the XYZ coordinates of the lower corner of the tile bounding box relative to the tile inventory origin.

[0419] Tile size xyz (tile_size_minus1_xyz[tileId][]): +1 indicates the th X, Y and Z dimensions of the tile bounding box.

[0420] Origin xyz(ti_origin_bits_minus1):+1 indicates the length in bits of each ti_origin_xyz syntax element excluding the sign bit.

[0421] The origin xyz and origin scale (ti_origin_xyz[] and ti_origin_log2_scale) indicate the XYZ origin of the sequence coordinate system specified by seq_origin_xyz[] and seq_origin_log2_scale. The values ​​of ti_origin_xyz[] and ti_origin_log2_scale are the same as seq_origin_xyz[] and seq_origin_log2_scale, respectively.

[0422] FIG. 22 illustrates a geometry slice header according to an embodiment.

[0423] In the process of encoding / decoding geometry information, relevant option information for generating a prediction tree is added to a geometry slice header and signaled. It can be efficiently signaled to support inter-prediction. The name of the signaling information can be understood within the meaning and function of the signaling information.

[0424] The geometry parameter set ID (gsh_geometry_parameter_set_id) indicates the gps_geom_parameter_set_id value of the active GPS.

[0425] The tile ID (gsh_tile_id) indicates the value of the tile ID referenced by GSH. The value of the tile ID (gsh_tile_id) is in the range of 0 to XX.

[0426] The slice ID (gsh_slice_id) identifies the slice header to be referenced in other syntax elements. The value of gsh_slice_id is in the range of 0 to XX.

[0427] Predicted Geometry Tree Build Method (pred_geom_tree_build_method): Indicates how to generate a predicted geometry tree for a slice. For example, it is signaled as 0 = prediction tree generation method without reference frame, 1 = integrated prediction tree generation method, 2 = generation method using reference prediction tree, etc. Each integer value can be changed.

[0428] The geometry slice header is also referred to as a geometry data unit header. The geometry slice header may further include the following parameters:

[0429] The geometry parameter set ID (gdu_geometry_parameter_set_id) indicates the active GPS with gps_geom_parameter_set_id.

[0430] The slice ID (slice_id) indicates the slice to which other DUs (data units, corresponding to slice units) refer.

[0431] The slice_tag identifies a slice that is a member of a slice group that has the same value for slice_tag. In the case of a tile inventory DU, the slice group is the tile identified by the tile ID. In the case of no tile inventory DU, the interpretation of slice_tag is application dependent.

[0432] frame_ctr_lsb specifies the frame_ctr_lsb_bits LSB of the conceptual frame counter FrameCtr. Consecutive slices with different frame_ctr_lsb values ​​form part of individual output point cloud frames. Consecutive slices with the same value of frame_ctr_lsb with no intermediate frame boundary marker data units form part of the same encoded point cloud frame.

[0433] slice_entropy_continuation, when set to 1, specifies that the entropy parsing state restoration process (11.6.2.2 and 11.6.3.2) is applied at the beginning of the slice's GDUs and ADUs. slice_entropy_continuation, when set to 0, specifies that the parsing of the slice's GDUs and ADUs is independent of other slices. If slice_entropy_continuation is absent, 0 is inferred.

[0434] If a GDU is the first GDU in a coded point cloud frame, it is a bitstream conformance requirement that slice_entropy_continuation be 0. Decoders MAY ignore (remove from the bitstream and discard) all slices in a coded point cloud frame with slice_entropy_continuation equal to 1 that are not preceded by a slice of the same frame with slice_entropy_continuation equal to 0.

[0435] The prev_slice_id is the same as the slice_id of the GDU of the previous slice in bitstream order. The decoder may ignore (remove from the bitstream and discard) slices where prev_slice_id is present and is not the same as the slice_id of the previous slice in the same frame.

[0436] If the slice_tag is not identical to the slice_tag of the GDU identified by prev_slice_id, it is RECOMMENDED that slice_entropy_continuation be 0. For example, when slice_tag is used to select a subset of slices, decoding is prevented if there is a dependency on a slice that is not selected.

[0437] slice_geom_origin_bits_minus1+1 specifies the length of each slice_geom_origin_xyz syntax element in bits.

[0438] slice_geom_origin_xyz[] and slice_geom_origin_log2_scale specify the th XYZ coordinates of the slice origin in the coding coordinate system. The slice origin in STV coordinates is indicated as SliceOrigin[]. If slice_geom_origin_log2_scale is not present, it is inferred as gps_geom_origin_log2_scale.

[0439] slice_angular_origin_bits_minus1+1 specifies the length of each slice_angular_origin_xyz syntax element in bits.

[0440] slice_angular_origin_xyz[] specifies the XYZ coordinates of the relative angle origin in the slice coordinate system. If slice_angular_origin_xyz[] is not present, it is inferred to be 0.

[0441] slice_geom_qp_offset specifies the slice geometry QP as an offset relative to the GPS geom_qp. If slice_geom_qp_offset is absent, it is inferred to be 0.

[0442] FIG. 23 shows geometry prediction tree data according to an embodiment.

[0443] Whether or not to use the reference prediction tree for each point for reconstruction is signaled in the geometry prediction tree data. It can be efficiently signaled to support inter-prediction. The name of the signaling information can be understood within the meaning and function of the signaling information.

[0444] Prediction tree node ID (PtnNodeIdx): Indicates the node index.

[0445] End of Trees (gpt_end_of_trees_flag): If 0, it indicates that other prediction trees follow in the data unit. If 1, gpt_end_of_trees_flag indicates that there are no more prediction trees in the data unit.

[0446] The number of children (ptn_child_cnt[nodeIdx]) is the number of direct child nodes of the current prediction tree node in the geometry prediction tree.

[0447] The prediction mode (ptn_pred_mode[nodeIdx]) indicates the mode used to predict the position relative to the current node.

[0448] Inter-reference flag (inter_reference_flag[nodeIdx]): Indicates whether to refer to information to reconstruct a point from the prediction tree generated by the reference frame.

[0449] FIG. 24 illustrates a method for transmitting point cloud data according to an embodiment.

[0450] The point cloud data transmission method / apparatus according to the embodiment (transmitting device 10000 of FIG. 1, point cloud video encoder 10002, transmitter 10003, acquisition-encoding-transmission 20000-20001-20002 of FIG. 2, encoder of FIG. 4, transmitting device of FIG. 12, device of FIG. 14, encoder of FIG. 16) corresponds to the method as shown in FIG. 24.

[0451] S2400, a method for transmitting point cloud data according to an embodiment, includes encoding the point cloud data.

[0452] In the encoding operation according to the embodiment, as described in Figures 1 to 14, point cloud data can be encoded. Also, as described in Figures 15 to 16, a prediction tree can be generated to efficiently compress point cloud data. The prediction tree according to the embodiment includes a prediction tree of a current frame, a prediction tree of a reference frame, and a prediction tree in which the prediction tree of the current frame and the prediction tree of the reference frame are integrated. As described in Figures 18 to 23, a bitstream including encoded point cloud data and related parameter information can be generated.

[0453] S2401, the point cloud data transmitting method according to the embodiment, further includes transmitting a bitstream including the point cloud data.

[0454] The transmission operation according to the embodiment can transmit encoded point cloud data as described in Figures 1 to 14. A bit stream including point cloud data encoded based on a prediction tree can be transmitted as described in Figures 15 to 16. The structure of the bit stream can be seen in the descriptions of Figures 18 to 23.

[0455] FIG. 25 illustrates a method for receiving point cloud data according to an embodiment.

[0456] The point cloud data receiving method / apparatus according to the embodiment (receiving device 10004, receiver 10005, point cloud video decoder 10006 in Figure 1, transmit-decode-render 20002-20003-20004 in Figure 2, decoder in Figures 10 to 11, receiving device in Figure 13, device in Figure 14, and decoder in Figure 17) corresponds to the method as shown in Figure 25.

[0457] S2500, a method for receiving point cloud data according to an embodiment, includes receiving a bitstream including point cloud data.

[0458] In the receiving operation according to the embodiment, a bit stream (FIGS. 18 to 23) including encoded point cloud data and parameters can be received as described in FIGs. 1 to 14. Point cloud data encoded based on a prediction tree can be received as described in FIGs. 15 and 17.

[0459] S2501, the point cloud data receiving method according to the embodiment, further includes a step of decoding the point cloud data.

[0460] In the decoding operation according to the embodiment, the point cloud data can be decoded as described in Figures 1 to 14. As described in Figures 15 and 17, a prediction tree can be generated and the point cloud data can be decoded based on the prediction tree. The point cloud data included in the bit stream of Figure 18 can be decoded based on the parameter information of Figures 19 to 23.

[0461] Referring to FIG. 1, a method for transmitting point cloud data according to an embodiment includes: encoding point cloud data; and transmitting a bitstream including the point cloud data.

[0462] 15, in relation to global and local motion estimation and compensation, encoding point cloud data includes performing inter-prediction between frames including point cloud data, and performing inter-prediction includes predicting global motion for the point cloud data based on a prediction unit, and dividing the prediction unit and predicting local motion for the point cloud data. The prediction unit includes an LPU, a PU, etc. depending on the size and the degree of division.

[0463] Referring to FIG. 15, in relation to the prediction tree generating method and the integrated prediction tree generating method, the encoding method according to the embodiment generates an integrated prediction tree using a prediction tree of a reference frame to predict a current point of a current frame. The step of encoding point cloud data includes the step of generating a prediction tree for the point cloud data, and the step of generating the prediction tree includes the step of searching for a parent node for a point included in a current frame including the point cloud data from a prediction tree for a reference frame of the current frame and registering the parent node in the prediction tree, and a parent node for a point included in the current frame is generated based on the number and distance of child nodes from at least one node included in the prediction tree for the reference frame, and inter-reference information is generated when prediction for a point included in the current frame is performed based on the reference frame. A parent node is also referred to as an upper node. A node includes one or more points. A node is also referred to as a point. The prediction tree includes a prediction tree for a current frame, a prediction tree for a reference frame, and an integrated prediction tree for the current frame and the reference frame. The integrated prediction tree is also referred to as a prediction tree. The combined prediction tree refers to a prediction tree generated from the prediction tree of the current frame and the prediction tree of the reference frame to code a point of the current frame.

[0464] 15, in relation to the prediction tree generation method and the integrated prediction tree generation method, the encoding method according to the embodiment generates an integrated prediction tree to continuously predict a next point of a current point. The step of generating a prediction tree further includes a step of registering a parent node for a next point in a current frame including point cloud data based on the prediction tree, and when the parent node for the next point is registered based on a reference frame, inter-reference information is generated. If points (or nodes) exist in the current frame, for example, a, b, and c, a is the current point and b is the next point.

[0465] Referring to FIG. 15, in relation to the generation method using a reference prediction tree, the step of encoding point cloud data includes generating a prediction tree for a current frame including the point cloud data, finding a parent node for a point included in the current frame based on the prediction tree for the current frame and the prediction tree of a reference frame for the current frame, searching for a parent node based on a distance between a node included in the prediction tree for the current frame and a node included in the prediction tree for the reference frame, and if a node included in the prediction tree of the reference frame is designated as the parent node, a virtual link between the prediction tree for the current frame and the prediction tree of the reference frame is generated, and inter-reference information is generated.

[0466] 18 to 23, the bitstream includes information related to prediction tree generation, such as a prediction geometry tree building method (pred_geom_tree_building_method) and inter-reference information (inter_reference_flag).

[0467] The point cloud data transmission method is performed by a transmitting device, which is an encoder including a memory and / or a processor, and includes an encoder for encoding the point cloud data, and a transmitter for transmitting a bitstream including the point cloud data.

[0468] The receiving method corresponds to the point cloud data transmitting method and performs the reverse process, including receiving a bitstream including the point cloud data; and decoding the point cloud data.

[0469] Referring to Figure 15, in relation to the prediction tree restoration method and motion compensation and prediction tree restoration, the step of decoding point cloud data includes a step of performing motion compensation on the point cloud data based on parameter information included in the bitstream, and a step of restoring a prediction tree for the point cloud data.

[0470] Referring to FIG. 15, in relation to the integrated prediction tree restoration method and inter reference information (reference_flag=1 or 0), the step of decoding point cloud data involves finding a parent node for a point included in a current frame including point cloud data from a prediction tree for a reference frame of the current frame or finding a parent node for the current frame based on the inter reference information included in the bitstream.

[0471] Referring to FIG. 15, in relation to the restoration method using a reference prediction tree, the step of decoding point cloud data restores a prediction tree for a current frame including the point cloud data, finds a parent node for a point included in the current frame based on the prediction tree for the current frame and the prediction tree of a reference frame for the current frame, and searches for a parent node based on a distance between a node included in the prediction tree for the current frame and a node included in the prediction tree for the reference frame. If a node included in the prediction tree of the reference frame is designated as the parent node, a virtual link between the prediction tree for the current frame and the prediction tree for the reference frame is generated.

[0472] The receiving method is performed by a receiving device configured as a decoder including a memory and / or a processor, the receiving device including a receiver for receiving a bitstream including the point cloud data, and a decoder for decoding the point cloud data.

[0473] When generating a prediction tree, a point close to the current point is searched for, but in the embodiment, the prediction tree itself is generated as a prediction tree for inter prediction, not as a tree of the current frame and a tree of the reference frame. That is, the encoder and / or decoder according to the embodiment generates a prediction tree taking into account the correlation of all points, enabling efficient compression / restoration. The compression efficiency of the geometry information compression method can be improved. Real-time point cloud data capture / compression / transmission / restoration / playback services can be efficiently supported.

[0474] The PCC encoding method, PCC decoding method, and signaling method of the above-mentioned embodiments provide the following effects. For scenarios requiring low-latency services, such as capturing and transmitting point cloud data from LiDAR in real time, or providing a service in which 3D map data is transmitted and processed in real time, it is necessary to shorten the encoding and decoding time. As a method of shortening the time, even if the entire point cloud information is not available, the latency can be reduced by starting encoding only for a part of the point cloud, streaming, and decoding.

[0475] The embodiment can provide a method for performing inter-prediction using a prediction tree generation method that reduces the latency of a geometry information compression method, such as Geometry-based Point Cloud Compression (G-PCC), for compressing 3D point cloud data for low-latency services, while maintaining the size of the bitstream.

[0476] As a result, the embodiment can improve the geometry compression efficiency of a geometry-based point cloud compression (G-PCC) encoder / decoder for compressing 3D point cloud data and provide a point cloud content stream.

[0477] The PCC encoder and / or PCC decoder according to the embodiment can provide an efficient prediction tree generation method and can provide the effect of increasing the efficiency of geometry compression encoding / decoding by taking into account correlation between frames.

[0478] The operation of the transceiver according to the above-described embodiment can be explained in conjunction with the following point cloud compression process.

[0479] Therefore, the transmitting method / apparatus according to the embodiment can efficiently compress point cloud data and transmit the data, and by transmitting signaling information accordingly, the receiving method / apparatus according to the embodiment can also efficiently decode / restore the point cloud data.

[0480] The embodiments are described in terms of methods and / or apparatus, and method descriptions and apparatus descriptions may be applied in a complementary manner.

[0481] Although the drawings have been described separately for the convenience of explanation, it is possible to combine the embodiments described in the drawings to design a new embodiment. In addition, it is also within the scope of the embodiments to design a computer readable recording medium in which a program for executing the previously described embodiments is recorded, as required by a person of ordinary skill in the art. The apparatus and method according to the embodiments are not limited to the configurations and methods of the described embodiments, as described above, and the embodiments may be configured by selectively combining all or part of each embodiment so as to be variously modified. Although preferred embodiments have been shown and described, the embodiments are not limited to the specific embodiments described above, and various modifications can be made by a person of ordinary skill in the art to which the invention belongs without departing from the gist of the embodiments claimed in the claims, and such modifications should not be understood individually from the technical ideas and prospects of the embodiments.

[0482] Various components of the apparatus according to the embodiment may be implemented by hardware, software, firmware, or a combination thereof. Various components of the embodiment may be implemented by one chip, for example, one hardware circuit. In the embodiment, the components according to the embodiment may be implemented by individual chips. In the embodiment, any of the components of the apparatus according to the embodiment may be implemented by one or more processors capable of executing one or more programs, and the one or more programs may include instructions for performing or causing any one or more of the operations / methods according to the embodiment to be performed. Executable instructions for performing the method / operation of the apparatus according to the embodiment may be stored in a non-transitory CRM or other computer program product configured to be executed by one or more processors, or may be stored in a transitory CRM or other computer program product configured to be executed by one or more processors. In addition, the memory according to the embodiment is used as a concept including not only volatile memory (e.g., RAM, etc.) but also non-volatile memory, flash memory, PROM, etc. It may also be implemented in the form of a carrier wave, such as transmission over the Internet. In addition, the recording medium read by the processor may be distributed to computer systems connected by a network, and the code read by the processor may be stored and executed in a distributed manner.

[0483] In this specification, " / " and "," are interpreted as "and / or." For example, "A / B" is interpreted as "A and / or B," and "A, B" is interpreted as "A and / or B." Furthermore, "A / B / C" means "any of A, B, and / or C." Also, "A, B, C" means "any of A, B, and / or C." Furthermore, in this specification, "or" is interpreted as "and / or." For example, "A or B" means 1) only "A," 2) only "B," or 3) "A and B." In other words, in this specification, "or" means "additionally or alternatively."

[0484] Terms such as first and second are used to describe various components of the embodiments. However, the various components according to the embodiments should not be limited in interpretation by the above terms. Such terms are merely used to distinguish one component from another. For example, a first user input signal can be referred to as a second user input signal. Similarly, a second user input signal can be referred to as a first user input signal. Use of such terms does not depart from the scope of various embodiments. Although the first user input signal and the second user input signal are both user input signals, they do not mean the same user input signal unless the context clearly indicates otherwise.

[0485] The terms used to describe the embodiments are used to describe specific embodiments and are not intended to limit the embodiments. As used in the description of the embodiments and the claims, the singular includes the plural unless the context clearly dictates otherwise. The term "and / or" is used to include all possible combinations between terms. "Comprises" describes the presence of a feature, number, step, element, and / or component, and does not mean that additional features, numbers, steps, elements, and / or components are not included. Conditional expressions such as "if" and "when" used to describe the embodiments are only intended to be interpreted as selective and limiting. It is intended that when a particular condition is met, a related action is performed in response to a particular condition, or a related definition is interpreted.

[0486] Moreover, the operations according to the embodiments described in this specification are performed by a transceiver device including a memory and / or a processor according to the embodiments. The memory stores a program for processing / controlling the operations according to the embodiments, and the processor controls various operations described in this specification. The processor is also referred to as a controller, etc. The operations of the embodiments are performed by firmware, software, and / or a combination thereof, and the firmware, software, and / or a combination thereof are stored in the processor or stored in the memory.

[0487] Meanwhile, the operations according to the above-described embodiments are performed by a transmitting device and / or a receiving device according to the embodiments. The transmitting / receiving device includes a transceiver for transmitting and receiving media data, a memory for storing instructions (program code, algorithms, flow charts and / or data) for a process according to the embodiment, and a processor for controlling the operation of the transmitting / receiving device.

[0488] The processor may also be referred to as a controller, and may be, for example, hardware, software, and / or a combination thereof. The operations according to the above-described embodiments are performed by the processor. The processor may also be embodied as an encoder / decoder for the operations of the above-described embodiments.

[0489] As mentioned above, the relevant content will be described with respect to the best mode for carrying out the embodiment.

[0490] As described above, the embodiments may be applied in whole or in part to point cloud data transmitting and receiving devices and systems.

[0491] Those skilled in the art can make various changes and modifications to the embodiments within the scope of the embodiments.

[0492] The embodiments include modifications / variations that are within the scope of the claims and the equivalents thereof.

[0493] [Claims (at the time of international application)] [Claim 1] A point cloud data transmission method, comprising: encoding the point cloud data; and transmitting a bitstream including the point cloud data. [Claim 2] The step of encoding the point cloud data includes performing inter-prediction between frames including the point cloud data, The step of performing inter prediction includes: predicting a global motion for the point cloud data based on a prediction unit; The method of claim 1 , further comprising: dividing the prediction unit and predicting local motion for the point cloud data. [Claim 3] encoding the point cloud data includes generating a prediction tree for the point cloud data; The step of generating a prediction tree comprises: searching for a parent node for a point included in a current frame including the point cloud data from a prediction tree for a reference frame of the current frame, and registering the parent node in the prediction tree; the parent node for the point included in the current frame is generated based on the number and distance of child nodes from at least one node included in the prediction tree for the reference frame; The method of claim 1 , wherein inter-reference information is generated when prediction of a point included in the current frame is performed based on a reference frame. [Claim 4] The generating the prediction tree further includes registering a parent node for a next point included in a current frame including the point cloud data based on the prediction tree; The point cloud data transmission method of claim 3 , wherein inter-reference information is generated when a parent node for the next point is registered based on the reference frame. [Claim 5] The step of encoding the point cloud data includes: generating a prediction tree for a current frame that includes the point cloud data; finding a parent node for a point included in the current frame based on a prediction tree for the current frame and a prediction tree of a reference frame for the current frame; finding the parent node based on a distance between a node included in a prediction tree for the current frame and a node included in a prediction tree for the reference frame; The point cloud data transmission method of claim 1, wherein when a node included in the prediction tree of the reference frame is designated as the parent node, a virtual link is generated between the prediction tree for the current frame and the prediction tree of the reference frame, and inter-reference information is generated. [Claim 6] The point cloud data transmission method of claim 1 , wherein the bit stream includes information regarding prediction tree generation. [Claim 7] A point cloud data transmission device, an encoder for encoding the point cloud data; and A point cloud data transmitting device comprising: a transmitter for transmitting a bit stream including the point cloud data. [Claim 8] A method for receiving point cloud data, comprising: receiving a bitstream comprising point cloud data; and A method for receiving point cloud data, comprising: a step of decoding the point cloud data. [Claim 9] The step of decoding the point cloud data includes: performing motion compensation on the point cloud data based on parameter information included in the bitstream; The method of claim 8 , further comprising: reconstructing a prediction tree for the point cloud data. [Claim 10] The step of decoding the point cloud data includes the steps of: decoding the point cloud data based on inter-reference information included in the bitstream; 10. The point cloud data receiving method of claim 9, further comprising: finding a parent node for a point included in a current frame including the point cloud data from a prediction tree for a reference frame of the current frame, or finding a parent node for the current frame from a prediction tree for the current frame. [Claim 11] The step of decoding the point cloud data includes: Reconstructing a prediction tree for a current frame including the point cloud data; finding a parent node for a point included in the current frame based on a prediction tree for the current frame and a prediction tree of a reference frame for the current frame; finding the parent node based on a distance between a node included in a prediction tree for the current frame and a node included in a prediction tree for the reference frame; 10. The point cloud data receiving method of claim 9, wherein when a node included in the prediction tree of the reference frame is designated as the parent node, a virtual link is generated between the prediction tree for the current frame and the prediction tree for the reference frame. [Claim 12] The method of claim 8 , wherein the bitstream includes information regarding prediction tree generation. [Claim 13] A point cloud data receiving device, a receiver for receiving a bitstream including point cloud data; and a decoder for decoding the point cloud data; [Claim 14] The decoder comprises: performing motion compensation on the point cloud data based on parameter information included in the bitstream; The point cloud data receiving device according to claim 13 , further comprising: a prediction tree for the point cloud data being reconstructed. [Claim 15] The decoder, based on inter-reference information included in the bitstream, The point cloud data receiving device of claim 14, wherein a parent node for a point included in a current frame including the point cloud data is found from a prediction tree for a reference frame of the current frame, or from a prediction tree for the current frame.

Claims

1. A point cloud data transmission method, comprising: encoding the point cloud data; and transmitting a bitstream including the point cloud data.

2. The step of encoding the point cloud data includes performing inter-prediction between frames including the point cloud data; The step of performing inter-prediction includes: predicting a global motion for the point cloud data based on a prediction unit; The method of claim 1 , further comprising: dividing the prediction unit and predicting local motion for the point cloud data.

3. encoding the point cloud data includes generating a prediction tree for the point cloud data; The step of generating a prediction tree comprises: searching for a parent node for a point included in a current frame including the point cloud data from a prediction tree for a reference frame of the current frame, and registering the parent node in the prediction tree; the parent node for the point included in the current frame is generated based on the number and distance of child nodes from at least one node included in the prediction tree for the reference frame; The method of claim 1 , wherein inter-reference information is generated when prediction of a point included in the current frame is performed based on a reference frame.

4. The generating the prediction tree further includes registering a parent node for a next point included in a current frame including the point cloud data based on the prediction tree; The method of claim 3 , wherein inter-reference information is generated when a parent node for the next point is registered based on the reference frame.

5. The step of encoding the point cloud data includes: generating a prediction tree for a current frame that includes the point cloud data; finding a parent node for a point included in the current frame based on a prediction tree for the current frame and a prediction tree of a reference frame for the current frame; finding the parent node based on a distance between a node included in a prediction tree for the current frame and a node included in a prediction tree for the reference frame; 2. The point cloud data transmission method of claim 1, wherein when a node included in a prediction tree of the reference frame is designated as the parent node, a virtual link is generated between the prediction tree for the current frame and the prediction tree of the reference frame, and inter-reference information is generated.

6. The method of claim 1 , wherein the bitstream includes information regarding prediction tree generation.

7. A point cloud data transmission device, an encoder for encoding the point cloud data; and A point cloud data transmitting device comprising: a transmitter for transmitting a bit stream including the point cloud data.

8. A method for receiving point cloud data, comprising: receiving a bitstream comprising point cloud data; and A method for receiving point cloud data comprising: decoding the point cloud data.

9. The step of decoding the point cloud data includes: performing motion compensation on the point cloud data based on parameter information included in the bitstream; The method of claim 8 , further comprising the step of: reconstructing a prediction tree for the point cloud data.

10. The step of decoding the point cloud data includes the steps of: decoding the point cloud data based on inter-reference information included in the bitstream; The point cloud data receiving method of claim 9, further comprising: searching for a parent node for a point included in a current frame including the point cloud data from a prediction tree for a reference frame of the current frame, or searching for a prediction tree for the current frame.

11. The step of decoding the point cloud data includes: Reconstructing a prediction tree for a current frame including the point cloud data; finding a parent node for a point included in the current frame based on a prediction tree for the current frame and a prediction tree of a reference frame for the current frame; finding the parent node based on a distance between a node included in a prediction tree for the current frame and a node included in a prediction tree for the reference frame; The method of claim 9 , further comprising: when a node included in the prediction tree of the reference frame is designated as the parent node, a virtual link is generated between the prediction tree for the current frame and the prediction tree for the reference frame.

12. The method of claim 8 , wherein the bitstream includes information regarding prediction tree generation.

13. A point cloud data receiving device, A receiver for receiving a bitstream including point cloud data; and A point cloud data receiving device comprising: a decoder for decoding the point cloud data.

14. The decoder comprises: performing motion compensation on the point cloud data based on parameter information included in the bitstream; The point cloud data receiving device according to claim 13 , further comprising: a predictive tree for the point cloud data.

15. The decoder, based on inter-reference information included in the bitstream, The point cloud data receiving device of claim 14 , further comprising: searching for a parent node for a point included in a current frame including the point cloud data from a prediction tree for a reference frame of the current frame, or searching for a prediction tree for the current frame.

Citation Information

Patent Citations

  • Predictive Coding For Point Cloud Compression

    US20210104075A1

  • Point cloud data transmission device, point cloud data transmission method, point cloud data reception device, and point cloud data reception method

    US20210105504A1

  • Predictive coding of point clouds using multiple frames of references

    US20210192798A1

  • Three-dimensional data encoding method, three-dimensional data decoding method, three-dimensional data encoding device, and three-dimensional data decoding device

    WO2022075319A1

Cited By

  • Method, apparatus, and medium for point cloud coding

    JP2025500353A

  • Method, apparatus and medium for point cloud coding

    JP2025501125A

  • Method, apparatus, and medium for point cloud coding

    JP7735574B2

  • Method, apparatus and medium for point cloud coding

    JP7827358B2

  • Method, apparatus, and medium for point cloud coding

    US12621495B2