Hybrid Tree Coding for Inter and Intra Prediction for Geometry Coding

Abstract

A device for decoding a bitstream including point cloud data is configured to determine an octree defining an octree-based division of a space including the point cloud, where leaf nodes of the octree include one or more points of the point cloud, and to directly decode a position of each of the one or more points in the leaf nodes, wherein, to directly decode a position of each of the one or more points in the leaf nodes, the one or more processors are further configured to generate predictions of the one or more points and determine the one or more points based on the predictions.

Description

[Technical field]

[0001] This application claims priority to U.S. Patent Application No. 17 / 562,398, filed December 27, 2021, and U.S. Provisional Patent Application No. 63 / 131,546, filed December 29, 2020, the entire contents of each of which are incorporated herein by reference. U.S. Patent Application No. 17 / 562,398, filed December 27, 2021, claims the benefit of U.S. Provisional Patent Application No. 63 / 131,546, filed December 29, 2020.

[0002] This disclosure relates to point cloud encoding and decoding. [Background technology]

[0003] [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] “Text of ISO / IEC FDIS 23090-9 Geometry-based Point Cloud Compression”, ISO / IEC JTC 1 / SC29 / WG 7 MDS19617, Teleconference, October 2020 [Non-Patent Document 2] G-PCC Codec Description, ISO / IEC JTC 1 / SC29 / WG 7 MDS19620, Teleconference, October 2020 Summary of the Invention [Means for solving the problem]

[0005] Generally, this disclosure describes a hybrid tree coding method that combines octree coding and predictive coding for advanced inter / intra prediction at the block level for point cloud compression.

[0006] In one example, the present disclosure describes a method for coding a point cloud, the method including the steps of determining an octree defining an octree-based partition of a space including the point cloud, where leaf nodes of the octree include one or more points of the point cloud, and where the positions of each of the one or more points in the leaf nodes are directly signaled; generating predictions of the one or more points using intra- or inter-prediction; and coding a syntax element indicating whether the one or more points are predicted using intra- or inter-prediction.

[0007] According to one example of the present disclosure, a device for decoding a bitstream including point cloud data includes a memory for storing the point cloud data and one or more processors coupled to the memory and implemented in a circuit, where the one or more processors are configured to determine an octree defining an octree-based partition of a space including the point cloud, where leaf nodes of the octree include one or more points of the point cloud, and directly decode a position of each of the one or more points in the leaf nodes, and for directly decoding a position of each of the one or more points in the leaf nodes, the one or more processors are further configured to generate predictions of the one or more points and determine the one or more points based on the predictions.

[0008] According to another example of the present disclosure, a method for decoding a point cloud includes determining an octree defining an octree-based partition of a space including the point cloud, where leaf nodes of the octree include one or more points of the point cloud, and directly decoding a position of each of the one or more points at the leaf nodes, where directly decoding a position of each of the one or more points at the leaf nodes includes generating a prediction of the one or more points and determining the one or more points based on the prediction.

[0009] According to another example of the present disclosure, a computer-readable storage medium stores instructions that, when executed by one or more processors, cause the one or more processors to determine an octree defining an octree-based partition of a space including a point cloud, where a leaf node of the octree includes one or more points of the point cloud, and directly decode a position of each of the one or more points at the leaf node; and, to directly decode a position of each of the one or more points at the leaf node, the instructions cause the one or more processors to generate a prediction of the one or more points and determine the one or more points based on the prediction.

[0010] According to another example of the present disclosure, an apparatus includes means for determining an octree defining an octree-based partition of a space including a point cloud, where a leaf node of the octree includes one or more points of the point cloud, and means for directly decoding a position of each of the one or more points at the leaf node, the means for directly decoding a position of each of the one or more points at the leaf node comprising: means for generating a prediction of the one or more points; and means for determining the one or more points based on the prediction.

[0011] The details of one or more examples are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will become apparent from the description, drawings, and claims. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram illustrating an example encoding and decoding system that may implement the techniques of this disclosure. [Diagram 2] FIG. 1 is a block diagram illustrating an example geometry point cloud compression (G-PCC) encoder. [Diagram 3] FIG. 2 is a block diagram illustrating an exemplary G-PCC decoder. [Figure 4] FIG. 1 is a conceptual diagram illustrating an example octree partitioning for geometry coding. [Diagram 5] FIG. 1 is a conceptual diagram illustrating an example of a prediction tree. [Figure 6] FIG. 1 is a conceptual diagram illustrating an example rotating Lidar acquisition model. [Figure 7] FIG. 13 is a conceptual diagram illustrating an example motion estimation flow chart for InterEM. [Figure 8] FIG. 2 is a conceptual diagram illustrating an exemplary algorithm for global motion estimation. [Figure 9] FIG. 2 is a conceptual diagram illustrating an example algorithm for estimation of local node motion vectors. [Figure 10] FIG. 1 is a conceptual diagram illustrating an example of high-level octree division. [Figure 11] FIG. 1 is a conceptual diagram illustrating an example of local prediction tree generation. [Figure 12] FIG. 1 is a conceptual diagram illustrating an exemplary current set of points (O0-O12) and a reference set of points (R0-R12), where N=M=13. [Figure 13] A conceptual diagram showing an example current set of points and a motion compensated reference set of points, where N=M=13. [Figure 14] FIG. 1 is a conceptual diagram illustrating an example distance measurement system that may be used with one or more techniques of this disclosure. [Figure 15] FIG. 1 is a conceptual diagram illustrating an example vehicle-based scenario in which one or more techniques of the present disclosure may be used. [Figure 16] FIG. 1 is a conceptual diagram illustrating an example extended reality system in which one or more techniques of this disclosure may be used. [Figure 17] FIG. 1 is a conceptual diagram illustrating an example mobile device system in which one or more techniques of the present disclosure may be used. [Figure 18] 5 is a flowchart illustrating an example operation for decoding a bitstream including point cloud data. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] A point cloud is a collection of points in three-dimensional (3D) space. The points may correspond to points on an object in the three-dimensional space. Thus, a point cloud may be used to represent the physical content of a three-dimensional space. Point clouds may have utility in a wide variety of situations. For example, a point cloud may be used in the context of an autonomous vehicle to represent the location of an object on a road. In another example, a point cloud may be used in the context of representing the physical content of an environment for the purpose of positioning virtual objects in an augmented reality (AR) or mixed reality (MR) application. Point cloud compression is the process for encoding and decoding a point cloud. Encoding a point cloud may reduce the amount of data required for storage and transmission of the point cloud.

[0014] There have been two major proposals for signaling the location of points in a point cloud before: octree coding and predictive tree coding. As part of encoding point cloud data using octree coding, the G-PCC encoder may generate an octree. Each node of the octree corresponds to a cuboid space. A node of the octree may have zero or eight child nodes. In other examples, a node may be divided into child nodes according to other tree structures. A child node of a parent corresponds to an equally sized cuboid within the cuboid corresponding to the parent node. The location of individual points of the point cloud may be signaled with respect to the origin of the node. A node is said to be unoccupied if it does not contain any points of the point cloud. If a node is unoccupied, there may be no need to signal additional data about the node. Conversely, a node is said to be occupied if it contains one or more points of the point cloud.

[0015] When encoding point cloud data using predictive tree coding, the G-PCC encoder determines a prediction mode for each point of the point cloud. The prediction mode for a point can be one of the following:

[0016] · No prediction / Zero prediction (0) Delta forecast (p0) Linear prediction (2*p0-p1) Parallelogram prediction (2*p0+p1-p2) If the prediction mode for a point is "no prediction / zero prediction", the point is considered a root point (i.e., root vertex), and the coordinates (e.g., x, y, z coordinates) of the point are signaled in the bitstream. If the prediction mode for a point is "delta prediction", the G-PCC encoder determines the difference (i.e., delta) between the coordinates of the point and the coordinates of a parent point, such as a root point or another point. If the prediction mode is "linear prediction", the G-PCC encoder determines the predicted coordinates of the point using linear prediction of the coordinates of the two parent points. The G-PCC encoder signals the difference between the predicted coordinates determined using linear prediction and the actual coordinates of the point. If the prediction mode is "parallelogram prediction", the G-PCC encoder determines the predicted coordinates using three parent points. The G-PCC encoder then signals the difference (e.g., "first order residual") between the predicted coordinates and the actual coordinates of the point. The predictive relationships between the points essentially define a tree of points.

[0017] It has been experimentally observed that octree coding may be more suitable for dense point clouds than predictive tree coding. Point clouds acquired using 3D modeling are often dense enough for octree coding to work better. However, for example, for automotive applications, point clouds acquired using LiDAR tend to be somewhat coarse, and therefore predictive coding may work better in these applications.

[0018] In some examples, the angular mode may be used to represent the coordinates of a point in a spherical coordinate system. Because the conversion process between the spherical coordinate system and the Cartesian (e.g., x, y, z) coordinate system is not perfect, information may be lost. However, because the G-PCC encoder can perform the conversion process, the G-PCC encoder can signal a "second-order residual" for a point that indicates the difference between the Cartesian coordinates of the point resulting from applying the conversion process to the spherical coordinates of the point and the original Cartesian coordinates of the point.

[0019] This disclosure relates to a hybrid coding model in which both octree coding and direct coding are used to code a point cloud. For example, octree coding may be used first to divide a space into nodes up to a certain level. The nodes at the certain level (and other occupied nodes of the octree that are not further divided) may be referred to as "leaf nodes." Points within the volume of the leaf nodes may be coded using the "direct" coding mode.

[0020] When encoding a point of a leaf node in a "direct" coding mode, the G-PCC encoder may select an intra prediction mode for the leaf node or an inter prediction mode for the leaf node. The G-PCC encoder may signal whether the point of the leaf node is encoded using an intra prediction mode or an inter prediction mode.

[0021] When the G-PCC encoder selects an intra-prediction mode for a leaf node, the G-PCC encoder may use predictive tree coding to encode the point in the leaf node in much the same manner as described above. That is, the G-PCC encoder may select among four prediction modes and signal coordinates for the point accordingly. However, instead of signaling coordinates relative to the origin of the entire space associated with the octree, the G-PCC encoder may signal coordinates relative to the origin of the leaf node. This may improve coding efficiency, especially for the root node.

[0022] If the G-PCC encoder selects an inter prediction mode for a leaf node, the G-PCC encoder may encode a point at the leaf node with respect to a set of points in a reference frame. The reference frame may be a previously coded frame similar to a previous frame of the video. The G-PCC encoder may perform motion estimation to identify a set of points in the reference frame that have a similar spatial arrangement as the points at the leaf node. The motion vector for the leaf node indicates the displacement between the point of the leaf node and the identified set of points in the reference frame.

[0023] The G-PCC encoder may signal a set of parameters for a leaf node. The parameters for the leaf node may include a reference index that identifies a reference frame. The parameters for the leaf node may also include a value that indicates the number of points in the leaf node.

[0024] The parameters of the leaf nodes may also include a residual value for each of the points in the leaf node. The residual value for a point in the leaf node indicates the difference between the predicted coordinates of the leaf node (determined by adding the motion vector of the leaf node to the point in the reference frame that corresponds to the point in the leaf node). In an example where the angular mode is used, the G-PCC encoder may also signal a second order residual for the point.

[0025] In some examples, the parameters for the leaf node also include a motion vector differential (MVD). The MVD indicates the difference between the motion vector of the leaf node and a predicted motion vector. The predicted motion vector is the motion vector of a neighboring node of the octree. The parameters for the leaf node may include an index that identifies the neighboring node.

[0026] In another example, similar to merge mode in conventional video coding, the parameters for the leaf node may not include MVD, and the motion vector of the leaf node may be assumed to be the same as the motion vector of the identified neighboring node.

[0027] In some examples, the signaling of the residual may be skipped. In some such examples where the angle mode is used, the signaling of the first order residual may be skipped while still signaling the second order residual.

[0028] 1 is a block diagram illustrating an example encoding and decoding system 100 that may perform the techniques of this disclosure. The techniques of this disclosure are generally directed to coding (encoding and / or decoding) point cloud data, i.e., supporting point cloud compression. In general, point cloud data includes any data for processing a point cloud. Coding may be useful for compressing and / or decompressing the point cloud data.

[0029] As shown in FIG. 1, the system 100 includes a source device 102 and a destination device 116. The source device 102 provides encoded point cloud data to be decoded by the destination device 116. In particular, in the example of FIG. 1, the source device 102 provides the point cloud data to the destination device 116 via a computer-readable recording medium 110. The source device 102 and the destination device 116 may comprise any of a wide range of devices, including desktop computers, notebook (i.e., laptop) computers, tablet computers, set-top boxes, telephone handsets such as smartphones, televisions, cameras, display devices, digital media players, video gaming consoles, video streaming devices, terrestrial or marine vehicles, spacecraft, aircraft, robots, LIDAR devices, satellites, and the like. In some cases, the source device 102 and the destination device 116 may be equipped for wireless communication.

[0030] In the example of FIG. 1, the source device 102 includes a data source 104, a memory 106, a G-PCC encoder 200, and an output interface 108. The destination device 116 includes an input interface 122, a G-PCC decoder 300, a memory 120, and a data consumer 118. According to this disclosure, the G-PCC encoder 200 of the source device 102 and the G-PCC decoder 300 of the destination device 116 may be configured to apply techniques of this disclosure related to a hybrid tree coding method that combines octree coding and predictive coding for enhanced inter / intra prediction at the block level for point cloud compression. Thus, the source device 102 represents an example of an encoding device, and the destination device 116 represents an example of a decoding device. In other examples, the source device 102 and the destination device 116 may include other components or arrangements. For example, the source device 102 may receive data (e.g., point cloud data) from an internal or external source. Similarly, destination device 116 may interface with an external data consumer rather than including the data consumer within the same device.

[0031] The system 100 as shown in FIG. 1 is only an example. In general, other digital encoding and / or decoding devices may perform the techniques of this disclosure related to a hybrid tree coding method that combines octree coding and predictive coding for enhanced inter / intra prediction at the block level for point cloud compression. The source device 102 and the destination device 116 are only examples of devices in which the source device 102 generates coded data for transmission to the destination device 116. This disclosure refers to devices that perform coding (encoding and / or decoding) of data as "coding" devices. Thus, the G-PCC encoder 200 and the G-PCC decoder 300 represent examples of coding devices, specifically, encoders and decoders, respectively. In some examples, the source device 102 and the destination device 116 may operate substantially symmetrically such that each of the source device 102 and the destination device 116 includes encoding and decoding components. Thus, the system 100 may support one-way or two-way transmission between the source device 102 and the destination device 116, for example, for streaming, playback, broadcast, telephony, navigation, and other applications.

[0032] In general, the data source 104 represents a source of data (i.e., raw, unencoded point cloud data) and may provide a sequential series of "frames" of data to the G-PCC encoder 200, which encodes the data for the frames. The data source 104 of the source device 102 may include a point cloud capture device, such as any of a variety of cameras or sensors, e.g., a 3D scanner or light detection and ranging (LIDAR) device, one or more video cameras, an archive containing previously captured data, and / or a data feed interface for receiving data from a data content provider. Alternatively or additionally, the point cloud data may be computer-generated from a scanner, camera, sensor, or other data. For example, the data source 104 may generate computer graphics-based data as source data, or result in a combination of live data, archived data, and computer-generated data. In each case, the G-PCC encoder 200 encodes the captured data, the pre-captured data, or the computer-generated data. The G-PCC encoder 200 may reorder the frames from the order in which they were received (sometimes referred to as "display order") into a coding order for coding. The G-PCC encoder 200 may generate one or more bitstreams including the encoded data. The source device 102 may then output the encoded data onto the computer-readable recording medium 110 via the output interface 108, for receipt and / or retrieval by, for example, the input interface 122 of the destination device 116.

[0033] The memory 106 of the source device 102 and the memory 120 of the destination device 116 may represent general-purpose memory. In some examples, the memory 106 and the memory 120 may store raw data, e.g., raw data from the data source 104 and raw decoded data from the G-PCC decoder 300. Additionally or alternatively, the memory 106 and the memory 120 may store software instructions executable by, e.g., the G-PCC encoder 200 and the G-PCC decoder 300, respectively. Although the memory 106 and the memory 120 are shown separately from the G-PCC encoder 200 and the G-PCC decoder 300 in this example, it should be understood that the G-PCC encoder 200 and the G-PCC decoder 300 may also include internal memory for functionally similar or equivalent purposes. Additionally, the memory 106 and the memory 120 may store, e.g., encoded data output from the G-PCC encoder 200 and input to the G-PCC decoder 300. In some examples, portions of memory 106 and memory 120 may be allocated as one or more buffers, e.g., for storing raw decoded and / or encoded data. For example, memory 106 and memory 120 may store data representing a point cloud.

[0034] The computer-readable medium 110 may represent any type of medium or device capable of transporting encoded data from the source device 102 to the destination device 116. In one example, the computer-readable medium 110 represents a communication medium that allows the source device 102 to transmit encoded data directly to the destination device 116 in real time, for example, via a radio frequency network or a computer-based network. In accordance with a communication standard, such as a wireless communication protocol, the output interface 108 may modulate a transmission signal including the encoded data, and the input interface 122 may demodulate a received transmission signal. The communication medium may comprise any wireless or wired communication medium, such as a radio frequency (RF) spectrum, or one or more physical transmission lines. The communication medium may form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. The communication medium may include routers, switches, base stations, or any other equipment that may be useful to facilitate communication from the source device 102 to the destination device 116.

[0035] In some examples, source device 102 may output the encoded data from output interface 108 to storage device 112. Similarly, destination device 116 may access the encoded data from storage device 112 via input interface 122. Storage device 112 may include any of a variety of distributed or locally accessed data storage media, such as a hard drive, Blu-ray disc, DVD, CD-ROM, flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded data.

[0036] In some examples, the source device 102 may output the encoded data to a file server 114 or another intermediate storage device that may store the encoded data generated by the source device 102. The destination device 116 may access the stored data from the file server 114 via streaming or download. The file server 114 may be any type of server device capable of storing the encoded data and transmitting the encoded data to the destination device 116. The file server 114 may represent a web server (e.g., for a website), a file transfer protocol (FTP) server, a content delivery network device, or a network attached storage (NAS) device. The destination device 116 may access the encoded data from the file server 114 through any standard data connection, including an Internet connection. This may include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., a digital subscriber line (DSL), a cable modem, etc.), or a combination of both suitable for accessing the encoded data stored on the file server 114. The file server 114 and the input interface 122 may be configured to operate according to a streaming transmission protocol, a download transmission protocol, or a combination thereof.

[0037] The output interface 108 and the input interface 122 may represent a wireless transmitter / receiver, a modem, a wired networking component (e.g., an Ethernet card), a wireless communication component operating according to any of the various IEEE 802.11 standards, or other physical components. In examples in which the output interface 108 and the input interface 122 comprise wireless components, the output interface 108 and the input interface 122 may be configured to transfer data, such as data encoded according to a cellular communication standard, such as 4G, 4G-LTE (Long Term Evolution), LTE Advanced, 5G, etc. In some examples in which the output interface 108 comprises a wireless transmitter, the output interface 108 and the input interface 122 may be configured to transfer data, such as data encoded according to other wireless standards, such as the IEEE 802.11 specification, the IEEE 802.15 specification (e.g., ZigBee™), the Bluetooth™ standard, etc. In some examples, the source device 102 and / or the destination device 116 may include respective system-on-chip (SoC) devices. For example, the source device 102 may include a SoC device for performing functions attributed to the G-PCC encoder 200 and / or the output interface 108, and the destination device 116 may include a SoC device for performing functions attributed to the G-PCC decoder 300 and / or the input interface 122.

[0038] The techniques of this disclosure may be applied to encoding and decoding in support of any of a variety of applications, such as communication between autonomous vehicles, communication between scanners, cameras, sensors and processing devices such as local or remote servers, geographic mapping, or other applications.

[0039] The input interface 122 of the destination device 116 receives the encoded bitstream from the computer-readable recording medium 110 (e.g., a communication medium, a storage device 112, a file server 114, etc.). The encoded bitstream may include signaling information defined by the G-PCC encoder 200 that is also used by the G-PCC decoder 300, such as syntax elements having values ​​that describe characteristics and / or processing of a coded unit (e.g., a slice, a picture, a group of pictures, a sequence, etc.). The data consumer 118 uses the decoded data. For example, the data consumer 118 may use the decoded data to determine the location of a physical object. In some examples, the data consumer 118 may include a display for presenting an image based on the point cloud.

[0040] The G-PCC encoder 200 and the G-PCC decoder 300 may each be implemented as any of a variety of suitable encoder and / or decoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. When the techniques are implemented partially in software, a device may store instructions for the software on a suitable non-transitory computer-readable recording medium and execute the instructions in hardware using one or more processors to perform the techniques of this disclosure. Each of the G-PCC encoder 200 and the G-PCC decoder 300 may be included in one or more encoders or decoders, any of which may be integrated as part of a composite encoder / decoder (codec) in the respective device. A device including the G-PCC encoder 200 and / or the G-PCC decoder 300 may comprise one or more integrated circuits, microprocessors, and / or other types of devices.

[0041] The G-PCC encoder 200 and the G-PCC decoder 300 may operate according to a coding standard, such as the Video Point Cloud Compression (V-PCC) standard or the Geometry Point Cloud Compression (G-PCC) standard. This disclosure may generally refer to coding (e.g., encoding and decoding) a picture to include the process of encoding or decoding data. The encoded bitstream generally includes a set of values ​​for syntax elements that represent coding decisions (e.g., coding modes).

[0042] This disclosure may generally refer to "signaling" some information, such as syntax elements. The term "signaling" may generally refer to communication of values ​​for syntax elements and / or other data used to decode encoded data. That is, the G-PCC encoder 200 may signal values ​​for syntax elements in a bitstream. In general, signaling refers to generating values ​​in a bitstream. As mentioned above, the source device 102 may transport the bitstream to the destination device 116 substantially in real time or not in real time, such as may occur when storing syntax elements in the storage device 112 for later retrieval by the destination device 116.

[0043] ISO / IEC MPEG (JTC1 / SC29 / WG11) is investigating the potential need for standardization of point cloud coding techniques with compression capabilities significantly beyond those of current methods. The group is working together on this research in a collaborative effort known as the 3-Dimensional Graphics Team (3DG) to evaluate compression design proposals proposed by experts in the field.

[0044] Point cloud compression activities are categorized into two different approaches. The first approach is "Video Point Cloud Compression" (V-PCC), which segments a 3D object and projects the segments into multiple 2D planes (represented as "patches" in a 2D frame), which are further coded by a legacy 2D video codec, such as the High Efficiency Video Coding (HEVC) (ITU-T H.265) codec. The second approach is "Geometry-Based Point Cloud Compression" (G-PCC), which directly compresses the 3D geometry, i.e., the positions of a set of points in 3D space and the associated attribute values ​​(for each point associated with the 3D geometry). G-PCC addresses the compression of point clouds in both category 1 (static point clouds) and category 3 (dynamically acquired point clouds). A recent draft of the G-PCC standard is available in "Text of ISO / IEC FDIS 23090-9 Geometry-based Point Cloud Compression", ISO / IEC JTC 1 / SC29 / WG 7 MDS19617, Teleconference, October 2020, and a description of the codec is available in G-PCC Codec Description, ISO / IEC JTC 1 / SC29 / WG 7 MDS19620, Teleconference, October 2020.

[0045] A point cloud includes a set of points in 3D space and may have attributes associated with the points. The attributes may be color information such as R, G, B, or Y, Cb, Cr, or reflectance information, or other attributes. Point clouds may be captured by various cameras or sensors, such as LIDAR sensors and 3D scanners, or may be computer generated. Point cloud data is used in a variety of applications, including but not limited to construction (modeling), graphics (3D models for visualization and animation), and the automotive industry (LIDAR sensors used to aid navigation).

[0046] The 3D space occupied by the point cloud data may be enclosed by a virtual bounding box. The positions of the points in the bounding box may be represented with a certain precision, and thus the positions of one or more points may be quantized based on the precision. At the smallest level, the bounding box is divided into voxels, which are the smallest units of space represented by a unit cube. A voxel in the bounding box may be associated with zero, one, or multiple points. The bounding box may be divided into multiple cubic / rectangular regions, which may be called tiles. Each tile may be coded into one or multiple slices. The partitioning of the bounding box into slices and tiles may be based on the number of points in each partition or based on other considerations (e.g., a particular region may be coded as a tile). The slice regions may be further partitioned using partitioning decisions similar to those in video codecs.

[0047] Figure 2 provides an overview of a G-PCC encoder 200. Figure 3 provides an overview of a G-PCC decoder 300. The modules shown are logical and do not necessarily correspond one-to-one to the code implemented in the reference implementation of the G-PCC codec, i.e., the TMC13 test model software studied by ISO / IEC MPEG (JTC1 / SC29 / WG11).

[0048] In both the G-PCC encoder 200 and the G-PCC decoder 300, the point cloud position is coded first. The attribute coding depends on the decoded geometry. The surface approximation analysis unit 212 and the RAHT unit 218 in FIG. 2, and the surface approximation synthesis unit 310 and the RAHT unit 314 in FIG. 3 are options typically used for category 1 data. The lifting units of the LOD generation units 220 and 222, and the LOD generation unit 316 and the inverse lifting unit 318 in FIG. 3 are options typically used for category 3 data. All other modules are common between categories 1 and 3.

[0049] There are two different types of coding techniques for geometry: octree and predictive tree coding. In the following, this disclosure focuses on octree coding. For category 3 data, the compressed geometry is typically represented as an octree from the root to the leaf level of individual voxels. For category 3 data, the compressed geometry is typically represented as an octree from the root to the leaf level of individual voxels. For category 1 data, the compressed geometry is typically represented by a pruned octree (i.e., an octree from the root down to the leaf level of a block larger than a voxel) plus a model that approximates the surface in each leaf of the pruned octree. In this way, both category 1 and category 3 data share the octree coding mechanism, and category 1 data may additionally approximate the voxels in each leaf with a surface model. The surface model used is a triangulation involving 1-10 triangles per block, resulting in a triangle soup. Thus, category 1 geometry codecs are known as Trisoup geometry codecs, and category 3 geometry codecs are known as octree geometry codecs.

[0050] 4 is a conceptual diagram illustrating an example octree partitioning for geometry coding. Octree 400 includes eight child nodes. Some of those child nodes, such as node 402, have no child nodes. However, others of the child nodes, such as node 404, have child nodes, some of the child nodes of node 404 also have child nodes, and so on.

[0051] At each node of the octree, occupancy is signaled (when not inferred) for one or more of the child nodes (up to 8 nodes). Multiple neighborhoods are specified, including (a) nodes that share faces with the current octree node, (b) nodes that share faces, edges, or vertices with the current octree node, etc. Within each neighborhood, the occupancy of the node and / or its children may be used to predict the occupancy of the current node or its children. For sparse points in some nodes of the octree, the codec also supports a direct coding mode, where the 3D position of the point is directly coded. A flag may be signaled to indicate that the direct mode is signaled. At the lowest level, the number of points associated with an octree node / leaf node may also be coded.

[0052] When geometry is coded, attributes corresponding to the geometry points are coded. When there are multiple attribute points corresponding to one reconstructed / decoded geometry point, an attribute value representing the reconstructed point can be derived.

[0053] There are three attribute coding methods in G-PCC: Region Adaptive Hierarchical Transform (RAHT) coding, Interpolation-based Hierarchical Nearest Neighbor Prediction (Prediction Transform), and Interpolation-based Hierarchical Nearest Neighbor Prediction with Update / Lifting Steps (Lifting Transform). RAHT and lifting are typically used for category 1 data, and prediction is typically used for category 3 data. However, either method may be used for any data, just as with geometry codecs in G-PCC, the attribute coding method used to code the point cloud is specified in the bitstream.

[0054] The coding of attributes may be performed at a certain level of detail (LOD), where each level of detail may be used to obtain a more precise representation of the point cloud attributes, which may be specified based on a distance metric from neighboring nodes or based on a sampling distance.

[0055] In the G-PCC encoder 200, the residual obtained as the output of the coding method for the attribute is quantized. The quantized residual may be coded using context-adaptive arithmetic coding.

[0056] In the example of FIG. 2, the G-PCC encoder 200 may include a coordinate transformation unit 202, a color transformation unit 204, a voxelization unit 206, an attribute transfer unit 208, an octree analysis unit 210, a surface approximation analysis unit 212, an arithmetic coding unit 214, a geometry reconstruction unit 216, a RAHT unit 218, an LOD generation unit 220, a lifting unit 222, a coefficient quantization unit 224, and an arithmetic coding unit 226.

[0057] As shown in the example of Figure 2, the G-PCC encoder 200 may receive a set of locations and a set of attributes. The locations may include coordinates of points in the point cloud. The attributes may include information about the points in the point cloud, such as a color associated with the points in the point cloud.

[0058] The coordinate transformation unit 202 may apply a transformation to the coordinates of the points to convert the coordinates from an initial domain to a transformation domain. This disclosure may refer to the transformed coordinates as transformed coordinates. The color transformation unit 204 may apply a transformation to convert color information of the attribute to a different domain. For example, the color transformation unit 204 may convert the color information from an RGB color space to a YCbCr color space.

[0059] Further, in the example of FIG. 2, the voxelization unit 206 may voxelize the transformed coordinates. The voxelization of the transformed coordinates may include quantization and removing some points of the point cloud. In other words, multiple points of the point cloud may be contained within a single "voxel", which may then be treated as one point in some respects. Further, the octree analysis unit 210 may generate an octree based on the voxelized transformed coordinates. Additionally, in the example of FIG. 2, the surface approximation analysis unit 212 may analyze the points to potentially determine a surface representation of the set of points. The arithmetic coding unit 214 may entropy code syntax elements that represent the octree and / or surface information determined by the surface approximation analysis unit 212. The G-PCC encoder 200 may output these syntax elements in a geometry bitstream.

[0060] The geometry reconstruction unit 216 may reconstruct transformation coordinates of points in the point cloud based on the octree, the data indicative of the surface determined by the surface approximation analysis unit 212, and / or other information. The number of transformation coordinates reconstructed by the geometry reconstruction unit 216 may differ from the original number of points in the point cloud due to voxelization and surface approximation. This disclosure may refer to the resulting points as reconstructed points. The attribute transfer unit 208 may transfer attributes of the original points of the point cloud to the reconstructed points of the point cloud.

[0061] Further, the RAHT unit 218 may apply RAHT coding to the attributes of the reconstructed points. Alternatively or additionally, the LOD generation unit 220 and the lifting unit 222 may apply LOD processing and lifting, respectively, to the attributes of the reconstructed points. The RAHT unit 218 and the lifting unit 222 may generate coefficients based on the attributes. The coefficient quantization unit 224 may quantize the coefficients generated by the RAHT unit 218 or the lifting unit 222. The arithmetic coding unit 226 may apply arithmetic coding to syntax elements representing the quantized coefficients. The G-PCC encoder 200 may output these syntax elements in an attribute bitstream.

[0062] In the example of FIG. 3, the G-PCC decoder 300 may include a geometry arithmetic decoding unit 302, an attribute arithmetic decoding unit 304, an octree synthesis unit 306, an inverse quantization unit 308, a surface approximation synthesis unit 310, a geometry reconstruction unit 312, a RAHT unit 314, a LoD generation unit 316, an inverse lifting unit 318, an inverse coordinate transformation unit 320, and an inverse color transformation unit 322.

[0063] The G-PCC decoder 300 may obtain a geometry bitstream and an attribute bitstream. The geometry arithmetic decoding unit 302 of the G-PCC decoder 300 may apply arithmetic decoding (e.g., context-adaptive binary arithmetic coding (CABAC) or other types of arithmetic decoding) to syntax elements in the geometry bitstream. Similarly, the attribute arithmetic decoding unit 304 may apply arithmetic decoding to syntax elements in the attribute bitstream.

[0064] The octree synthesis unit 306 may synthesize an octree based on syntax elements parsed from the geometry bitstream. In cases where surface approximation is used in the geometry bitstream, the surface approximation synthesis unit 310 may determine a surface model based on the syntax elements parsed from the geometry bitstream and based on the octree.

[0065] Further, the geometry reconstruction unit 312 may perform the reconstruction to determine coordinates of points in the point cloud. The inverse coordinate transformation unit 320 may apply an inverse transformation to the reconstructed coordinates to convert the reconstructed coordinates (positions) of the points in the point cloud from the transformed domain back to the original domain.

[0066] 3, the inverse quantization unit 308 may inverse quantize the attribute value. The attribute value may be based on a syntax element obtained from the attribute bitstream (e.g., including a syntax element decoded by the attribute arithmetic decoding unit 304).

[0067] Depending on how the attribute values ​​are encoded, the RAHT unit 314 may perform RAHT coding to determine color values ​​for the points of the point cloud based on the dequantized attribute values. Alternatively, the LOD generation unit 316 and the inverse lifting unit 318 may use level-of-detail based techniques to determine color values ​​for the points of the point cloud.

[0068] 3, the inverse color transformation unit 322 may apply an inverse color transformation to the color values. The inverse color transformation may be the inverse of the color transformation applied by the color transformation unit 204 of the G-PCC encoder 200. For example, the color transformation unit 204 may transform the color information from the RGB color space to the YCbCr color space. Accordingly, the inverse color transformation unit 322 may transform the color information from the YCbCr color space to the RGB color space.

[0069] The various units in FIG. 2 and FIG. 3 are shown to aid in understanding the operations performed by the G-PCC encoder 200 and the G-PCC decoder 300. The units may be implemented as fixed-function circuits, programmable circuits, or a combination thereof. A fixed-function circuit refers to a circuit that provides a specific function and is preset for the operations that may be performed. A programmable circuit refers to a circuit that may be programmed to perform various tasks and provides flexible functionality in the operations that may be performed. For example, a programmable circuit may execute software or firmware that causes the programmable circuit to operate in a manner defined by the software or firmware instructions. A fixed-function circuit may execute software instructions (e.g., to receive parameters or output parameters), but the type of operations that the fixed-function circuit performs is generally invariant. In some examples, one or more of the units may be different circuit blocks (fixed function or programmable), and in some examples, one or more of the units may be an integrated circuit.

[0070] Predictive geometry coding is introduced as an alternative to octree geometry coding, where nodes are arranged in a tree structure (which defines the predictor structure) and various prediction strategies are used to predict the coordinates of each node in the tree associated with that predictor.

[0071] 5 shows an example of a prediction tree 410, represented as a directed graph with arrows pointing in the prediction direction. Node 412 is the root vertex and has no predictors. Nodes 414A and 414B have two children. The dashed node has three children. The white node has one child, and nodes 418A-418E are leaf nodes with no children. Each node has only one parent node.

[0072] Four prediction strategies are specified for each node based on its parent (p0), grandfather (p1) and grandfather's father (p2).

[0073] No prediction / Zero prediction (0) Delta prediction (p0) Linear prediction (2*p0-p1) Parallelogram prediction (2*p0+p1-p2) The G-PCC encoder 200 may employ any algorithm to generate the prediction tree, and the algorithm used may be determined based on the application / use case, and several strategies may be used, some of which are described in G-PCC Codec Description, ISO / IEC JTC 1 / SC29 / WG 7 MDS19620, Teleconference, October 2020.

[0074] For each node, the residual coordinate values ​​are coded in the bitstream starting from the root node, depth-first.

[0075] Predictive geometry coding is primarily useful for category 3 (LIDAR-acquired) point cloud data, e.g., for low latency applications.

[0076] An angular mode may be used in predictive geometry coding, where the characteristics of the LIDAR sensor can be exploited in coding the prediction tree more efficiently. The coordinates of the position are converted to (r,φ,i) (radius, azimuth, and laser index) and the prediction is performed in this domain (residuals are coded in the r,φ,i domain). Due to errors in rounding, coding in r,φ,i is not lossless, and therefore a second set of residuals corresponding to Cartesian coordinates is coded. A description of the encoding and decoding strategy used for the angular mode for predictive geometry coding is reproduced below from G-PCC Codec Description, ISO / IEC JTC 1 / SC29 / WG 7 MDS19620, Teleconference, October 2020.

[0077] The method focuses on point clouds acquired using a rotating Lidar model, where the Lidar has N lasers (e.g., N=16, 32, 64) that rotate around the Z axis according to an azimuth angle φ (see FIG. 6). Each laser is rotated at a different elevation angle θ(i). i=1...N and height σ(i) i=1...N Assume that laser i strikes a point M with Cartesian integer coordinates (x, y, z) defined according to the coordinate system described in FIG.

[0078] This method provides for modeling the location of M with three parameters (r, φ, i), calculated as follows:

[0079]

number

[0080]

number

[0081]

number

[0082]

number

[0083]

number

[0084]

number

[0085]

number

[0086] · sign(t) is a function that returns 1 if t is positive, and (-1) otherwise.

[0087] |t| is the absolute value of t.

[0088] To avoid reconstruction inconsistencies due to the use of floating-point arithmetic, σ(i) i=1...N and tan(θ(i)) i=1...N is precomputed and quantized as follows:

[0089]

number

[0090]

number

[0091]

number

[0092] The reconstructed Cartesian coordinates are obtained as follows:

[0093]

number

[0094]

number

[0095] (r x ,r y ,r z ) be the reconstruction residual, defined as

[0096]

number

[0097] Model parameters

[0098]

number

[0099]

number

[0100]

number

[0101]

number

[0102]

number

[0103] o n(j) is the number of skipped points, which may be explicitly written into the bitstream or inferred from the context based on a deterministic strategy applied by both the encoder and the decoder. n(j) may also be referred to later as the "phi multiplier" and in some implementations may only be used with the delta predictor.

[0104] Using each node, we calculate the reconstruction residual (r x ,r y ,rz ) The G-PCC decoder 300 proceeds as follows.

[0105] Model parameters

[0106]

number

[0107]

number

[0108]

number

[0109] The reconstructed coordinates as explained above

[0110]

number

[0111]

number

[0112] The quantized reconstruction residual is calculated as follows:

[0113]

number

[0114]

number

[0115]

number

[0116]

number

[0117] To further improve the RD (rate-distortion) performance results, trellis quantization can be used.

[0118] The quantization parameters may be varied at the sequence / frame / slice / block level to achieve region adaptive quality and for rate control purposes.

[0119] The G-PCC encoder 200 may be configured to perform motion estimation for inter prediction. In the following, the motion estimation (global and local) process applied in the InterEM software is described. InterEM is based on an octree-based coding extension for inter prediction. Although the motion estimation is applied in an octree-based framework, a similar process (or at least a part of it) may also be applicable to predictive geometry coding.

[0120] There are two types of motion involved in the G-PCC InterEM software: global motion matrices and local nodal motion vectors. Global motion parameters include rotation matrices and translation vectors that may be applied to all points in the predicted (reference) frame. Local nodal motion vectors of a node in the octree are motion vectors that are applied only to points within the node in the predicted (reference) frame. Details of the motion estimation algorithm in InterEM are described below.

[0121] FIG. 7 shows a flow chart illustrating the motion estimation process. The input to the process includes a predicted frame 420 and a current frame 422. The G-PCC encoder 200 first estimates global motion at a global scale (424). After applying the estimated global motion to the predicted frame 420 (426), the G-PCC encoder 200 estimates local motion at a finer scale, the node level in the octree (428). Finally, the G-PCC encoder 200 applies motion compensation to the estimated local node motion and encodes the determined motion vectors and points (430).

[0122] The embodiment of Figure 7 is described in more detail below. The G-PCC encoder 200 may perform a process to estimate a global motion matrix and a translation vector. In the InterEM software, the global motion matrix is ​​defined to match feature points between a predicted frame (reference) and a current frame.

[0123] Figure 8 illustrates an example of a global motion estimation process that may be performed by the G-PCC encoder 200. In the example of Figure 8, the G-PCC encoder 200 finds feature points (432), samples the feature points (434), and performs motion estimation using a least mean squares (LMS) algorithm (436).

[0124] In the algorithm shown by Figure 8, points that have large position changes between the predicted frame and the current frame can be defined as feature points. For each point in the current frame, the closest point in the predicted frame is found, and point pairs are constructed between the current frame and the predicted frame. If the distance between the paired points is greater than a threshold, the paired points are considered as feature points.

[0125] After finding the feature points, sampling is performed on the feature points to reduce the scale of the problem (e.g., by selecting a subset of feature points to reduce the complexity of motion estimation).The LMS algorithm is then applied to derive motion parameters by attempting to reduce the error between each feature point in the predicted frame and the current frame.

[0126] FIG. 9 illustrates an exemplary process for estimating local node motion vectors. In the local node estimation algorithm illustrated in FIG. 9, motion vectors are estimated in a recursive manner. The cost function used to select the most suitable motion vector may be based on a rate-distortion cost. In FIG. 9, path 440 illustrates the process for a current node that is not split into eight children, and path 442 illustrates the process for a current node that is split into eight children.

[0127] If the current node is not split into 8 children (440), the motion vector that can result in the lowest cost between the current node and the predicted node is determined. If the current node is split into 8 children (442), a motion estimation algorithm is applied and the total cost under the split condition is obtained by adding the estimated cost value of each child node. The decision to split or not to split is reached by comparing the cost between split and no split; if split, each subnode is assigned its respective motion vector (or may be further split up to its children), if not split, the current node is assigned a motion vector.

[0128] Two parameters that affect the performance of motion vector estimation are block size (BlockSize) and minimum prediction unit size (MinPUSize). BlockSize defines the upper limit and MinPUSize defines the lower limit of the node size for applying motion vector estimation.

[0129] The InterEM software, which is basically an octree coder, performs occupied prediction and uses global / local motion and reference point cloud information while performing occupied prediction. Therefore, the InterEM software does not perform direct motion compensation of points, which may include, for example, applying motion to a point in a reference frame to project the point onto the current frame. The difference between the actual point and the predicted point may then be coded, which may be more effective in performing inter prediction.

[0130] One or more techniques disclosed in this document may be applied independently or combined. This disclosure proposes techniques for performing direct motion compensation while still benefiting from a flexible octree partition-based coding structure. In the following, these techniques are primarily presented in the context of octree partitioning, but may also be extended to OTQTBT (octree-quadtree-binary) partitioning scenarios.

[0131] The G-PCC encoder 200 and / or the G-PCC decoder 300 may be configured to perform high-level partitioning and process mode flags. In one example of the present disclosure, the G-PCC encoder 200 and / or the G-PCC decoder 300 may be configured to perform an octree-based partitioning (for occupancy prediction) on the current point cloud. However, it is possible to stop the octree partitioning at a level and then directly code the points inside the octree leaf volume (hereinafter referred to as "direct prediction") instead of coding the occupancy. A leaf node size, or an octree depth value, may be signaled to specify the level at which the octree partitioning stops and the points are coded as octree leaf volumes.

[0132] For each such octree leaf node, a flag may be signaled to indicate whether the set of points inside the octree leaf volume are intra-predicted or inter-predicted when the octree splitting stops and "direct prediction" is activated. In the geometry parameter set, maximum and minimum sizes for the octree leaves may be defined.

[0133] Figure 10 is a conceptual diagram illustrating a high-level octree partitioning of octree 444. Figure 10 is an example of an octree leaf node for direct prediction that contains 13 points (O0-O12). In a special case, the root node of the octree (no partition) may be coded using "direct prediction".

[0134] The G-PCC encoder 200 and / or the G-PCC decoder 300 may be configured to perform intra prediction. When the flag value is set to intra, all points inside the volume are intra predicted. For this purpose, a "local prediction tree" is generated. The generation of such a tree is non-normative (points can be traversed in different orders by azimuth, Morton, radial, etc., or in some other order). For each point, its prediction mode (0, 1, 2, 3), information on the number of children, the first order residual, and the second order residual (if the angular mode is enabled) are signaled. So, in summary, intra prediction is similar in its function to predictive geometry coding.

[0135] Alternatively, a single prediction mode may also be signaled for all points within the octree leaf volume, thereby reducing associated signaling costs. The radius value for the zero predictor (if the angle mode is enabled), or the (x, y, z) value (if the angle mode is disabled), may be set to, for example, the top-left point inside the octree leaf volume. Alternatively, the zero predictor may be signaled for the octree leaf volume, or an index indicating a point to use for the zero predictor inside the octree leaf volume may be signaled. Moreover, if the value is outside the octree leaf volume, clipping may be performed after performing prediction / reconstruction.

[0136] The syntax table for intra prediction may be similar to the syntax described for prediction trees in the Text of ISO / IEC FDIS 23090-9 Geometry-based Point Cloud Compression, ISO / IEC JTC 1 / SC29 / WG 7 MDS19617, Teleconference, October 2020, the entire contents of which are incorporated by reference.

[0137] The G-PCC encoder 200 and / or the G-PCC decoder 300 may be configured to perform inter prediction. Assuming that an octree leaf has N points inside, (O(0), ...., O(N-1)), in the case of inter prediction, motion estimation is performed at the encoder side with the current set of points in the octree leaf volume to find the best match with a similar set of points in the reference point cloud frame (where the reference point cloud can be either non-motion compensated or globally motion compensated). In the case of inter prediction of an octree leaf, the following is signaled:

[0138] i. Reference index (if there are multiple reference point cloud frames to predict from) ii. Motion Vector Difference (MVD). (The difference between the actual MV and the predicted MV (as explained above for performing MV prediction from neighbors) iii. The number of points in an octree leaf (N).

[0139] iv. The first order (and also second order, if angle mode is enabled) residuals (tuplets, differences between 3D coordinates) for the N points (R'i).

[0140] In the following, this disclosure describes the motion compensation process given the signaled reference indexes (if applicable) and MVs for the octree nodes.

[0141] The current octree leaf has its top-left point at (X0, Y0, Z0) and dimensions as (Sx, Sy, Sz), and the motion vector is MV = (MVx, MVy, MVz). Therefore, the corresponding reference block in the reference point cloud frame has its top-left point at (Xr, Yr, Zr) = (X0 - MVx, Y0 - MVy, Z0 - MVz) and size of (Sx, Sy, Sz).

[0142] b. Fetch all points inside this reference block and arranged in a 1D array, the order can be predetermined / fixed or signaled, for example, for octree leaves. For example, suppose there are M such points with coordinates (in the reference frame), R0,.....R(M-1), where Ri is a triplet providing the 3D coordinate of the i-th point, for i=0...(M-1) as shown in Figure 12.

[0143] c. All points are motion compensated by applying the signaled MV, which is used as the predicted geometric position (Pi), i.e., Pi = Ri + MV, as shown in Figure 13.

[0144] d. If angle mode is enabled, for all M points,

[0145]

number

[0146] In Figure 12, the current set of points is labeled O0-O12, the reference set of points is labeled R0-R12, and N = M = 13. In Figure 13, the current set of points is labeled O0-O12, and the motion compensated reference set of points is labeled P0-P12, and N = M = 13.

[0147] There are three possible scenarios here.

[0148] iN = M (the current octree node and the reference block have the same number of points).

[0149] ii.N > M.

[0150] iii. N < M.

[0151] The first scenario, where N=M, is described below: The residuals (first order and, if applicable, second order residuals) are added directly to the motion compensated points to produce the reconstruction = Pi + R'i.

[0152] The second scenario where N>M is described below. The motion compensated point (Pi) in the 1D array is expanded using the last value P(M-1), i.e., [P'0, ......P'(M-1), P'(M),.....P'(N-1)] = [P0, ......P(M-1), P(M-1),.....P(M-1)], and then the residuals are added directly to generate the reconstruction = P'i + R'i. Alternatively, a zero predictor is used for the expansion.

[0153] The third scenario, where N < M, is described below: The residual is added directly to the first N points, i.e., [P0, ....P(N-1)], to produce the reconstruction = Pi + Ri.

[0154] The G-PCC encoder 200 and / or the G-PCC decoder 300 may be configured to perform MV prediction from neighbors. It is possible to predict the MV of the current octree leaf from the MV of the spatiotemporal neighboring inter-octree leaf, and the corresponding MV difference may be signaled. The MV prediction index may be signaled when there are multiple spatiotemporal candidates. In addition to the spatiotemporal neighbor candidates, it is also possible to add previously used MV candidates in the MV candidate list based on recent history.

[0155] It is also possible to merge MV information with spatio-temporal neighbors by specifying a signaled "merge flag." A merge index can be signaled when there are multiple spatio-temporal candidates.

[0156] The G-PCC encoder 200 and / or the G-PCC decoder 300 may be configured to perform first order residual skipping.

[0157] For good inter prediction, the first order residual, which is applicable when the angle mode is enabled, is typically small or even close to 0. In such a case, it is also possible to skip the first order residual completely for all points in the octree leaf volume. Therefore, a primary_residual_skip flag can be signaled for the octree leaf volume. In this case, the difference between the original point and the predicted point is completely coded in the second order residual.

[0158] Alternatively, the primary_residual_skip_flag may be signaled at an octree level higher than the octree leaf volume and applies to one or more octree leaves associated with the octree level.

[0159] The following table is a syntax table for inter predicted octree leaves.

[0160] [Table 1] The examples in the various aspects of the present disclosure may be used individually or in any combination.

[0161] FIG. 14 is a conceptual diagram illustrating an example distance measurement system 600 that may be used with one or more techniques of the present disclosure. In the example of FIG. 14, the distance measurement system 600 includes an illuminator 602 and a sensor 604. The illuminator 602 may emit light 606. In some examples, the illuminator 602 may emit the light 606 as one or more laser beams. The light 606 may be in one or more wavelengths, such as infrared wavelengths or visible light wavelengths. In other examples, the light 606 is not a coherent laser light. When the light 606 encounters an object, such as an object 608, the light 606 produces return light 610. The return light 610 may include backscattered light and / or reflected light. The return light 610 may pass through a lens 611 that directs the return light 610 to produce an image 612 of the object 608 on the sensor 604. The sensor 604 generates a signal 618 based on the image 612. Image 612 may include a set of points (eg, as represented by the dots in image 612 of FIG. 14).

[0162] In some examples, the illuminator 602 and sensor 604 may be mounted on a rotating structure such that the illuminator 602 and sensor 604 capture a 360-degree view of the environment. In other examples, the distance measurement system 600 may include one or more optical components (e.g., mirrors, collimators, diffraction gratings, etc.) that enable the illuminator 602 and sensor 604 to detect objects within a certain range (e.g., up to 360 degrees). Although the example of FIG. 14 shows only a single illuminator 602 and sensor 604, the distance measurement system 600 may include multiple sets of illuminators and sensors.

[0163] In some examples, the illuminator 602 generates a structured light pattern. In such examples, the distance measurement system 600 may include multiple sensors 604 on which respective images of the structured light pattern are formed. The distance measurement system 600 may use the parallax between the images of the structured light pattern to determine the distance to an object 608 from which the structured light pattern is backscattered. The structured light based distance measurement system may have a high level of accuracy (e.g., accuracy in the sub-millimeter range) when the object 608 is relatively close (e.g., 0.2 meters to 2 meters) to the sensor 604. This high level of accuracy may be useful in facial recognition applications such as unlocking a mobile device (e.g., a mobile phone, a tablet computer, etc.) and for security applications.

[0164] In some examples, the distance measurement system 600 is a time-of-flight (ToF) based system. In some examples where the distance measurement system 600 is a ToF based system, the illuminator 602 generates a pulse of light. In other words, the illuminator 602 may modulate the amplitude of the emitted light 606. In such examples, the sensor 604 detects the return light 610 from the pulse of light 606 generated by the illuminator 602. The distance measurement system 600 may then determine the distance to the object 608 from which the light 606 is backscattered based on the delay between when the light 606 is emitted and when it is detected and the known speed of light in air. In some examples, rather than (or in addition to) modulating the amplitude of the emitted light 606, the illuminator 602 may modulate the phase of the emitted light 606. In such an example, the sensor 604 may detect the phase of the returning light 610 from the object 608 and determine the distance to a point on the object 608 using the speed of light and based on the time difference between when the illuminator 602 generates the light 606 at a particular phase and when the sensor 604 detects the returning light 610 at that particular phase.

[0165] In other examples, the point cloud may be generated without the use of the illuminator 602. For example, in some examples, the sensor 604 of the distance measurement system 600 may include two or more optical cameras. In such examples, the distance measurement system 600 may use the optical cameras to capture a stereo image of an environment including the object 608. The distance measurement system 600 (e.g., the point cloud generator 620) may then calculate the disparity between locations in the stereo image. The distance measurement system 600 may then use the disparity to determine distances to locations shown in the stereo image. From these distances, the point cloud generator 620 may generate a point cloud.

[0166] The sensor 604 may also detect other attributes of the object 608, such as color and reflectance information. In the example of Figure 14, the point cloud generator 620 may generate a point cloud based on the signal 618 generated by the sensor 604. The distance measurement system 600 and / or the point cloud generator 620 may form part of the data source 104 (Figure 1).

[0167] FIG. 15 is a conceptual diagram illustrating an example vehicle-based scenario in which one or more techniques of the present disclosure may be used. In the example of FIG. 15, a vehicle 700 includes a laser package 702, such as a LIDAR system. Although not shown in the example of FIG. 15, the vehicle 700 may also include a data source and a G-PCC encoder, such as G-PCC encoder 200 (FIG. 1). In the example of FIG. 15, the laser package 702 emits a laser beam 704 that reflects off pedestrians 706 or other objects in the road. The data source of the vehicle 700 may generate a point cloud based on a signal generated by the laser package 702. The G-PCC encoder of the vehicle 700 may encode the point cloud to generate a bit stream 708. The bit stream 708 may include many fewer bits than the unencoded point cloud obtained by the G-PCC encoder. An output interface of the vehicle 700 (e.g., output interface 108 (FIG. 1)) may transmit the bit stream 708 to one or more other devices. Thus, the vehicle 700 may be able to transmit the bitstream 708 to other devices more quickly than unencoded point cloud data. Additionally, the bitstream 708 may require less data storage capacity.

[0168] In the example of FIG. 15, the vehicle 700 may transmit a bitstream 708 to another vehicle 710. The vehicle 710 may include a G-PCC decoder, such as the G-PCC decoder 300 (FIG. 1). The G-PCC decoder of the vehicle 710 may decode the bitstream 708 to reconstruct a point cloud. The vehicle 710 may use the reconstructed point cloud for various purposes. For example, the vehicle 710 may determine that a pedestrian 706 is on the road ahead of the vehicle 700 based on the reconstructed point cloud, and thus begin to slow down, for example, even before the driver of the vehicle 710 recognizes that the pedestrian 706 is on the road. Thus, in some examples, the vehicle 710 may perform an autonomous navigation operation, generate a notification or warning, or take another action based on the reconstructed point cloud.

[0169] Additionally or alternatively, the vehicle 700 may transmit the bitstream 708 to the server system 712. The server system 712 may use the bitstream 708 for various purposes. For example, the server system 712 may store the bitstream 708 for subsequent reconstruction of the point cloud. In this example, the server system 712 may use the point cloud along with other data (e.g., vehicle telemetry data generated by the vehicle 700) to train an autonomous driving system. In other examples, the server system 712 may store the bitstream 708 for subsequent reconstruction for forensic crash investigation (e.g., if the vehicle 700 collides with the pedestrian 706) or transmit notifications or instructions for navigation to the vehicle 700 or the vehicle 710.

[0170] FIG. 16 is a conceptual diagram illustrating an example extended reality system in which one or more techniques of the present disclosure may be used. Extended reality (XR) is a term used to cover a variety of technologies including augmented reality (AR), mixed reality (MR), and virtual reality (VR). In the example of FIG. 16, a user 800 is located at a first location 802. The user 800 is wearing an XR headset 804. As an alternative to the XR headset 804, the user 800 may use a mobile device (e.g., a mobile phone, a tablet computer, etc.). The XR headset 804 includes a depth detection sensor, such as a LIDAR system, that detects the position of a point on an object 806 at the first location 802. A data source of the XR headset 804 may use a signal generated by the depth detection sensor to generate a point cloud representation of the object 806 at the first location 802. The XR headset 804 may include a G-PCC encoder (e.g., the G-PCC encoder 200 of FIG. 1) configured to encode the point cloud to generate a bitstream 808.

[0171] The XR headset 804 may transmit the bitstream 808 (e.g., over a network such as the Internet) to an XR headset 810 worn by a user 812 at a second location 814. The XR headset 810 may decode the bitstream 808 to reconstruct a point cloud. The XR headset 810 may use the point cloud to generate an XR visualization (e.g., AR, MR, VR visualization) representing the object 806 at the first location 802. Thus, in some examples, such as when the XR headset 810 generates a VR visualization, the user 812 at the location 814 may have a 3D immersive experience of the first location 802. In some examples, the XR headset 810 may determine a position of a virtual object based on the reconstructed point cloud. For example, the XR headset 810 may determine based on the reconstructed point cloud that an environment (e.g., the first location 802) includes a flat surface and then determine that a virtual object (e.g., a cartoon character) should be placed on the flat surface. The XR headset 810 may generate an XR visualization of the virtual object in the determined position. For example, the XR headset 810 may show a cartoon character sitting on a flat surface.

[0172] FIG. 17 is a conceptual diagram illustrating an example mobile device system in which one or more techniques of the present disclosure may be used. In the example of FIG. 17, a mobile device 900, such as a mobile phone or tablet computer, includes a depth-detecting sensor, such as a LIDAR system, that detects the location of points on an object 902 in an environment of the mobile device 900. A data source of the mobile device 900 may generate a point cloud representation of the object 902 using signals generated by the depth-detecting sensor. The mobile device 900 may include a G-PCC encoder (e.g., the G-PCC encoder 200 of FIG. 1) configured to encode the point cloud to generate a bitstream 904. In the example of FIG. 17, the mobile device 900 may transmit the bitstream to a remote device 906, such as a server system or another mobile device. The remote device 906 may decode the bitstream 904 to reconstruct the point cloud. The remote device 906 may use the point cloud for various purposes. For example, the remote device 906 may use the point cloud to generate a map of the environment of the mobile device 900. For example, the remote device 906 may generate a map of the interior of a building based on the reconstructed point cloud. In another example, the remote device 906 may generate imagery (e.g., computer graphics) based on the point cloud. For example, the remote device 906 may use the points of the point cloud as vertices of a polygon and use color attributes of the points as a basis for shading the polygon. In some examples, the remote device 906 may perform facial recognition using the point cloud.

[0173] Figure 18 is a flow chart illustrating example operations for decoding a bitstream that includes point cloud data. The G-PCC decoder 300 may perform the operations of Figure 18 as part of decoding a point cloud. In the example of Figure 18, the G-PCC decoder 300 determines (1000) an octree that defines an octree-based partitioning of a space that includes the point cloud. A leaf node of the octree contains one or more points of the point cloud.

[0174] The G-PCC decoder 300 directly codes the position of each of the one or more points in the leaf node (1002). To directly code the position of each of the one or more points in the leaf node, the G-PCC decoder 300 generates a prediction of the one or more points (1004) and determines the one or more points based on the prediction (1006). To directly decode the position of each of the one or more points in the leaf node, the G-PCC decoder 300 may be configured to receive a flag, where a first value for the flag indicates that the prediction of the one or more points is generated by intra prediction and a second value for the flag indicates that the prediction of the one or more points is generated by inter prediction, and to decode the one or more points using intra prediction or inter prediction based on the value of the flag.

[0175] The G-PCC decoder 300 may be configured to receive, in a bitstream for a point cloud, an octree leaf volume that specifies the volume of a leaf node. For example, assume that the entire point cloud is encapsulated within a W×W×W cuboid. The point cloud may be recursively partitioned, and for a given partition depth d, the octree leaf volume is W / 2 d ×W / 2 d ×W / 2 d At this level, an occupancy flag (binary) may be signaled, which indicates that the cuboid has at least one point when it is equal to 1. When the occupancy flag is 1, then a further intra or inter flag may be signaled, which indicates whether the points inside the cuboid are intra or inter predicted, respectively.

[0176] To generate a prediction of one or more points, the G-PCC decoder 300 may be further configured to generate a prediction of the one or more points using intra prediction, and to generate a prediction of the one or more points using intra prediction, the G-PCC decoder 300 may be further configured to determine a local prediction tree for the one or more points.

[0177] To determine the one or more points based on the prediction, the G-PCC decoder 300 may be configured to receive at least one of a prediction mode, a first order residual, and a second order residual for each of the one or more points in a bitstream for the point cloud. To generate a prediction of the one or more points, the G-PCC decoder 300 may be configured to generate a prediction of the one or more points using inter prediction, and to generate a prediction of the one or more points using inter prediction, the G-PCC decoder 300 may be further configured to perform motion estimation using the one or more points to determine a similar set of points in a reference point cloud frame.

[0178] To generate a prediction of one or more points, the G-PCC decoder 300 may be further configured to generate a prediction of one or more points using inter prediction, and to generate a prediction of one or more points using inter prediction, the G-PCC decoder 300 may be further configured to perform motion compensation to predict one or more points based on a set of points in a reference point cloud frame. To perform motion compensation, the G-PCC decoder 300 may be further configured to apply a motion vector to a set of points in a reference point cloud frame to determine a prediction of one or more points. The G-PCC decoder 300 may be configured to predict a motion vector based on the motion vectors of spatio-temporal neighboring inter octree leaves.

[0179] The G-PCC decoder 300 may be further configured to reconstruct a point cloud from the point cloud data. As part of reconstructing the point cloud, the G-PCC decoder 300 may be further configured to determine a position of one or more points of the point cloud based on the plane position.

[0180] It should be appreciated that, depending on the example, some acts or events of any of the techniques described herein may be performed in a different sequence and may be added, combined, or omitted entirely (e.g., not all acts or events described are necessary to the practice of the techniques). Moreover, in some examples, acts or events may be performed in parallel rather than sequentially, for example, through multithreaded processing, interrupt processing, or multiple processors.

[0181] In one or more examples, the functions described may be implemented as hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted via a computer-readable recording medium as one or more instructions or codes and executed by a hardware-based processing unit. A computer-readable recording medium may include a computer-readable storage medium corresponding to a tangible medium, such as a data storage medium, or a communication medium including any medium that facilitates transfer of a computer program from one place to another, for example according to a communication protocol. As such, a computer-readable recording medium may generally correspond to (1) a non-transitory tangible computer-readable storage medium, or (2) a communication medium such as a signal or carrier wave. A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementation of the techniques described in this disclosure. A computer program product may include a computer-readable recording medium.

[0182] By way of example and not limitation, such computer-readable storage media may comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and accessed by a computer. Also, any connection is properly referred to as a computer-readable storage medium. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of the medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transitory media, but instead cover non-transitory tangible storage media. As used herein, disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where disks typically reproduce data magnetically and discs reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer readable recording media.

[0183] The instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Thus, the terms "processor" and "processing circuitry" as used herein may refer to any of the above structures or any other structures suitable for implementing the techniques described herein. In addition, in some aspects, the functionality described herein may be provided within dedicated hardware and / or software modules configured for encoding and decoding, or may be incorporated into a composite codec. Also, the techniques may be fully implemented in one or more circuits or logic elements.

[0184] The techniques of this disclosure may be implemented in a wide variety of devices or apparatuses, including wireless handsets, integrated circuits (ICs), or sets of ICs (e.g., chipsets). Various components, modules, or units have been described in this disclosure to highlight functional aspects of devices configured to perform the disclosed techniques, but they do not necessarily require realization by different hardware units. Rather, as described above, the various units may be combined in a codec hardware unit or may be provided by a collection of interoperable hardware units including one or more processors as described above, along with suitable software and / or firmware.

[0185] The following numbered clauses refer to one or more aspects of the devices and techniques described in this disclosure.

[0186] Clause 1A: A method for coding a point cloud, comprising: determining an octree defining an octree-based division of a space containing the point cloud, wherein leaf nodes of the octree contain one or more points of the point cloud, and wherein the positions of each of the one or more points in the leaf nodes are directly signaled; generating a prediction of the one or more points using intra-prediction or inter-prediction; and coding a syntax element indicating whether the one or more points are predicted using intra-prediction or inter-prediction.

[0187] Clause 2A: The method of clause 1A, wherein an octree leaf volume that specifies a volume of a leaf node is signaled in the bitstream.

[0188] Clause 3A: The method of clause 1A or 2A, wherein generating a prediction for one or more points includes generating a prediction for the one or more points using intra prediction, and generating a prediction for the one or more points using intra prediction includes determining a local prediction tree for the one or more points.

[0189] Clause 4A: The method of clause 3A, wherein at least one of a prediction mode, a first order residual, and a second order residual is signaled for each of the one or more points.

[0190] Clause 5A: The method of clause 1A or 2A, wherein the step of generating a prediction of one or more points includes a step of generating a prediction of the one or more points using inter prediction, and the step of generating a prediction of the one or more points using inter prediction includes a step of performing motion estimation using the one or more points to determine a similar set of points in the reference point cloud frame.

[0191] Clause 6A: The method of any of clauses 1A, 2A, or 5A, wherein generating a prediction of one or more points includes generating a prediction of the one or more points using inter prediction, and generating a prediction of the one or more points using inter prediction includes performing motion compensation to predict the one or more points based on a set of points in a reference point cloud frame.

[0192] Clause 7A: The method of clause 6A, wherein performing motion compensation includes applying a motion vector to a set of points in the reference point cloud frame to determine a prediction of one or more points.

[0193] Clause 8A: The method of clause 7A, further comprising predicting the motion vector based on motion vectors of spatio-temporal neighboring inter-octree leaves.

[0194] Clause 9A: Any of the methods of clauses 1A to 8A, further comprising the step of generating a point cloud.

[0195] Clause 10A: A device for processing point clouds, comprising one or more means for performing the methods of any of clauses 1A to 9A.

[0196] Clause 11A: The device of clause 10A, wherein the one or more means comprise one or more processors implemented in circuitry.

[0197] Clause 12A: The device of any of clauses 10A or 11A, further comprising a memory for storing data representing the point cloud.

[0198] Clause 13A: A device according to any of clauses 10A to 12A, wherein the device comprises a decoder.

[0199] Clause 14A: The device of any of clauses 10A to 13A, wherein the device comprises an encoder.

[0200] Clause 15A: The device of any of clauses 10A to 14A, further comprising a device for generating a point cloud.

[0201] Clause 16A: The device of any of clauses 10A to 15A, further comprising a display for presenting an image based on the point cloud.

[0202] Clause 17A: A computer-readable storage medium having instructions stored thereon that, when executed, cause one or more processors to perform the method of any of clauses 1A to 9A.

[0203] Clause 1B: A device for decoding a bitstream including point cloud data, comprising: a memory for storing the point cloud data; and one or more processors coupled to the memory and implemented in circuitry, wherein the one or more processors are configured to determine an octree defining an octree-based partition of a space including the point cloud, wherein leaf nodes of the octree include one or more points of the point cloud, and to directly decode positions of each of the one or more points in the leaf nodes; and, in order to directly decode positions of each of the one or more points in the leaf nodes, the one or more processors are further configured to generate predictions of the one or more points and determine the one or more points based on the predictions.

[0204] Clause 2B: The device of clause 1B, wherein to directly decode a position of each of the one or more points in the leaf node, the one or more processors are further configured to receive a flag, where a first value for the flag indicates that a prediction of the one or more points is generated by intra prediction and a second value for the flag indicates that a prediction of the one or more points is generated by inter prediction, and to decode the one or more points using intra prediction or inter prediction based on the value of the flag.

[0205] Clause 3B: The device of clause 1B, wherein the one or more processors are further configured to receive an octree leaf volume in a bitstream including the point cloud, the octree leaf volume specifying a volume of a leaf node.

[0206] Clause 4B: The device of clause 1B, wherein, to generate a prediction of the one or more points, the one or more processors are further configured to generate a prediction of the one or more points using intra prediction, and, to generate a prediction of the one or more points using intra prediction, the one or more processors are further configured to determine a local prediction tree for the one or more points.

[0207] Clause 5B: The device of clause 1B, wherein the one or more processors are further configured to receive at least one of a prediction mode, a first order residual, and a second order residual for each of the one or more points in a bitstream including the point cloud, to determine the one or more points based on the prediction.

[0208] Clause 6B: The device of clause 1B, wherein to generate a prediction of the one or more points, the one or more processors are further configured to generate a prediction of the one or more points using inter prediction, and wherein to generate a prediction of the one or more points using inter prediction, the one or more processors are further configured to perform motion estimation using the one or more points to determine a similar set of points in a reference point cloud frame.

[0209] Clause 7B: The device of clause 1B, wherein to generate a prediction of one or more points, the one or more processors are further configured to generate a prediction of the one or more points using inter prediction, and wherein to generate a prediction of the one or more points using inter prediction, the one or more processors are further configured to perform motion compensation to predict the one or more points based on a set of points in a reference point cloud frame.

[0210] Clause 8B: The device of clause 7B, wherein to perform motion compensation, the one or more processors are further configured to apply motion vectors to a set of points in the reference point cloud frame to determine a prediction of one or more points.

[0211] Clause 9B: The device of clause 8B, wherein the one or more processors are further configured to predict a motion vector based on motion vectors of spatio-temporal neighboring inter-octree leaves.

[0212] Clause 10B: The device of clause 1B, wherein the one or more processors are further configured to reconstruct a point cloud from the point cloud data.

[0213] Clause 11B: The device of clause 10B, wherein the one or more processors are configured to determine positions of one or more points of the point cloud based on the planar positions as part of reconstructing the point cloud.

[0214] Clause 11B: The device of clause 10B, wherein the one or more processors are configured to determine, as part of reconstructing the point cloud, positions of one or more points of the point cloud based on directly decoded positions of each of the one or more points at the leaf node.

[0215] Clause 12B: The device of clause 11B, wherein the one or more processors are further configured to generate a map of the interior of the building based on the point cloud.

[0216] Clause 13B: The device of clause 11B, wherein the one or more processors are further configured to perform autonomous navigation operations based on the point cloud.

[0217] Clause 14B: The device of clause 11B, wherein the one or more processors are further configured to generate computer graphics based on the point cloud.

[0218] Clause 15B: The device of clause 11B, wherein the one or more processors are configured to determine a position of a virtual object based on the point cloud and generate an extended reality (XR) visualization in which the virtual object is at the determined position.

[0219] Clause 16B: The device of clause 11B, further comprising a display for presenting an image based on the point cloud.

[0220] Clause 17B: A device according to clause 1B, where the device is a mobile phone or a tablet computer.

[0221] Clause 18B: A device according to clause 1B where the device is a vehicle.

[0222] Clause 19B: A device according to clause 1B, where the device is an extended reality device.

[0223] Clause 20B: A method for decoding a point cloud, the method comprising: determining an octree defining an octree-based partition of a space containing the point cloud, wherein leaf nodes of the octree contain one or more points of the point cloud; and directly decoding a position of each of the one or more points at the leaf nodes, the directly decoding a position of each of the one or more points at the leaf nodes comprising: generating a prediction of the one or more points; and determining the one or more points based on the prediction.

[0224] Clause 21B: The method of clause 20B, wherein the step of directly decoding each position of the one or more points in the leaf node further includes the steps of receiving a flag, where a first value for the flag indicates that a prediction of the one or more points is generated by intra prediction and a second value for the flag indicates that a prediction of the one or more points is generated by inter prediction, and decoding the one or more points using intra prediction or inter prediction based on the value of the flag.

[0225] Clause 22B: The method of clause 20B, further comprising receiving an octree leaf volume in a bitstream for the point cloud that specifies a volume of a leaf node.

[0226] Clause 23B: The method of clause 20B, wherein generating a prediction for one or more points includes generating a prediction for the one or more points using intra prediction, and generating a prediction for the one or more points using intra prediction includes determining a local prediction tree for the one or more points.

[0227] Clause 24B: The method of clause 20B, wherein the step of determining one or more points based on the prediction includes a step of receiving at least one of a prediction mode, a first order residual, and a second order residual for each of the one or more points in a bitstream for the point group.

[0228] Clause 25B: The method of clause 20B, wherein the step of generating a prediction of one or more points includes a step of generating a prediction of the one or more points using inter prediction, and the step of generating a prediction of the one or more points using inter prediction includes a step of performing motion estimation using the one or more points to determine a similar set of points in the reference point cloud frame.

[0229] Clause 26B: The method of clause 20B, wherein generating a prediction of one or more points includes generating a prediction of the one or more points using inter prediction, and generating a prediction of the one or more points using inter prediction includes performing motion compensation to predict the one or more points based on a set of points in a reference point cloud frame.

[0230] Clause 27B: The method of clause 26B, wherein performing motion compensation includes applying motion vectors to a set of points in the reference point cloud frame to determine a prediction of one or more points.

[0231] Clause 28B: The method of clause 27B, further comprising predicting the motion vector based on the motion vectors of spatio-temporal neighboring inter-octree leaves.

[0232] Clause 29B: A computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to determine an octree defining an octree-based partition of a space including a point cloud, where a leaf node of the octree includes one or more points of the point cloud, and directly decode a position of each of the one or more points at the leaf node; and, in order to directly decode a position of each of the one or more points at the leaf node, generate a prediction of the one or more points, and determine the one or more points based on the prediction.

[0233] Clause 30B: An apparatus comprising: means for determining an octree defining an octree-based partition of a space including a point cloud, wherein leaf nodes of the octree include one or more points of the point cloud; and means for directly decoding a position of each of the one or more points at the leaf nodes, wherein the means for directly decoding a position of each of the one or more points at the leaf nodes comprises means for generating a prediction of the one or more points; and means for determining the one or more points based on the prediction.

[0234] Clause 1C: A device for decoding a bitstream including point cloud data, comprising: a memory for storing the point cloud data; and one or more processors coupled to the memory and implemented in circuitry, wherein the one or more processors are configured to determine an octree defining an octree-based partition of a space including the point cloud, wherein leaf nodes of the octree include one or more points of the point cloud, and to directly decode a position of each of the one or more points in the leaf nodes; and, in order to directly decode a position of each of the one or more points in the leaf nodes, the one or more processors are further configured to generate predictions of the one or more points and determine the one or more points based on the predictions.

[0235] Clause 2C: The device of Clause 1C, wherein to directly decode a position of each of the one or more points in the leaf node, the one or more processors are further configured to receive a flag, where a first value for the flag indicates that a prediction of the one or more points is generated by intra prediction and a second value for the flag indicates that a prediction of the one or more points is generated by inter prediction, and to decode the one or more points using intra prediction or inter prediction based on the value of the flag.

[0236] Clause 3C: The device of clause 1C or 2C, wherein the one or more processors are further configured to receive an octree leaf volume in a bitstream including the point cloud, the octree leaf volume specifying a volume of a leaf node.

[0237] Clause 4C: The device of any of clauses 1C to 3C, wherein, to generate a prediction of the one or more points, the one or more processors are further configured to generate a prediction of the one or more points using intra prediction, and wherein, to generate a prediction of the one or more points using intra prediction, the one or more processors are further configured to determine a local prediction tree for the one or more points.

[0238] Clause 5C: The device of any of clauses 1C to 4C, wherein the one or more processors are further configured to receive at least one of a prediction mode, a first order residual, and a second order residual for each of the one or more points in a bitstream including the point cloud, to determine the one or more points based on the prediction.

[0239] Clause 6C: The device of any of clauses 1C to 5C, wherein, to generate a prediction of the one or more points, the one or more processors are further configured to generate a prediction of the one or more points using inter prediction, and, to generate a prediction of the one or more points using inter prediction, the one or more processors are further configured to perform motion estimation using the one or more points to determine a similar set of points in a reference point cloud frame.

[0240] Clause 7C: The device of any of clauses 1C to 5C, wherein, to generate a prediction of the one or more points, the one or more processors are further configured to generate a prediction of the one or more points using inter prediction, and, to generate a prediction of the one or more points using inter prediction, the one or more processors are further configured to perform motion compensation to predict the one or more points based on a set of points in a reference point cloud frame.

[0241] Clause 8C: The device of clause 7C, wherein to perform motion compensation, the one or more processors are further configured to apply motion vectors to a set of points in the reference point cloud frame to determine a prediction of one or more points.

[0242] Clause 9C: The device of clause 8C, wherein the one or more processors are further configured to predict a motion vector based on motion vectors of spatio-temporal neighboring inter-octree leaves.

[0243] Clause 10C: The device of any of clauses 1C to 9C, wherein the one or more processors are further configured to reconstruct a point cloud from the point cloud data.

[0244] Clause 11C: The device of clause 10C, wherein the one or more processors are configured to determine, as part of reconstructing the point cloud, positions of one or more points of the point cloud based on directly decoded positions of each of the one or more points at the leaf node.

[0245] Clause 12C: The device of clause 11C, wherein the one or more processors are further configured to generate a map of the interior of the building based on the point cloud.

[0246] Clause 13C: The device of clause 11C, wherein the one or more processors are further configured to perform autonomous navigation operations based on the point cloud.

[0247] Clause 14C: The device of clause 11C, wherein the one or more processors are further configured to generate computer graphics based on the point cloud.

[0248] Clause 15C: The device of clause 11C, wherein the one or more processors are configured to determine a position of a virtual object based on the point cloud and generate an extended reality (XR) visualization in which the virtual object is at the determined position.

[0249] Clause 16C: The device of any of clauses 11C to 15C, further comprising a display for presenting an image based on the point cloud.

[0250] Clause 17C: Any device of clauses 1C to 16C, where the device is a mobile phone or a tablet computer.

[0251] Clause 18C: Any device of clauses 1C to 16C where the device is a vehicle.

[0252] Clause 19C: Any device of clauses 1C to 16C, where the device is an extended reality device.

[0253] Clause 20C: A method for decoding a point cloud, the method comprising: determining an octree defining an octree-based partition of a space containing the point cloud, wherein leaf nodes of the octree contain one or more points of the point cloud; and directly decoding a position of each of the one or more points at the leaf nodes, the directly decoding a position of each of the one or more points at the leaf nodes comprising: generating a prediction of the one or more points; and determining the one or more points based on the prediction.

[0254] Clause 21C: The method of clause 20C, wherein the step of directly decoding each position of the one or more points in the leaf node further includes a step of receiving a flag, where a first value for the flag indicates that a prediction of the one or more points is generated by intra prediction and a second value for the flag indicates that a prediction of the one or more points is generated by inter prediction, and a step of decoding the one or more points using intra prediction or inter prediction based on the value of the flag.

[0255] Clause 22C: The method of clause 20C or 21C, further comprising receiving an octree leaf volume in a bitstream for the point cloud, the octree leaf volume specifying a volume of a leaf node.

[0256] Clause 23C: Any of the methods of clauses 20C to 22C, wherein generating a prediction for one or more points includes generating a prediction for the one or more points using intra prediction, and generating a prediction for the one or more points using intra prediction includes determining a local prediction tree for the one or more points.

[0257] Clause 24C: Any of the methods of clauses 20C to 23C, wherein the step of determining one or more points based on a prediction includes a step of receiving at least one of a prediction mode, a first order residual, and a second order residual for each of the one or more points in a bitstream for the point group.

[0258] Clause 25C: Any of the methods of clauses 20C to 24C, wherein generating a prediction for one or more points includes generating a prediction for the one or more points using inter prediction, and generating a prediction for the one or more points using inter prediction includes performing motion estimation using the one or more points to determine a similar set of points in the reference point cloud frame.

[0259] Clause 26C: Any of the methods of clauses 20C to 25C, wherein generating a prediction for one or more points includes generating a prediction for the one or more points using inter prediction, and generating a prediction for the one or more points using inter prediction includes performing motion compensation to predict the one or more points based on a set of points in a reference point cloud frame.

[0260] Clause 27C: The method of clause 26C, wherein performing motion compensation includes applying a motion vector to a set of points in the reference point cloud frame to determine a prediction of one or more points.

[0261] Clause 28C: The method of clause 27C, further comprising predicting the motion vector based on the motion vectors of spatio-temporal neighboring inter-octree leaves.

[0262] Various examples have been described. These and other examples are within the scope of the following claims. [Explanation of symbols]

[0263] 100 Encoding and Decoding System, System 102 Source Device 104 Data Sources 106 Memory 108 Output Interface 110 Computer-readable recording medium 112 Storage Devices 114 File Server 116 Destination Device 118 Data Consumer 120 Memory 122 Input Interface 200G-PCC Encoder 202 Coordinate Transformation Unit 204 Color Conversion Unit 206 Voxelization Unit 208 Attribute Transfer Unit 210 8-tree analysis unit 212 Surface Approximation Analysis Unit 214, 226 Arithmetic Coding Unit 216, 312 Geometry Reconstruction Unit 218, 314 RAHT units 220 LOD generation units 222 Lifting Unit 224 Coefficient Quantization Unit 300 G-PCC Decoder 302 Geometry Arithmetic Decoding Unit 304 Attribute Arithmetic Decoding Unit 306 8th Tree Synthesis Unit 308 Inverse Quantization Unit 310 Surface Approximation Synthesis Unit 316 LoD Generation Units, LOD Generation Units 318 Reverse Lifting Unit 320 Inverse Coordinate Transformation Unit 322 Inverse Color Conversion Unit 400, 444 octree 402, 404 nodes 410 Prediction Trees 412, 414A, 414B, 418A~418E nodes 420 predicted frames 422 Current Frame 440, 442 Pass 600 Distance Measuring System 602 Illuminator 604 Sensor 606 Light, Synchrotron Radiation 608, 806, 902 objects 610 Return light 611 Lens 612 images 618 Signal 620 point cloud generator 700, 710 vehicles 702 Laser Package 704 Laser Beam 706 Pedestrians 708, 808, 904 bitstream 712 Server System 800, 812 users 802 1st Location 804, 810 XR Headset 814 Second Location 900 Mobile Devices 902 Objects 906 Remote Device

Claims

1. 1. A device for decoding a bitstream comprising point cloud data, comprising: A memory for storing the point cloud data; and one or more processors implemented in circuitry coupled to the memory, the one or more processors comprising: determining an octree defining an octree-based partition of a space containing the point cloud, a leaf node of the octree containing one or more points of the point cloud; receiving, in the bitstream including the point cloud, an octree leaf volume specifying a volume of the leaf node, for which a position of each of the one or more points in the leaf node is to be coded using a prediction within the leaf node; decoding a position of each of the one or more points in the leaf node; wherein in decoding the position of each of the one or more points in the leaf node, the one or more processors are configured to: generating a prediction of said one or more points; and determining said one or more points based on said prediction. The device is further configured to:

2. to directly decode the position of each of the one or more points in the leaf node, the one or more processors receiving a flag, a first value for the flag indicating that the prediction of the one or more points is generated by intra prediction and a second value for the flag indicating that the prediction of the one or more points is generated by inter prediction; and decoding the one or more points using intra prediction or inter prediction based on a value of the flag. The device of claim 1 , further configured to:

3. wherein, to generate the prediction of the one or more points, the one or more processors are further configured to generate the prediction of the one or more points using intra prediction; 13. The device of claim 1, wherein the one or more processors are further configured to: determine a local prediction tree for the one or more points to generate the prediction for the one or more points using intra prediction.

4. 2. The device of claim 1, wherein the one or more processors are further configured to receive at least one of a prediction mode, a first order residual, and a second order residual for each of the one or more points in the bitstream including the point cloud, to determine the one or more points based on the prediction.

5. To generate the prediction of the one or more points, the one or more processors are further configured to generate the prediction of the one or more points using inter prediction; 2. The device of claim 1, wherein to generate the prediction of the one or more points using inter prediction, the one or more processors are further configured to perform motion estimation using the one or more points to determine a similar set of points in a reference point cloud frame.

6. To generate the prediction of the one or more points, the one or more processors are further configured to generate the prediction of the one or more points using inter prediction; To generate the prediction of the one or more points using inter prediction, the one or more processors are further configured to perform motion compensation to predict the one or more points based on a set of points in a reference point cloud frame; To perform motion compensation, optionally the one or more processors are further configured to apply motion vectors to the set of points in the reference point cloud frame to determine a prediction of the one or more points; The one or more processors are optionally further configured to predict the motion vector based on motion vectors of spatio-temporal neighboring inter-octree leaves.

2. The device of claim 1.

7. The device of claim 1 , wherein the one or more processors are further configured to reconstruct the point cloud from the point cloud data.

8. 8. The device of claim 7, wherein the one or more processors are configured to determine, as part of reconstructing the point cloud, positions of one or more points of the point cloud based on the decoded positions of each of the one or more points in the leaf node.

9. the one or more processors are further configured to generate a map of an interior of a building based on the point cloud; or or or determining a position of a virtual object based on the point cloud; and generating an extended reality (XR) visualization of the virtual object at the determined position of the virtual object. The device of claim 8 , configured to:

10. The device of claim 8 , further comprising a display for presenting an image based on the point cloud.

11. The device of claim 1 , wherein the device is a mobile phone or a tablet computer.

12. The device of claim 1 , wherein the device is a vehicle.

13. The device of claim 1 , wherein the device is an extended reality device.

14. 1. A method for decoding a point cloud, comprising: determining an octree defining an octree-based partition of a space containing the point cloud, a leaf node of the octree containing one or more points of the point cloud; receiving, in a bitstream including the point cloud, an octree leaf volume specifying a volume of the leaf node, for which a position of each of the one or more points in the leaf node is to be coded using a prediction within the leaf node; decoding a position of each of the one or more points in the leaf node; and decoding the position of each of the one or more points in the leaf node comprises: generating a prediction of the one or more points; determining the one or more points based on the prediction; A method comprising:

15. 15. A computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform the method of claim 14.

Images