Locally Adaptive Inter Prediction for G-PCC
By allowing LPUs to have varying sizes and employing inter prediction, the method addresses the inefficiencies of strictly cubic LPUs, improving coding efficiency and reducing data requirements in point cloud compression.
Patent Information
- Application Number
- JP2024531165
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2022-11-30
- Filing Date
- 2022-12-01
- Publication Date
- 2024-11-21
AI Technical Summary
Existing point cloud compression techniques require LPUs to be strictly cubic, leading to misalignment of boundaries with octree cuboids, reducing coding efficiency and increasing data storage/transmission requirements.
Allowing LPUs to have different sizes along different directions, aligning with quadtree-binary tree splits, enabling flexible partitioning and inter prediction for improved coding efficiency.
Enhances coding efficiency by reducing data storage and transmission needs while maintaining effective representation of point cloud data.
Smart Images

Figure 2024543548000001_ABST
Abstract
Description
[Technical field]
[0001]
[0001] This application claims priority to U.S. Patent Application No. 18 / 060,208, filed November 30, 2022, and U.S. Provisional Patent Application No. 63 / 264,910, filed December 3, 2021, the entire contents of each of which are incorporated herein by reference. U.S. Patent Application No. 18 / 060,208, filed November 30, 2022, claims the benefit of U.S. Provisional Patent Application No. 63 / 264,910, filed December 3, 2021.
[0002]
[0002] This disclosure relates to point cloud encoding and decoding. [Background technology]
[0003]
[0003] A point cloud is a collection of points in a three-dimensional space. The points may correspond to points on an object in the three-dimensional space. Thus, a point cloud may be used to represent the physical content of a three-dimensional space. A point cloud may have utility in a wide variety of situations. For example, a point cloud may be used to represent the position of an object on a road in the context of an autonomous vehicle. In another example, a point cloud may be used to position virtual objects in an augmented reality (AR) or mixed reality (MR) application in the context of representing the physical content of an environment. Point cloud compression is the process of encoding and decoding a point cloud. Encoding a point cloud may reduce the amount of data required for storage and transmission of the point cloud. Summary of the Invention
[0004]
[0004] Generally, this disclosure describes techniques for Geometry Point Cloud Compression (G-PCC). The techniques of this disclosure may improve the performance of inter-coding using locally adaptive inter-prediction for G-PCC, such as the G-PCC standard being developed within MPEG by the 3-Dimensional Graphics (3DG) group. A point cloud may be divided into a number of Largest Prediction Units (LPUs). According to the techniques of this disclosure, at least two of the number of LPUs have different sizes along different directions. For example, an LPU may have a first size in a first direction (e.g., x, y, or z direction), and an LPU may have a second size in a second direction that is different from the first size. In other words, the LPUs need not be strictly cubic. Allowing the LPUs to have different sizes along different dimensions may allow the point cloud data to be divided into LPUs according to a quadtree-binary tree partitioning used for octree geometric coding of the point cloud data. The G-PCC encoder may perform inter prediction to determine a prediction point for the LPU and may encode points within the LPU based on the prediction point for the LPU. The G-PCC decoder may perform inter prediction to determine a prediction point for the LPU and may reconstruct points within the LPU based on the prediction point for the LPU. Allowing the LPUs to have different sizes along different directions may increase coding efficiency.
[0005]
[0005] In one example, the present disclosure describes a method for decoding point cloud data, the method including determining that the point cloud data is divided into multiple prediction units (LPUs), where at least two of the largest LPUs have different sizes along different directions, performing inter prediction to determine prediction points for the LPUs, and reconstructing points within the LPUs based on the prediction points for the LPUs.
[0006]
[0006] In another example, the present disclosure describes a method for encoding point cloud data, the method including determining that the point cloud data is divided into a plurality of largest prediction units (LPUs), where at least two of the plurality of LPUs have different sizes along different directions, performing inter prediction to determine prediction points for the LPUs, and encoding points within the LPUs based on the prediction points for the LPUs.
[0007]
[0007] In another example, the present disclosure describes a device for decoding point cloud data, the device comprising a memory configured to store the point cloud data and one or more processors implemented in a circuit configuration configured to determine that the point cloud data is divided into a plurality of largest prediction units (LPUs), at least two of the plurality of LPUs having different sizes along different directions, perform inter prediction to determine prediction points for the LPUs, and reconstruct points within the LPUs based on the prediction points for the LPUs.
[0008]
[0008] In another example, the present disclosure describes a device for encoding point cloud data, the device comprising a memory configured to store the point cloud data and one or more processors implemented in a circuit configuration configured to determine that the point cloud data is divided into a plurality of largest prediction units (LPUs), at least two of the plurality of LPUs having different sizes along different directions, perform inter prediction to determine prediction points for the LPUs, and encode points within the LPUs based on the prediction points for the LPUs.
[0009]
[0009] The details of one or more examples are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will become apparent from the description, drawings, and claims. [Brief description of the drawings]
[0010] [Figure 1]
[0010] FIG. 1 is a block diagram illustrating an example encoding and decoding system in which techniques of this disclosure may be implemented. [Diagram 2]
[0011] FIG. 1 is a block diagram illustrating an example coordinate-based point cloud compression (G-PCC) encoder. [Diagram 3]
[0012] FIG. 2 is a block diagram illustrating an exemplary G-PCC decoder. [Figure 4]
[0013] 4 is a flow chart illustrating an exemplary motion estimation algorithm. [Diagram 5]
[0014] FIG. 1 is a conceptual diagram illustrating an example of dividing a Largest Prediction Unit (LPU) into a Prediction Unit (PU) tree. [Figure 6]
[0015] FIG. 13 is a conceptual diagram illustrating the occupation of a parent node and its reference nodes to obtain the number of mispredicted siblings. [Figure 7A]
[0016] FIG. 1 is a conceptual diagram showing examples of a planar copy mode (PCM) mode and a non-PCM mode. [Figure 7B] FIG. 1 is a conceptual diagram showing examples of a planar copy mode (PCM) mode and a non-PCM mode. [Figure 8]
[0017] 1 is a flowchart illustrating an example operation of a G-PCC encoder in accordance with techniques of this disclosure. [Figure 9]
[0018] 11 is a flowchart illustrating an example operation of a G-PCC decoder in accordance with techniques of this disclosure. [Figure 10]
[0019] FIG. 1 is a conceptual diagram illustrating an example distance measurement system that may be used with one or more techniques of the present disclosure. [Figure 11]
[0020] FIG. 1 is a conceptual diagram illustrating an example vehicle-based scenario in which one or more techniques of the present disclosure may be used. [Figure 12]
[0021] FIG. 1 is a conceptual diagram illustrating an example extended reality system in which one or more techniques of this disclosure may be used. [Figure 13]
[0022] FIG. 1 is a conceptual diagram illustrating an example mobile device system in which one or more techniques of the present disclosure may be used. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0011]
[0023] A point cloud is a collection of points in three-dimensional space. In general, point cloud data includes any data for processing a point cloud, such as data representing a point cloud. The points may correspond to points on an object in three-dimensional space. Thus, point cloud data may be used to represent the physical content of a three-dimensional space. Point cloud data may have utility in a wide variety of situations. For example, point cloud data may be used to represent the location of an object on a road in the context of an autonomous vehicle. In another example, point cloud data may be used to place virtual objects in an augmented reality (AR) or mixed reality (MR) application in the context of representing the physical content of an environment. Point cloud compression is the process of encoding and decoding point cloud data. Encoding point cloud data may reduce the amount of data required to store and transmit point cloud data. Coordinate-based point cloud compression (G-PCC) is a technique for reducing the amount of data required to encode and store point cloud data.
[0012]
[0024] As part of encoding the point cloud data, the G-PCC encoder may generate an octree. Each node of the octree corresponds to a cuboid space. A cuboid is a hexahedron with six sides. Each face of the cuboid is a quadrilateral. For ease of explanation, this disclosure may, in some circumstances, interchangeably refer to a node and a cuboid space corresponding to the node. A node of the octree may have zero or eight child nodes. In other examples, a node may be divided into child nodes according to other tree structures. For example, a node of the octree may be divided into four or two child nodes. Thus, an octree may include a quadtree or binary tree division. A child node of a parent corresponds to a cuboid within a cuboid corresponding to a parent node. The location of an individual point of the point cloud may be signaled for a node corresponding to a cuboid that contains a point. If the node does not contain any point of the point cloud, the node is said to be unoccupied. If the node is unoccupied, there may be no need to signal additional data about the node. Conversely, a node is said to be occupied if it contains one or more points of the point cloud.
[0013]
[0025] In addition, the same point cloud data may be divided into equal-sized maximum prediction units (LPUs). Thus, each of the LPUs may be a strict cube. The G-PCC encoder may perform inter prediction to predict points of the LPU based on points in a reference frame. The reference frame includes previously reconstructed point cloud data. The G-PCC encoder may encode points within the LPU based on predicted points for the LPU. Similarly, the G-PCC decoder may perform inter prediction to predict points of the LPU based on points in the reference frame. The G-PCC decoder may reconstruct points within the LPU based on predicted points for the LPU.
[0014]
[0026] There are some drawbacks associated with requiring each of the LPUs to be a strict cube. For example, the boundaries of the LPUs may not align with the boundaries of the octree cuboid. In another example, the boundaries of the LPUs may not align squarely with the boundaries of the point cloud data. This misalignment of boundaries can reduce coding efficiency and lead to more data being stored or transmitted to represent the point cloud data.
[0015]
[0027] According to the techniques of this disclosure, at least two of the LPUs may have different sizes along different directions, so that the sizes of the LPUs align with the quadtree-binary tree partitioning of the point cloud data determined by octree geometry coding, which may increase coding efficiency and result in less data being stored or transmitted to represent the point cloud data.
[0016]
[0028] 1 is a block diagram illustrating an example encoding and decoding system 100 that may implement the techniques of this disclosure. The techniques of this disclosure generally relate to coding (encoding and / or decoding) point cloud data, i.e., supporting point cloud compression. In general, point cloud data includes any data for processing a point cloud. Coding may be useful for compressing and / or decompressing the point cloud data.
[0017]
[0029] As shown in Fig. 1, the system 100 includes a source device 102 and a destination device 116. The source device 102 provides encoded point cloud data that is decoded by the destination device 116. Specifically, in the example of Fig. 1, the source device 102 provides the point cloud data to the destination device 116 via a computer-readable medium 110. The source device 102 and the destination device 116 may include any of a wide range of devices, including desktop computers, notebook (i.e., laptop) computers, tablet computers, set-top boxes, telephone handsets such as smartphones, televisions, cameras, display devices, digital media players, video gaming consoles, video streaming devices, terrestrial or marine vehicles, spacecraft, aircraft, robots, LIDAR devices, satellites, and the like. In some cases, the source device 102 and the destination device 116 may be capable of wireless communication.
[0018]
[0030] In the example of FIG. 1, the source device 102 includes a data source 104, a memory 106, a G-PCC encoder 200, and an output interface 108. The destination device 116 includes an input interface 122, a G-PCC decoder 300, a memory 120, and a data consumer 118. According to this disclosure, the G-PCC encoder 200 of the source device 102 and the G-PCC decoder 300 of the destination device 116 may be configured to apply techniques of this disclosure related to inter-coding using local adaptive prediction of the GPCC standard. Thus, the source device 102 represents an example of an encoding device, and the destination device 116 represents an example of a decoding device. In other examples, the source device 102 and the destination device 116 may include other components or arrangements. For example, the source device 102 may receive data (e.g., point cloud data) from an internal or external source. Similarly, the destination device 116 may interface with an external data consumer rather than including the data consumer within the same device.
[0019]
[0031] The system 100 as shown in FIG. 1 is only an example. In general, other digital encoding and / or decoding devices may implement the techniques of this disclosure related to inter-coding using local adaptive prediction of the GPCC standard. The source device 102 and the destination device 116 are only examples of devices in which the source device 102 generates coded data for transmission to the destination device 116. This disclosure refers to devices that perform coding (encoding and / or decoding) of data as "coding" devices. Thus, the G-PCC encoder 200 and the G-PCC decoder 300 represent examples of coding devices, specifically, encoders and decoders, respectively. In some examples, the source device 102 and the destination device 116 may operate substantially symmetrically such that each of the source device 102 and the destination device 116 includes encoding and decoding components. Thus, the system 100 may support one-way or two-way transmission between the source device 102 and the destination device 116, for example, streaming, playback, broadcast, telephony, navigation, and other applications.
[0020]
[0032] In general, the data source 104 may represent a source of data (i.e., raw, unencoded point cloud data) and provide a sequential series of "frames" of data to the G-PCC encoder 200, which encodes the data of the frames. The data source 104 of the source device 102 may include a point cloud capture device, such as any of a variety of cameras or sensors, e.g., a 3D scanner or a light detection and ranging (LIDAR) device, one or more video cameras, an archive containing previously captured data, and / or a data feed interface for receiving data from a data content provider. Alternatively or in addition, the point cloud data may be computer-generated from scanner data, camera data, sensor data, or other data. For example, the data source 104 may generate computer graphics-based data as source data, or may generate a combination of live data, archive data, and computer-generated data. In each case, the G-PCC encoder 200 encodes the captured data, pre-captured data, or computer-generated data. The G-PCC encoder 200 may reorder the frames from reception order (sometimes called "display order") into a coding order for coding. The G-PCC encoder 200 may generate one or more bitstreams including the encoded data. The source device 102 may then output the encoded data onto a computer-readable medium 110 via an output interface 108 for receipt and / or retrieval by, for example, an input interface 122 of a destination device 116.
[0021]
[0033] The memory 106 of the source device 102 and the memory 120 of the destination device 116 may represent general purpose memories. In some examples, the memory 106 and the memory 120 may store raw data, e.g., raw data from the data source 104 and raw decoded data from the G-PCC decoder 300. Additionally or alternatively, the memory 106 and the memory 120 may store software instructions executable by, e.g., the G-PCC encoder 200 and the G-PCC decoder 300, respectively. Although the memory 106 and the memory 120 are shown in this example separately from the G-PCC encoder 200 and the G-PCC decoder 300, it should be understood that the G-PCC encoder 200 and the G-PCC decoder 300 may also include internal memories for functionally similar or equivalent purposes. Additionally, the memory 106 and the memory 120 may store encoded data, e.g., output from the G-PCC encoder 200 and input to the G-PCC decoder 300. In some examples, portions of memory 106 and memory 120 may be allocated as one or more buffers, e.g., for storing raw, decoded, and / or encoded data. For example, memory 106 and memory 120 may store data representing a point cloud.
[0022]
[0034] The computer-readable medium 110 may represent any type of medium or device capable of transporting encoded data from the source device 102 to the destination device 116. In one example, the computer-readable medium 110 represents a communication medium that allows the source device 102 to transmit encoded data directly to the destination device 116 in real time, for example, via a radio frequency network or a computer-based network. The output interface 108 may modulate and the input interface 122 may demodulate a received transmission signal including the encoded data according to a communication standard such as a wireless communication protocol. The communication medium may comprise any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission paths. The communication medium may form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. The communication medium may include routers, switches, base stations, or any other equipment that may be useful to facilitate communication from the source device 102 to the destination device 116.
[0023]
[0035] In some examples, source device 102 may output the encoded data from output interface 108 to storage device 112. Similarly, destination device 116 may access the encoded data from storage device 112 via input interface 122. Storage device 112 may include any of a variety of distributed or locally accessed data storage media, such as a hard drive, Blu-ray disc, DVD, CD-ROM, flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded data.
[0024]
[0036] In some examples, the source device 102 may output the encoded data to a file server 114 or another intermediate storage device that may store the encoded data generated by the source device 102. The destination device 116 may access the stored data from the file server 114 via streaming or download. The file server 114 may be any type of server device capable of storing the encoded data and transmitting the encoded data to the destination device 116. The file server 114 may represent a web server (e.g., for a website), a file transfer protocol (FTP) server, a content delivery network device, or a network attached storage (NAS) device. The destination device 116 may access the encoded data from the file server 114 through any standard data connection, including an Internet connection. This may include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., a digital subscriber line (DSL), a cable modem, etc.), or a combination of both suitable for accessing the encoded data stored in the file server 114. The file server 114 and the input interface 122 may be configured to operate according to a streaming transmission protocol, a download transmission protocol, or a combination thereof.
[0025]
[0037] The output interface 108 and the input interface 122 may represent a wireless transmitter / receiver, a modem, a wired networking component (e.g., an Ethernet card), a wireless communication component operating according to any of the various IEEE 802.11 standards, or other physical components. In examples where the output interface 108 and the input interface 122 comprise wireless components, the output interface 108 and the input interface 122 may be configured to transfer data, such as encoded data, according to a cellular communication standard, such as 4G, 4G-LTE (Long-Term Evolution), LTE Advanced, 5G, etc. In some examples where the output interface 108 comprises a wireless transmitter, the output interface 108 and the input interface 122 may be configured to transfer data, such as encoded data, according to other wireless standards, such as the IEEE 802.11 specification, the IEEE 802.15 specification (e.g., ZigBee™), the Bluetooth™ standard, etc. In some examples, the source device 102 and / or the destination device 116 may include respective system-on-a-chip (SoC) devices. For example, the source device 102 may include a SoC device for implementing functionality attributable to the G-PCC encoder 200 and / or the output interface 108, and the destination device 116 may include a SoC device for implementing functionality attributable to the G-PCC decoder 300 and / or the input interface 122.
[0026]
[0038] The techniques of this disclosure may be applied to encoding and decoding in support of any of a variety of applications, such as communication between autonomous vehicles, communication between scanners, cameras, sensors and processing devices such as local or remote servers, geographic mapping, or other applications.
[0027]
[0039] The input interface 122 of the destination device 116 receives the encoded bitstream from the computer-readable medium 110 (e.g., a communication medium, a storage device 112, a file server 114, etc.). The encoded bitstream may include signaling information defined by the G-PCC encoder 200, such as syntax elements having values that describe characteristics and / or processing of the coded units (e.g., slices, pictures, groups of pictures, sequences, etc.), that are also used by the G-PCC decoder 300. The data consumer 118 uses the decoded data. For example, the data consumer 118 may use the decoded data to determine the location of a physical object. In some examples, the data consumer 118 may include a display for displaying an image based on the point cloud.
[0028]
[0040] The G-PCC encoder 200 and the G-PCC decoder 300 may each be implemented as any of a variety of suitable encoder and / or decoder circuit configurations, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. When the techniques are implemented partially in software, a device may store software instructions on a suitable non-transitory computer-readable medium and execute the instructions in hardware using one or more processors to perform the techniques of this disclosure. Each of the G-PCC encoder 200 and the G-PCC decoder 300 may be included in one or more encoders or decoders, any of which may be integrated as part of a composite encoder / decoder (codec) in the respective device. A device including G-PCC encoder 200 and / or G-PCC decoder 300 may comprise one or more integrated circuits, microprocessors, and / or other types of devices.
[0029]
[0041] The G-PCC encoder 200 and the G-PCC decoder 300 may operate according to a coding standard, such as the video point cloud compression (V-PCC) standard or the coordinate-based point cloud compression (G-PCC) standard. This disclosure may generally refer to coding (e.g., encoding and decoding) of pictures, which includes the process of encoding or decoding data. An encoded bitstream generally includes a series of values of syntax elements that represent coding decisions (e.g., coding modes).
[0030]
[0042] This disclosure may generally refer to "signaling" some information, such as a syntax element. The term "signaling" may generally refer to communication of values of syntax elements and / or other data used to decode encoded data. That is, the G-PCC encoder 200 may signal values of syntax elements in a bitstream. In general, signaling refers to generating values in the bitstream. As discussed above, the source device 102 may transport the bitstream to the destination device 116 in substantially real-time or non-real-time, such as occurs when storing syntax elements in the storage device 112 for later retrieval by the destination device 116.
[0031]
[0043] Point cloud compression activities are categorized into two different approaches. The first approach is "Video Point Cloud Compression" (V-PCC), which segments a 3D object and projects the segments into multiple 2D planes (represented as "patches" in a 2D frame), which are further coded by a legacy 2D video codec, such as the High Efficiency Video Coding (HEVC) (ITU-T H.265) codec. The second approach is "Coordinate-Based Point Cloud Compression" (G-PCC), which directly compresses the 3D coordinates, i.e., the location of a set of points in 3D space, and the associated attribute values (for each point associated with the 3D coordinates). G-PCC addresses the compression of point clouds in both category 1 (static point clouds) and category 3 (dynamically acquired point clouds).
[0032]
[0044] A point cloud includes a set of points in 3D space and may have attributes associated with the points. The attributes may be color information such as R, G, B, or Y, Cb, Cr, or reflectance information, or other attributes. Point clouds may be captured by various cameras or sensors, such as LIDAR sensors and 3D scanners, or may be computer generated. Point cloud data is used in applications including, but not limited to, architecture (modeling), graphics (3D models for visualization and animation), automotive industry (LIDAR sensors used to assist navigation), and other applications including mobile phones, tablet computers, or other mobile computing devices.
[0033]
[0045] The 3D space occupied by the point cloud data may be enclosed by a virtual bounding box. The positions of points in the bounding box may be represented with a certain precision, and thus the positions of one or more points may be quantized based on that precision. At the smallest level, the bounding box is divided into voxels, which are the smallest units of space represented by a unit cube. A voxel in a bounding box may be associated with zero, one, or more than one point. The bounding box may be divided into multiple cubic / rectangular regions, sometimes called tiles. Each tile may be coded into one or more slices. The partitioning of the bounding box into slices and tiles may be based on the number of points in each section, or other considerations (e.g., a particular region may be coded as a tile). The slice regions may be further partitioned using partitioning decisions similar to those in video codecs.
[0034]
[0046] Figure 2 provides an overview of a G-PCC encoder 200. Figure 3 provides an overview of a G-PCC decoder 300. The illustrated modules are logical and do not necessarily correspond one-to-one to the implementation code in the reference implementation of the G-PCC codec, i.e., the TMC13 test model software considered by ISO / IEC MPEG (JTC1 / SC29 / WG11).
[0035]
[0047] In both the G-PCC Encoder 200 and the G-PCC Decoder 300, the location of the points is coded first. The attribute coding depends on the decoded geometry. In Figures 2 and 3, the grey shaded modules are typically options used for Category 1 data. The diagonal shaded modules are typically options used for Category 3 data. All other modules are common between Category 1 and Category 3.
[0036]
[0048] For category 3 data, the compressed geometry is typically represented as an octree from the root to the leaf level of the individual voxels. For category 1 data, the compressed geometry is typically represented by a pruned octree (i.e., an octree from the root to the leaf level of blocks larger than a voxel) plus a model that approximates the surface in each leaf of the pruned octree. In this way, both category 1 and category 3 data share the octree coding mechanism, while category 1 data may additionally approximate the voxels in each leaf with a surface model. The surface model used is a triangulation with 1 to 10 triangles per block, resulting in a triangle soup.
[0037]
[0049] At each node of the octree, the occupancy is signaled (when not guessed) for one or more of its child nodes (up to 8 nodes). A number of neighboring nodes are specified, including (a) nodes that share a face with the current octree node, (b) nodes that share a face, edge, or vertex with the current octree node, etc. Within each neighboring node, the occupancy of the node and / or its child nodes can be used to predict the occupancy of the current node or its children. For points that are sparsely distributed in some nodes of the octree, the codec (e.g., implemented by the G-PCC encoder 200 or the G-PCC decoder 300) also supports a direct coding mode in which the 3D positions of the points are directly coded. A flag can be signaled to indicate that the direct mode is signaled. At the lowest level, the number of points associated with an octree node / leaf node can also be coded.
[0038]
[0050] When geometry is coded, attributes corresponding to the geometry points are coded. When there are multiple attribute points corresponding to one reconstructed / decoded geometry point, attribute values representing the reconstructed points can be derived.
[0039]
[0051] In G-PCC, there are three attribute coding methods: Region Adaptive Hierarchical Transform (RAHT) coding, Interpolation-based Hierarchical Nearest Neighbor Prediction (Prediction Transform), and Interpolation-based Hierarchical Nearest Neighbor Prediction with Update / Lifting Steps (Lifting Transform). RAHT and lifting are typically used for category 1 data, and prediction is typically used for category 3 data. However, either method may be used for any data, and as in the geometry codec in G-PCC, the attribute coding method used to code the point cloud is specified in the bitstream.
[0040]
[0052] The coding of attributes can be done in levels of detail (LOD), with each level of detail being used to obtain a finer representation of the point cloud attributes. Each level of detail can be designated based on a distance metric from neighboring nodes or based on a sampling distance.
[0041]
[0053] In the G-PCC encoder 200, the residual obtained as the output of the attribute coding method is quantized. The residual may be obtained by subtracting the attribute value from a prediction derived based on the neighboring points of the current point and based on the attribute values of previously coded points. The quantized residual may be coded using context-adaptive arithmetic coding.
[0042]
[0054] In the example of FIG. 2, the G-PCC encoder 200 may include a coordinate transformation unit 202, a color transformation unit 204, a voxelization unit 206, an attribute transfer unit 208, an octree analysis unit 210, a surface approximation analysis unit 212, an arithmetic coding unit 214, a geometric reconstruction unit 216, a RAHT unit 218, an LOD generation unit 220, a lifting unit 222, a coefficient quantization unit 224, and an arithmetic coding unit 226.
[0043]
[0055] As shown in the example of FIG. 2, the G-PCC encoder 200 may obtain a set of positions and a set of attributes for points in a point cloud. The G-PCC encoder 200 may obtain the set of positions and the set of attributes for points in a point cloud from a data source 104 (FIG. 1). The positions may include coordinates of the points in the point cloud. The attributes may include information about the points in the point cloud, such as a color associated with the points in the point cloud. The G-PCC encoder 200 may generate a geometry bitstream 203 that includes an encoded representation of the positions of the points in the point cloud. The G-PCC encoder 200 may also generate an attribute bitstream 205 that includes an encoded representation of the set of attributes.
[0044]
[0056] The coordinate transformation unit 202 may apply a transformation to the coordinates of the points to convert the coordinates from an initial domain to a transformation domain. In this disclosure, the transformed coordinates may be referred to as transformed coordinates. The color transformation unit 204 may apply a transformation to convert color information of the attributes to a different domain. For example, the color transformation unit 204 may convert color information from an RGB color space to a YCbCr color space.
[0045]
[0057] Further, in the example of FIG. 2, the voxelization unit 206 may voxelize the transformed coordinates. The voxelization of the transformed coordinates may include quantization and removing some points of the point cloud. In other words, multiple points of the point cloud may be contained within a single "voxel" and then treated as one point in some regions. Further, the octree analysis unit 210 may generate an octree based on the voxelized transformed coordinates. Additionally, in the example of FIG. 2, the surface approximation analysis unit 212 may analyze the points to determine a surface representation of the set of points. The arithmetic coding unit 214 may entropy code syntax elements representing information of the surface determined by the octree and / or surface approximation analysis unit 212. The octree analysis unit 210 and the arithmetic coding unit 214 together may form a geometric coding unit. In some examples, the octree analysis unit 210 may perform inter prediction and / or intra prediction. G-PCC encoder 200 may output these syntax elements in geometry bitstream 203. Geometry bitstream 203 may also include other syntax elements, including syntax elements that are not arithmetically coded.
[0046]
[0058] The geometric reconstruction unit 216 may reconstruct transformation coordinates of points in the point cloud based on the octree, the data indicative of the surface determined by the surface approximation analysis unit 212, and / or other information. The number of transformation coordinates reconstructed by the geometric reconstruction unit 216 may differ from the original number of points in the point cloud due to voxelization and surface approximation. In this disclosure, the resulting points may be referred to as reconstructed points. The attribute transfer unit 208 may transfer attributes of the original points of the point cloud to the reconstructed points of the point cloud.
[0047]
[0059] Further, the RAHT unit 218 may apply RAHT coding to the attributes of the reconstructed points. In some examples, under RAHT, the attributes of a block of 2x2x2 point positions are taken and transformed along one direction to obtain four low-frequency nodes (L) and four high-frequency nodes (H). Then, the four low-frequency nodes (L) are transformed in a second direction to obtain two low-frequency nodes (LL) and two high-frequency nodes (LH). The two low-frequency nodes (LL) are transformed in a third direction to obtain one low-frequency node (LLL) and one high-frequency node (LLH). The low-frequency node LLL corresponds to the DC coefficient, and the high-frequency nodes H, LH, and LLH correspond to the AC coefficients. The transformation in each direction may be a 1-D transformation with two coefficient weights. The low-frequency coefficients may be taken as the coefficients of a 2x2x2 block for the next higher level of the RAHT transformation, and the AC coefficients are coded without modification, and such transformation continues up to the top-most root node. The tree traversal for encoding is a top-to-bottom traversal used to calculate the weights used for the coefficients, and the transform order is bottom-to-top. The coefficients can then be quantized and coded.
[0048]
[0060] Alternatively or additionally, the LOD generation unit 220 and the lifting unit 222 may apply LOD processing and lifting, respectively, to the attributes of the reconstructed points. LOD generation is used to split the attributes into different refinement levels. Each refinement level provides refinement to the attributes of the point cloud. The first refinement level provides a coarse approximation and includes a small number of points, subsequent refinement levels typically include more points, and so on. The refinement levels may be constructed using a distance-based metric or may use one or more other classification criteria (e.g., subsampling from a particular rank). Thus, a refinement level may include all reconstructed points. Each level of detail is generated by lifting the union of all points to a particular refinement level, e.g., LOD1 is obtained based on refinement levels RL1, LOD2 is obtained based on RL1 and RL2, and LODN is obtained by the union of RL1, RL2, ...RLN. In some cases, LOD generation is followed by a prediction scheme (such as a predictive transform) in which attributes associated with each point in the LOD are predicted from a weighted average of previous points, and the residual is quantized and entropy coded. Lifting schemes build on the predictive transform mechanism, where an update operator is used to update the coefficients and adaptive quantization of the coefficients is performed.
[0049]
[0061] The RAHT unit 218 and the lifting unit 222 may generate coefficients based on the attributes. The coefficient quantization unit 224 may quantize the coefficients generated by the RAHT unit 218 or the lifting unit 222. The arithmetic coding unit 226 may apply arithmetic coding to syntax elements representing the quantized coefficients. The G-PCC encoder 200 may output these syntax elements in the attribute bitstream 205. The attribute bitstream 205 may also include other syntax elements, including non-arithmetically coded syntax elements.
[0050]
[0062] In the example of FIG. 3, the G-PCC decoder 300 may include a geometry arithmetic decoding unit 302, an attribute arithmetic decoding unit 304, an octree synthesis unit 306, an inverse quantization unit 308, a surface approximation synthesis unit 310, a geometric reconstruction unit 312, a RAHT unit 314, a LoD generation unit 316, an inverse lifting unit 318, an inverse coordinate transformation unit 320, and an inverse color transformation unit 322.
[0051]
[0063] The G-PCC decoder 300 may obtain a geometry bitstream 203 and an attribute bitstream 205. A geometry arithmetic decoding unit 302 of the decoder 300 may apply arithmetic decoding (e.g., Context-Adaptive Binary Arithmetic Coding (CABAC) or other types of arithmetic decoding) to syntax elements in the geometry bitstream 203. Similarly, an attribute arithmetic decoding unit 304 may apply arithmetic decoding to syntax elements in the attribute bitstream 205.
[0052]
[0064] The octree synthesis unit 306 may synthesize an octree based on syntax elements parsed from the geometry bitstream 203. The octree synthesis unit 306, together with the geometry arithmetic decoding unit 302, may form the geometry decoding unit of the G-PCC decoder 300. Starting from the root node of the octree, the occupancy of each of the eight child nodes at each octree level is signaled in the bitstream. When the signaling indicates that a child node at a particular octree level is occupied, the occupancy of the children of this child node is signaled. The signaling of the nodes at each octree level is done before proceeding to a subsequent octree level. At the final level of the octree, each node corresponds to a voxel location, and when a leaf node is occupied, one or more points may be specified to be occupied at the voxel location. In some instances, some branches of the octree may terminate before the final level due to quantization. In such cases, the leaf node is considered an occupied node that does not have any child nodes. In some examples, the octree synthesis unit 306 may perform inter prediction and / or intra prediction. In instances where surface approximations are used in the geometry bitstream 203, the surface approximation synthesis unit 310 may determine a surface model based on syntax elements parsed from the geometry bitstream 203 and based on the octree. In some examples, inter prediction may be part of the process performed by the surface approximation synthesis unit 310 to determine the surface model.
[0053]
[0065] Further, the geometric reconstruction unit 312 may perform a reconstruction to determine coordinates of points in the point cloud. For each position in a leaf node of the octree, the geometric reconstruction unit 312 may reconstruct the node position by using a binary representation of the leaf node in the octree. At each respective leaf node, the number of points in the respective leaf node is signaled, which indicates the number of overlapping points at the same voxel position. When geometric quantization is used, the point positions are scaled to determine a reconstructed point position value.
[0054]
[0066] The inverse coordinate transformation unit 320 may apply an inverse transform to the reconstructed coordinates (positions) of the points in the point cloud to reverse them from the transformed domain back to the original domain. The positions of the points in the point cloud may be in the floating-point domain, whereas the point positions in the G-PCC codec are coded in the integer domain. The inverse transform may be used to transform the positions back to the original domain.
[0055]
[0067] 3, the inverse quantization unit 308 may inverse quantize the attribute value. The attribute value may be based on a syntax element obtained from the attribute bitstream 205 (e.g., including a syntax element decoded by the attribute arithmetic decoding unit 304).
[0056]
[0068] Depending on how the attribute values are coded, the RAHT unit 314 may perform RAHT coding to determine color values for the points of the point cloud based on the dequantized attribute values. RAHT decoding is performed from the top to the bottom of the tree. At each level, a configuration value is derived using the low-frequency and high-frequency coefficients derived from the dequantization process. At the leaf nodes, the derived values correspond to the attribute values for the coefficients. The weight derivation process for a point is similar to the process used in the G-PCC encoder 200. Instead, the LOD generation unit 316 and the inverse lifting unit 318 may use a level-of-detail based technique to determine color values for the points of the point cloud. The LOD generation unit 316 decodes each LOD to progressively more finely represent the attributes of the point. Using a predictive transform, the LOD generation unit 316 derives a predicted value for the point from a weighted sum of points in the previous LOD or previously reconstructed in the same LOD. The LOD generation unit 316 may add the predicted value to the residual (obtained after inverse quantization) to obtain a reconstructed value of the attribute. When a lifting scheme is used, the LOD generation unit 316 may also include an update operator to update the coefficients used to derive the attribute value. In this case, the LOD generation unit 316 may also apply inverse adaptive quantization.
[0057]
[0069] 3, the inverse color transformation unit 322 may apply an inverse color transformation to the color values. The inverse color transformation may be the inverse of the color transformation applied by the color transformation unit 204 of the encoder 200. For example, the color transformation unit 204 may transform the color information from the RGB color space to the YCbCr color space. In response, the inverse color transformation unit 322 may transform the color information from the YCbCr color space to the RGB color space.
[0058]
[0070] The various units in Figures 2 and 3 are shown to aid in understanding the operations performed by the encoder 200 and the decoder 300. The units may be implemented as fixed function circuits, programmable circuits, or a combination thereof. Fixed function circuits refer to circuits that provide specific functionality and are predefined in the operations they can perform. Programmable circuits refer to circuits that can be programmed to perform various tasks and provide flexible functionality in the operations they can perform. For example, a programmable circuit may execute software or firmware that causes the programmable circuit to operate in a manner defined by the software or firmware instructions. Although a fixed function circuit may execute software instructions (e.g., to receive parameters or output parameters), the type of operation the fixed function circuit performs is generally immutable. In some examples, one or more of the units may be separate circuit blocks (fixed function or programmable) and in some examples, one or more of the units may be an integrated circuit.
[0059]
[0071] The angle coding mode improves the coding efficiency of the planar mode by using the sensor characteristics of a typical LiDAR sensor. The angle coding mode is optionally used in conjunction with the planar mode to improve the coding of vertical (z) plane position syntax elements by utilizing knowledge of the position and angle of the sensing laser beam in a typical LiDAR sensor. Additionally, the angle coding mode may optionally be used to improve the coding of vertical z position bits in inferred direct coding mode (IDCM).
[0060]
[0072] The azimuth coding mode is similar to the angle mode and extends the angle mode to the coding of the (x) and (y) planar position syntax elements of the planar mode, improving the coding of the x or y position bits in the IDCM. "Angle mode" may also refer to azimuth mode in the following sections.
[0061]
[0073] Inter prediction is a technique that exploits redundancy between different frames of point cloud data. Some redundancy between frames of point cloud data is a result of translational or rotational motion of objects relative to the sensor that generates the point cloud data. Motion parameters can be used to characterize such motion. In some examples, the motion parameters involved in G-PCC are of two types: global motion parameters and local nodal motion vectors. Global motion parameters are defined as a rotation matrix and a translation vector, and are applied to all points in the predicted (reference) frame. Local nodal motion vectors of a node of an octree are motion vectors that are applied only to points within the node in the predicted (reference) frame. Details of an exemplary motion estimation algorithm are described below.
[0062]
[0074] 4 is a flow chart illustrating an exemplary motion estimation algorithm. Given an input predicted (reference) frame 400 and a current frame 402, the G-PCC encoder 200 may first estimate 404 a global motion at a global scale. The predicted frame 400 and the current frame 402 each represent a point cloud. In other words, the predicted frame 400 and the current frame 402 each comprise separate point cloud data. The current frame 402 may represent a point cloud that the G-PCC encoder 200 is currently encoding. The predicted frame 400 may represent a point cloud that the G-PCC encoder 200 previously reconstructed. The point cloud represented by the predicted frame 400 may correspond to an earlier time than the point cloud represented by the current frame 402.
[0063]
[0075] After estimating the global motion, the G-PCC encoder 200 may apply the estimated global motion to the predicted frame 400 (406). In other words, the G-PCC encoder 200 may perform motion compensation on the predicted frame 400 based on the estimated global motion. For example, the G-PCC encoder 200 may rotate or translate points in the predicted frame 400 based on the estimated block motion. The G-PCC encoder 200 may estimate the global motion based on a comparison between the predicted frame 400 and the current frame 402.
[0064]
[0076] After applying global motion to the prediction, the G-PCC encoder 200 may estimate local motion at a finer scale, for example at the node level of the octree. In other words, the G-PCC encoder 200 may perform local node motion estimation (408). For example, the G-PCC encoder 200 may determine, for an individual node, whether to perform motion compensation for the node based on an original version of the predicted frame 400 or a version of the predicted frame 400 to which global motion compensation has been applied. In some examples, as part of performing local node motion estimation, the G-PCC encoder 200 may determine a motion vector for the node.
[0065]
[0077] The G-PCC encoder 200 may apply the estimated local node motion to motion compensation (410). Applying the estimated location node motion may generate predicted positions of points within the node. In other words, the G-PCC encoder 200 may perform motion compensation on the node based on the local node motion estimation. The G-PCC encoder 200 may encode points within the node based on predicted positions of the points of the node. For example, the G-PCC encoder 200 may encode data indicating the displacement between the points of the node and the corresponding predicted points of the node. The G-PCC encoder 200 may also encode a motion vector of the node, if one exists.
[0066]
[0078] For the purpose of inter prediction, the point cloud may be divided into largest prediction units (LPUs). The LPUs may be recursively partitioned into sub-prediction units (PUs), thus forming a prediction tree. The "nodes" referred to in reference to FIG. 4 may be LPUs or PUs.
[0067]
[0079] In some examples, the LPU has two coding modes, including a vehicle LPU and a world LPU. In other words, each LPU can be either a vehicle LPU or a world LPU. The terms "vehicle LPU" and "world LPU" arise from a use case in which point cloud data is generated by a sensor on a moving vehicle, such as an autonomous vehicle. Generally, points of the point cloud data representing the vehicle itself do not move relative to the position of the sensor, since the sensor is at a fixed position on the vehicle. Similarly, points representing the road on which the vehicle is traveling tend not to change much from frame to frame (e.g., because roads are generally flat and smooth). On the other hand, points of the point cloud data representing other objects in the world other than the vehicle and the road move relative to the position of the sensor. Accordingly, there may be coding efficiencies associated with performing inter prediction differently for points representing vehicles and points representing objects in the world.
[0068]
[0080] Regardless of whether the LPU is a vehicle LPU or a world LPU, all PUs in an LPU are predicted using the same reference frame. The reference frame used to predict PUs in a vehicle LPU is a previously reconstructed point cloud, and the reference frame used to predict PUs in a world LPU is a compensated point cloud of a previously reconstructed point cloud using global motion. This is because the sensor has a fixed position on the vehicle, and the points in the point cloud data representing the vehicle do not rotate or translate relative to the sensor. Conversely, the sensor may rotate or translate relative to the world. Therefore, it may be more efficient to encode the points representing the vehicle without applying global motion estimation and to encode the points representing the world with global motion estimation.
[0069]
[0081] In some examples, the reference frame used to predict the PUs in the vehicle LPU or world LPU may be a frame generated using thresholds. These thresholds may be used to segment the point cloud into roads and objects. If the vertical coordinate of a point is between two thresholds, the point is classified as a road. The G-PCC encoder 200 may partition the point cloud into one or more vehicle LPUs and one or more world LPUs based on the amount of motion in different areas of the point cloud data. If the estimated motion of an LPU exceeds a threshold, the G-PCC encoder 200 may classify the LPU as a world LPU. Otherwise, the G-PCC encoder 200 may classify the LPU as a vehicle LPU. In this example, to generate the reference frame, global motion is applied to the object points and zero motion is applied to the road points (e.g., points representing the road on which the vehicle is traveling).
[0070]
[0082] FIG. 5 is a conceptual diagram illustrating an example of partitioning a maximum prediction unit (LPU) into a prediction unit (PU) tree. In other words, the partitioning process of an LPU into a PU tree is illustrated in FIG. 5. In the example of FIG. 5, the G-PCC encoder 200 has previously reconstructed a frame 500 representing a point cloud (shown in FIG. 5 as a "previously reconstructed group"). The G-PCC encoder 200 may apply global motion compensation to the frame 500 to generate a world reference frame 502. The vehicle reference frame 504 may simply be a copy of the frame 500.
[0071]
[0083] Further, in the example of FIG. 5, the G-PCC encoder 200 is encoding a current frame 506 (denoted as “current group” in FIG. 5). The current frame 506 includes a current LPU 508. The G-PCC encoder 200 may search a window 510 (e.g., active window Wv) surrounding a co-located LPU 512 in the vehicle reference frame 504 for points corresponding to points in the current LPU 508. In addition, the G-PCC encoder 200 may search a window 514 (e.g., active window Ww) surrounding a co-located LPU 516 in the world reference frame 502 for points corresponding to points in the current LPU 508. The co-located LPU 512 and the co-located LPU 516 are co-located with the current LPU 508. Thus, the active window is an extended area of the co-located reference LPU in the reference frame. The G-PCC encoder 200 may determine whether the points in window 510 or window 514 are a better match to the points in the current LPU 508. If the points in window 510 are a better match, the G-PCC encoder 200 may classify the current LPU 508 as a vehicle LPU. If the points in window 510 are a better match, the G-PCC encoder 200 may classify the current LPU 508 as a world LPU. All points in the active window are used to determine the LPU mode using a distortion comparison.
[0072]
[0084] After classifying the current LPU 508, the G-PCC encoder 200 may recursively partition the current LPU 508 into PUs (i.e., sub-PUs) (518). The partitioning of the LPU into sub-PUs starts from depth 0 or LPU level and proceeds to the maximum depth (or until the PU size reaches the minimum allowed size MinPUsize). The partitioning at each depth is determined using a rate-distortion optimization process. The rate-distortion cost without partitioning is calculated first. A motion search is performed to find the best motion with the minimum distortion between the current PU and the corresponding PU in the reference frame. The rate-distortion cost without partitioning is derived using the motion distortion and signaling overhead. Once the rate-distortion cost without partitioning is obtained, the G-PCC encoder 200 may test partitioning the PU into eight child nodes at the next depth. The rate-distortion cost with partitioning is the sum of the costs of the eight child nodes. To determine whether the PU is partitioned, the cost with partitioning and the cost without partitioning are compared. The following information may be signaled to the G-PCC decoder 300 for the current LPU 508. First, the LPU type is encoded. The LPU type may indicate whether the LPU is a vehicle LPU or a world LPU. Next, the split of the PUs of the current LPU 508 at each depth of the PU tree is signaled. If the PU is not split, the G-PCC encoder 200 may signal the associated motion (i.e., the motion associated with the PU).
[0073]
[0085] In some examples, inter prediction is used in occupation coding, including planar mode coding with angular mode disabled. In other words, inter prediction can be used in the process of coding whether a node in a geometric octree is occupied by one or more points. In some examples, the reference node is a collocated node in a reference frame that is derived by compensating a predicted frame using global motion. Once the reference node is obtained, it is divided into eight cubic child nodes of the same size.
[0074]
[0086] The points in each child node of the reference node are counted to form inter prediction occupancy (predOccupancy) and prediction occupancy strong (predOccupancyStrong). If there is at least one point in the child node, the corresponding bit (out of 8 bits) in predOccupancy is set equal to 1; otherwise, the corresponding bit is set equal to 0. On the other hand, if the number of points in the child node is greater than 2, the corresponding bit in predOccupancyStrong is set equal to 1; otherwise, the bit is set equal to 0.
[0075]
[0087] Next, the quality of inter prediction is evaluated by a parameter called "occupancyIsPredictable". The value of occupancyIsPredictable of a node is derived by the number of mispredicted siblings. As shown in Figure 6, the number of mispredicted siblings (e.g., numSiblingsMispredicted) is calculated by comparing the occupancy of the parent node with the occupancy of the parent reference node. If the predOccupancy of the current node is 0 or the number of mispredicted siblings is greater than 5, occupancyIsPredictable is set equal to 0. Otherwise, occupancyIsPredictable is set equal to 1.
[0076]
[0088] Figure 6 is a conceptual diagram illustrating the occupancy of a parent node 600 and a reference node 602 of the parent node 600 to obtain the number of mispredicted siblings. In the example of Figure 6, the G-PCC encoder 200 or the G-PCC decoder 300 may compare (604) the occupancy of the parent node 600 and the reference node 602 to obtain the number of mispredicted / mispredicted siblings (e.g., numSiblingsMispredicted). If the occupancy bits of a child node in the parent node 600 and the occupancy bits of the corresponding node in the reference node 602 are different, the child node is considered to be mispredicted.
[0077]
[0089] The G-PCC encoder 200 or the G-PCC decoder 300 may update occupancyIsPredictable to update predOccupancy, predOccupancyStrong, plane copy mode eligibility, and IDCM eligibility. If occupancyIsPredictable is equal to 0, predOccupancy, predOccupancyStrong, and plane copy mode eligibility are set equal to 0. If occupancyIsPredictable is 1, IDCM is disabled for this node in the current InterEM.
[0078]
[0090] If a node is coded using plane copy mode (PCM), the G-PCC encoder 200 does not signal the plane information of this node to the G-PCC decoder 300. Rather, the G-PCC encoder 200 and the G-PCC decoder copy the plane mode and plane position in three directions of the PCM coded node from the plane information of the reference node 602 generated from the predOccupancy.
[0079]
[0091] Examples of PCM and non-PCM nodes are shown in Fig. 7A and Fig. 7B, respectively. In other words, Fig. 7A and Fig. 7B are conceptual diagrams showing examples of PCM and non-PCM modes. In the example of Fig. 7A, the current node 700 only has points in the top block, and the reference node 702 also only has points in the top block. Thus, the current node 700 and the reference node 702 may share the same plane mode and plane position in three dimensions. In Fig. 7B, the current node 704 only has points in the top block. However, in Fig. 7B, the reference node 706 for the current node 704 has points in the top block and the bottom block. Thus, in Fig. 7B, the current node 704 and the reference node 706 do not share the plane mode and plane position in three dimensions. For non-PCM coded nodes, the plane information of the reference node is used to provide more flexibility in the context selection of the plane mode and plane position coding.
[0080]
[0092] In some examples, inter prediction is used to improve the occupancy coding. For example, inter prediction can be used in context selection to code the occupancy bits of the current node. This is shown below. GeometryOctreeEncoder::encodeOccupancyNeighNZ() and GeometryOctreeEncoder::encodeOccupancyNeighZ() work as follows: / / This part is for encoding the occupancy bits of the child nodes of the current node. int ctxIdxMapIdx=4 * idxAdj; / / This part is obtained using the occupancy of the adjacent nodes. ctxIdxMapIdx=4 * idxAdj+bitIsPredicted+bitPrediction / / Intra prediction + !!mappedPred+bitPred+bitPredStrong; / / Inter prediction ... auto& ctxIdxMap=_ctxIdxMaps[ctxIdxMapIdx]; ctxIdx=ctxIdxMap.evolve(bit,&ctxIdxMap[i][numOccupiedAcc]); _arithmeticEncoder->encode(bit,_ctxOccupancy[ctxIdx]);
[0081]
[0093] In the above code, idxAdj is determined using the occupancy of the neighboring nodes. bitIsPredicted and bitPrediction are the occupancy bits derived using intra prediction. For inter frames, these parameters are set equal to 0. !!mappedPred indicates whether the predicted occupancy of the inter reference block is 0 or not. ctxIdxMaps contains the context map address. bitPred and bitPredStrong are the corresponding bits of the child node in predOccupancy and predOccupancyStrong. In the above code, the evolve function calculates the context index used to code the bit value.
[0082]
[0094] A particular implementation of G-PCC may have one or more problems, including problems in partitioning and signaling of PU motion information. For example, in partitioning, an LPU may be a cubic node with the same size of the node in three directions. However, it may not be aligned with the size of the node when the point cloud is partitioned using QT-BT (quadtree, binary tree) partitioning, or when the node is located along the boundary of the point cloud. In signaling of PU motion information, the value of the translational motion of the PU is directly signaled. Note that the motion of the current PU may have a high correlation with the sibling PU or parent PU. Thus, motion predictive coding may have better performance compared to directly signaling the motion value. In addition, in one exemplary implementation of G-PCC, when a PU is not partitioned, the motion is always signaled. In some cases, there is no point in the co-located PU in the reference frame to perform motion compensation. Signaling the motion in these cases is redundant. This disclosure discloses several techniques to solve the above problems.
[0083]
[0095] According to the techniques of this disclosure, an LPU may have different sizes in different directions, defined as {lpuX, lpuY, lpuZ}, where lpuX, lpuY, and lpuZ are positive numbers. For example, an LPU may have a first size in a first direction (e.g., x, y, or z direction), and an LPU may have a second size in a second direction that is different from the first size. In one example, the LPU size may be aligned with the QT-BT partition of the group determined by octree geometric coding.
[0084]
[0096] Thus, the G-PCC encoder 200 may determine that the point cloud data is divided into a plurality of LPUs, at least two of the LPUs having different sizes along different directions. In this disclosure, a point cloud may refer to a complete point cloud or a portion of a point cloud. The G-PCC encoder 200 may perform inter prediction to determine a prediction point for the LPU. The G-PCC encoder 200 may encode points within the LPU based on the prediction point for the LPU. Similarly, the G-PCC decoder 300 may determine that the point cloud data is divided into a plurality of LPUs, at least two of the LPUs having different sizes along different directions. The G-PCC decoder 300 may perform inter prediction to determine a prediction point for the LPU. The G-PCC decoder 300 may reconstruct points within the LPU based on the prediction point for the LPU.
[0085]
[0097] The cloud may be flexibly partitioned into LPUs for inter prediction, for example, the partitioning of the point cloud into LPUs may be controlled by one or more parameters described below.
[0086]
[0098] For each dimension, i.e., x, y, and z, an indication of whether a split is performed along the dimension may be signaled. For example, a flag (e.g., split_lpu_x) may be signaled for the x dimension. In this example, split_lpu_x equal to 1 indicates that the point cloud may be split along the x dimension to generate LPUs. split_lpu_x equal to 0 indicates that the point cloud is not split along the x dimension to generate LPUs. Similar flags (e.g., split_lpu_y and split_lpu_z) may be signaled for the y and z dimensions.
[0087]
[0099] In some examples, for each dimension (i.e., x, y, and z), the LPU size along the dimension may be predetermined or signaled. The LPU size along a dimension may be specified in number of voxels, a logarithm of the number of voxels, or in another manner. For example, a syntax element (e.g., LPU_size_x) may be specified to indicate the LPU size along the x dimension. Similar syntax elements (e.g., LPU_size_y and LPU_size_z) may be signaled for the y and z dimensions. In this example, the syntax elements LPU_size_x, LPU_size_y, and LPU_size_z may have different values.
[0088]
[0100] In some examples, the LPU size along a dimension may be signaled only when splitting along that dimension is specified or indicated as being enabled. For example, LPU_size_x may be signaled to indicate the size of the LPU along the x dimension only when split_lpu_x indicates that splitting along the x dimension may be enabled. Further, in some examples, the logarithm of the LPU size along each dimension may be signaled instead of the LPU size. For example, if the value of the LPU size along the x direction is k, then the signaled value is log2(k).
[0089]
[0101] In some examples, for each dimension (i.e., x, y, and z), the value of the LPU size along the dimension may also be used to indicate (a) whether splitting is enabled along the dimension, and (b) the size of the LPU along the dimension. For example, LPU_size_x equal to 0 may specify that the point cloud is not split along the x dimension to generate LPUs. LPU_size_x greater than 0 may specify that the point cloud is split along the x dimension to generate LPUs, and that the size of the LPU along the x dimension is LPU_size_x. The same semantics may apply with respect to LPU_size_y and LPU_size_z.
[0090]
[0102] Thus, in such an example, the G-PCC encoder 200 may signal a syntax element for at least one of the x, y, and z directions indicating whether splitting of the point cloud data is enabled in that direction and the size of the LPU along that direction. The G-PCC decoder 300 may obtain a syntax element for at least one of the x, y, and z directions indicating both whether splitting of the point cloud data is enabled in that direction and the size of the LPU along that direction.
[0091]
[0103] In some examples, when splitting is not enabled along a dimension, the G-PCC encoder 200 and the G-PCC decoder 300 may infer that the value of the LPU size along the dimension is equal to the size of the point cloud along the dimension. For example, when the size of the point cloud is PC_size_x, PC_size_y, and PC_size_z along the x, y, and z dimensions, respectively, and split_lpu_x is equal to 0, the G-PCC encoder 200 and the G-PCC decoder 300 may infer that the value of LPU_size_x is equal to PC_size_x. Similar examples may be applied for split_lpu_y, PC_size_y, and LPU_size_y, and for split_lpu_z, PC_size_z, and LPU_size_z. Thus, in such an example, the G-PCC decoder 300 may determine that splitting of the point cloud data is not enabled along at least one of the x, y, or z directions, and based on a determination that splitting of the point cloud data is not enabled along that direction, the G-PCC decoder 300 may determine that the size of the LPU is equal to the size of the point cloud data along that direction.
[0092]
[0104] The division of LPUs into PUs may be determined similarly to the example described above with respect to dividing a point cloud into LPUs. For example, for each dimension (i.e., x, y, and z), an indication of whether the LPUs are to be divided along the dimension may be signaled. For example, a split_lpu_to_pu_x flag may be signaled for the x dimension. split_lpu_to_pu_x equal to 1 indicates that the division may be along the x dimension, and split_lpu_to_pu_x equal to 0 indicates that the division is not to be along the x dimension. Similarly, a split_lpu_to_pu_y flag may be signaled for the y dimension. split_lpu_to_pu_y equal to 1 indicates that the division may be along the y dimension, and split_lpu_to_pu_y equal to 0 indicates that the division is not to be along the y dimension. A split_lpu_to_pu_z flag may be signaled for the z dimension. split_lpu_to_pu_z equal to 1 indicates that the split may be made along the z dimension, and split_lpu_to_pu_z equal to 0 indicates that the split may not be made along the z dimension.
[0093]
[0105] In some examples, for each dimension (i.e., x, y, and z), a minimum PU size along the dimension may be predetermined or signaled. For example, PU_size_x may be specified to indicate the minimum PU size along the x dimension; PU_size_y may be specified to indicate the minimum PU size along the y dimension; and PU_size_z may be specified to indicate the minimum PU size along the z dimension. The terms "direction" and "dimension" are used interchangeably in this disclosure.
[0094]
[0106] In some examples, the G-PCC encoder 200 signals only the minimum PU size along a dimension when LPU-to-PU splitting along that dimension is specified or indicated as being enabled. For example, PU_size_x may be signaled to indicate the minimum size of PUs along the x dimension only when split_lpu_to_pu_x indicates that LPU-to-PU splitting along the x dimension may be enabled. Similar examples may apply for the y and z dimensions. In some examples, for each dimension, the minimum PU size may be specified as a ratio of the LPU size to the minimum PU size. For example, the G-PCC encoder 200 may signal a syntax element having a value equal to LPU_size_x divided by PU_size_x. In some examples, to further reduce signaling costs, the G-PCC encoder 200 may signal a syntax element having a value equal to log2(LPU_size_x / PU_size_x). This signaled value may be defined as the LPU_to_PU_size_ratio_log2 syntax element.
[0095]
[0107] In some examples, for each dimension, the value of the minimum PU size along a dimension may also be used to indicate (a) whether splitting is enabled along the dimension, and (b) the minimum size of a PU along the dimension. For example, PU_size_x equal to 0 may specify that LPU-to-PU splitting is not enabled along the x dimension. PU_size_x greater than 0 may specify that LPU-to-PU splitting is enabled along the x dimension, and the minimum size of a PU along the x dimension is PU_size_x. Alternatively, the same may apply to LPU_to_PU_size_ratio_log2. Similar examples may apply for the y and z dimensions.
[0096]
[0108] When splitting is not enabled along a dimension, the G-PCC encoder 200 and the G-PCC decoder 300 may infer that the value of the minimum PU size along the dimension is equal to the size of the LPU along the dimension. For example, when the sizes of the LPUs are LPU_size_x, LPU_size_y, and LPU_size_z along the x, y, and z dimensions, respectively, and split_lpu_to_pu_x is equal to 0, the minimum value of PU_size_x is inferred to be equal to LPU_size_x. A similar example may be applied for the y and z dimensions.
[0097]
[0109] In some examples, the group may be split vertically. In this example, the LPU sizes in the x and y directions may be set to be equal to the size of the group (or the group's bounding box) in the x and y directions, respectively. In some examples, the group may be split horizontally in the x direction. In this example, the LPU sizes in the y and z directions may be set to be equal to the size of the group (or the group's bounding box) in the y and z directions, respectively. In one example, the group may be split horizontally in the y direction. In this example, the LPU sizes in the x and z directions may be set to be equal to the size of the group (or the group's bounding box) in the x and z directions, respectively.
[0098]
[0110] Thus, in such an example, the G-PCC encoder 200 may signal a first instruction indicating whether the point cloud data is to be split into LPUs in the x-direction, a second signaled instruction indicating whether the point cloud data is to be split into LPUs in the y-direction, and / or a third signaled instruction indicating whether the point cloud data is to be split into LPUs in the z-direction. Similarly, the G-PCC decoder 300 may determine based on the first signaled instruction that the point cloud data is to be split into LPUs in the x-direction. The G-PCC decoder 300 may determine based on the second signaled instruction that the point cloud data is to be split into LPUs in the y-direction. The G-PCC decoder 300 may determine based on the third signaled instruction that the point cloud data is to be split into LPUs in the z-direction.
[0099]
[0111] In some examples, the splitting of groups in only one direction for inter prediction may be controlled by a flag that may be signaled at the header level (e.g., at the sequence parameter set (SPS) or geometry brick header (GBH) level). For example, the G-PCC encoder 200 may signal a syntax element at the header level indicating that the point cloud data is split into LPUs in only one of multiple directions. The G-PCC decoder 300 may determine that the point cloud data is split into LPUs in only one of multiple directions based on the syntax element signaled at the header level.
[0100]
[0112] In some examples, the PU movement may be set equal to 0. In such examples, the LPU is not split to a lower depth and the PU movement at the LPU level may not be signaled. This setting may be controlled by a flag that may be signaled at the header level (e.g., SPS or gbh level).
[0101]
[0113] In some examples, when a PU does not have a reference point, the motion of the PU may not be signaled. In other words, if there is no point in the area of the reference frame that is collocated with the PU, no motion information (e.g., motion vectors, etc.) is signaled for the PU.
[0102]
[0114] In some examples, the motion of the PU may be signaled using a prediction method. For example, the G-PCC encoder 200 and the G-PCC decoder 300 may determine a base motion of the PU. The G-PCC encoder 200 may signal a difference between the base motion and the actual motion of the PU. For example, the G-PCC encoder 200 may signal a motion vector difference (MVD) indicating the difference between the base motion vector and a motion vector indicating the actual motion of the PU. The G-PCC decoder 300 may reconstruct the actual motion vector of the PU based on the base motion vector and the MVD (e.g., by adding the corresponding components of the base motion vector and the MVD). In some examples, the base motion may be a translation part of the global motion. In some examples, a rotation matrix of the global motion may be applied in the motion compensation of the PU.
[0103]
[0115] In some examples, a group of PUs (e.g., PUs at the same depth) may share the same base motion. The base motion may be signaled in the bitstream or may be derived using neighboring PUs of the PU group. Thus, in some such examples, the G-PCC encoder 200 may signal different MVDs for different PUs in the PU group. For each PU in the PU group, the G-PCC decoder 300 may reconstruct an actual motion vector for the PU based on the MVD for the PU and the base motion vector for the PU group. In some examples, the base motion may be a translational part of the global motion. In some examples, a rotation matrix of the global motion may be applied in the motion compensation of the PU.
[0104]
[0116] In one exemplary G-PCC codec, reference points for a PU are derived first. Then, a set of these reference points is used for motion estimation and motion compensation. This disclosure proposes to use reference information, such as the number of reference points, to restrict the use of local prediction. For example, in one example, an LPU may not be predicted using local prediction if the number of reference points (or the number of points in the active window) is less than a predefined threshold, which may be signaled in the header information, e.g., the geometry slice header, the geometry parameter set, gbh, or the sequence parameter set (sps) level. In this example, if this condition is met, the LPU may not be further split into sub-PUs, and the LPU may be considered as a vehicle LPU with 0 local motion. The LPU mode and split flags and motion may not be signaled in this example.
[0105]
[0117] In some examples, splitting of a PU into PUs at higher depths may be limited by the number of reference points of the PU. For example, if the number of reference points is smaller than a predefined threshold, splitting of the PU may not be allowed. The predefined threshold for PU splitting restriction may depend on the PU size or the PU depth.
[0106]
[0118] FIG. 8 is a flowchart illustrating an example operation of the G-PCC encoder 200 according to the techniques of this disclosure. In the example of FIG. 8, a geometric coding unit (e.g., the octree analysis unit 210 and / or the arithmetic coding unit 214) of the G-PCC encoder 200 may determine that the point cloud data is to be divided into a plurality of LPUs (800). According to one or more techniques of this disclosure, at least two of the LPUs have different sizes along different directions. For example, the LPUs may have a first size in a first direction (e.g., the x, y, or z direction) and the LPUs may have a second size in a second direction that is different from the first size. In some examples, the sizes of the LPUs align with a quadtree-binary tree division of the point cloud data determined by octree geometric coding. The G-PCC encoder 200 may analyze different possible ways of dividing the point cloud data into a plurality of LPUs. The G-PCC encoder 200 may select a way to divide the point cloud based on a rate-distortion metric.
[0107]
[0119] The geometry coding unit of the G-PCC encoder 200 may signal that the point cloud data is split into multiple LPUs according to any of the examples provided elsewhere in this disclosure. For example, for at least one of the x, y, and z directions, the G-PCC encoder 200 may signal a syntax element indicating whether splitting of the point cloud data is enabled in that direction and the size of the LPU along that direction.
[0108]
[0120] Additionally, in the example of FIG. 8, the geometric coding unit of the G-PCC encoder 200 may perform inter prediction to determine a prediction point for the LPU (802). For example, the G-PCC encoder 200 may perform inter prediction, for example, as described with respect to FIG. 4. For example, the LPUs may include a first LPU (e.g., a vehicle LPU) and a second LPU (e.g., a world LPU), and the G-PCC encoder 200 may determine a prediction point in the first LPU based on a previously reconstructed point cloud without applying global motion compensation to the previously reconstructed point cloud. The G-PCC encoder 200 may determine a prediction point in the second LPU based on a previously reconstructed point cloud by applying global motion compensation to the previously reconstructed point cloud. Furthermore, in some examples, the G-PCC encoder 200 may apply location motion estimation and local motion compensation. For example, the G-PCC encoder 200 may determine the motion of one LPU of the multiple LPUs and use the motion of the LPU to predict points in the LPU. In examples in which one or more of the multiple LPUs are further divided into PUs, the G-PCC encoder 200 may perform inter prediction to determine prediction points for the PUs of the LPUs.
[0109]
[0121] The geometry coding unit of the G-PCC encoder 200 may encode a point in the LPU based on the predicted point for the LPU (804). For example, the G-PCC encoder 200 may signal an encoded residual value for the point in the LPU. The residual value for the point may indicate a difference between the actual position of the point and the position of the corresponding predicted point.
[0110]
[0122] FIG. 9 is a flowchart illustrating an example operation of the G-PCC decoder 300 according to the disclosed techniques. In the example of FIG. 9, a geometric decoding unit (e.g., a geometric arithmetic decoding unit 302 and an octree synthesis unit 306) of the G-PCC decoder 300 may determine that the point cloud data is divided into a plurality of LPUs (900). According to one or more techniques of the present disclosure, at least two of the plurality of LPUs have different sizes along different directions. The G-PCC decoder 300 may determine that the point cloud data is divided into a plurality of LPUs according to any of the examples provided elsewhere in the present disclosure. For example, the LPUs may have a first size in a first direction (e.g., x, y, or z direction), and the LPUs may have a second size in a second direction that is different from the first size. In some examples, the size of the LPUs aligns with a quadtree-binary tree division of the point cloud data determined by octree geometric coding. In some examples, as part of determining that the point cloud data is split into LPUs, the G-PCC decoder 300 may obtain a syntax element for at least one of the x, y, and z directions that indicates both whether splitting of the point cloud data is enabled in that direction and the size of the LPU along that direction.
[0111]
[0123] Further, in the example of FIG. 9, the geometric decoding unit of the G-PCC decoder 300 may perform inter prediction to determine a prediction point for the LPU (902). For example, the G-PCC decoder 300 may perform inter prediction as described above with respect to FIG. 4 and FIG. 5. For example, the LPUs may include a vehicle LPU and a world LPU, and the G-PCC encoder 200 may determine a prediction point in the vehicle LPU based on a previously reconstructed point cloud without applying global motion compensation to the previously reconstructed point cloud. The G-PCC encoder 200 may determine a prediction point in the world LPU based on a previously reconstructed point cloud by applying global motion compensation to the previously reconstructed point cloud. Further, in some examples, the G-PCC encoder 200 may apply location motion estimation and local motion compensation. For example, the G-PCC encoder 200 may determine the motion of one LPU of the multiple LPUs and use the motion of the LPU to predict points in the LPU.
[0112]
[0124] The G-PCC decoder 300 may reconstruct a point in the LPU based on the predicted point for the LPU (904). For example, the G-PCC decoder 300 may obtain an encoded residual value for the point in the LPU. The G-PCC decoder 300 may reconstruct a point in the LPU using the encoded residual value for the point and the predicted point for the LPU.
[0113]
[0125] FIG. 10 is a conceptual diagram illustrating an example distance measurement system 1000 that can be used in one or more techniques of the present disclosure. In the example of FIG. 10, the distance measurement system 1000 includes an illuminator 1002 and a sensor 1004. The illuminator 1002 can emit light 1006. In some examples, the illuminator 1002 can emit the light 1006 as one or more laser beams. The light 1006 can be one or more wavelengths, such as infrared wavelengths or visible light wavelengths. In other examples, the light 1006 is a non-coherent laser light. When the light 1006 hits an object, such as an object 1008, the light 1006 forms a return light 1010. The return light 1010 can include backscattered light and / or reflected light. The return light 1010 can pass through a lens 1011 that directs the return light 1010 to form an image 1012 of the object 1008 on the sensor 1004. The sensor 1004 generates a signal 1014 based on an image 1012. The image 1012 may comprise a set of points (e.g., as shown by the dots in the image 1012 of FIG. 10).
[0114]
[0126] In some examples, the illuminator 1002 and the sensor 1004 may be mounted on a rotating structure such that the illuminator 1002 and the sensor 1004 capture a 360-degree view of the environment. In other examples, the distance measurement system 1000 may include one or more optical components (e.g., mirrors, collimators, diffraction gratings, etc.) that enable the illuminator 1002 and the sensor 1004 to detect the distance of an object within a certain range (e.g., up to 360 degrees). Although the example of FIG. 10 shows only a single illuminator 1002 and sensor 1004, the distance measurement system 1000 may include multiple sets of illuminators and sensors.
[0115]
[0127] In some examples, the illuminator 1002 generates a structured light pattern. In such examples, the distance measurement system 1000 may include multiple sensors 1004 on which respective images of the structured light pattern are formed. The distance measurement system 1000 may use the parallax between the images of the structured light pattern to determine the distance to an object 1008 from which the structured light pattern is backscattered. The structured light based distance measurement system may have a high level of accuracy (e.g., accuracy in the sub-millimeter range) when the object 1008 is relatively close (e.g., 0.2 meters to 2 meters) to the sensor 1004. This high level of accuracy may be useful in facial recognition applications, such as unlocking a mobile device (e.g., a mobile phone, a tablet computer, etc.), and for security applications.
[0116]
[0128] In some examples, the distance measurement system 1000 is a time of flight (ToF) based system. In some examples where the distance measurement system 1000 is a ToF based system, the illuminator 1002 generates a pulse of light. In other words, the illuminator 1002 may modulate the amplitude of the emitted light 1006. In such examples, the sensor 1004 detects the return light 1010 from the pulse of light 1006 generated by the illuminator 1002. The distance measurement system 1000 may then determine the distance to the object 1008 from which the light 1006 is backscattered based on the delay between when the light 1006 is emitted and when it is detected, and the known speed of light in air. In some examples, rather than (or in addition to) modulating the amplitude of the emitted light 1006, the illuminator 1002 may modulate the phase of the emitted light 1006. In such an example, the sensor 1004 may detect the phase of the returning light 1010 from the object 1008 and determine the distance to a point on the object 1008 using the speed of light and based on the time difference between when the illuminator 1002 generates the light 1006 at a particular phase and when the sensor 1004 detects the returning light 1010 at that particular phase.
[0117]
[0129] In other examples, the point cloud data may be generated without the use of the illuminator 1002. For example, in some examples, the sensor 1004 of the distance measurement system 1000 may include two or more optical cameras. In such examples, the distance measurement system 1000 may use the optical cameras to capture a stereo image of an environment including the object 1008. The distance measurement system 1000 may include a point cloud generator 1020 capable of calculating disparity between locations in the stereo image. The distance measurement system 1000 may then use this disparity to determine distances to locations shown in the stereo image. From these distances, the point cloud generator 1020 may generate the point cloud data.
[0118]
[0130] The sensor 1004 may also detect other attributes of the object 1008, such as color and reflectance information. In the example of Figure 10, the point cloud generator 1020 may generate point cloud data based on the signal 1014 generated by the sensor 1004. The distance measurement system 1000 and / or the point cloud generator 1020 may form part of the data source 104 (Figure 1). Hence, the point cloud data generated by the distance measurement system 1000 may be encoded and / or decoded according to any of the techniques of this disclosure.
[0119]
[0131] FIG. 11 is a conceptual diagram illustrating an example vehicle-based scenario in which one or more techniques of the present disclosure may be used. In the example of FIG. 11, a vehicle 1100 includes a distance measurement system 1102. The distance measurement system 1102 may be implemented in the manner discussed with respect to FIG. 10. Although not shown in the example of FIG. 11, the vehicle 1100 may also include a data source, such as data source 104 (FIG. 1), and a G-PCC encoder, such as G-PCC encoder 200 (FIG. 1). In the example of FIG. 11, the distance measurement system 1102 emits a laser beam 1104 that reflects off a pedestrian 1106 or other object in a road. The data source of the vehicle 1100 may generate point cloud data based on a signal generated by the distance measurement system 1102. The G-PCC encoder of the vehicle 1100 may encode the point cloud data to generate a bitstream 1108, such as a geometry bitstream 203 (FIG. 2) and an attribute bitstream 205 (FIG. 2). The bitstream 1108 may contain many fewer bits than the unencoded point cloud data obtained by the G-PCC encoder.
[0120]
[0132] An output interface of the vehicle 1100 (e.g., output interface 108 (FIG. 1)) may transmit the bitstream 1108 to one or more other devices. The bitstream 1108 may include many fewer bits than the unencoded point cloud data obtained by the G-PCC encoder. Thus, the vehicle 1100 may be able to transmit the bitstream 1108 to other devices more quickly than the unencoded point cloud data. In addition, the bitstream 1108 may require less data storage capacity. As described elsewhere in this disclosure, techniques of this disclosure may further reduce the size of the encoded data.
[0121]
[0133] In the example of FIG. 11, the vehicle 1100 may transmit a bit stream 1108 to another vehicle 1110. The vehicle 1110 may include a G-PCC decoder, such as the G-PCC decoder 300 (FIG. 1). The G-PCC decoder of the vehicle 1110 may decode the bit stream 1108 to reconstruct the point cloud data. Reconstructing the point cloud data may include determining coordinates of points of the point cloud data and / or determining attribute values for the points of the point cloud data. The vehicle 1110 may use the reconstructed point cloud data for various purposes. For example, the vehicle 1110 may determine based on the reconstructed point cloud data that a pedestrian 1106 is on the road ahead of the vehicle 1100 and therefore begin to slow down even before the driver of the vehicle 1110 recognizes that the pedestrian 1106 is on the road. Thus, in some examples, the vehicle 1110 may perform an autonomous navigation operation based on the reconstructed point cloud data.
[0122]
[0134] Additionally or alternatively, the vehicle 1100 may transmit the bitstream 1108 to the server system 1112. The server system 1112 may use the bitstream 1108 for various purposes. For example, the server system 1112 may store the bitstream 1108 for subsequent reconstruction of the point cloud data. In this example, the server system 1112 may use the point cloud data along with other data (e.g., vehicle telemetry data generated by the vehicle 1100) to train an autonomous driving system. In another example, the server system 1112 may store the bitstream 1108 for subsequent reconstruction for forensic accident investigation (e.g., when the vehicle 1100 collides with the pedestrian 1106).
[0123]
[0135] The techniques of this disclosure may increase coding efficiency, which may reduce the size of the bitstream 1108 or may allow more detailed data to be sent in the bitstream 1108 without increasing the size of the bitstream 1108.
[0124]
[0136] FIG. 12 is a conceptual diagram illustrating an example extended reality system in which one or more techniques of the present disclosure may be used. Extended reality (XR) is a term used to cover a range of technologies including augmented reality (AR), mixed reality (MR), and virtual reality (VR). In the example of FIG. 12, a user 1200 is located at a first location 1202. The user 1200 is wearing an XR headset 1204. Instead of the XR headset 1204, the user 1200 may use a mobile device (e.g., a mobile phone, a tablet computer, etc.). The XR headset 1204 includes a depth detection sensor, such as a distance measurement system, that detects the position of a point on an object 1206 at the location 1202. A data source of the XR headset 1204 may use a signal generated by the depth detection sensor to generate point cloud data representing the object 1206 at the location 1202. The XR headset 1204 may include a G-PCC encoder (e.g., the G-PCC encoder 200 of FIG. 1) configured to encode the point cloud data to generate a bitstream 1208.
[0125]
[0137] The XR headset 1204 may transmit the bitstream 1208 (e.g., over a network such as the Internet) to an XR headset 1210 worn by a user 1212 at a second location 1214. The XR headset 1210 may decode the bitstream 1208 to reconstruct the point cloud data. The XR headset 1210 may use the point cloud data to generate an XR visualization (e.g., an AR, MR, VR visualization) representing the object 1206 at the location 1202. Thus, in some examples, such as when the XR headset 1210 generates a VR visualization, the user 1212 may have a 3D immersive experience of the location 1202. In some examples, the XR headset 1210 may determine a position of a virtual object based on the reconstructed point cloud data. For example, the XR headset 1210 may determine that an environment (e.g., the location 1202) includes a flat surface and then determine that a virtual object (e.g., an animated character) is to be placed on the flat surface based on the reconstructed point cloud data. The XR headset 1210 may generate an XR visualization with the virtual object in the determined position. For example, the XR headset 1210 may display an animated character sitting on a flat surface. Techniques of this disclosure may increase coding efficiency, which may reduce the size of the bitstream 1208 or allow more detailed data to be sent in the bitstream 1208 without increasing the size of the bitstream 1208.
[0126]
[0138] FIG. 13 is a conceptual diagram illustrating an example mobile device system in which one or more techniques of the present disclosure may be used. In the example of FIG. 13, a mobile device 1300, such as a mobile phone or tablet computer, includes a distance measurement system, such as a LIDAR system, that detects the position of a point on an object 1302 in an environment of the mobile device 1300. A data source of the mobile device 1300 may use a signal generated by a depth detection sensor to generate point cloud data representing the object 1302. The mobile device 1300 may include a G-PCC encoder (e.g., the G-PCC encoder 200 of FIG. 1) configured to encode the point cloud data to generate a bitstream 1304. In the example of FIG. 13, the mobile device 1300 may transmit the bitstream to a remote device 1306, such as a server system or another mobile device. The remote device 1306 may decode the bitstream 1304 to reconstruct the point cloud data. The remote device 1306 may use the point cloud data for various purposes. For example, the remote device 1306 may use the point cloud data to generate a map of the environment of the mobile device 1300. For example, the remote device 1306 may generate a map of the interior of a building based on the reconstructed point cloud data. In another example, the remote device 1306 may generate imagery (e.g., computer graphics) based on the point cloud data. For example, the remote device 1306 may use points of the point cloud data as vertices of a polygon and color attributes of the points as a basis for shading the polygon. In some examples, the remote device 1306 may use the reconstructed point cloud data for facial recognition or other security applications. The techniques of this disclosure may increase coding efficiency, thereby reducing the size of the bitstream 1304 or allowing more detailed data to be sent in the bitstream 1304 without increasing the size of the bitstream 1304.
[0127]
[0139] The embodiments of the various aspects of the present disclosure may be used individually or in any combination.
[0128]
[0140] The following is a non-limiting list of examples in accordance with one or more techniques of the present disclosure.
[0129]
[0141] Example 1A. A method for compounding point cloud data, comprising: determining that the point cloud data is divided into a plurality of largest prediction units (LPUs), at least two of the plurality of LPUs having different sizes along different directions; determining a motion vector of a prediction unit (PU) of one of the plurality of LPUs; using the motion vector of the PU to predict points in the PU; and reconstructing points in the PU based on the predicted points in the PU.
[0130]
[0142] Example 2A. A method for encoding point cloud data, comprising: dividing the point cloud data into a plurality of largest prediction units (LPUs), where at least two of the plurality of LPUs have different sizes along different directions; determining a motion vector for a prediction unit (PU) of one of the plurality of LPUs; using the motion vector of the PU to predict a point in the PU; and encoding a point in the PU based on the predicted point in the PU.
[0131]
[0143] Example 3A. A method for compounding point cloud data, comprising: determining a division of the point cloud data into a plurality of largest prediction units (LPUs); determining a motion of a prediction unit (PU) of one of the plurality of LPUs; using the motion of the PU to predict points in the PU; and reconstructing points in the PU based on the predicted points in the PU.
[0132]
[0144] Example 4A. The method of Example 3A, wherein determining the division of the point cloud data includes one or more of: determining whether the point cloud data is divided into LPUs in the x dimension based on a first signaled instruction; determining whether the point cloud data is divided into LPUs in the y dimension based on a second signaled instruction; or determining whether the point cloud data is divided into LPUs in the z dimension based on a third signaled instruction.
[0133]
[0145] Example 5A. The method of any one of Examples 3A or 4A, wherein determining a division of the point cloud data includes, for each dimension of the multiple dimensions, determining a size of the LPU along the dimension.
[0134]
[0146] Example 6A. The method of any one of Examples 3A to 5A, wherein the method further includes, for each dimension among the plurality of dimensions, determining a minimum PU size for the dimension, and determining a division of the point cloud data includes determining a division of the point cloud data based in part on the minimum PU size for the dimension.
[0135]
[0147] Example 7A. The method of example 6A, wherein determining a partitioning of the point cloud data includes, for each dimension among the multiple dimensions, determining whether partitioning is enabled along the dimension based on a minimum PU size for the dimension.
[0136]
[0148] Example 8A. The method of Example 7A, wherein determining a partitioning of the point cloud data includes, for at least one dimension of the plurality of dimensions, inferring a minimum PU size along the dimension to be equal to the size of an LPU along the dimension based on the fact that partitioning is not enabled for the dimension.
[0137]
[0149] Example 9A. The method of any one of Examples 3A to 8A, wherein determining the movement of the PU includes determining that the movement of the PU is set equal to 0.
[0138]
[0150] Example 10A. The method of any one of Examples 3A to 9A, wherein when the PU does not have a reference point, the movement of the PU is not signaled.
[0139]
[0151] Example 11A. The method of any one of Examples 3A to 10A, wherein the group of PUs includes a PU, and the group of PUs share the same base motion.
[0140]
[0152] Example 12A. The method of example 11A, wherein the base motion is a translational portion of the global motion.
[0141]
[0153] Example 13A. The method of any one of Examples 3A to 12A, wherein determining the motion of the PU includes applying a rotation matrix of the global motion in the motion compensation of the PU.
[0142]
[0154] Example 14A. The method of any one of Examples 3A to 13A, further comprising refraining from predicting the LPU using local prediction based on the number of reference points being less than a predefined threshold.
[0143]
[0155] Example 15A. The method of any one of Examples 3A to 13A, wherein determining a division of the point cloud data includes dividing a given PU into PUs of higher depth according to a restriction based on the number of reference points of the given PU.
[0144]
[0156] Example 16A. A method for encoding point cloud data, comprising: dividing the point cloud data into a plurality of largest prediction units (LPUs); determining a motion of a prediction unit (PU) of one of the plurality of LPUs; using the motion of the PU to predict points in the PU; and encoding points in the PU based on the predicted points in the PU.
[0145]
[0157] Example 17A. The method of Example 16A, further comprising one or more of: signaling a first instruction indicating whether the point cloud data is divided into LPUs in the x dimension; signaling a second instruction indicating whether the point cloud data is divided into LPUs in the y dimension; or signaling a third instruction indicating whether the point cloud data is divided into LPUs in the z dimension.
[0146]
[0158] Example 18A. The method of any one of Examples 16A or 17A, wherein splitting the point cloud data includes, for each dimension of the multiple dimensions, determining a size of the LPU along the dimension.
[0147]
[0159] Example 19A. The method of any one of Examples 16A to 18A, wherein the method further includes, for each dimension among the plurality of dimensions, determining a minimum PU size for the dimension, and wherein splitting the point cloud data includes splitting the point cloud data based in part on the minimum PU size for the dimension.
[0148]
[0160] Example 20A. The method of example 19A, wherein determining a partitioning of the point cloud data includes, for each dimension among the multiple dimensions, determining whether partitioning is enabled along the dimension based on a minimum PU size for the dimension.
[0149]
[0161] Example 21A. The method of example 20A, wherein splitting the point cloud data includes, for at least one dimension of the plurality of dimensions, inferring a minimum PU size along the dimension to be equal to the size of an LPU along the dimension based on the fact that splitting is not enabled for the dimension.
[0150]
[0162] Example 22A. The method of any one of Examples 16A to 21A, wherein determining the movement of the PU includes determining that the movement of the PU is set equal to 0.
[0151]
[0163] Example 23A. The method of any one of Examples 16A to 22A, further comprising refraining from signaling a movement of the PU based on the PU not having a reference point.
[0152]
[0164] Example 24A. The method of any one of Examples 16A to 23A, wherein the group of PUs includes a PU, and the group of PUs share the same base motion.
[0153]
[0165] Example 25A. The method of example 24A, wherein the base motion is a translational portion of the global motion.
[0154]
[0166] Example 26A. The method of any one of Examples 16A to 25A, wherein determining the motion of the PU includes applying a rotation matrix of the global motion in the motion compensation of the PU.
[0155]
[0167] Example 27A. The method of any one of Examples 16A to 26A, further comprising refraining from predicting the LPU using local prediction based on the number of reference points being less than a predefined threshold.
[0156]
[0168] Example 28A. The method of any one of Examples 16A to 27A, wherein splitting the point cloud data includes splitting a given PU into PUs of higher depth according to a restriction based on the number of reference points of the given PU.
[0157]
[0169] Example 29A. A device for processing point clouds, the device comprising one or more means for carrying out the method according to any one of Examples 1A to 28A.
[0158]
[0170] Example 30A. The device of Example 29A, wherein the one or more means comprises one or more processors implemented in the circuit configuration.
[0159]
[0171] Example 31A. The device of any one of Examples 29A or 30A, further comprising a memory for storing data representing the point cloud data.
[0160]
[0172] Example 32A. The device of any one of Examples 29A to 31A, wherein the device comprises a decoder.
[0161]
[0173] Example 33A. A device described in any one of Examples 29A to 32A, wherein the device comprises an encoder.
[0162]
[0174] Example 34A. The device of any one of Examples 29A to 33A, further comprising a device for generating point cloud data.
[0163]
[0175] Example 35A. The device of any one of Examples 29A to 34A, further comprising a display for displaying an image based on the point cloud data.
[0164]
[0176] Example 36A. A computer-readable storage medium storing instructions that, when executed, cause one or more processors to perform the method of any one of Examples 1A to 28A.
[0165]
[0177] Example 1B: A method for compounding point cloud data includes determining that the point cloud data is divided into a plurality of largest prediction units (LPUs), where at least two of the plurality of LPUs have different sizes along different directions, performing inter prediction to determine prediction points for the LPUs, and reconstructing points within the LPUs based on the prediction points for the LPUs.
[0166]
[0178] Example 2B: The method of example 1B, wherein the size of the LPU is aligned with a quadtree-binary tree division of the point cloud data determined by octree geometric coding.
[0167]
[0179] Example 3B: A method as described in any one of Examples 1B and 2B, wherein the LPU includes a first LPU and a second LPU, and performing inter prediction to determine a prediction point within the LPU includes determining a prediction point within the first LPU based on a previously reconstructed point cloud without applying global motion compensation to the previously reconstructed point cloud, and determining a prediction point within the second LPU based on the previously reconstructed point cloud by applying global motion compensation to the previously reconstructed point cloud.
[0168]
[0180] Example 4B: A method as described in any one of Examples 1B to 3B, wherein performing inter prediction to determine a predicted point within an LPU includes determining a movement of one LPU among a plurality of LPUs, and using the movement of the LPU to predict a point within the LPU.
[0169]
[0181] Example 5B: A method according to any one of Examples 1B to 4B, wherein determining that the point cloud data is divided into multiple LPUs includes determining that the point cloud data is divided into LPUs in the x direction based on a first signaled instruction, determining that the point cloud data is divided into LPUs in the y direction based on a second signaled instruction, or determining that the point cloud data is divided into LPUs in the z direction based on a third signaled instruction.
[0170]
[0182] Example 6B: A method as described in any one of Examples 1B to 4B, wherein determining that the point cloud data is divided into LPUs includes obtaining a syntax element indicating, for at least one of the x, y, and z directions, whether division of the point cloud data is enabled in that direction and the size of the LPU along that direction.
[0171]
[0183] Example 7B: A method as described in any one of Examples 1B to 4B, wherein determining that the point cloud data is divided into LPUs includes determining that division of the point cloud data is not enabled along at least one of the x, y, or z directions, and based on a determination that division of the point cloud data is not enabled along that direction, determining that the size of the LPU is equal to the size of the point cloud data along that direction.
[0172]
[0184] Example 8B: A method as described in any one of Examples 1B to 7B, including one of: the directions include x, y, and z directions, and the point cloud data is divided in a vertical direction and the size of the LPU in the x and y directions is equal to the size of the point cloud data in the x and y directions; the point cloud data is divided in a horizontal x direction and the size of the LPU in the y and z directions is equal to the size of the point cloud data in the x and y directions; or the point cloud data is divided in a horizontal y direction and the size of the LPU in the x and z directions is equal to the size of the point cloud data in the x and y directions.
[0173]
[0185] Example 9B: A method as described in any one of Examples 1B to 8B, wherein determining that the point cloud data is divided into multiple LPUs includes determining that the point cloud data is divided into LPUs in only one of multiple directions based on syntax elements signaled at the header level.
[0174]
[0186] Example 10B: A method for encoding point cloud data includes determining that the point cloud data is divided into a plurality of largest prediction units (LPUs), where at least two of the plurality of LPUs have different sizes along different directions, performing inter prediction to determine prediction points for the LPUs, and encoding points within the LPUs based on the prediction points for the LPUs.
[0175]
[0187] Example 11B: The method described in Example 10B, wherein the size of the LPU is aligned with a quadtree-binary tree division of the point cloud data determined by octree geometric coding.
[0176]
[0188] Example 12B: A method described in any one of Examples 10B and 11B, wherein the LPU includes a first LPU and a second LPU, and performing inter prediction to determine a prediction point within the LPU includes determining a prediction point within the first LPU based on a previously reconstructed point cloud without applying global motion compensation to the previously reconstructed point cloud, and determining a prediction point within the second LPU based on the previously reconstructed point cloud by applying global motion compensation to the previously reconstructed point cloud.
[0177]
[0189] Example 13B: A method described in any one of Examples 10B to 12B, wherein performing inter prediction to determine a predicted point within an LPU includes determining a movement of one LPU among a plurality of LPUs, and using the movement of the LPU to predict a point within the LPU.
[0178]
[0190] Example 14B: The method of any one of Examples 10B to 13B further includes signaling a first instruction indicating whether the point cloud data is divided into LPUs in the x direction, signaling a second signaled instruction indicating whether the point cloud data is divided into LPUs in the y direction, or signaling a third signaled instruction indicating whether the point cloud data is divided into LPUs in the z direction.
[0179]
[0191] Example 15B: A method as described in any one of Examples 10B to 13B, further comprising signaling a syntax element indicating, for at least one of the x, y, and z directions, whether splitting of the point cloud data is enabled in that direction and the size of the LPU along that direction.
[0180]
[0192] Example 16B: A method as described in any one of Examples 10B to 15B, wherein the directions include x, y, and z directions, and the point cloud data is divided in a vertical direction and the size of the LPU in the x and y directions is equal to the size of the point cloud data in the x and y directions; the point cloud data is divided in a horizontal X direction and the size of the LPU in the y and z directions is equal to the size of the point cloud data in the x and y directions; or the point cloud data is divided in a horizontal Y direction and the size of the LPU in the x and z directions is equal to the size of the point cloud data in the x and y directions.
[0181]
[0193] Example 17B: A method according to any one of Examples 10B to 16B, further comprising signaling a syntax element at the header level indicating that the point cloud data is divided into LPUs in only one of the multiple directions.
[0182]
[0194] Example 18B: A device for decoding point cloud data comprises a memory configured to store the point cloud data and one or more processors implemented in a circuit configuration configured to determine that the point cloud data is divided into a plurality of largest prediction units (LPUs), where at least two of the plurality of LPUs have different sizes along different directions, perform inter prediction to determine prediction points for the LPUs, and reconstruct points within the LPUs based on the prediction points for the LPUs.
[0183]
[0195] Example 19B: A device as described in Example 18B, wherein one or more processors are configured, as part of determining that the point cloud data is divided into multiple LPUs, to determine based on a first signaled instruction that the point cloud data is divided into LPUs in the x direction, based on a second signaled instruction that the point cloud data is divided into LPUs in the y direction, or based on a third signaled instruction that the point cloud data is divided into LPUs in the z direction.
[0184]
[0196] Example 20B: The device described in Example 18B, wherein one or more processors are configured to obtain, as part of determining that the point cloud data is divided into LPUs, a syntax element for each of the x, y, and z directions indicating whether division of the point cloud data is enabled in that direction and the size of the LPU along that direction.
[0185]
[0197] Example 21B: A device as described in Example 18B, wherein one or more processors are configured to determine, as part of determining that the point cloud data is divided into LPUs, that division of the point cloud data is not enabled along at least one of the x, y, or z directions, and to determine that the size of the LPU is equal to the size of the point cloud data along that direction.
[0186]
[0198] Example 22B: A device described in any one of Examples 18B to 21B, including one of: the directions include x, y, and z directions, and the point cloud data is divided in a vertical direction and the size of the LPU in the x and y directions is equal to the size of the point cloud data in the x and y directions; the point cloud data is divided in a horizontal X direction and the size of the LPU in the y and z directions is equal to the size of the point cloud data in the x and y directions; or the point cloud data is divided in a horizontal Y direction and the size of the LPU in the x and z directions is equal to the size of the point cloud data in the x and y directions.
[0187]
[0199] Example 23B: A device described in any one of Examples 18B to 22B, further comprising a display for displaying an image based on the point cloud data.
[0188]
[0200] Example 24B: A device for encoding point cloud data comprises a memory configured to store the point cloud data, and one or more processors implemented in a circuit configuration configured to determine that the point cloud data is divided into a plurality of largest prediction units (LPUs), at least two of the plurality of LPUs having different sizes along different directions, perform inter prediction to determine prediction points for the LPUs, and encode points within the LPUs based on the prediction points for the LPUs.
[0189]
[0201] Example 25B: A device as described in Example 24B, wherein one or more processors are configured to signal, as part of determining that the point cloud data is divided into multiple LPUs, a first indication indicating whether the point cloud data is divided into LPUs in the x direction, a second signaled indication indicating whether the point cloud data is divided into LPUs in the y direction, or a third signaled indication indicating whether the point cloud data is divided into LPUs in the z direction.
[0190]
[0202] Example 26B: A device as described in Example 24B, wherein one or more processors are configured to signal a syntax element indicating, for at least one of the x, y, or z directions, whether splitting of the point cloud data is enabled in that direction and the size of the LPU along that direction.
[0191]
[0203] Example 27B: A device described in any one of Examples 24B to 26B, including one of: the directions include x, y, and z directions, and the point cloud data is divided in a vertical direction and the size of the LPU in the x and y directions is equal to the size of the point cloud data in the x and y directions; the point cloud data is divided in a horizontal X direction and the size of the LPU in the y and z directions is equal to the size of the point cloud data in the x and y directions; or the point cloud data is divided in a horizontal Y direction and the size of the LPU in the x and z directions is equal to the size of the point cloud data in the x and y directions.
[0192]
[0204] Example 28B: A device described in any one of Examples 24B to 27B, further comprising a device for generating point cloud data.
[0193]
[0205] Example 1C: A method for compounding point cloud data includes determining that the point cloud data is divided into a plurality of largest prediction units (LPUs), where at least two of the LPUs have different sizes along different directions, performing inter prediction to determine predicted points for the LPUs, and reconstructing points within the LPUs based on the predicted points for the LPUs.
[0194]
[0206] Example 2C: The method of example 1C, wherein the size of the LPU is aligned with a quadtree-binary tree division of the point cloud data determined by octree geometric coding.
[0195]
[0207] Example 3C: The method described in Example 1C, wherein the LPU includes a first LPU and a second LPU, and performing inter prediction to determine a prediction point within the LPU includes determining a prediction point within the first LPU based on a previously reconstructed point cloud without applying global motion compensation to the previously reconstructed point cloud, and determining a prediction point within the second LPU based on the previously reconstructed point cloud by applying global motion compensation to the previously reconstructed point cloud.
[0196]
[0208] Example 4C: The method described in Example 1C, wherein performing inter prediction to determine a predicted point within an LPU includes determining a motion of one LPU among a plurality of LPUs, and using the motion of the LPU to predict a point within the LPU.
[0197]
[0209] Example 5C: The method of Example 1C, wherein determining that the point cloud data is divided into multiple LPUs includes determining that the point cloud data is divided into LPUs in the x direction based on a first signaled instruction, determining that the point cloud data is divided into LPUs in the y direction based on a second signaled instruction, or determining that the point cloud data is divided into LPUs in the z direction based on a third signaled instruction.
[0198]
[0210] Example 6C: The method described in Example 1C, wherein determining that the point cloud data is divided into LPUs includes obtaining a syntax element that indicates, for at least one of the x, y, and z directions, whether division of the point cloud data is enabled in that direction and the size of the LPU along that direction.
[0199]
[0211] Example 7C: The method of Example 1C, wherein determining that the point cloud data is divided into LPUs includes determining that division of the point cloud data is not enabled along at least one of the x, y, or z directions, and based on a determination that division of the point cloud data is not enabled along that direction, determining that the size of the LPU is equal to the size of the point cloud data along that direction.
[0200]
[0212] Example 8C: The method of Example 1C, including one of: the directions include x, y, and z directions, and the point cloud data is divided in a vertical direction and the size of the LPU in the x and y directions is equal to the size of the point cloud data in the x and y directions; the point cloud data is divided in a horizontal x direction and the size of the LPU in the y and z directions is equal to the size of the point cloud data in the x and y directions; or the point cloud data is divided in a horizontal y direction and the size of the LPU in the x and z directions is equal to the size of the point cloud data in the x and y directions.
[0201]
[0213] Example 9C: The method described in Example 1C, wherein determining that the point cloud data is divided into multiple LPUs includes determining that the point cloud data is divided into LPUs in only one of the directions based on syntax elements signaled at the header level.
[0202]
[0214] Example 10C: A method for encoding point cloud data includes determining that the point cloud data is divided into a plurality of largest prediction units (LPUs), where at least two of the plurality of LPUs have different sizes along different directions, performing inter prediction to determine prediction points for the LPUs, and encoding points within the LPUs based on the prediction points for the LPUs.
[0203]
[0215] Example 11C: The method described in Example 10C, wherein the size of the LPU is aligned with a quadtree-binary tree division of the point cloud data determined by octree geometric coding.
[0204]
[0216] Example 12C: The method described in Example 10C, wherein the LPU includes a first LPU and a second LPU, and performing inter prediction to determine a prediction point within the LPU includes determining a prediction point within the first LPU based on a previously reconstructed point cloud without applying global motion compensation to the previously reconstructed point cloud, and determining a prediction point within the second LPU based on the previously reconstructed point cloud by applying global motion compensation to the previously reconstructed point cloud.
[0205]
[0217] Example 13C: The method described in Example 10C, wherein performing inter prediction to determine a predicted point within an LPU includes determining a movement of one LPU among a plurality of LPUs, and using the movement of the LPU to predict a point within the LPU.
[0206]
[0218] Example 14C: The method of Example 10C, further comprising signaling a first signaled instruction indicating whether the point cloud data is divided into LPUs in the x direction, signaling a second signaled instruction indicating whether the point cloud data is divided into LPUs in the y direction, or signaling a third signaled instruction indicating whether the point cloud data is divided into LPUs in the z direction.
[0207]
[0219] Example 15C: The method described in Example 10C, further comprising signaling a syntax element indicating, for at least one of the x, y, and z directions, whether splitting of the point cloud data is enabled in that direction and the size of the LPU along that direction.
[0208]
[0220] Example 16C: The method of Example 10C, including one of: the directions include x, y, and z directions, and the point cloud data is divided in a vertical direction and the size of the LPU in the x and y directions is equal to the size of the point cloud data in the x and y directions; the point cloud data is divided in a horizontal X direction and the size of the LPU in the y and z directions is equal to the size of the point cloud data in the x and y directions; or the point cloud data is divided in a horizontal Y direction and the size of the LPU in the x and z directions is equal to the size of the point cloud data in the x and y directions.
[0209]
[0221] Example 17C: The method of example 10C, further comprising signaling a syntax element at the header level indicating that the point cloud data is divided into LPUs in only one of the directions.
[0210]
[0222] Example 18C: A device for combining point cloud data comprises a memory configured to store the point cloud data, and one or more processors implemented in a circuit configuration configured to determine that the point cloud data is divided into a plurality of largest prediction units (LPUs), where at least two of the plurality of LPUs have different sizes along different directions, perform inter prediction to determine predicted points for the LPUs, and reconstruct points within the LPUs based on the predicted points for the LPUs.
[0211]
[0223] Example 19C: The device described in Example 18C, wherein one or more processors are configured, as part of determining that the point cloud data is divided into multiple LPUs, to determine based on a first signaled instruction that the point cloud data is divided into LPUs in the x direction, based on a second signaled instruction that the point cloud data is divided into LPUs in the y direction, or based on a third signaled instruction that the point cloud data is divided into LPUs in the z direction.
[0212]
[0224] Example 20C: The device described in Example 18C, wherein one or more processors are configured to obtain, as part of determining that the point cloud data is divided into LPUs, a syntax element for each of the x, y, and z directions indicating whether division of the point cloud data is enabled in that direction and the size of the LPU along that direction.
[0213]
[0225] Example 21C: The device described in Example 18C, wherein the one or more processors are configured to determine, as part of determining that the point cloud data is divided into LPUs, that division of the point cloud data is not enabled along at least one of the x, y, or z directions, and to determine that the size of the LPU is equal to the size of the point cloud data along that direction.
[0214]
[0226] Example 22C: The device of Example 18C, including one of: the directions include x, y, and z directions, and the point cloud data is divided in a vertical direction and the size of the LPU in the x and y directions is equal to the size of the point cloud data in the x and y directions; the point cloud data is divided in a horizontal X direction and the size of the LPU in the y and z directions is equal to the size of the point cloud data in the x and y directions; or the point cloud data is divided in a horizontal Y direction and the size of the LPU in the x and z directions is equal to the size of the point cloud data in the x and y directions.
[0215]
[0227] Example 23C: A device described in Example 18C, further comprising a display for displaying an image based on the point cloud data.
[0216]
[0228] Example 24C: A device for encoding point cloud data comprises a memory configured to store the point cloud data, and one or more processors implemented in a circuit configuration configured to determine that the point cloud data is divided into a plurality of largest prediction units (LPUs), at least two of the plurality of LPUs having different sizes along different directions, perform inter prediction to determine prediction points for the LPUs, and encode points within the LPUs based on the prediction points for the LPUs.
[0217]
[0229] Example 25C: The device described in Example 24C, wherein one or more processors are configured to signal, as part of determining that the point cloud data is divided into multiple LPUs, a first indication indicating whether the point cloud data is divided into LPUs in the x direction, a second signaled indication indicating whether the point cloud data is divided into LPUs in the y direction, or a third signaled indication indicating whether the point cloud data is divided into LPUs in the z direction.
[0218]
[0230] Example 26C: The device described in Example 24C, wherein one or more processors are configured to signal a syntax element indicating, for at least one of the x, y, or z directions, whether splitting of the point cloud data is enabled in that direction and the size of the LPU along that direction.
[0219]
[0231] Example 27C: The device of Example 24C, including one of: the directions include x, y, and z directions, and the point cloud data is divided in a vertical direction and the size of the LPU in the x and y directions is equal to the size of the point cloud data in the x and y directions; the point cloud data is divided in a horizontal X direction and the size of the LPU in the y and z directions is equal to the size of the point cloud data in the x and y directions; or the point cloud data is divided in a horizontal Y direction and the size of the LPU in the x and z directions is equal to the size of the point cloud data in the x and y directions.
[0220]
[0232] Example 28C: A device described in Example 24C, further comprising a device for generating point cloud data.
[0221]
[0233] By way of example, it should be appreciated that any acts or events of the techniques described herein may be performed in a different order, may be added, combined, or omitted entirely (e.g., not all acts or events described are necessary to the practice of the techniques). Moreover, in some examples, acts or events may be performed in parallel rather than sequentially, for example, through multithreaded processing, interrupt processing, or multiple processors.
[0222]
[0234] In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted via a computer-readable medium as one or more instructions or code and executed by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium, which corresponds to a tangible medium, such as a data storage medium, or a communication medium, which includes any medium that facilitates transfer of a computer program from one place to another, for example, according to a communication protocol. As such, a computer-readable medium may generally correspond to (1) a tangible computer-readable storage medium that is non-transitory, or (2) a communication medium, such as a signal or carrier wave. A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementation of the techniques described in this disclosure. A computer program product may include a computer-readable medium.
[0223]
[0235] By way of example, and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is properly referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included within the definition of the medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transitory media, but instead refer to non-transitory, tangible storage media. As used herein, disk and disc include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically and discs reproduce data optically using lasers. Combinations of the above are also intended to be included within the scope of computer readable media.
[0224]
[0236] The instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Accordingly, the terms "processor" and "processing circuitry" as used herein may refer to any of the above structures, or any other structure suitable for implementing the techniques described herein. In addition, in some aspects, the functionality described herein may be provided within dedicated hardware and / or software modules configured for encoding and decoding, or may be incorporated into a composite codec. The techniques may also be fully implemented in one or more circuits or logic elements.
[0225]
[0237] The techniques of this disclosure may be implemented in a wide variety of devices or apparatuses, including a wireless handset, an integrated circuit (IC), or a set of ICs (e.g., a chipset). Various components, modules, or units are described in this disclosure to highlight functional aspects of devices configured to implement the disclosed techniques, but do not necessarily require realization by different hardware units. Rather, as described above, the various units may be combined in a codec hardware unit or may be provided by a collection of interoperable hardware units including one or more processors as described above, in conjunction with suitable software and / or firmware.
[0226]
[0238] Various examples have been described. These and other examples are within the scope of the following claims.
Claims
1. 1. A method for decoding point cloud data, comprising: determining that the point cloud data is partitioned into a plurality of largest prediction units (LPUs), at least two of the plurality of LPUs having different sizes along different directions; performing inter prediction to determine a prediction point for the LPU; and reconstructing points within the LPU based on the predicted points for the LPU.
2. The method of claim 1 , wherein the size of the LPU aligns with a quadtree-binary tree partitioning of the point cloud data determined by octree geometric coding.
3. the LPU includes a first LPU and a second LPU; performing inter prediction to determine the prediction point within the LPU; determining a prediction point in the first LPU based on a previously reconstructed point cloud without applying global motion compensation to the previously reconstructed point cloud; and applying global motion compensation to the previously reconstructed point cloud to determine a prediction point within the second LPU based on the previously reconstructed point cloud. The method of claim 1.
4. performing inter prediction to determine a prediction point within the LPU; determining a movement of one of the plurality of LPUs; using the motion of the LPU to predict points within the LPU; The method of claim 1 , comprising:
5. Determining that the point cloud data is divided into a plurality of LPUs, determining, based on a first signaled instruction, that the point cloud data is to be divided into the LPUs in an x-direction; determining based on a second signaled instruction that the point cloud data is divided into the LPUs in a y direction; or The method of claim 1 , further comprising: determining, based on a third signaled instruction, that the point cloud data is divided into the LPUs in the z direction.
6. 2. The method of claim 1 , wherein determining that the point cloud data is divided into LPUs includes obtaining a syntax element for at least one of an x, y, and z direction that indicates whether division of the point cloud data is enabled in the direction and a size of the LPU along the direction.
7. determining that the point cloud data is divided into the LPUs; determining that segmentation of the point cloud data is not enabled along at least one of an x, y, or z direction; and determining, based on the determination that splitting of the point cloud data is not enabled along the direction, that a size of the LPU is equal to a size of the point cloud data along the direction.
8. the directions include x, y and z directions; the point cloud data is divided vertically, and the size of the LPU in x and y directions is equal to the size of the point cloud data in the x and y directions; the point cloud data is divided horizontally in the x direction, and the size of the LPU in the y and z directions is equal to the size of the point cloud data in the x and y directions; or the point cloud data is divided in a horizontal y direction, and the size of the LPU in the y and z directions is equal to the size of the point cloud data in the x and y directions; including one of The method of claim 1.
9. 2. The method of claim 1 , wherein determining that the point cloud data is divided into the multiple LPUs comprises determining that the point cloud data is divided into the LPUs in only one of the directions based on syntax elements signaled at a header level.
10. 1. A method for encoding point cloud data, comprising the steps of: determining that the point cloud data is partitioned into a plurality of largest prediction units (LPUs), at least two of the plurality of LPUs having different sizes along different directions; performing inter prediction to determine a prediction point for the LPU; and encoding a point in the LPU based on the predicted point for the LPU.
11. The method of claim 10, wherein the size of the LPU aligns with a quadtree-binary tree partitioning of the point cloud data determined by octree geometric coding.
12. the LPU includes a first LPU and a second LPU; performing inter prediction to determine the prediction point within the LPU; determining a prediction point in the first LPU based on a previously reconstructed point cloud without applying global motion compensation to the previously reconstructed point cloud; and applying global motion compensation to the previously reconstructed point cloud to determine a prediction point within the second LPU based on the previously reconstructed point cloud. The method of claim 10.
13. performing inter prediction to determine a prediction point within the LPU; determining a movement of one of the plurality of LPUs; and using the motion of the LPU to predict points within the LPU.
14. signaling a first indication indicating whether the point cloud data is split into the LPUs in an x-direction; signaling a second signaled indication indicating whether the point cloud data is split into the LPUs in the y direction; or signaling a third signaled indication indicating whether the point cloud data is split into the LPUs in a z-direction. The method of claim 10.
15. 11. The method of claim 10, further comprising signaling a syntax element for at least one of an x, y, and z direction indicating whether splitting of the point cloud data is enabled in the direction and the size of an LPU along the direction.
16. the directions include x, y and z directions; the point cloud data is divided vertically, the size of the LPU in x and y directions being equal to the size of the point cloud data in the x and y directions; the point cloud data is divided in a horizontal x-direction, and the size of the LPU in y and z directions is equal to the size of the point cloud data in the x and y directions; or the point cloud data is divided in a horizontal Y direction, and the size of the LPU in the y and z directions is equal to the size of the point cloud data in the x and y directions; including one of The method of claim 10.
17. The method of claim 10 , further comprising signaling a syntax element at a header level indicating that the point cloud data is divided into the LPUs in only one of the directions.
18. 1. A device for decoding point cloud data, comprising: a memory configured to store the point cloud data; One or more processors implemented in a circuit arrangement, determining that the point cloud data is partitioned into a plurality of largest prediction units (LPUs), at least two of the plurality of LPUs having different sizes along different directions; performing inter prediction to determine a prediction point for the LPU; one or more processors configured to reconstruct points in the LPU based on the predicted points for the LPU; A device comprising:
19. As part of the one or more processors determining that the point cloud data is divided into a plurality of LPUs, determining, based on a first signaled indication, that the point cloud data is to be divided into the LPUs in an x-direction; determining based on a second signaled instruction that the point cloud data is to be divided into the LPUs in a y direction; or The device of claim 18 , configured to determine, based on a third signaled instruction, that the point cloud data is divided into the LPUs in the z direction.
20. 20. The device of claim 18, wherein the one or more processors are configured to obtain, as part of determining that the point cloud data is divided into LPUs, for each of x, y, and z directions, a syntax element indicating whether division of the point cloud data is enabled in the direction and a size of the LPU along the direction.
21. 20. The device of claim 18, wherein the one or more processors are configured to determine, as part of determining that the point cloud data is divided into LPUs, that division of the point cloud data along at least one of an x, y, or z direction is not enabled, and to determine that a size of the LPU is equal to a size of the point cloud data along the direction.
22. the directions include x, y and z directions; the point cloud data is divided vertically, and the size of the LPU in x and y directions is equal to the size of the point cloud data in the x and y directions; the point cloud data is divided in a horizontal x-direction, and the size of the LPU in y and z-directions is equal to the size of the point cloud data in the x and y-directions; or the point cloud data is divided in a horizontal Y direction, and the size of the LPU in the y and z directions is equal to the size of the point cloud data in the x and y directions; including one of 20. The device of claim 18.
23. The device of claim 18 , further comprising a display for displaying an image based on the point cloud data.
24. 1. A device for encoding point cloud data, comprising: a memory configured to store the point cloud data; One or more processors implemented in a circuit arrangement, determining that the point cloud data is partitioned into a plurality of largest prediction units (LPUs), at least two of the plurality of LPUs having different sizes along different directions; performing inter prediction to determine a prediction point for the LPU; one or more processors configured to encode points in the LPU based on the predicted points for the LPU.
25. As part of the one or more processors determining that the point cloud data is divided into a plurality of LPUs, signaling a first indication of whether the point cloud data is split into the LPUs in an x-direction; signaling a second signaling indication indicating whether the point cloud data is split into the LPUs in the y direction; or 25. The device of claim 24, configured to signal a third signaled indication indicating whether the point cloud data is split into the LPUs in the z direction.
26. 25. The device of claim 24, wherein the one or more processors are configured to signal a syntax element for at least one of an x, y, or z direction indicating whether splitting of the point cloud data is enabled in the direction and a size of the LPU along the direction.
27. the directions include x, y and z directions; the point cloud data is divided vertically, and the size of the LPU in x and y directions is equal to the size of the point cloud data in the x and y directions; the point cloud data is divided in a horizontal x-direction, and the size of the LPU in y and z-directions is equal to the size of the point cloud data in the x and y-directions; or the point cloud data is divided in a horizontal Y direction, and the size of the LPU in the y and z directions is equal to the size of the point cloud data in the x and y directions; including one of 25. The device of claim 24.
28. The device of claim 24 , further comprising a sensor for generating the point cloud data.
