A hybrid framework for point cloud compression

Abstract

In one implementation, a hybrid architecture for compressing and decompressing point clouds is proposed. In particular, a first decoding block is for the most significant bits, typically coded by a tree-based coding method; a second decoding block is for the mid-range bits, typically coded by a voxel-based method; and a third decoding block is for the least significant bits, typically coded by a point-based method. For example, the decoder configures a network of decoders according to the total number of bits and bit distribution positions, uses a tree-based decoding block to decode the coarse point cloud and its associated point-wise features, upsamples the coarse point cloud to a denser point cloud, uses a voxel-based decoding block to update the point-wise features, and uses a point-based decoding block to refine the accuracy of the coordinates of the dense but low bit-depth point cloud to a high bit-depth point cloud.

Description

[Technical field]

[0001] The present embodiments relate generally to methods and apparatus for point cloud compression and processing. [Background technology]

[0002] The Point Cloud (PC) data format is a universal data format across several business domains, for example, from autonomous driving, to robotics, augmented reality / virtual reality (AR / VR), civil engineering, computer graphics, and animation / film industries. 3D LiDAR (Light Detection and Ranging) sensors are being deployed in self-driving cars, and affordable LiDAR sensors have been released, for example, from Velodyne Velabit, Apple iPad Pro 2020, and Intel RealSense LiDAR Camera L515. With advances in sensing technology, 3D point cloud data can be more practical than ever and is expected to be the ultimate enabler in the applications discussed herein. Summary of the Invention [Problem to be solved by the invention]

[0003] According to one embodiment, there is provided a method of decoding point cloud data for a point cloud, the method including: decoding a first set of data using a first decoding strategy, the first set of data corresponding to a first bit-level subset of the point cloud data, and decoding a second set of data using a second decoding strategy, the second set of data corresponding to a second bit-level subset of the point cloud data. The method may further include decoding a third set of data using a third decoding strategy, the third set of data corresponding to a third bit-level subset of the point cloud data.

[0004] According to another embodiment, there is provided a method of encoding point cloud data for a point cloud, the method including encoding a first set of data using a first encoding strategy, the first set of data corresponding to a first subset of a bit level of the point cloud data, and encoding a second set of data using a second encoding strategy, the second set of data corresponding to a second subset of a bit level of the point cloud data. The method may further include encoding a third set of data using a third encoding strategy, the third set of data corresponding to a third subset of a bit level of the point cloud data.

[0005] According to another embodiment, there is provided an apparatus for decoding point cloud data for a point cloud, the apparatus comprising one or more processors configured to: decode a first set of data using a first decoding strategy, the first set of data corresponding to a first bit-level subset of the point cloud data, and decode a second set of data using a second decoding strategy, the second set of data corresponding to a second bit-level subset of the point cloud data. The apparatus may be further configured to decode a third set of data using a third decoding strategy, the third set of data corresponding to a third bit-level subset of the point cloud data.

[0006] According to another embodiment, there is provided an apparatus for encoding point cloud data for a point cloud, the apparatus comprising one or more processors configured to encode a first set of data using a first encoding strategy, the first set of data corresponding to a first subset of a bit level of the point cloud data, and to encode a second set of data using a second encoding strategy, the second set of data corresponding to a second subset of a bit level of the point cloud data. The apparatus may be further configured to encode a third set of data using a third encoding strategy, the third set of data corresponding to a third subset of a bit level of the point cloud data.

[0007] One or more embodiments also provide a computer program comprising instructions that, when executed by one or more processors, cause the one or more processors to perform an encoding or decoding method according to any of the embodiments described above. One or more of the embodiments also provide a computer readable storage medium having stored thereon instructions for encoding or decoding point cloud data according to the methods described above.

[0008] One or more embodiments also provide a computer-readable storage medium storing video data generated by the methods described above. One or more embodiments also provide methods and apparatus for transmitting or receiving video data generated by the methods described above. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 illustrates a block diagram of a system in which aspects of the present embodiments may be implemented. [Figure 2A] FIG. 2A illustrates point-based, octree-based, voxel-based, and sparse voxel-based point cloud representations, respectively. [Figure 2B]FIG. 2B illustrates point-based, octree-based, voxel-based, and sparse voxel-based point cloud representations, respectively. [Figure 2C] FIG. 2C illustrates point-based, octree-based, voxel-based, and sparse voxel-based point cloud representations, respectively. [Figure 2D] FIG. 2D illustrates point-based, octree-based, voxel-based, and sparse voxel-based point cloud representations, respectively. [Diagram 3] FIG. 3 illustrates a bit-level example. [Figure 4] FIG. 4 illustrates an encoding diagram according to one embodiment. [Diagram 5] FIG. 5 illustrates a PN block according to one embodiment. [Figure 6] FIG. 6 illustrates a VN block according to one embodiment. [Figure 7] FIG. 7 illustrates an ON block according to one embodiment. [Figure 8] FIG. 8 illustrates a decoding diagram according to one embodiment. [Figure 9] FIG. 9 illustrates an ON* block according to one embodiment. [Figure 10] FIG. 10 illustrates a VN* block according to one embodiment. [Figure 11] FIG. 11 illustrates a PN* block according to one embodiment. [Figure 12] FIG. 12 illustrates an encoding diagram according to another embodiment. [Figure 13] FIG. 13 illustrates a decoding diagram according to another embodiment. [Figure 14] FIG. 14 illustrates a method for encoding point cloud data according to one embodiment. [Figure 15] FIG. 15 illustrates a method for decoding point cloud data according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] FIG. 1 illustrates a block diagram of an example of a system in which various aspects and embodiments may be implemented. System 100 may be embodied as a device including various components described below and configured to perform one or more of the aspects described herein. Examples of such devices include various electronic devices, such as, but not limited to, personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected appliances, and servers. Elements of system 100, alone or in combination, may be embodied in a single integrated circuit, multiple ICs, and / or separate components. For example, in at least one embodiment, the processing elements and encoder / decoder elements of system 100 are distributed across multiple ICs and / or separate components. In various embodiments, system 100 is communicatively coupled to other systems or other electronic devices, for example, via a communication bus or through dedicated input and / or output ports. In various embodiments, system 100 is configured to implement one or more of the aspects described in the present application.

[0011] System 100 includes at least one processor 110 configured to execute instructions loaded therein, for example, to implement various aspects described herein. Processor 110 may include embedded memory, input / output interfaces, and various other circuits as known in the art. System 100 includes at least one memory 120 (e.g., volatile and / or non-volatile memory devices). System 100 includes storage device 140, which may include non-volatile and / or volatile memory, including, but not limited to, EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic disk drives, and / or optical disk drives. Storage device 140 may include, by way of non-limiting example, an internal storage device, an attached storage device, and / or a network-accessible storage device.

[0012] System 100 includes, for example, an encoder / decoder module 130 configured to process data to provide encoded or decoded video, which may include its own processor and memory. Encoder / decoder module 130 represents a module that may be included within a device to perform encoding and / or decoding functions. As is known, a device may include one or both of an encoding and decoding module. Additionally, encoder / decoder module 130 may be implemented as a separate element of system 100 or may be incorporated within processor 110 as a combination of hardware and software, as is known to those skilled in the art.

[0013] Program code to be loaded onto the processor 110 or the encoder / decoder 130 to perform various aspects described herein may be stored in the storage device 140 and then loaded onto the memory 120 for execution by the processor 110. According to various embodiments, one or more of the processor 110, the memory 120, the storage device 140, and the encoder / decoder module 130 may store one or more of various items during execution of the processes described herein. Such stored items may include, but are not limited to, input video, decoded video, or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, expressions, operations, and operational logic.

[0014] In some embodiments, memory internal to the processor 110 and / or the encoder / decoder module 130 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device may be either the processor 110 or the encoder / decoder module 130) is used for one or more of these functions. The external memory may be the memory 120 and / or the storage device 140, e.g., dynamic volatile memory and / or non-volatile flash memory. In some embodiments, the external non-volatile flash memory is used to store the television's operating system. In at least one embodiment, a high-speed external dynamic volatile memory, such as RAM, is used as working memory for video coding and decoding operations, such as MPEG-2, JPEG Pleno, MPEG-1, HEVC, or VVC.

[0015] Input to the elements of system 100 may be provided through various input devices, as indicated at block 105. Such input devices may include, but are not limited to, (i) an RF section that receives, for example, an RF signal transmitted over the air by a broadcast station, (ii) a composite input, (iii) a USB input, and / or (iv) an HDMI input.

[0016] In various embodiments, the input devices of block 105 have associated respective input processing elements as known in the art. For example, the RF section may be associated with elements suitable for (i) selecting a desired frequency (also referred to as selecting a signal or band-limiting a signal to a frequency band), (ii) down-converting the selected signal, (iii) band-limiting again to a narrower frequency band to select a signal frequency band that may be referred to (for example) as a channel in a particular embodiment, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired stream of data packets. The RF section of various embodiments includes one or more elements that perform these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF section may include a tuner that performs these various functions, including, for example, down-converting a received signal to a lower frequency (e.g., an intermediate frequency or a near baseband frequency) or to baseband. In one embodiment of a set-top box, the RF section and its associated input processing elements perform frequency selection by receiving, filtering, downconverting, and refiltering RF signals transmitted over a wired (e.g., cable) medium to a desired frequency band. Various embodiments rearrange the order of the above-described (and other) elements, remove some of these elements, and / or add other elements that perform similar or different functions. Adding elements may include inserting elements between existing elements, for example, inserting amplifiers and analog-to-digital converters. In various embodiments, the RF section includes an antenna.

[0017] Additionally, the USB and / or HDMI terminals may include respective interface processors for connecting the system 100 to other electronic devices across USB and / or HDMI connections. It should be appreciated that various aspects of the input processing, e.g., Reed-Solomon error correction, may be implemented, for example, in a separate input processing IC or in the processor 110, as desired. Similarly, aspects of the USB or HDMI interface processing may be implemented, as desired, in a separate interface IC or in the processor 110. For example, the processor 110 and various processing elements, including the encoder / decoder 130, which operates in combination with memory and storage elements, to process the data stream as desired for presentation on an output device by providing a demodulated, error corrected, and demultiplexed stream to various processing elements, including the encoder / decoder 130, which operates in combination with memory and storage elements.

[0018] The various elements of the system 100 may be provided within an integrated housing in which the various elements may be interconnected and transmit data between each other using a suitable connection arrangement 115, such as an internal bus known in the art, including an I2C bus, wiring, and printed circuit boards.

[0019] System 100 includes a communication interface 150 that enables communication with other devices over a communication channel 190. Communication interface 150 may include, but is not limited to, a transceiver configured to transmit and receive data over communication channel 190. Communication interface 150 may include, but is not limited to, a modem or a network card, and communication channel 190 may be implemented in a wired and / or wireless medium, for example.

[0020] Data is streamed to the system 100 in various embodiments using a Wi-Fi network such as IEEE 802.11. The Wi-Fi signal in these embodiments is received by a communication channel 190 and communication interface 150 adapted for Wi-Fi communication. The communication channel 190 in these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to enable streaming applications and other over-the-top communications. In other embodiments, a set-top box that delivers data via an HDMI connection in the input block 105 is used to provide the streamed data to the system 100. In yet other embodiments, an RF connection in the input block 105 is used to provide the streamed data to the system 100.

[0021] System 100 may provide output signals to a variety of output devices, including display 165, speakers 175, and other peripheral devices 185. Other peripheral devices 185, in various example embodiments, include one or more of a standalone DVR, a disc player, a stereo system, a lighting system, and other devices that provide functionality based on the output of system 100. In various embodiments, control signals are communicated between system 100 and display 165, speakers 175, or other peripheral devices 185 using signaling such as AV.Link, CEC, or other communication protocols that allow inter-device control with or without user intervention. Output devices may be communicatively coupled to system 100 via dedicated connections through respective interfaces 160, 170, and 180. Alternatively, output devices may be connected to system 100 via communication interface 150 using communication channel 190. Display 165 and speakers 175 may be integrated into a single unit with other components of system 100, for example, in an electronic device such as a television. In various embodiments, the display interface 160 includes a display driver, such as a timing controller (T Con) chip.

[0022] Display 165 and speakers 175 may alternatively be separate from one or more of the other components, for example, if the RF portion of input 105 is part of a separate set-top box. In various embodiments in which display 165 and speakers 175 are external components, output signals may be provided via dedicated output connections including, for example, an HDMI port, a USB port, or a COMP output.

[0023] It is believed that point cloud data may consume a large portion of network traffic, for example between cars connected via 5G networks and for immersive communications (VR / AR). Efficient representation formats are necessary for understanding and communicating point clouds. In particular, raw point cloud data needs to be properly organized and processed for world modeling and sensing purposes. Compression of raw point clouds is essential if storage and transmission of the data is required in relevant scenarios.

[0024] Furthermore, point clouds may represent successive scans of the same scene containing multiple moving objects. They are called dynamic point clouds, in comparison to static point clouds captured from a static scene or objects. Dynamic point clouds are typically organized into frames, with different frames captured at different times. Dynamic point clouds may require real-time or low latency processing and compression.

[0025] A 3D point cloud consists of a set of 3D points. Each point is defined by its 3D position (x,y,z). The 3D point cloud represents the geometry of an object or a scene. Optionally, each point may be further associated with some attributes, e.g., RGB color (r,g,b), normal (nx,ny,nz), and / or reflectance (r), depending on the application. In this work, we mainly focus on processing and compressing point cloud geometry.

[0026] The automotive industry and autonomous vehicles are areas where point clouds can be used. Autonomous vehicles should be able to "explore" their environment and make good driving decisions based on the reality of their immediate surroundings. Typical sensors like LiDAR generate (dynamic) point clouds that are used by perception engines. These point clouds are not intended to be seen by the human eye, they are usually sparse, not necessarily colored, and dynamic with a high capture frequency. They may have other attributes like reflectivity provided by LiDAR, since this attribute indicates the material of the sensed object and can help make the decision.

[0027] Virtual reality (VR) and immersive worlds are predicted by many as the future of 2D flat video. With VR and immersive worlds, the viewer is immersed in the environment all around them, as opposed to standard TV, where the viewer can only see the virtual world in front of them. There are several degrees of immersion, depending on the viewer's degrees of freedom in the environment. Point clouds are a good format candidate for delivering virtual reality worlds. Point clouds used in VR can be static or dynamic, and are typically of average size, e.g., at most a few million points at a time.

[0028] Point clouds can also be used for various purposes such as cultural heritage / architecture, where objects such as statues or buildings are scanned in 3D to share the spatial configuration of the object without sending or visiting the object. Also, when an object may be destroyed, for example a temple may be destroyed by an earthquake, point clouds can be used to ensure the preservation of the knowledge of the object. Such point clouds are usually static, color-coded and huge.

[0029] Another use case is in topography and cartography, where using 3D representations maps are not limited to flat surfaces and can include relief. Google Maps is a good example of a 3D map, but it uses meshes instead of point clouds. Nevertheless, point clouds can be the preferred data format for 3D maps, where such point clouds are usually static, color-coded, and large.

[0030] For the applications considered herein, world modeling and sensing via point clouds can be a useful technique to enable a machine to gain knowledge about the 3D world around it.

[0031] 3D point cloud data is essentially a set of discrete samples on the surface of an object or scene. To fully represent the real world with point samples, a huge number of points is actually required. For example, a typical VR immersive scene contains millions of points, while a point cloud typically contains hundreds of millions of points. Therefore, processing such large-scale point clouds is computationally expensive, especially for consumer devices with limited computing power, such as smartphones, tablets, and automobile navigation systems.

[0032] To perform processing or inference on point clouds, an efficient storage method is needed. To store and process the input point cloud at an affordable computational cost, one solution is to first downsample the point cloud, where the downsampled point cloud summarizes the geometry of the input point cloud while having very few points. The downsampled point cloud is then fed to a subsequent machine task for further consumption. However, further reduction in storage space can be achieved by converting the raw point cloud data (original or downsampled) into a bitstream through entropy encoding techniques for lossless compression. A better entropy model results in a smaller bitstream and therefore more efficient compression. In addition, the entropy model can also be paired with downstream tasks that allow the entropy encoder to maintain task-specific information while compressing.

[0033] In addition to lossless encoding, many scenarios call for lossy encoding for significantly improved compression ratios while maintaining the induced distortion under a certain quality level.

[0034] point cloud representation In recent years, learning-based point cloud compression has emerged. Existing methods can be mainly classified into three categories according to the point cloud representation format: point-based, voxel-based, and octree-based, as shown in Figures 2A, 2B, 2C, and 2D.

[0035] Point-Based Representation In a native point-based representation, such as that illustrated in FIG. 2A, points are specified directly by their coordinates in 3D and point sizes are not defined.

[0036] Octree-Based Representation In an octree-based representation as illustrated in FIG. 2B, the entire space, i.e., the 3D bounding box, is recursively partitioned into an octree structure to represent the point cloud. If the bounding box has a scale of 1×1×1, then the octree leaf nodes have a scale of 1 / (2 d )×1 / (2 d )×1 / (2 d ), where the index d represents the depth level in the octree, counting from 0.

[0037] In an octree decomposition tree, the root node covers the entire 3D bounding box. This 3D space is divided equally in every direction, i.e., x-direction, y-direction, and z-direction, resulting in eight voxels. For each voxel, if there is at least one point in it, the voxel is marked as occupied and represented by "1". Otherwise, it is marked as empty and represented by "0". The octree root node is described by an 8-bit integer that indicates the occupancy information.

[0038] To move from one octree level to the next, the space of each occupied voxel is further divided into eight child voxels in the same way. If occupied, each child voxel is further represented by an 8-bit integer. The division of occupied voxels continues until the final octree depth level is reached. The leaves of the octree ultimately represent the point cloud.

[0039] Octree-based point cloud compression algorithms aim to code the octree nodes using an entropy coder. For efficient entropy coding of the octree nodes, a probability distribution model is typically utilized to assign shorter symbols to octree node values ​​that occur with higher probability. A decoder can reconstruct the point cloud from the decoded octree nodes.

[0040] Voxel-based representation In a voxel-based representation such as that illustrated in Fig. 2C, 3D point coordinates are uniformly quantized by a quantization step, where each point corresponds to an occupied voxel with a size equal to the quantization step.

[0041] A naive voxel representation may not be efficient in memory usage due to large empty spaces. Next, a sparse voxel representation is introduced, in which occupied voxels are arranged in a sparse tensor format for efficient storage and processing. An example of a sparse voxel representation is depicted in FIG. 2D, where empty voxels do not consume any memory or storage.

[0042] Note that even in a sparse voxel representation, the voxels are still arranged in an organized manner.

[0043] Traditionally, there are three main categories of point cloud representations: Each category of point cloud representation has a specific encoding / decoding backbone, and how to integrate the different processing backbones into a unified processing or coding framework has not been fully explored.

[0044] In the following, we review some prior work in dealing with point cloud compression depending on the point cloud representation used.

[0045] Octree-based point cloud compression Deep entropy models refer to a category of learning-based approaches that use neural network modules to formulate context models and attempt to predict probability distributions of node occupancy values.

[0046] One deep entropy model is known as OctSqueeze, as described in a paper by Huang, Lila, et al. entitled "OctSqueeze: Octree-Structured Entropy Model for LiDAR Compression" in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2020. It utilizes ancestor nodes hierarchically, including parent nodes, grandparent nodes, etc. Three MLP-based (multi-layer perceptron) neural network modules are used. The first MLP module takes as input the context of the current node to generate output features. The second MLP module takes the output features of two such first MLP modules. One of the output features operates at the current octree depth level and the other at the previous octree depth level. The second MLP module also generates output features. The last third MLP module takes the output features of two such second MLP modules. One of the output features is from the current octree depth level, and the other is from the previous octree depth level. Finally, the third MLP module generates a predicted probability distribution. The estimated probability distribution is used for efficient arithmetic coding.

[0047] Another deep entropy model is known as VoxelContextNet, as described in the paper by Que, Zizheng, et al., entitled "VoxelContext-Net: An Octree based Framework for Point Cloud Compression," in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pp. 6042-6051, 2021. Unlike OctSqueeze, which uses ancestor nodes, VoxelContextNet adopts an approach that first analyzes the local surface shape and then uses spatially neighboring voxels to predict a probability distribution.

[0048] In our previous work, we proposed a self-supervised compression model consisting of an adaptive entropy coder operating on a tree-structured conditional entropy model. Information from local neighborhoods, as well as the global topology, is exploited from the octree structure.

[0049] Voxel-based point cloud compression Given that 2D convolution has been successfully employed in learning-based image compression (see the paper by Balle, Johannes, et al. entitled "Variational Image Compression with a Scale Hyperprior", arXiv preprint arXiv:1802.01436 (2018)), 3D convolution has been investigated for point cloud compression. For this purpose, the point cloud needs to be represented by voxels.

[0050] In regular 3D convolution, a 3D kernel is overlaid at every location specified by the stride step, regardless of whether the voxel is occupied or empty, as described in the paper by Wang, Jianqiang, et al., entitled "Learned Point Cloud Geometry Compression," arXiv preprint arXiv:1909.12037 (2019). To avoid the computation and memory consumption caused by empty voxels, we apply sparse convolution, as described in the paper by Wang, Jianqiang, et al., entitled "Multiscale Point Cloud Geometry Compression," 2021 Data Compression Conference (DCC), pp. 73-82. IEEE, 2021 (hereafter "Wang"), where point cloud voxels can be represented by a sparse tensor.

[0051] However, even in sparse convolution with sparse tensors, it is typically inefficient to apply large convolution kernels. This is because the representation of the convolution kernel is still a dense tensor, and more kernel parameters can be mapped to empty voxels when the kernel size is increasing. This fact can lead to inefficient training or even training failure. In Wang, the kernel size employed in sparse convolution is only 3 × 3 × 3. A kernel size that is too small can lead to a small receptive field and even less representative latent descriptors.

[0052] Point-based point cloud compression Learning-based point cloud compression has also been studied using native point cloud representations. Point-based architectures such as multi-layer perceptrons (MLPs) are used in frameworks such as those described in Yan, Wei et al., “Deep Autoencoder-Based Lossy Geometry Compression for Point Cloud,” arXiv preprint arXiv:1905.03691 (2019), and Gao, Linyao et al., “Point Cloud Geometry Compression Via Neural Graph Sampling,” In 2021 IEEE International Conference on Image Processing (ICIP), pp. 3373-3377, IEEE, 2021.

[0053] In Yan, the encoder employs a five-layer MLP-based PointNet module to extract feature vectors for a subsampled set of point clouds. The decoder employs a MLP-based LatentGAN module to reconstruct the point cloud. Due to the design of LatentGAN, the size of the reconstructed point cloud cannot be adjusted without updating the dimensions of the last layer of LatentGAN.

[0054] During encoding as described in Gao, two downsampling modules are first deployed using an approach called neural graph sampling (NGS) that uses point-based convolutions. Then, a PointNet-like module is applied to extract feature vectors that are entropy coded.

[0055] Although point-based architectures (MLP) may be more efficient from a training perspective, existing point-based architectures exhibit limited compression performance for point clouds compared to voxel-based or sparse convolution-based architectures.

[0056] In this work, we propose a hybrid coding framework that intends to fully exploit different types of point cloud compression approaches, each of which targets dealing with a specific bit level of the point coordinates.

[0057] Considering point clouds with high bit depth, i.e., D bits, we propose to use different coding strategies for different bit levels. Let index d represent the bit level or corresponding depth level of the octree decomposition. For the most significant bit or root level, d=0. Figure 3 is an illustrative diagram showing how the bits of a coordinate number with D=18 bits are indexed in this work.

[0058] We propose the following hybrid coding framework: a For d -1, a tree-based coding strategy is used, e.g., an octree-based coding method. No downsampling or abstraction is required. The next few bits, i.e., d = d a , ..., d b For d -1, there are a significant number of occupied voxels, and voxels are typically connected to each other. Therefore, voxel-based coding strategies, such as regular convolution or sparse convolution, are used. The last few bits, i.e., d = d b For D-1, ..., D-1, the points are very sparse and normal or sparse convolutions would fail due to lack of valid neighboring voxels or points. As a result, a point-based coding strategy using, for example, a multi-layer perceptron (MLP) is used. In the following, the proposed hybrid encoding and decoding process is detailed.

[0059] encoding On the encoder side, the bit-level parameters (d a ,d b ) is determined according to the characteristics of the input point cloud. The selected bit-level parameters (d a ,d b ) and the total bit depth D must be known to the decoder and are transmitted, for example, via the sequence parameter set, picture parameter set, or supplementary enhancement information (SEI) message.

[0060] FIG. 4 shows a block diagram implementing the proposed encoding procedure according to one embodiment. In particular, the “DB” block (410) determines the parameters (d a ,d b ) is determined by the parameter d. a can be empirically selected, for example, d a = 7. Parameter d a can be selected by checking the number of occupied voxels at the bit level, for example when that number starts to exceed a certain threshold. b To select a bit level, the voxel density (density of occupied voxels) is checked. The rate of increase in voxel density from the previous bit level is calculated. If the rate of increase starts to fall below a threshold, the current level is increased to d b Basically, level d b The remaining bits from do not increase the point density but mainly contribute to increasing the point precision. (d a ,d b ) parameter selection is typically done by a and d b -1. In addition, the "DB" block uses the parameter (d a ,d b) as input. For example, the configuration parameters may define the rate points at which the codec operates. Typically, the higher the bit rate, the higher the d a and d b The value of becomes higher.

[0061] The specific design of the "PN" (420), "VN" (430), and "ON" (440) blocks may differ from the example provided, but they are proposed to address three different bit levels. The "PN" block (420) processes the last few bits, i.e., D-1 and d b Since the points in this range are very sparse, it is proposed to use point-based feature extraction for the "PN" block, for example, mainly based on MLP network layers. Instead, the "VN" block (430) encodes the vectors d b -1 and d a It is proposed to use a voxel-based feature extraction module. The "ON" block is used to encode the bits between d a The first d between -1 and 0 a The three blocks are designed to code bits. For example, it is proposed to use tree-based coding for geometric information and arithmetic coding for related features. The three blocks correspond to three coding modes suitable for different bit levels. In the following, some exemplary implementations of the three blocks are presented.

[0062] The "PN" block consists of D-1 and d b An exemplary design of the “PN” block, which mainly consists of several MLP layers, is shown in FIG. 5. For the input point cloud PC, it is first quantized (510) by a quantization step Q, (d bA point cloud PCv with a bit depth of 1−1) bits and n points is output. A nearest neighbor module NN (520) then finds the k nearest neighbors of each point in PCv from PC, forming n point sets PCv,1, PCv,2, ..., PCv,n for each of the n points in PCv. These n point sets are then fed to a series of point-wise MLP layers (530) to extract point-wise features, followed by a global max pooling (540) and another series of MLP layers (550) to extract point-based features Fv={fv,1, fv,2, ..., fv,n} for each point in PCv. V The number of points in can be very close to the number of points in the input point cloud PC for the point cloud captured from the LiDAR sensor. In fact, a similar number of points is due to the high sparsity in the last few bits of such a point cloud. Both the point cloud PCv and the features Fv are output (Fv can be considered as attributes of the points in PCv). PCv is a feature vector with reduced precision (bit depth 0 to d b -1), and Fv represents the input point cloud PC at bit depth d b Note that the feature represents detailed information in the output PC V Each point in is now associated with a point-based feature extracted by a neural network module, "PN". With the PN module, the details from the original input PCs are now represented as point-wise features in PCv. Thus, PCv is a quantized point cloud, but contains detailed information from the PCs in the associated features (Fv).

[0063] Next, PC Vis fed as input to the "VN" block (430). Unlike the "PN" block, which operates on a point-based representation, the "VN" block operates on voxels, and thus PCv is transformed to be represented in a voxel grid with the corresponding features in Fv associated with the voxels as attributes. The "VN" block employs regular convolution or sparse convolution as the processing backbone. An example design of the "VN" block is shown in Figure 6. It is composed of several sub-blocks (610, 620, 630), each of which consists of a convolution layer for feature extraction, a downsampling function, and a ReLU activation function.

[0064] Note that in this example, the downsampling ratio of each downsampling function is set to 1 / 2. Note that each downsampling reduces the spatial resolution by half, which effectively reduces the bit depth by 1. Sub-blocks can use other types of activation functions, such as logistic sigmoid and tanh. Convolutional layer sizes (kernel size and number of output channels) can also vary.

[0065] The "VN" block specifies the bit depth as d a From d b Since we intend to code the difference (d b -d a Each sub-block also outputs more abstract features for the downsampled point cloud. Finally, the “VN” block extracts the lower resolution point cloud PC O and a set of abstract features Fo. These features are O The PCo can be seen as a more abstract version of the PCv at a reduced spatial resolution, and the PCo is sparser than the PCv. Furthermore, the PCo has a set of features Fo associated with each of its points. As a coarse version of the input point cloud PC, the PC O d a It only has a bit depth of 128 bits.

[0066] Next, PC O is provided as an input to the "ON" block 440. An example of the design of the "ON" block is shown in FIG. O The geometric positions of are losslessly coded into a bitstream using an octree-based encoder or another tree-based point cloud encoder. In one implementation, the "ON" block uses an adaptive arithmetic coder (720) to encode the octree node symbols generated by the octree splitter (710). In another embodiment, a deep entropy model may be utilized to promote the arithmetic coding efficiency of the octree coding. In yet another embodiment, a different tree structure may be applied, such as a QTBT tree (quarter tree and binary tree) or a KD tree. In addition, the PC O The features F associated with are quantized (730) and then also coded (740) into a bitstream using another adaptive arithmetic coder. In one embodiment, the coding of F may be facilitated by utilizing a factorized prior model or a hyper-prior model. In FIG. 7, two bitstreams are generated. Note that these two bitstreams may be multiplexed into one bitstream.

[0067] Decryption The proposed decoding procedure according to one embodiment is shown in Figure 8. This is the reverse procedure of the encoding. Corresponding to the encoder, the decoder has a total number of bits D and bit allocation positions d a and d b Then, the decoder obtains a parameter set including the total number of bits D and the bit allocation position d a and d b According to the above, we construct the decoder network and call the decoding process in the following order: - decoding the coarse point cloud PCo and its associated point-wise features using a tree-based decoding block; - Compile a coarse point cloud into a denser point cloud PC' V Then, we use the voxel-based decoding block to update the point-wise features. - Using a point-based decoding block to decode dense but low bit depth point clouds (PC's V ) coordinate precision is refined into a high bit depth point cloud PC'.

[0068] More generally, the point cloud PCo can be seen as a first reconstructed version of the point cloud, the point cloud PC' V can be seen as a second reconstructed version of the point cloud, and the point cloud PC' can be seen as a third reconstructed version of the point cloud. V Over time, the points become denser, and the second version, PC' V From version 1 to the third version PC', the accuracy of the point cloud coordinates is refined.

[0069] In this embodiment, the proposed decoder first receives a parameter set and calculates the parameters (d a ,d b ) and the total bit depth D. Then, the decoding process proceeds according to the analyzed parameters.

[0070] First, select the block "ON * " (810) uses tree-based decoding to decode point clouds O The first d a The block “ON” is reconstructed as shown in Figure 9. * 9 shows an exemplary design of the entropy decoder (910) and the octree de-splitter (920) for the PC. O It is used to reconstruct the "ON * " is also generated by the entropy decoder (930) and the inverse quantizer (940) in the PC O For each point in we also decode a set of pointwise features F'o.

[0071] PC O And the feature F'o is then added to the next block "VN * (820) as input to the block "VN *The structure of " is the parameter (d a ,d b ) is constructed based on the block “VN * ” is an exemplary design of the block “VN * "Inside (d b -d a ) sub-blocks (1010, 1020, 1030), each of which performs upsampling, voxel-based convolution, and ReLU activation function to progressively obtain a point cloud PC'v (a reconstruction of PCv) from PCo. In this example, the upsampling ratio of the upsampling function is set to 2. VN * Note that also outputs the features F'v associated with each point of PC'v. * " may utilize sparse convolutions.

[0072] PC' V Finally, add the block "PN * ” (830) to obtain the final reconstructed point cloud PC’. b Reconstruct the last few bits from D to D-1. * We propose to use a point-based architecture for the PN * 11 shows an exemplary design of a LatentGAN network for refining the accuracy of the points. In one embodiment, this can be implemented by a LatentGAN network or several MLP layers to refine the accuracy of the points. Assuming that PC'v has n' points, the MLP layer (1110) takes as input the features F'v and calculates a set of offset coordinates {s1, s2, ..., s n’}. By adding (1120, 1130, 1140) the offset coordinates with the associated points in PC'v, a reconstructed point cloud PC' having n' points is obtained. In another embodiment, the MLP layer may be configured to output two or more offset points for each point in PC'v. Then, by adding all offsets with the associated points in PC'v, a reconstructed point cloud PC' having more than n' points may be obtained. In this case, the reconstructed point cloud PC' not only has a higher accuracy than PC'v, but also has a higher density than PC'v.

[0073] A generalized hybrid coding framework Above we have described a hybrid coding framework based on three different ranges of bit levels. In the following, we present some extended hybrid coding frameworks.

[0074] Space-level coding In one embodiment, we propose to generalize the three sets of bit levels to three spatial resolutions, each corresponding to a different quantization step on the input point cloud.

[0075] We assume that the input points are still represented by D-bit integers. a and d b The bit levels are respectively two quantization steps Q a =pow(2,Dd a -1) and Q b =pow(2,Dd b -1). Using the generalized hybrid coding framework, a and Q b can be chosen more flexibly and does not necessarily have to be a power of two.

[0076] Reduced Hybrid Coding In one embodiment, it is not necessary to include all three coding modules. Instead, it may have only two coding modules. For example, if the target point cloud is very dense, only the "ON" block and the "VN" block may be needed, as illustrated in FIG. 12. When using the VN block as illustrated in FIG. 6, Fv can be considered to be filled with 1s. In this case, the decoder needs two parameters d a On the other hand, if the target point cloud is sparse, then only the "ON" and "PN" blocks may be used, the "PN" blocks being particularly useful for handling sparsity in the input point cloud. Similarly, in this case, the decoder only needs to signal two parameters d a and D need only be signaled.

[0077] Complexity Scalable Decoding When the bitstream is provided to a decoder, the decoder can decide at which step to stop the decoding process. Early stopping of the decoding results in a coarser point cloud, but consumes less computational cost for decoding.

[0078] FIG. 14 illustrates a method of encoding point cloud data according to one embodiment. In this embodiment, for a point cloud or a portion of a point cloud to be encoded, the encoder encodes (1410) a first set of data corresponding to a first subset of the bit level of the point cloud data using a first encoding strategy. The encoder then encodes (1420) a second set of data corresponding to a second subset of the bit level of the point cloud data using a second encoding strategy. The point cloud can be encoded with only two sets of data, where the first set of data includes the most significant bits and the second set of data includes the remaining bits. The first and second encoding strategies can be octree-based encoding and point-based network, respectively.

[0079] The encoder may split the point cloud into more data sets. When the point cloud data is split into three data sets, the first set of data includes the most significant bits, the third set of data includes the least significant bits, and the second set of data includes the remaining bits. The encoder encodes the third set of data (1430) using a third encoding strategy. As described above, the first, second, and third encoding strategies can be octree-based encoding, voxel-based encoding, and point-based network, respectively.

[0080] FIG. 15 illustrates a method of decoding point cloud data according to one embodiment. In this embodiment, for a point cloud or a portion of a point cloud to be decoded, the decoder uses a first decoding strategy to decode a first set of data corresponding to a bit-level first subset of the point cloud data (1510). The decoder then uses a second decoding strategy to decode a second set of data corresponding to a bit-level second subset of the point cloud data (1520). The point cloud can consist of only two sets of data, where the first set of data includes the most significant bits and the second set of data includes the remaining bits. The first and second decoding strategies can be octree-based decoding and point-based network, respectively.

[0081] The point cloud may include more sets of data. When the point cloud data is divided into three parts, the first set of data includes the most significant bits, the third set of data includes the least significant bits, and the second set of data includes the remaining bits. The decoder decodes the third set of data (1530) using a third decoding strategy. As described above, the first, second, and third decoding strategies can be octree-based decoding, voxel-based decoding, and point-based network, respectively.

[0082] Various numerical values ​​are used in this application. The specific values ​​are for illustrative purposes and the described aspects are not limited to these specific values.

[0083] Various methods are described herein, each of which includes one or more steps or actions for achieving the described method. The order and / or use of specific steps and / or actions may be modified or combined, unless a specific order of steps or actions is required for proper operation of the method. In addition, terms such as "first", "second", etc. may be used in various embodiments to modify elements, components, steps, operations, etc., such as, for example, "first decoding" and "second decoding". The use of such terms does not imply a modified ordering of operations, unless specifically required. Thus, in this embodiment, the first decoding need not be performed before the second decoding, but may occur, for example, before, during, or during an overlapping time with the second decoding.

[0084] The implementations and aspects described herein may be implemented in, for example, a method or process, an apparatus, a software program, a data stream, or a signal. Even if discussed in the context of only a single implementation form (e.g., discussed only as a method), the implementation of the discussed features may also be implemented in other forms (e.g., an apparatus or a program). An apparatus may be implemented in, for example, appropriate hardware, software, and firmware. The method may be implemented in an apparatus such as a processor, which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include, for example, communication devices such as computers, mobile phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate communication of information between end users.

[0085] References to "one embodiment" or "an embodiment" or "one implementation" or "an implementation," as well as other variations thereof, mean that a particular feature, structure, characteristic, etc. described in connection with that embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment" or "an embodiment," or "in one implementation," as well as other variations thereof, appearing in various places throughout this application are not necessarily all referring to the same embodiment.

[0086] Additionally, the application may refer to "determining" various information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from memory.

[0087] Additionally, the application may refer to "accessing" various information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0088] Additionally, the application may refer to "receiving" various information. Receiving, like "accessing," is intended to be a broad term. Receiving information may include, for example, one or more of accessing information or retrieving information (e.g., from a memory). Furthermore, "receiving" typically involves some form of activity, such as storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0089] For example, in the case of "A / B," "A and / or B," and "at least one of A and B," it should be understood that the use of any of the following " / ," "and / or," and "at least one of" is intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of both alternatives (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such language is intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of only the third listed alternative (C), or the selection of only the first and second listed alternatives (A and B), or the selection of only the first and third listed alternatives (A and C), or the selection of only the second and third listed alternatives (B and C), or the selection of all three alternatives (A and B and C). This may be expanded as many times as the number of items listed, as would be apparent to one of ordinary skill in the art and related arts.

[0090] As will be apparent to one of ordinary skill in the art, implementations may generate a variety of signals formatted to carry information that may be, for example, stored or transmitted. Information may include, for example, instructions for performing a method or data generated by one of the described implementations. For example, a signal may be formatted to carry a bit stream of the described embodiments. Such a signal may be formatted, for example, as an electromagnetic wave (e.g., using a radio frequency portion of the spectrum) or as a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier wave with the encoded data stream. The information that the signal carries may be, for example, analog information or digital information. The signal may be transmitted by a variety of different wired or wireless links, as is known. The signal may be stored in a processor-readable medium.

Claims

1. 1. A method of decoding point cloud data for a point cloud, comprising: Decoding a first set of data using a first decoding strategy, the first set of data corresponding to a bit-level first subset of the point cloud data; and decoding a second set of data using a second decoding strategy, the second set of data corresponding to a second bit-level subset of the point cloud data.

2. 2. The method of claim 1 , wherein the first set of data corresponds to a first version of the point cloud and the second set of data corresponds to a second version of the point cloud, the second version of the point cloud being denser than the first version of the point cloud.

3. The first subset of bit levels comprises bit levels 0 to d ａ −1, and the second subset of bit levels is bit level d ａ ~d ｂ −1, bit 0 represents the most significant bit, and d ａ and d ｂ is an integer, and d ａ <d ｂ The method according to claim 1 or 2,

4. d ｂ 4. The method of claim 3, wherein: D is equal to D, where D represents a total bit depth of the point cloud data, and bit D-1 represents the least significant bit.

5. The method according to any one of claims 1 to 4, wherein the first decoding strategy is a reversible decoding strategy.

6. The method according to any one of claims 1 to 5, wherein the second decoding strategy is a voxel-based decoding strategy or a point-based decoding strategy.

7. The voxel-based decoding strategy, 7. The method of claim 6, comprising a plurality of sub-blocks, each sub-block of the plurality of sub-blocks comprising an upsampling function, a convolutional layer, and an activation function, and a number of sub-blocks in the plurality of sub-blocks is the same as a number of bit levels in the second subset of bit levels.

8. A method according to any one of claims 1 to 7, wherein an output of the first decoding strategy is used as an input to the second decoding strategy.

9. 9. The method of claim 1, further comprising: decoding a third set of data using a third decoding strategy, the third set of data corresponding to a bit-level third subset of the point cloud data.

10. 10. The method of claim 9, wherein the third set of data corresponds to a third version of the point cloud, the third version of the point cloud data refining coordinate accuracy of the second version of the point cloud.

11. 11. The method of claim 9 or 10, further comprising determining that the third set of data is to be decoded, wherein the decoding of the third set of data is performed only in response to the third set of data being to be decoded.

12. The third subset of bit levels is bit level d ｂ 12. The method according to claim 9, wherein bit D-1 represents the least significant bit, D representing a total bit depth of the point cloud data.

13. The method according to any one of claims 9 to 12, wherein the third decoding strategy is a point-based decoding strategy.

14. The point-based decoding strategy, Obtaining an initial version of a set of features and a reconstructed point cloud; generating offset coordinates for each point in the initial version of a point cloud reconstructed from the set of features; and refining coordinates of the initial version of the reconstructed point cloud using the offset coordinates.

15. The method of claim 14, wherein a MLP (Multi-Layer Perceptron) layer or a LatentGAN network is used to generate the offset coordinates.

16. A method according to any one of claims 9 to 15, wherein an output of the second decoding strategy is used as an input to the third decoding strategy.

17. D, d ａ and d ｂ The method of any one of claims 3 to 16, further comprising decoding information indicative of values ​​for one or more of:

18. 18. The method of claim 1, further comprising determining that a second set of data is to be decoded, wherein the decoding of the second set is performed only in response to the second set of data being decoded.

19. 1. A method of encoding point cloud data for a point cloud, comprising the steps of: encoding a first set of data using a first encoding strategy, the first set of data corresponding to a bit-level first subset of the point cloud data; encoding a second set of data using a second encoding strategy, the second set of data corresponding to a second bit-level subset of the point cloud data.

20. 20. The method of claim 19, wherein the first set of data corresponds to a first version of the point cloud and the second set of data corresponds to a second version of the point cloud, the second version of the point cloud being denser than the first version of the point cloud.

21. The first subset of bit levels comprises bit levels 0 to d ａ −1, and the second subset of bit levels is bit level d ａ ~d ｂ −1, bit 0 represents the most significant bit, and d ａ and d ｂ is an integer, and d ａ <d ｂ The method according to claim 19 or 20, wherein

22. d ａ 22. The method of claim 21, wherein the value of is selected based on at least one of a number of occupied voxels at the bit level and a configuration parameter.

23. d ｂ 22. The method of claim 21 , wherein: D is equal to D, where D represents a total bit depth of the point cloud data, and where bit D−1 represents the least significant bit.

24. d ｂ The method according to any one of claims 20 to 23, wherein the value of x is selected based on a voxel density increase rate or a configuration parameter at the bit level.

25. The method according to any one of claims 19 to 24, wherein the first encoding strategy is a lossless encoding strategy.

26. The method according to any one of claims 19 to 25, wherein the second encoding strategy is a voxel-based encoding strategy.

27. The voxel-based coding strategy comprises:

27. The method of claim 26, comprising a plurality of sub-blocks, each sub-block of the plurality of sub-blocks comprising a convolutional layer, a downsampling function, and an activation function, and a number of sub-blocks in the plurality of sub-blocks is the same as a number of bit levels in the second subset of bit levels.

28. A method according to any one of claims 19 to 27, wherein an output of the second encoding strategy is used as an input to the first encoding strategy.

29. 29. The method of any one of claims 19 to 28, further comprising encoding a third set of data using a third encoding strategy, the third set of data corresponding to a bit-level third subset of the point cloud data.

30. 30. The method of claim 29, wherein the third set of data corresponds to a third version of the point cloud, the third version of the point cloud data having greater coordinate precision than the second version of the points.

31. The third subset of bit levels is bit level d ｂ 31. The method of claim 29 or 30, wherein bit D-1 represents the least significant bit, with D representing a total bit depth of the point cloud data.

32. The method according to any one of claims 29 to 31, wherein the third encoding strategy is a point-based encoding strategy.

33. The point-based encoding strategy comprises: d ｂ quantizing the point cloud data of the point cloud to form another point cloud at a bit depth of -1; and for each point in the other point cloud, extracting a respective feature using its k nearest neighbors in the point cloud.

34. 34. The method according to claim 32 or 33, wherein a MLP (Multi-Layer Perceptron) layer is used to extract the respective features.

35. D, d ａ and d ｂ The method of any one of claims 21 to 34, further comprising encoding information indicative of values ​​for one or more of:

36. A method according to any one of claims 29 to 35, wherein an output of the third encoding strategy is used as an input to the second encoding strategy.

37. An apparatus comprising one or more processors and at least one memory coupled to said one or more processors, said one or more processors configured to perform the method of any one of claims 1 to 36.

38. A signal comprising video data formed by carrying out a method according to any one of claims 19 to 36.

39. A computer readable storage medium having stored thereon instructions for encoding or decoding a point cloud according to the method of any one of the preceding claims.

Images