Learning-based image coding for 3D point cloud data, with each color component of a 2d image corresponding to a dimension of 3D cartesian coordinates of a point position

Abstract

This work concerns the compression of point cloud data via an alternate raster image representation, specifically for point clouds acquired from a LiDAR sensor. In one implementation, we propose an image-based learned point cloud compression architecture that makes use of a 3D to 2D image projection module called GridAlign. Our proposal resolves the issues with the existing range image-based point cloud compression through a one-to-one assignment of the original cartesian coordinates onto a pre-defined grid. Our proposed architecture can handle high bit depth LiDAR data and can also operate in a block-based fashion.

Description

[0001] LEARNING-BASED IMAGE CODING FOR 3D POINT CLOUD DATA, WITH EACH COLOR COMPONENT OF A 2D IMAGE CORRESPONDING TO A DIMENSION OF 3D CARTESIAN COORDINATES OF A POINT POSITION

[0002] TECHNICAL FIELD

[0003] [1] The present embodiments generally relate to a method and an apparatus for point cloud compression and processing.

[0004] BACKGROUND

[0005] [2] 3D point cloud is a universal data format across several business domains, e.g., from autonomous driving, robotics, augmented reality / virtual reality (AR / VR), civil engineering, computer graphics, to the animation / movie industry. 3D LiDAR (Light Detection and Ranging) sensors have been widely deployed in self-driving cars. Affordable LiDAR sensors are released from Velodyne Velabit, Apple iPad Pro 2020 and Intel RealSense LiDAR camera L515. With great advances in sensing technologies, 3D point cloud data has become more practical than ever and is expected to be an ultimate enabler in the applications mentioned.

[0006] [3] Point cloud data is also believed to consume a large portion of network traffic, e.g., among connected cars over 5G network, and immersive communications (VR / AR). Efficient representation formats are necessary for point cloud understanding and communication. In particular, raw point cloud data need to be properly organized and processed for the purposes of world modeling and sensing. Compression on raw point clouds is essential when storage and transmission of the data are required in the related scenarios.

[0007] [4] Furthermore, point clouds may represent a sequential scan of the same scene, which contains multiple moving objects. They are called dynamic point clouds as compared to static point clouds captured from a static scene or static objects. Dynamic point clouds are typically organized into frames, with different frames being captured at different time. Dynamic point clouds may require the processing and compression to be in real-time or with low delay.

[0008] SUMMARY

[0009] [5] According to an embodiment, a method of decoding point cloud data is presented, comprising: decoding at least a 2D image, wherein each color component of said 2D image corresponds to a dimension of 3D cartesian coordinates of point cloud data of a 3D point cloud; obtaining said 3D cartesian coordinates of said point cloud data of said 3D point cloud from said color components of said at least a 2D image; and reconstructing said 3D point cloud from said 3D cartesian coordinates of said 3D point cloud.

[0010] [6] According to another embodiment, a method of encoding point cloud data is presented, comprising: obtaining said point cloud data with 3D cartesian coordinates; mapping said 3D cartesian coordinates to cartesian coordinates on a 2D grid to form a 2D image, wherein a respective color component of said 2D image corresponds to a dimension of said 3D cartesian coordinates; and encoding said 2D image.

[0011] [7] According to another embodiment, an apparatus for decoding point cloud data is presented, comprising one or more processors and at least one memory coupled to said one or more processors, wherein said one or more processors are configured to decode at least a 2D image, wherein each color component of said 2D image corresponds to a dimension of 3D cartesian coordinates of point cloud data of a 3D point cloud; obtain said 3D cartesian coordinates of said point cloud data of said 3D point cloud from said color components of said at least a 2D image; and reconstruct said 3D point cloud from said 3D cartesian coordinates of said 3D point cloud.

[0012] [8] According to another embodiment, an apparatus for encoding point cloud data is presented, comprising one or more processors and at least one memory coupled to said one or more processors, wherein said one or more processors are configured to obtain said point cloud data with 3D cartesian coordinates; map said 3D cartesian coordinates to cartesian coordinates on a 2D grid to form a 2D image, wherein a respective color component of said 2D image corresponds to a dimension of said 3D cartesian coordinates; and encoding said 2D image.

[0013] [9] One or more embodiments also provide a computer program comprising instructions which when executed by one or more processors cause the one or more processors to perform the encoding method or decoding method according to any of the embodiments described above. One or more of the present embodiments also provide a computer readable storage medium having stored thereon instructions for encoding or decoding point cloud data according to the methods described above.

[0014]

[0010] One or more embodiments also provide a computer readable storage medium having stored thereon video data generated according to the methods described above. One or more embodiments also provide a method and apparatus for transmitting or receiving the video data generated according to the methods described above.

[0015] BRIEF DESCRIPTION OF THE DRAWINGS

[0016]

[0011] FIG. 1 illustrates a block diagram of a system within which aspects of the present embodiments may be implemented.

[0017]

[0012] FIG. 2 illustrates a diagram of an image-based point cloud compression system where Proj2D is projection onto spherical coordinates.

[0018]

[0013] FIG. 3 illustrates a diagram of an image-based point cloud compression system where SphQ is projection onto spherical coordinates.

[0019]

[0014] FIG. 4 illustrates projection of 3D cartesian coordinates into the spherical domain coordinates.

[0020]

[0015] FIG. 5 illustrates a diagram of an image-based point cloud compression system, according to an embodiment.

[0021]

[0016] FIG. 6A illustrates a diagram of the GridAlign module, and FIG. 6B illustrates the data flow for the GridAlign module, according to an embodiment.

[0022]

[0017] FIG. 7 illustrates a diagram of an image-based point cloud compression system, where MSB and LSB are coded separately, according to an embodiment.

[0023]

[0018] FIG. 8 illustrates a diagram of an image-based point cloud compression system, where coding of the LSB depends on MSB, according to an embodiment.

[0024]

[0019] FIG. 9 illustrates an example of an image-based point cloud compression system, according to an embodiment.

[0025]

[0020] FIG. 10 and FIG. 11 illustrate an encoder and a decoder, respectively, in an architecture of an image-based point cloud compression system, according to an embodiment.

[0026]

[0021] FIG. 12 illustrates a diagram of a point cloud compression system where blocks of the point cloud are compressed via image-based compression, according to an embodiment.

[0027] DETAILED DESCRIPTION

[0028]

[0022] FIG. 1 illustrates a block diagram of an example of a system in which various aspects and embodiments can be implemented. System 100 may be embodied as a device including the various components described below and is configured to perform one or more of the aspects described in this application. Examples of such devices, include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. Elements of system 100, singly or in combination, may be embodied in a single integrated circuit, multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 100 are distributed across multiple Ics and / or discrete components. In various embodiments, the system 100 is communicatively coupled to other systems, or to other electronic devices, via, for example, a communications bus or through dedicated input and / or output ports. In various embodiments, the system 100 is configured to implement one or more of the aspects described in this application.

[0029] [231 The system 100 includes at least one processor 110 configured to execute instructions loaded therein for implementing, for example, the various aspects described in this application. Processor 110 may include embedded memory, input output interface, and various other circuitries as known in the art. The system 100 includes at least one memory 120 (e.g., a volatile memory device, and / or a non-volatile memory device). System 100 includes a storage device 140, which may include non-volatile memory and / or volatile memory, including, but not limited to, EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic disk drive, and / or optical disk drive. The storage device 140 may include an internal storage device, an attached storage device, and / or a network accessible storage device, as non-limiting examples.

[0030]

[0024] System 100 includes an encoder / decoder module 130 configured, for example, to process data to provide an encoded video or decoded video, and the encoder / decoder module 130 may include its own processor and memory. The encoder / decoder module 130 represents module(s) that may be included in a device to perform the encoding and / or decoding functions. As is known, a device may include one or both of the encoding and decoding modules. Additionally, encoder / decoder module 130 may be implemented as a separate element of system 100 or may be incorporated within processor 110 as a combination of hardware and software as known to those skilled in the art.

[0031]

[0025] Program code to be loaded onto processor 110 or encoder / decoder 130 to perform the various aspects described in this application may be stored in storage device 140 and subsequently loaded onto memory 120 for execution by processor 1 10. In accordance with various embodiments, one or more of processor 110, memory 120, storage device 140, and encoder / decoder module 130 may store one or more of various items during the performance of the processes described in this application. Such stored items may include, but are not limited to, the input video, the decoded video or portions of the decoded video, the bitstream, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.

[0032] [26J In several embodiments, memory inside of the processor 110 and / or the encoder / decoder module 130 is used to store instructions and to provide working memory for processing that is needed during encoding or decoding. In other embodiments, however, a memory external to the processing device (for example, the processing device may be either the processor 1 10 or the encoder / decoder module 130) is used for one or more of these functions. The external memory may be the memory 120 and / or the storage device 140, for example, a dynamic volatile memory and / or a non-volatile flash memory. In several embodiments, an external non-volatile flash memory is used to store the operating system of a television. In at least one embodiment, a fast external dynamic volatile memory such as a RAM is used as working memory for video coding and decoding operations, such as for MPEG-2, JPEG Pleno, MPEG-I, V-DMC, HEVC, or VVC.

[0033]

[0027] The input to the elements of system 100 may be provided through various input devices as indicated in block 105. Such input devices include, but are not limited to, (i) an RF portion that receives an RF signal transmitted, for example, over the air by a broadcaster, (ii) a Composite input terminal, (iii) a USB input terminal, and / or (iv) an HDMI input terminal.

[0034]

[0028] In various embodiments, the input devices of block 105 have associated respective input processing elements as known in the art. For example, the RF portion may be associated with elements suitable for (i) selecting a desired frequency (also referred to as selecting a signal, or band-limiting a signal to a band of frequencies), (ii) down converting the selected signal, (iii) bandlimiting again to a narrower band of frequencies to select (for example) a signal frequency band which may be referred to as a channel in certain embodiments, (iv) demodulating the down converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select the desired stream of data packets. The RF portion of various embodiments includes one or more elements to perform these functions, for example, frequency selectors, signal selectors, band- limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF portion may include a tuner that performs various of these functions, including, for example, down converting the received signal to a lower frequency (for example, an intermediate frequency or a near-baseband frequency) or to baseband. In one set-top box embodiment, the RF portion and its associated input processing element receives an RF signal transmitted over a wired (for example, cable) medium, and performs frequency selection by filtering, down converting, and filtering again to a desired frequency band. Various embodiments rearrange the order of the above-described (and other) elements, remove some of these elements, and / or add other elements performing similar or different functions. Adding elements may include inserting elements in between existing elements, for example, inserting amplifiers and an analog- to-digital converter. In various embodiments, the RF portion includes an antenna.

[0035]

[0029] Additionally, the USB and / or HDMI terminals may include respective interface processors for connecting system 100 to other electronic devices across USB and / or HDMI connections. It is to be understood that various aspects of input processing, for example, Reed-Solomon error correction, may be implemented, for example, within a separate input processing IC or within processor 110 as necessary. Similarly, aspects of USB or HDMI interface processing may be implemented within separate interface Ics or within processor 110 as necessary. The demodulated, error corrected, and demultiplexed stream is provided to various processing elements, including, for example, processor 110, and encoder / decoder 130 operating in combination with the memory and storage elements to process the datastream as necessary for presentation on an output device.

[0036]

[0030] Various elements of system 100 may be provided within an integrated housing, Within the integrated housing, the various elements may be interconnected and transmit data therebetween using suitable connection arrangement 115, for example, an internal bus as known in the art, including the I2C bus, wiring, and printed circuit boards.

[0037]

[0031] The system 100 includes communication interface 150 that enables communication with other devices via communication channel 190. The communication interface 150 may include, but is not limited to, a transceiver configured to transmit and to receive data over communication channel 190. The communication interface 150 may include, but is not limited to, a modem or network card and the communication channel 190 may be implemented, for example, within a wired and / or a wireless medium.

[0032] Data is streamed to the system 100, in various embodiments, using a Wi-Fi network such as IEEE 802. 11. The Wi-Fi signal of these embodiments is received over the communications channel 190 and the communications interface 150 which are adapted for Wi-Fi communications. The communications channel 190 of these embodiments is typically connected to an access point or router that provides access to outside networks including the Internet for allowing streaming applications and other over-the-top communications. Other embodiments provide streamed data to the system 100 using a set-top box that delivers the data over the HDMI connection of the input block 105. Still other embodiments provide streamed data to the system 100 using the RF connection of the input block 105.

[0038]

[0033] The system 100 may provide an output signal to various output devices, including a display 165, speakers 175, and other peripheral devices 185. The other peripheral devices 185 include, in various examples of embodiments, one or more of a stand-alone DVR, a disk player, a stereo system, a lighting system, and other devices that provide a function based on the output of the system 100. In various embodiments, control signals are communicated between the system 100 and the display 165, speakers 175, or other peripheral devices 185 using signaling such as AV. Link, CEC, or other communications protocols that enable device-to-device control with or without user intervention. The output devices may be communicatively coupled to system 100 via dedicated connections through respective interfaces 160, 170, and 180. Alternatively, the output devices may be connected to system 100 using the communications channel 190 via the communications interface 150. The display 165 and speakers 175 may be integrated in a single unit with the other components of system 100 in an electronic device, for example, a television. In various embodiments, the display interface 160 includes a display driver, for example, a timing controller (T Con) chip.

[0039]

[0034] The display 165 and speaker 175 may alternatively be separate from one or more of the other components, for example, if the RF portion of input 105 is part of a separate set-top box. In various embodiments in which the display 165 and speakers 175 are external components, the output signal may be provided via dedicated output connections, including, for example, HDMI ports, USB ports, or COMP outputs.

[0040]

[0035] Point Cloud Data Use Cases

[0041]

[0036] The automotive industry and autonomous car are domains in which point clouds may be used. Autonomous cars should be able to “probe” their environment to make good driving decisions based on the reality of their immediate surroundings. Typical sensors like LiDARs produce (dynamic) point clouds that are used by the perception engine. These point clouds are not intended to be viewed by human eyes and they are typically sparse, not necessarily colored, and dynamic with a high frequency of capture. They may have other attributes like the reflectance ratio provided by the LiDAR as this attribute is indicative of the material of the sensed object and may help in making a decision.

[0042] [37J Virtual Reality (VR) and immersive worlds are foreseen by many as the future of 2D video. The basic idea is to immerse the viewers in an environment all around them as opposed to the standard TV where they can only look at the virtual world in the front. There are several gradations in the immersivity depending on the freedom of the viewer in the environment. Point cloud is a good format candidate to distribute VR contents. They may be static or dynamic and are typically of moderate size, no more than millions of points at a time.

[0043]

[0038] Point clouds may be also used for various purposes such as culture heritage / building in which objects like statues or buildings are scanned in 3D in order to share the spatial configuration of the object without moving or visiting them. Also, it is a way to ensure preserving the knowledge of the object in case it may be destroyed, for instance, a temple by an earthquake. Such point clouds are typically static, colored, and huge.

[0044]

[0039] Another use case is in topography and cartography in which using 3D representations, maps are not limited to the plane and may include the relief. Google Maps is now a good example of 3D maps but uses meshes instead of point clouds. Nevertheless, point clouds may be a suitable data format for 3D maps and such point clouds are typically static, colored, and huge.

[0045]

[0040] World modeling and sensing via point clouds could be an essential technology to allow machines to gain knowledge about the 3D world around them, which is crucial for the applications discussed herein.

[0046]

[0041] 3D point cloud data are essentially discrete samples on the surfaces of objects or scenes. To fully represent the real world with point samples, in practice it requires a huge number of points. For instance, a typical VR immersive scene contains millions of points, while point clouds typically contain hundreds of millions of points. Therefore, the processing of such large-scale point clouds is computationally expensive, especially for consumer devices, e.g., smartphone, tablet, and automotive navigation system, that have limited computational power.

[0047]

[0042] The first step for any processing or inference on the point cloud is to have efficient storage methodologies. To store and process the input point cloud with affordable computational cost, one solution is to down-sample it first, where the down-sampled point cloud summarizes the geometry of the input point cloud while having much fewer points. The down-sampled point cloud is then fed to the subsequent machine task for further consumption. However, further reduction in storage space can be achieved by converting the raw point cloud data (original or downsampled) into a bitstream through either entropy coding techniques for lossless compression or feature map coding techniques for lossy compression.

[0048]

[0043] There are three main approaches for point cloud compression, namely: octree-based compression, voxel-based compression, and image-based compression. Octree-based compression first represents the point cloud as an occupancy tree and then losslessly compresses the tree structure up to a desirable depth based on the context of each node. In contrast, both voxel- and image-based compression approaches extract features from the voxel and image representations (respectively) of the point cloud, and then encode a quantized version of these features into a bitstream using entropy coding.

[0049]

[0044] A typical diagram for image-based point cloud compression is depicted in FIG. 2. As can be seen from the figure, the first step in image-based point cloud compression is to project (210) a 3D point cloud (Xc) onto a 2D image (X2D). Afterwards, the 2D image can be encoded (220) and decoded (230) using any image encoder-decoder pair. Finally, the decoded 2D image (X2D) is projected (240) back to 3D domain to obtain a reconstructed 3D point cloud (Xc).

[0050]

[0045] A traditional well-known way to achieve the projection is via projection (310) of the 3D cartesian coordinates (Xc) into the spherical domain coordinates (XSq), as shown in FIG. 3. Then XSqcan be encoded (320) and decoded (330) using an image encoder-decoder pair. Finally, decoded XSqis projected (340) back to 3D domain to obtain a reconstructed 3D point cloud (Xc).

[0051]

[0046] From FIG. 4, we can see the SphQ module (310) which consists of two steps. In the first step (410), the cartesian coordinates ( Xc) are converted to their corresponding spherical coordinates (Xs) which are azimuth, elevation and range / depth values. Then, in the second step only the elevation and azimuth angles are quantized to a desired grid resolution followed by placing (420) the respective range / depth values on the quantized grid coordinates to form XSq. This conversion is usually lossy in nature since (1) some precision of elevation and azimuth is lost in quantization, and (2) some depth values are also lost depending on the specified resolution of the grid, i.e., quantization of the elevation and azimuth (e.g., two points in Xsare quantized into the same point and one depth of these two points is covered therefore lost). It should be noted here Proj3D module in this case takes the decoded range values, along with the quantized elevation (rows) and azimuth (columns), to convert from spherical coordinates back to 3D cartesian coordinates. In this work, we focus on a better image-based representation of the point cloud which can circumvent both issues and can directly project the cartesian coordinates onto an image without any loss of information.

[0052]

[0047] FIG. 5 illustrates an overall diagram of our proposed architecture for image-based point cloud compression system, according to an embodiment. Compared with the image-based compression systems described briefly above, we can see that a contribution lies in the projection of the 3D point cloud to the 2D image. The proposed GridAlign module (510) places the cartesian coordinates (Xc) directly onto an image grid (XCg) and then aims to compress the grid aligned cartesian coordinates instead of some projected coordinates (e.g., depth values in range image compression). A typical image compression pipeline, either learning-based or non-learning based, can be used to operate on the output of our proposed GridAlign module to compress (520) and decompress (530) the original point cloud. Then, the role of Proj3D (540) is to just extract the three color components from each pixel of the decoded image (corresponding to x, y, and z coordinates) and place them in a list of 3D point locations.

[0053]

[0048] Image Generation Module

[0054]

[0049] We now describe the novel image generation, i.e., GridAlign module shown in FIG. 6A. As we can see, the first step (610) in the GridAlign module is to obtain a spherical domain projection (XSq) of the cartesian coordinates (Xc). Afterwards, these spherical coordinates (elevation and azimuth coordinates) are one-to-one matched to a grid (630) using a Grid Matching module (620). Through the matching process, each spherical coordinate is associated with a grid point and a matching index representing grid point index is obtained (640). The grid’s size is typically the same or larger than the number of points in the point cloud to assure that no points are lost in the conversion. Finally, using the matching index, the corresponding cartesian coordinates are assigned (650) to the respective grid points. This way all cartesian coordinates are one-to-one matched to the grid points while possibly having some empty grid points (which carry zero values by default). We have thus effectively converted x-, y- and z-coordinates (Xc) of a 3D cartesian point cloud to a 2D image (XCfJ) with 3 channels (color components), as shown in FIG. 6A, for example, the x-, y-, and z- coordinates become the R, G, and B components for the 2D image. That is, each dimension of the cartesian coordinates of the 3D point cloud becomes a color component of the 2D image.

[0055]

[0050] A mathematical representation is also shown in FIG. 6B where a 3D point ptin the input point cloud (Xc) is converted to spherical coordinates st(in XS£ / ) from which the elevation angle 0, is converted into a laser index lt(the number of angles equal to the number of lasers so that the laser index can be used in space of laser angles) and the azimuth angle (p is quantized by a factor qn to produce mt, where q is chosen depending on the required resolution. Then, based on the results of grid matching (620), all st= [f;,;] are one-to-one mapped to the pre-defined grid points Qj = [pj, qj\ (in Grid, 630) to produce mapping from point index i in the input cloud to grid point index in the grid j i -> j (640) which is used to obtain the 2D image (XC£,) as [x^y,, z is assigned to grid point [pj, Qy],

[0056]

[0051] Grid Matching

[0057]

[0052] The Grid Matching module (620) in our proposed architecture has the functionality of one- to-one mapping between the projected spherical coordinates (only elevation and azimuth values) to the pre-defined grid. To achieve this, we can exploit both traditional and learning based methods. It should be noted that the pre-defined grid is defined, for example, using the known knowledge about the LiDAR sensor. For example, Velodyne64 sensor has 64 lasers with an average of around 2000 points per laser. Thus, a grid of 64 rows and 2048 columns would be sufficient for point clouds from this sensor.

[0058]

[0053] It should be emphasized here that the goal of grid matching module is an appropriate matching of the 3D points to a 2D image such that the neighborhood relation in the 3D space is still approximately preserved in the 2D image. With the neighborhood relation preserved, the resulting images retain the spatial correlation in the point cloud and can facilitate efficient compression of the resulting 2D images. A random mapping from 3D coordinates to 2D image would not be suitable for compression.

[0059]

[0054] In one embodiment, we use the well-known Earth Mover Distance (EMD) (see an article by Rubner, Yossi, et al., “A metric for distributions with applications to image databases,” sixth international conference on computer vision, IEEE, 1998) to find the corresponding mapping. The EMD is a measure of dissimilarity between two distributions (continuous or discrete) in a metric space. It can be seen as the minimum amount of work required to change one distribution into the other. This distance implicitly preserves the neighborhood information as the amount of work to move two neighboring points in one domain to far-away points in another domain would be larger.

[0060]

[0055] In another embodiment, we can use methods like RasterFairy or IsoMatch (see an article by Fried, Ohad, et al., “IsoMatch: Creating informative grid layouts,” Computer graphics forum, Vol. 34. No. 2. 2015) which can efficiently transform a 2D point cloud to a 2D grid whilst trying to preserve the neighborhood relations present in the original 2D point cloud. These methods work by trying to minimize an approximate objective function which acts as a proxy to the pairwise assignment objective function.

[0061]

[0056] In yet another embodiment, we can use the learning based FoldingNet (see an article by Yang, Yaoqing, et al., “Foldingnet: Point cloud auto-encoder via deep grid deformation,” Proceedings of the IEEE conference on computer vision and pattern recognition, 2018) or TearingNet (see an article by Pang, Jiahao, et al., “Tearingnet: Point cloud autoencoder to learn topology-friendly representations,” Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2021) to transform a 2D grid into the projected spherical coordinates and thus find the associated mapping. Both methods learn a non-linear mapping using a few learnable MLP (MultiLayer Perceptron) layers to fold (or tear then fold) a fixed 2D grid to a given point cloud. The learnable folding operation is not perfect but using a nearest neighbor search after the folding can lead to the desired output of a mapping between a 2D grid and the projected spherical coordinates.

[0062]

[0057] High Bit Depth Compression with Low Bit Depth Coding

[0063]

[0058] In a typical LiDAR system, the data is acquired as floating-point values in meters which are then quantized to the resolution of the sensor, typically 1mm. Given the typical range of the LiDAR sensor is 120m, the quantized data ends up having a bit depth of around 18 bits. Traditional image codecs typically deal with lower bit depth data, e.g., 8 bits or 10 bits. To perform compression of this high bit depth data using typical image compression pipelines, we propose separating the bits into several smaller low bit depth chunks containing most and least (and possibly middle) significant bits. The choice of bit allocation between the MSB and LSB can be user-defined and may depend on the type of LiDAR data being compressed.

[0064]

[0059] FIG. 7 illustrates a diagram of such an architecture, where the most and least significant bits are compressed separately using two separate image compressors, according to an embodiment. As we can see from FIG. 7, after the high bit depth coordinates (Xc) are projected (710) onto an image / grid, the bits are separated into two images MSB The MSB images is usually very smooth and is compressed losslessly or with high fidelity via ENCLL (720) as it contains more significant information. On the other hand, the LSB image is compressed in a lossy fashion via ENCL (730). At the decoder side after both images are decoded by corresponding decoder DECLL (740) and DECL (750), the information in the most (XggB) and least significant images is combined (760) to produce a lossy reconstruction (c) of the original 3D cartesian coordinates.

[0065]

[0060] In another embodiment as shown in FIG. 8, instead of being completely independent, the encoder / decoder of the LSB takes input from the MSB branch. The pipeline is similar to FIG. 7, except that the encoding (830) and decoding (850) of the LSB image is conditioned upon the encoded (820) and decoded MSB image (840), respectively.

[0066]

[0061] It should be noted here that the lossless encoder / decoder for MSB in FIG. 7 and FIG. 8 can be any lossless (traditional or learning-based) image encoder / decoder pair. Moreover, for the lossy encoder / decoder for LSB any of the existing state-of-the-art lossy (traditional or learning-based) image encoder / decoder pair can be employed.

[0067]

[0062] An example architecture of the setup in FIG. 7 with traditional image compression codec JPEG2000 is shown in FIG. 9, where the said codec operates in lossless mode (920, 940) for MSB and lossy mode (930, 950) for LSB.

[0068]

[0063] An example architecture of the setup in FIG. 8 with traditional image compression codec JPEG2000 (lossless) for MSB and learning based scale hyperprior model (lossy) (see an article by Hu, Yueyu, et al., "Learning end-to-end lossy image compression: A benchmark." IEEE Transactions on Pattern Analysis and Machine Intelligence 44.8 (2021): 4194-421 1) for LSB is shown in FIG. 10 and FIG. 11, for the encoder and decoder, respectively.

[0069]

[0064] Here, the learning-based model is conditioned on input from the MSB branch during encoding / decoding. Within the learning based model, Conv N,k,k / d| represents 2D convolution with N output channels, kernel size k x k, and downsampling factor d, Conv N,k,k / d) represents 2D convolution with N output channels, kernel size k x k, and upsampling factor d. Moreover, ReLU (rectified linear unit) is the nonlinearity, Q represents quantization, AE / AD represent arithmetic encoding / decoding, and abs represents the absolute value.

[0070]

[0065] Specifically, at the encoder side, the LSB is encoded with a feature encoder (1010) and a hyper prior encoder (1020). Both the feature and hyper feature are quantized (1030, 1050) and arithmetic encoded (1040, 1060). Further, the hyper feature is arithmetic decoded (1070) and restored (1080) to be used as input to the arithmetic encoder (1040) for the feature. At the decoder side, the MSB is downscaled (1110) to be used as prediction for the LSB branch. The hyper feature is arithmetic decoded (1140) and restored (1130) as input to the arithmetic decoder (1150) for the feature. The feature decoder (1120) output the reconstructed LSB, which is combined (1160) with the MSB to reconstruct the point cloud.

[0071]

[0066] Block Based Coding

[0072]

[0067] The description so far has been about the scenario when a whole point cloud (frame) is converted into an image representation and compressed. However, in some cases it is desirable to first split the point cloud into several regions / blocks (% ), and then compress each region / block, as shown in FIG. 12. This can be achieved, for example, by first using an octree for partitioning (1210) the point cloud, followed by conversion and compression of each block (X5) via proposed image-based compression. At the decoder side, the decoded blocks (A^) are combined (1220) to reconstruct the point cloud. It should be noted that for coding the octree partitioning information, either a traditional or a learning-based octree coding scheme can be used.

[0073]

[0068] Various numeric values are used in the present application. The specific values are for example purposes and the aspects described are not limited to these specific values.

[0074]

[0069] Various methods are described herein, and each of the methods comprises one or more steps or actions for achieving the described method. Unless a specific order of steps or actions is required for proper operation of the method, the order and / or use of specific steps and / or actions may be modified or combined. Additionally, terms such as “first”, “second”, etc. may be used in various embodiments to modify an element, component, step, operation, etc., such as, for example, a “first decoding” and a “second decoding”. Use of such terms does not imply an ordering to the modified operations unless specifically required. So, in this example, the first decoding need not be performed before the second decoding, and may occur, for example, before, during, or in an overlapping time period with the second decoding.

[0075]

[0070] The implementations and aspects described herein may be implemented in, for example, a method or a process, an apparatus, a software program, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (for example, discussed only as a method), the implementation of features discussed may also be implemented in other forms (for example, an apparatus or program). An apparatus may be implemented in, for example, appropriate hardware, software, and firmware. The methods may be implemented in, for example, an apparatus, for example, a processor, which refers to processing devices in general, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, for example, computers, cell phones, portable / personal digital assistants (“PDAs”), and other devices that facilitate communication of information between end-users.

[0076]

[0071] Reference to “one embodiment” or “an embodiment” or “one implementation” or “an implementation”, as well as other variations thereof, means that a particular feature, structure, characteristic, and so forth described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrase “in one embodiment” or “in an embodiment” or “in one implementation” or “in an implementation”, as well any other variations, appearing in various places throughout this application are not necessarily all referring to the same embodiment.

[0077]

[0072] Additionally, this application may refer to “determining” various pieces of information. Determining the information may include one or more of, for example, estimating the information, calculating the information, predicting the information, or retrieving the information from memory.

[0078]

[0073] Further, this application may refer to “accessing” various pieces of information. Accessing the information may include one or more of, for example, receiving the information, retrieving the information (for example, from memory), storing the information, moving the information, copying the information, calculating the information, determining the information, predicting the information, or estimating the information.

[0079]

[0074] Additionally, this application may refer to “receiving” various pieces of information. Receiving is, as with “accessing”, intended to be a broad term. Receiving the information may include one or more of, for example, accessing the information, or retrieving the information (for example, from memory). Further, “receiving” is typically involved, in one way or another, during operations, for example, storing the information, processing the information, transmitting the information, moving the information, copying the information, erasing the information, calculating the information, determining the information, predicting the information, or estimating the information.

[0080]

[0075] It is to be appreciated that the use of any of the following “ / ”, “and / or”, and “at least one of’, for example, in the cases of “A / B”, “A and / or B” and “at least one of A and B”, is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of both options (A and B). As a further example, in the cases of “A, B, and / or C” and “at least one of A, B, and C”, such phrasing is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of the third listed option (C) only, or the selection of the first and the second listed options (A and B) only, or the selection of the first and third listed options (A and C) only, or the selection of the second and third listed options (B and C) only, or the selection of all three options (A and B and C). This may be extended, as is clear to one of ordinary skill in this and related arts, for as many items as are listed.

[0081]

[0076] As will be evident to one of ordinary skill in the art, implementations may produce a variety of signals formatted to carry information that may be, for example, stored or transmitted. The information may include, for example, instructions for performing a method, or data produced by one of the described implementations. For example, a signal may be formatted to carry the bitstream of a described embodiment. Such a signal may be formatted, for example, as an electromagnetic wave (for example, using a radio frequency portion of spectrum) or as a baseband signal. The formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information that the signal carries may be, for example, analog or digital information. The signal may be transmitted over a variety of different wired or wireless links, as is known. The signal may be stored on a processor-readable medium.

Claims

CLAIMS1. A method of decoding point cloud data, comprising: decoding at least a 2D image, wherein each color component of said 2D image corresponds to a dimension of 3D cartesian coordinates of point cloud data of a 3D point cloud; obtaining said 3D cartesian coordinates of said point cloud data of said 3D point cloud from said color components of said at least a 2D image; and reconstructing said 3D point cloud from said 3D cartesian coordinates of said 3D point cloud.

2. The method of claim 1, wherein said decoding comprises: decoding at least a first 2D image and a second 2D image, wherein said first 2D image corresponds to a first range of bit depths of said point cloud data of said 3D point cloud, and said second 2D image corresponds to a second range of bit depths of said point cloud data of said 3D point cloud, and wherein said 3D cartesian coordinates of said point cloud data of said 3D point cloud are obtained from both said first 2D image and second 2D image.

3. The method of claim 2, wherein said first 2D image corresponds to a range of higher bit depth and said second 2D image corresponds to a range of lower bit depths, and wherein said first 2D image is decoded with a lossless decoder, and said second 2D image is decoded with a lossy decoder.

4. The method of claim 2 or 3, wherein said first 2D image is decoded with a neural network based decoder and said second 2D image is decoded by a non-neural network based decoder.

5. The method of any one of claims 2-4, wherein decoding of said second 2D image depends on said first 2D image.

6. The method of any one of claims 1-5, wherein said 3D point cloud corresponds to a region of another 3D point cloud.

7. The method of any one of claims 1-6, wherein x-, y- and z-coordinates of said point cloud data correspond to R, G and B components of said 2D image.

8. A method of encoding point cloud data of a 3D point cloud, comprising: obtaining said point cloud data with 3D cartesian coordinates; mapping said 3D cartesian coordinates to cartesian coordinates on a 2D grid to form a 2D image, wherein a respective color component of said 2D image corresponds to a dimension of said 3D cartesian coordinates; and encoding said 2D image.

9. The method of claim 8, wherein said mapping comprises: projecting said 3D cartesian coordinates to spherical coordinates to obtain elevation and azimuth values; matching said elevation and azimuth values one-to-one to grid points in said 2D grid; and placing said 3D cartesian coordinates onto matched grid coordinates in said 2D grid to output grid aligned cartesian coordinates for said 2D image, based on results of said matching.

10. The method of claim 9, wherein said matching is performed to preserve neighborhood relation in said 3D point cloud.

11. The method of any one of claims 8-10, wherein each spherical coordinate is associated with a grid point in said 2D grid and a matching index representing said association is obtained.

12. The method of any one of claims 8-11, wherein said 2D grid is defined based on information about LiDAR sensors.

13. The method of any one of claims 8-12, further comprising: partitioning said 2D image into at least a first 2D image and a second 2D image, wherein said first 2D image corresponds to a first range of bit depths, and said second 2D image corresponds to a second range of bit depths; and encoding said first 2D image and said second 2D image.

14. The method of claim 13, wherein said first 2D image corresponds to a range of higher bit depth, and said second 2D image corresponds to a range of lower bit depths, and wherein said first 2D image is encoded with a lossless encoder, and said second 2D image is encoded with a lossy encoder.

15. The method of any one of claims 8-14, wherein said encoding of said second 2D image depends on said first 2D image.

16. The method of any one of claims 8-15, wherein said 3D point cloud corresponds to a region of another 3D point cloud.

17. The method of any one of claims 8-16, wherein x-, y- and z-coordinates of said 3D cartesian coordinates of said point cloud data correspond to R, G and B components of said 2D image.

18. An apparatus, comprising one or more processors and at least one memory coupled to said one or more processors, wherein said one or more processors are configured to perform the method of any of claims 1-17.

19. A non-transitory computer readable medium comprising instructions which, when the instructions are executed by a computer, cause the computer to perform the method of any of claims 1-17.

Images