An extended framework for point cloud compression

Abstract

In one implementation, a lossy point cloud compression scheme is proposed to encode the geometry of a point cloud using deep neural networks. The encoder first encodes a coarser version of the input point cloud as a bitstream. It then represents the residuals (fine geometric details) of the input point cloud as per-point features of the encoded coarser point cloud, and then encodes the features as a second bitstream. At the decoder side, the coarser point cloud is first decoded from the first bitstream. Then, its per-point features are decoded. Finally, the residuals are decoded from the per-point features and added back to the coarser point cloud, resulting in a high-quality decoded point cloud. The encoding and / or decoding of the features can be further augmented by feature aggregation, such as a Transformer block.

Description

[Technical field]

[0001] The present embodiments relate generally to methods and apparatus for point cloud compression and processing. Summary of the Invention [Problem to be solved by the invention]

[0002] The Point Cloud (PC) data format is a universal data format across several business domains, for example, from autonomous driving, to robotics, Augmented Reality / Virtual Reality (AR / VR), civil engineering, computer graphics, and animation / film industries. 3D Light Detection and Ranging (LiDAR) sensors are being deployed in autonomous vehicles, and affordable LiDAR sensors have been released from Velodyne Velabit, Apple iPad Pro 2020, and Intel RealSense LiDAR Camera L515. With advances in sensing technology, 3D point cloud data is more practical than ever and is expected to be the ultimate enabler in the applications discussed herein.

[0003] According to one embodiment, a method for decoding point cloud data is presented, the method comprising: decoding a first version of a point cloud, obtaining a set of point-wise features for the first version of the point cloud, obtaining refinement information for the first version of the point cloud from the set of point-wise features, and obtaining a second version of the point cloud based on the refinement information and the first version of the point cloud.

[0004] According to another embodiment, a method for encoding point cloud data is presented, the method comprising: encoding a first version of a point cloud; reconstructing a second version of the point cloud for the point cloud; obtaining refinement information based on the second version of the point cloud and the point cloud; obtaining point-wise features for the second version of the point cloud from the refinement information; and encoding the point-wise features.

[0005] According to another embodiment, an apparatus for decoding point cloud data is presented, comprising one or more processors and at least one memory coupled to the one or more processors, the one or more processors configured to: decode a first version of a point cloud, obtain a set of point-wise features for the first version of the point cloud, obtain refinement information for the first version of the point cloud from the set of point-wise features, and obtain a second version of the point cloud based on the refinement information and the first version of the point cloud.

[0006] According to another embodiment, an apparatus for encoding point cloud data is presented, comprising one or more processors and at least one memory coupled to the one or more processors, the one or more processors configured to encode a first version of a point cloud, reconstruct a second version of the point cloud for the point cloud, obtain refinement information based on the second version of the point cloud and the point cloud, obtain point-wise features for the second version of the point cloud from the refinement information, and encode the point-wise features.

[0007] One or more embodiments also provide a computer program comprising instructions that, when executed by one or more processors, cause the one or more processors to perform an encoding or decoding method according to any of the embodiments described herein. One or more of the embodiments also provide a computer readable storage medium storing instructions for encoding or decoding point cloud data according to the methods described herein.

[0008] One or more embodiments also provide a computer-readable storage medium that stores video data generated by the methods described herein. One or more embodiments also provide methods and apparatus for transmitting or receiving video data generated by the methods described herein. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 illustrates a block diagram of a system in which aspects of the present embodiments may be implemented. [Figure 2A] FIG. 2A shows point-based, octree-based, voxel-based, and sparse voxel-based point cloud representations, respectively. [Figure 2B] FIG. 2B shows point-based, octree-based, voxel-based, and sparse voxel-based point cloud representations, respectively. [Figure 2C] FIG. 2C shows point-based, octree-based, voxel-based, and sparse voxel-based point cloud representations, respectively. [Figure 2D] FIG. 2D shows point-based, octree-based, voxel-based, and sparse voxel-based point cloud representations, respectively. [Diagram 3] FIG. 3 illustrates an encoder architecture according to one embodiment. [Figure 4] FIG. 4 illustrates a decoder architecture according to one embodiment. [Diagram 5] FIG. 5 illustrates an encoder architecture with a lossless base codec according to one embodiment. [Figure 6] FIG. 6 is a diagram of a subtraction module, according to one embodiment. [Figure 7] FIG. 7 is a diagram of a residual-to-feature converter according to one embodiment. [Figure 8] FIG. 8 is a diagram of a residual-to-feature converter for sparse point clouds according to one embodiment. [Figure 9] FIG. 9 illustrates a feature encoder architecture according to one embodiment. [Figure 10] FIG. 10 illustrates a feature decoder architecture according to one embodiment. [Figure 11] FIG. 11 illustrates an example application of sparse tensor operations, according to one embodiment. [Figure 12] FIG. 12 is a diagram of a feature-to-residual transformer according to one embodiment. [Figure 13] FIG. 13 is a diagram of a summation module, according to one embodiment. [Figure 14] FIG. 14 is a decoder architecture in skip mode according to one embodiment. [Figure 15] FIG. 15 is a feature decoder architecture in skip mode according to one embodiment. [Figure 16] FIG. 16 illustrates another feature encoder architecture according to one embodiment. [Figure 17] FIG. 17 illustrates a feature decoder architecture according to one embodiment. [Figure 18] FIG. 18 illustrates the cascading of several feature aggregation modules according to one embodiment. [Figure 19] FIG. 19 shows the Transformer block for feature aggregation. [Figure 20] Figure 20 shows the Inception-ResNet block for feature aggregation. [Figure 21] Figure 21 shows the ResNet block for feature aggregation. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] FIG. 1 illustrates a block diagram of an example of a system in which various aspects and embodiments may be implemented. System 100 may be embodied as a device including various components described below and configured to perform one or more of the aspects described herein. Examples of such devices include various electronic devices, such as, but not limited to, personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected appliances, and servers. The elements of system 100 may be embodied, alone or in combination, in a single integrated circuit, multiple ICs, and / or separate components. For example, in at least one embodiment, the processing elements and encoder / decoder elements of system 100 are distributed across multiple ICs and / or separate components. In various embodiments, system 100 is communicatively coupled to other systems or other electronic devices, for example, through a communication bus or dedicated input and / or output ports. In various embodiments, system 100 is configured to implement one or more of the aspects described herein.

[0011] The system 100 includes at least one processor 110 configured to execute instructions loaded therein, for example, to implement various aspects described herein. The processor 110 may include embedded memory, input / output interfaces, and various other circuits known in the art. The system 100 includes at least one memory 120 (e.g., volatile and / or non-volatile memory devices). The system 100 includes a storage device 140, which may include non-volatile and / or volatile memory, including, but not limited to, EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic disk drives, and / or optical disk drives. The storage device 140 may include, by way of non-limiting example, an internal storage device, an attached storage device, and / or a network-accessible storage device.

[0012] The system 100 includes, for example, an encoder / decoder module 130 configured to process data to provide encoded or decoded video, which may include its own processor and memory. The encoder / decoder module 130 represents a module that may be included within a device to perform encoding and / or decoding functions. As is known, a device may include one or both of an encoding module and a decoding module. It is noted that the encoder / decoder module 130 may be implemented as a separate element of the system 100 or may be incorporated within the processor 110 as a combination of hardware and software known to those skilled in the art.

[0013] Program code to be loaded onto the processor 110 or the encoder / decoder 130 to perform various aspects described herein may be stored in the storage device 140 and then loaded onto the memory 120 for execution by the processor 110. According to various embodiments, one or more of the processor 110, the memory 120, the storage device 140, and the encoder / decoder module 130 may store one or more of various items during execution of the processes described herein. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from processing of equations, expressions, operations, and operational logic.

[0014] In some embodiments, memory internal to the processor 110 and / or the encoder / decoder module 130 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device can be either the processor 110 or the encoder / decoder module 130) is used for one or more of these functions. The external memory may be the memory 120 and / or the storage device 140, e.g., dynamic volatile memory and / or non-volatile flash memory. In some embodiments, the external non-volatile flash memory is used to store the television's operating system. In at least one embodiment, a high-speed external dynamic volatile memory, such as RAM, is used as working memory for video encoding and decoding operations, such as MPEG-2, JPEG Pleno, MPEG-1, HEVC, or VVC.

[0015] Input to the elements of system 100 may be provided through various input devices, as indicated at block 105. Such input devices may include, but are not limited to, (i) an RF section that receives, for example, an RF signal transmitted over the air by a broadcast station, (ii) a composite input, (iii) a USB input, and / or (iv) an HDMI input.

[0016] In various embodiments, the input devices of block 105 have associated respective input processing elements as known in the art. For example, the RF section may be associated with elements suitable for (i) selecting a desired frequency (also referred to as selecting a signal or band-limiting a signal to a frequency band), (ii) down-converting the selected signal, (iii) band-limiting again to a narrower frequency band to select a signal frequency band that may be referred to (for example) as a channel in a particular embodiment, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired stream of data packets. The RF section of various embodiments includes one or more elements that perform these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF section may include a tuner that performs these various functions, including, for example, down-converting a received signal to a lower frequency (e.g., an intermediate frequency or a nearby baseband frequency) or to baseband. In one embodiment of a set-top box, the RF section and its associated input processing elements perform frequency selection by receiving, filtering, downconverting, and refiltering RF signals transmitted over a wired (e.g., cable) medium to a desired frequency band. Various embodiments rearrange the order of the above-mentioned (and other) elements, remove some of these elements, and / or add other elements that perform similar or different functions. Adding elements may include inserting elements between existing elements, for example, inserting amplifiers and analog-to-digital converters. In various embodiments, the RF section includes an antenna.

[0017] It should be noted that the USB and / or HDMI terminals may include respective interface processors for connecting the system 100 to other electronic devices across USB and / or HDMI connections. It should be understood that various aspects of the input processing, e.g., Reed-Solomon error correction, may be implemented, for example, in a separate input processing IC or in the processor 110, as desired. Similarly, aspects of the USB or HDMI interface processing may be implemented, as desired, in a separate interface IC or in the processor 110. For example, the processor 110 and various processing elements, including the encoder / decoder 130, which operates in combination with memory and storage elements, to process the data stream as desired for presentation on an output device by providing a demodulated, error corrected, and demultiplexed stream.

[0018] The various elements of the system 100 may be provided within an integrated housing in which the various elements may be interconnected and transmit data between each other using suitable connection arrangements 115, such as internal buses well known in the art, including an I2C bus, wiring, and printed circuit boards.

[0019] System 100 includes a communication interface 150 that enables communication with other devices over a communication channel 190. Communication interface 150 may include, but is not limited to, a transceiver configured to transmit and receive data over communication channel 190. Communication interface 150 may include, but is not limited to, a modem or a network card, and communication channel 190 may be implemented in a wired and / or wireless medium, for example.

[0020] Data is streamed to the system 100 in various embodiments using a Wi-Fi network such as IEEE 802.11. The Wi-Fi signal in these embodiments is received by a communication channel 190 and communication interface 150 adapted for Wi-Fi communication. The communication channel 190 in these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to enable streaming applications and other over-the-top communications. In other embodiments, a set-top box delivering data through an HDMI connection of the input block 105 is used to provide the streamed data to the system 100. In still other embodiments, an RF connection of the input block 105 is used to provide the streamed data to the system 100.

[0021] System 100 may provide output signals to various output devices, including display 165, speakers 175, and other peripheral devices 185. Other peripheral devices 185, in various examples of embodiments, include one or more of a standalone DVR, a disc player, a stereo system, a lighting system, and other devices that provide functionality based on the output of system 100. In various embodiments, control signals are communicated between system 100 and display 165, speakers 175, or other peripheral devices 185 using signaling such as AV.Link, CEC, or other communication protocols that allow inter-device control with or without user intervention. Output devices may be communicatively coupled to system 100 via dedicated connections through respective interfaces 160, 170, and 180. Alternatively, output devices may be connected to system 100 via communication interface 150 using communication channel 190. Display 165 and speakers 175 may be integrated into a single unit along with other components of system 100, for example in an electronic device such as a television. In various embodiments, display interface 160 includes a display driver, for example a Timing Controller (T Con) chip.

[0022] Display 165 and speakers 175 may alternatively be separate from one or more of the other components, for example, if the RF portion of input 105 is part of a separate set-top box. In various embodiments where display 165 and speakers 175 are external components, the output signals may be provided via dedicated output connections including, for example, an HDMI port, a USB port, or a COMP output.

[0023] It is believed that point cloud data may consume a large portion of network traffic, for example between cars connected via 5G networks and in immersive communications (VR / AR). Efficient representation formats are necessary for understanding and communicating point clouds. In particular, raw point cloud data needs to be properly organized and processed for world modeling and sensing purposes. Compression of raw point clouds is essential if storage and transmission of the data is required in relevant scenarios.

[0024] Furthermore, point clouds may represent successive scans of the same scene containing multiple moving objects. They are called dynamic point clouds, in comparison to static point clouds captured from a static scene or static objects. Dynamic point clouds are typically organized into frames, with different frames captured at different times. Dynamic point clouds may require processing and compression to be done in real-time or with low latency.

[0025] The automotive industry and autonomous vehicles are areas where point clouds can be used. Autonomous vehicles should be able to "probe" their environment and make good driving decisions based on the reality of their immediate surroundings. Typical sensors like LiDAR generate (dynamic) point clouds that are used by perception engines. These point clouds are not intended to be seen by the human eye, they are usually sparse, not necessarily colored, and dynamic with a high capture frequency. They may have other attributes like reflectivity provided by LiDAR, since this attribute indicates the material of the object being sensed and can help to make decisions.

[0026] Virtual reality (VR) and immersive worlds are predicted by many as the future of 2D flat video. With VR and immersive worlds, the viewer is immersed in the environment all around them, as opposed to standard TV, where the viewer can only see the virtual world in front of them. There are several degrees of immersion, depending on the viewer's degrees of freedom in the environment. Point clouds are a good format candidate for delivering virtual reality worlds. Point clouds used in VR can be static or dynamic, and are typically of average size, e.g., at most a few million points at a time.

[0027] Point clouds can also be used for various purposes such as cultural heritage / architecture, where objects such as statues or buildings are scanned in 3D to share the spatial configuration of the object without sending or visiting the object. Point clouds can also be used to ensure preservation of the knowledge of an object when it may be destroyed, for example a temple may be destroyed by an earthquake. Such point clouds are usually static, color-coded and huge.

[0028] Another use case is in topography and cartography, where using 3D representations maps are not limited to flat surfaces but can include relief. Google Maps is a good example of a 3D map, but it uses meshes instead of point clouds. Nevertheless, point clouds can be the preferred data format for 3D maps, and such point clouds are usually static, color-coded, and large.

[0029] World modeling and sensing via point clouds can be a useful technique for the applications considered herein, enabling a machine to gain knowledge about the 3D world around it.

[0030] 3D point cloud data is essentially a set of discrete samples on the surface of an object or scene. To fully represent the real world with point samples, a huge number of points is actually required. For example, a typical VR immersive scene contains millions of points, while a point cloud typically contains hundreds of millions of points. Thus, processing such large-scale point clouds is computationally expensive, especially for consumer devices with limited computing power, such as smartphones, tablets, and automobile navigation systems.

[0031] To perform processing or inference on point clouds, an efficient storage method is needed. To store and process the input point cloud at an affordable computational cost, one solution is to first downsample the point cloud, where the downsampled point cloud summarizes the geometry of the input point cloud while having very few points. The downsampled point cloud is then fed to a subsequent machine task for further consumption. However, further reduction in storage space can be achieved by converting the raw point cloud data (original or downsampled) into a bitstream through entropy encoding techniques for lossless compression. A better entropy model results in a smaller bitstream and therefore more efficient compression. Note that the entropy model can also be paired with downstream tasks that allow the entropy encoder to maintain task-specific information while compressing.

[0032] In addition to lossless coding, many scenarios call for lossy coding for significantly improved compression ratios while maintaining the induced distortion under a certain quality level.

[0033] We propose a learning-based PCC framework that can perform compression using different point cloud representations. In the following, we first consider different point cloud representations and their use in learning-based PCC.

[0034] Point-based representation A point cloud is essentially a collection of 3D coordinates that sample the surface of an object or scene. In this native representation, each point is directly specified by its x, y, and z coordinates in 3D space. However, the points in a point cloud are usually unorganized and sparsely distributed in 3D space, making it difficult to directly process the point coordinates. Figure 2A provides an example of a point-based representation. For simplicity, all figures in Figures 2A, 2B, 2C, and 2D show the corresponding point cloud representation in 2D space.

[0035] With the development of deep learning, point cloud processing and compression has also been studied using unique point-based representations. One of the most representative works in this thread is PointNet, which is a point-based processing architecture based on Multi-Layer Perceptrons (MLP) and a global max pooling operator for feature extraction. Subsequent works such as PointNet++, DGCNN, and KP-Conv extend PointNet to more complex point-based operations that consider neighboring information. These point-based processing architectures can be utilized for PCC. In the work described in the paper titled "Deep autoencoder-based lossy geometry compression for point clouds" by Yan, Wei et al. (arXiv preprint arXiv:1905.03691, 2019), the encoder employs a five-layer PointNet to extract feature vectors to compress the input point cloud. The decoder employs a series of MLPs to decode the point cloud. Another study, DECOPO (see the paper entitled “Deep compression for dense point cloud maps” by Wiesmann, Louis et al., IEEE Robotics and Automation Letters 6, no. 2, pp. 2060-2067, 2021), employs KP-Conv to process the input point cloud and generate a bitstream.

[0036] Octree-based representation Besides unique point coordinates, a point cloud can be represented by an octree decomposition tree, as shown in the example of FIG. 2B. First, a root node is constructed that covers the complete 3D space within a bounding box. This space is then divided equally in all directions, i.e., in the x-, y-, and z-directions, resulting in eight voxels. For each voxel, if there is at least one point, the voxel is marked as occupied and represented by a value of "1", otherwise it is marked as empty and represented by a value of "0". The voxel division is then continued until a pre-specified condition is met. FIG. 2B provides a simple example, which shows a quadtree, which is the 2D counterpart of an octree. By encoding the octree, its associated point cloud is losslessly encoded. A common approach to encoding an octree is by encoding each occupied voxel with an 8-bit value that indicates the occupancy of its individual octant. In this way, we first encode the root voxel node by an 8-bit value. Then, for each occupied voxel in the next level, we encode its 8-bit occupancy symbol and then move on to the next level.

[0037] Deep entropy models refer to a category of learning-based approaches that use neural network modules to formulate context models and attempt to predict the probability distribution of octet-bit occupancy symbols. One deep entropy model is known as OctSqueeze (see the paper by Huang, Lila et al., entitled "OctSqueeze: Octree-structured entropy model for LiDAR compression," in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2020). It utilizes ancestor nodes, including parent nodes, grandparent nodes, etc. Three MLP-based modules are used to estimate the probability distribution of occupancy symbols for the current octree node. Another deep entropy model is known as VoxelContextNet (see the paper by Que, Zizheng et al., entitled "VoxelContext-Net: An octree-based framework for point cloud compression," in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pp. 6042-6051, 2021). Unlike OctSqueeze, which uses ancestor nodes, VoxelContextNet adopts an approach that first analyzes the local surface shape and then uses spatial neighboring voxels to predict the probability distribution.

[0038] Voxel-based representation In the voxel-based representation, the 3D point coordinates are uniformly quantized by a quantization process. Each point corresponds to an occupied voxel with a size equal to the quantization process (FIG. 2C). The naive voxel representation may not be efficient in memory usage due to large empty spaces. Then, a sparse voxel representation is introduced, in which the occupied voxels are arranged in a sparse tensor format for efficient storage and processing. An example of the sparse voxel representation is depicted in FIG. 2C, where the empty voxels (with dotted lines) do not consume any memory or storage.

[0039] Considering that 2D convolution has been successfully used in learning-based image compression, its extension 3D convolution has also been investigated for point cloud compression. For this purpose, the point cloud needs to be represented by voxels. In regular 3D convolution, a 3D kernel is overlaid on all positions specified by the stride step, regardless of whether the voxel is occupied or empty. To avoid the computation and memory consumption due to empty voxels, sparse 3D convolution may be applied, where the point cloud voxels are represented by a sparse tensor.

[0040] In the study, pcc_geo_cnn_v2 (see the paper entitled “Improved deep point cloud geometry compression” by Quach, Maurice et al., 2020 IEEE 22nd International Workshop on Multimedia Signal Processing (MMSP)), the authors propose to encode / decode the point cloud using regular 3D convolutions. To avoid the large computational cost and memory consumption, another study, PCGCv2 (see the paper entitled “Multiscale point cloud geometry compression” by Wang, Jianqiang et al., 2021 Data Compression Conference (DCC)), encodes / decodes the point cloud using sparse 3D convolutions. It also employs an octree coding scheme to losslessly encode the low bit depth part of the input point cloud.

[0041] In this paper, we propose an extended coding framework for lossy point cloud compression using deep neural networks. In the following, we first provide a general architecture and then provide details.

[0042] Overview of the proposed extended coding framework In the proposed coding framework, instead of directly encoding / decoding the point cloud, we convert the point cloud into a coarser / simplified point cloud with per-point features as point attributes for encoding / decoding. The features represent refinement information, e.g., residuals (or geometric details) from the input point cloud. Specifically, we use a base layer for encoding / decoding the coarser point cloud and an enhancement layer for encoding / decoding the per-point features, as shown in Fig. 3 and Fig. 4.

[0043] Input point cloud to be compressed PC 0Given , the encoder first converts it to a coarser point cloud. This coarser point cloud, which is easier to compress, is first encoded as a bitstream. Then, for each point in the coarser point cloud, the encoder generates a point-wise feature (or PC 0 The obtained point-wise features are further encoded as a second bitstream.

[0044] At the decoder side, we first decode a coarser point cloud from the first bit stream. We then proceed to decode point-wise features from the second bit stream based on the coarser point cloud. We then derive the residual components (PC 0 Finally, the decoded point cloud is obtained by adding the residual back to the coarser point cloud.

[0045] encoding The encoder architecture, according to one embodiment, is provided in Figure 3. Input point cloud PC 0 To compress, we first quantize it to a step size s (s>1) using a quantizer (310). In one embodiment, we use a PC 0 For every point, such as A in , the quantizer divides the coordinates by s and then converts them to integers to derive [x / s],[y / s],[z / s]), where the function [·] can be a floor function, ceiling function, or rounding function. The quantizer then removes duplicate points with the same 3D coordinates, i.e., if there are several points with the same coordinates, it keeps only one of them and removes the rest, and then calculates the PC 1 get.

[0046] The obtained quantized point cloud PC 1 is then compressed by a base point cloud encoder (320), which generates a bitstream BS 0 Then, BS 0 is decoded using the base point cloud decoder (330) to obtain another point cloud PC' 1Then, the step size s is used to output PC' 1 is inversely quantized (340) to give the first reconstructed point cloud PC 2 In one embodiment, a PC having 3D coordinates (x, y, z) is obtained. 0 For every point, such as A in , the inverse quantizer multiplies the coordinate by s, resulting in (xs, ys, zs). 2 is the original point cloud PC 0 Note that this is a coarser / simpler version of . Furthermore, as the quantization step size s increases, the PC 2 becomes even coarser. PC 2 The base layer is then completed.

[0047] In the enhancement layer, first, PC 0 and its rougher version PC 2 to the subtraction module (350). Intuitively, the subtraction module 0 From PC 2 The purpose of the method is to subtract the PC to be coded by the enhancement layer, and output the residual R. 0 The residual R is then fed to a residual-to-feature converter (360) to 2 Generate a point-wise feature vector for each point in 2 Point A in is the input PC close to point A. 0 A feature vector f that is a high-level descriptor of local fine details in A Then, the PC 2 Based on,the pointwise feature set (denoted by F) is encoded into another bitstream,BS,using a feature encoder (370). 1 It will be encoded as:

[0048] Two bitstreams BS 0 and B.S. 1 are the outputs of the encoder. They can also be merged into one unified bitstream.

[0049] Decryption The decoder architecture according to one embodiment is shown in Figure 4. Bitstream BS 0 and B.S. 1 When BS 1 is received, the base layer starts first and then proceeds to the enhancement layer. 0 From PC' 1 is decoded by a base decoder (410), and then an inverse quantizer (420) is applied with a step size s to obtain a coarser point cloud PC 2 get.

[0050] In the enhancement layer, the feature decoder module 430 first decodes the already decoded coarser point cloud PC 2 Using BS 1 is applied to decode the vector, outputting a set of point-wise features F'. The set of features F' is 2 For example, PC 2 A point A in has its own feature vector f' A The decoded feature vector f' A is the corresponding feature vector f A Note that f may have a different size than -. A and f' A Both are PCs close to point A. 0 The aim of the decoded feature set F' is to describe the local fine geometric details of the point cloud PC. The decoded feature set F' is then passed to a feature-to-residual converter (440) to generate the residual component R'. Finally, the coarser point cloud PC 2 and the residual R' is provided to an adder module (450). The adder module adds the residual R' to the PC 2 The final decoded point cloud PC' is obtained by adding it back to 0 results.

[0051] Module In the following, we provide details of the individual modules in the proposed framework and present possible variations of the modules.

[0052] Base PCC Codec The base codec (base encoder and base decoder in Figs. 3 and 4) can be any PCC codec. In one embodiment, the base codec is selected to be a lossy PCC codec, such as pcc_geo_cnn_v2 and PCGCv2. In this case, PC' 1 PC 1 (see FIG. 3). In another embodiment, the base codec is selected to be a lossless PCC codec, such as the MPEG G-PCC standard, or a deep entropy model with an octree representation, such as OctSqueeze and VoxelContext-Net. In this case, PC' 1 is essentially a PC 1 In that case, the encoder architecture can be simplified by removing the base decoder, as shown in FIG.

[0053] Subtraction Module The subtraction module, i.e.,

[0054]

number

[0055] In one embodiment, the subtraction module performs a k-nearest neighbor (kNN) search to find the PC 0 To extract the geometric details of the PC 2 For each 3D point A in ,, PC 0 Search (610) for the k-nearest neighbors in B 0 ,B 1 ,...,B k-1 Then, B 0 ,B 1 ,...,Bk-1 Subtracting point A from (620, 630, 640) gives the residual point B' 0 ,B' 1 ,...,B' k-1 Here, the term "subtraction" simply refers to point-wise subtraction, e.g., B 0 Subtracting A from B 0 Note that this means subtracting the (x, y, z) coordinates of A from the (x, y, z) coordinates of B. The resulting 3D point set B' 0 ,B' 1 ,...,B' k-1 , S A Then, PC 2 There are n points in A 0 ,A 1 ,...,A n-1 Then, each residual set is S 0 ,S 1 ,...,S n-1 and together they constitute the residual component R, which is the output of the subtraction module.

[0056] The value of k is the input point cloud PC 0 Note that k may be chosen based on the density level of . For dense point clouds, the value may be larger (e.g., k=10). In contrast, for sparse point clouds such as LiDAR sweeps, 0 In the case of , the value of k can be very small, such as k = 1, which is 2 All points in are 0 This means that the object is associated with only one point in the object.

[0057] In one embodiment, the PC closest to point A 0 Instead of using a kNN search to search k points in A, we search for PCs within distance r from A. 0 This operation is called a ball query. The value (or radius) r for the ball query can be determined by the quantization step size s of the quantizer. For example, if s is larger, i.e., PC 2 If is coarser, the value of r is 0In another embodiment, we still use a kNN search to find the k closest points to a query point, say A. However, we then keep only those points that are within a distance r from A. The value of r is determined in the same way as when using a ball query.

[0058] Also note that the distance metric used by the kNN search or ball query can be any distance metric, such as the L-1 norm, the L-2 norm, and the L-infinity norm.

[0059] Residual-to-feature converter Using the residual component R, the residual-to-feature converter 2 Calculate the set of feature vectors for points in (PC 2 We use a deep neural network to generate point-wise feature vectors by taking as input the residual set (associated with the points in 2 For point A, 3D point B' 0 ,B' 1 ,...,B' k-1 The residual set S A is the residual set S A A feature vector f describing the geometric shape of A The output is processed by a deep neural network. 2 n points in A 0 ,A 1 ,...,A n-1 For all of them, their corresponding feature vectors f 0 ,f 1 ,...,f n-1 gives us the feature set F, which is the output of the residual-feature module.

[0060] In one embodiment, the deep neural network processing S A uses the PointNet architecture, as shown in Figure 7. It consists of a set of shared MLPs (710). The perceptrons are 0 ,B' 1,...,B' k-1 The outputs of the set of shared MLPs (715, k × 32) are aggregated by a global max-pooling operation (720) that extracts a global feature vector (725) of length 32. This global feature vector is further processed (730) by another set of MLP layers to produce an output feature vector f A is brought about (735).

[0061] In the above, the PointNet architecture is used to extract features. Note that different network structures or configurations can be used. For example, the MLP dimension can be adjusted according to the complexity of the actual scenario, or more sets of MLPs can be used. In general, any network structure that meets the input / output requirements can be used.

[0062] In one embodiment, the residual set S A The number of points in the vector f A Add to the set S A This results in an extended feature vector with one additional dimension that indicates the density of . It is particularly useful when the ball query is used in the subtraction module, since the number of points retrieved by the ball query can be different for different residual sets.

[0063] Assuming we are dealing with a sparse point set and k=1, this means that the residual set S A Itself is a single 3D point B' 0 Then, in one embodiment, the deep neural network is simplified as a set of MLP layers, as shown in Figure 8. In another embodiment, the deep neural network is further pruned to obtain the feature vector f A Directly to B' 0 Let the (x,y,z) coordinates be the coordinates of the point.

[0064] Feature Codec The purpose of a feature codec (feature encoder and feature decoder) is to encode / decode a set of features. Specifically, on the encoder side, we take a feature vector f 0 ,f 1 ,...,f n-1 A feature set F including 1 (Fig. 3), and the decoder generates the bitstream BS 1 Let f' be a feature vector of n elements. 0 ,f' 1 ,...,f' n-1 (FIG. 4). The feature codec may be implemented based on a deep neural network.

[0065] In one embodiment, the feature encoder applies sparse 3D convolution with a downsampling operator to shrink and then encode the feature set F, while the feature decoder applies sparse 3D convolution with an upsampling operator to expand the received feature set. This is to exploit spatial redundancy between adjacent feature vectors to improve compression performance. To apply sparse 3D convolution, a sparse 3D (or 4D) tensor needs to be constructed first to represent the input point cloud. The sparse tensor stores only the occupied voxel coordinates and their associated features (e.g., FIG. 2D). The sparse 3D convolution layer operates only on the occupied voxels to reduce computational cost. In the simple case where the point cloud contains only geometric shapes (3D point coordinates), a sparse 3D tensor is constructed and the value "1" is placed in those occupied voxels. If the point cloud contains features / attributes in the form of 1D vectors, a sparse 4D tensor is constructed and the feature vectors are assigned to the corresponding occupied voxels.

[0066] Feature Encoder A feature encoder based on sparse 3D convolution and downsampling according to one embodiment is shown in Figure 9. Intuitively, we successively downsample a set of features F to make them smaller, and then decompose the downsampled version Fdown In Fig. 9, first, PC 2 9, a sparse 4D tensor is constructed (910) using F and the feature set F. Specifically, each voxel of the tensor represents a cube of size s×s×s in 3D space, with length equal to the quantization step size s of the quantizer / inverse quantizer in FIG. 3. It is then passed to two downsampling processing blocks (920, 930), each of which includes two sparse 3D convolutional layers (921, 923, 931, 933) and one downsampling operator (925, 935). In FIG. 9, “CONV N” (921, 923, 931, 933) represents a sparse 3D convolutional layer with N output channels and stride 1, while “ReLU” (922, 924, 932, 934) is a ReLU nonlinear activation function. The block "Downsampling2" (925, 935) is a sparse tensor downsampling operator with a ratio of 2. It reduces the size of the sparse tensor by a factor of 2 along each dimension, similar to the downsampling operator for regular 2D images (see FIG. 11 (1110) for an example example).

[0067] PC down The output tensor of the second processing block of FIG. 9, represented as down We read all feature vectors from the occupied voxels of and create a downsampled feature set F down It is then passed to a quantization module (950) followed by an entropy encoder (960) to produce the output bitstream BS 1 The target of the feature encoder is to encode a set of features F, but 2 provides the voxel geometry on which tensor operations (e.g., convolution and downsampling) operate, so the feature encoder still requires PCs as input. 2 It is clear that it requires

[0068] Feature Decoder A feature decoder based on sparse 3D convolution, downsampling and upsampling is shown in Fig. 10. Intuitively, first, a set of features F' is downsampled from the bitstream. down Then, to gradually expand and refine the features, 2 In FIG. 10, first, the bitstream BS 1 is decoded by an entropy decoder (1010), followed by an inverse quantizer (1020), resulting in a downsampled feature set F' down is obtained.

[0069] On the other hand, PC 2 We also construct a 3D sparse tensor (1090) based solely on the geometry (coordinates) of , and then sequentially downsample the tensors (1070, 1045) to obtain the tensor PC' down PC' is obtained. down (Fig. 10), feature encoder (Fig. 9) on PC down Note that although F' and F' have the same geometric shape, their features are different. down To upsample, we use PC' down This is accomplished by the feature replacement module (1030), which places the PC' down The original features of F' down Replace by another sparse tensor PC'' down results.

[0070] Then, PC'' is processed by two upsampling processing blocks (1040, 1060), each of which contains one upsampling operator and two sparse 3D convolutional layers. downIn FIG. 10, "upsampling2" is a sparse tensor upsampling operator with a ratio of 2. It expands the size of the sparse tensor by a factor of 2 along each dimension, similar to the upsampling operator for normal 2D images (see FIG. 11 (1120) for an example example). After each upsampling processing block, the resulting tensor is refined using a coordinate pruning module (1055, 1080). FIG. 11 (1130) provides an example example of coordinate pruning, which is to remove a portion of the occupied voxels of the input tensor and keep the rest based on the set of input coordinates obtained from the coordinate reader (1050, 1075). Using this module, in FIG. 10, "PC'' down We remove some voxels (and associated features) from the upsampled version of PC 2 After the second coordinate pruning module (1080), we keep only voxels that also appear in the downsampled version of 2 We obtain a tensor that has the same geometric shape as F′, and then apply the feature reader (1085) to it to obtain the decoded feature set F′.

[0071] The entropy coder in the feature codec can be non-trained and can be an entropy coder based on a deep neural network, e.g., a factorized prior, or a hyper-prior (see Balle, Johannes et al., "Variational image compression with a scale hyperprior," arXiv preprint, arXiv:1802.01436, 2018).

[0072] 9 and 10 show an example of applying two downsampling / upsampling processing blocks for the feature codec. However, under the same principle, it is possible to use fewer or more processing blocks. In one embodiment, we do not apply any downsampling / upsampling in the feature codec. In this embodiment, the feature encoder simply applies some convolutional layers between the sparse tensor construction and the feature reader, while the feature decoder applies another set of convolutional layers between the feature replacement module and the feature reader. Instead of the ReLU activation function, other activation functions such as the logistic sigmoid function and the hyperbolic tangent function can also be used. The number of convolutional layers, the kernel size of the convolutional layers and the number of output channels, as well as the way of combining them, can also be changed. The downsampling / upsampling processing block can also perform downsampling / upsampling by a ratio (denoted as α) different from 2, for example, α=4 or α=8. In one embodiment, a downsampling operator (with downsampling ratio α) may be absorbed into its previous convolutional layer (with stride 1) by changing the stride of the convolutional layer to α. Similarly, an upsampling operator (with upsampling ratio α) may be absorbed into its subsequent convolutional layer (with stride 1), both of which become a deconvolutional layer with stride α.

[0073] FIG. 11 is used to illustrate the downsampling, upsampling, coordinate reading, and coordinate pruning processes. For simplicity, the operations in FIG. 11 are shown in 2D space, but the same principles apply to 3D space. In this example, the input point cloud A0 occupies voxels at positions (0,2), (0,3), (0,4), (0,5), (1,1), ..., (7,4). Thus, the coordinate reader (1140) outputs the occupied coordinates as (0,2), (0,3), (0,4), (0,5), (1,1), ..., (7,4). By downsampling (1110) A0 by a factor of 2, the number of voxels is reduced by half in each dimension in the downsampled point cloud A1, and a voxel is considered occupied if any of the corresponding four points are occupied in A0. After upsampling (1120) A1, the number of voxels restarts in A2, and a voxel in A2 is considered occupied if its corresponding voxel is occupied in A1. Note that A2 is denser than the input point cloud A0. The occupied coordinate information is used by the coordinate pruning module (1130) to remove / prune from A2 the unoccupied points (occupied voxels) in A0. That is, in the resulting point cloud A3, only voxels that were occupied in the original point cloud A0 are considered occupied.

[0074] Feature-to-residual converter The feature-to-residual converter on the decoder (Figure 4) converts the decoded features F' back into residual components R'. Specifically, it applies a deep neural network to 2 All feature vectors f' in F' associated with point A in A Transform it to the corresponding residual point set S' A Return to.

[0075] In one embodiment, we implement a feature-to-residual transformer with a series of MLP layers, as shown in Figure 12. In this case, a feature vector in F', e.g., f' Ais fed into a series of MLP layers (1210), which directly convert the m 3D point clouds C 0 ,C 1 ,...,C m-1 which outputs the decoded residual set S' A Therefore, n points A 0 ,A 1 ,...,A n-1 For the feature to residual conversion, S' 0 ,S' 1 ,...,S' n-1 These residual sets together constitute the decoded residual component R'.

[0076] The number of decoded points m can be a fixed constant, such as m=5, or it can be 0 Note that the density level of the PC may be adaptively selected based on prior knowledge of the density level of the PC. 0 If it is known that x is very sparse, then m may be set to a small number, such as m=2.

[0077] In one embodiment, we also remove 3D points in R′ that are too far from the origin. i If the distance to the origin of a vector is greater than a threshold t, it is considered an outlier and is removed from R'. The threshold t can be a predetermined constant. It can also be selected according to the quantization step size s of the quantizer on the encoder (Figure 3). For example, the larger s, the smaller the PC 2 This means that R is coarser, in which case the threshold t is also larger to retain more points in R'.

[0078] Instead of simply using MLP layers, in another embodiment, the feature-to-residual converter may use a more advanced architecture such as a FoldingNet decoder (see Yang, Yaoqing et al., “FoldingNet: Point cloud auto-encoder via deep grid deformation,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018).

[0079] Addition Module After decoding the residual R', the summation module (Fig. 4

[0080]

number

[0081] In one embodiment, the summing module sums the points in R′ to the PC 2 to their associated points in R to generate new 3D points. A For each point C 0 ,C 1 ,...,C m-1 (1310, 1320, 1330), and another group C' of m points 0 ,C' 1 ,...,C' m-1 Note that the summation in this step simply means pointwise summation, i.e. the (x,y,z) coordinates of two 3D points are summed respectively. 2 For , this procedure ends with a total of n × m points, which together form the decoded point cloud PC' 0 In one embodiment, the summation module has an extra step at the end, which gives0 From the acquired n×m points, duplicate 3D points are removed to obtain

[0082] Skip Mode In one embodiment, the decoder receives the bitstream BS 1 Without taking as input, we can obtain a coarser point cloud PC 2 The skip mode is switched to by BS 0 This can be indicated by adding a flag to the BS or by a Supplementary Enhancement Information (SEI) message. 1 Since the decoder skips the first bit of the BS, this decoding mode is called skip mode. 0 This is particularly useful when the bit rate budget is tight, since only

[0083] Figure 14 shows the feature decoder taking PC 2 1 shows a decoder architecture in skip mode that receives only the PC 2 Based only on the geometric shape of F, we generate a set of feature vectors F'. Under skip mode, the architecture of the feature decoder is also simplified, as shown in Figure 15. In Figure 15, F const Note that, represents a predefined set of invariant features, e.g., the set of features with all 1s. Unlike in FIG. 10, under skip mode, the feature decoder down The characteristics of F const By directly substituting the predefined features in the PC'' down get.

[0084] Multi-scale coding The previous embodiment includes two scales of granularity, the base layer being a coarser point cloud PC 2 The first layer deals with encoding the fine geometric details, and the second layer deals with encoding the enhancement details. This two-scale encoding scheme may have limitations in practice.

[0085] In one embodiment, we extend our proposal to a three-scale encoding scheme. This is achieved by encapsulating our two-scale encoder (Fig. 3) and our two-scale decoder (Fig. 4) as a base encoder and a base decoder, respectively. In this case, three bitstreams corresponding to the three different scales of the input point cloud are generated by the codec.

[0086] Using the same principle, in another embodiment, we further extend our proposal to encoding schemes with four or more scales, which is achieved by recursively replacing the base encoder and the base decoder with our two-layer encoder (FIG. 3) and our two-layer decoder (FIG. 4).

[0087] Simplified feature codec In one embodiment, the feature codec is simplified and the set of features F is directly entropy coded / entropy decoded. In this case, the feature encoder takes only the set of features F as input. Inside the feature encoder, F is directly quantized and the bitstream BS 1 On the other hand, the feature decoder uses the BS 1 The feature decoder receives only the BS 1 is entropy decoded and then dequantized to obtain the decoded feature set F'. 1 In skip mode, where the enhancement layer is not available, the decoder will skip the enhancement layer and the final decoder output will be the coarse point cloud PC. 2 It is.

[0088] Improvements from feature aggregation modules The features generated in the feature encoder (FIG. 9) and the feature decoder (FIG. 10) can be further aggregated / refined by introducing an additional feature aggregation module. The feature aggregation module receives as input a sparse tensor with features having N channels, and then modifies the features to better fulfill the compression task. Note that the output features still have N channels, i.e., the feature aggregation module does not change the shape of the sparse tensor.

[0089] Integration of feature aggregation module The location of the feature aggregation module in the feature encoder and / or decoder may be changed. Also, the feature aggregation module may be included only on the encoder side, only on the decoder side, or both on the encoder side and the decoder side. In one embodiment, the feature encoder shown in FIG. 9 is adjusted with a feature aggregation module (926, 936) inserted after the downsampling operator of each downsampling processing block, as shown in FIG. 16. In one embodiment, the feature decoder shown in FIG. 10 is adjusted with a feature aggregation module (1046, 1066) inserted after each coordinate pruning operator, as shown in FIG. 17. It should be noted that by placing the feature aggregation module after (and not before) the coordinate pruning operator, the computational cost can be reduced. Compared to FIG. 10, in FIG. 17, the coordinate pruning and feature aggregation modules are absorbed into the preceding upsampling processing block.

[0090] In another embodiment, instead of having only one feature aggregation module in each upsampling / downsampling processing block, several feature aggregation modules can be cascaded to achieve better compression performance, as shown in FIG. 18.

[0091] Design of feature aggregation module There are different design options for the feature aggregation module. In one embodiment, it uses a Transformer architecture similar to the Voxel Transformer as described in the paper "Voxel transformer for 3D object detection" by Mao, Jiageng et al. (Proceedings of the IEEE / CVF International Conference on Computer Vision, 2021). A diagram of the Transformer block is shown in FIG. 19, which consists of a Self-Attention block (1910) with residual connections (1920) and an MLP block (consisting of MLP layers 1930) with residual connections (1940). Details of the Self-Attention block are described below.

[0092] The current feature vector f associated with the voxel location A, A and voxel position A i The k features f related to A - i (Here, A i (0≦i≦k-1) are the k nearest neighbors of A in the input sparse tensor. The attention block then calculates all the neighboring features f A Based on the feature f Ai First, we try to update the query embedding Q for A. A is calculated by the following formula: Q A =MLP Q (f A )

[0093] Then, all the nearest neighbor key embeddings of A, K Ai and value embedding V Ai is calculated by the following formula:

[0094]

number

[0095]

number

[0096]

number

[0097] The Transformer block updates the features for all occupied positions in the sparse tensor in the same way, and then outputs the updated sparse tensor. Q (·), MLP K (·), MLP V (·), and MLP P Note that (·) may contain only one fully connected layer, which corresponds to a linear projection.

[0098] In another embodiment, the feature aggregation module uses an Inception-ResNet (IRN) architecture (see Wang, Jianqiang et al., "Multiscale point cloud geometry compression," 2021 Data Compression Conference (DCC), IEEE, 2021), as shown in FIG. 20. In this example, we show the architecture of an IRN block for aggregating features with D channels. Again, "CONV N" represents a 3D convolutional layer with N output channels.

[0099] In another embodiment, the feature aggregation module uses a ResNet architecture (see He, Kaiming et al., “Deep residual learning for image recognition,” Proceedings of the IEEE conference on computer vision and pattern recognition, 2016) as shown in Figure 21. In this example, we show the architecture of the ResNet block for aggregating features with D channel.

[0100] Various numerical values ​​are used in this application. The specific values ​​are for illustrative purposes and the described aspects are not limited to these specific values.

[0101] Various methods are described herein, each of which includes one or more steps or acts for achieving the described method. The order and / or use of specific steps and / or acts may be modified or combined, unless a specific order of steps or acts is required for proper operation of the method. It should be noted that terms such as "first", "second", etc. may be used in various embodiments to modify elements, components, steps, operations, etc., such as, for example, "first decode" and "second decode". The use of such terms does not imply any ordering to the modified operations, unless specifically required. Thus, in this embodiment, the first decode need not be performed before the second decode, but may occur, for example, before, during, or during a period of overlap with the second decode.

[0102] The implementations and aspects described herein may be implemented in, for example, a method or process, an apparatus, a software program, a data stream, or a signal. Even if only discussed in the context of a single implementation (e.g., discussed only as a method), the implementation of the discussed features may also be implemented in other forms (e.g., an apparatus or a program). An apparatus may be implemented in, for example, appropriate hardware, software, and firmware. The method may be implemented in, for example, an apparatus such as a processor, which generally refers to processing devices including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include, for example, communication devices such as computers, mobile phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate communication of information between end users.

[0103] References to "one embodiment" or "an embodiment" or "one implementation" or "an implementation," as well as other variations thereof, mean that a particular feature, structure, characteristic, etc. described in connection with that embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" or "in one implementation" or "in an implementation," as well as other variations, appearing in various places throughout this application are not necessarily all referring to the same embodiment.

[0104] It should be noted that the application may refer to "determining" various information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from memory.

[0105] Additionally, the application may refer to "accessing" various information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0106] It should be noted that the application may refer to "receiving" various information. Receiving, like "accessing," is intended to be a broad term. Receiving information may include, for example, one or more of accessing the information or retrieving the information (e.g., from a memory). Furthermore, "receiving" typically involves in some way an operation, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0107] For example, in the case of "A / B," "A and / or B," and "at least one of A and B," it should be understood that the use of any of the following " / ," "and / or," and "at least one of" is intended to encompass the selection of only the first enumerated alternative (A), or the selection of only the second enumerated alternative (B), or the selection of both alternatives (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such language is intended to encompass the selection of only the first enumerated alternative (A), or the selection of only the second enumerated alternative (B), or the selection of only the third enumerated alternative (C), or the selection of only the first and second enumerated alternatives (A and B), or the selection of only the first and third enumerated alternatives (A and C), or the selection of only the second and third enumerated alternatives (B and C), or the selection of all three alternatives (A and B and C). This can be expanded as many times as the items listed, as would be apparent to one of ordinary skill in this and related arts.

[0108] As will be apparent to one of ordinary skill in the art, implementations may generate a variety of signals formatted to carry information that may be, for example, stored or transmitted. Information may include, for example, instructions for performing a method or data generated by one of the described implementations. For example, a signal may be formatted to carry a bit stream of the described embodiments. Such a signal may be formatted, for example, as an electromagnetic wave (e.g., using a radio frequency portion of the spectrum) or as a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier wave with the encoded data stream. The information that the signal carries may be, for example, analog information or digital information. The signal may be transmitted by a variety of different wired or wireless links, as is well known. The signal may be stored in a processor-readable medium.

Claims

1. 1. A method for decoding point cloud data, comprising: Decoding a first version of the point cloud; obtaining a set of point-wise features for the first version of the point cloud; and obtaining refinement information for the first version of the point cloud from the point-by-point features; and and obtaining a second version of the point cloud based on the refinement information and the first version of the point cloud.

2. The method of claim 1 , wherein the first version of the point cloud is obtained based on a third version of the point cloud and refinement information related to the third version of the point cloud.

3. Obtaining the pointwise feature set comprises: Obtaining a set of features; and upsampling the set of features to form an upsampled set of features.

4. obtaining coordinate information indicative of which points are within the first version of the point cloud; The method of claim 3 , further comprising pruning the upsampled features to form a refined feature set in response to the coordinate information.

5. The method of claim 4 , wherein obtaining the coordinate information comprises downsampling the first version of the point cloud.

6. The method according to any one of claims 3 to 5, wherein the features are decoded from a bitstream.

7. The method according to any one of claims 3 to 5, wherein said set of features is immutable.

8. Obtaining the refinement information A method according to any preceding claim, comprising converting each feature in the point-wise feature set into one or more 3D points.

9. The method of claim 8 , further comprising removing points from the one or more 3D points according to their distance from an origin.

10. The method of claim 9 , wherein the distance is adaptive to a quantization step size used in the inverse quantization.

11. The method of any one of claims 3 to 10, wherein the upsampling comprises tensor upsampling and convolutional layers.

12. The method according to any one of claims 3 to 10, wherein the upsampling comprises at least a deconvolution layer.

13. The method according to any one of claims 3 to 10, wherein said upsampling applies feature aggregation.

14. The method of claim 13 , wherein the feature aggregation is based on a Transformer module, an Inception-ResNet module, or a ResNet module.

15. The method of claim 13 or 14, wherein the feature aggregation is applied after the pruning of the upsampled features.

16. 1. A method for encoding point cloud data, comprising the steps of: encoding a first version of the point cloud; reconstructing a second version of the point cloud; and obtaining refinement information based on the second version of the point cloud and the point cloud; obtaining point-by-point features for the second version of the point cloud from the refinement information; and and encoding the point-by-point features.

17. The method of claim 16 , wherein the refinement information corresponds to a residual component.

18. reconstructing a second version of the point cloud 18. A method according to claim 16 or 17, comprising dequantising the decoded first version of the point cloud to form the second version of the point cloud.

19. said encoding said first version of said point cloud; encoding a third version of the point cloud, the third version being a coarser version of the first version of the point cloud; reconstructing a fourth version of the point cloud from the third version of the point cloud; and obtaining further refinement information based on the first version of the point cloud and the fourth version of the point cloud; obtaining from the further refinement a further set of point-wise features for the fourth version of the point cloud; and and encoding said further pointwise feature set.

20. The obtaining of the refinement information comprises, for points in the second version of the point cloud: obtaining one or more nearest neighbors in the first version of the point cloud; and obtaining respective differences between the 3D coordinates of said point and the 3D coordinates of each of said one or more nearest neighbours.

21. A method according to any one of claims 16 to 20, further comprising obtaining the point-wise features for the second version of the point cloud from the refinement information using a point-based neural network architecture.

22. The method of claim 21 , wherein the point-based neural network architecture comprises PointNet.

23. encoding the pointwise features, A method according to any one of claims 16 to 22, comprising downsampling the point-wise features based on the second version of the point cloud.

24. forming a sparse tensor based on the point-by-point feature set and the second version of the point cloud, wherein the downsampling is performed on the sparse tensor.

24. The method of claim 23.

25. Obtaining a feature vector from the downsampled sparse tensor; and quantizing and encoding the feature vector; 25. The method of claim 24, further comprising:

26. The method of any one of claims 23 to 25, wherein there is more than one iteration of downsampling.

27. The method of any one of claims 16 to 26, wherein the residual components are indicative of geometric details of the first version of the point cloud.

28. A method according to any one of claims 16 to 27, further comprising encoding a signal indicating that the point-wise features are encoded.

29. The method of any one of claims 23 to 27, wherein the downsampling comprises tensor downsampling and convolutional layers.

30. The method according to any one of claims 23 to 27, wherein the downsampling comprises at least a convolutional layer with a stride of 2 or more.

31. The method according to any one of claims 23 to 27, wherein said downsampling applies feature aggregation.

32. The method of claim 31 , wherein the feature aggregation is based on a Transformer module, an Inception-ResNet module, or a ResNet module.

33. An apparatus comprising one or more processors and at least one memory coupled to said one or more processors, said one or more processors configured to execute a method according to any one of claims 1 to 32.

34. A signal comprising point cloud data formed by carrying out a method according to any one of claims 16 to 32.

35. A computer readable storage medium storing instructions for encoding or decoding a point cloud according to the method of any one of the preceding claims.

Images