Deep wavelet point cloud attribute compression

Abstract

A method and an apparatus for compressing point cloud attributes are provided. The method and apparatus are based on a wavelet-like architecture that uses one or more lifting stages. Data representative of at least one attribute of a point cloud are obtained and transformed into at least two latents by applying at least one neural network-based lifting stage. The at least two latents are then entropy encoded. A non-transitory computer readable medium storing coded data representative of the at least two latents is also provided. The at least two latents comprise at least one latent comprising residual prediction between a first set of the at least one attribute and a prediction obtained from the second set of the at least one attribute, and at least one latent comprising an updated version of the second set of the at least one attribute obtained using the residual prediction.

Description

[0001] DEEP WAVELET POINT CLOUD ATTRIBUTE COMPRESSION

[0002] This application claims the priority to European Application No. 24305149.7, filed on 29 January 2024, which is incorporated herein by reference in its entirety.

[0003] TECHNICAL FIELD

[0004] The present embodiments generally relate to point cloud attribute compression, and more particularly to deep wavelet-based architecture for encoding point cloud attribute.

[0005] BACKGROUND

[0006] The use of 3D applications is becoming more popular every day, and to be able to exploit said applications different data formats are being used. One of the main data formats are point clouds. Point clouds (PCs) are a set of unordered points with coordinates x, y, z, corresponding to the location of the point in the space and its attributes (colors, normal vectors, etc.).

[0007] The use of this new types of data requires the creation of new compression methods to efficiently store and transmit data, especially since point clouds can have millions of points.

[0008] Different methods have been studied, and between those methods, learning based architectures are gaining strength. Those architectures are often extensions of learning-based methods that were already explored in the 2D image domain.

[0009] The use of Variational Auto Encoders (VAEs) (for example as in J. Wang, Z. Ma, H. Wei, Y. Yu, V. Zakharchenko and D. Wang, [AI-3DGC] Point Cloud Attribute Compression using Sparse Tensor-Representation, Moving Pictures Expert Group, 2022) and Normalizing Flows (NFs) (for example as in A. Borba Pinheiro, J.-E. Marvie, G. Valenzise andF. Dufaux, "NF- PCAC: Normalizing Flow Based Point Cloud Attribute Compression, " in ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023) as compression architectures have been explored in the point cloud domain; however, these methods are not really interpretable and have some disadvantages when it comes to performance in high bitrates in the VAE case and model size and complexity in the NF case.

[0010] Another type of architecture that uses a learning-based wavelet-like architecture has been explored in the 2D image compression domain, such as in H. Ma, D. Liu, H. Li and F. Wu, "End-to-End Optimized Versatile Image Compression With Wavelet-Like Transform, " IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, pp. 1247-1263, 2022.

[0011] The application of learning-based algorithms for point cloud attribute compression was not extensively explored in the literature, the two main works on that specific case-scenario are VAE and NF as mentionned above. And neither manages to outperform the state-of-the-art GPCC algorithm for attribute compression.

[0012] SUMMARY

[0013] According to an aspect, a method for encoding point cloud attributes is provided. The method comprises obtaining data representative of at least one attribute of a point cloud, transforming the data representative of the at least one attribute of the point cloud into at least two latents by applying at least one neural network-based lifting stage, encoding the at least two latents.

[0014] According to another aspect, an apparatus for encoding point cloud attributes is provided. The apparatus comprises one or more processors operable to obtain data representative of at least one attribute of a point cloud, transform the data representative of the at least one attribute of the point cloud into at least two latents by applying at least one neural network-based lifting stage, encode the at least two latents.

[0015] According to an aspect, a method for decoding point cloud attributes is provided. The method comprises decoding at least two latents representative of at least one attribute of a point cloud and transforming the at least two latents into data representative of the at least one attribute of the point cloud by applying at least one neural network-based lifting stage.

[0016] According to another aspect, an apparatus for decoding point cloud attributes is provided. The apparatus comprises one or more processors operable to decode at least two latents representative of at least one attribute of a point cloud and transform the at least two latents into data representative of the at least one attribute of the point cloud by applying at least one neural network-based lifting stage.

[0017] Further embodiments that can be used alone or in combination are described herein.

[0018] One or more embodiments also provide a computer program comprising instructions which when executed by one or more processors cause the one or more processors to perform any one of the methods for encoding or decoding point cloud attributes according to any of the embodiments described herein. One or more of the present embodiments also provide a non- transitory computer readable medium and / or a computer readable storage medium having stored thereon instructions for encoding or decoding point cloud attributes according to the methods described herein.

[0019] One or more embodiments also provide a non-transitory computer readable storage medium having stored thereon a bitstream generated according to the methods described herein. One or more embodiments also provide a method and apparatus for transmitting or receiving the bitstream generated according to the methods described above.

[0020] BRIEF DESCRIPTION OF THE DRAWINGS

[0021] FIG. 1A illustrates a block diagram of a system within which aspects of the present embodiments may be implemented according to an embodiment.

[0022] FIG. IB illustrates a block diagram of a system within which aspects of the present embodiments may be implemented according to another embodiment.

[0023] FIG. 1C illustrates a block diagram of a system within which aspects of the present embodiments may be implemented according to another embodiment.

[0024] FIG. 2A illustrates an example of a wavelet like architecture for compressing point cloud attributes according to an embodiment.

[0025] FIG. 2B illustrates an example of a method for encoding point cloud attributes according to an embodiment, for example using the wavelet like architecture illustrated on FIG. 2A.

[0026] FIG. 2C illustrates an example of a method for decoding point cloud attributes according to an embodiment, for example using the wavelet like architecture illustrated on FIG. 2A.

[0027] FIG. 3 illustrates an example of a dense block that can be used for a feature extraction block in the architecture illustrated on FIG. 2A.

[0028] FIG. 4 illustrates an example of a lift module on the encoder side, according to an embodiment.

[0029] FIG. 5 illustrates an example of a splitting according to a Morton order.

[0030] FIG. 6 illustrates an example of several lifting steps inside a lifting block, according to an embodiment.

[0031] FIG. 7A illustrates an example of a prediction operation in a lifting block according to an embodiment.

[0032] FIG. 8 illustrates an example of a splitting process according to an embodiment.

[0033] FIG. 9 illustrates an example of changes of a tensor stride for convolution in a prediction or update operation in a lifting block according to an embodiment.

[0034] FIG. 10 illustrates an example of a masked convolution for an entropy coding according to an embodiment.

[0035] FIG. 11 illustrates an example of an entropy encoder and entropy decoder that use mean and scale hyperprior, according to an embodiment.

[0036] FIG. 12 illustrates an example of an entropy encoder and entropy decoder that use context model, according to an embodiment. FIG. 13 illustrates an example of an entropy encoder and entropy decoder that use context model and hyperprior, according to an embodiment.

[0037] FIG. 14 illustrates an example of an architecture for entropy encoding or entropy decoding using context model, according to an embodiment.

[0038] FIG. 15A illustrates an example of an architecture for entropy encoding using hyperprior, according to an embodiment.

[0039] FIG. 15B illustrates an example of an architecture for entropy decoding using hyperprior, according to an embodiment.

[0040] FIG. 16 illustrates an example of a lift module on the decoder side, according to an embodiment. FIG. 17 illustrates an example of an hyperprior context encoder and decoder (HCE used in FIG. 19), according to an embodiment.

[0041] FIG. 18 illustrates an overview of the architecture, according to an embodiment.

[0042] FIG. 19 shows two remote devices communicating over a communication network in accordance with an example of the present principles.

[0043] FIG. 20 shows the syntax of a signal in accordance with an example of the present principles.

[0044] DETAILED DESCRIPTION

[0045] This application describes a variety of aspects, including tools, features, embodiments, models, approaches, etc. Many of these aspects are described with specificity and, at least to show the individual characteristics, are often described in a manner that may sound limiting. However, this is for purposes of clarity in description, and does not limit the application or scope of those aspects. Indeed, all of the different aspects can be combined and interchanged to provide further aspects. Moreover, the aspects can be combined and interchanged with aspects described in earlier filings as well.

[0046] The aspects described and contemplated in this application can be implemented in many different forms. FIG. 1A, IB and 1C below provide some embodiments, but other embodiments are contemplated and the discussion of FIGs. 1A, IB, 1C does not limit the breadth of the implementations. At least one of the aspects generally relates to point cloud encoding and decoding, and at least one other aspect generally relates to transmitting a bitstream generated or encoded. These and other aspects can be implemented as a method, an apparatus, a computer readable storage medium having stored thereon instructions for encoding or decoding point cloud data according to any of the methods described, and / or a computer readable storage medium having stored thereon a bitstream generated according to any of the methods described.

[0047] In the present application, the terms “reconstructed” and “decoded” may be used interchangeably, the terms “pixel” and “sample” may be used interchangeably, the terms “image,” “picture” and “frame” may be used interchangeably.

[0048] Various methods are described herein, and each of the methods comprises one or more steps or actions for achieving the described method. Unless a specific order of steps or actions is required for proper operation of the method, the order and / or use of specific steps and / or actions may be modified or combined. Additionally, terms such as “first”, “second”, etc. may be used in various embodiments to modify an element, component, step, operation, etc., such as, for example, a “first decoding” and a “second decoding”. Use of such terms does not imply an ordering to the modified operations unless specifically required. So, in this example, the first decoding need not be performed before the second decoding, and may occur, for example, before, during, or in an overlapping time period with the second decoding.

[0049] The present aspects are not limited to MPEG Point cloud attribute compression standard, and can be applied, for example, to other standards and recommendations, whether pre-existing or future-developed, and extensions of any such standards and recommendations. Unless indicated otherwise, or technically precluded, the aspects described in this application can be used individually or in combination.

[0050] FIG. 1A-1C illustrates block diagrams of examples of systems in which various aspects and embodiments can be implemented. Any one of the systems 100 A, 100B or 100B may be embodied as a device including the various components described below and is configured to perform one or more of the aspects described in this application. Examples of such devices, include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. In various embodiments, the system 100 A, 100B or 100C is communicatively coupled to other systems, or to other electronic devices, via, for example, a communications bus or through dedicated input and / or output ports. In various embodiments, the system 100 A, 100B or 100C is configured to implement one or more of the aspects described in this application.

[0051] FIG. 1 A illustrates a block diagram of an example of a system in which various aspects and embodiments can be implemented. The system 100A includes at least one processor 110 configured to execute instructions loaded therein for implementing, for example, the various aspects described in this application. Processor 110 may include embedded memory, input output interface, and various other circuitries as known in the art. The system 100 A includes at least one memory 120, e.g., a volatile memory device, and / or a non-volatile memory device, including, but not limited to, EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic disk drive, and / or optical disk drive. The memory 120 may include an internal storage device, an attached storage device, and / or a network accessible storage device, as non-limiting examples. The processor 110 may be interconnected to the memory 120 by an interconnection bus 115.

[0052] Program code to be loaded onto processor 110 to perform the various aspects described in this application is subsequently loaded onto memory 120 for execution by processor 110.

[0053] In some embodiments, memory inside of the processor 110 is used to store program code instructions and to provide working memory for processing that is needed during encoding or decoding. The input to the elements of system 100 A may be provided through various input devices (not represented). Both Processor 110 and memory 120 can also have one or more additional interconnections to external connections.

[0054] FIG. IB illustrates a block diagram of an example of a system 100B in which various aspects and embodiments can be implemented. The system 100B includes the processor 110 and memory 120 as described in relation with FIG. 1 A. The input to the elements of system 100B may be provided through various input devices as indicated in block 105 which is described further below with FIG. 1C. Such input devices include, but are not limited to, (i) a radio frequency (RF) portion that receives an RF signal transmitted, for example, over the air by a broadcaster, (ii) a Component (COMP) input terminal (or a set of COMP input terminals), (iii) a Universal Serial Bus (USB) input terminal, and / or (iv) a High Definition Multimedia Interface (HDMI) input terminal. Other examples, not shown in FIG. IB, include composite video.

[0055] The various elements may be interconnected and transmit data therebetween using suitable connection arrangement 115, for example, an internal bus as known in the art, including the I2C bus, wiring, and printed circuit boards.

[0056] The system 100B includes communication interface 150 that enables communication with other devices via communication channel 190. The communication interface 150 may include, but is not limited to, a transceiver configured to transmit and to receive data over communication channel 190. The communication interface 150 may include, but is not limited to, a modem or network card and the communication channel 190 may be implemented, for example, within a wired and / or a wireless medium.

[0057] The system 100B may provide an output signal to various output devices, including a display, speakers, and other peripheral devices. The output devices may be communicatively coupled to system 100B via dedicated connections through respective interfaces 160, 170, and 180. Alternatively, the output devices may be connected to system 100B using the communications channel 190 via the communications interface 150.

[0058] FIG. 1C illustrates a block diagram of an example of a system 100C in which various aspects and embodiments can be implemented according to another embodiment. Elements of system 100C, singly or in combination, may be embodied in a single integrated circuit, multiple Ics, and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 100C are distributed across multiple Ics and / or discrete components.

[0059] The system 100C includes the processor 110 and memory 120 as described in relation with FIG. 1 A or IB.

[0060] System 100C includes a storage device 140, which may include non-volatile memory and / or volatile memory, including, but not limited to, EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic disk drive, and / or optical disk drive. The storage device 140 may include an internal storage device, an attached storage device, and / or a network accessible storage device, as non-limiting examples.

[0061] System 100C includes an encoder / decoder module 130 configured, for example, to process data to provide an encoded video or decoded video, and the encoder / decoder module 130 may include its own processor and memory. The encoder / decoder module 130 represents module(s) that may be included in a device to perform the encoding and / or decoding functions. As is known, a device may include one or both of the encoding and decoding modules. Additionally, encoder / decoder module 130 may be implemented as a separate element of system 100C or may be incorporated within processor 110 as a combination of hardware and software as known to those skilled in the art.

[0062] Program code to be loaded onto processor 110 or encoder / decoder 130 to perform the various aspects described in this application may be stored in storage device 140 and subsequently loaded onto memory 120 for execution by processor 110. In accordance with various embodiments, one or more of processor 110, memory 120, storage device 140, and encoder / decoder module 130 may store one or more of various items during the performance of the processes described in this application. Such stored items may include, but are not limited to, the input data (image, video, volumetric content), the decoded data (image, video, volumetric content) or portions of the decoded data, the bitstream, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.

[0063] In some embodiments, memory inside of the processor 110 and / or the encoder / decoder module 130 is used to store instructions and to provide working memory for processing that is needed during encoding or decoding. In other embodiments, however, a memory external to the processing device (for example, the processing device may be either the processor 110 or the encoder / decoder module 130) is used for one or more of these functions. The external memory may be the memory 120 and / or the storage device 140, for example, a dynamic volatile memory and / or a non-volatile flash memory. In several embodiments, an external non-volatile flash memory is used to store the operating system of a television. In at least one embodiment, a fast external dynamic volatile memory such as a RAM is used as working memory for data encoding and decoding operations, such as for MPEG-2, HEVC (HEVC refers to High Efficiency Video Coding, also known as H.265 and MPEG-HPart 2), or VVC (Versatile Video Coding also known as H.266, standard developed by JVET, the Joint Video Experts Team).

[0064] The input to the elements of system 100C may be provided through various input devices as indicated in block 105, also mentioned in FIG. IB. Such input devices of system 100B or 100C include, but are not limited to, (i) a radio frequency (RF) portion that receives an RF signal transmitted, for example, over the air by a broadcaster, (ii) a Component (COMP) input terminal (or a set of COMP input terminals), (iii) a Universal Serial Bus (USB) input terminal, and / or (iv) a High Definition Multimedia Interface (HDMI) input terminal. Other examples, not shown in FIG. IB or 1C, include composite video.

[0065] In various embodiments, the input devices of block 105 in system 100B or 100C have associated respective input processing elements as known in the art. For example, the RF portion may be associated with elements suitable for (i) selecting a desired frequency (also referred to as selecting a signal, or band-limiting a signal to a band of frequencies), (ii) down converting the selected signal, (iii) band-limiting again to a narrower band of frequencies to select (for example) a signal frequency band which can be referred to as a channel in certain embodiments, (iv) demodulating the down converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select the desired stream of data packets. The RF portion of various embodiments includes one or more elements to perform these functions, for example, frequency selectors, signal selectors, band-limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF portion may include a tuner that performs various of these functions, including, for example, down converting the received signal to a lower frequency (for example, an intermediate frequency or a near-baseband frequency) or to baseband. In one set-top box embodiment, the RF portion and its associated input processing element receives an RF signal transmitted over a wired (for example, cable) medium, and performs frequency selection by filtering, down converting, and filtering again to a desired frequency band. Various embodiments rearrange the order of the above-described (and other) elements, remove some of these elements, and / or add other elements performing similar or different functions. Adding elements may include inserting elements in between existing elements, for example, inserting amplifiers and an analog-to- digital converter. In various embodiments, the RF portion includes an antenna.

[0066] Additionally, the USB and / or HDMI terminals may include respective interface processors for connecting system 100B or 100C to other electronic devices across USB and / or HDMI connections. It is to be understood that various aspects of input processing, for example, Reed- Solomon error correction, may be implemented, for example, within a separate input processing IC or within processor 110 as necessary. Similarly, aspects of USB or HDMI interface processing may be implemented within separate interface ICs or within processor 110 as necessary. The demodulated, error corrected, and demultiplexed stream is provided to various processing elements, including, for example, processor 110, and encoder / decoder 130 operating in combination with the memory and storage elements to process the data stream as necessary for presentation on an output device.

[0067] Various elements of the systems 100A, 100B or 100C may be provided within an integrated housing, Within the integrated housing, the various elements may be interconnected and transmit data therebetween using the suitable connection arrangement 115, for example, an internal bus as known in the art, including the I2C bus, wiring, and printed circuit boards.

[0068] Similarly as for the system 100B of FIG. IB, the system 100C includes communication interface 150 that enables communication with other devices via communication channel 190. The communication interface 150 may include, but is not limited to, a transceiver configured to transmit and to receive data over communication channel 190. The communication interface 150 may include, but is not limited to, a modem or network card and the communication channel 190 may be implemented, for example, within a wired and / or a wireless medium. Data is streamed to the system 100B or 100C, in various embodiments, using a Wi-Fi network such as IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). The Wi-Fi signal of these embodiments is received over the communications channel 190 and the communications interface 150 which are adapted for Wi-Fi communications. The communications channel 190 of these embodiments is typically connected to an access point or router that provides access to outside networks including the Internet for allowing streaming applications and other over-the-top communications. Other embodiments provide streamed data to the system 100B or 100C using a set-top box that delivers the data over the HDMI connection of the input block 105. Still other embodiments provide streamed data to the system 100B or 100C using the RF connection of the input block 105. As indicated above, various embodiments provide data in a non-streaming manner. Additionally, various embodiments use wireless networks other than Wi-Fi, for example a cellular network or a Bluetooth network. The system 100C may provide an output signal to various output devices, including a display 165, speakers 175, and other peripheral devices 185. The display 165 of various embodiments includes one or more of, for example, a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. The display 165 can be for a television, a tablet, a laptop, a cell phone (mobile phone), or other devices. The display 165 can also be integrated with other components (for example, as in a smart phone), or separate (for example, an external monitor for a laptop). The other peripheral devices 185 include, in various examples of embodiments, one or more of a stand-alone digital video disc (or digital versatile disc) (DVR, for both terms), a disk player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 185 that provide a function based on the output of the system 100C. For example, a disk player performs the function of playing the output of the system 100C.

[0069] In various embodiments, control signals are communicated between the system 100C and the display 165, speakers 175, or other peripheral devices 185 using signaling such as AV.Link, CEC, or other communications protocols that enable device-to-device control with or without user intervention. The output devices may be communicatively coupled to system 100C via dedicated connections through respective interfaces 160, 170, and 180. Alternatively, the output devices may be connected to system 100C using the communications channel 190 via the communications interface 150. The display 165 and speakers 175 may be integrated in a single unit with the other components of system 100C in an electronic device, for example, a television. In various embodiments, the display interface 160 includes a display driver, for example, a timing controller (T Con) chip.

[0070] The display 165 and speaker 175 may alternatively be separate from one or more of the other components, for example, if the RF portion of input 105 is part of a separate set-top box. In various embodiments in which the display 165 and speakers 175 are external components, the output signal may be provided via dedicated output connections, including, for example, HDMI ports, USB ports, or COMP outputs.

[0071] In any of the systems 100A, 100B or 100C, the embodiments can be carried out by computer program product comprising code instructions that implements any one of embodiments described herein. The computer program product may be computer software implemented by the processor 110 or by hardware, or by a combination of hardware and software. As a nonlimiting example, the embodiments can be implemented by one or more integrated circuits. The memory 120 of any one of the systems 100A, 100B or 100C can be of any type appropriate to the technical environment and can be implemented using any appropriate data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory, as non-limiting examples. The processor 110 of any one of the systems 100 A, 100B or 100C can be of any type appropriate to the technical environment, and can encompass one or more of microprocessors, general purpose computers, special purpose computers, and processors based on a multi-core architecture, as non-limiting examples.

[0072] Some embodiments described herein provide for compressing point cloud attributes using learning-based wavelet-like architecture. The wavelet-like architecture uses a predict-update block to produce a latent space the same size as the input point cloud (PC). The latent space is a representation of the input with different coefficients. A goal is to produce a latent space that is easier to compress than the original input.

[0073] Like in the normalizing flow architecture cited previously, the wavelet-like architecture provides invertible transformations, meaning that the blocks used in the network are completely invertible and can be used also in a lossless approach.

[0074] Wavelets can be used in image compression to code different coefficients sub bands of an image. The wavelet-like architecture is adapted herein to the domain of 3D point cloud compression.

[0075] An aim of some embodiments provided herein is to have an architecture that is more interpretable in the sense of signal processing and offer an alternative to models such as NFs and VAEs.

[0076] The Wavelet-Like architecture provided herein, which can be named Lift-PCAC in the following, performs several transformations on the data, producing a latent space which is encoded. The architecture performs a split of the data, and a predict-update operation.

[0077] FIG. 2A illustrates an example of an architecture to code Point Cloud attributes using a learning-based approach, according to an embodiment. The architecture comprises a feature extaction (Feat Extraction) module that processes an input Point Cloud or point cloud attributes and provides the extracted features to a lifting module (lift module). The lifting module outputs one or more latents that are quantized (Q) and arithmetically encoded (AE) using for example an hyperprior encoder / decoder and context model. On the decoder, the encoded latents are arithmetically decoded and provided to the lifting module and then to the feature extraction module to reconstruct the point cloud or point cloud attributes.

[0078] The learning-based wavelet framework is adapted to the compression of 3D point cloud color attributes. The architecture of FIG.2A, is further described below in the order of the path that is shown in FIG.2A: from the input PC until the Reconstructed one on the decoder side of the architecture.

[0079] FIG. 2B illustrates an example of a method 200 for encoding point cloud attributes according to an embodiment, for example using the wavelet like architecture illustrated on FIG. 2A. At

[0080] 210, data representative of at least one attribute of an input point cloud is obtained, for example using the feature extraction module of FIG. 2A. At 220, the data is transformed with one or more lifting stages that use trained neural networks to produce a set of latents or subbands. At 230, the latents or subbands are encoded into a stream for storage or transmission, for example with quantization and entropy encoding.

[0081] FIG. 2C illustrates an example of a method 201 for decoding point cloud attributes according to an embodiment, for example using the wavelet like architecture illustrated on FIG. 2A. At

[0082] 211, a stream that comprises the encoded latents or subbands is decoded, for example with entropy decoding and dequantization. At 221, the latents or subbands are transformed with one or more lifting stages that use trained neural networks to produce the data representative of the attributes of the point cloud. At 231, the attributes of the point cloud are reconstructed from the transformed data.

[0083] The architecture and methods described herein for compressing attributes of a point cloud can be implemented in a point cloud compression system, for example in a G-PCC or V-PCC coding standards or any other point cloud compression system. As shown on FIG. 2A, according to an embodiment, the architecture comprises a feature extraction block, a Lift Module on both the encoder and decoder, a hyperprior encoder and a hyperprior decoder.

[0084] A goal of the feature extraction block is to help extracting more non-linear features from the input point cloud. An example of a feature extraction block is illustrated on FIG. 3. The feature extraction block is composed of several convolutions in a dense block architecture. The dense block has the goal of preserving initial features along the convolutions by concatenating the output of previous convolutions in the output of current convolution. This block has been proven to enhance performance of learning-based architectures. The feature extraction block illustrated on FIG. 3 can be used in the architecture of FIG. 2A to produce better features for the Lift Module, which is the core of the provided network. The same architecture is used on the decoder side to reconstruct the point cloud.

[0085] The lift module used in the architecture of FIG. 2A is a neural network architecture with different blocks. It produces as outputs several latent spaces or subbands, which are then processed by an hyperprior encoder / decoder to be sent to the decoder side of the architecture. These latent spaces are the equivalent of sub-bands in the conventional wavelet transforms (in a loosely sense, given the nonlinear filters here might not have a band-pass behavior). In the present document, the terms latents, latent spaces or subbands can be used interchageably. A goal of the lift module is to create the coefficients which are going to be encoded.

[0086] An example of a lift module that can be used in the architecutre of FIG. 2A is illustrated on FIG. 4. The division performed in the Split block divides the point cloud attributes into 2 sets of point cloud attributes with half the number of significant points as the input one.

[0087] The Lift Module is composed of several Lift Blocks subsequently applied to one output of a preceding lift block. The number of Lift Blocks, and thus the number of decomposition levels, can be variable, and any number can be chosen, as long as it fits the memory and is smaller than the resolution of the Point Cloud.

[0088] For example, 4 lifting blocks can be used. In this example, this means that the number of points in the smaller subbands is the original number of points in the PC divided by 16.

[0089] A first operation in the Lift Block is the split. The splitting of the PC separates the points in two downsampled point clouds, called Xo for odds and Xe for evens. After producing Xo and Xe, their corresponding binary masks are also obtained. Such binary masks indicate where the points of the downsampled point clouds Xo and Xe are located. In other words, a binary mask for Xo, respectively Xe, indicates for a location in the input point cloud whether or not the location is filled or not.

[0090] The splitting can be performed in several different ways, as long as the neighbor points stay separated during the downsampling process.

[0091] In a variant, the splitting can be done by scanning the points of the point cloud according to a Morton Order, also known as Z-order. According to this variant, the points are ordered in the Point Cloud according to their Morton order, by mapping the data in a one dimensional way and dividing it by separating it into Xo and Xe according to their index in the ordered PC, as illustrated on FIG. 5 showing an example of a mortong code splitting of a 2D point cloud. For example points with odd index go into Xo and points with even index go into Xe.

[0092] In another variant, the splitting can be done according to a KD Tree splitting. In this variant, the space or bounding box bounding the point cloud is split until each obtained split units comprises only one pair of points in the same unit. One point of the pair is then put into Xo and the other point of the pair is put into Xe.

[0093] In another variant, the splitting can be done using a Farthest Point Sampling. According to this variant, a random point is picked up from the PC as a first point of one set, for example Xo. Then, at each iteration, the process samples from the remaining points of the point cloud the point that is the farthest from the set of sampled points. The process is repeated until half the points opf the poit cloud is sampled. In that way the point cloud is split into the sampled points which form the set Xo and non-sampled points which form the set Xe.

[0094] It is to be noted that Xo and Xe in the splitting can be inversed, as long as the same splitting method is used for the subsequent blocks.

[0095] After producing the two point clouds Xe and Xo (they can also be called tensors in a learningbased context), the lifting operations are performed as illustrated on FIG. 5. The idea behind the lifting operation is to have a predict-update duo that produces two tensors as output with the same format and filled points as the tensors provided as input to the lifting operation. For example, on FIG. 4 and FIG. 5, the lift block comprises one lifting step that includes one prediction Po and one update Uo. In other examples, the lift block can have an arbitrary number of lifting operations: meaning the tensor can be predicted and updated several times before being sent to the next stage (next lift block or output of the lift module), as shown in FIG. 6 showing several lifting steps inside a lift block. As can be seen on FIG. 6, the lift block comprises n+1 lifting steps, each one comprises one prediction Pi and one update Ui with i=0, 1... n.

[0096] In the Predict transformation, represented by a Pi, with I being, 0, 1 or n in FIG. 5 or 6, an aim of the network Pi is to “predict” the points on the other side of the branch. This means in the example of FIG. 5 or 6 that the points in Xe are predicted with the points in Xo.

[0097] An example of an architecture for a predict transformation is illustrated on FIG. 7A. The network inside Pi can have any configuration, as long as it outputs the same number of significant coefficients that are on the other side of the network. In a variant, a small combination of sparse convolutions can be used.

[0098] From an implementation point of view, since the transformation is done in the sparse domain, masks are used to obtain the desired effect.

[0099] As can be seen on FIG. 7A, the split operation takes as a sparse input wherein black squares indicate occupied points or voxels of the point cloud while hatched squares indicate void or empty points or voxels (in other words voxels wherein there is no point cloud data). After the split, two tensors Xo and Xe with a same number of coefficients are provided as input tensors to the lifting operation. In the respective tensors Xo and Xe illustrated on FIG. 7A, black squares indicate occupied voxels in respectively Xo and Xe, hatched squares indicate empty voxels as in the input point cloud data, and white squares indicates occupied voxels in the other tensors (respectively Xe and Xo). Also, a first binary mask Be is produced with the same size as the tensor Xo and Xe. The first binary mask Be indicates using a first value (for example 1) the locations in tensor Xe that are occupied (black squares in Xe on FIG. 7A). In a variant, a second value (for example 0) can be used to indicate the locations that are occupied in tensor Xo (white squares in Xe on FIG. 7A).

[0100] A second binary mask Bo can be produced also to indicate the locations in tensor Xo that are occupied (black squares in Xo on FIG. 7A). The second binary mask Bo is complementary to the first binary mask Be.

[0101] On FIG. 7A, the tensor Xo is provided as input to the predict transformation Po, as well as the binary mask Be. The Predict transformation Po comprises a Neural Network (NN) and a filter (Filter).

[0102] The NN applies one or more sparse convolutions to Xo that predicts Xe from Xo. An example of an architecture for the prediction transformation is illustrated on FIG. 7B. FIG. 7B shows in a variant an IRN (Inception Residual Network) block. In other variants, the predict transformation can use any type of convolution in the block as long as the input and output have a same number of channels. The NN output is a tensor that comprises the predicted values. The filter uses the binary mask Be (Mask on FIG. 7A) to filter the coefficients of the NN output that are in Xe. This is done using a pointwise multiplication on all the points that were initially occupied. By doing this, the sparse convolution framework can be used to perform predictions without needing to create new tensors. The sparse convolution framework can be done using only binary masks.

[0103] After performing the transformation in Po, a pointwise subtraction is done between Xe and the output of Po, providing the resulting tensor X’eO on FIG. 5 and 6. The output of Po corresponds to the output of the NN and the filtering that occurs right after.

[0104] The exact same process is used for the second part of the lifting step: the update, represented by a Ui, with i being, 0, 1 or n in FIG. 5 or 6. But, the update takes as input the result of the subtraction (X’eO) and the second binary mask Bo obtained from Xo. The block Uo has the same configuration as Po, and also possesses a filter to match the result to Xo. The output of the block Uo is then summed with Xo, producing the tensor X’oO. The same architecture for the update transformation can be used as for the predict transformation, for example as illustrated on FIG. 7B.

[0105] After the lifting step, with both results X’oO and X’eO at hand, X’eO is sent to be encoded and te lifting process continues using X’oO as input to the next lifting block.

[0106] As illustrated on FIG. 4, the exact same process is repeated again, producing X’el and X’ol. X’el is sent to be encoded and the process continues with Xol as input to the new lifting block. This continues until the number of Lift blocks ends. For example, if the architecture comprises 5 lift blocks, the process is repeated 5 times and 6 tensors are output and sent to be encoded.

[0107] The NN blocks inside the predict transformation Pi and update transformation Ui have the same architecture and configuration in the successive lift blocks, except for the tensor stride that is performed in the convolutions. The stride of the convolutions defines how far a point needs to be to be considered as a neighbor. When the process goes deeper in the network, the point cloud has less and less significant coefficients, as shown in FIG. 8. As can be seen in FIG. 8, after several splitting, the point cloud has a smaller amount of significant coefficients that are further away from each other. This means that convolutions performed on such tensors have to look for neighborhood information further away.

[0108] Therefore, in a variant, the stride of the convolution used in the successive lift block increases with the decomposition level. After several splitting, the behavior of the convolutions changes, and the convolutions search for further away neighbors, as shown in FIG. 9. FIG. 9 illustrates an example of how the tensor stride changes how the convolution operates on the tensor. The top part of FIG. 9 illustrates how the convolution works when the tensor has a normal stride, for example 1, the neighbors are immediate neighbors (adjacent to a current point of the convolution). The bottom part of FIG. 9 illustrates the case where the kernel of the convolution looks for neighbors that are not immediate / adjacent neighbors to a current point of the convolution, for example using a stride of 2.

[0109] After going through all the lifting transformations, latents or sub-bands of the point cloud are prvied as output. These latents or subbands are to be encoded and sent to the decoder side. The latents or subbands are quantized and arithmetically encoded to produce a binary stream that can be stored or transmitted.

[0110] The quantization process is different during training and during inference of the model as will be explained further below. When performing inference in the model, the quantization process consists in a “round” of the coefficients in the sub-bands. The coefficients are then sent to the encoder with the side information that are needed for proper decoding.

[0111] To produce the side information that pilots the encoder, a hyperprior / context coding architecture can be used to produce the context information for the entropy encoder and decoder.

[0112] Hyperprior / Context Coding

[0113] After producing all the sub-bands (output of the lift module on FIG. 2A, or for example referenced by X’eO, X’el ...X’eN and X’oN from FIG. 4), the subbands are processed by a quantization and an entropy encoder. For example, the entropy encoder can use an hyperprior encoder (Hyper Encoder on FIG. 2A) to generate coefficients that pilot the entropy encoder and decoder.

[0114] This hyperprior encoder generates a side bitstream that is sent to the decoder side along with the bitstream that comprises the data encoding the subbands.

[0115] In a variant, the hyperprior encoder can include a context model to also help the entropy encoder with the past coefficients that were already coded in an autoregressive fashion. An example of such an entropy encoder and decoder is illustrated on FIG. 13 in the context of an image encoder. Only the right part of (in the dashed box) is used for the entropy encoder of FIG. 2A. In another variant, the hyperprior encoder does not include a context model, as illustrated on FIG. 11. In another variant, only the context model part can be used without the hyperprior, generating only one main bitstream, as illustrated on FIG. 12.

[0116] In an embodiment, the design of the hyperprior encoder can be the same as the one used in D. Minnen, J. Balle and G. Toderici, Joint Autoregressive and Hierarchical Priors for Learned Image Compression, arXiv, 2018 proposed for image compression. As mentionned above, three different versions can be used. One with only the context model (FIG. 12), one with a mean scale hyperprior (FIG. 11) or one with both together (FIG. 13). FIG. 11-13 are extracted from Appendix D of the mentionned document.

[0117] To adapt these entropy encoders / decoders to the case of point cloud used in the present document, all the convolutions used in the entropy encoder / decoder are replaced by Sparse Convolutions. The architecture of the context model block and hyperprior block are shown on FIG. 14 and 15A and 15B. FIG. 14 shows an example of a context model block with the first convolution is masked convolution that takes the context of previously coded features. FIG. 15A and 15B repsectively show an example of hyperprior architectures for the encoding (FIG. 15 A) and the decoding (FIG. 15B).

[0118] However, the convolutions shown in these blocks are only examles and other variants with other architecture can be used. As long as they produce the correct coefficients for the Entropy Encoder and Decoder (in FIG. 2A an arithmetic encoder / decoder). For example, the arithmetic encoder / decoder (AE) / AD) can be an rANS encoder and decoder. But other examples are possible and the entropy encoder can be changed.

[0119] Regarding the context model, it is to be noted that the first convolution in the block illustrated on FIG. 14 needs to be a masked convolution, as shown in FIG. 10. FIG. 10 illustrates a masked convolution wherein the coefficients that are not yet coded by the entropy encoder are not taken into consideration for the constitution of the context. As shown on FIG. 10, only the dotted squares of the kernel are used for obtaining the context in the convolution with the tensor to arithmetically encode / decode. This is because the AE does not have access to future coefficients to encode / decode.

[0120] One example of context model can be seen in FIG. 14. The architecture is similar as in J. Wang, Z. Ma, H. Wei, Y. Yu, V. Zakharchenko and D. Wang, [AI-3DGC] Point Cloud Attribute Compression using Sparse Tensor -Representation, Moving Pictures Expert Group, 2022, except from the size of the masked convolution. On an embodiment, the masked convolution used for the architecture proposed herein is a 7x7x7 to capture a larger context. In a variant wherein the hyperprior is used as well, the context is concatenated with the hyperprior the necessary information to code the coefficients are produced.

[0121] In a variant wherein there is no hyperprior, the prior information for the subsequent coefficient to be en coded / decoded is produced only by the previously encoded / decoded coefficients (from the context model).

[0122] One example of the architecture used in the Hyperprior Encoder and Decoder can be seen in FIG. 15 A and FIG.15B . Sparse convolutions are used that produce a latent that is encoded with a factorized entropy model, i.e. the factorized entropy model considers there is no correlation between the different coefficients of the latent.

[0123] The hyperprior architectures can be used in any of the combinations illustrated in FIG. 11-13. Since multiple subbands are obtained at the ouptu of the lift module encoder, depending on the size of the lift module encoder used, a different number of hyperprior / context architectures can be used. In one variant, one hyperprior / context architecture is used for each one of the subbands, in this way the subbands are coded separately. In another variant, a same hyperprior / context architecture can be used for all the subbands, in this way, the subbands are coded jointly.

[0124] Entropy Encoder and Decoder

[0125] In an embodiment, after producing the latents or subbands of the point cloud and producing the hyperparameters for piloting the entropy encoder and entropy decoder, an rANS encoder and decoder is used for arithmetic encoding and decoding to produce the stream. But it can be any entropy encoder. Another arithmetic encoder could also be used, provided that it can take as input the information generated in the hyperprior.

[0126] On the decoder side, the side information, if any, is decoded by the hyperprior decoder (Hyper Decoder on FIG. 2A) to provide the entropy parameters that pilot the arithmetic decoder. The stream that comprises the encoded latents / subbands is decoded using the arithmetic decoder and context model if any.

[0127] As shown on FIG. 2 A, the decoded latents / subbands are provided as inputs to the lift module decoder and then to the feature extraction module for reconstructing the point cloud attributes.

[0128] Reverse Lift Block On the reverse direction, the lift module has a different path that is followed by the tensors. The filters are the same, and the predict and update operators are also the same. The difference is the path followed by the tensors and the split module which is replaced by a merging module that has the opposite function, to get the two tensors and merge them into one. An example of the lift module on the decoder side is shown on FIG. 16: the Ifit blocks are used in the reverse order and instead of having a splitting operation, a merging operation is done after the lifting operations in each lift block which transforms Xe and Xo back into the input tensor. The merging operation used follows the same scanning of the points of the point cloud as the scanning order used in the splitting operation.

[0129] Complete Overview of the Framework

[0130] Considering the hyperprior / context encoder and decoder as the block defined in FIG. 17, remembering that any combination between Hyperprior and context model can be used as mentionned above, a detailed overview of the entire network in illustrated on FIG. 18.

[0131] HCE from FIG. 18 refers to the Hyperprior Context Encoder / Decoder illustrated on FIG. 17. On FIG. 18, the top part shows the encoder and the bottom part shows the decoder.

[0132] Training

[0133] The network as shown on FIG. 18 can be trained with any static point cloud dataset. In the training stage of the network, the quantization block is replaced for the sake of the back propagation needed to train the network. The quantization step is simulated as a random normal noise between [-0.5, 0.5], Since the entropy encoder and decoder are lossless, they are not modelized.

[0134] A rate distortion loss : L = D + lambda*R is used for training, where D is the distortion between the reconstructed point cloud and the input, for example Mean Square Error MSE can be used. R stands for the rate which is estimated in the entropy model, and lambda the Lagrangian parameter.

[0135] A RANS encoder / decoder is used for inference. The training uses partitions of size 128x128x128 of the original point clouds to fit the memory, but the point cloud can be an input of any size. The network can also be trained and can function in different modes. In a variant, the network is trained for RGB / YUV attributes in Full Channels mode. In another variant, the network is trained for RGB / YUV attributes in Single Channel mode.

[0136] In the Full Channels Mode, the network is trained to process all the channels in the point cloud at once, meaning the network ingests the 3 input channels (RGB or YUV) and the latents have 3 channels for every point. By doing this, the convolutional filters in the network need to have 3 input channels to ingest the data.

[0137] In the Single Channel Mode, the network is trained for one channel at once. The same coordinates of the point cloud are used and 3 “one channel” point clouds to be processed are produced. The network needs to be trained by using the same method as well. In the inference, the channels are not correlated to one another, therefore, one bitstream for each channel is obtained.

[0138] In the Single Channel mode, in a variant, one network can be trained per channel, that is one network specialized for the Red channel, one for Green and one for Blue (in the RGB case). In another variant, one single network that codes one channel is used for the 3 of them.

[0139] In the YUV case, one network trained for the luma channel and one network trained for the chroma channels can be used, reducing the number of networks trained to 2.

[0140] Signaling / Syntax

[0141] Current activities in MPEG AI-PCC works on defining some Al models for compression and decompression of Point Clouds geometry and photometry. The MPEG group currently handle photometry separately from the geometry. The architecture provided herein allows to use a Wavelet-type architecture instead of a Variational Auto-Encoder or a Normalizing Flow to perform the coding / decoding of the Photometry. Any of these solutions could be an option in the codec. Therefore, a signaling flag noticing which deep decoder architecture to use could be present in the bitstream.

[0142] If the Wavelet-type architecture is selected, any of the different variations of the architecture that could be implemented can be signaled. For example, the splitting method can be specified. The color space the architecture is trained for could also be specified, as well as the channel mode mentioned above.

[0143] In a variant, the number of lifting blocks that are used, the number of lift steps can also be signaled to the decoder. So that when several network configurations are possible, the correct network is selected on the decoder side. An example of signaling that can be used is illustrated on Table 1 below, one or more of the syntax elements can be used in the standard specification.

[0144] Table 1 - Signaling the use of the VAE, Normalizing Flow or Wavelet in PC Learning Based bitstream and decoding parameters

[0145] The architecture described herein provides a learning-based wavelet-like transform for point cloud attribute compression. Some embodiments provide modular network that can be size adjusted according to the type of data and also depending on how many blocks are used. In the wavelet-like architecture, sparse convolutions are used. The provided architecture can be applied for example to the G-PCC coding standard extension based on Deep Learning.

[0146] In an embodiment, illustrated in FIG. 19, in a transmission context between two remote devices A and B over a communication network NET, the device A comprises a processor in relation with memory RAM and ROM which are configured to implement a method for encoding a point cloud according to any one of the embodiments described herein and the device B comprises a processor in relation with memory RAM and ROM which are configured to implement a method for decoding a point cloud according to any one of the embodiments described herein. In accordance with an example, the network is a broadcast network, adapted to broadcast / transmit a coded point cloud from device A to decoding devices including the device B.

[0147] FIG. 20 shows an example of the syntax of a signal transmitted over a packet-based transmission protocol. Each transmitted packet P comprises a header H and a payload PAYLOAD. In some embodiments, the payload PAYLOAD may comprise data representative of point cloud attributes encoded according to any one of the embodiments described above. The payload can comprise encoded latents or subbands as well as any side information for hyperprior and signaling mentionned above.

[0148] Various implementations involve decoding. “Decoding”, as used in this application, can encompass all or part of the processes performed, for example, on a received encoded sequence in order to produce a final output suitable for display. In various embodiments, such processes include one or more of the processes typically performed by a decoder, for example, entropy decoding, inverse quantization, inverse transformation, and differential decoding. In various embodiments, such processes also, or alternatively, include processes performed by a decoder of various implementations described in this application, for example, entropy decoding a sequence of binary symbols to reconstruct image, video or point cloud data.

[0149] As further examples, in one embodiment “decoding” refers only to entropy decoding, in another embodiment “decoding” refers only to differential decoding, and in another embodiment “decoding” refers to a combination of entropy decoding and differential decoding, and in another embodiment “decoding” refers to the whole reconstructing picture process including entropy decoding. Whether the phrase “decoding process” is intended to refer specifically to a subset of operations or generally to the broader decoding process will be clear based on the context of the specific descriptions and is believed to be well understood by those skilled in the art.

[0150] Various implementations involve encoding. In an analogous way to the above discussion about “decoding”, “encoding” as used in this application can encompass all or part of the processes performed, for example, on an input video sequence in order to produce an encoded bitstream. In various embodiments, such processes include one or more of the processes typically performed by an encoder, for example, partitioning, differential encoding, transformation, quantization, and entropy encoding. In various embodiments, such processes also, or alternatively, include processes performed by an encoder of various implementations described in this application, for example, determining re-sampling filter coefficients, re-sampling a decoded picture.

[0151] As further examples, in one embodiment “encoding” refers only to entropy encoding, in another embodiment “encoding” refers only to differential encoding, and in another embodiment “encoding” refers to a combination of differential encoding and entropy encoding. Whether the phrase “encoding process” is intended to refer specifically to a subset of operations or generally to the broader encoding process will be clear based on the context of the specific descriptions and is believed to be well understood by those skilled in the art.

[0152] Note that the syntax elements as used herein, are descriptive terms. As such, they do not preclude the use of other syntax element names.

[0153] This disclosure has described various pieces of information, such as for example syntax, that can be transmitted or stored, for example. This information can be packaged or arranged in a variety of manners, including for example manners common in video standards such as putting the information into an SPS, a PPS, a NAL unit, a header (for example, a NAL unit header, picture header or a slice header), or an SEI message. Other manners are also available, including for example manners common for system level or application level standards such as putting the information into one or more of the following: a. SDP (session description protocol), a format for describing multimedia communication sessions for the purposes of session announcement and session invitation, for example as described in RFCs and used in conjunction with RTP (Real-time Transport Protocol) transmission. b. DASH MPD (Media Presentation Description) Descriptors, for example as used in DASH and transmitted over HTTP, a Descriptor is associated to a Representation or collection of Representations to provide additional characteristic to the content Representation. c. RTP header extensions, for example as used during RTP streaming. d. ISO Base Media File Format, for example as used in OMAF and using boxes which are object-oriented building blocks defined by a unique type identifier and length also known as 'atoms' in some specifications. e. HLS (HTTP live Streaming) manifest transmitted over HTTP. A manifest can be associated, for example, to a version or collection of versions of a content to provide characteristics of the version or collection of versions.

[0154] When a figure is presented as a flow diagram, it should be understood that it also provides a block diagram of a corresponding apparatus. Similarly, when a figure is presented as a block diagram, it should be understood that it also provides a flow diagram of a corresponding method / process.

[0155] Some embodiments refer to rate distortion optimization. In particular, during the encoding process, the balance or trade-off between the rate and distortion is usually considered, often given the constraints of computational complexity. The rate distortion optimization is usually formulated as minimizing a rate distortion function, which is a weighted sum of the rate and of the distortion. There are different approaches to solve the rate distortion optimization problem. For example, the approaches may be based on an extensive testing of all encoding options, including all considered modes or coding parameters values, with a complete evaluation of their coding cost and related distortion of the reconstructed signal after coding and decoding. Faster approaches may also be used, to save encoding complexity, in particular with computation of an approximated distortion based on the prediction or the prediction residual signal, not the reconstructed one. Mix of these two approaches can also be used, such as by using an approximated distortion for only some of the possible encoding options, and a complete distortion for other encoding options. Other approaches only evaluate a subset of the possible encoding options. More generally, many approaches employ any of a variety of techniques to perform the optimization, but the optimization is not necessarily a complete evaluation of both the coding cost and related distortion.

[0156] The implementations and aspects described herein can be implemented in, for example, a method or a process, an apparatus, a software program, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (for example, discussed only as a method), the implementation of features discussed can also be implemented in other forms (for example, an apparatus or program). An apparatus can be implemented in, for example, appropriate hardware, software, and firmware. The methods can be implemented in, for example, a processor, which refers to processing devices in general, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, such as, for example, computers, cell phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate communication of information between end-users.

[0157] Reference to “one embodiment” or “an embodiment” or “one implementation” or “an implementation”, as well as other variations thereof, means that a particular feature, structure, characteristic, and so forth described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrase “in one embodiment” or “in an embodiment” or “in one implementation” or “in an implementation”, as well any other variations, appearing in various places throughout this application are not necessarily all referring to the same embodiment.

[0158] Additionally, this application may refer to “determining” various pieces of information. Determining the information can include one or more of, for example, estimating the information, calculating the information, predicting the information, or retrieving the information from memory.

[0159] Further, this application may refer to “accessing” various pieces of information. Accessing the information can include one or more of, for example, receiving the information, retrieving the information (for example, from memory), storing the information, moving the information, copying the information, calculating the information, determining the information, predicting the information, or estimating the information.

[0160] Additionally, this application may refer to “receiving” various pieces of information. Receiving is, as with “accessing”, intended to be a broad term. Receiving the information can include one or more of, for example, accessing the information, or retrieving the information (for example, from memory). Further, “receiving” is typically involved, in one way or another, during operations such as, for example, storing the information, processing the information, transmitting the information, moving the information, copying the information, erasing the information, calculating the information, determining the information, predicting the information, or estimating the information.

[0161] It is to be appreciated that the use of any of the following “ / ”, “and / or”, and “at least one of’, for example, in the cases of “A / B”, “A and / or B” and “at least one of A and B”, is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of both options (A and B). As a further example, in the cases of “A, B, and / or C” and “at least one of A, B, and C”, such phrasing is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of the third listed option (C) only, or the selection of the first and the second listed options (A and B) only, or the selection of the first and third listed options (A and C) only, or the selection of the second and third listed options (B and C) only, or the selection of all three options (A and B and C). This may be extended, as is clear to one of ordinary skill in this and related arts, for as many items as are listed.

[0162] Also, as used herein, the word “signal” refers to, among other things, indicating something to a corresponding decoder. In this way, in an embodiment the same parameter is used at both the encoder side and the decoder side. Thus, for example, an encoder can transmit (explicit signaling) a particular parameter to the decoder so that the decoder can use the same particular parameter. Conversely, if the decoder already has the particular parameter as well as others, then signaling can be used without transmitting (implicit signaling) to simply allow the decoder to know and select the particular parameter. By avoiding transmission of any actual functions, a bit savings is realized in various embodiments. It is to be appreciated that signaling can be accomplished in a variety of ways. For example, one or more syntax elements, flags, and so forth are used to signal information to a corresponding decoder in various embodiments. While the preceding relates to the verb form of the word “signal”, the word “signal” can also be used herein as a noun.

[0163] As will be evident to one of ordinary skill in the art, implementations can produce a variety of signals formatted to carry information that can be, for example, stored or transmitted. The information can include, for example, instructions for performing a method, or data produced by one of the described implementations. For example, a signal can be formatted to carry the bitstream of a described embodiment. Such a signal can be formatted, for example, as an electromagnetic wave (for example, using a radio frequency portion of spectrum) or as a baseband signal. The formatting can include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information that the signal carries can be, for example, analog or digital information. The signal can be transmitted over a variety of different wired or wireless links, as is known. The signal can be stored on a processor-readable medium.

[0164] A number of embodiments has been described above. Features of these embodiments can be provided alone or in any combination, across various claim categories and types.

Claims

CLAIMS1. A method, comprising:Obtaining data representative of at least one attribute of a point cloud, Transforming the data representative of the at least one attribute of the point cloud into at least two latents by applying at least one neural network-based lifting stage,Encoding the at least two latents.

2. An apparatus comprising one or more processors operable to:Obtain data representative of at least one attribute of a point cloud,Transform the data representative of the at least one attribute of the point cloud into at least two latents by applying at least one neural network-based lifting stage,Encode the at least two latents.

3. A method, comprising:Decoding at least two latents representative of at least one attribute of a point cloud,Transforming the at least two latents into data representative of the at least one attribute of the point cloud by applying at least one neural network-based lifting stage.

4. An apparatus comprising one or more processors operable to: decode at least two latents representative of at least one attribute of a point cloud, transform the at least two latents into data representative of the at least one attribute of the point cloud by applying at least one neural network-based lifting stage.

5. The method of claim 1 or the apparatus of claim 2, wherein applying at least one neural network-based lifting stage comprises: splitting the data representative of the at least one attribute of the point cloud into a first set and a second set, applying at least one lifting operation to the first set and the second set, wherein the at least one lifting operation is based on a neural network.

6. The method of claim 3 or the apparatus of claim 4, wherein applying at least oneneural network-based lifting stage comprises: applying at least one lifting operation to two latents of the at least two latents, wherein the at least one lifting operation is based on a neural network, merging outputs of the at least one lifting operation.

7. The method or the apparatus of claims 5 or 6, wherein applying at least one lifting operation comprises a prediction operation that uses a first neural network and an update operation that uses a second neural network.

8. The method or the apparatus of claim 7, wherein the prediction operation and the update operation respectively apply a masked filtering on an output of the first and second neural network.

9. The method or the apparatus of claim 7 or 8, wherein at least one of the first or second neural network uses sparse convolutions.

10. The method of any one of claims 1, 3 or 5-9 or the apparatus of any one of claims 2 or 4-9, wherein when more than one neural network-based lifting stage is applied, a stride of neural networks used in the neural network-based lifting stage increases with a number of neural network-based lifting stages.

11. The method of any one of claims 1, 3 or 5-10 or the apparatus of any one of claims 2 or 4-10, wherein encoding or decoding the one or more latents comprises arithmetically encoding or arithmetically decoding the one or more latents.

12. A non-transitory computer readable medium storing coded data representative of at least two latents encoding at least one attribute of a point cloud, the at least two latents comprising at least one latent comprising residual prediction between a first set of the at least one attribute and a prediction obtained from a second set of the at least one attribute, and at least one latent comprising an updated version of the second set of the at least one attribute obtained using the residual prediction.

13. The non-transitory computer readable medium of claim 12, comprising coded data representative of an hyperprior latent for entropy decoding the coded datarepresentative of at least two latents.

14. The method of claims 1, 3 or 5-11 or the apparatus of claims 2, or 4-11 or the non- transitory computer readable medium of claims 12 or 13-14, wherein one or more syntax elements are signaled to indicate a full channel mode or a single channel mode, wherein the full channel mode indicates that the at least one neural network-based lifting stage processes all channels of the at least one attribute of the point cloud and the single channel mode indicates tha the at least one neural network-based lifting stage processes one channel of the at least one attribute of the point cloud.

15. The method or the apparatus or the non-transitory computer readable medium of claim 14, wherein the single channel mode, a flag is signaled to indicate whether a same network is used for all channels of the at least one attribute or different networks are used.

16. The method of claims 1, 3 or 5-11 or 14-15 or the apparatus of claims 2, or 4-11 or 14-15 or the non-transitory computer readable medium of claims 12 or 14-15, wherein one or more syntax elements are signaled to indicate at least one of a number of lifting blocks or a number of lifting steps.

17. A computer program product including instructions for causing one or more processors to carry out the method of any of claims 1, 3 or 5-11 or 14-16.

18. A non-transitory computer readable medium storing executable program instructions to cause a computer executing the program instructions to perform the method of any of claims 1, 3 or 5-11 or 14-16.

19. A device comprising: an apparatus according to claim 4; and at least one of (i) an antenna configured to receive or transmit a signal, the signal including data representative of the point cloud, (ii) a band limiter configured to limit the signal to a band of frequencies that includes the data representative of the point cloud, or (iii) a display configured to display the point cloud.

20. A device according to claim 19, wherein the device comprises at least one of a television, a cell phone, a tablet, a set-top box.

Images