A rate-adaptive codec for dynamic point cloud compression

Abstract

An electronic device and method using a rate-adaptive codec for dynamic point cloud compression are provided. The electronic device receives a point cloud frame (PCF) sequence including a reference PCF set and a current PCF to be encoded. The electronic device generates 3D feature data based on applying a neural network-based encoder to at least one of the reference PCF set and the current PCF. The electronic device selects an encoding path for the current PCF from an encoding pipeline as an I-frame encoding path or a P-frame encoding path. The selected encoding path can extend from an output of the neural network-based encoder and includes a first set of transformation operations that can be applied to the 3D feature data. The electronic device generates a bitstream of coded PCFs based on applying the first set of transformation operations to the 3D feature data.

Description

[Technical Field]

[0001] Cross-reference to related applications / incorporation by reference This application claims the benefit of priority to U.S. Patent Application No. 18 / 509,138, filed with the U.S. Patent and Trademark Office on November 14, 2023, which claims priority to U.S. Provisional Patent Application Serial No. 63 / 476,589, filed on December 21, 2022, the contents of which are incorporated herein by reference in their entireties. [Background technology]

[0002] Advances in the field of dynamic point cloud compression (PCC) have led to the development of point cloud coding techniques for efficiently representing point cloud data associated with 3D objects. Point cloud data can be compressed (i.e., encoded) to generate coded point cloud data for storage, processing, or transmission. The original point cloud can then be reconstructed based on the coded point cloud data. Point cloud data associated with a current point cloud frame can be encoded or decoded based on the current point cloud frame (i.e., I-frame codec) or a previous point cloud frame set (i.e., P-frame codec). Each of these encoding or decoding schemes (I-frame or P-frame codec) can have its own advantages and disadvantages. However, using both encoding or decoding schemes in combination can provide certain benefits that would not be available if either were used alone. However, when used in combination, separate codec networks may be required for each of the I-frame and P-frame codecs, even if the architecture of each codec network includes the same encoder and decoder. Each codec network may include its own machine learning-based encoder and decoder, and each codec network may need to be trained separately. Therefore, having separate codecs for encoding / decoding I-frames and P-frames can increase the latency associated with training. The increased latency can be due to the increased number of network parameters whose values ​​need to be calculated to obtain efficient I-frame and P-frame codecs. The increased latency and computational cost can discourage the combined use of I-frame and P-frame codecs.

[0003] The limitations and disadvantages of conventional approaches will become apparent to those skilled in the art by comparing the described system with certain aspects of the present disclosure illustrated in the remainder of this application and with reference to the drawings. Summary of the Invention

[0004] An electronic device and method using a rate adaptive codec for dynamic point cloud compression is provided substantially as shown and / or described in connection with at least one of the figures and more fully set forth in the claims.

[0005] These and other features and advantages of the present disclosure will become apparent from a consideration of the following detailed description of the disclosure when taken in conjunction with the accompanying drawings, in which like reference characters refer to like elements throughout. [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 1 illustrates an exemplary network environment including a rate-adaptive codec for dynamic point cloud compression, according to an embodiment of the present disclosure. [Figure 2] FIG. 1 is a block diagram illustrating an exemplary electronic device including a rate-adaptive codec for dynamic point cloud compression, according to an embodiment of the present disclosure. [Figure 3A] FIG. 1 illustrates an exemplary rate-adaptive unified I-frame and P-frame codec architecture for dynamic point cloud compression and point cloud reconstruction, according to an embodiment of the present disclosure. [Figure 3B] FIG. 1 illustrates an exemplary rate-adaptive unified I-frame and P-frame codec architecture for dynamic point cloud compression and point cloud reconstruction, according to an embodiment of the present disclosure. [Figure 4] 10A-10C are block diagrams illustrating example operations for generating gain vector sets and encoding point cloud frames at different encoding rates based on the gain vector sets, according to embodiments of the present disclosure. [Figure 5] 10A-10C are block diagrams illustrating example operations for generating an inverse gain vector set and decoding a point cloud frame based on the inverse gain vector set, according to embodiments of the present disclosure. [Figure 6] 1 is a flowchart illustrating the operation of an exemplary method for dynamic point cloud compression using a rate-adaptive unified I-frame and P-frame codec architecture, according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0007] The disclosed electronic device and method for providing a rate-adaptive codec for dynamic point cloud compression may include the following embodiments. An exemplary aspect of the present disclosure provides an electronic device (e.g., a computer device, a game console, or virtual reality goggles) including a rate-adaptive codec. The rate-adaptive codec may include an encoding pipeline and a decoding pipeline. The encoding pipeline may integrate an intra-frame (i.e., I-frame) encoder and an inter-frame (i.e., P-frame) encoder. Similarly, the decoding pipeline may integrate an intra-frame (i.e., I-frame) decoder and an inter-frame (i.e., P-frame) decoder. The encoding pipeline may include a set of gain vectors. The gain vector set may modulate feature data associated with the point cloud frame such that the point cloud frame is encoded at a desired coding rate to generate encoded point cloud data. The decoding pipeline may include a set of inverse-gain vectors. The inverse gain vector set can demodulate the encoded point cloud data at rates corresponding to different encoding rates for decoding or reconstructing the point cloud frame.

[0008] The electronic device may receive a PCF sequence including a set of reference point cloud frames (PCFs) and a current PCF. The current PCF may be a 3D frame to be encoded (or compressed). After receiving the PCFs, the electronic device may generate three-dimensional (3D) feature data based on applying a neural network-based encoder included in an encoding pipeline to at least one of the reference PCF set and the current PCF. Once the 3D feature data is generated, the electronic device may select a coding path from the encoding pipeline. The coding path may be selected as an I-frame encoding path (i.e., an intra-frame encoder) or a P-frame encoding path (i.e., an inter-frame encoder) for encoding the current PCF. The selected coding path may extend from the output of the neural network-based encoder and may include a first series of transformation operations that can be applied to the 3D feature data. Finally, the electronic device may generate a bitstream of coded PCF (i.e., encoded point cloud data) based on applying the first series of transformation operations to the 3D feature data.

[0009] The electronic device can apply decoding operations to the bitstream of the coded PCF to reconstruct the current PCF. Based on this application, a set of decoded features for the current PCF can be generated. The electronic device can then select a decoding path from the decoding pipeline as an I-frame decoding path (i.e., an intra-frame decoder) or a P-frame decoding path (i.e., an inter-frame decoder). The selected decoding path can include a second set of transformation operations that can be applied to the decoded feature set. The electronic device can reconstruct the current PCF of the PCF sequence based on applying the second set of transformation operations to the decoded feature set.

[0010] Typically, a PCF can be encoded or decoded using an I-frame coder or a P-frame coder. Because a PCF contains a substantial amount of data, such compression and reconstruction may be necessary to enable storage, rendering, transmission, or reception of the PCF. An I-frame coder can encode / reconstruct a PCF without a reference PCF, whereas a P-frame coder can encode / reconstruct a PCF by using a set of reference PCFs that precede or follow the PCF in the PCF's timeline. The architectures of the I-frame coder and the P-frame coder can each include the same machine learning-based encoder and decoder (i.e., the encoder in each of the I-frame coder and the P-frame coder can be the same, and the decoder in each of the I-frame coder and the P-frame coder can be the same). Although the architectures of the I-frame and P-frame coders include the same encoder / decoder, using them together may require separate encoder networks for each of the I-frame and P-frame coders. This requirement can lead to redundancy in the coder network due to the inclusion of redundant encoders and decoders. Furthermore, these redundancies may increase the number of coder network parameters, storage costs (memory requirements), computational costs, and latency associated with training a coder network. Coder networks (including I-frame coders and P-frame coders) can be trained to encode a PCF at a specific bit rate and to reconstruct a PCF from point cloud data encoded at a specific bit rate. To encode a PCF at a different bit rate or to reconstruct a PCF from point cloud data encoded at a different bit rate, it may be necessary to retrain the coder network. In some scenarios, multiple coder networks can be used when a PCF is to be encoded at different bit rates or reconstructed from point cloud data encoded at different bit rates.Each coder network may include identical encoders and decoders trained to encode a PCF at a particular bit rate or to reconstruct a PCF from point cloud data encoded at a particular bit rate. Thus, redundancy may be introduced if the PCF needs to be encoded at a different bit rate or reconstructed from point cloud data encoded at a different bit rate.

[0011] To address the above-mentioned problems, an electronic device may provide a rate-adaptive codec that includes a compression network trained to encode a PCF at a desired bit rate (i.e., a tunable or adjustable rate) and a reconstruction network trained to reconstruct the PCF regardless of the coding rate that could have been used to encode the PCF. The compression network may integrate two encoding paths, i.e., an I-frame encoding path and a P-frame encoding path, into an encoding pipeline. The reconstruction network may integrate two decoding paths, i.e., an I-frame decoding path and a P-frame decoding path, into a decoding pipeline. The rate-adaptive codec may enable seamless transitions between the I-frame encoding path and the P-frame encoding path, or between the I-frame decoding path and the P-frame decoding path. By integrating the encoding and decoding paths, the number of encoders and decoders included in the encoding and decoding pipelines may be minimized, respectively. The performance of the rate-adaptive codec may be improved by reducing the computational cost and latency associated with training the encoding and decoding pipelines of the rate-adaptive codec. The reduction in both computational cost and latency can be attributed to a significant reduction in the number of network parameters (due to minimizing the number of encoders and decoders involved in the rate adaptive codec).

[0012] The rate-adaptive codec may enable encoding of a PCF at an adjustable coding rate (which may be continuous rather than discrete) through the use of a modulation network. The modulation network may provide a set of gain vectors that may be generated based on training of a fully connected neural network. Based on a selection of a gain vector from the set of gain vectors, features associated with the current PCF or a previous set of PCFs may be modulated so that the current PCF can be coded at a different coding rate. The rate-adaptive codec may further enable reconstruction of a PCF from a bitstream of coded PCFs (generated based on coding of the current PCF) through the use of a demodulation network. The demodulation network may include a set of inverse gain vectors, each of which may correspond to a gain vector from the set of gain vectors that may be used to code the current PCF. An inverse gain vector from the set of inverse gain vectors may be selected such that the PCF is reconstructed from the bitstream of coded PCFs. The coded PCF may be generated based on modulation using a gain vector corresponding to the selected inverse gain vector. Thus, the rate-adaptive codec may enable coding over a discrete series of rate points (coding rates).

[0013] The electronic device can generate further gain vectors based on the set of gain vectors and can generate corresponding inverse gain vectors based on the set of inverse gain vectors. These generation can be based on interpolation and / or extrapolation using gain vectors from the set of gain vectors and inverse gain vectors from the set of inverse gain vectors. Generating the further gain vectors and inverse gain vectors can enable encoding of the PCF at any target bit rate.

[0014] FIG. 1 illustrates an exemplary network environment including a rate-adaptive codec for dynamic point cloud compression, according to an embodiment of the present disclosure. FIG. 1 illustrates a network environment 100. The network environment 100 may include an electronic device 102 and a server 104. The electronic device 102 may communicate with the server 104 over one or more networks (e.g., a communication network 106). The electronic device 102 may include an encoding pipeline 108 and a decoding pipeline 110. The encoding pipeline 108 may include a neural network-based encoder 112, a first neural network-based predictor 114, and a gain vector set 116. The decoding pipeline 110 may include a neural network-based decoder 118, a second neural network-based predictor 120, and an inverse gain vector set 122. The electronic device 102 may receive a point cloud frame (PCF) sequence 124 as input. The PCF sequence 124 may include a current PCF 124A and a reference point cloud frame set (PCF) 124B...124N. The encoding pipeline 108 may receive as input a PCF sequence 124 and produce as output a bitstream of coded PCFs 126 associated with the current PCF 124A. The decoding pipeline 110 may receive as input the bitstream of coded PCFs 126 and a set of reconstructed reference PCFs 128. The decoding pipeline 110 may produce as output a reconstructed PCF 130 (i.e., a reconstructed point cloud corresponding to the current PCF 124A).

[0015] The electronic device 102 may include suitable logic, circuitry, interfaces, and / or code that may be configured to generate 3D feature data based on applying the neural network-based encoder 112 to the PCF sequence 124. The electronic device 102 may further select a coding path as an I-frame coding path or a P-frame coding path for the current PCF 124A. Based on the selection, a first set of transform operations may be applied to the 3D feature data to generate a bitstream of the coded PCF 126. The electronic device 102 may further apply decoding operations to the bitstream of the coded PCF 126 to generate a decoded feature set for the current PCF 124A. The electronic device 102 may further select a decoding path as an I-frame decoding path or a P-frame decoding path for decoding the current PCF 124A. Based on the selection, a second set of transform operations may be applied to the decoded feature set. Based on the application of the second set of transform operations, a reconstructed PCF 130 corresponding to the current PCF 124A may be generated. Examples of electronic devices 102 may include, but are not limited to, computer devices, video conferencing systems, augmented reality (AR) devices, virtual reality (VR) devices, mixed reality (MR) devices, gaming consoles, servers, smart wearable devices, mainframe machines, computer workstations, and / or consumer electronics (CE) devices.

[0016] The server 104 may include suitable logic, circuitry, interfaces, and / or code that may be configured to store the PCF sequences 124 (i.e., the current PCF 124A and the reference PCF set 124B...124N) and information related to the PCF sequences 124. The server 104 may be further configured to receive a request for the PCF sequences 124 from the electronic device 102. The server 104 may transmit the PCF sequences 124 to the electronic device 102 based on the request. In some embodiments, the server 104 may include an encoding pipeline 108 and / or a decoding pipeline 110 and perform associated operations of the encoding pipeline 108 and / or the decoding pipeline 110. The server 104 may generate a bitstream of the coded PCF 126 and / or a reconstructed PCF 130 based on the execution. The server 104 may transmit the bitstream of the coded PCF 126 and / or the reconstructed PCF 130 to the electronic device 102. The server 104 may perform operations via a web application, a cloud application, an HTTP request, a repository operation, a file transfer, etc. Examples of implementations of server 104 include, but are not limited to, a database server, a file server, a web server, an application server, a mainframe server, a cloud computing server, or combinations thereof.

[0017] In at least one embodiment, the server 104 may be implemented as multiple distributed cloud-based resources using a number of techniques known to those skilled in the art. Those skilled in the art will appreciate that the scope of the present disclosure may not be limited to the implementation of the server 104 and the electronic device 102 as two independent entities. In some embodiments, the functionality of the server 104 may be incorporated, in whole or at least in part, into the electronic device 102 without departing from the scope of the present disclosure.

[0018] The communication network 106 may include a communication medium that enables the electronic device 102 and the server 104 to communicate with each other. The communication network 106 may be a wired or wireless communication network. Examples of the communication network 106 may include, but are not limited to, the Internet, a Wireless Fidelity (Wi-Fi) network, a personal area network (PAN), a local area network (LAN), or a metropolitan area network (MAN). The electronic device 102 may be configured to connect to the communication network 106 according to various wired and / or wireless communication protocols. Examples of such communication protocols include, but are not limited to, at least one of Transmission Control Protocol and Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), ZigBee, EDGE, IEEE802.11, Light Fidelity (Li-Fi), 802.16, IEEE802.11s, IEEE802.11g, multi-hop communication, wireless access point (AP), device-to-device communication, cellular communication protocols, and Bluetooth (BT) communication protocols.

[0019] The neural network-based encoder 112 of the encoding pipeline 108 may include suitable logic, circuitry, interfaces, and / or code that may be configured to encode each PCF included in the PCF sequence 124 to generate a feature set associated with the 3D points of each PCF. The feature sets may include a first feature set associated with the current PCF 124A and a second feature set associated with the set of reference PCFs 124B...124N. The neural network-based encoder 112 may encode the current PCF 124A to generate a first feature set associated with the 3D points in the current PCF 124A. Similarly, the neural network-based encoder 112 may encode each reference PCF in the set of reference PCFs 124B...124N to generate a second feature set associated with the 3D points of each reference PCF. The neural network-based encoder 112 may be implemented as a deep neural network (DNN) in the form of computer-executable code on a graphics processing unit (GPU), central processing unit (CPU), tensor processing unit (TPU), reduced instruction set computing (RISC), application specific integrated circuit (ASIC), or complex instruction set computing (CISC) processor, coprocessor, and / or combination thereof. In other embodiments, the neural network-based encoder 112 may be implemented as a DNN on dedicated hardware in conjunction with other computational circuitry of the electronic device 102. In such implementations, the neural network-based encoder 112 may be associated with a particular form factor on a particular computational circuit. Examples of particular computational circuitry may include, but are not limited to, a field programmable gate array (FPGA), a programmable logic device (PLD), an ASIC, a programmable ASIC (PL-ASIC), an application specific integrated component (ASSP), and a system-on-chip (SOC) based on a standard microprocessor (MPU) or digital signal processor (DSP).According to one embodiment, the neural network-based encoder 112 may also be interfaced with a GPU to parallelize the operation of the neural network-based encoder 112 .

[0020] Each of the first neural network-based predictor 114 and the second neural network-based predictor 120 can be a system of computational networks or artificial neurons, typically arranged in multiple layers. Each of the first neural network-based predictor 114 and the second neural network-based predictor 120 can be defined by hyperparameters, such as activation function(s), number of weights, cost function, regularization function, input size, and number of layers. Furthermore, these layers can include an input layer, one or more hidden layers, and an output layer. Each of the multiple layers can include one or more nodes (or artificial neurons). The output of every node in the input layer can be connected to at least one node in the hidden layer(s). Similarly, the input of each hidden layer can be connected to the output of at least one node in another layer of each of the first neural network-based predictor 114 and the second neural network-based predictor 120. The output of each hidden layer can be coupled to the input of at least one node in another layer of each of the first neural network-based predictor 114 and the second neural network-based predictor 120. The node(s) in the final layer can receive input from at least one hidden layer and output a result. The number of layers and the number of nodes in each layer can be determined from hyperparameters of each of the first neural network-based predictor 114 and the second neural network-based predictor 120. Such hyperparameters can be set before, during, or after training of each of the first neural network-based predictor 114 and the second neural network-based predictor 120.

[0021] Each node may correspond to a mathematical function (e.g., a sigmoid function or a rectified linear unit) having parameters that can be adjusted during training of each of the first neural network-based predictor 114 and the second neural network-based predictor 120. The set of parameters may include a weight parameter, a regularization parameter, etc. Each node may calculate an output using a mathematical function based on one or more inputs from nodes in other layer(s) (e.g., previous layer(s)) of each of the first neural network-based predictor 114 and the second neural network-based predictor 120. All or some of the nodes in each of the first neural network-based predictor 114 and the second neural network-based predictor 120 may correspond to the same or different mathematical functions. In training each of the first neural network-based predictor 114 and the second neural network-based predictor 120, one or more parameters of each node of each of the first neural network-based predictor 114 and the second neural network-based predictor 120 can be updated based on whether the output of the final layer for a given input (from the training dataset) matches the correct result according to the loss function of each of the first neural network-based predictor 114 and the second neural network-based predictor 120. The above process can be repeated for the same or different inputs until a minimum of the loss function is achieved and the training error is minimized. Several training methods are known in the art, such as gradient descent, stochastic gradient descent, batch gradient descent, gradient boosting, and metaheuristic methods.

[0022] Each of the first neural network-based predictor 114 and the second neural network-based predictor 120 can be a machine learning model trained to generate multi-scale features associated with a reference PCF. The first neural network-based predictor 114 can receive as input a second feature set associated with 3D points of each reference PCF in the set of reference PCFs 124B...124N and target coordinate information associated with the current PCF 124A. The second neural network-based predictor 120 can receive as input a feature set associated with 3D points of each reconstructed reference PCF in the set of reconstructed reference PCFs 128 and decoded target coordinate information associated with the current PCF 124A. Each of the first neural network-based predictor 114 and the second neural network-based predictor 120 can generate a predicted value as an output. The prediction can indicate coding features associated with the current PCF 124A.

[0023] In some embodiments, each of the first neural network-based predictor 114 and the second neural network-based predictor 120 may include electronic data that may be implemented as a software component of an application executable on the electronic device 102. Each of the first neural network-based predictor 114 and the second neural network-based predictor 120 may rely on libraries, external scripts, or logic / instructions for execution by a processing unit included in the electronic device 102. In one or more embodiments, each of the first neural network-based predictor 114 and the second neural network-based predictor 120 may be implemented using hardware that may include a processor, a microprocessor (e.g., that performs or controls the execution of one or more operations), an FPGA, or an ASIC. Alternatively, in some embodiments, each of the first neural network-based predictor 114 and the second neural network-based predictor 120 may be implemented using a combination of hardware and software. Examples of each of the first neural network-based predictor 114 and the second neural network-based predictor 120 may include, but are not limited to, a DNN, a convolutional neural network (CNN), an artificial neural network (ANN), a fully connected neural network, a deep Bayesian neural network, or a combination of such networks. In an embodiment, each of the first neural network-based predictor 114 and the second neural network-based predictor 120 may correspond to a learning engine capable of performing numerical computation techniques using a data flow graph. In some embodiments, each of the first neural network-based predictor 114 and the second neural network-based predictor 120 may be based on a hybrid architecture of multiple DNNs.

[0024] The encoding pipeline 108 includes a gain vector set 116 that can be used to modulate features associated with the PCF sequence 124. The gain vector set 116 can be generated based on training a first neural network model (e.g., a multilayer perceptron). The first neural network model can be trained based on a first set of bits-per-point (BPP) values, which can indicate a range of coding rates. The first set of BPP values ​​can be encoded based on a one-hot coding scheme to generate a set of one-hot vectors. The one-hot vector set can be provided as an input to the first neural network model to train it. The gain vector set 116 can be generated based on this training. Each gain vector in the gain vector set 116 can be associated with a coding rate, and can modulate a first feature set associated with the current PCF 124A or a second feature set associated with a reference PCF set 124B...124N so that the current PCF 124A is encoded at that coding rate.

[0025] The neural network-based decoder 118 may include suitable logic, circuitry, and / or interfaces that can be configured to reconstruct the current PCF 124A based on at least one of the bitstream of the coded PCF 126 and the set of reconstruction reference PCFs 128. The neural network-based decoder 118 may be implemented as a DNN (computer-executable code format) on a GPU, a CPU, a TPU, a RISC processor, an ASIC processor, a CISC processor, a coprocessor, and / or a combination thereof. In other embodiments, the neural network-based decoder 118 may be implemented as a DNN on dedicated hardware in conjunction with other computational circuitry of the electronic device 102. In such implementations, the neural network-based decoder 118 may be associated with a particular form factor on a particular computational circuit. Examples of particular computational circuitry include, but are not limited to, an FPGA, a PLD, an ASIC, a PL-ASIC, an ASSP, and a standard MPU- or DSP-based SOC. In some embodiments, the neural network-based decoder 118 may interface with a GPU to parallelize the operation of the neural network-based decoder 118.

[0026] The inverse gain vector set 122 included in the decoding pipeline 110 can be used to demodulate features associated with a decoded feature set of the PCF sequence 124 (which can be generated based on the bitstream of the coded PCF 126). The inverse gain vector set 122 can be generated based on training a second neural network model (e.g., a multilayer perceptron). The second neural network model can be trained based on a second set of BPP values ​​corresponding to a range of decoding rates. The range of decoding rates can correspond to a range of coding rates. The second set of BPP values ​​can be encoded based on a one-hot coding scheme to generate the one-hot vector set. The one-hot vector set can be provided as an input to the second neural network model to train it. The inverse gain vector set 122 can be generated based on this training. Each inverse gain vector 122 in the inverse gain vector set 122 can be associated with a decoding rate, and the features associated with the decoded feature set can be demodulated so that the current PCF 124A is decoded at that decoding rate to generate the reconstructed PCF 130.

[0027] Each PCF in the PCF sequence 124 (i.e., the current PCF 124A and the set of reference PCFs 124B...124N) may correspond to a set of geometric representations of a set of 3D objects (e.g., a mouse) in a 3D environment. Each PCF may comprise a set of 3D points located at different locations in 3D space of the 3D environment. In some embodiments, the electronic device 102 may generate the PCF sequence 124 or obtain the PCF sequence 124 from the server 104. Each 3D point in each PCF in the PCF sequence 124 may include geometric information (which may include coordinates of the corresponding 3D point in the corresponding PCF) and attribute information associated with the corresponding 3D point. The attribute information may include color information, reflectance information, opacity information, normal vector information, material identifier information, or texture information.

[0028] The bitstream of the coded PCF 126 may be generated based on encoding each PCF of the PCF sequence 124 at one or more coding rates. The bitstream of the coded PCF 126 may be generated based on applying a first series of transformation operations to at least one of a first feature set associated with the current PCF 124A and a second feature set associated with the reference PCF set 124B...124N. The first series of transformation operations may include modulating the first feature set or the second feature set using the gain vector set 116 such that the current PCF 124A is coded at one or more coding rates. The bitstream of the coded PCF 126 may be generated as an output of the encoding pipeline 108 and may be received as an input by the decoding pipeline 110. The decoding pipeline 110 may generate a decoded feature set of the current PCF 124A based on the bitstream of the coded PCF 126.

[0029] The reconstructed reference PCF set 128 may include PCFs reconstructed through the decoding pipeline 110 prior to the reconstruction of the current PCF 124A. The PCFs in the reconstructed reference PCF set 128 may be generated based on decoding a bitstream of coded PCFs. The bitstream of coded PCFs may be generated based on encoding the reference PCF set 124B...124N. The reconstructed reference PCF set 128 may be used to reconstruct the current PCF 124A (i.e., to generate the reconstructed PCF 130) based on the selection of a P-frame decoding path in the decoding pipeline 110.

[0030] The reconstructed PCF 130 may be a frame of reconstructed 3D data corresponding to the current PCF 124A. The electronic device 102 may reconstruct the current PCF 124A based on the bitstream of the coded PCF 126 via the neural network-based decoder 118. The reconstruction may be based on demodulation of decoded features associated with the current PCF 124A (which may be generated based on the bitstream of the coded PCF 126).

[0031] In operation, the electronic device 102 may be configured to receive a PCF sequence 124 that may include a reference PCF set 124B...124N and a current PCF 124A to be encoded. According to an embodiment, each PCF in the reference PCF set 124B...124N may precede the current PCF 124A in the reception timeline of the PCF sequence 124.

[0032] The electronic device 102 may be further configured to generate 3D feature data based on applying the neural network-based encoder 112 to at least one of the reference PCF set 124B...124N and the current PCF 124A. According to an embodiment, the 3D feature data may include a first feature set associated with the current PCF 124A. The first feature set may be generated as an output of the neural network-based encoder 112 based on applying the neural network-based encoder 112 to the current PCF 124A. The first feature set may be related to the occupancy of 3D points within the current PCF 124A. In some embodiments, the first feature set may include features related to the occupancy of a set of 3D points within the current PCF 124A to be encoded. The set of 3D points to be encoded may be received as user input. The 3D feature data may further include a second feature set associated with the reference PCF set 124B...124N. The second feature set can be generated as an output of the neural network-based encoder 112 based on applying the neural network-based encoder 112 to each reference PCF in the reference PCF set 124B...124N. The second feature set can include reference features related to the occupancy of the 3D points in each reference PCF in the reference PCF set 124B...124N.

[0033] The electronic device 102 may be further configured to select an encoding path from the encoding pipeline 108 as either an I-frame encoding path or a P-frame encoding path for the current PCF 124A. The selected encoding path may extend from the output of the neural network-based encoder 112 and may include a first set of transformation operations that may be applied to the generated 3D feature data. According to an embodiment, the first set of transformation operations may be applicable to the first feature set (generated as the output of the neural network-based encoder 112) based on the selection of the encoding path as the I-frame encoding path. Meanwhile, the first set of transformation operations may be applicable to the first feature set and the second feature set (generated as the output of the neural network-based encoder 112) based on the selection of the encoding path as the P-frame encoding path.

[0034] According to one embodiment, the first series of transformation operations may include applying a gain vector from the gain vector set 116 to the first feature set to generate a first set of gain-modulated features based on the selection of the coding path as the I-frame coding path. The gain vector may be selected from the gain vector set 116. The selection may be based on an association between the gain vector and a coding rate and on receiving an input by the electronic device 102. For example, the input may indicate a selected coding rate that matches the coding rate associated with the gain vector.

[0035] According to an embodiment, the first set of transformation operations may include generating a first predicted feature set based on applying a first neural network-based predictor 114 to the second feature set based on selection of the encoding path as the P-frame encoding path. In some embodiments, the first predicted feature set may be associated with a set of 3D points in each reference PCF of the set of reference PCFs 124B...124N that corresponds to a set of 3D points in the current PCF 124A to be encoded. The first predicted feature set may be generated as an output of the first neural network-based predictor 114.

[0036] The first series of transformation operations may further include applying one gain vector from the gain vector set 116 to each of the first feature set and the first predicted feature set. Applying the gain vector to the first feature set may generate a first gain-modulated feature set, and applying the gain vector to the first predicted feature set may generate a second gain-modulated feature set. The first gain-modulated feature set may be generated based on modulating the first feature set with the gain vector. Similarly, the second gain-modulated feature set may be generated based on modulating the first predicted feature set with the gain vector. A gain vector may be selected from the gain vector set 116 based on an association between the gain vector and a coding rate that matches a selected coding rate (indicated in the input received by the electronic device 102). A difference operation may then be applied to the first gain-modulated feature set and the second gain-modulated feature set to generate a set of residual features such that the current PCF 124A is coded at the selected coding rate.

[0037] The electronic device 102 may be further configured to generate a bitstream of the coded PCF 126 based on applying a first set of transformation operations to the 3D feature data. The application of the first set of transformation operations may be based on a selection of an I-frame encoding path or a P-frame encoding path. Thus, generation of the bitstream of the coded PCF 126 may be based on a first gain modulation feature set (if the I-frame encoding path is selected) or a residual feature set (if the P-frame encoding path is selected).

[0038] In the decoding stage, the electronic device 102 may be configured to apply a decoding operation to the bitstream of the coded PCF 126 to generate a decoded feature set for the current PCF 124A. The generation of the decoded feature set may be based on whether an I-frame coding path or a P-frame coding path is selected to generate the bitstream of the coded PCF 126. If the P-frame coding path is selected, the decoded feature set may correspond to a set of reconstructed residual features (reconstructed based on the residual feature set). On the other hand, if the I-frame coding path is selected, the decoded feature set may correspond to a set of reconstructed gain-modulated features (reconstructed based on the first gain-modulated feature set).

[0039] The electronic device 102 may be further configured to select a decoding path from the decoding pipeline 110 as one of an I-frame decoding path or a P-frame decoding path. The decoding path may include a second set of transform operations applicable to the decoded feature set. The selection of the decoding path as the I-frame decoding path may be based on selecting the I-frame encoding path as the encoding path. Meanwhile, the selection of the decoding path as the P-frame decoding path may be based on selecting the P-frame encoding path as the encoding path. Thus, the second set of transform operations may be applicable to the reconstructed gain modulation features or the reconstructed residual feature set.

[0040] Based on the selection of the I-frame decoding path, a second set of transformation operations may be applied to the reconstructed gain-modulated feature set (i.e., the decoded feature set). The second set of transformation operations may include applying a selected inverse gain vector to the decoded feature set to generate a set of gain-demodulated features. The selected inverse gain vector may correspond to a gain vector selected to modulate the first feature set (included in the 3D feature data). The inverse gain vector may be selected from the inverse gain vector set 122 based on receiving an input that may indicate an association between the inverse gain vector and a decoding rate and a selected decoding rate that matches the decoding rate associated with the inverse gain vector. The input may be based on a coding rate associated with the gain vector corresponding to the selected inverse gain vector. The gain-demodulated feature set may be generated based on demodulating the reconstructed gain-modulated feature set so that the current PCF 124A can be decoded at the selected decoding rate.

[0041] Based on the selection of the P frame decoding path, a second series of transformation operations may be applied to the reconstructed residual feature set (i.e., the decoded feature set). The second series of transformation operations may include generating a third feature set based on applying the neural network-based encoder 112 to the reconstructed reference PCF set 128. Then, a second predicted feature set may be generated based on applying the second neural network-based predictor 120 to the third feature set. In some embodiments, the third predicted feature set may be associated with a set of 3D points in each reconstructed reference PCF of the reference PCF set 124B...124N that corresponds to a set of 3D points in the current PCF 124A that needs to be decoded. The second predicted feature set may be generated as the output of the second neural network-based predictor 120. The second series of transformation operations may further include applying one gain vector from the gain vector set 116 to the third feature set. This application may generate a third gain-modulated feature set. The third gain-modulated feature set may be generated based on modulating the third feature set using the gain vector. This gain vector can be the same as the gain vector used to modulate the first feature set and the first predicted feature set.

[0042] The second series of transformation operations may further include applying an addition operation to the third gain-modulated feature set and the reconstructed residual feature set (i.e., the decoded feature set) to generate a fourth feature set. The second series of transformation operations may further include applying a selected inverse gain vector to the fourth feature set to generate a gain-demodulated feature set. The selected inverse gain vector may correspond to the gain vector selected to modulate the first feature set and the first predicted feature set (included in the 3D feature data) and the gain vector selected to modulate the third feature set. The inverse gain vector may be selected from the inverse gain vector set 122 based on receiving an input indicating an association between the inverse gain vector and a decoding rate and a selected decoding rate that matches the decoding rate associated with the inverse gain vector. The input may be based on a coding rate associated with the gain vector corresponding to the selected inverse gain vector. The gain-demodulated feature set may be generated based on demodulation of the fourth feature set to enable decoding of the current PCF 124A at the selected decoding rate.

[0043] The electronic device 102 may be further configured to reconstruct a current PCF 124A of the PCF sequence 124 based on applying a second set of transformation operations to the decoded feature set (i.e., the reconstructed gain modulation features or the reconstructed residual feature set). The reconstruction of the current PCF 124A may correspond to generating the reconstructed PCF 130. The application of the second set of transformation operations may be based on selecting an I-frame decoding path or a P-frame decoding path. Thus, if the I-frame encoding path is selected, the reconstructed PCF 130 may be generated using a gain demodulation feature set generated based on the reconstructed gain modulation feature set. On the other hand, if the P-frame encoding path is selected, the reconstructed PCF 130 may be generated using a gain demodulation feature set generated based on a fourth feature set.

[0044] According to one embodiment, a reconstructed PCF 130 corresponding to the current PCF 124A can be generated based on applying a neural network-based decoder 118 to a gain demodulation feature set (generated based on selecting either the I-frame decoding path or the P-frame decoding path).

[0045] FIG. 2 is a block diagram illustrating an exemplary electronic device including a rate-adaptive codec for dynamic point cloud compression, according to an embodiment of the present disclosure. The description of FIG. 2 is provided with reference to the elements of FIG. 1. FIG. 2 illustrates a block diagram 200 of an electronic device 102. The electronic device 102 may include a circuit 202, a memory 204, an input / output (I / O) device 206, and a network interface 208. In at least one embodiment, the memory 204 may include the encoding pipeline 108 and the decoding pipeline 110. In at least one embodiment, the I / O device 206 may include a display device 210. The circuit 202 may be communicatively coupled to the memory 204, the I / O device 206, and the network interface 208 via wired or wireless communication of the electronic device 102.

[0046] The circuit 202 may include suitable logic, circuits, and interfaces that may be configured to execute program instructions related to different operations performed by the electronic device 102. These operations include receiving a PCF sequence 124, which may include a set of reference PCFs 124B...124N and a current PCF 124A that needs to be encoded, generating 3D feature data based on applying the neural network-based encoder 112 to at least one of the set of reference PCFs 124B...124N and the current PCF 124A, selecting an encoding path (including a first series of transformation operations that can be applied to the 3D feature data) from the encoding pipeline 108 as one of an I-frame encoding path or a P-frame encoding path for the current PCF 124A, and encoding the 3D feature data based on the first series of transformation operations that can be applied to the 3D feature data. The circuit 202 may include generating a bitstream of the coded PCF 126 based on applying a second set of transform operations to the bitstream of the coded PCF 126 to generate a decoded feature set for the current PCF 124A, selecting a decoding path from the decoding pipeline 110 as one of an I-frame decoding path or a P-frame decoding path (including a second set of transform operations applicable to the decoded feature set), and reconstructing the current PCF 124A of the PCF sequence 124 (and generating the reconstructed PCF 130) based on applying the second set of transform operations to the decoded feature set. The circuit 202 may include one or more processing units, which may be implemented as an integrated processor or a group of processors that collectively perform the functions of one or more dedicated processing units. The circuit 202 may be implemented based on multiple processor technologies known in the art. An example implementation of the circuit 202 may be an x86-based processor, a GPU, a CPU, a RISC processor, an ASIC processor, a CISC processor, a microcontroller, and / or other computing circuit.

[0047] The memory 204 may include suitable logic, circuitry, and / or interfaces that may be configured to store instructions executable by the circuit 202. The memory 204 may be configured to store an operating system and associated applications. In at least one embodiment, the memory may store the gain vector set 116 and the inverse gain vector set 122. The memory 204 may be further configured to store the PCF sequence 124, the first feature set, the second feature set, the third feature set, the fourth feature set, the first predicted feature set, the second predicted feature set, the first gain modulation feature set, the second gain modulation feature set, the third gain modulation feature set, the gain demodulation feature set, the residual feature set, the bitstream of the coded PCF 126, the reconstructed reference PCF set 128, the reconstructed PCF 130, and coordinate information associated with a set of 3D points within the current PCF 124A to be encoded. In at least one embodiment, the neural network-based encoder 112, the first neural network-based predictor 114, the second neural network-based predictor 120, and the neural network-based decoder 118 included in the memory 204 may be implemented as a combination of programmable instructions stored in the memory 204 or logic units on hardware circuitry (i.e., programmable logic units) of the electronic device 102. Example implementations of the memory 204 may include, but are not limited to, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), a hard disk drive (HDD), a solid-state drive (SSD), a CPU cache, and / or a secure digital (SD) card.

[0048] The I / O device 206 may include suitable logic, circuitry, interfaces, and / or code that may be configured to receive user input that may trigger reception of the PCF sequence 124, selection of an I-frame encoding path, selection of a P-frame encoding path, selection of an I-frame decoding path, selection of a P-frame decoding path, selection of a coding rate (for selecting a gain vector in the gain vector set 116), or selection of a decoding rate (for selecting an inverse gain vector in the inverse gain vector set 122). The I / O device 206 may be configured to provide output in response to the user input. The I / O device 206 may include various input and output devices that may be configured to communicate with the circuit 202. Examples of input devices may include, but are not limited to, a touch screen, a keyboard, a mouse, a joystick, and / or a microphone. Examples of output devices may include a display device 210.

[0049] The display device 210 may include suitable logic, circuitry, interfaces, and / or code that can be configured to render each PCF included in the PCF sequence 124 (i.e., the current PCF 124A and the reference PCF set 124B...124N) on a display screen of the display device 210. The display device 210 may be further configured to render the reconstructed PCF 130 on a display screen of the display device 210. According to some embodiments, the display device 210 may include a touch screen for receiving user input. The display device 210 may be implemented through a number of known technologies, such as, but not limited to, liquid crystal display (LCD) display, light emitting diode (LED) display, plasma display, and / or organic LED (OLED) display technology, and / or other display technologies. According to some embodiments, the display device 210 may be a display screen of a smart glasses device, a 3D display, a see-through display, a projection display, an electrochromic display, and / or a transparent display.

[0050] The network interface 208 may include suitable logic, circuitry, interfaces, and / or code that may be configured to establish communications between the electronic device 102 and the server 104 over the communications network 106. The network interface 208 may be implemented using various known technologies to support wired or wireless communications between the electronic device 102 and the communications network 106. The network interface 208 may include, but is not limited to, an antenna, a radio frequency (RF) transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a coder-decoder (CODEC) chipset, a subscriber identity module (SIM) card, and / or a local buffer.

[0051] The network interface 208 can communicate via wireless communication with networks such as the Internet, an intranet, and / or wireless networks such as a cellular network, a wireless local area network (LAN), and / or a metropolitan area network (MAN). The wireless communication can use any of a number of communication standards, protocols, and technologies, such as Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (W-CDMA), Long Term Evolution (LTE), Fifth Generation (5G) New Radio (NR), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wireless Fidelity (WiFi) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, and / or IEEE 802.11n), Voice over Internet Protocol (VoIP), Light Fidelity (Li-Fi), Wi-MAX, protocols for email, instant messaging, and / or short message service.

[0052] The functions or operations performed by first electronic device 102 described in Figure 1 may be performed by circuitry 202. The operations performed by circuitry 202 are described in detail, for example, in Figures 3A and 3B, 4 and 5.

[0053] Figures 3A and 3B illustrate an exemplary rate-adaptive unified I-frame and P-frame codec architecture for dynamic point cloud compression and point cloud reconstruction, according to an embodiment of the present disclosure. Figures 3A and 3B are described with reference to elements of Figures 1 and 2. Figures 3A and 3B illustrate an exemplary architecture 300 for dynamic point cloud compression and point cloud reconstruction. Architecture 300 shows an electronic device 102 that includes an encoding pipeline 108 (see Figure 3A) and a decoding pipeline 110 (see Figure 3B).

[0054] The encoding pipeline 108 may include a P-frame encoding path 302 and an I-frame encoding path 304. The P-frame encoding path 302 may include a neural network-based encoder 112, a first neural network-based predictor 114, a gain vector set 116, a subtractor 306, a quantizer 308, and an autoencoder 310. The I-frame encoding path 304 may include the neural network-based encoder 112, the gain vector set 116, a quantizer 312, and an autoencoder 314. The encoding pipeline 108 may further include an octree-based encoder 316.

[0055] The decoding pipeline 110 (see FIG. 3B) may include a P-frame decoding path 318 and an I-frame decoding path 320. The P-frame decoding path 318 may include a neural network-based encoder 112, a second neural network-based predictor 120, a gain vector set 116, an autodecoder 322, an accumulator 324, an inverse gain vector set 122, and the neural network-based decoder 118. The I-frame decoding path 320 may include an autodecoder 326, an inverse gain vector set 122, and the neural network-based decoder 118. The decoding pipeline 110 may further include an octree-based decoder 328.

[0056] The circuit 202 may be configured to receive, at any point in time, a PCF sequence including a set of reference PCFs (e.g., P(t-1), P(t-2), ..., and P(tN)) and a current PCF to be encoded (e.g., P(t)). The neural network-based encoder 112 may receive the set of reference PCFs and the current PCF as input. Each reference PCF in the set of reference PCFs may precede the current PCF. The neural network-based encoder 112 may generate 3D feature data as output. The 3D feature data may include a first feature set (e.g., F(t)) associated with the 3D points of the current PCF (i.e., P(t)). The first feature set may be associated with the occupancy of the 3D points of the current PCF. According to an embodiment, the 3D points may be a set of 3D points of the current PCF that need to be encoded. The circuit 202 may determine that a set of 3D points should be encoded based on an input that may include coordinate information (e.g., C(t)) of the set of 3D points of the current PCF.

[0057] The 3D feature data may further include a second feature set associated with the set of reference PCFs (i.e., P(t-1), P(t-2), ..., P(tN)). The second feature set may include a feature set associated with a 3D point of each reference PCF in the set of reference PCFs. For example, the feature set associated with the 3D point of the reference PCF P(t-1) may be F(t-1). Similarly, the feature set associated with the 3D point of the reference PCF P(tN) may be F(tN). The feature set associated with each reference PCF may include reference features associated with the occupancy of the 3D point in the corresponding reference PCF. The 3D point may correspond to a set of 3D points in the current PCF. The second feature set may be represented as {F(t-1)...F(tN)}.

[0058] The circuit 202 may be further configured to select a coding path from the coding pipeline 108 as a P-frame coding path 302 or an I-frame coding path 304 for coding the current PCF. The P-frame coding path 302 and the I-frame coding path 304 each may include a first set of transformation operations that may be applied to the 3D feature data. The first set of transformation operations may include applying a first neural network-based predictor 114 to a second feature set based on the selection of the P-frame coding path 302. Based on this application, a first predicted feature set (e.g., F ~ (t)) can be generated as output.

[0059] The feature sets associated with the 3D points of each reference PCF may be downsampled by at least one scaling factor to generate a first predicted feature set. According to one embodiment, the feature sets may be downsampled by scaling factors 2, 3, ..., and K. For example, F(t-1) may be downsampled by scaling factors 2, 3, ..., K to generate multi-scale feature sets F(t-1), ... 2 (t-1), F 3 (t-1), ..., F KSimilarly, we can downsample F(tN) to generate a multi-scale feature set F(t-1). 2 (tN), F 3 (tN),...,F K (tN) can be generated.

[0060] For each scaling factor (i.e., 2, 3, ..., or K), a space-time tensor can be generated using feature sets associated with all reference PCFs in the reference PCF set downsampled by the same factor. For example, for a scaling factor of "2," F 2 (t-1), ..., and F 2 (tN) can be used to generate a space-time tensor. Similarly, for the scaling factor "K", F K (t-1), ..., and F K (tN). According to one embodiment, the generation of the space-time tensor for a particular scaling factor can be based on the spatiotemporal concatenation of feature sets associated with all reference PCFs of the reference PCF set downsampled by the same scaling factor. For example, the space-time tensor for a scaling factor of 2 is 2 (t-1), ..., F 2 (tN). Similarly, the space-time tensor for the scaling factor "K" can be constructed as F K (t-1), ..., F K It can be constructed based on the spatiotemporal connection of (tN).

[0061] After generating the (K-1) space-time tensors (for scaling factors 2, 3, ..., and K), space-time analysis can be performed on each space-time tensor. According to one embodiment, the space-time analysis can include performing a sparse convolution or self-attention operation on each space-time tensor to generate a modified space-time tensor. The features represented by each modified space-time tensor can be downsampled by a particular scaling factor. The scaling factor can be based on the highest scaling factor (i.e., K) by which the feature set of each reference PCF in the set of reference PCFs is downsampled. An inception residual network (IRN) can then be applied to each downsampled modified space-time tensor to generate a final space-time tensor. Thus, the (K-1) final space-time tensors can be generated and concatenated to generate a multi-scale feature concatenation vector. The sparse convolution or self-attention operation can be performed on this multi-scale feature concatenation vector.

[0062] The first neural network-based predictor 114 may further receive coordinate information (i.e., C(t)) of a set of 3D points (to be encoded) of the current PCF (i.e., P(t)) to generate a first predicted feature set. The first neural network-based predictor 114 may generate a first predicted feature set (i.e., F(t)) based on C(t) and the results of a sparse convolution operation or a self-attention operation on the multi-scale feature connection vector. ~ (t)) can be generated.

[0063] The first series of transform operations may further include applying a selected gain vector to the first feature set (i.e., F(t)) to generate a first gain-modulated feature set. According to an embodiment, the circuit 202 may be configured to retrieve the gain vector set 116 from the memory 204. Each gain vector in the gain vector set 116 may correspond to a coding rate associated with the encoding pipeline 108. Once the gain vector set 116 is retrieved, the circuit 202 may select a gain vector from the gain vector set 116. The selected gain vector may modulate the features of the first feature set such that the current PCF (i.e., P(t)) is encoded at a coding rate corresponding to the selected gain vector.

[0064] According to one embodiment, the circuit 202 may receive a first input that may include a selection of a first coding rate for the current PCF (i.e., P(t)). The circuit 202 may determine a gain vector from the gain vector set 116 that corresponds to a coding rate that matches the first coding rate. Once the circuit 202 determines such a gain vector from the gain vector set 116, the circuit 202 may select the gain vector. Thus, the gain vector may be selected based on the first input. A first gain modulation feature set may be generated based on modulating the first feature set (i.e., F(t)) with the selected gain vector.

[0065] The first series of transformation operations is performed on a first set of predicted features (i.e., F ~ The method may further include applying the selected gain vector to the current PCF (i.e., P(t)) to generate a second gain-modulated feature set. The selected gain vector may modulate features of the first predicted feature set such that the current PCF (i.e., P(t)) is coded at a coding rate (first coding rate) corresponding to the selected gain vector.

[0066] The first series of transformation operations may further include applying a difference operation to the first set of gain modulation features and the second set of gain modulation features. Based on the application, a residual feature set (i.e., R(t)) may be generated. According to an embodiment, the subtractor 306 may receive the first set of gain modulation features and the second set of gain modulation features as inputs and apply a difference operation between these inputs to generate a residual feature set. The first set of gain modulation features and the second set of gain modulation features may be represented as F(t) and F(t), respectively. ~ Since the residual feature set is obtained based on (t), it can indicate the difference between the actual feature set and the predicted feature set, which can be used to compensate for possible errors in predicting features associated with the current PCF (i.e., P(t)).

[0067] The quantizer 308 may receive a residual feature set (i.e., R(t)) as an input and quantize each residual feature of the residual feature set to one quantization level from a set of predetermined quantization levels. The quantizer 308 may generate a set of quantized residual features as an output based on applying a quantization scheme to the residual feature set. The quantization scheme may be based on an entropy model. The autoencoder 310 may receive the quantized residual feature set as an input and generate a bitstream of coded PCFs as an output. This generation may be based on applying an encoding scheme to the quantized residual feature set. The encoding scheme may be based on an entropy model. The encoding scheme may compress the features of the quantized residual feature set to generate a bitstream of coded PCFs.

[0068] The coded PCF bitstream can be generated based on the residual feature set (i.e., R(t)) upon selection of the P frame coding path 302. The coded PCF bitstream can be encoded at a coding rate corresponding to the selected gain vector. Therefore, the coded PCF bitstream can be generated based on the selected gain vector. Note that the coded PCF bitstream is generated based on the first feature set (i.e., F(t)) and the first predicted feature set (i.e., F(t)). ~ (t)) and the gain vector set F(t) and F ~ The features of (t) may also be coded at other coding rates based on a selection of other gain vectors 116 corresponding to other coding rates at which the features of (t) may be modulated. The other gain vectors may be selected based on a first input that may indicate a particular value of the first coding rate that matches the coding rate corresponding to one of the other gain vectors.

[0069] Meanwhile, the first series of transformation operations may include applying a selected gain vector to the first feature set (i.e., F(t)) based on the selection of the I-frame encoding path 304 to generate a first gain-modulated feature set. The quantizer 312 may receive the first gain-modulated feature set as an input and quantize each feature of the first gain-modulated feature set to one quantization level from a set of predetermined quantization levels. The quantizer 308 may generate a set of quantized gain-modulated features as an output based on applying a quantization scheme to the first gain-modulated feature set. The quantization scheme may be based on an entropy model. The auto-encoder 314 may receive the quantized gain-modulated feature set as an input and generate a bitstream of coded PCFs as an output. The generation of the bitstream of coded PCFs may be based on application of an encoding scheme to the quantized gain-modulated feature set. The encoding scheme may be based on an entropy model. The encoding scheme can compress the features of the quantized gain modulation feature set to generate a bitstream of coded PCFs.

[0070] The bitstream of the coded PCF can be generated based on the first set of gain modulation features upon selection of the I-frame encoding path 304. The bitstream of the coded PCF can be encoded at a coding rate corresponding to the selected gain vector. Therefore, the bitstream of the coded PCF can be generated based on the selected gain vector. Note that the bitstream of the coded PCF can also be encoded at other coding rates based on modulation of the features of the first set of features (i.e., F(t)) and selection of other gain vectors from the gain vector set 116 corresponding to other coding rates.

[0071] According to one embodiment, the octree-based encoder 316 may receive as input coordinate information (i.e., C(t)) of a series of 3D points of the current PCF (i.e., P(t)) to be encoded. The octree-based encoder 316 may encode the coordinate information to generate encoded coordinate information. The decoding pipeline 110 may receive the coded PCF bitstream (generated based on the residual feature set (i.e., R(t)) or the first gain modulation feature set) and the encoded coordinate information. The octree-based decoder 328 of the decoding pipeline 110 may receive the encoded coordinate information as input and reconstruct the coordinate information (i.e., C(t)). The circuit 202 may determine a series of 3D points of the current PCF (i.e., P(t)) to be decoded based on the reconstructed coordinate information (e.g., Ĉ(t)).

[0072] According to an embodiment, the circuit 202 can be configured to apply a decoding operation to a bitstream of the coded PCF to generate a decoded feature set for the current PCF (i.e., P(t)). To apply the decoding operation, a decoding path can be selected from the decoding pipeline 110. The decoding path can be selected as a P-frame decoding path 318 or an I-frame decoding path 320. The autodecoder 322 can receive as input the bitstream of the coded PCF generated based on the residual feature set (i.e., R(t)) based on selecting the P-frame decoding path 318 as the decoding path. The autodecoder 322 can apply a decoding operation to the bitstream of the coded PCF to generate a decoded feature set for the current PCF. The autodecoder 322 can select the reconstructed residual feature set (e.g., R ~ (t)) can be generated as a decoded feature set. The reconstruction (i.e., decoding operation) can be based on a decoding scheme. The decoding scheme can be based on an entropy model. The decoding scheme can decompress the features of the quantized residual feature set to generate a decoded feature set (i.e., R ~ (t)) can be generated.

[0073] Meanwhile, the autodecoder 326 may receive as input a bitstream of a coded PCF generated based on the first gain modulation feature set based on selecting the I-frame decoding path 320 as the decoding path. The autodecoder 326 may apply a decoding operation to the bitstream of the coded PCF to generate a decoded feature set for the current PCF (i.e., P(t)). The decoding operation may be based on a decoding scheme. The decoding scheme may be based on an entropy model. The decoding scheme may decompress features of the quantized gain modulation feature set to generate the decoded feature set.

[0074] The selected decoding path may include a second set of transform operations that can be applied to the decoded feature set. A current PCF (i.e., P(t)) of the PCF sequence may be reconstructed based on applying the second set of transform operations to the decoded feature set. In the decoding path selected as the P frame decoding path 318, the second set of transform operations may include receiving a PCF sequence including a reconstructed reference PCF set (e.g., P'(t-1), P'(t-2), ..., and P'(tN)). This reception may be based on reconstructing the reference PCF set (i.e., P(t-1), P(t-2), ..., and P(tN)) before reconstructing the current PCF (i.e., P(t)). The neural network-based encoder 112 may receive the reconstructed reference PCF set as an input.

[0075] The second series of transformation operations may further include generating a third feature set (e.g., {F'(t-1)...F'(tN)}) based on applying the neural network-based encoder 112 to the set of reconstructed reference PCFs (i.e., P'(t-1), P'(t-2), ..., and P'(tN)). The third feature set may include a feature set associated with a 3D point of each reconstructed reference PCF in the set of reconstructed reference PCFs. The feature sets associated with the set of reconstructed reference PCFs may be generated as an output of the neural network-based encoder 112. For example, the feature set associated with the 3D point of the reconstructed reference PCF P'(t-1) may be F'(t-1). Similarly, the feature set associated with the 3D point of the reconstructed reference PCF P'(tN) may be F'(tN). The feature set associated with the 3D point of each reconstructed reference PCF may include features related to the occupancy of the 3D point of the corresponding reconstructed reference PCF. Each 3D point of the reconstruction reference PCF may correspond to a set of 3D points of the current PCF to be decoded (ie, P(t)).

[0076] The second series of transformation operations may further include applying a second neural network-based predictor 120 to the third feature set (i.e., {F'(t-1)...F'(tN)}) to generate a second predicted feature set (e.g., F'(t)). The second neural network-based predictor 120 may receive as input the third feature set (i.e., a feature set associated with a 3D point of each reconstructed reference PCF in the set of reconstructed reference PCFs) and the reconstructed coordinate information (i.e., Ĉ(t)). The second neural network-based predictor 120 may predict the second predicted feature set (i.e., F'(t)) as an output. The generation of the second predicted feature set (based on the third feature set) may involve applying a second neural network-based predictor 114 (based on the second feature set, i.e., {F(t-1)...F(tN)}) to the first predicted feature set (i.e., F'(t)) using the first neural network-based predictor 114. ~ It can be identical to the production of (t).

[0077] The second series of transformation operations can further include applying a selected gain vector to the second predicted feature set (i.e., F'(t)) to generate a third gain-modulated feature set. The selected gain vector is a function of the first feature set (i.e., F(t)) and the first predicted feature set (i.e., F ~ (t)) to generate the first and second gain-modulated feature sets, respectively. The second series of transformation operations may be the same gain vectors used to modulate the first and second gain-modulated feature sets (i.e., the reconstructed residual feature set or R ~ The accumulator 324 may further include applying an additive operation to the third gain modulation feature set and the reconstructed residual feature set (i.e., R ~ The accumulator 324 may receive as input a fourth feature set (i.e., F″(t)) associated with 3D points having coordinate information included in the reconstructed coordinate information (i.e., Ĉ(t)). The accumulator 324 may generate as output a fourth feature set (i.e., F″(t)) associated with 3D points having coordinate information included in the reconstructed coordinate information (i.e., Ĉ(t)).

[0078] The second series of transform operations may further include applying the selected inverse gain vector to the fourth feature set (i.e., F″(t)) to generate a gain demodulation feature set. According to an embodiment, the circuit 202 may be configured to retrieve the inverse gain vector set 122 from the memory 204. Each inverse gain vector in the inverse gain vector set 122 may correspond to a decoding rate associated with the decoding pipeline 110. Once the inverse gain vector set 122 is retrieved, the circuit 202 may select one inverse gain vector from the inverse gain vector set 122. The selected inverse gain vector may correspond to the gain vector selected (in generating the third gain modulation feature set). The selected inverse gain vector may demodulate the features of the fourth feature set such that the current PCF (i.e., P(t)) is reconstructed at the decoding rate corresponding to the selected inverse gain vector.

[0079] According to an embodiment, the circuit 202 can receive a third input, which can include a selection of a first decoding rate for reconstruction of the current PCF (i.e., P(t)). The selected first decoding rate can correspond to the first encoding rate. The circuit 202 can determine an inverse gain vector from the inverse gain vector set 122 that corresponds to a decoding rate matching the first decoding rate. Once the circuit 202 determines such an inverse gain vector from the inverse gain vector set 122, it can select the inverse gain vector. Thus, the inverse gain vector can be selected based on the third input. Thus, a gain demodulation feature set can be generated based on demodulation of features of the fourth feature set (i.e., F″(t)) with the selected inverse gain vector. The current PCF of the PCF sequence can be reconstructed (at the first decoding rate) based on the selected inverse gain vector.

[0080] The second series of transformation operations may further include applying a neural network-based decoder 118 to the gain demodulation feature set to reconstruct the current PCF (i.e., P(t)). The neural network-based decoder 118 may receive the gain demodulation feature set as an input and generate a reconstructed PCF (e.g., P'(t)) corresponding to the current PCF as an output.

[0081] Meanwhile, in the decoding path selected as the I-frame decoding path 320, the second series of transformation operations may include applying a selected inverse gain vector to the decoded feature set (generated based on decoding of the first gain-modulated feature set) to generate a gain-demodulated feature set. The selected inverse gain vector may demodulate the features of the decoded feature set such that the current PCF (i.e., P(t)) is reconstructed at a decoding rate corresponding to the selected inverse gain vector. Thus, the current PCF of the PCF sequence may be reconstructed (at the first decoding rate) based on the selected inverse gain vector. The second series of transformation operations may further include applying a neural network-based decoder 118 to the gain-demodulated feature set to reconstruct the current PCF (i.e., P(t)). The neural network-based decoder 118 may receive the gain-demodulated feature set as an input and generate a reconstructed PCF (i.e., P'(t)) corresponding to the current PCF as an output.

[0082] FIG. 4 is a block diagram illustrating an example operation for generating a gain vector set and encoding a point cloud frame at different encoding rates based on the gain vector set, according to an embodiment of the present disclosure. The description of FIG. 4 is provided in relation to elements in FIGS. 1, 2, 3A, and 3B. FIG. 4 illustrates an example scenario 400. The example scenario 400 illustrates an example module, which may be a neural network-based encoder 112, a first neural network-based predictor 114, or a second neural network-based predictor 120. The example module may receive an input (represented by X), which may be a PCF sequence (input of the neural network-based encoder 112), a second feature set (input of the first neural network-based predictor 114), or a third feature set (input of the second neural network-based predictor 120). Also illustrated is a one-hot encoder 402 and a first neural network (e.g., a multilayer perceptron 404). The multi-layer perceptron 404 can be trained based on inputs received from the one-hot encoder 402. The trained multi-layer perceptron 404 can generate a gain vector set 116. The gain vector set 116 can modulate features generated by the neural network-based encoder 112, the first neural network-based predictor 114, or the second neural network-based predictor 120.

[0083] At any point in time, the circuit 202 may retrieve a first set of bits per point (BPP) values ​​associated with a set of encoding rates for the encoding pipeline 108. The first set of BPP values ​​may be provided as an input to the one-hot encoder 402. The first set of BPP values ​​may be received as an input indicating a rate at which the current PCF needs to be encoded. The one-hot encoder 402 may encode the first set of BPP values ​​based on a one-hot encoding scheme to generate an encoded BPP value set. The one-hot encoder 402 may generate an encoded BPP value of the encoded BPP value set for each input (i.e., a BPP value represented as β in the first BPP value set). The encoded BPP values ​​may be one-hot vectors. The circuit 202 supplies each encoded BPP value of the encoded BPP value set to a first neural network, i.e., a multilayer perceptron 404. The encoded BPP value set (i.e., a one-hot vector set) may be supplied as an input to the first neural network to train the first neural network.

[0084] The first neural network (i.e., the multilayer perceptron 404) can generate an output for each coded BPP value (input) in the coded BPP value set. Thus, a series of outputs can be generated for the coded BPP value set. The circuit 202 can calculate products (denoted as Yβ) of the generated outputs with either the first feature set (the output of the neural network-based encoder 112, denoted as Y), the first predicted feature set (the output of the first neural network-based predictor 114, denoted as Y), or the second predicted feature set (the output of the second neural network-based predictor 120, denoted as Y). Thus, a series of products can be calculated for the series of outputs. The first neural network (i.e., the multilayer perceptron 404) can be trained based on the calculated series of products. Once the training of the first neural network (i.e., the multilayer perceptron 404) is complete, a gain vector 116 can be generated. Thus, each gain vector in the gain vector set 116 may be based on the training of a first neural network (ie, multi-layer perceptron 404).

[0085] After generating the gain vector set 116, the first neural network (i.e., the multi-layer perceptron 404) can be discarded. The circuit 202 discards the first feature set (i.e., F(t)), the first predicted feature set (i.e., F(t)), and the first predicted feature set (i.e., F(t)) from the gain vector set 116. ~ A gain vector can be selected that modulates the first predicted feature set (i.e., F(t)), or the second predicted feature set (i.e., F'(t)). ~ At least one feature of F'(t) can be modulated such that the current PCF (i.e., P(t)) is coded at a coding rate corresponding to the selected gain vector, while features of F'(t) can be modulated (at a coding rate corresponding to the selected gain vector) such that P(t) is decoded at a decoding rate corresponding to the coding rate or a reconstructed PCF (i.e., P'(t)) is generated.

[0086] However, in some scenarios, it may be necessary to encode P(t) at a coding rate that does not match any coding rate corresponding to a gain vector in the gain vector set 116. Such a scenario may occur by receiving a second input. The second input may include a selection of a second coding rate for which a corresponding gain vector does not exist in the gain vector set 116. In such a scenario, the circuit 202 may generate an intermediate gain vector based on interpolation or extrapolation using at least one gain vector from the gain vector set 116. For example, the gain vector set 116 may include g1 corresponding to 0.04 BPP, g2 corresponding to 0.05 BPP, g3 corresponding to 0.06 BPP, g4 corresponding to 0.07 BPP, and g5 corresponding to 0.08 BPP. In the first scenario, the second input may include a selection of a second coding rate that is 0.055 BPP for which a corresponding gain vector does not exist in the gain vector set 116. The second coding rate does not match any coding rate corresponding to a gain vector in the gain vector set 116. Therefore, an intermediate gain vector can be generated based on interpolation using two gain vectors g2 and g3. According to one embodiment, the intermediate gain vector (e.g., g 23 ) can be generated as follows: TIFF2026502167000002.tif13150

[0087] In a second scenario, the second input may include a selection of a second encoding rate that is 0.035 BPP. For this second encoding rate, an intermediate gain vector may be generated based on extrapolation using gain vector g. According to one embodiment, the intermediate gain vector (e.g., g 11 ) to g 11= g1 × s, s∈(0.8,1). In a third scenario, the second input may include a selection of a second coding rate of 0.085 BPP. For this second coding rate, an intermediate gain vector may be generated based on extrapolation using the gain vector g5. According to one embodiment, the intermediate gain vector (e.g., g 55 ) to g 55 = g5 × s, s∈(1,1.15).

[0088] The circuit 202 calculates the generated intermediate gain vector (i.e., g 23 , g 11 , or g 55 ) can be further configured to update the gain vector set 116 to include F(t), F(t). Thus, the gain vector set 116 can be expanded to include additional gain vectors that allow the current PCF to be coded at a wide range of coding rates without the need to retrain any of the neural network-based encoder 112, the first neural network-based predictor 114, or the second neural network-based predictor 120. ~ The selected gain vector that can be used to modulate the characteristics of F'(t), or F'(t), can be the intermediate gain vector generated.

[0089] FIG. 5 is a block diagram illustrating an example operation for generating an inverse gain vector set and decoding a point cloud frame based on the inverse gain vector set, according to an embodiment of the present disclosure. FIG. 5 is described with reference to elements in FIGS. 1, 2, 3A, 3B, and 4. FIG. 5 illustrates an example scenario 500. The example scenario 500 illustrates a neural network-based decoder 118. The neural network-based decoder 118 can generate a reconstructed PCF (represented by X′) that corresponds to the current PCF. Also shown is a one-hot encoder 502 and a second neural network (e.g., a multilayer perceptron 504). The multilayer perceptron 504 can be trained based on inputs received from the one-hot encoder 502. The trained multilayer perceptron 504 can then calculate the reconstructed PCF (Q(Y β )) or (Q(Y β ) can be used to demodulate the features of the decoded feature set.

[0090] At any point in time, the circuit 202 can retrieve a second BPP value set associated with a set of decoding rates of the decoding pipeline 110. The second BPP value set can be provided as an input to the one-hot encoder 502. The second BPP value set can be received as an input indicating a decoding rate at which the current PCF should be reconstructed. The one-hot encoder 502 can encode the second BPP value set based on a one-hot encoding scheme to generate an encoded BPP value set (i.e., a one-hot vector set). The one-hot encoder 502 can generate an encoded BPP value (i.e., a one-hot vector) of the encoded BPP value set for each input (i.e., a BPP value of the second BPP value set, represented as β). The circuit 202 can provide each encoded BPP value of the encoded BPP value set to a second neural network, i.e., a multilayer perceptron 504. The encoded BPP value set (i.e., a one-hot vector set) can be provided as an input to the second neural network to train the second neural network. The second neural network (i.e., multi-layer perceptron 504) can generate an output for each coded BPP value (input) in the coded BPP value set. Thus, a series of outputs can be generated for the coded BPP value set. The circuit 202 then calculates the generated outputs and the (Q(Y β )) a fourth feature set or (Q(Y β ) with either the decoded feature set (represented by ) or the decoded feature set (represented by ). Thus, a series of products can be calculated for a series of outputs. A second neural network (i.e., multilayer perceptron 504) can be trained based on the calculated series of products. Once the training of the second neural network (i.e., multilayer perceptron 504) is complete, an inverse gain vector set 122 can be generated. Thus, each inverse gain vector in the inverse gain vector set 122 can be generated based on the training of the second neural network (i.e., multilayer perceptron 504).

[0091] After generating the inverse gain vector set 122, the second neural network (i.e., the multilayer perceptron 504) can be discarded. The circuit 202 can select an inverse gain vector from the inverse gain vector set 122 that demodulates features of a fourth feature set or features of a decoded feature set (which can be generated based on decoding the first gain modulation feature set). The features of the fourth feature set or features of the decoded feature set can be demodulated so that the current PCF (i.e., P(t)) is reconstructed at a decoding rate corresponding to the selected inverse gain vector. However, in some scenarios, it may be necessary to reconstruct (decode) P(t) at a decoding rate that does not match any of the decoding rates corresponding to the inverse gain vectors in the inverse gain vector set 122. This is due to the receipt of a second input including a selection of a second coding rate (not present in the corresponding gain vector set 116) and a fourth input. The fourth input can include a selection of a second decoding rate for which a corresponding inverse gain vector is not present in the inverse gain vector set 122. In such a scenario, circuit 202 may generate intermediate inverse gain vectors based on extrapolation using one inverse gain vector from inverse gain vector set 122. For example, inverse gain vector set 122 may include ig1 corresponding to 25 units, ig2 corresponding to 20 units, ig3 corresponding to 16.67 units, ig4 corresponding to 14.28 units, and ig5 corresponding to 12.5 units.

[0092] In a first scenario, the fourth input may include a selection of a second decoding rate of 30 units. For this second decoding rate, an intermediate inverse gain vector may be generated based on extrapolation using the inverse gain vector ig1. According to an embodiment, the intermediate inverse gain vector (e.g., ig 11 )of In a second scenario, a fourth input may include a selection of a second decoding rate of 10 units. For this second decoding rate, an intermediate inverse gain vector may be generated based on extrapolation using the inverse gain vector ig5. According to an embodiment, the intermediate inverse gain vector (e.g., ig 55 )of The file can be generated as TIFF2026502167000004.tif13150.

[0093] The circuit 202 generates the intermediate inverse gain vector (i.e., ig 11 , or ig 55 ) The inverse gain vector set 122 can be further configured to update the inverse gain vector set 122 to include the inverse gain vectors ( ) and ( ). Thus, the inverse gain vector set 122 can be expanded to include additional inverse gain vectors that allow the current PCF to be decoded at a wide range of decoding rates without the need to retrain the neural network-based decoder 118. The selected inverse gain vectors that can be used to demodulate features of the fourth Feature Set or features of the decoded Feature Set can be the generated intermediate inverse gain vectors.

[0094] Figure 6 is a flowchart illustrating operations of an exemplary method for dynamic point cloud compression using a rate-adaptive unified I-frame and P-frame codec architecture, according to an embodiment of the present disclosure. Figure 6 is described with reference to elements of Figures 1, 2, 3A, 3B, 4, and 5. Figure 6 shows a flowchart 600. Operations 602-610 can be performed by any computer system, such as the electronic device 102 or the circuitry 202 of the electronic device 102. Operations can begin at 602 and proceed to 604.

[0095] At 604, a PCF sequence may be received that may include a reference PCF set and a current PCF to be encoded. In at least one embodiment, circuit 202 may be configured to receive a PCF sequence that may include a reference PCF set and a current PCF to be encoded. Details of receiving the reference PCF set and the current PCF are described, for example, in Figures 1 and 3A.

[0096] At 606, 3D feature data may be generated based on applying the neural network-based encoder 112 to at least one of the reference PCF set and the current PCF. In at least one embodiment, the circuit 202 may be configured to generate the 3D feature data based on applying the neural network-based encoder 112 to at least one of the reference PCF set and the current PCF. Details of generating the 3D feature data are described, for example, in Figures 1 and 3A.

[0097] At 608, a coding path may be selected from the coding pipeline 108 as one of an I-frame coding path or a P-frame coding path for the current PCF. In at least one embodiment, the circuit 202 may be configured to select a coding path from the coding pipeline 108 as one of an I-frame coding path or a P-frame coding path for the current PCF. The selected coding path extends from the output of the neural network-based encoder 112 and may include a first series of transformation operations that may be applied to the 3D feature data. Details of the coding selection are described, for example, in FIGS. 1 and 3A.

[0098] At 610, a bitstream of coded PCFs may be generated based on applying a first series of transformation operations to the 3D feature data. In at least one embodiment, the circuit 202 may be configured to generate a bitstream of coded PCFs based on applying a first series of transformation operations to the 3D feature data. Details of generating a bitstream of coded PCFs are described, for example, in Figures 1, 3A, 3B, 4, and 5. Control may proceed to an end.

[0099] Although flowchart 600 is depicted as discrete operations such as 604, 606, 608, and 610, the disclosure is not so limited. Thus, in some embodiments, such discrete operations may be further divided into additional operations, combined into fewer operations, or eliminated, depending on the implementation, without departing from the essence of the disclosed embodiments.

[0100] Various embodiments of the present disclosure may provide a non-transitory computer-readable medium and / or storage medium and / or computer-readable recording medium storing computer-executable instructions executable by a machine and / or a computer to operate an electronic device (e.g., electronic device 102 of FIG. 1 ). Such instructions may cause electronic device 102 to perform operations including storing an encoding pipeline (e.g., encoding pipeline 108 of FIG. 1 ) including a neural network-based encoder (e.g., neural network-based encoder 112 of FIG. 1 ) and a gain vector set (e.g., gain vector set 116 of FIG. 1 ). The operations may further include storing a decoding pipeline (e.g., decoding pipeline 110 of FIG. 1 ) including a neural network-based decoder (e.g., neural network-based decoder 118 of FIG. 1 ) and an inverse gain vector set (e.g., inverse gain vector set 122 of FIG. 1 ). The operations may further include receiving a PCF sequence including a reference PCF set and a current PCF to be encoded, and generating 3D feature data based on applying neural network-based encoder 112 to at least one of the reference PCF set and the current PCF. The 3D feature data may include a first feature set associated with the current PCF and a second feature set associated with a reference PCF set. The operations may further include selecting an encoding path for the current PCF from the encoding pipeline 108 as one of an I-frame encoding path or a P-frame encoding path. The selected encoding path may extend from an output of the neural network-based encoder 112 and may include a first series of transform operations that may be applied to the 3D feature data. In the encoding path selected as the I-frame encoding path, the first series of transform operations may include applying a selected gain vector to the first feature set to generate a first gain-modulated feature set.In the coding path selected as the P frame coding path, the first series of transformation operations can include applying a first neural network-based predictor 114 to the second feature set to generate a first predicted feature set, applying a selected gain vector to the first feature set to generate a first gain modulated feature set, applying the selected gain vector to the first predicted feature set to generate a second gain modulated feature set, and applying a difference operation to the first gain modulated feature set and the second gain modulated feature set to generate a residual feature set. The operations can further include generating a bitstream of a coded PCF based on applying the first series of transformation operations to the 3D feature data. The bitstream of the coded PCF can be generated based on the first gain modulated feature set or the residual feature set.

[0101] An exemplary embodiment of the present disclosure may include an electronic device (such as the electronic device 102 of FIG. 1) that may include a circuit (such as the circuit 202 of FIG. 2). The electronic device 102 may further include a memory (such as the memory 204 of FIG. 2) that may be configured to store an encoding pipeline (such as the encoding pipeline 108 of FIG. 1) that includes a neural network-based encoder (such as the neural network-based encoder 112 of FIG. 1) and a gain vector set (such as the gain vector set 116 of FIG. 1). The memory 204 may be further configured to store a decoding pipeline (such as the decoding pipeline 110) that includes a neural network-based decoder (such as the neural network-based decoder 118 of FIG. 1) and an inverse gain vector set (such as the inverse gain vector set 122 of FIG. 1). The circuit 202 may be configured to receive a PCF sequence that includes a reference PCF set and a current PCF to be encoded. The circuit 202 may be further configured to generate 3D feature data based on applying the neural network-based encoder 112 to at least one of the reference PCF set and the current PCF. The 3D feature data may include a first feature set associated with the current PCF and a second feature set associated with a reference PCF set. The circuit 202 may be further configured to select an encoding path for the current PCF from the encoding pipeline 108 as one of an I-frame encoding path or a P-frame encoding path. The selected encoding path may extend from an output of the neural network-based encoder 112 and may include a first series of transform operations that may be applied to the 3D feature data. In the encoding path selected as the I-frame encoding path, the first series of transform operations may include applying a selected gain vector to the first feature set to generate a first gain-modulated feature set.In the coding path selected as the P frame coding path, the first series of transformation operations may include applying a first neural network-based predictor 114 to the second feature set to generate a first predicted feature set, applying a selected gain vector to the first feature set to generate a first gain modulated feature set, applying the selected gain vector to the first predicted feature set to generate a second gain modulated feature set, and applying a difference operation to the first gain modulated feature set and the second gain modulated feature set to generate a residual feature set. The circuit 202 may be further configured to generate a bitstream of coded PCFs based on applying the first series of transformation operations to the 3D feature data. The bitstream of coded PCFs may be generated based on the first gain modulated feature set or the residual feature set.

[0102] According to an embodiment, the circuit 202 may be further configured to retrieve the gain vector set 116 from the memory 204. Each gain vector in the gain vector set 116 may correspond to a coding rate associated with the coding pipeline 108. The circuit 202 may be further configured to select a gain vector from the gain vector set 116. The bitstream of the coded PCF may be generated further based on the selected gain vector.

[0103] According to an embodiment, the circuit 202 can be further configured to receive a first input that can include a selection of a first coding rate for the current PCF. A gain vector can be selected based on the first input.

[0104] According to an embodiment, the circuit 202 can be further configured to receive a second input, which can include a selection of a second coding rate for which a corresponding gain vector is not present in the gain vector set 116. The circuit 202 can be further configured to generate an intermediate gain vector based on interpolation or extrapolation using at least one gain vector of the gain vector set 116. The circuit 202 can be further configured to update the gain vector set 116 to include the generated intermediate gain vector. The selected gain vector can be the intermediate gain vector.

[0105] According to an embodiment, the circuit 202 may be further configured to retrieve a first set of BPP values ​​associated with a set of encoding rates for the encoding pipeline 108. The circuit 202 may be further configured to encode the first set of BPP values ​​based on a one-hot encoding scheme to generate an encoded BPP value set. The circuit 202 may be further configured to provide each encoded BPP value of the encoded BPP value set to a first neural network. The circuit 202 may be further configured to generate an output of the first neural network for each encoded BPP value of the encoded BPP value set. The circuit 202 may be further configured to calculate a product of the generated output and the first feature set. The circuit 202 may be further configured to train the first neural network based on the calculated product. The circuit 202 may be further configured to generate each gain vector of the gain vector set based on the training of the first neural network.

[0106] According to one embodiment, the circuit 202 may be further configured to apply a decoding operation to the bitstream of the coded PCF to generate a decoded Feature Set for the current PCF. The circuit 202 may be further configured to select a decoding path from the decoding pipeline 110 as either an I-frame decoding path or a P-frame decoding path. The decoding path may include a second set of transform operations that may be applied to the decoded Feature Set. For a decoding path selected as the I-frame decoding path, the second set of transform operations may include applying a selected inverse gain vector to the decoded Feature Set to generate a gain-demodulated Feature Set and applying the neural network-based decoder 118 to the gain-demodulated Feature Set to reconstruct the current PCF. In the decoding path selected as the P-frame decoding path, the second series of transformation operations may include receiving a PCF sequence that may include a reconstructed reference PCF set, generating a third feature set based on applying a neural network-based encoder 112 to the reconstructed reference PCF set, applying a second neural network-based predictor 120 to the third feature set to generate a second predicted feature set, applying a selected gain vector to the second predicted feature set to generate a third gain-modulated feature set, applying an additive operation to the third gain-modulated feature set and the decoded feature set to generate a fourth feature set, applying the selected inverse gain vector to the fourth feature set to generate a gain-modulated feature set, and applying a neural network-based decoder 118 to the gain-modulated feature set to reconstruct a current PCF. The circuit 202 may be further configured to reconstruct a current PCF of the PCF sequence based on applying the second series of transformation operations to the decoded feature set.

[0107] According to an embodiment, the circuit 202 may be further configured to retrieve a gain vector set from the memory 204. Each gain vector in the gain vector set 116 may correspond to a coding rate associated with the encoding pipeline 108. The circuit 202 may be further configured to select a gain vector from the gain vector set 116. The circuit 202 may be further configured to retrieve an inverse gain vector set from the memory 204. Each inverse gain vector in the inverse gain vector set 122 may correspond to a decoding rate associated with the decoding pipeline 110. The circuit 202 may be further configured to select an inverse gain vector from the inverse gain vector set 122. The selected inverse gain vector may correspond to the selected gain vector. A current PCF of the PCF sequence may be further reconstructed based on the selected inverse gain vector.

[0108] According to an embodiment, the circuit 202 may be further configured to receive a third input including a selection of a first decoding rate for the reconstruction of the current PCF, and the inverse gain vector may be selected based on the third input.

[0109] According to an embodiment, the circuit 202 may be further configured to receive a fourth input, which may include a selection of a second decoding rate for which a corresponding inverse gain vector is not present in the inverse gain vector set 122. The circuit 202 may be further configured to generate an intermediate inverse gain vector based on extrapolation using an inverse gain vector of the inverse gain vector set 122. The circuit 202 may be further configured to update the inverse gain vector set 122 to include the generated intermediate inverse gain vector. The selected inverse gain vector may be the intermediate inverse gain vector.

[0110] According to an embodiment, the circuit 202 may be further configured to retrieve a second set of BPP values ​​associated with a set of decoding rates for the decoding pipeline 110. The circuit 202 may be further configured to encode the second set of BPP values ​​based on a one-hot encoding scheme to generate an encoded BPP value set. The circuit 202 may be further configured to provide each encoded BPP value of the encoded BPP value set to a second neural network. The circuit 202 may be further configured to generate an output of the second neural network for each encoded BPP value of the encoded BPP value set. The circuit 202 may be further configured to calculate a product of the generated output and the decoded feature set. The circuit 202 may be further configured to train the second neural network based on the calculated product. The circuit 202 may be further configured to generate each inverse gain vector of the inverse gain vector set 122 based on training the second neural network.

[0111] The present disclosure can be implemented in hardware or a combination of hardware and software. The present disclosure can be implemented in a centralized manner in at least one computer system, or in a distributed manner where different elements can be distributed across several interconnected computer systems. Any computer system or other device adapted to perform the methods described herein can be suitable. The combination of hardware and software can be a general-purpose computer system that includes a computer program that, when loaded and executed, can control the computer system to perform the methods described herein. The present disclosure can be implemented in hardware, including portions of integrated circuits that also perform other functions.

[0112] The present disclosure may also be embodied in a computer program product, which includes all features that enable the implementation of the methods described herein and which is capable of executing these methods when loaded into a computer system. A computer program in this context means any expression, in any language, code or notation, of a set of instructions intended to cause a system having information processing capabilities to perform a particular function, either directly, or after a) conversion into another language, code or notation, or b) reproduction in a different content form, or both.

[0113] While the present disclosure has been described with reference to several embodiments, those skilled in the art will recognize that various modifications may be made and equivalents may be substituted without departing from the scope of the disclosure. Additionally, many modifications may be made to adapt a particular situation or material to the teachings of the disclosure without departing from the scope of the disclosure. Therefore, it is not intended that the disclosure be limited to the particular embodiments disclosed, but rather, it is intended to include all embodiments falling within the scope of the appended claims. [Explanation of symbols]

[0114] 600 Flowchart 602 start 604 receives a sequence of point cloud frames (PCFs) including a set of reference (PCFs) and a current PCF to be encoded 606 Generate 3D feature data based on applying a neural network-based encoder to at least one of the reference PCF set and the current PCF. 608 selects a coding path from the coding pipeline that extends from the output of the neural network-based encoder and includes a first series of transformation operations applicable to the 3D feature data as either an I-frame coding path or a P-frame coding path for the current PCF. 610 generating a bitstream of coded PCFs based on applying a first series of transformation operations to the 3D feature data;

Claims

1. 1. An electronic device comprising: a memory configured to store an encoding pipeline including a neural network based encoder and a set of gain vectors; The circuit and The circuit comprises: receiving a sequence of point cloud frames (PCFs) including a set of reference point cloud frames (PCFs) and a current PCF to be encoded; generating 3D feature data based on applying the neural network-based encoder to at least one of the reference PCF set and the current PCF; selecting an encoding path from the encoding pipeline that extends from an output of the neural network-based encoder and includes a first series of transform operations applicable to the 3D feature data as one of an I-frame encoding path or a P-frame encoding path for the current PCF; generating a coded PCF bitstream based on applying the first series of transformation operations to the 3D feature data; It is configured as follows: An electronic device characterized by:

2. The 3D feature data is a first feature set associated with the current PCF; a second feature set associated with the reference PCF set; The electronic device of claim 1 , comprising:

3. The circuit comprises: retrieving from the memory the set of gain vectors, each gain vector corresponding to a coding rate associated with the coding pipeline; selecting a gain vector from the set of gain vectors; The electronic device of claim 2 further configured to:

4. the circuitry is further configured to receive a first input including a selection of a first coding rate for the current PCF, and the gain vector is selected based on the first input. The electronic device of claim 3 .

5. The circuit comprises: receiving a second input including a selection of a second coding rate for which a corresponding gain vector is not present in the set of gain vectors; generating an intermediate gain vector based on interpolation or extrapolation using at least one gain vector from the set of gain vectors; updating the gain vector set to include the generated intermediate gain vector; wherein the selected gain vector is the intermediate gain vector. The electronic device of claim 3 .

6. The circuit comprises: retrieving a first set of bits per point (BPP) values ​​associated with a set of coding rates for the coding pipeline; encoding the first set of BPP values ​​based on a one-hot encoding scheme to generate an encoded set of BPP values; providing each coded BPP value of the set of coded BPP values ​​to a first neural network; generating an output of the first neural network for each coded BPP value in the set of coded BPP values; calculating a product of the generated output and the first feature set; training the first neural network based on the calculated products; generating each gain vector of the set of gain vectors based on training of the first neural network; The electronic device of claim 3 further configured to:

7. the coded PCF bitstream is generated further based on the selected gain vector. The electronic device of claim 3 .

8. For the coding path selected as the I-frame coding path, the first series of transformation operations includes applying the selected gain vector to the first feature set to generate first gain modulation features; the coded PCF bitstream is generated based on the first set of gain modulation characteristics; The electronic device of claim 3 .

9. For the coding path selected as the P frame coding path, the first series of transform operations comprises: applying a first neural network based predictor to the second feature set to generate a first predicted feature set; applying the selected gain vector to the first feature set to generate a first gain modulated feature set; applying the selected gain vector to the first set of predicted features to generate a second set of gain modulated features; applying a difference operation to the first gain modulated feature set and the second gain modulated feature set to generate a residual feature set; wherein the coded PCF bitstream is generated based on the residual feature set. The electronic device of claim 3 .

10. the memory is further configured to store a decoding pipeline including a neural network based decoder and an inverse gain vector set. The electronic device of claim 1 .

11. The circuit comprises: applying a decoding operation to the bitstream of the coded PCF to generate a decoded feature set of the current PCF; selecting a decoding path from the decoding pipeline that includes a second series of transform operations applicable to the decoded feature set as one of an I-frame decoding path or a P-frame decoding path; reconstructing the current PCF of the sequence of PCFs based on applying the second series of transformation operations to the decoded feature set; The electronic device of claim 10 further configured to:

12. The circuit comprises: retrieving from the memory the set of gain vectors, each gain vector corresponding to a coding rate associated with the coding pipeline; selecting a gain vector from the set of gain vectors; retrieving from the memory the set of inverse gain vectors, each inverse gain vector corresponding to a decoding rate associated with the decoding pipeline; selecting an inverse gain vector from the set of inverse gain vectors that corresponds to the selected gain vector; The electronic device of claim 11 further configured to:

13. the circuit is further configured to receive a third input including a selection of a first decoding rate for reconstruction of the current PCF, and the inverse gain vector is selected based on the third input.

13. The electronic device of claim 12.

14. The circuit comprises: receiving a fourth input including a selection of a second decoding rate for which a corresponding inverse gain vector is not present in the set of inverse gain vectors; generating an intermediate inverse gain vector based on extrapolation using an inverse gain vector from the set of inverse gain vectors; updating the set of inverse gain vectors to include the generated intermediate inverse gain vectors; wherein the selected inverse gain vector is the intermediate inverse gain vector.

13. The electronic device of claim 12.

15. The circuit comprises: retrieving a second set of bits per point (BPP) values ​​associated with a range of decoding rates for the decoding pipeline; encoding the second set of BPP values ​​based on a one-hot encoding scheme to generate an encoded set of BPP values; providing each coded BPP value of the set of coded BPP values ​​to a second neural network; generating an output of the second neural network for each coded BPP value in the set of coded BPP values; computing a product of the generated output and the decoded feature set; training the second neural network based on the calculated products; generating each inverse gain vector of the set of inverse gain vectors based on training of the second neural network; The electronic device of claim 12 further configured to:

16. the current PCF of the PCF sequence is reconstructed further based on the selected inverse gain vector.

13. The electronic device of claim 12.

17. For the decoding path selected as the I-frame decoding path, the second series of conversion operations comprises: applying the selected inverse gain vector to the decoded feature set to generate a gain-demodulated feature set; applying the neural network based decoder to the gain demodulation feature set to reconstruct the current PCF; 13. The electronic device of claim 12, comprising:

18. For the decoding path selected as the P frame decoding path, the second series of transformation operations comprises: receiving a PCF sequence including a reconstructed reference PCF set; generating a third feature set based on applying the neural network-based encoder to the reconstructed reference PCF set; applying a second neural network based predictor to the third feature set to generate a second predicted feature set; applying the selected gain vector to the second set of predicted features to generate a third set of gain modulated features; applying an additive operation to the third gain modulated feature set and the decoded feature set to generate a fourth feature set; applying the selected inverse gain vector to the fourth feature set to generate a gain demodulated feature set; applying the neural network based decoder to the gain demodulation feature set to reconstruct the current PCF; 13. The electronic device of claim 12, comprising:

19. In an electronic device including a memory, storing in the memory an encoding pipeline including a neural network based encoder and a set of gain vectors; receiving a sequence of point cloud frames (PCFs) including a set of reference point cloud frames (PCFs) and a current PCF to be encoded; generating 3D feature data based on applying the neural network-based encoder to at least one of the reference PCF set and the current PCF; selecting a coding path from the coding pipeline extending from an output of the neural network-based encoder and including a first series of transformation operations applicable to the 3D feature data as one of an I-frame coding path or a P-frame coding path for the current PCF; generating a coded PCF bitstream based on applying the first series of transformation operations to the 3D feature data; A method comprising:

20. retrieving from the memory the set of gain vectors, each gain vector corresponding to a coding rate associated with the coding pipeline; selecting a gain vector from the set of gain vectors; further comprising the coded PCF bitstream is generated further based on the selected gain vector.

20. The method of claim 18.

Images