Rate adaptive codec for dynamic point cloud compression

Abstract

An electronic device and method for dynamic point cloud compression using a rate adaptive codec are provided. An electronic device receives a sequence of point cloud frames (PCFs) that includes a set of reference PCFs and a current PCF to encode. The electronic device generates 3D feature data according to an application of the neural network-based encoder to at least one of the current PCF and the set of reference PCFs. The electronic device selects a coding path from the coding pipeline as an I-frame coding path or a P-frame coding path for the current PCF. The selected encoding path extends from an output of the neural network-based encoder and includes a first sequence of transform operations applicable to the 3D feature data. The electronic device generates a bitstream of the encoded PCF based on application of the first sequence of transformation operations to the 3D feature data.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS / INCORPORATED BY REFERENCE

[0002] This application claims the benefit of priority to U.S. Application No. 18 / 509,138 filed in the U.S. Patent and Trademark Office on November 14, 2023, which claims the benefit of priority to U.S. Provisional Patent Application Serial No. 63 / 476,589 filed on December 21, 2022, the entire contents of which are incorporated herein by reference. Background Art

[0003] Advances in the field of dynamic point cloud compression (PCC) have led to the development of point cloud encoding methods for efficient representation of point cloud data associated with 3D objects. Point cloud data can be compressed (i.e., encoded) to generate encoded point cloud data, which can be stored, processed, or transmitted. Subsequently, the original point cloud can be reconstructed based on the encoded point cloud data. Point cloud data associated with a current point cloud frame can be encoded or decoded based on a current point cloud frame (i.e., an I-frame codec) or a set of previous point cloud frames (i.e., a P-frame codec). Each of these encoding or decoding schemes (I-frame or P-frame codec) can have its own unique advantages and disadvantages. However, using the two encoding or decoding schemes in combination can provide certain advantages that may not be obtained if either is used alone. However, for combined use, each of the I-frame codec and the P-frame codec may require a separate codec network even if the architecture of each codec network includes the same encoder and decoder. Each codec network may include its own machine learning-based encoder and decoder, and each codec network may require to be trained separately. Therefore, having separate codecs for I-frame and P-frame encoding / decoding may increase the latency involved in training. The increased latency may be due to the increase in several network parameters whose values ​​may be required to be calculated to obtain an efficient I-frame and P-frame codec. The increased latency and computational cost may discourage the use of a combined I-frame and P-frame codec.

[0004] By comparing the described system with certain aspects of the present disclosure as set forth in the remainder of this application and with reference to the accompanying figures, the limitations and disadvantages of conventional and traditional approaches will become clear to those skilled in the art. Summary of the invention

[0005] As more fully set forth in the claims, substantially as shown in and / or described in conjunction with at least one of the figures, there is provided an electronic device and method for dynamic point cloud compression using a rate adaptive codec.

[0006] These and other features and advantages of the present disclosure may be understood by reviewing the following detailed description of the disclosure and the accompanying drawings, in which like reference numerals refer to like parts throughout. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 is a diagram illustrating an exemplary network environment including a rate adaptive codec for dynamic point cloud compression according to an embodiment of the present disclosure.

[0008] Figure 2 is a block diagram illustrating an exemplary electronic device including a rate adaptive codec for dynamic point cloud compression according to an embodiment of the present disclosure.

[0009] Figure 3A and Figure 3B is a diagram illustrating an exemplary rate adaptive unified I-frame and P-frame codec architecture for dynamic point cloud compression and point cloud reconstruction according to an embodiment of the present disclosure.

[0010] Figure 4 is a block diagram illustrating exemplary operations for generating a set of gain vectors and encoding a point cloud frame at different encoding rates based on the set of gain vectors according to an embodiment of the present disclosure.

[0011] Figure 5 is a block diagram illustrating exemplary operations for generating a set of inverse gain vectors and decoding a point cloud frame based on the set of inverse gain vectors according to an embodiment of the present disclosure.

[0012] Figure 6 is a flow chart illustrating the operation of an exemplary method for dynamic point cloud compression using a rate adaptive unified I-frame and P-frame codec architecture in accordance with an embodiment of the present disclosure. DETAILED DESCRIPTION

[0013] The embodiments described below may be found in the disclosed electronic devices and methods for providing a rate adaptive codec for dynamic point cloud compression. Exemplary aspects of the present disclosure provide an electronic device (e.g., a computing device, a gaming console, or a virtual reality goggles) comprising a rate adaptive codec. The rate adaptive codec may include an encoding pipeline and a decoding pipeline. The encoding pipeline may unify an intra-frame (i.e., I-frame) encoder and an inter-frame (i.e., P-frame) encoder. Similarly, the decoding pipeline may unify an intra-frame (i.e., I-frame) decoder and an inter-frame (i.e., P-frame) decoder. The encoding pipeline may include a set of gain vectors. The set of gain vectors may modulate feature data associated with a point cloud frame so that the point cloud frame is encoded at a desired encoding rate for generating encoded point cloud data. The decoding pipeline may include a set of inverse gain vectors. In order to decode or reconstruct a point cloud frame, the set of inverse gain vectors may demodulate the encoded point cloud data at rates corresponding to different encoding rates.

[0014] An electronic device may receive a sequence of point cloud frames (PCFs), which may include a set of reference PCFs and a current PCF. The current PCF may be a 3D frame to be encoded (or compressed). After receiving, the electronic device may generate three-dimensional (3D) feature data based on the application of a neural network-based encoder included in the encoding pipeline to the current PCF and at least one of the reference PCFs in the set. Once the 3D feature data is generated, the electronic device may select an encoding path from the encoding pipeline. The encoding path may be selected as an encoding path (i.e., an intra-frame encoder) or a P-frame encoding path (i.e., an inter-frame encoder) for (encoding of) the current PCF. The selected encoding path may extend from the output of the neural network-based encoder and may include a first transform operation sequence that may be applicable to 3D feature data. Finally, the electronic device may generate a bitstream of the encoded PCF (i.e., the encoded point cloud data) based on the application of the first transform operation sequence to the 3D feature data.

[0015] In order to reconstruct the current PCF, the electronic device may apply a decoding operation to the bit stream of the encoded PCF. Based on the application, a set of decoded features of the current PCF may be generated. Thereafter, the electronic device may select a decoding path from the decoding pipeline as an I frame decoding path (i.e., an intra-frame decoder) or a P frame decoding path (i.e., an inter-frame decoder). The selected decoding path may include a second transform operation sequence that may be applicable to the set of decoded features. The electronic device may reconstruct the current PCF of the PCF sequence based on the application of the second transform operation sequence to the set of decoded features.

[0016] Typically, a PCF may be encoded or decoded using an I-frame codec or a P-frame codec. Since the PCF includes a large amount of data, such compression and reconstruction may be necessary to enable storage, rendering, transmission, or reception of the PCF. The I-frame codec may encode / reconstruct the PCF without any reference PCF, while the P-frame codec may encode / reconstruct the PCF by using a set of reference PCFs that may be located before or after the PCF in the timeline of the PCF. The architecture of each of the I-frame and P-frame codecs may include the same machine learning-based encoder and decoder (i.e., the encoder in each of the I-frame and P-frame codecs may be the same, and the decoder in each of the I-frame and P-frame codecs may be the same). Although the same encoder / decoder is included in the architecture of the I-frame and P-frame codecs, a separate codec network may be required for each of the I-frame and P-frame codecs for combined use. Due to the inclusion of repeated encoders and decoders, this requirement may result in redundancy in the codec network. In addition, due to redundancy, the number of codec network parameters, storage costs (memory requirements), computational costs, and delays involved in training the codec network may increase. A codec network (including an I-frame codec and a P-frame codec) can be trained to encode a PCF at a specific bit rate and reconstruct a PCF from point cloud data encoded at a specific bit rate. In order to encode a PCF at a different bit rate or reconstruct a PCF from point cloud data encoded at a different bit rate, retraining of the codec network may be required. In some scenarios, if a PCF is to be encoded at different bit rates or reconstructed from point cloud data encoded at different bit rates, multiple codec networks may be used. Each of the codec networks may include the same encoder and decoder, which are trained to encode a PCF at a specific bit rate or reconstruct a PCF from point cloud data encoded at a specific bit rate. Therefore, if a PCF needs to be encoded at different bit rates or reconstructed from point cloud data encoded at different bit rates, redundancy is introduced.

[0017] In order to solve the above-mentioned problems, an electronic device may provide a rate adaptive codec, which may include a compression network trained to encode PCF at a desired bit rate (i.e., an adjustable or tunable rate), and a reconstruction network trained to reconstruct PCF regardless of the encoding rate that may be used to encode PCF. The compression network may be a unification of two encoding paths (i.e., an I frame encoding path and a P frame encoding path) into an encoding pipeline. The reconstruction network may be a unification of two decoding paths (i.e., an I frame decoding path and a P frame decoding path) into a decoding pipeline. The rate adaptive codec may allow seamless transitions between I frame and P frame encoding paths or between I frame and P frame decoding paths. The unification of the encoding path and the unification of the decoding path may lead to minimization of the number of encoders and decoders included in the encoding pipeline and the decoding pipeline, respectively. The performance of the rate adaptive codec may be improved by reducing the computational cost and latency that may be involved in training the encoding pipeline and the decoding pipeline of the rate adaptive codec. The reduction in both computational cost and latency may be due to a significant reduction in the number of network parameters (which may be due to minimization of the number of encoders and decoders included in a rate adaptive codec).

[0018] The rate adaptive codec can enable the PCF to be encoded with an adjustable coding rate (which can be a continuous value rather than a discrete value) by using a modulation network. The modulation network can provide a set of gain vectors, which can be generated based on the training of a fully connected neural network. Based on selecting a gain vector from the set of gain vectors, the features associated with the current PCF or a set of previous PCFs can be modulated for encoding the current PCF at different coding rates. The rate adaptive codec can also enable the PCF to be reconstructed from the bit stream of the coded PCF (generated based on the coding of the current PCF) by using a demodulation network. The demodulation network can include a set of inverse gain vectors, and each inverse gain vector can correspond to a gain vector in a set of gain vectors that can be used to encode the current PCF. The inverse gain vector in the set of inverse gain vectors can be selected for reconstructing the PCF from the bit stream of the coded PCF. The coded PCF can be generated based on the modulation using a gain vector that can correspond to the selected inverse gain vector. Therefore, the rate adaptive codec can enable encoding at a set of discrete rate points (coding rates).

[0019] The electronic device can generate an additional gain vector based on a set of gain vectors and generate a corresponding inverse gain vector based on a set of inverse gain vectors. The generation can be based on interpolation and / or extrapolation using gain vectors in the set of gain vectors and inverse gain vectors in the set of inverse gain vectors. The generation of additional gain vectors and inverse gain vectors can allow the PCF to be encoded at any arbitrary target bit rate.

[0020] Figure 1 is a diagram illustrating an exemplary network environment including a rate adaptive codec for dynamic point cloud compression according to an embodiment of the present disclosure. Figure 1 , a network environment 100 is shown. The network environment 100 may include an electronic device 102 and a server 104. The electronic device 102 may communicate with the server 104 via one or more networks, such as a communication network 106. The electronic device 102 may include an encoding pipeline 108 and a decoding pipeline 110. The encoding pipeline 108 may include an encoder 112 based on a neural network, a predictor 114 based on a first neural network, and a set of gain vectors 116. The decoding pipeline 110 may include a decoder 118 based on a neural network, a predictor 120 based on a second neural network, and a set of inverse gain vectors 122. The electronic device 102 may receive a point cloud frame (PCF) sequence 124 as input. The PCF sequence 124 may include a current PCF 124A and a set of reference point cloud frames (PCFs) 124B ... 124N. The encoding pipeline 108 may receive the PCF sequence 124 as input and generate a bitstream 126 of an encoded PCF associated with the current PCF 124A as output. The decoding pipeline 110 may receive as input a bitstream of encoded PCFs 126 and a set of reconstructed reference PCFs 128. The decoding pipeline 110 may generate as output a reconstructed PCF 130 (ie, a reconstructed point cloud corresponding to the current PCF 124A).

[0021] The electronic device 102 may include suitable logic, circuit systems, interfaces and / or codes that may be configured to generate 3D feature data based on the application of the PCF sequence 124 by the neural network-based encoder 112. The electronic device 102 may also select an encoding path as an I-frame encoding path or a P-frame encoding path for the current PCF 124A. Based on the selection, a first transform operation sequence may be applied to the 3D feature data for generating a bitstream 126 of the encoded PCF. The electronic device 102 may also apply a decoding operation to the bitstream 126 of the encoded PCF to generate a set of decoded features of the current PCF 124A. The electronic device 102 may also select a decoding path as an I-frame decoding path or a P-frame decoding path to decode the current PCF 124A. Based on the selection, a second transform operation sequence may be applied to the set of decoded features. Based on the application of the second transform operation sequence, a reconstructed PCF 130 that may correspond to the current PCF 124A may be generated. Examples of electronic device 102 may include, but are not limited to, computing devices, video conferencing systems, augmented reality (AR) devices, virtual reality (VR devices), mixed reality (MR) devices, gaming consoles, servers, smart wearable devices, mainframes, computer workstations, and / or consumer electronics (CE) devices.

[0022] The server 104 may include suitable logic, circuitry, interfaces, and / or code that may be configured to store the PCF sequence 124 (i.e., the current PCF 124A and a set of reference PCFs 124B ... 124N) and information associated with the PCF sequence 124. The server 104 may also be configured to receive a request for the PCF sequence 124 from the electronic device 102. The server 104 may transmit the PCF sequence 124 to the electronic device 102 based on the receipt of the request. In some embodiments, the server 104 may include an encoding pipeline 108 and / or a decoding pipeline 110, and perform associated operations of the encoding pipeline 108 and / or the decoding pipeline 110. Based on the execution, the server 104 may generate a bitstream 126 of the encoded PCF and / or a reconstructed PCF 130. The server 104 may transmit the bitstream 126 of the encoded PCF and / or the reconstructed PCF 130 to the electronic device 102. Server 104 may perform operations via web applications, cloud applications, HTTP requests, repository operations, file transfers, etc. Example implementations of server 104 may include, but are not limited to, database servers, file servers, web servers, application servers, mainframe servers, cloud computing servers, or combinations thereof.

[0023] In at least one embodiment, the server 104 can be implemented as a plurality of distributed cloud-based resources using several techniques well known to those skilled in the art. Those skilled in the art will appreciate that the scope of the present disclosure may not be limited to implementing the server 104 and the electronic device 102 as two separate entities. In some embodiments, the functionality of the server 104 may be fully or at least partially incorporated into the electronic device 102 without departing from the scope of the present disclosure.

[0024] The communication network 106 may include a communication medium through which the electronic device 102 and the server 104 can communicate with each other. The communication network 106 may be a wired or wireless communication network. Examples of the communication network 106 may include, but are not limited to, the Internet, a wireless fidelity (Wi-Fi) network, a personal area network (PAN), a local area network (LAN), or a metropolitan area network (MAN). The electronic device 102 may be configured to connect to the communication network 106 according to various wired and / or wireless communication protocols. Examples of such communication protocols may include, but are not limited to, at least one of the transmission control protocol and Internet protocol (TCP / IP), user datagram protocol (UDP), hypertext transfer protocol (HTTP), file transfer protocol (FTP), Zig Bee, EDGE, IEEE 802.11, light fidelity (Li-Fi), 802.16, IEEE 802.11s, IEEE 802.11g, multi-hop communication, wireless access point (AP), device-to-device communication, cellular communication protocol, and Bluetooth (BT) communication protocol.

[0025] The neural network based encoder 112 of the encoding pipeline 108 may include suitable logic, circuitry, interfaces and / or code that may be configured to encode each PCF included in the PCF sequence 124 to generate a set of features associated with the 3D points of each PCF. The set of features may include a first set of features associated with the current PCF 124A and a second set of features associated with a set of reference PCFs 124B ... 124N. The neural network based encoder 112 may encode the current PCF 124A to generate a first set of features associated with the 3D points in the current PCF 124A. Similarly, the neural network based encoder 112 may encode each reference PCF in the set of reference PCFs 124B ... 124N for generating a second set of features associated with the 3D points of each reference PCF. The neural network-based encoder 112 may be implemented as a deep neural network (DNN) in the form of computer executable code on a graphics processing unit (GPU), a central processing unit (CPU), a tensor processing unit (TPU), a reduced instruction set computing (RISC), an application-specific integrated circuit (ASIC), or a complex instruction set computing (CISC) processor, a coprocessor, and / or a combination thereof. In some other embodiments, the neural network-based encoder 112 may be implemented as a DNN on dedicated hardware that interfaces with other computing circuit systems of the electronic device 102. In such embodiments, the neural network-based encoder 112 may be associated with a specific form factor on a specific computing circuit system. Examples of specific computing circuit systems may include, but are not limited to, a field programmable gate array (FPGA), a programmable logic device (PLD), an ASIC, a programmable ASIC (PL-ASIC), an application-specific integrated part (ASSP), and a system on a chip (SOC) based on a standard microprocessor (MPU) or a digital signal processor (DSP). According to an embodiment, the neural network-based encoder 112 may also interface with a GPU to parallelize the operation of the neural network-based encoder 112.

[0026] Each of the first neural network-based predictor 114 and the second neural network-based predictor 120 can be a system or computational network of artificial neurons that can be generally arranged in multiple layers. Each of the first neural network-based predictor 114 and the second neural network-based predictor 120 can be defined by its hyperparameters, such as (one or more) activation functions, the number of weights, the cost function, the regularization function, the input size, the number of layers, etc. In addition, the layers can include an input layer, one or more hidden layers, and an output layer. Each of the multiple layers can include one or more nodes (or artificial neurons). The outputs of all nodes in the input layer can be coupled to at least one node of the (one or more) hidden layers. Similarly, the input of each hidden layer can be coupled to the output of at least one node in the other layers of each of the first neural network-based predictor 114 and the second neural network-based predictor 120. The output of each hidden layer can be coupled to the input of at least one node in the other layers of each of the first neural network-based predictor 114 and the second neural network-based predictor 120. The (one or more) nodes in the final layer can receive inputs from at least one hidden layer to output results. The number of layers and the number of nodes in each layer may be determined according to hyperparameters of each of the first neural network-based predictor 114 and the second neural network-based predictor 120. Such hyperparameters may be set before, during, or after training each of the first neural network-based predictor 114 and the second neural network-based predictor 120 on a training dataset.

[0027] Each node may correspond to a mathematical function (e.g., a sigmoid function or a rectified linear unit) whose parameters are adjustable during the training of each of the first neural network-based predictor 114 and the second neural network-based predictor 120. The set of parameters may include weight parameters, regularization parameters, etc. Each node may use the mathematical function to calculate an output based on one or more inputs from nodes in (one or more) other layers (e.g., (one or more) previous layers) of each of the first neural network-based predictor 114 and the second neural network-based predictor 120. All or some of the nodes of each of the first neural network-based predictor 114 and the second neural network-based predictor 120 may correspond to the same or different mathematical functions. When training each of the first neural network-based predictor 114 and the second neural network-based predictor 120, one or more parameters of each node of each of the first neural network-based predictor 114 and the second neural network-based predictor 120 may be updated based on whether the output of the final layer for a given input (from the training data set) matches the correct result according to the loss function for each of the first neural network-based predictor 114 and the second neural network-based predictor 120. The above process may be repeated for the same or different inputs until a minimum value of the loss function is achieved and the training error is minimized. Several methods for training are known in the art, such as gradient descent, stochastic gradient descent, batch gradient descent, gradient boosting, meta-heuristics, etc.

[0028] Each of the first neural network-based predictor 114 and the second neural network-based predictor 120 may be a machine learning model trained to generate multi-scale features associated with a reference PCF. The first neural network-based predictor 114 may receive as input a second set of features associated with 3D points of each reference PCF in a set of reference PCFs 124B ... 124N and target coordinate information associated with the current PCF 124A. The second neural network-based predictor 120 may receive as input a set of features associated with 3D points of each reconstructed reference PCF in a set of reconstructed reference PCFs 128 and decoded target coordinate information associated with the current PCF 124A. Each of the first neural network-based predictor 114 and the second neural network-based predictor 120 may generate a prediction as an output. The prediction may indicate the encoded features associated with the current PCF 124A.

[0029] In some embodiments, each of the first neural network-based predictor 114 and the second neural network-based predictor 120 may include electronic data that may be implemented as a software component of an application executable on the electronic device 102. Each of the first neural network-based predictor 114 and the second neural network-based predictor 120 may rely on a library, an external script, or logic / instructions for execution by a processing device included in the electronic device 102. In one or more embodiments, each of the first neural network-based predictor 114 and the second neural network-based predictor 120 may be implemented using hardware, which may include a processor, a microprocessor (e.g., for performing one or more operations or controlling the execution of one or more operations), an FPGA, or an ASIC. Alternatively, in some embodiments, each of the first neural network-based predictor 114 and the second neural network-based predictor 120 may be implemented using a combination of hardware and software. Examples of each of the first neural network-based predictor 114 and the second neural network-based predictor 120 may include, but may not be limited to, a DNN, a convolutional neural network (CNN), an artificial neural network (ANN), a fully connected neural network, a deep Bayesian neural network, or a combination of such networks. In an embodiment, each of the first neural network-based predictor 114 and the second neural network-based predictor 120 may correspond to a learning engine that can perform numerical computing techniques using a data flow graph. In some embodiments, each of the first neural network-based predictor 114 and the second neural network-based predictor 120 may be based on a hybrid architecture of multiple DNNs.

[0030] A set of gain vectors 116 included in the encoding pipeline 108 can be used to modulate features associated with the PCF sequence 124. The set of gain vectors 116 can be generated based on training of a first neural network model (e.g., a multilayer perceptron). The first neural network model can be trained based on a first set of bits per point (BPP) values, which can indicate a set of coding rates. The first set of BPP values ​​can be encoded based on a one-hot encoding scheme to generate a set of one-hot vectors. The set of one-hot vectors can be fed as input to the first neural network model for training the first neural network model. Based on the training, a set of gain vectors 116 can be generated. Each gain vector in the set of gain vectors 116 can be associated with a coding rate and modulate a first set of features associated with a current PCF 124A or a second set of features associated with a set of reference PCFs 124B ... 124N so that the current PCF 124A is encoded at the coding rate.

[0031] The neural network based decoder 118 may include suitable logic, circuit systems and / or interfaces that may be configured to reconstruct the current PCF 124A based on the bitstream 126 of the encoded PCF and at least one of a set of reconstructed reference PCFs 128. The neural network based decoder 118 may be implemented as a DNN (computer executable code format) on a GPU, a CPU, a TPU, a RISC processor, an ASIC processor, a CISC processor, a coprocessor and / or a combination thereof. In some other embodiments, the neural network based decoder 118 may be implemented as a DNN on dedicated hardware that interfaces with other computing circuit systems of the electronic device 102. In such embodiments, the neural network based decoder 118 may be associated with a specific form factor on a specific computing circuit system. Examples of specific computing circuit systems may include, but are not limited to, FPGAs, PLDs, ASICs, PL-ASICs, ASSPs, and SOCs based on standard MPUs or DSPs. In some embodiments, the neural network based decoder 118 may interface with a GPU to parallelize the operation of the neural network based decoder 118.

[0032] A set of inverse gain vectors 122 included in the decoding pipeline 110 can be used to demodulate features associated with a set of decoded features of the PCF sequence 124 (which can be generated based on the bitstream 126 of the encoded PCF). The set of inverse gain vectors 122 can be generated based on the training of a second neural network model (e.g., a multilayer perceptron). The second neural network model can be trained based on a second set of BPP values ​​that can correspond to a set of decoding rates. The set of decoding rates can correspond to a set of encoding rates. The second set of BPP values ​​can be encoded based on a one-hot encoding scheme to generate a set of one-hot vectors. The set of one-hot vectors can be fed as input to the second neural network model for training the second neural network model. Based on the training, a set of inverse gain vectors 122 can be generated. Each inverse gain vector 122 in the set of inverse gain vectors 122 can be associated with a decoding rate and demodulate features associated with a set of decoded features so that the current PCF 124A is decoded at the decoding rate for generating a reconstructed PCF 130.

[0033] Each PCF in the PCF sequence 124 (i.e., the current PCF 124A and a set of reference PCFs 124B ... 124N) may correspond to a set of geometric representations of a set of 3D objects (such as a mouse) in a 3D environment. Each PCF may constitute a set of 3D points located at different positions in the 3D space of the 3D environment. In some embodiments, the electronic device 102 may generate the PCF sequence 124 or obtain the PCF sequence 124 from the server 104. Each 3D point in each PCF in the PCF sequence 124 may include geometric information (which may include the coordinates of the corresponding 3D point in the corresponding PCF) and attribute information associated with the corresponding 3D point. The attribute information may include color information, reflectivity information, opacity information, normal vector information, material identifier information, or texture information.

[0034] The bitstream 126 of the encoded PCFs may be generated based on encoding each PCF in the sequence of PCFs 124 at one or more coding rates. The bitstream 126 of the encoded PCFs may be generated based on application of a first sequence of transform operations to at least one of a first set of features associated with the current PCF 124A and a second set of features associated with a set of reference PCFs 124B ... 124N. The first sequence of transform operations may include modulating the first set of features or the second set of features using a set of gain vectors 116 so that the current PCF 124A is encoded at one or more coding rates. The bitstream 126 of the encoded PCFs may be generated as an output of the encoding pipeline 108 and received as an input by the decoding pipeline 110. Based on the bitstream 126 of the encoded PCFs, the decoding pipeline 110 may generate a set of decoded features for the current PCF 124A.

[0035] Before reconstructing the current PCF 124A, a set of reconstructed reference PCFs 128 may include PCFs that may have been reconstructed via the decoding pipeline 110. The PCFs in the set of reconstructed reference PCFs 128 may be generated based on the decoding of the bitstream of the encoded PCF. The bitstream of the encoded PCF may be generated based on the encoding of a set of reference PCFs 124B ... 124N. The set of reconstructed reference PCFs 128 may be used to reconstruct the current PCF 124A (i.e., the generation of the reconstructed PCF 130) based on the selection of the P frame decoding path of the decoding pipeline 110.

[0036] The reconstructed PCF 130 may be a frame of reconstructed 3D data corresponding to the current PCF 124A. The electronic device 102 may reconstruct the current PCF 124A based on the encoded PCF bitstream 126 via the neural network-based decoder 118. The reconstruction may be based on demodulation of decoded features associated with the current PCF 124A (which may be generated based on the encoded PCF bitstream 126).

[0037] In operation, the electronic device 102 may be configured to receive a PCF sequence 124, which may include a set of reference PCFs 124B...124N and a current PCF 124A to be encoded. According to an embodiment, each PCF in the set of reference PCFs 124B...124N may precede the current PCF 124A on the timeline of reception of the PCF sequence 124.

[0038] The electronic device 102 may also be configured to generate 3D feature data based on the application of the neural network-based encoder 112 to at least one of the current PCF 124A and the set of reference PCFs 124B ... 124N. According to an embodiment, the 3D feature data may include a first set of features associated with the current PCF 124A. The first set of features may be generated as an output of the neural network-based encoder 112 based on the application of the neural network-based encoder 112 to the current PCF 124A. The first set of features may be associated with the occupancy of 3D points in the current PCF 124A. In some embodiments, the first set of features may include features associated with the occupancy of a set of 3D points to be encoded in the current PCF 124A. The set of 3D points to be encoded may be received as user input. The 3D feature data may also include a second set of features associated with the set of reference PCFs 124B ... 124N. The second set of features may be generated as an output of the neural network-based encoder 112 based on the application of the neural network-based encoder 112 to each reference PCF in the set of reference PCFs 124B ... 124N. The second set of features may include reference features that may be associated with the occupancy of the 3D point in each reference PCF in the set of reference PCFs 124B...124N.

[0039] The electronic device 102 may also be configured to select a coding path as one of an I-frame coding path or a P-frame coding path for the current PCF 124A from the coding pipeline 108. The selected coding path may extend from the output of the neural network-based encoder 112 and may include a first sequence of transform operations that may be applicable to the generated 3D feature data. According to an embodiment, based on selecting the coding path as an I-frame coding path, the first sequence of transform operations may be applicable to a first set of features (generated as an output of the neural network-based encoder 112). On the other hand, based on selecting the coding path as a P-frame coding path, the first sequence of transform operations may be applicable to a first set of features and a second set of features (generated as an output of the neural network-based encoder 112).

[0040] According to an embodiment, based on selecting the coding path as the I-frame coding path, the first sequence of transform operations may include applying a gain vector from a set of gain vectors 116 to the first set of features to generate a first set of gain modulated features. The gain vector may be selected from the set of gain vectors 116. The selection may be based on an association of the gain vector with a coding rate and receipt of an input by the electronic device 102. For example, the input may indicate a selected coding rate that matches a coding rate associated with the gain vector.

[0041] According to an embodiment, based on selecting the encoding path as a P-frame encoding path, the first sequence of transform operations may include generating a first set of predicted features based on applying the first neural network-based predictor 114 to the second set of features. In some embodiments, the first set of predicted features may be associated with a set of 3D points for each reference PCF in the set of reference PCFs 124B...124N, which may correspond to a set of 3D points in the current PCF 124A to be encoded. The first set of predicted features may be generated as an output of the first neural network-based predictor 114.

[0042] The first sequence of transform operations may also include applying a gain vector in a set of gain vectors 116 to each of the first set of features and the first set of predicted features. Applying the gain vector to the first set of features may result in generating a first set of gain-modulated features, while applying the gain vector to the first set of predicted features may result in generating a second set of gain-modulated features. The first set of gain-modulated features may be generated based on the modulation of the first set of features using the gain vector. Similarly, the second set of gain-modulated features may be generated based on the modulation of the first set of predicted features using the gain vector. The gain vector may be selected from a set of gain vectors 116 based on the association of the gain vector with a coding rate that matches the selected coding rate (indicated in the input received by the electronic device 102). Thereafter, a differential operation may be applied to the first set of gain-modulated features and the second set of gain-modulated features to generate a set of residual features so that the current PCF 124A is encoded at the selected coding rate.

[0043] The electronic device 102 may also be configured to generate a bitstream 126 of the coded PCF based on the application of the first transform operation sequence to the 3D feature data. The application of the first transform operation sequence may be based on the selection of the I frame encoding path or the P frame encoding path. Therefore, the generation of the bitstream 126 of the coded PCF may be based on the first set of gain modulated features (if the I frame encoding path is selected) or a set of residual features (if the P frame encoding path is selected).

[0044] In the decoding stage, the electronic device 102 can be configured to apply a decoding operation to the bitstream 126 of the encoded PCF to generate a set of decoded features of the current PCF 124A. The generation of the set of decoded features can be based on whether an I frame encoding path or a P frame encoding path is selected for the generation of the bitstream 126 of the encoded PCF. If the P frame encoding path is selected, then the set of decoded features can correspond to a set of reconstructed residual features (reconstructed based on the set of residual features). On the other hand, if the I frame encoding path is selected, then the set of decoded features can correspond to a set of reconstructed gain modulated features (reconstructed based on the first set of gain modulated features).

[0045] The electronic device 102 may also be configured to select a decoding path as one of an I frame decoding path or a P frame decoding path from the decoding pipeline 110. The decoding path may include a second transform operation sequence applicable to a set of decoded features. Selecting the decoding path as an I frame decoding path may be based on selecting an I frame encoding path as an encoding path. On the other hand, selecting the decoding path as a P frame decoding path may be based on selecting a P frame encoding path as an encoding path. Therefore, the second transform operation sequence may be applicable to the reconstructed gain modulated features or a set of reconstructed residual features.

[0046] Based on the selection of the I frame decoding path, a second transform operation sequence can be applied to a set of reconstructed gain modulated features (i.e., a set of decoded features). The second transform operation sequence may include applying the selected inverse gain vector to a set of decoded features to generate a set of gain demodulated features. The selected inverse gain vector may correspond to the gain vector selected for modulation (included in the 3D feature data) of the first set of features. The inverse gain vector can be selected from a set of inverse gain vectors 122 based on the association of the inverse gain vector with the decoding rate and the reception of the input of the selected decoding rate that can indicate the decoding rate associated with the inverse gain vector. The input can be based on the coding rate associated with the gain vector corresponding to the selected inverse gain vector. A set of gain demodulated features can be generated based on the demodulation of the features of a set of reconstructed gain modulation, so that the current PCF 124A can be decoded at the selected decoding rate.

[0047] Based on the selection of the P-frame decoding path, the second sequence of transform operations may be applied to a set of reconstructed residual features (i.e., a set of decoded features). The second sequence of transform operations may include generating a third set of features based on the application of the neural network-based encoder 112 to a set of reconstructed reference PCFs 128. Thereafter, a second set of predicted features may be generated based on the application of the second neural network-based predictor 120 to the third set of features. In some embodiments, the third set of predicted features may be associated with a set of 3D points of each reconstructed reference PCF in a set of reference PCFs 124B...124N, which may correspond to a set of 3D points in the current PCF 124A that needs to be decoded. The second set of predicted features may be generated as an output of the second neural network-based predictor 120. The second sequence of transform operations may also include applying a gain vector in a set of gain vectors 116 to the third set of features. The application may result in the generation of a third set of gain modulated features. The third set of gain modulated features may be generated based on the modulation of the third set of features using the gain vector. The gain vector may be the same as the gain vector used to modulate the first set of features and the first set of predicted features.

[0048] The second transform operation sequence may also include: an addition operation may be applied to the features of the third group of gain modulation and a group of reconstructed residual features (i.e., a group of decoded features) to generate a fourth group of features. The second transform operation sequence may also include applying the selected inverse gain vector to the fourth group of features to generate a group of gain demodulated features. The selected inverse gain vector may correspond to the gain vector selected for modulating the first group of features (included in the 3D feature data) and the first group of predicted features, and the gain vector selected for modulating the third group of features. The inverse gain vector may be selected from a group of inverse gain vectors 122 based on its association with the decoding rate and the reception of the input indicating the selected decoding rate matched with the decoding rate associated with the inverse gain vector. The input may be based on the coding rate associated with the gain vector corresponding to the selected inverse gain vector. A group of gain demodulated features may be generated based on the demodulation of the fourth group of features so that the current PCF 124A may be decoded at the selected decoding rate.

[0049] The electronic device 102 may also be configured to reconstruct a current PCF 124A of the PCF sequence 124 based on the application of a set of decoded features (i.e., reconstructed gain modulated features or a set of reconstructed residual features) by a second transform operation sequence. The reconstruction of the current PCF 124A may correspond to the generation of the reconstructed PCF 130. The application of the second transform operation sequence may be based on the selection of an I-frame decoding path or a P-frame decoding path. Thus, if the I-frame encoding path is selected, a set of gain demodulated features generated based on a set of reconstructed gain modulated features may be used for the generation of the reconstructed PCF 130. On the other hand, if the P-frame encoding path is selected, a set of gain demodulated features generated based on a fourth set of features may be used for the generation of the reconstructed PCF 130.

[0050] According to an embodiment, a reconstructed PCF 130 corresponding to the current PCF 124A may be generated based on application of a set of gain demodulated features (generated based on selection of an I-frame decoding path or a P-frame decoding path) by the neural network-based decoder 118.

[0051] Figure 2 is a block diagram illustrating an exemplary electronic device including a rate adaptive codec for dynamic point cloud compression according to an embodiment of the present disclosure. Figure 2 is a combination of Figure 1 The elements of the explanation. Figure 2 , a block diagram 200 of an electronic device 102 is shown. The electronic device 102 may include a circuit system 202, a memory 204, an input / output (I / O) device 206, and a network interface 208. In at least one embodiment, the memory 204 may include an encoding pipeline 108 and a decoding pipeline 110. In at least one embodiment, the I / O device 206 may include a display device 210. The circuit system 202 may be communicatively coupled to the memory 204, the I / O device 206, and the network interface 208 through wired or wireless communication of the electronic device 102.

[0052] The circuit system 202 may include suitable logic, circuit systems and interfaces that may be configured to execute program instructions associated with different operations to be performed by the electronic device 102. The operations may include receiving a PCF sequence 124, which may include a set of reference PCFs 124B...124N and a current PCF 124A that may be required to be encoded; generating 3D feature data based on the application of the neural network-based encoder 112 to the current PCF 124A and at least one of the set of reference PCFs 124B...124N; selecting an encoding path (including a first transform operation sequence that may be applicable to the 3D feature data) as one of an I frame encoding path or a P frame encoding path for the current PCF 124A from the encoding pipeline 108; generating a bitstream 126 of the encoded PCF based on the application of the first transform operation sequence to the 3D feature data; applying a decoding operation to the bitstream 126 of the encoded PCF to generate a set of decoded features of the current PCF 124A; selecting the current PCF from the encoding pipeline 108 124A; selecting a decoding path as one of the I frame decoding path or the P frame decoding path from the decoding pipeline 110 (including a second transform operation sequence that can be applied to a set of decoded features); and reconstructing the current PCF 124A of the PCF sequence 124 based on the application of the second transform operation sequence to the set of decoded features (to generate the reconstructed PCF 130). The circuit system 202 may include one or more processing units, which may be implemented as an integrated processor or a cluster of processors that collectively perform the functions of one or more dedicated processing units. The circuit system 202 may be implemented based on a variety of processor technologies known in the art. Examples of implementations of the circuit system 202 may be an x86-based processor, a GPU, a CPU, a RISC processor, an ASIC processor, a CISC processor, a microcontroller, and / or other computing circuits.

[0053] The memory 204 may include suitable logic, circuitry, and / or interfaces that may be configured to store instructions executable by the circuitry 202. The memory 204 may be configured to store an operating system and associated applications. In at least one embodiment, the memory may store a set of gain vectors 116 and a set of inverse gain vectors 122. The memory 204 may also be configured to store a PCF sequence 124, a first set of features, a second set of features, a third set of features, a fourth set of features, a first set of predicted features, a second set of predicted features, a first set of gain modulated features, a second set of gain modulated features, a third set of gain modulated features, a set of gain demodulated features, a set of residual features, a bitstream 126 of the coded PCF, a set of reconstructed reference PCFs 128, a reconstructed PCF 130, and coordinate information associated with a set of 3D points in the current PCF 124A to be coded. In at least one embodiment, the neural network-based encoder 112, the first neural network-based predictor 114, the second neural network-based predictor 120, and the neural network-based decoder 118 included in the memory 204 may be implemented as a combination of programmable instructions stored in the memory 204 or logic units (i.e., programmable logic units) on the hardware circuit system of the electronic device 102. Examples of implementations of the memory 204 may include, but are not limited to, a random access memory (RAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a hard disk drive (HDD), a solid-state drive (SSD), a CPU cache, and / or a secure digital (SD) card.

[0054] The I / O device 206 may include suitable logic, circuit systems, interfaces and / or codes that may be configured to receive user input that may trigger receiving a PCF sequence 124, selecting an I frame encoding path, a P frame encoding path, an I frame decoding path, a P frame decoding path, an encoding rate (for selecting a gain vector in a set of gain vectors 116) or a decoding rate (for selecting an inverse gain vector in a set of inverse gain vectors 122). The I / O device 206 may be configured to provide an output in response to the user input. The I / O device 206 may include various input and output devices that may be configured to communicate with the circuit system 202. Examples of input devices may include, but are not limited to, a touch screen, a keyboard, a mouse, a joystick and / or a microphone. Examples of output devices may include a display device 210.

[0055] The display device 210 may include suitable logic, circuit systems, interfaces and / or codes that may be configured to render each PCF included in the PCF sequence 124 (i.e., the current PCF 124A and a set of reference PCFs 124B ... 124N) onto a display screen of the display device 210. The display device 210 may also be configured to render the reconstructed PCF 130 onto a display screen of the display device 210. According to an embodiment, the display device 210 may include a touch screen to receive user input. The display device 210 may be implemented by several known technologies, such as, but not limited to, a liquid crystal display (LCD) display, a light emitting diode (LED) display, a plasma display and / or an organic LED (OLED) display technology and / or other display technologies. According to an embodiment, the display device 210 may be a display screen of a smart glasses device, a 3D display, a perspective display, a projection-based display, an electrochromic display and / or a transparent display.

[0056] The network interface 208 may include suitable logic, circuitry, interfaces, and / or code that may be configured to establish communication between the electronic device 102 and the server 104 via the communication network 106. The network interface 208 may be implemented using various known techniques to support wired or wireless communication between the electronic device 102 and the communication network 106. The network interface 208 may include, but is not limited to, an antenna, a radio frequency (RF) transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a coder-decoder (CODEC) chipset, a subscriber identity module (SIM) card, and / or a local buffer.

[0057] The network interface 208 may communicate with a network such as the Internet, an intranet, and / or a wireless network such as a cellular telephone network, a wireless local area network (LAN), and / or a metropolitan area network (MAN) via wireless communications. The wireless communications may use any of a variety of communication standards, protocols, and technologies, such as Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (W-CDMA), Long Term Evolution (LTE), Fifth Generation (5G) New Radio (NR), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (such as IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, and / or IEEE 802.11n), Voice over Internet Protocol (VoIP), Light Fidelity (Li-Fi), Wi-MAX, email protocols, instant messaging, and / or short message service (SMS).

[0058] like Figure 1As described in , the functions or operations performed by the electronic device 102 may be performed by the circuit system 202. For example, in Figure 3A and Figure 3B , Figure 4 and Figure 5 The operations performed by circuit system 202 are described in detail in FIG.

[0059] Figure 3A and Figure 3B is a diagram illustrating an exemplary rate adaptive unified I-frame and P-frame codec architecture for dynamic point cloud compression and point cloud reconstruction according to an embodiment of the present disclosure.

[0060] Figure 3A and Figure 3B is a combination of Figure 1 and Figure 2 Reference Figure 3A and Figure 3B , an exemplary architecture 300 for dynamic point cloud compression and point cloud reconstruction is shown. In the architecture 300, the electronic device 102 is shown to include an encoding pipeline 108 (see Figure 3A ) and decoding pipeline 110 (see Figure 3B ).

[0061] The encoding pipeline 108 may include a P-frame encoding path 302 and an I-frame encoding path 304. The P-frame encoding path 302 may include a neural network-based encoder 112, a first neural network-based predictor 114, a set of gain vectors 116, a subtractor 306, a quantizer 308, and an autoencoder 310. The I-frame encoding path 304 may include a neural network-based encoder 112, a set of gain vectors 116, a quantizer 312, and an autoencoder 314. The encoding pipeline 108 may also include an octree-based encoder 316.

[0062] Decoding pipeline 110 (see Figure 3B ) may include a P-frame decoding path 318 and an I-frame decoding path 320. The P-frame decoding path 318 may include a neural network-based encoder 112, a second neural network-based predictor 120, a set of gain vectors 116, an auto decoder 322, an accumulator 324, a set of inverse gain vectors 122, and a neural network-based decoder 118. The I-frame decoding path 320 may include an auto decoder 326, a set of inverse gain vectors 122, and a neural network-based decoder 118. The decoding pipeline 110 may also include an octree-based decoder 328.

[0063] At any time, the circuit system 202 can be configured to receive a PCF sequence including a set of reference PCFs (e.g., P(t-1), P(t-2), ... and P(tN)) and a current PCF to be encoded (e.g., P(t)). The set of reference PCFs and the current PCF can be received as input by the neural network-based encoder 112. Each reference PCF in the set of reference PCFs can be located before the current PCF. The neural network-based encoder 112 can generate 3D feature data as output. The 3D feature data can include a first set of features (e.g., F(t)) that can be associated with 3D points of the current PCF (i.e., P(t)). The first set of features can be associated with the occupancy of the 3D points in the current PCF. According to an embodiment, the 3D points can be a set of 3D points of the current PCF that can be required to be encoded. The circuit system 202 can determine that the set of 3D points is to be encoded based on an input that can include coordinate information (e.g., C(t)) of a set of 3D points of the current PCF.

[0064] The 3D feature data may also include a second set of features associated with the set of reference PCFs (i.e., P(t-1), P(t-2), ... and P(tN)). The second set of features may include a feature group associated with the 3D points of each reference PCF in the set of reference PCFs. For example, the feature group associated with the 3D points of the reference PCF P(t-1) may be F(t-1). Similarly, the feature group associated with the 3D points of the reference PCF P(tN) may be F(tN). The feature group associated with each reference PCF may include reference features associated with the occupancy of the 3D points in the corresponding reference PCF. The 3D point may correspond to a set of 3D points of the current PCF. The second set of features may be represented as {F(t-1)...F(tN)}.

[0065] The circuit system 202 may also be configured to select an encoding path from the encoding pipeline 108 as the P frame encoding path 302 or the I frame encoding path 304 for encoding the current PCF. Each of the P frame encoding path 302 and the I frame encoding path 304 may include a first sequence of transform operations that may be applied to the 3D feature data. Based on the selection of the P frame encoding path 302, the first sequence of transform operations may include applying the first neural network-based predictor 114 to the second set of features. Based on the application, a first set of predicted features (e.g., F ～ (t)) as output.

[0066] To generate the first set of predicted features, the feature set associated with the 3D points of each reference PCF may be downsampled by at least one scaling factor. According to an embodiment, the feature set may be downsampled by scaling factors 2, 3, ..., and K. For example, F(t-1) may be downsampled by scaling factors 2, 3, ..., and K to generate multi-scale feature sets F, respectively. 2 (t-1), F 3 (t-1), ... and F K (t-1). Similarly, F(tN) can be downsampled to generate a multi-scale feature set F 2 (tN), F 3 (tN), ... and F K (tN).

[0067] For each scaling factor (i.e., 2, 3, ..., or K), a spatiotemporal tensor may be generated using a feature set associated with all reference PCFs in a set of reference PCFs downsampled by the same factor. For example, for scaling factor "2", a spatiotemporal tensor may be generated based on F 2 (t-1), … and F 2 (tN) generates a space-time tensor. Similarly, for the scaling factor “K”, it can be based on F K (t-1), … and F K (tN) Generate a spatiotemporal tensor. According to an embodiment, the generation of a spatiotemporal tensor for a certain scaling factor may be based on the spatiotemporal concatenation of feature groups associated with all reference PCFs in a set of reference PCFs downsampled by the same scaling factor. For example, the spatiotemporal tensor for scaling factor "2" may be based on F 2 (t-1), …, and F 2 Similarly, the space-time tensor for the scaling factor “K” can be constructed based on F K (t-1), …, and F K (tN) space-time cascade construction.

[0068] After generating (K-1) spatiotemporal tensors (for scaling factors 2, 3, ... and K), a spatiotemporal analysis may be performed on each spatiotemporal tensor. According to an embodiment, the spatiotemporal analysis may include performing a sparse convolution operation or a self-attention operation on each spatiotemporal tensor to generate a modified spatiotemporal tensor. The features represented by each modified spatiotemporal tensor may be downsampled by a certain scaling factor. The scaling factor may be based on the highest scaling factor (i.e., K) at which the feature group of each reference PCF in a set of reference PCFs is downsampled. Thereafter, an initial residual network (IRN) may be applied to each downsampled modified spatiotemporal tensor to generate a final spatiotemporal tensor. In this way, (K-1) final spatiotemporal tensors may be generated, and the (K-1) final spatiotemporal tensors may be cascaded to generate a multi-scale feature cascade vector, and a sparse convolution operation or a self-attention operation may be performed on the multi-scale feature cascade vector.

[0069] The first neural network-based predictor 114 may also receive coordinate information (i.e., C(t)) of a set of 3D points of the current PCF (i.e., P(t)) (to be encoded) for generating a first set of predicted features. The first neural network-based predictor 114 may generate the first set of predicted features (i.e., F) based on C(t) and the result of a sparse convolution operation or a self-attention operation on the multi-scale feature concatenation vector. ～ (t)).

[0070] The first sequence of transform operations may also include applying the selected gain vector to the first set of features (i.e., F(t)) to generate a first set of gain modulated features. According to an embodiment, the circuit system 202 may be configured to retrieve a set of gain vectors 116 from the memory 204. Each gain vector in the set of gain vectors 116 may correspond to a coding rate associated with the encoding pipeline 108. Once the set of gain vectors 116 is retrieved, the circuit system 202 may select a gain vector from the set of gain vectors 116. The selected gain vector may modulate the features in the first set of features so that the current PCF (i.e., P(t)) may be encoded at the coding rate corresponding to the selected gain vector.

[0071] According to an embodiment, the circuit system 202 may receive a first input, which may include a selection of a first coding rate for a current PCF (i.e., P(t)). The circuit system 202 may determine a gain vector corresponding to a coding rate that matches the first coding rate from a set of gain vectors 116. When determining such a gain vector from the set of gain vectors 116, the circuit system 202 may select a gain vector. Thus, the gain vector may be selected based on the first input. The first set of gain modulated features may be generated based on modulation of features in the first set of features (i.e., F(t)) by the selected gain vector.

[0072] The first sequence of transform operations may also include applying the selected gain vector to the first set of predicted features (ie, F ～ (t)) to generate a second set of gain modulated features. The selected gain vector can modulate the features in the first set of predicted features so that the current PCF (ie, P(t)) can be encoded at the corresponding coding rate (first coding rate) with the selected gain vector.

[0073] The first sequence of transform operations may also include applying a differential operation to the first set of gain modulated features and the second set of gain modulated features. Based on the application, a set of residual features (i.e., R(t)) may be generated. According to an embodiment, the subtractor 306 may receive the first set of gain modulated features and the second set of gain modulated features as inputs and apply a differential operation between the inputs to generate a set of residual features. Since the first set of gain modulated features and the second set of gain modulated features are based on F(t) and F(t), respectively, ～ (t) is obtained, so the set of residual features can indicate the difference between the actual feature set and the predicted feature set. This difference can be used to compensate for any errors that may have caused the prediction of the features associated with the current PCF (ie, P(t)).

[0074] The quantizer 308 may receive the set of residual features (i.e., R(t)) as input and quantize each of the residual features in the set of residual features to a quantization level in a set of predefined quantization levels. The quantizer 308 may generate a set of quantized residual features as output based on the application of a quantization scheme to the set of residual features. The quantization scheme may be based on an entropy model. The autoencoder 310 may receive the set of quantized residual features as input and generate a bitstream of the encoded PCF as output. The generation may be based on the application of a coding scheme to the set of quantized residual features. The coding scheme may be based on an entropy model. The coding scheme may compress the features in the set of quantized residual features to generate a bitstream of the encoded PCF.

[0075] After selecting the P frame encoding path 302, a bitstream of the encoded PCF can be generated based on the set of residual features (i.e., R(t)). The bitstream of the encoded PCF can be encoded at a coding rate corresponding to the selected gain vector. Therefore, the bitstream of the encoded PCF can be generated based on the selected gain vector. It can be noted that the features in the first set of features (i.e., F(t)) and the first set of predicted features (i.e., F ～ (t)) and a set of gain vectors 116 that can modulate F(t) and F ～(t) and encoding the bit stream of the encoded PCF at the other coding rate. The other gain vectors can be selected based on a first input that can indicate a specific value of the first coding rate, which specific value can match the coding rate that one of the other gain vectors can correspond to.

[0076] On the other hand, based on the selection of the I frame encoding path 304, the first sequence of transform operations may include applying the selected gain vector to the first set of features (i.e., F(t)) to generate a first set of gain modulated features. The quantizer 312 may receive the first set of gain modulated features as input and quantize each feature in the first set of gain modulated features to a quantization level in a set of predefined quantization levels. The quantizer 308 may generate a set of quantized gain modulated features as output based on the application of a quantization scheme to the first set of gain modulated features. The quantization scheme may be based on an entropy model. The autoencoder 314 may receive the set of quantized gain modulated features as input and generate a bitstream of the encoded PCF as output. The generation of the bitstream of the encoded PCF may be based on the application of a coding scheme to the set of quantized gain modulated features. The coding scheme may be based on an entropy model. The coding scheme may compress the features in the set of quantized gain modulated features to generate a bitstream of the encoded PCF.

[0077] After selecting the I-frame encoding path 304, a bitstream of the encoded PCF can be generated based on the features of the first set of gain modulations. The bitstream of the encoded PCF can be encoded at a coding rate corresponding to the selected gain vector. Therefore, the bitstream of the encoded PCF can be generated based on the selected gain vector. It can be noted that the bitstream of the encoded PCF can be encoded at other coding rates based on the modulation of the features in the first set of features (i.e., F(t)) and the selection of other gain vectors in the set of gain vectors 116 corresponding to other coding rates.

[0078] According to an embodiment, the octree-based encoder 316 may receive as input coordinate information (i.e., C(t)) of a set of 3D points of a current PCF (i.e., P(t)) to be encoded. The octree-based encoder 316 may encode the coordinate information to generate encoded coordinate information. The decoding pipeline 110 may receive a bitstream of the encoded PCF (generated based on a set of residual features (i.e., R(t)) or a first set of gain modulated features) and the encoded coordinate information. The octree-based decoder 328 of the decoding pipeline 110 may receive as input the encoded coordinate information and reconstruct the coordinate information (i.e., C(t)). Based on the reconstructed coordinate information (e.g., C ～ (t)), circuitry 202 may determine a set of 3D points for the current PCF (ie, P(t)) to be decoded.

[0079] According to an embodiment, the circuit system 202 can be configured to apply a decoding operation to the bitstream of the encoded PCF to generate a set of decoded features of the current PCF (i.e., P(t)). To apply the decoding operation, a decoding path can be selected from the decoding pipeline 110. The decoding path can be selected as the P frame decoding path 318 or the I frame decoding path 320. Based on selecting the P frame decoding path 318 as the decoding path, the automatic decoder 322 can receive as input the bitstream of the encoded PCF generated based on a set of residual features (i.e., R(t)). The automatic decoder 322 can apply the decoding operation to the bitstream of the encoded PCF to generate a set of decoded features of the current PCF. The automatic decoder 322 can generate a set of reconstructed residual features (e.g., R ～ (t)) as the set of decoded features. The reconstruction (i.e., the decoding operation) may be based on a decoding scheme. The decoding scheme may be based on an entropy model. The decoding scheme may decompress features in a set of quantized residual features to generate a set of decoded features (i.e., R ～ (t)).

[0080] On the other hand, based on selecting the I frame decoding path 320 as the decoding path, the automatic decoder 326 can receive as input the bitstream of the encoded PCF generated based on the first set of gain modulated features. The automatic decoder 326 can apply a decoding operation to the bitstream of the encoded PCF to generate a set of decoded features of the current PCF (i.e., P(t)). The decoding operation can be based on a decoding scheme. The decoding scheme can be based on an entropy model. The decoding scheme can decompress features in a set of quantized gain modulated features to generate a set of decoded features.

[0081] The selected decoding path may include a second sequence of transform operations that may be applicable to a set of decoded features. The current PCF of the PCF sequence (i.e., P(t)) may be reconstructed based on the application of the second sequence of transform operations to the set of decoded features. For the selected decoding path as the P-frame decoding path 318, the second sequence of transform operations may include receiving a PCF sequence including a set of reconstructed reference PCFs (e.g., P'(t-1), P'(t-2), ..., and P'(tN)). The reception may be based on the reconstruction of a set of reference PCFs (i.e., P(t-1), P(t-2), ..., and P(tN)) before the reconstruction of the current PCF (i.e., P(t)). The neural network-based encoder 112 may receive a set of reconstructed reference PCFs as input.

[0082] The second sequence of transform operations may also include generating a third set of features (e.g., {F'(t-1)...F'(tN)}) based on the application of the neural network-based encoder 112 to a set of reconstructed reference PCFs (i.e., P'(t-1), P'(t-2),..., and P'(tN)). The third set of features may include a set of features associated with the 3D points of each reconstructed reference PCF in the set of reconstructed reference PCFs. The set of features associated with the set of reconstructed reference PCFs may be generated as an output of the neural network-based encoder 112. For example, the set of features associated with the 3D points of the reconstructed reference PCF P'(t-1) may be F'(t-1). Similarly, the set of features associated with the 3D points of the reconstructed reference PCF P'(tN) may be F'(tN). The set of features associated with the 3D points of each reconstructed reference PCF may include features associated with the occupancy of the 3D points in the corresponding reconstructed reference PCF. The 3D points of each reconstructed reference PCF may correspond to a set of 3D points of the current PCF (ie, P(t)) to be decoded.

[0083] The second sequence of transformation operations may also include applying the second neural network-based predictor 120 to the third set of features (i.e., {F'(t-1) ... F'(tN)}) to generate a second set of predicted features (e.g., F'(t)). The second neural network-based predictor 120 may receive the third set of features (i.e., the set of features associated with each 3D point of the set of reconstructed reference PCFs) and the reconstructed coordinate information (i.e., C ～ (t)) as input. The second neural network-based predictor 120 may predict a second set of predicted features (i.e., F'(t)) as output. The generation of the second set of predicted features (based on the third set of features) may be similar to the generation of the first set of predicted features (i.e., F'(t)) by using the first neural network-based predictor 114. ～ (t)) (based on the second set of features, i.e., {F(t-1)…F(tN)}) are the same.

[0084] The second sequence of transform operations may also include applying the selected gain vector to the second set of predicted features (i.e., F'(t)) to generate a third set of gain modulated features. The selected gain vector may be a vector modulating the first set of features (i.e., F(t)) and the first set of predicted features (i.e., F ～ (t)) to generate the same gain vectors for the first set of gain modulated features and the second set of gain modulated features, respectively. The second sequence of transform operations may also include applying an addition operation to a third set of gain modulated features and a set of decoded features (i.e., a set of reconstructed residual features or R ～(t)) to generate a fourth set of features (e.g., F”(t)). The accumulator 324 may receive the third set of gain modulated features and a set of reconstructed residual features (i.e., R ～ (t)) as input. The accumulator 324 may generate a coordinate information whose coordinate information may be included in the reconstructed coordinate information (ie, C ～ A fourth set of features (i.e., F”(t)) associated with the 3D points in (t)) is taken as output.

[0085] The second sequence of transform operations may also include applying the selected inverse gain vector to a fourth set of features (i.e., F"(t)) to generate a set of gain-modulated features. According to an embodiment, the circuit system 202 may be configured to retrieve a set of inverse gain vectors 122 from the memory 204. Each inverse gain vector in the set of inverse gain vectors 122 may correspond to a decoding rate associated with the decoding pipeline 110. Once a set of inverse gain vectors 122 is retrieved, the circuit system 202 may select an inverse gain vector from the set of inverse gain vectors 122. The selected inverse gain vector may correspond to the selected gain vector (based on which the third set of gain-modulated features is generated). The selected inverse gain vector may demodulate features in the fourth set of features so that the current PCF (i.e., P(t)) may be reconstructed at a decoding rate corresponding to the selected inverse gain vector.

[0086] According to an embodiment, the circuit system 202 may receive a third input, which may include a selection of a first decoding rate for reconstructing the current PCF (i.e., P(t)). The selected first decoding rate may correspond to the first encoding rate. The circuit system 202 may determine an inverse gain vector corresponding to a decoding rate that matches the first decoding rate from a set of inverse gain vectors 122. When determining such an inverse gain vector from a set of inverse gain vectors 122, the circuit system 202 may select the inverse gain vector. Therefore, the inverse gain vector may be selected based on the third input. Therefore, a set of gain demodulated features may be generated based on the demodulation of features in a fourth set of features (i.e., F" (t)) by the selected inverse gain vector. The current PCF (of the PCF sequence) may be reconstructed at the first decoding rate) based on the selected inverse gain vector.

[0087] The second sequence of transform operations may also include applying a neural network-based decoder 118 to a set of gain-demodulated features to reconstruct the current PCF (ie, P(t)). The neural network-based decoder 118 may receive the set of gain-demodulated features as input and generate a reconstructed PCF (eg, P'(t)) corresponding to the current PCF as output.

[0088] On the other hand, for the selected decoding path as the I frame decoding path 320, the second transform operation sequence may include applying the selected inverse gain vector to a set of decoded features (generated based on the decoding of the first set of gain modulated features) to generate a set of gain demodulated features. The selected inverse gain vector can demodulate the features in the set of decoded features so that the current PCF (i.e., P(t)) can be reconstructed at the corresponding decoding rate with the selected inverse gain vector. Therefore, the current PCF of the PCF sequence can be reconstructed based on the selected inverse gain vector (at the first decoding rate). The second transform operation sequence may also include applying the neural network-based decoder 118 to a set of gain demodulated features to reconstruct the current PCF (i.e., P(t)). The neural network-based decoder 118 can receive a set of gain demodulated features as input and generate a reconstructed PCF (i.e., P'(t)) corresponding to the current PCF as output.

[0089] Figure 4 is a block diagram illustrating exemplary operations for generating a set of gain vectors and encoding a point cloud frame at different encoding rates based on the set of gain vectors according to an embodiment of the present disclosure. Figure 4 is a combination of Figure 1 , Figure 2 , Figure 3A and Figure 3B Reference Figure 4 , an exemplary scenario 400 is shown. An exemplary module is shown in the exemplary scenario 400, which may be a neural network-based encoder 112, a first neural network-based predictor 114, or a second neural network-based predictor 120. The exemplary module may receive an input (represented by X), which may be a PCF sequence (input to the neural network-based encoder 112), a second set of features (input to the first neural network-based predictor 114), or a third set of features (input to the second neural network-based predictor 120). A one-hot encoder 402 and a first neural network (e.g., a multi-layer perceptron 404) are also shown. The multi-layer perceptron 404 may be trained based on the input received from the one-hot encoder 402. The trained multi-layer perceptron 404 may generate a set of gain vectors 116. The set of gain vectors 116 may modulate the features generated by the neural network-based encoder 112, the first neural network-based predictor 114, or the second neural network-based predictor 120.

[0090] At any time instance, the circuit system 202 can retrieve a first set of bits per point (BPP) values ​​associated with a set of encoding rates for the encoding pipeline 108. The first set of BPP values ​​can be provided as input to the one-hot encoder 402. The first set of BPP values ​​can be received as input indicating the rate at which the current PCF can be required to be encoded. The one-hot encoder 402 can encode the first set of BPP values ​​based on a one-hot encoding scheme to generate a set of encoded BPP values. For each input (i.e., a BPP value in the first set of BPP values ​​represented as β), the one-hot encoder 402 can generate an encoded BPP value in a set of encoded BPP values. The encoded BPP value can be a one-hot vector. The circuit system 202 can feed each encoded BPP value in the set of encoded BPP values ​​to a first neural network, i.e., a multilayer perceptron 404. The set of encoded BPP values ​​(i.e., a set of one-hot vectors) can be fed as input to the first neural network to train the first neural network.

[0091] For each encoded BPP value (input) in the set of encoded BPP values, the first neural network (i.e., multilayer perceptron 404) may generate an output. Thus, a set of outputs may be generated for the set of encoded BPP values. Circuitry 202 may calculate the product (denoted as Y) of the generated output and the first set of features (the output of neural network-based encoder 112, denoted as Y), the first set of predicted features (the output of the first neural network-based predictor 114, denoted as Y), or the second set of predicted features (the output of the second neural network-based predictor 120, denoted as Y). β ). Thus, a set of products may be calculated for the set of outputs. The first neural network (i.e., multilayer perceptron 404) may be trained based on the calculated set of products. Once the training of the first neural network (i.e., multilayer perceptron 404) is complete, a set of gain vectors 116 may be generated. Thus, each gain vector in the set of gain vectors 116 may be based on the training of the first neural network (i.e., multilayer perceptron 404).

[0092] After generating the set of gain vectors 116, the first neural network (i.e., multilayer perceptron 404) may be discarded. Circuitry 202 may select gain vectors from the set of gain vectors 116 to modulate the first set of features (i.e., F(t)), the first set of predicted features (i.e., F ～ (t)) or a second set of predicted features (i.e., F'(t)). F(t) and F ～(t) for encoding the current PCF (i.e., P(t)) at a coding rate corresponding to the selected gain vector. On the other hand, the characteristics of F'(t) can be modulated (at a coding rate corresponding to the selected gain vector) for decoding P(t) at a decoding rate corresponding to the coding rate or generating a reconstructed PCF (i.e., P'(t)).

[0093] However, in some scenarios, it may be required to encode P(t) at a coding rate that does not match any coding rate corresponding to a gain vector in a set of gain vectors 116. Such a scenario may occur due to the reception of a second input. The second input may include a selection of a second coding rate for which there is no corresponding gain vector in the set of gain vectors 116. In such a scenario, the circuit system 202 may generate an intermediate gain vector based on interpolation or extrapolation using at least one gain vector in the set of gain vectors 116. For example, the set of gain vectors 116 may include g1 corresponding to 0.04BPP, g2 corresponding to 0.05BPP, g3 corresponding to 0.06BPP, g4 corresponding to 0.07BPP, and g5 corresponding to 0.08BPP. In the first scenario, the second input may include a selection of a second coding rate that is 0.055BPP. For this second coding rate, there is no corresponding gain vector in the set of gain vectors 116. The second coding rate does not match any coding rate corresponding to a gain vector in the set of gain vectors 116. Therefore, an intermediate gain vector may be generated based on interpolation using the two gain vectors g2 and g3. 23 ) can be generated as:

[0094]

[0095] In a second scenario, the second input may include a selection of a second coding rate as 0.035 BPP. For this second coding rate, an intermediate gain vector may be generated based on extrapolation using the gain vector g1. According to an embodiment, the intermediate gain vector (e.g., g 11 ) can be generated as: g 11 =g1×s, s∈(0.8,1). In a third scenario, the second input may include a selection of a second coding rate as 0.085 BPP. For this second coding rate, an intermediate gain vector may be generated based on extrapolation using the gain vector g5. According to an embodiment, the intermediate gain vector (e.g., g 55 ) can be generated as: g 55 =g5×s,s∈(1,1.15).

[0096] Circuitry 202 may also be configured to update set of gain vectors 116 to include the generated intermediate gain vectors (ie, g 23 , g 11 or 55 ). Thus, the set of gain vectors 116 can be extended to include additional gain vectors so that the current PCF can be encoded at coding rates falling within a wide range without requiring retraining of the neural network-based encoder 112, the first neural network-based predictor 114, or the second neural network-based predictor 120. ～ The selected gain vector for the feature in F(t) or F'(t) may be the generated intermediate gain vector.

[0097] Figure 5 is a block diagram illustrating exemplary operations for generating a set of inverse gain vectors and decoding a point cloud frame based on the set of inverse gain vectors according to an embodiment of the present disclosure. Figure 5 is a combination of Figure 1 , Figure 2 , Figure 3A , Figure 3B and Figure 4 Reference Figure 5 , an exemplary scenario 500 is shown. In the exemplary scenario 500, a neural network-based decoder 118 is shown. The neural network-based decoder 118 can generate a reconstructed PCF (represented by X') that can correspond to the current PCF. A one-hot encoder 502 and a second neural network (e.g., a multilayer perceptron 504) are also shown. The multilayer perceptron 504 can be trained based on the input received from the one-hot encoder 502. The trained multilayer perceptron 504 can generate a set of inverse gain vectors 122, which can be used to demodulate the fourth set of features (represented by Q(Y β )) or a set of decoded features (denoted by Q(Y β ) represents the features in ).

[0098] At any time instance, the circuit system 202 can retrieve a second set of BPP values ​​associated with a set of decoding rates for the decoding pipeline 110. The second set of BPP values ​​can be provided as input to the one-hot encoder 502. The second set of BPP values ​​can be received as an input indicating a decoding rate based on which the current PCF will be reconstructed. The one-hot encoder 502 can encode the second set of BPP values ​​based on a one-hot encoding scheme to generate a set of encoded BPP values ​​(i.e., a set of one-hot vectors). For each input (i.e., a BPP value in the second set of BPP values ​​represented as β), the one-hot encoder 502 can generate an encoded BPP value in a set of encoded BPP values ​​(i.e., a one-hot vector). The circuit system 202 can feed each encoded BPP value in a set of encoded BPP values ​​to a second neural network, i.e., a multilayer perceptron 504. A set of encoded BPP values ​​(i.e., a set of one-hot vectors) can be fed as input to the second neural network to train the second neural network. For each encoded BPP value (input) in the set of encoded BPP values, the second neural network (i.e., multilayer perceptron 504) can generate an output. Thus, a set of outputs can be generated for the set of encoded BPP values. The circuit system 202 can calculate the generated outputs and the fourth set of features (given by Q(Y β ) or a set of decoded features (denoted by Q(Y β ) represents the product of ). Therefore, a set of products can be calculated for the set of outputs. The second neural network (i.e., multilayer perceptron 504) can be trained based on the calculated set of products. Once the training of the second neural network (i.e., multilayer perceptron 504) is completed, a set of inverse gain vectors 122 can be generated. Therefore, each inverse gain vector in a set of inverse gain vectors 122 can be generated based on the training of the second neural network (i.e., multilayer perceptron 504).

[0099] After generating a set of inverse gain vectors 122, the second neural network (i.e., multilayer perceptron 504) can be discarded. Circuit system 202 can select an inverse gain vector in the set of inverse gain vectors 122 to demodulate features in a fourth set of features or features in a set of decoded features (which can be generated based on the decoding of features of the first set of gain modulations). Features in the fourth set of features or features in a set of decoded features can be demodulated at a decoding rate corresponding to the selected inverse gain vector to reconstruct the current PCF (i.e., P(t)). However, in some scenarios, it may be required to reconstruct (decode) P(t) at a decoding rate that does not match any decoding rate corresponding to the inverse gain vector in a set of inverse gain vectors 122. This can be due to the reception of a second input and the reception of a fourth input, the second input including a selection of a second coding rate (for which a corresponding gain vector does not exist in a set of gain vectors 116). The fourth input may include a selection of a second decoding rate, for which a corresponding inverse gain vector does not exist in a set of inverse gain vectors 122. In such a scenario, the circuit system 202 may generate the intermediate inverse gain vector based on extrapolation using the inverse gain vectors in the set of inverse gain vectors 122. For example, the set of inverse gain vectors 122 may include ig1 corresponding to 25 units, ig2 corresponding to 20 units, ig3 corresponding to 16.67 units, ig4 corresponding to 14.28 units, and ig5 corresponding to 12.5 units.

[0100] In the first scenario, the fourth input may include a selection of a second decoding rate of 30 units. For this second decoding rate, an intermediate inverse gain vector may be generated based on extrapolation using the inverse gain vector ig1. According to an embodiment, the intermediate inverse gain vector (eg, ig 11 ): In the second scenario, the fourth input may include a selection of a second decoding rate as 10 units. For this second decoding rate, an intermediate inverse gain vector may be generated based on extrapolation using the inverse gain vector ig5. According to an embodiment, the intermediate inverse gain vector (eg, ig 55 ):

[0101]

[0102] The circuit system 202 may also be configured to update the set of inverse gain vectors 122 to include the generated intermediate inverse gain vector (ie, ig 11 orig 55). Thus, the set of inverse gain vectors 122 can be expanded to include additional inverse gain vectors so that the current PCF can be decoded at decoding rates falling within a wide range without requiring retraining of the neural network-based decoder 118. The selected inverse gain vector that can be used to demodulate features in the fourth set of features or features in a set of decoded features can be a generated intermediate inverse gain vector.

[0103] Figure 6 is a flow chart illustrating the operation of an exemplary method for dynamic point cloud compression using a rate adaptive unified I-frame and P-frame codec architecture in accordance with an embodiment of the present disclosure. Figure 6 is a combination of Figure 1 , Figure 2 , Figure 3A , Figure 3B , Figure 4 and Figure 5 Reference Figure 6 , a flowchart 600 is shown. The operations from 602 to 610 may be implemented by any computing system, such as by the electronic device 102 or the circuit system 202 of the electronic device 102. The operations may start at 602 and may proceed to 604.

[0104] At 604, a PCF sequence may be received, which may include a set of reference PCFs and a current PCF to be encoded. In at least one embodiment, circuit system 202 may be configured to receive a PCF sequence that may include the set of reference PCFs and the current PCF to be encoded. Details of the reception of the set of reference PCFs and the current PCF may be provided, for example, in Figure 1 and Figure 3A Described in.

[0105] At 606, 3D feature data may be generated based on the application of the neural network-based encoder 112 to at least one of the current PCF and the set of reference PCFs. In at least one embodiment, the circuit system 202 may be configured to generate the 3D feature data based on the application of the neural network-based encoder 112 to at least one of the current PCF and the set of reference PCFs. The details of generating the 3D feature data are described in detail, for example, Figure 1 and Figure 3A Described in.

[0106] At 608, an encoding path may be selected from the encoding pipeline 108 as one of an I-frame encoding path or a P-frame encoding path for the current PCF. In at least one embodiment, the circuit system 202 may be configured to select an encoding path from the encoding pipeline 108 as one of an I-frame encoding path or a P-frame encoding path for the current PCF. The selected encoding path may extend from the output of the neural network-based encoder 112 and include a first sequence of transform operations that may be applicable to the 3D feature data. The details of the encoding selection are described, for example, in Figure 1 and Figure 3A Described in.

[0107] At 610, a bitstream of the encoded PCF may be generated based on the application of the first sequence of transform operations to the 3D feature data. In at least one embodiment, the circuit system 202 may be configured to generate a bitstream of the encoded PCF based on the application of the first sequence of transform operations to the 3D feature data. The details of the generation of the bitstream of the encoded PCF are described, for example, in Figure 1 , Figure 3A , Figure 3B , Figure 4 and Figure 5 Control can be passed to the end.

[0108] Although flowchart 600 is shown as discrete operations, such as 604, 606, 608, and 610, the present disclosure is not limited thereto. Thus, in certain embodiments, such discrete operations may be further divided into additional operations, combined into fewer operations, or eliminated, depending on the implementation, without departing from the essence of the disclosed embodiments.

[0109] Various embodiments of the present disclosure may provide a non-transitory computer-readable medium and / or storage medium, and / or a computer-readable recording medium having stored thereon information executable by a machine and / or a computer to operate an electronic device (e.g., Figure 1 Such instructions may cause the electronic device 102 to perform operations, including storing an encoding pipeline (such as Figure 1 The encoding pipeline 108 includes a neural network based encoder (such as Figure 1 A neural network based encoder 112) and a set of gain vectors (such as Figure 1 The operation may also include storing a decoding pipeline (such as decoding pipeline 110) including a neural network based decoder (such as Figure 1 A neural network-based decoder 118) and a set of inverse gain vectors (such as Figure 1The operation may also include receiving a PCF sequence including a set of reference PCFs and a current PCF to be encoded, and generating 3D feature data based on applying the neural network-based encoder 112 to the current PCF and at least one of the set of reference PCFs. The 3D feature data may include a first set of features associated with the current PCF and a second set of features associated with the set of reference PCFs. The operation may also include selecting a coding path from the coding pipeline 108 for the current PCF as one of an I-frame coding path or a P-frame coding path. The selected coding path may extend from the output of the neural network-based encoder 112 and may include a first sequence of transform operations that may be applicable to the 3D feature data. For the selected coding path as the I-frame coding path, the first sequence of transform operations may include applying the selected gain vector to the first set of features to generate a first set of gain modulated features. For the selected coding path as the P frame coding path, the first sequence of transform operations may include applying the first neural network-based predictor 114 to the second set of features to generate a first set of predicted features, applying the selected gain vector to the first set of features to generate a first set of gain modulated features, applying the selected gain vector to the first set of predicted features to generate a second set of gain modulated features, applying a differential operation to the first set of gain modulated features and the second set of gain modulated features to generate a set of residual features. The operation may also include generating a bitstream of the encoded PCF based on the application of the first sequence of transform operations to the 3D feature data. The bitstream of the encoded PCF may be generated based on the first set of gain modulated features or the set of residual features.

[0110] Exemplary aspects of the present disclosure may include electronic devices such as Figure 1 An electronic device 102), which may include a circuit system (such as Figure 2 The electronic device 102 may also include a memory (such as Figure 2 204), which may be configured to store the encoding pipeline (such as Figure 1 The encoding pipeline 108 includes a neural network based encoder (such as Figure 1 A neural network based encoder 112) and a set of gain vectors (such as Figure 1 The memory 204 may also be configured to store a decoding pipeline (such as the decoding pipeline 110) including a neural network based decoder (such as Figure 1 A neural network-based decoder 118) and a set of inverse gain vectors (such as Figure 1A set of inverse gain vectors 122 of the set of inverse gain vectors 122). The circuit system 202 can be configured to receive a PCF sequence including a set of reference PCFs and a current PCF to be encoded. The circuit system 202 can also be configured to generate 3D feature data based on the application of the neural network-based encoder 112 to the current PCF and at least one of the set of reference PCFs. The 3D feature data may include a first set of features associated with the current PCF and a second set of features associated with the set of reference PCFs. The circuit system 202 can also be configured to select a coding path as one of an I-frame coding path or a P-frame coding path for the current PCF from the coding pipeline 108. The selected coding path can extend from the output of the neural network-based encoder 112 and can include a first sequence of transform operations that can be applied to the 3D feature data. For the selected coding path as an I-frame coding path, the first sequence of transform operations can include applying the selected gain vector to the first set of features to generate a first set of gain-modulated features. For the selected encoding path as the P frame encoding path, the first transform operation sequence may include applying the first neural network-based predictor 114 to the second set of features to generate a first set of predicted features, applying the selected gain vector to the first set of features to generate a first set of gain modulated features, applying the selected gain vector to the first set of predicted features to generate a second set of gain modulated features, and applying a differential operation to the first set of gain modulated features and the second set of gain modulated features to generate a set of residual features. The circuit system 202 may also be configured to generate a bitstream of the encoded PCF based on applying the first transform operation sequence to the 3D feature data. The bitstream of the encoded PCF may be generated based on the first set of gain modulated features or the set of residual features.

[0111] According to an embodiment, the circuit system 202 may also be configured to retrieve the set of gain vectors 116 from the memory 204. Each gain vector in the set of gain vectors 116 may correspond to a coding rate associated with the encoding pipeline 108. The circuit system 202 may also be configured to select a gain vector from the set of gain vectors 116. The bitstream of the encoded PCF may be further generated based on the selected gain vector.

[0112] According to an embodiment, the circuit system 202 may also be configured to receive a first input, which may include a selection of a first coding rate for a current PCF.The gain vector may be selected based on the first input.

[0113] According to an embodiment, the circuit system 202 may also be configured to receive a second input, which may include a selection of a second coding rate for which there is no corresponding gain vector in the set of gain vectors 116. The circuit system 202 may also be configured to generate an intermediate gain vector based on interpolation or extrapolation using at least one gain vector in the set of gain vectors 116. The circuit system 202 may also be configured to update the set of gain vectors 116 to include the generated intermediate gain vector. The selected gain vector may be an intermediate gain vector.

[0114] According to an embodiment, the circuit system 202 may also be configured to retrieve a first set of BPP values ​​associated with a set of encoding rates for the encoding pipeline 108. The circuit system 202 may also be configured to encode the first set of BPP values ​​based on a one-hot encoding scheme to generate a set of encoded BPP values. The circuit system 202 may also be configured to feed each encoded BPP value in the set of encoded BPP values ​​to the first neural network. The circuit system 202 may also be configured to generate an output of the first neural network for each encoded BPP value in the set of encoded BPP values. The circuit system 202 may also be configured to calculate the product of the generated output and the first set of features. The circuit system 202 may also be configured to train the first neural network based on the calculated products. The circuit system 202 may also be configured to generate each gain vector in the set of gain vectors based on the training of the first neural network.

[0115] According to an embodiment, the circuit system 202 may also be configured to apply a decoding operation to the bitstream of the encoded PCF to generate a set of decoded features of the current PCF. The circuit system 202 may also be configured to select a decoding path as one of an I frame decoding path or a P frame decoding path from the decoding pipeline 110. The decoding path may include a second sequence of transform operations that may be applied to the set of decoded features. For the selected decoding path as the I frame decoding path, the second sequence of transform operations may include applying the selected inverse gain vector to the set of decoded features to generate a set of gain demodulated features, and applying the neural network-based decoder 118 to the set of gain demodulated features to reconstruct the current PCF. For the selected decoding path as the P-frame decoding path, the second sequence of transform operations may include receiving a PCF sequence that may include a set of reconstructed reference PCFs, generating a third set of features based on the application of the neural network-based encoder 112 to the set of reconstructed reference PCFs, applying the second neural network-based predictor 120 to the third set of features to generate a second set of predicted features, applying the selected gain vector to the second set of predicted features to generate a third set of gain modulated features, applying an addition operation to the third set of gain modulated features and the set of decoded features to generate a fourth set of features, applying the selected inverse gain vector to the fourth set of features to generate a set of gain demodulated features, and applying the neural network-based decoder 118 to the set of gain demodulated features to reconstruct a current PCF. The circuit system 202 may also be configured to reconstruct a current PCF of the PCF sequence based on the application of the second sequence of transform operations to the set of decoded features.

[0116] According to an embodiment, the circuit system 202 may also be configured to retrieve a set of gain vectors from the memory 204. Each gain vector in the set of gain vectors 116 may correspond to a coding rate associated with the encoding pipeline 108. The circuit system 202 may also be configured to select a gain vector from the set of gain vectors 116. The circuit system 202 may also be configured to retrieve a set of inverse gain vectors from the memory 204. Each inverse gain vector in the set of inverse gain vectors 122 may correspond to a decoding rate associated with the decoding pipeline 110. The circuit system 202 may also be configured to select an inverse gain vector from the set of inverse gain vectors 122. The selected inverse gain vector may correspond to the selected gain vector. The current PCF of the PCF sequence may be further reconstructed based on the selected inverse gain vector.

[0117] According to an embodiment, the circuit system 202 may be further configured to receive a third input comprising a selection of a first decoding rate for reconstructing the current PCF. The inverse gain vector may be selected based on the third input.

[0118] According to an embodiment, the circuit system 202 may also be configured to receive a fourth input, which may include a selection of a second decoding rate for which a corresponding inverse gain vector may not be present in the set of inverse gain vectors 122. The circuit system 202 may also be configured to generate an intermediate inverse gain vector based on an extrapolation using an inverse gain vector in the set of inverse gain vectors 122. The circuit system 202 may also be configured to update the set of inverse gain vectors 122 to include the generated intermediate inverse gain vector. The selected inverse gain vector may be an intermediate inverse gain vector.

[0119] According to an embodiment, the circuit system 202 may also be configured to retrieve a second set of BPP values ​​associated with a set of decoding rates for the decoding pipeline 110. The circuit system 202 may also be configured to encode the second set of BPP values ​​based on a one-hot encoding scheme to generate a set of encoded BPP values. The circuit system 202 may also be configured to feed each encoded BPP value in the set of encoded BPP values ​​to a second neural network. The circuit system 202 may also be configured to generate an output of the second neural network for each encoded BPP value in the set of encoded BPP values. The circuit system 202 may also be configured to calculate the product of the generated output and the set of decoded features. The circuit system 202 may also be configured to train the second neural network based on the calculated product. The circuit system 202 may also be configured to generate each inverse gain vector in the set of inverse gain vectors 122 based on the training of the second neural network.

[0120] The present disclosure may be implemented in hardware, or in a combination of hardware and software. The present disclosure may be implemented in at least one computer system in a centralized manner, or in a distributed manner in which different elements may be distributed among several interconnected computer systems. A computer system or other device suitable for performing the methods described herein may be suitable. The combination of hardware and software may be a general-purpose computer system with a computer program that, when loaded and executed, can control the computer system so that it performs the methods described herein. The present disclosure may be implemented in hardware that includes a portion of an integrated circuit that also performs other functions.

[0121] The present disclosure may also be embedded in a computer program product, which comprises all the features enabling the implementation of the methods described herein and which, when loaded in a computer system, is able to carry out these methods. In this context, a computer program refers to any expression of a set of instructions in any language, code or notation, which is intended to cause a system with information processing capabilities to perform certain functions, either directly or after either or both of the following: a) conversion into another language, code or notation; b) reproduction in a different material form.

[0122] Although the present disclosure is described with reference to certain embodiments, it will be appreciated by those skilled in the art that various changes may be made and equivalent forms may be substituted without departing from the scope of the present disclosure. In addition, many modifications may be made to adapt specific circumstances or materials to the teachings of the present disclosure without departing from its scope. Therefore, it is intended that the present disclosure is not limited to the disclosed embodiments, but that the present disclosure will include all embodiments falling within the scope of the appended claims.

Claims

1. An electronic device, comprising: A memory configured to store an encoding pipeline, the encoding pipeline comprising a neural network-based encoder and a set of gain vectors; as well as A circuit system is configured to: Receiving a point cloud frame (PCF) sequence including a set of reference point cloud frames (PCFs) and a current PCF to be encoded; generating 3D feature data based on application of a neural network-based encoder to at least one of a current PCF and the set of reference PCFs; Selecting a coding path from the coding pipeline for the current PCF as one of the I-frame coding path or the P-frame coding path, wherein the selected encoding path extends from an output of the neural network based encoder and comprises a first sequence of transform operations applied to the 3D feature data; and A bitstream of an encoded PCF is generated based on application of the first sequence of transform operations to the 3D feature data.

2. The electronic device according to claim 1, wherein the 3D feature data comprises: the first set of features associated with the current PCF, and A second set of features is associated with the set of reference PCFs.

3. The electronic device of claim 2, wherein the circuit system is further configured to: retrieving the set of gain vectors from memory, wherein each gain vector in the set of gain vectors corresponds to an encoding rate associated with an encoding pipeline, and A gain vector is selected from the set of gain vectors.

4. The electronic device of claim 3, wherein the circuit system is further configured to receive a first input, the first input comprising a selection of a first coding rate for a current PCF, wherein the gain vector is selected based on the first input.

5. The electronic device of claim 3, wherein the circuit system is further configured to: receiving a second input, the second input comprising a selection of a second coding rate, for which there is no corresponding gain vector in the set of gain vectors; generating an intermediate gain vector based on interpolation or extrapolation using at least one gain vector from the set of gain vectors; and The set of gain vectors is updated to include the generated intermediate gain vector, wherein the selected gain vector is the intermediate gain vector.

6. The electronic device of claim 3, wherein the circuit system is further configured to: retrieving a first set of bits per point (BPP) values ​​associated with a set of encoding rates for an encoding pipeline; encoding the first set of BPP values ​​based on a one-hot encoding scheme to generate a set of encoded BPP values; feeding each encoded BPP value in the set of encoded BPP values ​​to a first neural network; generating an output of the first neural network for each encoded BPP value in the set of encoded BPP values; Calculate the product of the generated output and the first set of features; training a first neural network based on the calculated product; as well as Each gain vector in the set of gain vectors is generated based on training of the first neural network.

7. The electronic device of claim 3, wherein a bit stream of the encoded PCF is further generated based on the selected gain vector.

8. The electronic device according to claim 3, wherein: For the selected coding path as the I-frame coding path, the first sequence of transform operations includes applying the selected gain vector to the first set of features to generate a first set of gain modulated features, The bit stream of the encoded PCF is generated based on the characteristics of the first group of gain modulation.

9. The electronic device according to claim 3, wherein: For the selected coding path as the P-frame coding path, the first sequence of transform operations includes: applying the first neural network-based predictor to the second set of features to generate a first set of predicted features, applying the selected gain vector to the first set of features to generate a first set of gain modulated features, applying the selected gain vector to the first set of predicted features to generate a second set of gain modulated features, and applying a difference operation to the first set of gain modulated features and the second set of gain modulated features to generate a set of residual features, The bit stream of the encoded PCF is generated based on the set of residual features.

10. The electronic device of claim 1, wherein the memory is further configured to store a decoding pipeline, the decoding pipeline comprising a neural network-based decoder and a set of inverse gain vectors.

11. The electronic device of claim 10, wherein the circuit system is further configured to: Applying a decoding operation to the bitstream of the encoded PCF to generate a set of decoded features of the current PCF; Selecting a decoding path from the decoding pipeline as one of an I frame decoding path or a P frame decoding path, wherein the decoding path comprises a second sequence of transform operations applicable to the set of decoded features; and A current PCF of the sequence of PCFs is reconstructed based on application of a second sequence of transform operations to the decoded set of features.

12. The electronic device of claim 11, wherein the circuit system is further configured to: retrieving the set of gain vectors from a memory, wherein each gain vector in the set of gain vectors corresponds to a coding rate associated with the coding pipeline, Selecting a gain vector from the set of gain vectors, Retrieving the set of inverse gain vectors from memory, wherein each inverse gain vector in the set of inverse gain vectors corresponds to a decoding rate associated with a decoding pipeline; and An inverse gain vector is selected from the set of inverse gain vectors, wherein the selected inverse gain vector corresponds to the selected gain vector.

13. The electronic device of claim 12, wherein the circuit system is further configured to receive a third input, the third input comprising a selection of a first decoding rate for reconstructing the current PCF, wherein the inverse gain vector is selected based on the third input.

14. The electronic device of claim 12, wherein the circuit system is further configured to: receiving a fourth input, the fourth input comprising a selection of a second decoding rate, for which there is no corresponding inverse gain vector in the set of inverse gain vectors; generating an intermediate inverse gain vector based on extrapolation using an inverse gain vector from the set of inverse gain vectors; and The set of inverse gain vectors is updated to include the generated intermediate inverse gain vector, wherein the selected inverse gain vector is the intermediate inverse gain vector.

15. The electronic device of claim 12, wherein the circuit system is further configured to: retrieving a second set of bits per point (BPP) values ​​associated with a set of decoding rates for a decoding pipeline; encoding the second set of BPP values ​​based on a one-hot encoding scheme to generate a set of encoded BPP values; feeding each encoded BPP value in the set of encoded BPP values ​​to a second neural network; generating an output of a second neural network for each encoded BPP value in the set of encoded BPP values; Calculate the product of the generated output and the set of decoded features; training a second neural network based on the calculated product; as well as Each inverse gain vector in the set of inverse gain vectors is generated based on training of a second neural network.

16. The electronic device of claim 12, wherein a current PCF of the PCF sequence is further reconstructed based on the selected inverse gain vector.

17. The electronic device according to claim 12, wherein: For the selected decoding path as the I-frame decoding path, the second sequence of transform operations includes: applying the selected inverse gain vector to the set of decoded features to generate a set of gain demodulated features, and A neural network based decoder is applied to the set of gain demodulated features to reconstruct the current PCF.

18. The electronic device according to claim 12, wherein: For the selected decoding path as the P-frame decoding path, the second sequence of transform operations includes: receiving a PCF sequence including a set of reconstructed reference PCFs, A third set of features is generated based on the application of a neural network-based encoder to the set of reconstructed reference PCFs, applying a second neural network-based predictor to the third set of features to generate a second set of predicted features, applying the selected gain vector to the second set of predicted features to generate a third set of gain modulated features, applying an addition operation to the third set of gain modulated features and the set of decoded features to generate a fourth set of features, applying the selected inverse gain vector to the fourth set of features to generate a set of gain-demodulated features, and A neural network based decoder is applied to the set of gain demodulated features to reconstruct the current PCF.

19. A method comprising: In an electronic device including a memory: storing an encoding pipeline in a memory, the encoding pipeline comprising a neural network-based encoder and a set of gain vectors; as well as Receiving a point cloud frame (PCF) sequence including a set of reference point cloud frames (PCFs) and a current PCF to be encoded; generating 3D feature data based on application of a neural network-based encoder to at least one of a current PCF and the set of reference PCFs; Selecting a coding path from the coding pipeline for the current PCF as one of the I-frame coding path or the P-frame coding path, The selected encoding path extends from the output of the neural network-based encoder and includes a first sequence of transform operations applied to the 3D feature data; and a bitstream of the encoded PCF is generated based on the application of the first sequence of transform operations to the 3D feature data.

20. The method of claim 18, further comprising: retrieving the set of gain vectors from a memory, wherein each gain vector in the set of gain vectors corresponds to an encoding rate associated with an encoding pipeline; as well as Selecting a gain vector from the set of gain vectors, The bit stream of the encoded PCF is further generated based on the selected gain vector.

Images