Multiscale Interpretation for Dynamic Point Cloud Compression

Abstract

An electronic device and method for multi-scale inter-prediction for dynamic point cloud compression are provided. The electronic device receives a set of reference point cloud frames and a current point cloud frame. The electronic device generates reference frame data including a feature set for each reference point cloud frame and a first feature set for the current point cloud frame. The electronic device predicts a second feature set for the current point cloud frame based on the reference frame data using a first neural network predictor. The electronic device calculates a residual feature set based on the first feature set and the second feature set. The electronic device generates a quantized residual feature set based on the residual feature set, and generates a bitstream of encoded point cloud data for the current 3D point cloud frame based on the quantized residual feature set.

Description

Technical Field

[0001] 〔Cross - Reference to Related Applications / Incorporation by Reference〕 This application claims priority to U.S. Patent Application No. 18 / 471,753, filed on September 21, 2023, which claims priority to U.S. Provisional Patent Application Serial No. 63 / 380,089, filed on October 19, 2022, and the contents of these documents are hereby incorporated by reference in their entirety.

Background Art

[0002] Advances in the field of Dynamic Point Cloud Compression (PCC) have led to the development of techniques that enable efficient representation of data associated with the 3D points of a point cloud. Typically, a point cloud contains a large number of unstructured 3D points. Each 3D point can include geometric information and attribute information (e.g., color, transparency, reflectivity, opacity, texture, and material, etc.) associated with the corresponding 3D point. Thus, each 3D point of a point cloud can contain a significant amount of data. Point cloud data may require compression (i.e., encoding) using a PCC encoder for storage, processing, or transmission of the point cloud. Subsequently, a PCC decoder can reconstruct the point cloud based on the encoded point cloud data received from the PCC encoder. The PCC encoder can generate encoded point cloud data based on the current point cloud frame and previously decoded point cloud frames in order to encode the current point cloud frame. The encoded point cloud data can be transmitted to a PCC decoder that can reconstruct the corresponding points by decoding the encoded point cloud data. Reconstruction of the current point cloud frame based on the encoded point cloud data (generated based on previously decoded point cloud frames) is prone to errors, and artifacts or irregularities may be seen in the geometry of the reconstructed point cloud.

Summary of the Invention

[0003] [[ID=+]] Those skilled in the art will be able to see the limitations and disadvantages of conventional methods by comparing the described system with some aspects of the disclosure shown with reference to the drawings in the remainder of this application.

[0004] The present invention provides an electronic apparatus and method for multiscale interpretation for dynamic point cloud compression, as illustrated and / or described in substantially relation to at least one figure and more fully provided in the claims.

[0005] These and other features and advantages of the disclosure can be understood by considering the following detailed description of the disclosure with reference to the accompanying drawings, which indicate the same elements throughout by the same reference numerals. [Brief explanation of the drawing]

[0006] [Figure 1] This figure shows an exemplary network environment for multiscale interpretation for dynamic point cloud compression according to embodiments of the present disclosure. [Figure 2] This is a block diagram illustrating an exemplary first electronic device for multiscale interpretation for dynamic 3D point cloud frame compression according to an embodiment of the present disclosure. [Figure 3] This is a block diagram illustrating an exemplary second electronic device for multiscale interpretation for 3D point cloud frame reconstruction according to embodiments of the present disclosure. [Figure 4] This figure shows an exemplary architecture for multiscale interpretation for dynamic 3D point cloud compression and 3D point cloud reconstruction according to embodiments of the present disclosure. [Figure 5] This block diagram shows an exemplary operation according to an embodiment of the present disclosure, in which a feature set associated with the current 3D point cloud frame is predicted based on multiscale features associated with a reference 3D point cloud frame set. [Figure 6]This block diagram shows an exemplary operation according to an embodiment of the present disclosure, in which a feature set associated with the current 3D point cloud frame is predicted based on multiscale features associated with a reference 3D point cloud frame set. [Figure 7A] This figure shows an exemplary scenario for encoding or decoding a 3D point cloud frame based on a prior 3D point cloud frame according to embodiments of the present disclosure. [Figure 7B] This figure shows an exemplary scenario for encoding or decoding a 3D point cloud frame based on two preceding 3D point cloud frames, according to embodiments of the present disclosure. [Figure 8] This figure shows an exemplary scenario for encoding / decoding a 3D point cloud frame based on a preceding 3D point cloud frame and a subsequent 3D point cloud frame, according to embodiments of the present disclosure. [Figure 9] This flowchart shows the operation of an exemplary method for multiscale interpretation for dynamic 3D point cloud frame compression according to embodiments of the present disclosure. [Figure 10] This flowchart shows the operation of an exemplary method for multiscale interpretation for 3D point cloud frame reconstruction according to embodiments of the present disclosure. [Modes for carrying out the invention]

[0007] Embodiments described below can be found in the first electronic device, the second electronic device, and the method for multiscale interpretation for dynamic point cloud compression disclosed. Exemplary aspects of the disclosure provide a first electronic device (e.g., a computer device, a game console, or virtual reality goggles) that predicts features associated with a 3D point cloud frame based on multiscale features of a reference 3D point cloud frame and determines the difference between the actual features associated with the 3D point cloud frame and the predicted features. Specifically, the first electronic device can receive a three-dimensional (3D) point cloud sequence that includes a reference 3D point cloud frameset and the current 3D point cloud frame to be encoded. After receiving this sequence, the first electronic device can generate reference frame data that includes a set of features associated with each reference 3D point cloud frame in the reference 3D point cloud frameset. The first electronic device can further generate current frame data associated with the 3D points of the current 3D point cloud frame. The current frame data may include a first set of features (i.e., an actual set of features) associated with the occupancy rate of 3D points in the current 3D point cloud frame. The first electronic device can predict a second set of features associated with the 3D points of the current 3D point cloud frame. The prediction of the second set of features can be based on the application of the first neural network predictor to reference frame data. The first electronic device can compute a set of residual features based on the generated first set of features and the predicted second set of features. Subsequently, the first electronic device can generate a quantized residual feature set based on the application of a quantization scheme to the residual feature set. Finally, the first electronic device can generate a bitstream of encoded point cloud data for the current 3D point cloud frame based on the application of an encoding scheme to the quantized residual feature set.

[0008] An exemplary aspect of the present disclosure further provides a second electronic device (e.g., a computer device, a game console, virtual reality goggles, or a smart wearable device) that predicts features associated with a current 3D point cloud frame based on multiscale features of a reference 3D point cloud frameset. The current 3D point cloud frame can be reconstructed based on the predicted features and encoded point cloud data. Specifically, the second electronic device can receive a 3D point cloud sequence containing a reference 3D point cloud frameset. After receiving the sequence, the second electronic device can generate reference frame data containing a set of features associated with the 3D points of each reference 3D point cloud frame in the reference 3D point cloud frameset. The second electronic device can then receive (from the first electronic device) a bitstream of encoded point cloud data associated with the current 3D point cloud frame to be decoded. The second electronic device can predict a third set of features associated with the 3D points of the current 3D point cloud frame based on the application of a second neural network predictor to the reference frame data. The second electronic device can generate a fourth set of features related to the 3D points of the current 3D point cloud frame based on the received bitstream of encoded point cloud data and the predicted third set of features. The second electronic device can generate (i.e., reconstruct) the current 3D point cloud frame based on the application of a decoding scheme to the determined fourth set of features.

[0009] Typically, 3D point clouds can be compressed using point cloud compression (PCC) techniques and then reconstructed using decoding techniques. Encoding is necessary because each 3D point in a 3D point cloud can contain a significant amount of point data, making it impossible to transmit point cloud data over communication channels with limited bandwidth. Encoding can involve predicting the features of the current point cloud frame based on the features of a previously decoded point cloud frame. Based on the predicted features, a bitstream of encoded point cloud data can be generated. A PCC decoder can then reconstruct the current point cloud frame based on this bitstream. However, if there is a large difference between the current point cloud frame and a previous point cloud frame, the reconstructed point cloud frame may contain artifacts or surface irregularities. To prevent the appearance of artifacts or irregularities, the difference between the current point cloud frame and a previously decoded point cloud frame can be determined. This difference can be used to estimate the motion of objects in the current point cloud frame relative to the previously decoded point cloud frame. Based on the motion estimation, another bitstream of encoded point cloud data can be generated. Since multiple bitstreams are generated, lossless reconstruction or decoding of each point cloud frame may require additional bandwidth resources to send the multiple bitstreams of encoded point cloud data to the PCC decoder.

[0010] To address these issues, the first electronic device can perform dynamic point cloud compression based on predictions of latent features associated with the current point cloud frame using multi-scale latent features associated with a previously decoded reference point cloud frameset that can precede the current point cloud frame. Furthermore, the second electronic device can determine the multi-scale latent features associated with the reference point cloud frameset and reconstruct the current point cloud frame using the determined multi-scale latent features. The determination of multi-scale features by the first electronic device can facilitate a reduction in the size of the bitstream representing the encoded point cloud data associated with the current point cloud frame, thereby reducing transmission throughput. This reduction can be achieved because the PCC encoder on the first electronic device side can be used to determine the actual features associated with each point cloud frame, and the residuals generated based on the actual and predicted features can be used to generate the encoded point cloud data. The residuals can represent the difference between the actual and predicted features. The first electronic device can further compress the residuals using an entropy encoder and transmit the compressed residuals as encoded point cloud data to the second electronic device. On the second electronic device side, a PCC encoder can be used to determine the multiscale latent features associated with the reference point cloud frameset. Based on the determined multiscale latent features associated with the reference point cloud frameset, the latent features associated with the current point cloud frame can be predicted. Subsequently, on the second electronic device side, the received residuals can be reconstructed, and the reconstructed residuals can be accumulated together with the predicted features to determine the actual features associated with the current point cloud frame. On the second electronic device side, a PCC decoder can be used to reconstruct the current point cloud frame based on the determined actual features.

[0011] Figure 1 shows an exemplary network environment for multiscale interprediction for dynamic point cloud compression according to an embodiment of the present disclosure. Figure 1 shows the network environment 100. The network environment 100 includes a first electronic device 102, a second electronic device 104, and a server 106. The first electronic device 102 can communicate with the second electronic device 104 and the server 106 via one or more networks (such as a communication network 108). The first electronic device 102 may include a first point cloud compression (PCC) encoder 110, a first neural network predictor 112, and an octree-based encoder 114. The second electronic device 104 may include a second PCC encoder 116, an octree-based decoder 118, a second neural network predictor 120, and a PCC decoder 122. The first electronic device 102 can receive a reference 3D point cloud frameset 124 and the current 3D point cloud frame 126 as inputs and generate encoded point cloud data 128 associated with the current 3D point cloud frame 126 as an output. The second electronic device 104 can receive the reference 3D point cloud frameset 124 and the encoded point cloud data 128 as inputs and generate a decoded 3D point cloud frame 130 as an output.

[0012] In Figure 1, the first electronic device 102 is responsible for encoding point cloud data (e.g., point cloud sequences), and the second electronic device 104 is responsible for decoding and reconstructing point cloud data from the compressed point cloud representation shared by the first electronic device 102.

[0013] In some embodiments, the first electronic device 102 and the second electronic device 104 may be the same device. In such cases, either the first electronic device 102 or the second electronic device 104 may be responsible for both encoding and reconstructing the point cloud data. The first PCC encoder 110 may be the same as the second PCC encoder 116, and the first neural network predictor 112 may be the same as the second neural network predictor 120.

[0014] The first electronic device 102 may include preferred logic, circuitry, interfaces, and / or code that can be configured to receive a reference 3D point cloud frameset 124, the current 3D point cloud frame 126, and target coordinate information associated with the current 3D point cloud frame 126. The first electronic device 102 may further determine features associated with each reference 3D point cloud frame of the reference 3D point cloud frameset 124 and features associated with the current 3D point cloud frame 126 via a first PCC encoder 110. The first electronic device 102 may further predict features associated with the current 3D point cloud frame 126 based on the features associated with each reference 3D point cloud frame of the reference 3D point cloud frameset 124 via a first neural network predictor 112. The first electronic device 102 may further be configured to determine residuals based on the features determined by the first PCC encoder 110 and the features predicted by the first neural network predictor 112, and to transmit the residuals. Examples of the first electronic device 102 include, but are not limited to, servers, video conferencing systems, augmented reality (AR) devices, virtual reality (VR) devices, mixed reality (MR) devices, game consoles, servers, smart wearable devices, mainframe machines, computer workstations, and / or consumer electronic (CE) devices.

[0015] The second electronic device 104 may include suitable logic, circuitry, interfaces, and / or code that can be configured to receive a reference 3D point cloud frameset 124, encoded target coordinate information, and compressed residuals. The second electronic device 104 can determine the features associated with each reference 3D point cloud frame of the reference 3D point cloud frameset 124 via a second PCC encoder 116 and predict the features associated with the current 3D point cloud frame 126 via a second neural network predictor 120. Furthermore, the second electronic device 104 can reconstruct (i.e., decode) the current 3D point cloud frame 126 via a PCC decoder 122 to obtain a decoded 3D point cloud frame 130. The reconstruction may be based on the features and residuals predicted by the second neural network predictor 120. Examples of the second electronic device 104 include, but are not limited to, computer devices, video conferencing systems, AR devices, VR devices, MR devices, game consoles, smart wearable devices, servers, mainframe machines, computer workstations, and / or CE devices.

[0016] Server 106 may include preferred logic, circuitry, interfaces, and / or code that can be configured to generate a reference 3D point cloud frameset 124 of 3D objects in 3D space. Server 106 may be configured to generate each 3D reference point cloud frame of the reference 3D point cloud frameset 124 using image and depth information of the objects. Server 106 may be configured to store the reference 3D point cloud frameset 124 and information associated with the reference 3D point cloud frameset 124. Server 106 may be further configured to receive requests from a first electronic device 102 or a second electronic device 104 for the reference 3D point cloud frameset 124. Based on the request, Server 106 may transmit the reference 3D point cloud frameset 124 to the first electronic device 102 or the second electronic device 104. In some embodiments, the server 106 may include a first PCC encoder 110, a first neural network predictor 112, an octree-based encoder 114, a second PCC encoder 116, an octree-based decoder 118, a second neural network predictor 120, and a PCC decoder 122.

[0017] Server 106 can perform operations via web applications, cloud applications, HTTP requests, repository operations, and file transfers. Examples of implementations of Server 106 include, but are not limited to, database servers, file servers, web servers, application servers, mainframe servers, cloud computing servers, or combinations thereof. In at least one embodiment, Server 106 can be implemented as multiple distributed cloud-based resources by using several technologies well known to those skilled in the art. Those skilled in the art will understand that the scope of this disclosure is not limited to implementing Server 106 and the first electronic device 102 as two separate entities, implementing Server 106 and the second electronic device 104 as two separate entities, or implementing Server 106, the first electronic device 102, and the second electronic device 104 as three separate entities. In some embodiments, the functionality of Server 106 can be incorporated, in whole or at least partially, into the first electronic device 102 or the second electronic device 104 without departing from the scope of this disclosure.

[0018] The communication network 108 may include a communication medium that enables the first electronic device 102, the second electronic device 104, and the server 106 to communicate with each other. The communication network 108 may be a wired or wireless communication network. Examples of the communication network 108 include, but are not limited to, the Internet, Wireless Fidelity (Wi-Fi) networks, Personal Area Networks (PANs), Local Area Networks (LANs), or Metropolitan Area Networks (MANs). The first electronic device 102 and the second electronic device 104 may be configured to connect to the communication network 108 according to various wired and wireless communication protocols. Examples of such wired and wireless communication protocols include, but are not limited to, at least one of the following: Transmission Control Protocol and Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), ZigBee, EDGE, IEEE 802.11, Light Fidelity (Li-Fi), 802.16, IEEE 802.11s, IEEE 802.11g, Multihop Communication, Wireless Access Point (AP), Device-to-Device Communication, Cellular Communication Protocol, and Bluetooth (BT) Communication Protocol.

[0019] Each of the first PCC encoder 110 and the second PCC encoder 116 may include suitable logic, circuitry, interfaces and / or code that can be configured to encode each reference 3D point cloud frame of the reference 3D point cloud frameset 124 to generate a feature set associated with the 3D points of the corresponding reference 3D point cloud frame of the reference 3D point cloud frameset 124. The first PCC encoder 110 may be further configured to encode the current 3D point cloud frame 126 to generate a first feature set associated with the 3D points of the current 3D point cloud frame 126.

[0020] Each of the first PCC encoder 110 and the second PCC encoder 116 can be implemented as a deep neural network (including model files and associated inference code) that can run on a graphics processing unit (GPU), central processing unit (CPU), tensor processing unit (TPU), reduced instruction set computing (RISC) processor, application-specific integrated circuit (ASIC) processor, or composite instruction set computing (CISC) processor, coprocessor, and / or a combination thereof. In some embodiments, the first PCC encoder 110 can be implemented as a deep neural network on dedicated hardware that works in conjunction with other computing circuits of the first electronic device 102. Similarly, the second PCC encoder 116 can be implemented as a deep neural network on dedicated hardware that works in conjunction with other computing circuits of the second electronic device 104. In such implementations, the first PCC encoder 110 and the second PCC encoder 116 can be associated with a specific form factor on a specific computing circuit. Examples of specific computing circuits include, but are not limited to, field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), ASICs, programmable ASICs (PL-ASICs), application-specific integrated circuits (ASSPs), and systems-on-chip (SOCs) based on standard microprocessors (MPUs) or digital signal processors (DSPs). In one embodiment, the first PCC encoder 110 or the second PCC encoder 116 can be coupled with a GPU in order to parallelize the operation of the first PCC encoder 110 or the second PCC encoder 116, respectively.

[0021] Each of the first neural network predictor 112 and the second neural network predictor 120 can be called a neural network, which is typically a system of computational networks or artificial neurons that can be arranged in multiple layers. A neural network can be defined by hyperparameters such as (single or multiplicative) activation functions, the number of weights, the cost function, the regularization function, the input size, and the number of layers. Furthermore, these layers may include an input layer, one or more hidden layers, and an output layer. Each of the multiple layers may include one or more nodes (or artificial neurons). The output of all nodes in the input layer can be coupled to at least one node in the (single or multiplicative) hidden layers. Similarly, the input of each hidden layer can be coupled to the output of at least one node in the other layers of the neural network. The output of each hidden layer can be coupled to the input of at least one node in the other layers of the neural network. The (single or multiplicative) nodes in the final layer can receive input from at least one hidden layer and output a result. The number of layers and the number of nodes in each layer can be determined from the hyperparameters of the neural network. These hyperparameters can be set before or after training the neural network.

[0022] Each node can correspond to a mathematical function (e.g., a sigmoid function or a rectified linear unit) with parameters that can be adjusted during training of the neural network. The parameter set may include, for example, weight parameters and regularization parameters. Each node can compute an output using a mathematical function based on one or more inputs from nodes in other (one or multiple) layers of the neural network (e.g., previous (one or multiple) layers). All or some of the nodes in the neural network can correspond to the same or different mathematical functions. During training of the neural network, one or more parameters of each node in the neural network can be updated based on whether the output of the final layer for a given input (from the training dataset) matches the correct result according to the neural network's loss function. The above process can be repeated for the same or different inputs until the minimum value of the loss function is achieved and the training error is minimized. Several training methods are known in this field, including gradient descent, stochastic gradient descent, batch gradient descent, gradient boosting, and metaheuristic methods.

[0023] Each of the first neural network predictor 112 and the second neural network predictor 120 can be a machine learning model trained to generate multiscale features associated with an input 3D point cloud frame. Each of the first neural network predictor 112 and the second neural network predictor 120 can take as input a feature set associated with a 3D point in each reference 3D point cloud frame of a reference 3D point cloud frame set 124, and target coordinate information associated with the current 3D point cloud frame 126. Each of the first neural network predictor 112 and the second neural network predictor 120 can generate a prediction as an output in response to the input. The prediction can indicate features associated with the current 3D point cloud frame 126. The first neural network predictor 112 can predict a second feature set associated with the current 3D point cloud frame 126 based on the application of the first neural network predictor 112 to the feature set (generated by the first PCC encoder 110) and target coordinate information. The second neural network predictor 120 can predict a third feature set associated with the current 3D point cloud frame 126 based on its application to the feature set (generated by the second PCC encoder 116) and the target coordinate information (decoded by the octvine-based decoder 118).

[0024] In some embodiments, each of the first neural network predictor 112 and the second neural network predictor 120 may include electronic data that can be implemented as a software component of an application executable on the first electronic device 102 and the second electronic device 104. Each of the first neural network predictor 112 and the second neural network predictor 120 may rely on libraries, external scripts, or logic / instructions for execution by the processing units included in the first electronic device 102 and the second electronic device 104. In one or more embodiments, each of the first neural network predictor 112 and the second neural network predictor 120 may be implemented using hardware that includes a processor, a microprocessor (e.g., one or more operations that perform or control the execution of), an FPGA, or an ASIC. Alternatively, in some embodiments, each of the first neural network predictor 112 and the second neural network predictor 120 may be implemented using a combination of hardware and software. Examples of the first neural network predictor 112 and the second neural network predictor 120 include, but are not limited to, deep neural networks (DNNs), convolutional neural networks (CNNs), artificial neural networks (ANNs), fully connected neural networks, and / or combinations of such networks.

[0025] The octree-based encoder 114 may include preferred logic, circuitry, interfaces, and / or code that can be configured to encode data based on the use of a data structure (i.e., an octree). The octree-based encoder 114 can recursively partition 3D space (such as a portion of the current 3D point cloud frame 126 containing a 3D point located at a target coordinate specified in the target coordinate information) into smaller regions (such as blocks) known as octants. This partitioning can continue until each block satisfies a stopping criterion (the points within each block represent a similar density or texture). In one embodiment, the octant-based encoder 114 can encode partitioning hierarchy information and point cloud data associated with a 3D point located at a target coordinate to generate encoded target coordinate information.

[0026] In some embodiments, the octree-based encoder 114 may be a machine learning-based octree encoder capable of encoding (compressing) 3D points in the target coordinates of the current 3D point cloud frame 126 using techniques such as deep octree coding. Examples of such machine learning-based octree encoders include, but are not limited to, G-PCC, OctSqueeze, and VoxelContext-Net.

[0027] The octree-based decoder 118 may include suitable logic, circuitry, interfaces, and / or code that can be configured to decode the encoded target coordinate information. At each level of the octree (from which the encoded target coordinate information can be decoded), the octants can be reconstructed until the entire volume (i.e., the portion of the current 3D point cloud frame 126 containing the 3D points located at the target coordinates) is reconstructed. Based on the octants, the reconstructed decoded target coordinate information can be obtained.

[0028] In some embodiments, the octree-based encoder 114 may be a machine learning-based octree encoder capable of decoding coded target coordinate information associated with the current 3D point cloud frame 126 using deep octree decoding techniques. Examples of such machine learning-based octree decoders include, but are not limited to, G-PCC, OctSqueeze, and VoxelContext-Net.

[0029] The PCC decoder 122 may include suitable logic, circuitry, and / or interfaces that can be configured to reconstruct the current 3D point cloud frame 126 based on encoded point cloud data 128. The PCC decoder 122 may accept a fourth feature set as input. The fourth feature set may be generated based on a third feature set and decompressed residuals. Based on the application of the PCC decoder 122 to the fourth feature set, a decoded 3D point cloud frame 130 can be generated. The PCC decoder 122 may be implemented as a deep neural network on a GPU, CPU, TPU, RISC processor, ASIC processor, CISC processor, coprocessor, and / or a combination thereof. In some other embodiments, the PCC decoder 122 may be implemented as a deep neural network on dedicated hardware that works in conjunction with other computing circuits of the second electronic device 104. In such implementations, the PCC decoder 122 may be associated with a specific form factor on a particular computing circuit. Examples of specific computing circuits include, but are not limited to, FPGAs, PLDs, ASICs, PL-ASICs, ASSPs, and SOCs based on standard MPUs or DSPs. In some embodiments, the PCC decoder 122 can be linked with a GPU to parallelize its operation.

[0030] Each of the reference 3D point cloud frames of the reference 3D point cloud frameset 124 and the current 3D point cloud frame 126 can correspond to a geometric representation of one or more 3D objects in a 3D environment (e.g., a real-world environment). Each 3D point cloud frame can constitute a set of 3D points positioned at different locations according to a 3D coordinate system. According to one embodiment, a first electronic device 102 can obtain the reference 3D point cloud frameset 124 and the current 3D point cloud frame 126 from the server 106. Similarly, a second electronic device 104 can also obtain the reference 3D point cloud frameset 124 from the server 106. Each 3D point in the reference 3D point cloud frame can include geometric information (i.e., the coordinates of the corresponding 3D point in the corresponding reference 3D point cloud frame) and attribute information associated with the corresponding 3D point. The attribute information can include, for example, color information, reflectivity information, opacity information, normal vector information, material identifier information, or texture information.

[0031] The encoded point cloud data 128 can be generated based on the encoding of each reference 3D point cloud frame of the reference 3D point cloud frameset 124 and the current 3D point cloud frame 126. The encoded point cloud data 128 can also be generated based on the prediction of multiscale features associated with the current 3D point cloud frame 126 and the encoding of target coordinate information associated with the current 3D point cloud frame 126. The encoded point cloud data 128 may include compressed residuals (generated based on the generated first feature set and the predicted second feature set) and encoded target coordinate information. The first electronic device 102 can transmit the encoded point cloud data 128 as a bitstream to the second electronic device 104.

[0032] The decoded 3D point cloud frame 130 can be a reconstructed point cloud frame that corresponds to the current 3D point cloud frame 126. The second electronic device 104 can reconstruct the current 3D point cloud frame 126 based on the encoded point cloud data 128 via the PCC decoder 122. Reconstruction can be based on decompressing the compressed residuals and accumulating the decompressed residuals (i.e., the original generated residuals) with a predicted third feature set (i.e., a fourth feature set). The decoded 3D point cloud frame 130 can be generated based on the application of the PCC decoder 122 to this accumulation.

[0033] During operation, the first electronic device 102 can be configured to receive a 3D point cloud sequence that includes a reference 3D point cloud frameset 124 and the current 3D point cloud frame 126 to be encoded. According to one embodiment, each reference 3D point cloud frame in the reference 3D point cloud frameset 124 may be a previously decoded 3D point cloud frame and may precede (i.e., be received earlier than) or follow the current 3D point cloud frame 126 in the timeline for receiving the 3D point cloud sequence.

[0034] The first electronic device 102 can be further configured to generate reference frame data, which includes a feature set associated with the 3D points of each reference 3D point cloud frame in the reference 3D point cloud frameset 124. The generation of reference frame data can be performed based on the application of the first PCC encoder 110 to each reference 3D point cloud frame in the reference 3D point cloud frameset 124. The reference frame data can be generated as the output of the first PCC encoder 110. The feature set associated with the 3D points of each reference 3D point cloud frame may include reference features related to the occupancy rate of 3D points in the corresponding reference 3D point cloud frame of the reference 3D point cloud frameset 124, and reference coordinate information associated with the 3D points of the corresponding reference 3D point cloud frame. The reference coordinate information may include, for example, the coordinates of the 3D points in the corresponding reference 3D point cloud frame.

[0035] The first electronic device 102 can be further configured to generate current frame data relating to the 3D points of the current 3D point cloud frame 126. The generation of current frame data can be based on the application of the first PCC encoder 110 to the current 3D point cloud frame 126. The generated current frame data may include a first feature set relating to the occupancy rate of the 3D points of the current 3D point cloud frame 126. The first feature set can be generated as the output of the first PCC encoder 110. The first feature set may be called the actual features relating to the 3D points of the current 3D point cloud frame 126. According to one embodiment, the first feature set may include features relating to the occupancy rate of 3D points having coordinates included in target coordinate information.

[0036] The first electronic device 102 can be further configured to predict a second feature set associated with the 3D points of the current 3D point cloud frame 126, based on the application of the first neural network predictor 112 to reference frame data. Each feature set associated with a 3D point in each reference 3D point cloud frame of the reference 3D point cloud frame set 124 can be supplied as input to the first neural network predictor 112. Each feature set can be associated with a 3D point in the corresponding reference 3D point cloud frame having coordinates included in the reference coordinate information. The feature sets may include reference features related to the occupancy rate of 3D points in the corresponding reference 3D point cloud frame.

[0037] According to one embodiment, the first electronic device 102 can provide coordinate information (i.e., target coordinate information) related to 3D points in the current 3D point cloud frame 126 as input to the first neural network predictor 112. The 3D points can be 3D points for which features need to be predicted, and the target coordinate information can include the coordinates of these 3D points. The target coordinate information can include target coordinates contained in the current 3D point cloud frame 126. Based on the application of the first neural network predictor 112 to the input, a second set of features can be predicted as the output of the first neural network predictor 112. The second set of features can include features related to the occupancy rate of 3D points having coordinates contained in the target coordinate information.

[0038] According to one embodiment, the feature set associated with each reference 3D point cloud frame can be downsampled by scales of 2, 3, ..., K for prediction. For example, the feature set associated with the first reference 3D point cloud frame of reference 3D point cloud frameset 124 can be downsampled by scales of 2, 3, ..., K. Thus, (K-1) feature sets can be generated for the first reference 3D point cloud frame. Similarly, (K-1) feature sets can be generated for each of the other reference 3D point cloud frames of reference 3D point cloud frameset 124. Then, a spatiotemporal tensor can be constructed for each downsampling scale based on the feature sets of all reference 3D point cloud frames downsampled by the same scale. Since the feature set associated with each reference 3D point cloud frame is downsampled by scales of 2, 3, ..., K, (K-1) spatiotemporal tensors can be constructed. For example, for a downsampling scale of "2", a first spatiotemporal sensor can be constructed using the feature sets associated with all reference 3D point cloud frames downsampled by scale 2. Similarly, for a downsampling scale "K", a (K-1) spatiotemporal sensor can be constructed using the feature sets associated with all 124 reference 3D point cloud frames downsampled by the "K" scale. The construction of a spatiotemporal sensor for a corresponding downsampling scale can be performed based on the spatiotemporal concatenation of the feature sets of all reference 3D point cloud frames that can be downsampled by the corresponding downsampling scale. Subsequently, spatiotemporal tensor analysis can be performed based on the application of sparse convolution or self-attention operations to each of the (K-1) spatiotemporal tensors.After spatiotemporal tensor analysis, each of the downsampled (K-1) spatiotemporal tensors can be further downsampled (separately from the spatiotemporal tensor constructed based on the spatiotemporal concatenation of the feature sets downsampled by a scale of K). Subsequently, the (K-1) downsampled spatiotemporal tensors can be concatenated to generate a multi-scale feature concatenation vector.

[0039] Based on the target coordinate information and multiscale feature linkage vector associated with the 3D points in the current 3D point cloud frame 126, a second feature set associated with the 3D points in the current 3D point cloud frame 126 can be predicted as the output of the first neural network predictor 112. The second feature set may include features associated with 3D points having coordinates included in the target coordinate information.

[0040] The first electronic device 102 can be further configured to calculate a residual feature set based on a first feature set (i.e., current frame data relating to the 3D points of the current 3D point cloud frame 126) and a predicted second feature set. According to one embodiment, the first electronic device 102 can calculate the difference between the first feature set and the predicted second feature set. The calculated difference can represent the error relating the second feature set (i.e., features relating to the 3D points of the current 3D point cloud frame 126 predicted based on reference frame data) to the first feature set (i.e., actual features relating to the 3D points of the current 3D point cloud frame 126). This difference can also correspond to a residual feature set relating to the 3D points of the current 3D point cloud frame 126 having coordinates included in the target coordinate information.

[0041] The first electronic device 102 can be further configured to generate a quantized residual feature set based on the application of a quantization scheme to the residual feature set. The quantization scheme can be based on an entropy model and may include a quantization level set. According to one embodiment, the value of each residual feature in the residual feature set can be quantized to a certain quantization level of the quantization level set. The residual feature set can then be quantized for subsequent compression and encoding of the residual feature set.

[0042] The first electronic device 102 can be further configured to generate a bitstream of encoded point cloud data 128 for the current 3D point cloud frame 126, based on the application of an encoding scheme to the generated quantization residual feature set. The encoding scheme can be based on an entropy model (on which the quantization scheme can be based). Based on the encoding scheme, each quantization residual feature of the quantization residual feature set can be compressed for the generation of encoded point cloud data 128. The bitstream of encoded point cloud data 128 can constitute a compressed quantization residual feature set.

[0043] According to one embodiment, the encoded point cloud data 128 may further include encoded target coordinate information. The target coordinate information can be encoded based on the application of an octvine-based encoder 114 to the target coordinate information. The first electronic device 102 can transmit a bitstream of the encoded point cloud data 128 to the second electronic device 104.

[0044] According to one embodiment, the second electronic device 104 can be configured to receive (from the first electronic device 102) encoded point cloud data 128 and a 3D point cloud sequence that includes a reference 3D point cloud frameset 124. The second electronic device 104 can be configured to extract a compressed quantized residual feature set and encoded target coordinate information from the encoded point cloud data 128 (i.e., the bitstream generated by the first electronic device 102). The second electronic device 104 can then apply a decoding scheme to the compressed quantized residual feature set for decompression. The decoding scheme can generate a residual feature set that corresponds to the residual feature set computed (by the first electronic device 102) based on the first feature set and the predicted second feature set. The second electronic device 104 can apply an octree-based decoder 118 to the encoded target coordinate information to generate target coordinate information as the output of the octree-based decoder 118.

[0045] The second electronic device 104 can be further configured to generate reference frame data that can include a feature set associated with the 3D points of each reference 3D point cloud frame in the reference 3D point cloud frameset 124. The feature set associated with each reference 3D point cloud frame can be generated based on the application of the second PCC encoder 116 to the corresponding reference 3D point cloud frame. The feature set can be generated as the output of the second PCC encoder 116. The features associated with each reference 3D point cloud frame can include reference features related to the occupancy rate of the 3D points in the corresponding reference 3D point cloud frame, and reference coordinate information (including the coordinates of the 3D points in the corresponding reference 3D point cloud frame).

[0046] The second electronic device 104 can be further configured to predict a third feature set associated with the 3D points of the current 3D point cloud frame 126, based on the application of the second neural network predictor 120 to reference frame data. The third feature set can be associated with 3D points having coordinates contained in target coordinate information decoded by the octree-based decoder 118. According to one embodiment, the second neural network predictor 120 can take as input the feature set associated with the 3D points of each reference 3D point cloud frame of the reference 3D point cloud frameset 124 (i.e., the output of the second PCC encoder 116) and target coordinate information (i.e., the output of the octree-based decoder 118). Based on the application of the second neural network predictor 120 to the inputs, the third feature set can be predicted as the output of the second neural network predictor 120. If the first neural network predictor 112 and the second neural network predictor 120 are identical, then the prediction of the third feature set (by the second neural network predictor 120) can be identical to the prediction of the second feature set (by the first neural network predictor 112). Furthermore, if the first PCC encoder 110 and the second PCC encoder 116 are identical, and the decoded target coordinate information (generated by the octree-based decoder 118) matches the target coordinate information encoded by the octree-based encoder 114, then the second feature set and the third feature set can be identical.

[0047] The second electronic device 104 can be further configured to generate a fourth feature set relating to the 3D points of the current 3D point cloud frame 126 (having coordinates included in the target coordinate information) based on the bitstream of the received encoded point cloud data 128 (i.e., the residual feature set that can be generated by the decoding scheme) and the predicted third feature set. According to one embodiment, the residual feature set and the predicted third feature set can be accumulated to generate the fourth feature set. The 3D points of the current 3D point cloud frame 126 can be reconstructed based on the application of the PCC decoder 122 to the fourth feature set. The PCC decoder 122 can generate a decoded 3D point cloud frame 130 as an output. The decoded 3D point cloud frame 130 can be a reconstructed version of the current 3D point cloud frame 126.

[0048] Figure 2 is a block diagram illustrating an exemplary first electronic device for multiscale interpretation for dynamic 3D point cloud frame compression according to an embodiment of the present disclosure. The description of Figure 2 will be made in relation to the elements of Figure 1. Figure 2 shows a block diagram 200 of the first electronic device 102. The first electronic device 102 may include a circuit 202, a memory 204, an input / output (I / O) device 206, and a network interface 208. In at least one embodiment, the memory 204 may include a first PCC encoder 110, a first neural network predictor 112, and an octree-based encoder 114. In at least one embodiment, the I / O device 206 may include a display device 210. The circuit 202 may be communicably coupled to the memory 204, the I / O device 206, and the network interface 208 via wired or wireless communication of the first electronic device 102.

[0049] Circuit 202 may include suitable logic, circuits, and interfaces that can be configured to execute program instructions related to different operations performed by the first electronic device 102. Circuit 202 may include one or more processing units, which may be implemented as an integrated processor or processor group that collectively performs the functions of one or more dedicated processing units. Circuit 202 may be implemented based on several processor technologies well known in the art. Examples of implementations of Circuit 202 may be x86-based processors, GPUs, CPUs, RISC processors, ASIC processors, CISC processors, microcontrollers, and / or other computing circuits.

[0050] Memory 204 may include preferred logic, circuits and / or interfaces that can be configured to store instructions executable by circuit 202. Memory 204 may be configured to store an operating system and associated applications. Memory 204 may be further configured to store a 3D point cloud sequence, generated reference frame data, generated current frame data, coordinate information related to 3D points in the current 3D point cloud frame 126, a second feature set, a residual feature set, a quantized residual feature set, and a bitstream of encoded point cloud data 128, etc. In at least one embodiment, the first PCC encoder 110, the first neural network predictor 112, and the octree-based encoder 114 contained in memory 204 may be implemented as a combination of programmable instructions stored in memory 204 or logic units (i.e., programmable logic units) on the hardware circuits of the first electronic device 102. Examples of memory 204 implementations include, but are not limited to, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), hard disk drives (HDDs), solid-state drives (SSDs), CPU caches, and / or secure digital (SD) cards.

[0051] The I / O device 206 may include suitable logic, circuitry, interfaces, and / or code that can be configured to receive user input that can trigger the reception of a 3D point cloud sequence or the transmission of a bitstream of encoded point cloud data 128. The I / O device 206 may be further configured to provide outputs in response to user input. The I / O device 206 may include various input and output devices that can be configured to communicate with circuitry 202. Examples of input devices include, but are not limited to, a touch screen, keyboard, mouse, joystick, and / or microphone. An example of an output device is a display device 210.

[0052] The display device 210 may include suitable logic, circuitry, interfaces, and / or code that can be configured to render each reference 3D point cloud frame of a reference 3D point cloud frameset included in a 3D point cloud sequence onto the display screen of the display device 210. According to one embodiment, the display device 210 may include a touch screen for receiving user input. The display device 210 can be implemented through a number of known technologies, including, but not limited to, liquid crystal displays (LCDs), light-emitting diode (LEDs), plasma displays, and / or organic LED (OLED) display technologies, and / or other display technologies. According to one embodiment, the display device 210 may mean a display screen for smart glasses, a 3D display, a see-through display, a projected display, an electrochromic display, and / or a transparent display.

[0053] The network interface 208 may include suitable logic, circuitry, interfaces, and / or code that can be configured to establish communication between the first electronic device 102, the second electronic device 104, and the server 106 via the communication network 108. The network interface 208 may be implemented to support wired or wireless communication between the first electronic device 102 and the communication network 108 using various known techniques. The network interface 208 may include, but is not limited to, an antenna, a radio frequency (RF) transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a coder-decoder (CODEC) chipset, a subscriber identification module (SIM) card, and / or a local buffer.

[0054] The network interface 208 can communicate wirelessly with networks such as the Internet, intranets, and / or wireless networks such as cellular networks, wireless local area networks (LANs), and / or metropolitan area networks (MANs). Wireless communication can use any of several communication standards, protocols, and technologies, including Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (W-CDMA), Long-Term Evolution (LTE), 5th Generation (5G) New Radio (NR), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wireless Fidelity (WiFi) (such as IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, and / or IEEE 802.11n), Voice over Internet Protocol (VoIP), Light Fidelity (Li-Fi), Wi-MAX, and protocols for email, instant messaging, and / or short message services.

[0055] The functions or operations performed by the first electronic device 102, as described in Figure 1, can be performed by the circuit 202. The operations performed by the circuit 202 will be described in detail, for example, in Figures 3, 4, 5, 6, 7A, 7B, and 8.

[0056] Figure 3 is a block diagram illustrating an exemplary second electronic device for multiscale interpretation for 3D point cloud frame reconstruction according to an embodiment of the present disclosure. The description of Figure 3 will be made in relation to the elements of Figure 1. Figure 3 shows a block diagram 300 of the second electronic device 104. The second electronic device 104 may include a circuit 302, a memory 204, an I / O device 306, and a network interface 308. In one embodiment, the memory 304 may include a second PCC encoder 116, an octree-based decoder 118, a second neural network predictor 120, and a PCC decoder 122. In at least one embodiment, the I / O device 306 may include a display device 310. The circuit 302 may be communicably coupled to the memory 304, the I / O device 306, and the network interface 308 via wired or wireless communication of the second electronic device 104.

[0057] Circuit 302 may include suitable logic, circuits, and interfaces that can be configured to execute program instructions related to different operations performed by the second electronic device 104. Circuit 302 may include one or more processing units, which may be implemented as an integrated processor or processor group that collectively performs the functions of one or more dedicated processing units. Circuit 302 may be implemented based on several processor technologies well known in the art. Examples of implementations of Circuit 302 may be x86-based processors, GPUs, RISC processors, ASIC processors, CISC processors, microcontrollers, CPUs, and / or other computing circuits.

[0058] Memory 304 may include preferred logic, circuits, and / or interfaces that can be configured to store instructions executable by circuit 302. Memory 304 may be configured to store an operating system and associated applications. Memory 304 may be further configured to store a 3D point cloud sequence, reference frame data, decoded coordinate information related to the 3D points of the current 3D point cloud frame to be decoded, a predicted third feature set, a decompressed residual feature set, and a 3D decoded point cloud frame 130. In at least one embodiment, the second PCC encoder 116, octree-based decoder 118, second neural network predictor 120, and PCC decoder 122 included in memory 304 are implemented as programmable instructions stored in memory 304 or as a combination of logic units (i.e., programmable logic units) on the hardware circuit of the second electronic device 104. Examples of implementations of memory 304 include, but are not limited to, RAM, ROM, EEPROM, HDD, SSD, CPU cache, and / or SD card.

[0059] The I / O device 306 may include suitable logic, circuitry, interfaces, and / or code that can be configured to receive user input that can trigger the reception of a bitstream of a 3D point cloud sequence and encoded point cloud data 128. The I / O device 306 may be further configured to provide an output in response to the user input. The I / O device 306 may include various input and output devices that can be configured to communicate with circuitry 302. Examples of input devices include, but are not limited to, a touch screen, keyboard, mouse, joystick, and / or microphone. An example of an output device is a display device 310.

[0060] The display device 310 may include suitable logic, circuitry, interfaces, and / or code that can be configured to render each reference 3D point cloud frame and the decoded 3D point cloud frame 130 of the reference 3D point cloud frameset onto the display screen of the display device 310. According to one embodiment, the display device 310 may include a touch screen for receiving user input. The display device 310 can be implemented through a number of known technologies, including, but not limited to, LCD displays, LED displays, plasma displays, and / or OLED display technologies, and / or other display technologies. According to one embodiment, the display device 310 may mean a display screen for smart glasses, a 3D display, a see-through display, a projected display, an electrochromic display, and / or a transparent display.

[0061] The network interface 308 may include suitable logic, circuitry, interfaces, and / or code that can be configured to establish communication between the first electronic device 102, the second electronic device 104, and the server 106 via the communication network 108. The network interface 308 may be implemented to support wired or wireless communication between the second electronic device 104 and the communication network 108 using various known techniques. The network interface 308 may include, but is not limited to, an antenna, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a CODEC chipset, a SIM card, and / or a local buffer.

[0062] The network interface 308 can communicate wirelessly with networks such as the Internet, intranets, and / or wireless networks such as cellular networks, Wi-Fi, and / or MAN. Wireless communication can use any of several communication standards, protocols, and technologies, including GSM, EDGE, W-CDMA, LTE, 5G NR, CDMA, TDMA, Bluetooth, Wi-Fi (such as IEEE802.11a, IEEE802.11b, IEEE802.11g, and / or IEEE802.11n), VoIP, Li-Fi, Wi-MAX, and protocols for email, instant messaging, and / or SMS.

[0063] The functions or operations performed by the second electronic device 104, as described in Figure 1, can be performed by the circuit 302. The operations performed by the circuit 302 will be described in detail, for example, in Figures 4, 5, 6, 7A, 7B, and 8.

[0064] Figure 4 shows an exemplary architecture for multiscale interprediction for dynamic 3D point cloud compression and 3D point cloud reconstruction according to embodiments of the present disclosure. The description of Figure 4 will be made in relation to the elements of Figures 1, 2 and 3. Figure 4 shows an exemplary architecture 400 for dynamic 3D point cloud compression and 3D point cloud reconstruction. Architecture 400 includes a first PCC encoder 110, a first neural network predictor 112, an octree-based encoder 114, a second PCC encoder 116, an octree-based decoder 118, a second neural network predictor 120, a PCC decoder 122, a subtractor 402, a quantizer 404, an autoencoder 406, an autodecoder 408, and an accumulator 410.

[0065] At any given time, the first PCC encoder 110 can receive a reference 3D point cloud frameset and the current 3D point cloud frame to be encoded, i.e., P(t). The reference 3D point cloud frameset can contain "N" 3D point cloud frames, i.e., P(t-1), P(t-2), ..., and P(tN). The reference 3D point cloud frameset, i.e., P(t-1), P(t-2), ..., or P(tN), can precede or follow the current 3D point cloud frame. In some cases, such frames can be called previously decoded 3D point cloud frames (i.e., frames decoded before the current 3D point cloud frame was received). The first PCC encoder 110 can receive the reference 3D point cloud frameset and the current 3D point cloud frame as inputs. The first PCC encoder 110 can generate reference frame data and current frame data as its outputs, respectively. Reference frame data can include feature sets associated with the 3D points of each reference 3D point cloud frame in the reference 3D point cloud frameset. For example, the output feature set associated with the 3D points of the input reference 3D point cloud frame P(t-1) may be F(t-1). Similarly, the output feature set associated with the 3D points of the input reference 3D point cloud frame P(tN) may be F(tN). The feature set (such as F(t-1)) associated with the 3D points of each reference 3D point cloud frame (such as P(t-1)) in the reference 3D point cloud frameset can include reference features related to the occupancy rate of 3D points in the corresponding reference 3D point cloud frame, and reference coordinate information related to the coordinates of the 3D points in the corresponding reference 3D point cloud frame. Current frame data can be associated with the 3D points of the current 3D point cloud frame (i.e., P(t)) and can include a first feature set, i.e., F(t). The first feature set can be related to the occupancy rate of 3D points in the current 3D point cloud frame and can be generated as the output of the first PCC encoder 110. According to one embodiment, the 3D point can be a point in the current 3D point cloud frame that needs to be encoded.

[0066] The first neural network predictor 112 can receive, as input, a set of features associated with the 3D points of each reference 3D point cloud frame in a reference 3D point cloud frame set (i.e., F(t - 1) ··· F(t - N)). The set of features associated with the 3D points of each reference 3D point cloud frame in the reference 3D point cloud frame set can be downsampled by at least one scaling coefficient. According to one embodiment, the set of features associated with the 3D points of each reference 3D point cloud frame (having coordinates included in the reference coordinate information) can be downsampled by scaling coefficients such as 2, 3, ..., K. For example, F(t - 1) can be downsampled by scaling coefficients 2, 3, ..., K so as to generate multi-scale feature sets F 2 (t - 1), F 3 (t - 1), ..., F K (t - 1) respectively. Similarly, F(t - N) can be downsampled so as to generate multi-scale feature sets F 2 (t - N), F 3 (t - N), ..., F K (t - N).

[0067] For each scaling coefficient (i.e., 2, 3, ..., or K), the first neural network predictor 112 can generate a spatio-temporal tensor using the sets of features associated with all the reference 3D point cloud frames in the reference 3D point cloud frame set. For example, for the scaling coefficient "2", a spatio-temporal tensor can be generated based on F 2 (t - 1), ..., and F 2 (t - N). Similarly, for the scaling coefficient "K", a spatio-temporal tensor can be generated based on F[[ID=K1]] K (t - 1), ..., F KA spatiotemporal tensor can be generated based on (tN). According to one embodiment, the generation of a spatiotemporal tensor for a particular scaling factor can be based on the spatiotemporal concatenation of feature sets associated with all reference 3D point cloud frames of a reference 3D point cloud frameset downsampled by the same scaling factor. For example, the spatiotemporal tensor for a scaling factor "2" is F 2 (t-1), ..., F 2 It can be constructed based on the spatiotemporal connection of (tN). Similarly, the spatiotemporal tensor with respect to the scaling coefficient "K" is F K (t-1), ..., F K It can be constructed based on the spatiotemporal connection of (tN). Thus, (K-1) spatiotemporal tensors can be constructed for scaling factors 2, 3, ..., and K. Each spatiotemporal tensor (of the (K-1) spatiotemporal tensors) for a particular scaling factor can represent the features of all reference 3D point cloud frames that can be downsampled by the scaling factor.

[0068] The first neural network predictor 112 may further receive (target) coordinate information (i.e., C(t)) related to a 3D point in the current 3D point cloud frame (i.e., P(t)). C(t) may indicate coordinates in the current 3D point cloud frame that can include a 3D point having features that need to be predicted (or encoded) based on the feature set related to the 3D point in each reference 3D point cloud frame (which has coordinates included in the reference coordinate information). A 3D point in each reference 3D point cloud frame with coordinates related to the reference coordinate information can correspond to a 3D point in the current 3D point cloud frame with coordinates included in C(t). In some embodiments, the spatiotemporal tensor of each scaling coefficient can be constructed based on the feature set related to all reference 3D point cloud frames that are downsampled by the corresponding scaling coefficient, and the current feature set related to the 3D point in the current 3D point cloud frame. The coordinates of the 3D point are included in C(t), and the features of the current feature set related to the 3D point can be padded with "1" for all scaling coefficients (because the features of the current feature set should be predicted).

[0069] After the construction of the spatiotemporal tensors, spatiotemporal analysis can be performed on each of the (K-1) spatiotemporal tensors. According to one embodiment, spatiotemporal analysis may include performing a sparse convolution operation or a self-attention operation on each of the spatiotemporal tensors (i.e., the (K-1) spatiotemporal tensors). The sparse convolution operation of a spatiotemporal tensor can be performed based on a filter. The filter can weight features represented by spatiotemporal tensors in different spatial domains with the same weight value. The sparse convolution operation can modify each feature based on the corresponding feature, the set of neighboring features of the corresponding feature, and the weight values ​​used to weight the corresponding feature and the neighboring features, respectively. The total number of neighboring features may be based on the size of the filter. By modifying each feature, a modified spatiotemporal tensor can be generated. Thus, by performing a sparse operation on the (K-1) spatiotemporal tensors, (K-1) modified spatiotemporal tensors can be generated. Similarly, by performing a self-aware operation on the (K-1) spatiotemporal tensors, the features represented by the spatiotemporal tensors can be modified. This modification can be based on a filter of a predetermined size that weights the features represented by the spatiotemporal tensors. The filter can modify each feature based on the corresponding feature, the weight of the corresponding feature, the neighboring features of the corresponding feature, and the respective weights of the neighboring features. By modifying the features, a modified spatiotemporal tensor can be generated. Thus, (K-1) modified spatiotemporal tensors can be generated based on the application of a self-aware network to the (K-1) spatiotemporal tensors.

[0070] The features represented by each modified spatiotemporal tensor (generated based on sparse convolution or self-attention) can be downsampled by a specific scaling factor. The scaling factor for which the modified spatiotemporal tensor should be downsampled can be based on a scaling factor (such as 2, 3, ..., or K) that allows the original version of the modified spatiotemporal tensor to be constructed, and the highest scaling factor (i.e., K) that downsamples the feature set of each reference 3D point cloud frame in the reference 3D point cloud frameset. An inception residual network can then be applied to each downsampled modified spatiotemporal tensor to generate the final spatiotemporal tensors. Thus, (K-1) final spatiotemporal tensors can be generated as the output of the inception residual network.

[0071] The (K-1) final spatiotemporal tensors can be concatenated to generate a multiscale feature concatenation vector. A sparse convolution or self-attention operation can be performed on the multiscale feature concatenation vector. Based on the results of the sparse convolution or self-attention operation, the first neural network predictor 112 generates a second set of features (i.e., F) related to the 3D points (having coordinates included in C(t)) of the current 3D point cloud frame (i.e., P(t)). ~ (t) can be predicted. Thus, the second feature set can be predicted based on the coordinate information (i.e., C(t)) associated with the 3D points of the current 3D point cloud frame.

[0072] The subtractor 402 takes the first feature set (i.e., F(t)) and the second feature set (i.e., F ~It can accept (t)) as input. Based on the first feature set and the second feature set, the residual feature set (i.e., R(t)) can be calculated. The residual feature set consists of the actual feature set (i.e., F(t) generated by the first PCC encoder 110) and the predicted feature set (i.e., F) generated by the first neural network predictor 112. ~ The difference between (t) and (t) can be shown. This difference can be used to correct for any errors that may occur in predicting features associated with the current 3D point cloud frame (i.e., P(t)). The predicted features can be used to reconstruct the current 3D point cloud frame (i.e., the 3D points of P(t) with coordinates contained in C(t)).

[0073] Quantizer 404 can accept a residual feature set (i.e., R(t)) and quantize each residual feature of the residual feature set to a predetermined quantization level set. Quantizer 404 can generate a quantized residual feature set as output based on the application of a quantization scheme to the residual feature set. The quantization scheme can be based on an entropy model. Autoencoder 406 can accept the quantized residual feature set as input and generate a bitstream of encoded point cloud data of the current 3D point cloud frame as output. This generation can be based on the application of an encoding scheme to the quantized residual feature set. The encoding scheme can be based on an entropy model and can encode the quantized residual feature set by compressing each quantized residual feature of the quantized residual feature set.

[0074] According to one embodiment, the octree-based encoder 114 can receive coordinate information (i.e., C(t)) as input, encode the coordinate information, and generate encoded coordinate information. The circuit 202 can include the encoded coordinate information in the bitstream of the generated encoded point cloud data and transmit the generated bitstream to the second electronic device 104.

[0075] Circuit 302 can receive the generated bitstream and extract encoded coordinate information and encoded point cloud data from the bitstream. The octree-based decoder 118 can receive the encoded coordinate information as input and decode the encoded coordinate information to reconstruct the coordinate information (i.e., C(t)). The coordinate information can be associated with a 3D point in the current 3D point cloud frame (i.e., P(t)) to be decoded. Based on the reconstructed coordinate information, the target coordinates in the current 3D point cloud frame containing the 3D point to be decoded can be determined.

[0076] The autodecoder 408 receives encoded point cloud data as input and generates a residual feature set (i.e., R~ (t)) can be reconstructed as output. This reconstruction can be based on the application of a decoding scheme to encoded point cloud data. The decoding scheme can be based on an entropy model and can decompress the quantized residual feature set. The residual feature set (i.e., R ~ Using (t), the 3D points of the current 3D point cloud frame (i.e., P(t)) (with coordinates included in C(t)) can be reconstructed.

[0077] The second PCC encoder 116 can accept a 3D point cloud sequence as input, which may include a reference 3D point cloud frameset, i.e., P(t-1), P(t-2), ..., and P(tN). Based on the input, it can generate reference frame data as output, which includes a feature set associated with the 3D points of each reference 3D point cloud frame in the reference 3D point cloud frameset. The reference frame data can be generated based on the application of the second PCC encoder 116 to the reference 3D point cloud frameset. For example, the feature set associated with the 3D points of reference 3D point cloud frame P(t-1) may be F(t-1). Similarly, the feature set associated with the 3D points of reference 3D point cloud frame P(tN) may be F(tN). The feature set associated with the 3D points of each reference 3D point cloud frame in the reference 3D point cloud frameset may include reference features related to the occupancy rate of 3D points in the corresponding reference 3D point cloud frame of the reference 3D point cloud frameset, and reference coordinate information related to the 3D points of the corresponding reference 3D point cloud frame. These 3D points can correspond to 3D points in the current 3D point cloud frame (i.e., P(t)) that have coordinates contained in the regenerated coordinate information (i.e., C(t)).

[0078] The second neural network predictor 120 can receive as input a feature set associated with the 3D points of each reference 3D point cloud frame in the reference 3D point cloud frameset. The second neural network predictor 120 can then predict as output a third feature set (i.e., F'(t)) associated with the 3D points of the current 3D point cloud frame (i.e., P(t)). The 3D points of P(t) need to be decoded, and the coordinates of the 3D points may be contained in C(t). The prediction can be based on the application of the second neural network predictor 120 to the reference frame data, i.e., the feature set associated with the 3D points of each reference 3D point cloud frame. The third feature set can be further predicted based on the (re)generated coordinate information (i.e., C(t)). The generation of the third feature set can be identical to the generation of the second feature set by the first neural network predictor 112.

[0079] The accumulator 410 processes the third feature set (i.e., F'(t)) and the reconstructed residual feature set (i.e., R ~ The accumulator 410 can accept (t)) as input. The accumulator 410 can generate a fourth feature set (i.e., F''(t)) as output, relating to the 3D points (i.e., those having coordinates contained in C(t)) of the current 3D point cloud frame (i.e., P(t)). Generation can be based on the accumulation of the third feature set and the reconstructed residual feature set. The PCC decoder 122 can accept the fourth feature set as input and reconstruct the current 3D point cloud frame (i.e., P(t)) based on the application of a decoding scheme to the fourth feature set. The PCC decoder 122 can generate a 3D point cloud frame (i.e., P'(t)) as output, which can correspond to P(t).

[0080] Figure 5 is a block diagram illustrating an exemplary operation according to an embodiment of the present disclosure, in which a feature set associated with a current 3D point cloud frame is predicted based on multiscale features associated with a reference 3D point cloud frameset. The description of Figure 5 will be made in relation to the elements of Figures 1, 2, 3 and 4. Figure 5 shows an exemplary block diagram 500 for predicting a feature set associated with a current 3D point cloud frame. The exemplary block diagram 500 may include a set of blocks that can represent a set of operations starting from 502 and ending at 508. The set of operations can be performed by a first neural network predictor 112 or a second neural network predictor 120. The first neural network predictor 112 or the second neural network predictor 120 may take two feature sets, namely F(t-1) and F(t-2), as input. F(t-1) may be associated with 3D points in a reference 3D point cloud frame P(t-1). F(t-2) may be associated with 3D points in a reference 3D point cloud frame P(t-2). Both P(t-2) and P(t-1) can precede the current 3D point cloud frame P(t). P(t) should be encoded or decoded based on predictions of features associated with the 3D points in P(t).

[0081] F(t-1) can include reference features related to the occupancy rate of 3D points in P(t-1) and reference coordinate information related to the 3D points in P(t-1). For example, F(t-1) can include reference features f related to four 3D points in P(t-1). 11 ,f 13 ,f 14 and f 15 This can include the reference coordinates of four 3D points. On the other hand, F(t-2) can include reference features related to the occupancy rate of the 3D points of P(t-2) and reference coordinate information related to the 3D points of P(t-2). For example, F(t-2) can include reference features f related to the five 3D points of P(t-2). 23 ,f 24 ,f 25 ,f 26 and f 28 This can include the reference coordinates of five 3D points.

[0082] A feature set associated with the 3D points of P(t) can be constructed based on (target) coordinate information C(t). C(t) can include the coordinates of the 3D points of P(t). The first neural network predictor 112 or the second neural network predictor 120 can accept C(t) as input. For example, C(t) can include the coordinates of five 3D points. These coordinates are c 33 , c 34 , c 35 , c36 and c 39 It can be that the feature set can include features related to five 3D points of P(t). According to one embodiment, the feature set can be constructed by padding the features related to the five 3D points with "1". The constructed feature set is F(t), i.e., [1 33 ,1 34 ,1 35 ,1 36 ,1 39 ]) can be expressed as ]). The first neural network predictor 112 or the second neural network predictor 120 downsamples F(t-2), F(t-1), and F(t) respectively by scaling factors of "2" and "3" for encoding or decoding P(t), and then expresses a feature set associated with five 3D points of P(t) (i.e., F ~ (t))(For example, [f 33 ,f 34 ,f 35 ,f 36 ,f 39 It is possible to predict ]).

[0083] The reference feature of F(t-2) (i.e., f 11 ,f 13 ,f 14 and f 15 ) is F 2 (t-2) and F 3 To generate (t-2), we can downsample using scaling factors of "2" and "3", respectively. Similarly, the reference feature of F(t-1) (i.e., f23 ,f 24 ,f 25 ,f 26 and f 28 ) is F 2 (t-1) and F 3 (t-1) can be generated by downsampling using scaling factors of "2" and "3", respectively. Since F(t) is constructed by padding the 3D point features of P(t) with "1", downsampling F(t) using scaling factors of "2" and "3" will return F(t).

[0084] In 502A, F(t), F 2 (t-1) and F 2 (t-2) can be spatiotemporally concatenated. This spatiotemporal concatenation can generate a first spatiotemporal tensor. The first spatiotemporal tensor can be constructed with a scaling factor of "2" and can be generated based on the features of all reference 3D point cloud frames (i.e., P(t-2) and P(t-1)) downsampled by the scaling factor of "2". For example, the features represented by the first spatiotemporal tensor can include feature vectors such as [f11,0,0], [0,0,0], [f13,f23,1], [f14,f24,1], [f15,f25,1], [0,f26,1], [0,0,0], [0,f28,0] and [0,0,1]. Here, [0,0,0] represents an empty region.

[0085] In 502B, F(t), F 3 (t-1) and F 3 (t-2) can be spatiotemporally concatenated. This spatiotemporal concatenation can generate a second spatiotemporal tensor. The second spatiotemporal tensor can be constructed with a scaling factor of "3" and can be generated based on the features of all reference 3D point cloud frames (i.e., P(t-2) and P(t-1)) downsampled by the scaling factor of "3".

[0086] In 504A, a spatiotemporal analysis can be performed on the first spatiotemporal tensor. The spatiotemporal analysis can be based on self-attentional actions and can modify the features represented by the first spatiotemporal tensor. The modification of each feature can be based on one or more of the following: the size of the filter used to perform the self-attentional action, the weights assigned to the corresponding feature, and the weights assigned to each adjacent feature of the corresponding feature. Based on the modification of each feature represented by the first spatiotemporal tensor, a modified first spatiotemporal tensor can be generated.

[0087] In 504B, spatiotemporal analysis can be performed on the second spatiotemporal tensor. This spatiotemporal analysis can be based on self-attentional actions, and each feature represented by the second spatiotemporal tensor can be modified. Based on the modification of each feature, a modified second spatiotemporal tensor can be generated.

[0088] In 506, a multiscale feature-connected vector can be generated based on the spatiotemporal connection between the modified first spatiotemporal tensor and the modified second spatiotemporal tensor.

[0089] In 508, features associated with the 3D points of the current 3D point cloud frame (i.e., P(t)) can be predicted. The prediction can be based on the application of a sparse convolution operation or a self-attention operation to the generated multiscale feature-connected vector. The coordinates of the 3D points of the current 3D point cloud frame (from which features are predicted) can be contained in C(t). Thus, the prediction can be further based on coordinate information (i.e., C(t)). The predicted features are f 33 ,f 34 ,f 35 ,f 36 and f 39 It can include such predicted features, 3D points (i.e., c 33 , c 34 , c 35 , c 36 and c 39 ) can be encoded or decoded.

[0090] Figure 6 is a block diagram illustrating an exemplary operation according to an embodiment of the present disclosure, in which a feature set associated with a current 3D point cloud frame is predicted based on multiscale features associated with a reference 3D point cloud frame set. The description of Figure 6 will be made in relation to the elements of Figures 1, 2, 3, 4, and 5. Figure 6 shows an exemplary block diagram 600 for predicting a feature set associated with a current 3D point cloud frame (i.e., P(t)). Block diagram 600 may include a set of blocks that can represent a series of operations starting from 602 and ending at 608. The series of operations can be performed by a first neural network predictor 112 or a second neural network predictor 120. The first neural network predictor 112 or the second neural network predictor 120 may take two feature sets, namely F(t-1) and F(t-2), as input. The first neural network predictor 112 or the second neural network predictor 120 downsamples F(t-2) and F(t-1) respectively with scaling factors of "2" and "3", respectively, to the feature set (i.e., F) related to the 3D of P(t) having coordinates contained in C(t). ~ (t)) can be predicted F ~ (t) can be predicted for encoding or decoding P(t).

[0091] In 602A, F 2 (t-1) and F 2 (t-2) can be spatiotemporally concatenated. This spatiotemporal concatenation can generate a first spatiotemporal tensor. The first spatiotemporal tensor can be constructed with a scaling factor of "2" and can be generated based on the features of all reference 3D point cloud frames (i.e., P(t-2) and P(t-1)) downsampled by the scaling factor of "2". For example, the features represented by the first spatiotemporal tensor are [f11,0], [0,0], [f13,f 23 ], [f 14 ,f 24 ], [f15 ,f 25 ], [0,f 26 ], [0,0], [0,f 28 Feature vectors such as [0,0] can be included. Here, [0,0,0] represents an empty region.

[0092] In 602B, F 3 (t-1), F 3 (t-2) can be spatiotemporally concatenated. This spatiotemporal concatenation can generate a second spatiotemporal tensor. The second spatiotemporal tensor can be constructed with a scaling factor of "3" and can therefore be generated based on the features of all reference 3D point cloud frames (i.e., P(t-2) and P(t-1)) downsampled by the scaling factor of "3".

[0093] In 604A, a spatiotemporal analysis can be performed on the first spatiotemporal tensor. In one embodiment, the spatiotemporal analysis can be based on a sparse convolution operation to modify each feature represented by the first spatiotemporal tensor. The modification of each feature can be based on the size of the filter used to perform the sparse convolution operation, and one or more of the weights assigned to each of the corresponding features and their adjacent features. Based on the modification of each feature represented by the first spatiotemporal tensor, a modified first spatiotemporal tensor can be generated. Subsequently, the modified first spatiotemporal tensor can be downsampled to generate a downsampled modified first spatiotemporal tensor. The downsampled first spatiotemporal tensor can be passed as input to a first inception residual network, which generates the final spatiotemporal tensor as output.

[0094] In 604B, spatiotemporal analysis can be performed on the second spatiotemporal tensor. The spatiotemporal analysis can be based on sparse convolution operations and can modify each feature represented by the second spatiotemporal tensor. Based on the modification of each feature, a modified second spatiotemporal tensor can be generated. The modified second spatiotemporal tensor can be passed as input to the first inception residual network, which generates the final spatiotemporal tensor as output.

[0095] In 606, a multiscale feature-connected vector can be generated based on the spatiotemporal connection between the final spatiotemporal tensor generated by the first inception residual network and the final spatiotemporal tensor generated by the second inception residual network.

[0096] In 608, features associated with the 3D points of the current 3D point cloud frame (i.e., P(t)) can be predicted. The prediction can be based on applying a sparse convolution operation or a self-attention operation to the generated multiscale feature-connected vector. The coordinates of the 3D points of the current 3D point cloud frame (whose features are predicted) can be contained in C(t). Thus, the prediction can be further based on coordinate information (i.e., C(t)). Based on the predicted features, the 3D points of P(t) can be encoded or decoded.

[0097] Figure 7A shows an exemplary scenario for encoding or decoding a 3D point cloud frame based on a preceding 3D point cloud frame according to embodiments of the present disclosure. The description of Figure 7A is made in relation to the elements of Figures 1, 2, 3, 4, 5, and 6. Figure 7A shows exemplary scenario 700A. Exemplary scenario 700A shows an exemplary sequence of 3D point cloud frames, which may include an "I" 3D point cloud frame and a "P" 3D point cloud frameset. An "I" 3D point cloud frame can be encoded / decoded individually, while each "P" 3D point cloud frame can be encoded / decoded based on a previous 3D point cloud frame. For example, at time t0, an "I" 3D point cloud frame can be encoded / decoded. At time t1, the "P" 3D point cloud frames of the "P" 3D point cloud frameset can be encoded / decoded based on predicted features related to the occupancy rate of 3D points in the "I" 3D point cloud frame. At time t2, another "P"3D point cloud frame in the "P"3D point cloud frameset can be encoded / decoded based on the predicted features related to the occupancy rate of 3D points in the "P"3D point cloud frame encoded / decoded at time t1. Other "P"3D point cloud frames in the "P"3D point cloud frameset can be encoded / decoded in the same manner.

[0098] Figure 7B shows an exemplary scenario for encoding or decoding a 3D point cloud frame based on two preceding 3D point cloud frames according to embodiments of the present disclosure. The description of Figure 7B is made in relation to the elements of Figures 1, 2, 3, 4, 5, 6 and 7A. Figure 7B shows exemplary scenario 700B. Exemplary scenario 700B shows an exemplary sequence of 3D point cloud frames that may include two "I" 3D point cloud frames and a "P" 3D point cloud frameset. The "I" 3D point cloud frames can be encoded / decoded individually, while each "P" 3D point cloud frame can be encoded / decoded based on two preceding 3D point cloud frames.

[0099] For example, at time t0, a first "I" 3D point cloud frame can be encoded or decoded, and at time t1, a second "I" 3D point cloud frame can be encoded or decoded. At time t2, a "P" 3D point cloud frame from the "P" 3D point cloud frameset can be encoded / decoded based on the predicted features related to the 3D point occupancy of the first "I" 3D point cloud frame and the predicted features related to the 3D point occupancy of the second "I" 3D point cloud frame. At time t3, another "P" 3D point cloud frame from the "P" 3D point cloud frameset can be encoded / decoded based on the predicted features related to the 3D point occupancy of the second "I" 3D point cloud frame (encoded / decoded at time t1) and the predicted features related to the 3D point occupancy of the "P" 3D point cloud frame (encoded / decoded at time t2). At time t4, another "P"3D point cloud frame in the "P"3D point cloud frameset can be encoded / decoded based on the predicted features related to the 3D point occupancy of the "P"3D point cloud frame encoded / decoded at time t2, and the predicted features related to the 3D point occupancy of the "P"3D point cloud frame encoded / decoded at time t2. Other "P"3D point cloud frames in the "P"3D point cloud frameset can be encoded / decoded in the same manner.

[0100] Figure 8 shows an exemplary scenario for encoding / decoding 3D point cloud frames based on preceding and succeeding 3D point cloud frames according to embodiments of the present disclosure. The description of Figure 8 is made in relation to the elements of Figures 1, 2, 3, 4, 5, 6, 7A, and 7B. Figure 8 shows exemplary scenario 800. Exemplary scenario 800 shows an exemplary sequence of 3D point cloud frames that may include two "I" 3D point cloud frames, three "P" 3D point cloud frames, and three "B" 3D point cloud frames. Each "I" 3D point cloud frame can be encoded / decoded individually. Each "P" 3D point cloud frame can be encoded / decoded based on a preceding 3D point cloud frame, such as an "I" 3D point cloud frame or a "P" 3D point cloud frame. Each "B" 3D point cloud frame can be encoded / decoded based on a previous 3D point cloud frame (such as an "I" 3D point cloud frame or a "P" 3D point cloud frame) and a subsequent 3D point cloud frame (such as a "P" 3D point cloud frame).

[0101] For example, at time t0, the first "I" 3D point cloud frame (i.e., the first frame with frame index -0) can be encoded / decoded, and at time t1, the second "I" 3D point cloud frame (i.e., the second frame with frame index -1) can be encoded / decoded. At time t2, the first "P" 3D point cloud frame (i.e., the fourth frame with frame index -3) can be encoded or decoded based on predicted features related to the occupancy rate of 3D points in the second "I" 3D point cloud frame. The encoding / decoding order of the first "P" 3D point cloud frame can be "2" because the first "P" 3D point cloud frame is a third frame that is encoded / decoded (after the encoding / decoding of the first and second "I" 3D point cloud frames). At time t3, the first "B" 3D point cloud frame (i.e., the third frame with frame index -2) can be encoded or decoded based on the predicted features related to the occupancy rate of 3D points in the second "I" 3D point cloud frame (i.e., the preceding frame) and the predicted features related to the occupancy rate of 3D points in the first "P" 3D point cloud frame (i.e., the succeeding frame). The encoding / decoding order of the first "B" 3D point cloud frame can be "3" (i.e., the fourth frame to be encoded or decoded). At time t4, the second "P" 3D point cloud frame (i.e., the sixth frame with frame index -5) can be encoded / decoded based on the predicted features related to the occupancy rate of 3D points in the first "P" 3D point cloud frame. The encoding / decoding order of the second "P" 3D point cloud frame can be "4" (i.e., the fifth frame to be encoded or decoded). At time t5, a second "B" 3D point cloud frame (i.e., a fifth frame with frame index -4) can be encoded or decoded based on the predicted features related to the occupancy rate of 3D points in the first "P" 3D point cloud frame (i.e., the preceding frame) and the predicted features related to the occupancy rate of 3D points in the second "P" 3D point cloud frame (i.e., the succeeding frame). The encoding / decoding order of the second "B" 3D point cloud frame can be "5" (i.e., the sixth frame to be encoded or decoded).The third "P" 3D point cloud frame (i.e., the eighth frame with frame index -7) and the third "B" 3D point cloud frame (i.e., the seventh frame with frame index -6) can be encoded / decoded in the same manner.

[0102] Figure 9 is a flowchart illustrating the operation of an exemplary method for multiscale interpretation for dynamic 3D point cloud frame compression according to an embodiment of the present disclosure. The description of Figure 9 is made in relation to the elements of Figures 1, 2, 3, 4, 5, 6, 7A, 7B and 8. Figure 9 shows flowchart 900. The operations 902-916 can be performed by either a computer system, such as the first electronic device 102 or the circuit 202 of the first electronic device 102. The operation can start from 902 and proceed to 904.

[0103] At 904, a 3D point cloud sequence can be received that includes a reference 3D point cloud frameset and the current 3D point cloud frame to be encoded. In at least one embodiment, circuit 202 can be configured to receive a 3D point cloud sequence that includes a reference 3D point cloud frameset and the current 3D point cloud frame to be encoded. Details on receiving the reference 3D point cloud frameset and the current 3D point cloud frame are illustrated, for example, in Figures 1 and 4.

[0104] In 906, reference frame data can be generated, which includes a feature set associated with each reference 3D point cloud frame in the reference 3D point cloud frameset. In at least one embodiment, the circuit 202 can be configured to generate reference frame data, which includes a feature set associated with each reference 3D point cloud frame in the reference 3D point cloud frameset. Details of the generation of reference frame data are illustrated, for example, in Figures 1 and 4.

[0105] In 908, current frame data related to the 3D points of the current 3D point cloud frame can be generated. In at least one embodiment, circuit 202 can be configured to generate current frame data related to the 3D points of the current 3D point cloud frame. The current frame data may include a first set of features related to the occupancy rate of 3D points in the current 3D point cloud frame. Details of the generation of current frame data are illustrated, for example, in Figures 1 and 4.

[0106] In 910, a second set of features associated with the 3D points of the current 3D point cloud frame can be predicted based on the application of a first neural network predictor to reference frame data. In at least one embodiment, the circuit 202 can be configured to predict a second set of features associated with the 3D points of the current 3D point cloud frame based on the application of a first neural network predictor to reference frame data. Details of the prediction of the second set of features are illustrated, for example, in Figures 1, 4, 5, and 6.

[0107] In 912, the residual feature set can be calculated based on the first feature set and the second feature set. In at least one embodiment, the circuit 202 can be configured to calculate the residual feature set based on the first feature set and the second feature set. Details of the calculation of the residual feature set are illustrated, for example, in Figures 1 and 4.

[0108] In 914, a quantized residual feature set can be generated based on the application of a quantization scheme to the residual feature set. In at least one embodiment, the circuit 202 can be configured to generate a quantized residual feature set based on the application of a quantization scheme to the residual feature set. Details of the generation of the quantized residual feature set are illustrated, for example, in Figures 1 and 4.

[0109] In 916, a bitstream of encoded point cloud data can be generated for the current 3D point cloud frame based on the application of an encoding scheme to the quantized residual feature set. In at least one embodiment, circuit 202 can be configured to generate a bitstream of encoded point cloud data for the current 3D point cloud frame based on the application of an encoding scheme to the quantized residual feature set. Details of the generation of the bitstream of encoded point cloud data are illustrated, for example, in Figures 1 and 4. Control can then proceed to termination.

[0110] While flowchart 700 shows discrete operations such as 904, 906, 908, 910, 912, 914, and 916, the disclosure is not limited in this way. Accordingly, in some embodiments, such discrete operations can be further divided into further operations, combined into fewer operations, or deleted, depending on the implementation, without impairing the essence of the disclosed embodiments.

[0111] Figure 10 is a flowchart illustrating the operation of an exemplary method for multiscale interpretation for 3D point cloud frame reconstruction according to embodiments of the present disclosure. The description of Figure 10 is made in relation to the elements of Figures 1, 2, 3, 4, 5, 6, 7A, 7B, 8, and 9. Figure 10 shows flowchart 1000. The operations 1002-1014 can be performed by a computer system, such as a second electronic device 104 or circuit 302 of the second electronic device 104. The operation can start from 1002 and proceed to 1004.

[0112] In 1004, a 3D point cloud sequence can be received, which may include a reference 3D point cloud frameset. In at least one embodiment, the circuit 302 can be configured to receive a 3D point cloud sequence, which may include a reference 3D point cloud frameset. Details on receiving the 3D point cloud sequence are illustrated, for example, in Figures 1 and 4.

[0113] In 1006, reference frame data can be generated, which includes a feature set associated with the 3D points of each reference 3D point cloud frame in the reference 3D point cloud frameset. In at least one embodiment, the circuit 302 can be configured to generate reference frame data, which includes a feature set associated with the 3D points of each reference 3D point cloud frame in the reference 3D point cloud frameset. Details of the generation of reference frame data are illustrated, for example, in Figures 1 and 4.

[0114] At 1008, a bitstream of encoded point cloud data related to the current 3D point cloud frame to be decoded can be received. In at least one embodiment, circuit 302 can be configured to receive a bitstream of encoded point cloud data related to the current 3D point cloud frame to be decoded. Details of receiving the bitstream are illustrated, for example, in Figures 1 and 4.

[0115] In 1010, a third set of features associated with the 3D points of the current 3D point cloud frame can be predicted based on the application of a second neural network predictor to reference frame data. In at least one embodiment, the circuit 302 can be configured to predict a third set of features associated with the 3D points of the current 3D point cloud frame based on the application of a second neural network predictor to reference frame data. Details of the prediction of the third set of features are illustrated, for example, in Figures 1, 4, 5, and 6.

[0116] In 1012, a fourth feature set related to the 3D points of the current 3D point cloud frame can be generated based on the received encoded point cloud data bitstream and the predicted third feature set. In at least one embodiment, the circuit 302 can be configured to generate a fourth feature set related to the 3D points of the current 3D point cloud frame based on the received encoded point cloud data bitstream and the predicted third feature set. Details of the generation of the fourth feature set are illustrated, for example, in Figures 1 and 4.

[0117] In 1014, the current 3D point cloud frame can be reconstructed based on the application of the decoding scheme to the determined fourth set of features. In at least one embodiment, the circuit 302 can be configured to reconstruct the current 3D point cloud frame based on the application of the decoding scheme to the determined fourth set of features. Details of the reconstruction of the current 3D point cloud frame are illustrated, for example, in Figures 1 and 4. Control can then proceed to termination.

[0118] While flowchart 1000 shows discrete operations such as 1004, 1006, 1008, 1010, 1012, and 1014, the disclosure is not limited in this way. Accordingly, in some embodiments, such discrete operations can be further divided into further operations, combined into fewer operations, or deleted, depending on the implementation, without impairing the essence of the disclosed embodiments.

[0119] Exemplary embodiments of the present disclosure include an electronic device (such as the first electronic device 102 in Figure 1) which may include a circuit (such as the circuit 202 in Figure 2) which may be communicatively coupled to another electronic device (such as the second electronic device 104 in Figure 1). The first electronic device 102 may further include a memory (such as the memory 204 in Figure 2) which may be configured to store a predictor (such as the first neural network predictor 112 in Figure 1). The memory 204 may be configured to store a PCC encoder (such as the first PCC encoder 110 in Figure 1). The circuit 202 may be configured to receive a 3D point cloud sequence which may include a reference 3D point cloud frameset and the current 3D point cloud frame to be encoded. The reference 3D point cloud frameset may include at least one reference 3D point cloud frame which may precede the current 3D point cloud frame, or at least one reference 3D point cloud frame which may follow the current 3D point cloud frame. Circuit 202 can be further configured to generate reference frame data, which includes a feature set associated with the 3D points of each reference 3D point cloud frame in the reference 3D point cloud frameset. The reference frame data can be generated based on the application of the first PCC encoder 110 to the reference 3D point cloud frameset. The feature set associated with the 3D points of each reference 3D point cloud frame in the reference 3D point cloud frameset may include reference features related to the occupancy rate of 3D points in the corresponding reference 3D point cloud frame of the reference 3D point cloud frameset, and reference coordinate information related to the 3D points of the corresponding reference 3D point cloud frame. Circuit 202 can be further configured to generate current frame data, which includes the 3D points of the current 3D point cloud frame. The current frame data can be generated based on the application of the first PCC encoder 110 to the current 3D point cloud frame. The current frame data may include a first feature set related to the occupancy rate of 3D points in the current 3D point cloud frame. The circuit 202 can be further configured to predict a second set of features associated with the 3D points of the current 3D point cloud frame, based on the application of the first neural network predictor 112 to reference frame data.A second feature set can be predicted based on coordinate information associated with the 3D points of the current 3D point cloud frame. Circuit 202 can be further configured to compute a residual feature set based on the first and second feature sets. Circuit 202 can be further configured to generate a quantized residual feature set based on the application of a quantization scheme to the residual feature set. Circuit 202 can be further configured to generate a bitstream of encoded point cloud data for the current 3D point cloud frame based on the application of an encoding scheme to the quantized residual feature set.

[0120] According to one embodiment, the circuit 202 can be further configured to encode the coordinate information based on the application of an octree-based encoder (such as the octree-based encoder 114) to the coordinate information. The bitstream of encoded point cloud data can contain the encoded coordinate information.

[0121] According to one embodiment, the circuit 202 can be further configured to downsample the feature set associated with the 3D points of each reference 3D point cloud frame in the reference 3D point cloud frameset by at least one scaling factor.

[0122] Exemplary embodiments of the present disclosure include electronic devices (such as a second electronic device 104 in Figure 1) which may include circuits (such as circuit 302 in Figure 3) which may be communicatively coupled to another electronic device (such as a first electronic device 102 in Figure 1). The second electronic device 104 may further include a memory (such as memory 304 in Figure 3) which may be configured to store a predictor (such as a second neural network predictor 120 in Figure 1). The memory 204 may be configured to store a PCC encoder (such as a second PCC encoder 116 in Figure 1) and a PCC decoder (such as a PCC decoder 122 in Figure 1). Circuit 302 may be configured to receive a 3D point cloud sequence which may include a reference 3D point cloud frameset. Circuit 302 may be further configured to generate reference frame data which includes a set of features associated with each reference 3D point cloud frame of the reference 3D point cloud frameset. The reference frame data may be generated based on the application of the second PCC encoder 116 to the reference 3D point cloud frameset. The feature set associated with the 3D points of each reference 3D point cloud frame in the reference 3D point cloud frameset may include reference features related to the occupancy rate of 3D points in the corresponding reference 3D point cloud frame of the reference 3D point cloud frameset, and reference coordinate information associated with the 3D points of the corresponding reference 3D point cloud frame. Circuit 302 can be further configured to receive a bitstream of encoded point cloud data related to the current 3D point cloud frame to be decoded. The received bitstream of encoded point cloud data may further include encoded coordinate information associated with the 3D points of the current 3D point cloud frame. Circuit 302 can be further configured to predict a third feature set associated with the 3D points of the current 3D point cloud frame based on the application of a second neural network predictor 120 to the reference frame data. Circuit 302 can be further configured to generate a fourth feature set associated with the 3D points of the current 3D point cloud frame based on the received bitstream of encoded point cloud data and the predicted third feature set.Circuit 302 can be further configured to reconstruct the current 3D point cloud frame based on the application of a decoding scheme to the determined fourth feature set.

[0123] According to one embodiment, the circuit 302 can be further configured to generate coordinate information relating to 3D points in the current 3D point cloud frame based on the application of an octree-based decoder (such as the octree-based decoder 118) to encoded coordinate information. A third set of features can be predicted based on the generated coordinate information.

[0124] This disclosure can be implemented in hardware or in a combination of hardware and software. This disclosure can be implemented centrally within at least one computer system or in a distributed manner, where different elements can be distributed across multiple interconnected computer systems. A computer system or other device adapted to perform the methods described herein may be suitable. The hardware-software combination may be a general-purpose computer system including a computer program that, when loaded and executed, can control the computer system to perform the methods described herein. This disclosure can also be implemented in hardware, including a portion of an integrated circuit that also performs other functions.

[0125] This disclosure includes all features that enable the implementation of the methods described herein and can be incorporated into a computer program product that can perform these methods when loaded onto a computer system. In this context, a computer program means any expression in any language, code, or notation of an instruction set intended to be executed directly, or after either a) conversion to another language, code, or notation, or b) reproduction in a different content form, on a system having information processing capabilities.

[0126] While this disclosure has been described with reference to several embodiments, those skilled in the art will understand that various modifications can be made and equivalents can be substituted without departing from the scope of this disclosure. Furthermore, many modifications can be made without departing from the scope of this disclosure to suit specific circumstances or content to the teachings of this disclosure. Accordingly, this disclosure is not limited to the specific embodiments disclosed, but is intended to include all embodiments that fall within the scope of the appended claims. [Explanation of symbols]

[0127] 102 First Electronic Device 104 Second Electronic Device 110 First PCC encoder 112 First NN Predictor 114 Octvine-based encoder 116 Second PCC encoder 118 Octvine-based decoder 120 Second NN Predictor 122 PCC Decoders 402 Subtractor 404 Quantizer 406 Autoencoder 408 Autodecoder 410 Accumulator

Claims

1. The first electronic device, A memory configured to store a first neural network predictor, Circuits and, The circuit is equipped with, It receives a reference 3D point cloud frameset and a 3D point cloud sequence containing the current 3D point cloud frame to be encoded. A reference frame data is generated that includes a feature set associated with the 3D points of each reference 3D point cloud frame in the aforementioned reference 3D point cloud frameset. Current frame data is generated, which includes a first set of features related to the 3D points in the current 3D point cloud frame and the occupancy rate of the 3D points in the current 3D point cloud frame. Based on the application of the first neural network predictor to the reference frame data, a second set of features related to the 3D points in the current 3D point cloud frame is predicted. Based on the first feature set and the second feature set, calculate the residual feature set. A quantized residual feature set is generated based on the application of a quantization scheme to the residual feature set. Based on the application of the encoding scheme to the quantized residual feature set, a bitstream of encoded point cloud data is generated for the current 3D point cloud frame. It is configured in such a way. A first electronic device characterized by the following:

2. The second set of features is predicted based on the coordinate information associated with the 3D points in the current 3D point cloud frame. The first electronic device according to claim 1.

3. The circuit is further configured to encode the coordinate information based on the application of an octvine-based encoder to the coordinate information, The bitstream of the encoded point cloud data includes the encoded coordinate information. The first electronic device according to claim 2.

4. The memory is further configured to store a first point cloud compression (PCC) encoder, The reference frame data is generated based on the application of the first PCC encoder to the reference 3D point cloud frameset. The current frame data is generated based on the application of the first PCC encoder to the current 3D point cloud frame. The first electronic device according to claim 1.

5. The feature set associated with the 3D points of each reference 3D point cloud frame in the aforementioned reference 3D point cloud frameset is: A reference feature related to the occupancy rate of the 3D points in the corresponding reference 3D point cloud frame of the aforementioned reference 3D point cloud frameset, The reference coordinate information relating to the 3D points of the corresponding reference 3D point cloud frame, The first electronic device according to claim 1, including the following:

6. The reference 3D point cloud frameset includes at least one reference 3D point cloud frame preceding the current 3D point cloud frame, or at least one reference 3D point cloud frame succeeding the current 3D point cloud frame. The first electronic device according to claim 1.

7. The circuit is further configured to downsample the feature set associated with the 3D points of each reference 3D point cloud frame in the reference 3D point cloud frameset by at least one scaling factor. The first electronic device according to claim 1.

8. A second electronic device, A memory configured to store a second neural network predictor, Circuits and, The circuit is equipped with, We receive a 3D point cloud sequence that includes a reference 3D point cloud frameset, A reference frame data is generated that includes a feature set associated with the 3D points of each reference 3D point cloud frame in the aforementioned reference 3D point cloud frameset. Receive a bitstream of encoded point cloud data associated with the current 3D point cloud frame to be decoded. Based on the application of the second neural network predictor to the reference frame data, a third set of features related to the 3D points of the current 3D point cloud frame is predicted. Based on the received bitstream of encoded point cloud data and the predicted third feature set, a fourth feature set related to the 3D points of the current 3D point cloud frame is generated. The current 3D point cloud frame is reconstructed based on the application of the decoding scheme to the determined fourth feature set. It is configured in such a way. A second electronic device characterized by the following:

9. The bitstream of the received encoded point cloud data further includes encoded coordinate information relating to the 3D points of the current 3D point cloud frame. The second electronic device according to claim 8.

10. The circuit is further configured to generate coordinate information relating to the 3D points of the current 3D point cloud frame based on the application of an octvine-based decoder to the encoded coordinate information. The second electronic device according to claim 9.

11. The third set of features is predicted based on the generated coordinate information, The second electronic device according to claim 10.

12. The memory is further configured to store a point cloud compression (PCC) encoder. The aforementioned reference frame data is generated based on the application of the PCC encoder to the reference 3D point cloud frameset. The second electronic device according to claim 8.

13. The feature set associated with the 3D points of each reference 3D point cloud frame in the aforementioned reference 3D point cloud frameset is: A reference feature related to the occupancy rate of the 3D points in the corresponding reference 3D point cloud frame of the aforementioned reference 3D point cloud frameset, The reference coordinate information relating to the 3D points of the corresponding reference 3D point cloud frame, The second electronic device according to claim 8, including the following:

14. In the first electronic device, It receives a reference 3D point cloud frameset and a 3D point cloud sequence containing the current 3D point cloud frame to be encoded, To generate reference frame data that includes a feature set associated with the 3D points of each reference 3D point cloud frame in the aforementioned reference 3D point cloud frameset, To generate current frame data, which includes a first set of features relating to the 3D points of the current 3D point cloud frame and relating to the occupancy rate of the 3D points in the current 3D point cloud frame, Based on the application of a first neural network predictor to the aforementioned reference frame data, predict a second set of features associated with the 3D points in the current 3D point cloud frame, Calculating the residual feature set based on the first feature set and the second feature set, A quantized residual feature set is generated based on the application of a quantization scheme to the residual feature set, Based on the application of the encoding scheme to the quantized residual feature set, a bitstream of encoded point cloud data is generated for the current 3D point cloud frame, A method characterized by including the following.

15. The second set of features is predicted based on the coordinate information associated with the 3D points in the current 3D point cloud frame. The method according to claim 14.

16. The method further includes encoding the coordinate information based on the application of an octave-based encoder to the coordinate information, The bitstream of the encoded point cloud data includes the encoded coordinate information. The method according to claim 15.

17. The aforementioned reference frame data is generated based on the application of a first point cloud compression (PCC) encoder to the reference 3D point cloud frameset. The current frame data is generated based on the application of the first PCC encoder to the current 3D point cloud frame. The method according to claim 14.

18. The feature set associated with the 3D points of each reference 3D point cloud frame in the aforementioned reference 3D point cloud frameset is: A reference feature related to the occupancy rate of the 3D points in the corresponding reference 3D point cloud frame of the aforementioned reference 3D point cloud frameset, The reference coordinate information relating to the 3D points of the corresponding reference 3D point cloud frame, The method according to claim 14, including the method described in claim 14.

19. The reference 3D point cloud frameset includes at least one reference 3D point cloud frame preceding the current 3D point cloud frame, or at least one reference 3D point cloud frame succeeding the current 3D point cloud frame. The method according to claim 14.

20. The further includes downsampling the feature set associated with the 3D points of each reference 3D point cloud frame in the reference 3D point cloud frameset by at least one scaling factor. The method according to claim 14.

Images