Multi-scale inter prediction for dynamic point cloud compression

Abstract

An electronic device and method for multi-scale inter prediction for dynamic point cloud compression are provided. The electronic device receives a set of reference point cloud frames and a current point cloud frame. The electronic device generates reference frame data that includes a feature set for each reference point cloud frame and a first set of features for a current point cloud frame. The electronic device predicts a second set of features of the current point cloud frame using the first neural network predictor based on the reference frame data. The electronic device calculates a set of residual features based on the first set of features and the second set of features. The electronic device generates a set of quantized residual features based on the set of residual features and generates a bitstream of encoded point cloud data for the current 3D point cloud frame based on the set of quantized residual features.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS / INCORPORATED BY REFERENCE

[0002] This application claims priority to U.S. Application No. 18 / 471,753 filed with the U.S. Patent and Trademark Office on September 21, 2023, which claims priority to U.S. Provisional Patent Application Serial No. 63 / 380,089 filed on October 19, 2022, the entire contents of which are hereby incorporated by reference into this document. Background Art

[0003] Advances in the field of dynamic point cloud compression (PCC) have led to the development of methods that enable efficient representation of data associated with 3D points of a point cloud. Typically, a point cloud may include a large number of unstructured 3D points. Each 3D point may include geometric information and attribute information (e.g., color, transparency, reflectivity, opacity, texture, material, etc.) associated with the corresponding 3D point. Therefore, each 3D point of the point cloud may include a large amount of data. The point cloud data may need to be compressed (i.e., encoded) using a PCC encoder in order to store, process, or transmit the point cloud. Subsequently, a PCC decoder may reconstruct the point cloud based on the encoded point cloud data received from the PCC encoder. In order to encode the current point cloud frame, the PCC encoder may generate encoded point cloud data based on the current point cloud frame and the previously decoded point cloud frame. The encoded point cloud data may be sent to a PCC decoder, which may reconstruct the corresponding points by decoding the encoded point cloud data. Reconstructing the current point cloud frame based on the encoded point cloud data (generated based on the previously decoded point cloud frame) may be prone to errors and result in artifacts or irregularities in the geometry of the reconstructed point cloud.

[0004] As explained in the remainder of this application and with reference to the accompanying drawings, limitations and disadvantages of conventional traditional approaches will become apparent to those skilled in the art by comparing the described system with some aspects of the present disclosure. Summary of the invention

[0005] As more fully set forth in the claims, there is provided an electronic device and method for multi-scale inter-frame prediction for dynamic point cloud compression substantially as shown in and / or described in conjunction with at least one of the accompanying drawings.

[0006] These and other features and advantages of the present disclosure will be understood by a careful study of the following detailed description of the disclosure and the accompanying drawings, in which like reference numerals refer to like parts throughout. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 is a diagram illustrating an exemplary network environment for multi-scale inter-frame prediction for dynamic point cloud compression according to an embodiment of the present disclosure.

[0008] Figure 2 is a block diagram illustrating an exemplary first electronic device for multi-scale inter-frame prediction for dynamic 3D point cloud frame compression according to an embodiment of the present disclosure.

[0009] Figure 3 is a block diagram illustrating an exemplary second electronic device for multi-scale inter-frame prediction for 3D point cloud frame reconstruction according to an embodiment of the present disclosure.

[0010] Figure 4 is a diagram illustrating an exemplary architecture for multi-scale inter-frame prediction for dynamic 3D point cloud compression and 3D point cloud reconstruction according to an embodiment of the present disclosure.

[0011] Figure 5 is a block diagram illustrating exemplary operations for predicting a set of features associated with a current 3D point cloud frame based on multi-scale features associated with a set of reference 3D point cloud frames according to an embodiment of the present disclosure.

[0012] Figure 6 is a block diagram illustrating exemplary operations for predicting a set of features associated with a current 3D point cloud frame based on multi-scale features associated with a set of reference 3D point cloud frames according to an embodiment of the present disclosure.

[0013] Fig. 7A is a diagram illustrating an exemplary scenario for encoding or decoding a 3D point cloud frame based on a previous 3D point cloud frame according to an embodiment of the present disclosure.

[0014] Figure 7B is a diagram illustrating an exemplary scenario for encoding or decoding a 3D point cloud frame based on two previous 3D point cloud frames according to an embodiment of the present disclosure.

[0015] Figure 8 is a diagram illustrating an exemplary scenario for encoding / decoding a 3D point cloud frame based on a previous 3D point cloud frame and a subsequent 3D point cloud frame according to an embodiment of the present disclosure.

[0016] Fig. 9 is a flow chart illustrating the operation of an exemplary method for multi-scale inter-frame prediction for dynamic 3D point cloud frame compression according to an embodiment of the present disclosure.

[0017] Fig.10 is a flow chart illustrating the operation of an exemplary method for multi-scale inter-frame prediction for 3D point cloud frame reconstruction according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0018] The implementation scheme described below can be found in the disclosed first electronic device, second electronic device and method for multi-scale inter-frame prediction of dynamic point cloud compression. Exemplary aspects of the present disclosure provide a first electronic device (e.g., a computing device, a game console or virtual reality glasses) for predicting features associated with a 3D point cloud frame based on multi-scale features of a reference 3D point cloud frame, and determining the difference between actual features associated with the 3D point cloud frame and predicted features. Specifically, the first electronic device can receive a three-dimensional (3D) point cloud sequence, and the 3D point cloud sequence can include a set of reference 3D point cloud frames and a current 3D point cloud frame to be encoded. After receiving, the first electronic device can generate reference frame data, and the reference frame data can include a feature set associated with 3D points of each reference 3D point cloud frame in the set of reference 3D point cloud frames. The first electronic device can also generate current frame data associated with 3D points of the current 3D point cloud frame. The current frame data can include a first set of features (i.e., a set of actual features) associated with the occupancy of 3D points in the current 3D point cloud frame. The first electronic device can predict a second set of features associated with 3D points of the current 3D point cloud frame. The prediction of the second set of features can be based on applying the first neural network predictor to the reference frame data. Based on the generated first set of features and the predicted second set of features, the first electronic device can calculate a set of residual features. Thereafter, the first electronic device can generate a set of quantized residual features based on applying a quantization scheme to the set of residual features. Finally, the first electronic device can generate a bitstream of encoded point cloud data for the current 3D point cloud frame based on applying a coding scheme to the set of quantized residual features.

[0019] Exemplary aspects of the present disclosure also provide a second electronic device (e.g., a computing device, a game console, virtual reality glasses, or a smart wearable device) for predicting features associated with a current 3D point cloud frame based on multi-scale features of a set of reference 3D point cloud frames. The current 3D point cloud frame can be reconstructed based on the predicted features and the encoded point cloud data. Specifically, the second electronic device can receive a 3D point cloud sequence, which can include a set of reference 3D point cloud frames. After receiving, the second electronic device can generate reference frame data, which includes a feature set associated with 3D points of each reference 3D point cloud frame in the set of reference 3D point cloud frames. Thereafter, the second electronic device can receive (from the first electronic device) a bitstream of the encoded point cloud data associated with the current 3D point cloud frame to be decoded. The second electronic device can predict a third set of features associated with the 3D points of the current 3D point cloud frame based on applying the second neural network predictor to the reference frame data. The second electronic device can generate a fourth set of features associated with the 3D points of the current 3D point cloud frame based on the received bitstream of the encoded point cloud data and the predicted third set of features. The second electronic device may generate (ie, reconstruct) the current 3D point cloud frame based on applying the decoding scheme to the determined fourth set of features.

[0020] Typically, a 3D point cloud can be compressed using a point cloud compression (PCC) technique, and then a decoding technique can be used to reconstruct the 3D point cloud. Encoding may be necessary because each 3D point of a 3D point cloud may include a large amount of point data, and it may not be feasible to send the point cloud data through a communication channel with limited bandwidth. Encoding may involve predicting features of a current point cloud frame based on features of a previously decoded point cloud frame. Based on the predicted features, a bitstream of encoded point cloud data may be generated. The PCC decoder can reconstruct the current point cloud frame based on the bitstream. However, if there is a significant difference between the current point cloud frame and the previous point cloud frame, the reconstructed point cloud frame may include artifacts or surface irregularities. In order to prevent artifacts or irregularities, the difference between the current point cloud frame and the previously decoded point cloud frame can be determined. The difference can be used for motion estimation of objects in the current point cloud frame relative to the previously decoded point cloud frame. Based on the motion estimation, another bitstream of encoded point cloud data can be generated. Since multiple bitstreams are generated, additional bandwidth resources may be required to send multiple bitstreams of encoded point cloud data to the PCC decoder for lossless reconstruction or decoding of each point cloud frame.

[0021] In order to solve such problems, the first electronic device can use multi-scale potential features associated with a set of reference point cloud frames that can be decoded before the current point cloud frame and previously, and perform dynamic point cloud compression based on the prediction of potential features associated with the current point cloud frame. In addition, the second electronic device can determine the multi-scale potential features associated with the set of reference point cloud frames, and can use the determined multi-scale potential features to reconstruct the current point cloud frame. The determination of the multi-scale features by the first electronic device can facilitate the reduction of the size of the bit stream representing the encoded point cloud data associated with the current point cloud frame, thereby reducing the transmission throughput. Since the actual features associated with each point cloud frame can be determined using a PCC encoder on the first electronic device side, and the residual generated based on the actual features and the predicted features can be used to generate the encoded point cloud data, the reduction can be achieved. The residual can represent the difference between the actual feature and the predicted feature. The first electronic device can also use an entropy encoder to compress the residual and send the compressed residual as encoded point cloud data to the second electronic device. On the second electronic device side, a PCC encoder can be used to determine the multi-scale potential features associated with the set of reference point cloud frames. Based on the determined multi-scale potential features associated with the set of reference point cloud frames, potential features associated with the current point cloud frame can be predicted. Thereafter, the received residual can be reconstructed, and the reconstructed residual can be accumulated with the predicted features to determine the actual features associated with the current point cloud frame on the second electronic device side. Based on the determined actual features, the current point cloud frame can be reconstructed using a PCC decoder on the second electronic device side.

[0022] Figure 1 is a diagram illustrating an exemplary network environment for multi-scale inter-frame prediction for dynamic point cloud compression according to an embodiment of the present disclosure. Figure 1 , a network environment 100 is shown. The network environment 100 includes a first electronic device 102, a second electronic device 104, and a server 106. The first electronic device 102 can communicate with the second electronic device 104 and the server 106 through one or more networks (such as a communication network 108). The first electronic device 102 may include a first point cloud compression (PCC) encoder 110, a first neural network predictor 112, and an octree-based encoder 114. The second electronic device 104 may include a second PCC encoder 116, an octree-based decoder 118, a second neural network predictor 120, and a PCC decoder 122. The first electronic device 102 may receive a set of reference 3D point cloud frames 124 and a current 3D point cloud frame 126 as input, and may generate encoded point cloud data 128 associated with the current 3D point cloud frame 126 as output. The second electronic device 104 may receive a set of reference 3D point cloud frames 124 and encoded point cloud data 128 as input, and may generate a decoded 3D point cloud frame 130 as output.

[0023] exist Figure 1 In the embodiment, the first electronic device 102 is responsible for encoding point cloud data (eg, a point cloud sequence), while the second electronic device 104 is responsible for decoding and reconstructing the point cloud data from the compressed point cloud representation shared by the first electronic device 102 .

[0024] In some embodiments, the first electronic device 102 and the second electronic device 104 may be the same device. In such cases, the first electronic device 102 or the second electronic device 104 may be responsible for both encoding and reconstruction of the point cloud data. The first PCC encoder 110 may be the same as the second PCC encoder 116, and the first neural network predictor 112 may be the same as the second neural network predictor 120.

[0025] The first electronic device 102 may include appropriate logic, circuitry, interfaces, and / or code that may be configured to receive a set of reference 3D point cloud frames 124, a current 3D point cloud frame 126, and target coordinate information associated with the current 3D point cloud frame 126. The first electronic device 102 may also determine, via the first PCC encoder 110, features associated with each reference 3D point cloud frame in the set of reference 3D point cloud frames 124, and features associated with the current 3D point cloud frame 126. The first electronic device 102 may also predict, via the first neural network predictor 112, features associated with the current 3D point cloud frame 126 based on features associated with each reference 3D point cloud frame in the set of reference 3D point cloud frames 124. The first electronic device 102 may also be configured to determine a residual based on the features determined by the first PCC encoder 110 and the features predicted by the first neural network predictor 112, and transmit the residual. Examples of the first electronic device 102 may include, but are not limited to, computing devices such as servers, video conferencing systems, augmented reality (AR) devices, virtual reality (VR devices), mixed reality (MR) devices, gaming consoles, servers, smart wearable devices, mainframe machines, computer workstations, and / or consumer electronics (CE) devices.

[0026] The second electronic device 104 may include appropriate logic, circuitry, interfaces and / or code that may be configured to receive a set of reference 3D point cloud frames 124, encoded target coordinate information, and compressed residuals. The second electronic device 104 may determine features associated with each reference 3D point cloud frame in the set of reference 3D point cloud frames 124 via the second PCC encoder 116, and may predict features associated with the current 3D point cloud frame 126 via the second neural network predictor 120. The second electronic device 104 may also reconstruct (i.e., decode) the current 3D point cloud frame 126 via the PCC decoder 122 to obtain a decoded 3D point cloud frame 130. The reconstruction may be based on the features predicted by the second neural network predictor 120 and the residuals. Examples of the second electronic device 104 may include, but are not limited to, computing devices, video conferencing systems, AR devices, VR devices, MR devices, gaming consoles, smart wearable devices, servers, mainframe machines, computer workstations, and / or CE devices.

[0027] The server 106 may include appropriate logic, circuitry, interfaces and / or code that may be configured to generate a set of reference 3D point cloud frames 124 of a 3D object in a 3D space. The server 106 may be configured to generate each 3D reference point cloud frame in the set of reference 3D point cloud frames 124 using the image and depth information of the object. The server 106 may be configured to store a set of reference 3D point cloud frames 124 and information associated with a set of reference 3D point cloud frames 124. The server 106 may also be configured to receive a request for a set of reference 3D point cloud frames 124 from the first electronic device 102 or the second electronic device 104. The server 106 may send a set of reference 3D point cloud frames 124 to the first electronic device 102 or the second electronic device 104 based on the request. In some embodiments, the server 106 may include a first PCC encoder 110, a first neural network predictor 112, an octree-based encoder 114, a second PCC encoder 116, an octree-based decoder 118, a second neural network predictor 120, and a PCC decoder 122.

[0028] The server 106 may perform operations through web applications, cloud applications, HTTP requests, repository operations, file transfers, etc. Example implementations of the server 106 may include, but are not limited to, a database server, a file server, a web server, an application server, a mainframe server, a cloud computing server, or a combination thereof. In at least one embodiment, the server 106 may be implemented as a plurality of distributed cloud-based resources using several technologies known to those of ordinary skill in the art. It will be appreciated by those of ordinary skill in the art that the scope of the present disclosure may not be limited to implementing the server 106 and the first electronic device 102 as two independent entities, implementing the server 106 and the second electronic device 104 as two independent entities, or implementing the server 106, the first electronic device 102, and the second electronic device 104 as three independent entities. In some embodiments, the functions of the server 106 may be incorporated in whole or in part into the first electronic device 102 or the second electronic device 104 without departing from the scope of the present disclosure.

[0029] The communication network 108 may include a communication medium through which the first electronic device 102, the second electronic device 104, and the server 106 can communicate with each other. The communication network 108 may be a wired or wireless communication network. Examples of the communication network 108 may include, but are not limited to, the Internet, a wireless fidelity (Wi-Fi) network, a personal area network (PAN), a local area network (LAN), or a metropolitan area network (MAN). The first electronic device 102 and the second electronic device 104 may be configured to connect to the communication network 108 according to various wired and wireless communication protocols. Examples of such wired and wireless communication protocols may include, but are not limited to, at least one of the transmission control protocol and Internet protocol (TCP / IP), the user datagram protocol (UDP), the hypertext transfer protocol (HTTP), the file transfer protocol (FTP), Zig Bee, EDGE, IEEE 802.11, light fidelity (Li-Fi), 802.16, IEEE 802.11s, IEEE802.11g, multi-hop communication, wireless access point (AP), device-to-device communication, cellular communication protocol, and Bluetooth (BT) communication protocol.

[0030] Each of the first PCC encoder 110 and the second PCC encoder 116 may include suitable logic, circuitry, interfaces and / or code that may be configured to encode each reference 3D point cloud frame in the set of reference 3D point cloud frames 124 to generate a set of features associated with 3D points of a corresponding reference 3D point cloud frame in the set of reference 3D point cloud frames 124. The first PCC encoder 110 may also be configured to encode the current 3D point cloud frame 126 to generate a first set of features associated with the 3D points in the current 3D point cloud frame 126.

[0031] Each of the first PCC encoder 110 and the second PCC encoder 116 can be implemented as a deep neural network (including model files and associated reasoning codes) that can be executed on a graphics processing unit (GPU), a central processing unit (CPU), a tensor processing unit (TPU), a reduced instruction set computing (RISC), an application-specific integrated circuit (ASIC) or a complex instruction set computer (CISC) processor, a coprocessor and / or a combination thereof. In some embodiments, the first PCC encoder 110 can be implemented as a deep neural network on dedicated hardware that interfaces with other computing circuits of the first electronic device 102. Similarly, the second PCC encoder 116 can be implemented as a deep neural network on dedicated hardware that interfaces with other computing circuits of the second electronic device 104. In such an implementation, the first PCC encoder 110 and the second PCC encoder 116 can be associated with a specific form factor on a specific computing circuit. Examples of specific computing circuits may include, but are not limited to, field programmable gate arrays (FPGAs), programmable logic devices (PLDs), ASICs, programmable ASICs (PL-ASICs), application-specific integrated components (ASSPs), and systems on chips (SOCs) based on standard microprocessors (MPUs) or digital signal processors (DSPs). According to an embodiment, the first PCC encoder 110 or the second PCC encoder 116 may also interface with a GPU to parallelize the operations of the first PCC encoder 110 or the second PCC encoder 116, respectively.

[0032] Each of the first neural network predictor 112 and the second neural network predictor 120 may be referred to as a neural network as a computational network or system of artificial neurons that may generally be arranged in a plurality of layers. The neural network may be defined by its hyperparameters, such as activation functions, the number of weights, cost functions, regularization functions, input sizes, the number of layers, etc. In addition, the layers may include an input layer, one or more hidden layers, and an output layer. Each of the plurality of layers may include one or more nodes (or artificial neurons). The outputs of all nodes in the input layer may be coupled to at least one node in the hidden layer. Similarly, the inputs of each hidden layer may be coupled to the outputs of at least one node in the other layers of the neural network. The outputs of each hidden layer may be coupled to the inputs of at least one node in the other layers of the neural network. The nodes in the final layer may receive inputs from at least one hidden layer to output results. The number of layers and the number of nodes in each layer may be determined according to the hyperparameters of the neural network. Such hyperparameters may be set before or after training of the neural network.

[0033] Each node can correspond to a mathematical function (e.g., a sigmoid function or a rectified linear unit) with parameters that are adjustable during the training of the neural network. The set of parameters may include, for example, weight parameters, regularization parameters, etc. Each node can use the mathematical function to calculate an output based on one or more inputs from nodes in other layers of the neural network (e.g., the previous (one or more) layers). All or some nodes of the neural network may correspond to the same or different mathematical functions. In the training of the neural network, one or more parameters of each node of the neural network can be updated based on whether the output of the final layer for a given input (from the training data set) matches the correct result according to the loss function of the neural network. The above process can be repeated for the same or different inputs until the minimum value of the loss function is reached and the training error is minimized. Several training methods are known in the art, for example, gradient descent, stochastic gradient descent, batch gradient descent, gradient boosting, meta-heuristics, etc.

[0034] Each of the first neural network predictor 112 and the second neural network predictor 120 may be a machine learning model trained to generate multi-scale features associated with an input 3D point cloud frame. Each of the first neural network predictor 112 and the second neural network predictor 120 may receive as input a feature set associated with 3D points of each reference 3D point cloud frame in a set of reference 3D point cloud frames 124 and target coordinate information associated with the current 3D point cloud frame 126. Each of the first neural network predictor 112 and the second neural network predictor 120 may generate a prediction as an output in response to the input. The prediction may indicate features associated with the current 3D point cloud frame 126. The first neural network predictor 112 may predict a second set of features associated with the current 3D point cloud frame 126 based on applying the first neural network predictor 112 to the feature set (generated by the first PCC encoder 110) and the target coordinate information. The second neural network predictor 120 may predict a third set of features associated with the current 3D point cloud frame 126 based on applying the second neural network predictor 120 to the feature set (generated by the second PCC encoder 116) and the target coordinate information (decoded by the octree-based decoder 118).

[0035] In some embodiments, each of the first neural network predictor 112 and the second neural network predictor 120 may include electronic data that may be implemented as a software component of an application executable on the first electronic device 102 and the second electronic device 104. Each of the first neural network predictor 112 and the second neural network predictor 120 may rely on a library, an external script, or logic / instructions to be executed by a processing device included in the first electronic device 102 and the second electronic device 104. In one or more embodiments, each of the first neural network predictor 112 and the second neural network predictor 120 may be implemented using hardware, which may include a processor, a microprocessor (e.g., for performing one or more operations or controlling the performance of one or more operations), an FPGA, or an ASIC. Alternatively, in some embodiments, each of the first neural network predictor 112 and the second neural network predictor 120 may be implemented using a combination of hardware and software. Examples of the first neural network predictor 112 and the second neural network predictor 120 may include, but are not limited to, a deep neural network (DNN), a convolutional neural network (CNN), an artificial neural network (ANN), a fully connected neural network, and / or a combination of such networks.

[0036] The octree-based encoder 114 may include appropriate logic, circuits, interfaces and / or codes that can be configured to encode data based on the use of a data structure (i.e., an octree). The octree-based encoder 114 may recursively partition a 3D space (such as a portion of the current 3D point cloud frame 126, which includes a 3D point located at a target coordinate specified in the target coordinate information) into smaller areas (such as blocks) called octants. The partitioning may continue until each block meets the es stopping criterion (the points within each block represent similar density or texture). In an embodiment, the octree-based encoder 114 may encode the point cloud data and partition hierarchy information associated with the 3D point located at the target coordinate to generate encoded target coordinate information.

[0037] In some embodiments, the octree-based encoder 114 may be a machine-learning-based octree encoder that may use techniques such as deep octree encoding to encode (compress) 3D points at target coordinates of the current 3D point cloud frame 126. Examples of such machine-learning-based octree encoders may include, but are not limited to, G-PCC, OctSqueeze, and VoxelContext-Net.

[0038] The octree-based decoder 118 may include suitable logic, circuitry, interfaces and / or code that may be configured to decode the encoded target coordinate information. At each level of the octree (based on which the encoded target coordinate information may be decoded), an octant may be reconstructed until the entire volume (i.e., the portion of the current 3D point cloud frame 126 that includes the 3D points located at the target coordinates) is reconstructed. Based on the octant, the reconstructed decoded target coordinate information may be obtained.

[0039] In some embodiments, the octree-based encoder 114 may be a machine learning-based octree encoder that may use deep octree decoding techniques to decode the encoded object coordinate information associated with the current 3D point cloud frame 126. Examples of such machine learning-based octree decoders may include, but are not limited to, G-PCC, OctSqueeze, and VoxelContext-Net.

[0040] The PCC decoder 122 may include appropriate logic, circuits, and / or interfaces that may be configured to reconstruct the current 3D point cloud frame 126 based on the encoded point cloud data 128. The PCC decoder 122 may receive a fourth set of features as input. The fourth set of features may be generated based on the third set of features and the decompressed residual. Based on applying the PCC decoder 122 to the fourth set of features, a decoded 3D point cloud frame 130 may be generated. The PCC decoder 122 may be implemented as a deep neural network on a GPU, a CPU, a TPU, a RISC processor, an ASIC processor, a CISC processor, a coprocessor, and / or a combination thereof. In some other embodiments, the PCC decoder 122 may be implemented as a deep neural network on dedicated hardware that interfaces with other computing circuits of the second electronic device 104. In such an implementation, the PCC decoder 122 may be associated with a specific form factor on a specific computing circuit. Examples of specific computing circuits may include, but are not limited to, FPGAs, PLDs, ASICs, PL-ASICs, ASSPs, and SOCs based on standard MPUs or DSPs. In some embodiments, PCC decoder 122 may interface with a GPU to parallelize the operation of PCC decoder 122 .

[0041] Each reference 3D point cloud frame in a set of reference 3D point cloud frames 124 and the current 3D point cloud frame 126 may correspond to a geometric representation of one or more 3D objects in a 3D environment (e.g., a real-world environment). Each 3D point cloud frame may constitute a set of 3D points located at different positions according to a 3D coordinate system. According to an embodiment, the first electronic device 102 may obtain a set of reference 3D point cloud frames 124 and the current 3D point cloud frame 126 from the server 106. Similarly, the second electronic device 104 may obtain a set of reference 3D point cloud frames 124 from the server 106. Each 3D point in the reference 3D point cloud frame may include attribute information and geometric information associated with the corresponding 3D point (i.e., the coordinates of the corresponding 3D point in the corresponding reference 3D point cloud frame). The attribute information may include, for example, color information, reflectivity information, opacity information, normal vector information, material identifier information, or texture information.

[0042] The encoded point cloud data 128 may be generated based on the encoding of each reference 3D point cloud frame in a set of reference 3D point cloud frames 124 and the current 3D point cloud frame 126. In addition, the encoded point cloud data 128 may also be generated based on the prediction of the multi-scale features associated with the current 3D point cloud frame 126 and the encoding of the target coordinate information associated with the current 3D point cloud frame 126. The encoded point cloud data 128 may include a compressed residual (generated based on the generated first set of features and the predicted second set of features) and the encoded target coordinate information. The first electronic device 102 may send the encoded point cloud data 128 as a bitstream to the second electronic device 104.

[0043] The decoded 3D point cloud frame 130 may be a reconstructed point cloud frame that may correspond to the current 3D point cloud frame 126. The second electronic device 104 may reconstruct the current 3D point cloud frame 126 via the PCC decoder 122 based on the encoded point cloud data 128. The reconstruction may be based on the decompression of the compressed residual and the accumulation of the decompressed residual (i.e., the originally generated residual) and the predicted third set of features (i.e., the fourth set of features). The decoded 3D point cloud frame 130 may be generated based on applying the PCC decoder 122 to the accumulation.

[0044] In operation, the first electronic device 102 may be configured to receive a 3D point cloud sequence, which may include a set of reference 3D point cloud frames 124 and a current 3D point cloud frame to be encoded 126. According to an embodiment, each reference 3D point cloud frame in the set of reference 3D point cloud frames 124 may be a previously decoded 3D point cloud frame, and may be before (i.e., received earlier than) or after the current 3D point cloud frame 126 in the reception timeline of the 3D point cloud sequence.

[0045] The first electronic device 102 may also be configured to generate reference frame data including a feature set associated with 3D points of each reference 3D point cloud frame in a set of reference 3D point cloud frames 124. Generation of the reference frame data may be based on applying the first PCC encoder 110 to each reference 3D point cloud frame in a set of reference 3D point cloud frames 124. The reference frame data may be generated as an output of the first PCC encoder 110. The feature set associated with the 3D points of each reference 3D point cloud frame may include reference features associated with the occupancy of the 3D points in a corresponding reference 3D point cloud frame in a set of reference 3D point cloud frames 124, and reference coordinate information associated with the 3D points of the corresponding reference 3D point cloud frame. The reference coordinate information may, for example, include the coordinates of the 3D points in the corresponding reference 3D point cloud frame.

[0046] The first electronic device 102 may also be configured to generate current frame data associated with the 3D points of the current 3D point cloud frame 126. The generation of the current frame data may be based on applying the first PCC encoder 110 to the current 3D point cloud frame 126. The generated current frame data may include a first set of features associated with the occupancy of the 3D points in the current 3D point cloud frame 126. The first set of features may be generated as an output of the first PCC encoder 110. The first set of features may be referred to as actual features associated with the 3D points of the current 3D point cloud frame 126. According to an embodiment, the first set of features may include features associated with the occupancy of those 3D points whose coordinates are included in the target coordinate information.

[0047] The first electronic device 102 may also be configured to predict a second set of features associated with 3D points of the current 3D point cloud frame 126 based on applying the first neural network predictor 112 to the reference frame data. The feature set associated with the 3D points of each reference 3D point cloud frame in the set of reference 3D point cloud frames 124 may be fed as input to the first neural network predictor 112. It should be noted that each feature set may be associated with those 3D points of the corresponding reference 3D point cloud frame whose coordinates may be included in the reference coordinate information. The feature set may include reference features associated with the occupancy of the 3D points in the corresponding reference 3D point cloud frame.

[0048] According to an embodiment, the first electronic device 102 may provide coordinate information associated with 3D points of the current 3D point cloud frame 126 (i.e., target coordinate information) as input to the first neural network predictor 112. The 3D points may be those 3D points whose features may need to be predicted, and the coordinates of the 3D points may be included in the target coordinate information. The target coordinate information may include the target coordinates included in the current 3D point cloud frame 126. Based on applying the first neural network predictor 112 to the input, a second set of features may be predicted as an output of the first neural network estimator 112. The second set of features may include features associated with the occupancy of the 3D points whose coordinates are included in the target coordinate information.

[0049] According to an embodiment, the feature set associated with each reference 3D point cloud frame may be downsampled at scales of 2, 3, ..., and K for prediction. For example, the feature set associated with the first reference 3D point cloud frame in a set of reference 3D point cloud frames 124 may be downsampled at scales of 2, 3, ..., and K. Therefore, (K-1) feature sets may be generated for the first reference 3D point cloud frame. Similarly, (K-1) feature sets may be generated for each other reference 3D point cloud frame in a set of reference 3D point cloud frames 124. Thereafter, a spatiotemporal tensor may be constructed for each downsampled scale based on the feature sets of all reference 3D point cloud frames downsampled at the same scale. Since the feature set associated with each reference 3D point cloud frame is downsampled at scales of 2, 3, ..., and K, (K-1) spatiotemporal tensors may be constructed. For example, using the feature set associated with all reference 3D point cloud frames downsampled at a scale of 2, a first spatiotemporal sensor may be constructed for a downsampled scale of "2". Similarly, using the feature sets associated with all reference 3D point cloud frames 124 downsampled at a scale of "K", the (K-1)th spatiotemporal sensor can be constructed for the downsampled scale "K". Based on the spatiotemporal concatenation of the feature sets of all reference 3D point cloud frames that can be downsampled at the corresponding downsampled scale, the construction of the spatiotemporal sensor for the corresponding downsampled scale can be performed. Thereafter, spatiotemporal tensor analysis can be performed based on applying sparse convolution or self-attention operations to each of the (K-1) spatiotemporal tensors. After the spatiotemporal tensor analysis, each of the downsampled (K-1) spatiotemporal tensors (except the spatiotemporal tensor constructed based on the spatiotemporal concatenation of the feature sets downsampled at the scale of K) can be further downsampled. Thereafter, the (K-1) downsampled spatiotemporal tensors can be spliced ​​to generate a multi-scale feature splicing vector.

[0050] Based on the target coordinate information and the multi-scale feature stitching vector associated with the 3D points of the current 3D point cloud frame 126, a second set of features associated with the 3D points of the current 3D point cloud frame 126 may be predicted as an output of the first neural network predictor 112. The second set of features may include features associated with those 3D points whose coordinates may be included in the target coordinate information.

[0051] The first electronic device 102 may also be configured to calculate a set of residual features based on the first set of features (i.e., current frame data associated with the 3D points of the current 3D point cloud frame 126) and the predicted second set of features. According to an embodiment, the first electronic device 102 may calculate the difference between the first set of features and the predicted second set of features. The calculated difference may indicate an error associated with the second set of features (i.e., features associated with the 3D points of the current 3D point cloud frame 126 predicted based on the reference frame data) relative to the first set of features (i.e., actual features associated with the 3D points of the current 3D point cloud frame 126). In addition, the difference may correspond to a set of residual features associated with the 3D points of the current 3D point cloud frame 126 whose coordinates may be included in the target coordinate information.

[0052] The first electronic device 102 may also be configured to generate a set of quantized residual features based on applying a quantization scheme to the set of residual features. The quantization scheme may be based on an entropy model and may include a set of quantization levels. According to an embodiment, the value of each residual feature in the set of residual features may be quantized at a certain quantization level in the set of quantization levels. The set of residual features may be quantized so that the set of residual features may be subsequently compressed and encoded.

[0053] The first electronic device 102 may also be configured to generate a bitstream of encoded point cloud data 128 for the current 3D point cloud frame 126 based on applying a coding scheme to the generated set of quantized residual features. The coding scheme may be based on an entropy model (the quantization scheme may be based on the entropy model). Based on the coding scheme, each quantized residual feature in the set of quantized residual features may be compressed to generate the encoded point cloud data 128. The bitstream of the encoded point cloud data 128 may constitute a set of compressed quantized residual features.

[0054] According to an embodiment, the encoded point cloud data 128 may also include encoded target coordinate information. The target coordinate information may be encoded based on applying the octree-based encoder 114 to the target coordinate information. The first electronic device 102 may send a bit stream of the encoded point cloud data 128 to the second electronic device 104.

[0055] According to an embodiment, the second electronic device 104 may be configured to receive the encoded point cloud data 128 (from the first electronic device 102) and receive a 3D point cloud sequence that may include a set of reference 3D point cloud frames 124. The second electronic device 104 may be configured to extract a set of compressed quantized residual features and encoded target coordinate information from the encoded point cloud data 128 (i.e., the bitstream generated by the first electronic device 102). Thereafter, the second electronic device 104 may apply a decoding scheme to the set of compressed quantized residual features to decompress the set of compressed quantized residual features. The decoding scheme may generate a set of residual features that may correspond to a set of residual features calculated (by the first electronic device 102) based on the first set of features and the predicted second set of features. The second electronic device 104 may apply an octree-based decoder 118 to the encoded target coordinate information to generate the target coordinate information as an output of the octree-based decoder 118.

[0056] The second electronic device 104 may also be configured to generate reference frame data, which may include a feature set associated with the 3D points of each reference 3D point cloud frame in a set of reference 3D point cloud frames 124. The feature set associated with each reference 3D point cloud frame may be generated based on applying the second PCC encoder 116 to the corresponding reference 3D point cloud frame. The feature set may be generated as an output of the second PCC encoder 116. The features associated with each reference 3D point cloud frame may include reference features associated with the occupancy of the 3D points of the corresponding reference 3D point cloud frame and reference coordinate information (which includes the coordinates of the 3D points of the 3D points of the corresponding reference 3D point cloud frame).

[0057] The second electronic device 104 may also be configured to predict a third set of features associated with the 3D points of the current 3D point cloud frame 126 based on applying the second neural network predictor 120 to the reference frame data. The third set of features may be associated with those 3D points whose coordinates may be included in the target coordinate information decoded by the octree-based decoder 118. According to an embodiment, the second neural network predictor 120 may receive as input a feature set associated with the 3D points of each reference 3D point cloud frame in a set of reference 3D point cloud frames 124 (i.e., the output of the second PCC encoder 116) and the target coordinate information (i.e., the output of the octree-based decoder 118). Based on applying the second neural network predictor 120 to the input, the third set of features may be predicted as the output of the second neural network predictor 120. If the first neural network predictor 112 and the second neural network predictor 120 are the same, the prediction of the third set of features (performed by the second neural network predictor 120) may be the same as the prediction of the second set of features (performed by the first neural network predictor 112). Furthermore, if the first PCC encoder 110 and the second PCC encoder 116 are the same, and the decoded target coordinate information (generated by the octree-based decoder 118) matches the target coordinate information encoded by the octree-based encoder 114, then the second set of features and the third set of features may be the same.

[0058] The second electronic device 104 may also be configured to generate a fourth set of features associated with the 3D points of the current 3D point cloud frame 126 (whose coordinates are included in the target coordinate information) based on the received bitstream of the encoded point cloud data 128 (i.e., a set of residual features that can be generated by the decoding scheme) and the predicted third set of features. According to an embodiment, the set of residual features and the predicted third set of features may be accumulated to generate a fourth set of features. The 3D points of the current 3D point cloud frame 126 may be reconstructed based on applying the PCC decoder 122 to the fourth set of features. The PCC decoder 122 may generate a decoded 3D point cloud frame 130 as an output. The decoded 3D point cloud frame 130 may be a reconstructed version of the current 3D point cloud frame 126.

[0059] Figure 2 is a block diagram illustrating an exemplary first electronic device for multi-scale inter-frame prediction for dynamic 3D point cloud frame compression according to an embodiment of the present disclosure. Figure 1 To illustrate the elements Figure 2 . refer to Figure 2, a block diagram 200 of a first electronic device 102 is shown. The first electronic device 102 may include circuitry 202, memory 204, input / output (I / O) devices 206, and a network interface 208. In at least one embodiment, the memory 204 may include a first PCC encoder 110, a first neural network predictor 112, and an octree-based encoder 114. In at least one embodiment, the I / O device 206 may include a display device 210. The circuitry 202 may be communicatively coupled to the memory 204, the I / O device 206, and the network interface 208 via wired or wireless communication of the first electronic device 102.

[0060] The circuit 202 may include appropriate logic, circuits, and interfaces that may be configured to execute program instructions associated with different operations to be performed by the first electronic device 102. The circuit 202 may include one or more processing units that may be implemented as an integrated processor or a processor cluster that collectively performs the functions of one or more dedicated processing units. The circuit 202 may be implemented based on a variety of processor technologies known in the art. Example implementations of the circuit 202 may be x86-based processors, GPUs, CPUs, RISC processors, ASIC processors, CISC processors, microcontrollers, and / or other computing circuits.

[0061] The memory 204 may include appropriate logic, circuitry, and / or interfaces that may be configured to store instructions that may be executed by the circuit 202. The memory 204 may be configured to store an operating system and associated applications. The memory 204 may also be configured to store a 3D point cloud sequence, generated reference frame data, generated current frame data, coordinate information associated with 3D points of the current 3D point cloud frame 126, a second set of features, a set of residual features, a set of quantized residual features, a bitstream of the encoded point cloud data 128, etc. In at least one embodiment, the first PCC encoder 110, the first neural network predictor 112, and the octree-based encoder 114 included in the memory 204 may be implemented as a combination of programmable instructions stored in the memory 204 or logic units (i.e., programmable logic units) on the hardware circuit of the first electronic device 102. Examples of implementations of the memory 204 may include, but are not limited to, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), a hard disk drive (HDD), a solid-state drive (SSD), a CPU cache, and / or a secure digital (SD) card.

[0062] The I / O device 206 may include suitable logic, circuitry, interfaces, and / or code that may be configured to receive user input that may trigger the reception of a 3D point cloud sequence or the transmission of a bitstream of encoded point cloud data 128. The I / O device 206 may also be configured to provide output in response to the user input. The I / O device 206 may include various input and output devices that may be configured to communicate with the circuit 202. Examples of input devices may include, but are not limited to, a touch screen, a keyboard, a mouse, a joystick, and / or a microphone. Examples of output devices may include a display device 210.

[0063] The display device 210 may include appropriate logic, circuitry, interfaces and / or code that may be configured to render each reference 3D point cloud frame in a set of reference 3D point cloud frames contained in a 3D point cloud sequence onto a display screen of the display device 210. According to an embodiment, the display device 210 may include a touch screen that receives user input. The display device 210 may be implemented by several known technologies, such as, but not limited to, a liquid crystal display (LCD) display, a light emitting diode (LED) display, a plasma display, and / or an organic LED (OLED) display technology, and / or other display technologies. According to an embodiment, the display device 210 may refer to a display screen, a 3D display, a perspective display, a projection-based display, an electrochromic display and / or a transparent display of a smart glasses device.

[0064] The network interface 208 may include suitable logic, circuitry, interfaces and / or code that may be configured to establish communication between the first electronic device 102, the second electronic device 104 and the server 106 via the communication network 108. The network interface 208 may be implemented using various known technologies to support wired or wireless communication between the first electronic device 102 and the communication network 108. The network interface 208 may include, but is not limited to, an antenna, a radio frequency (RF) transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a codec (CODEC) chipset, a subscriber identity module (SIM) card and / or a local buffer.

[0065] The network interface 208 can communicate with a network, such as the Internet, an intranet, and / or a wireless network such as a cellular telephone network, a wireless local area network (LAN), and / or a metropolitan area network (MAN) via wireless communication. The wireless communication can use any of a variety of communication standards, protocols, and technologies, such as Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (W-CDMA), Long Term Evolution (LTE), Fifth Generation (5G) New Radio (NR), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (such as IEEE802.11a, IEEE 802.11b, IEEE 802.11g, and / or IEEE 802.11n), Voice over Internet Protocol (VoIP), Light Fidelity (Li-Fi), Wi-MAX, email protocols, instant messaging, and / or short message service (SMS).

[0066] like Figure 1 The functions or operations performed by the first electronic device 102 shown in FIG. 1 may be performed by the circuit 202. The operations performed by the circuit 202 are, for example, Figure 3 , Figure 4 , Figure 5 , Figure 6 , Fig. 7A , Figure 7B and Figure 8 Described in detail in.

[0067] Figure 3 is a block diagram illustrating an exemplary second electronic device for multi-scale inter-frame prediction for 3D point cloud frame reconstruction according to an embodiment of the present disclosure. Figure 1 To illustrate the elements Figure 3 . refer to Figure 3 , a block diagram 300 of a second electronic device 104 is shown. The second electronic device 104 may include a circuit 302, a memory 204, an I / O device 306, and a network interface 308. In an embodiment, the memory 304 may include a second PCC encoder 116, an octree-based decoder 118, a second neural network predictor 120, and a PCC decoder 122. In at least one embodiment, the I / O device 306 may include a display device 310. The circuit 302 may be communicatively coupled to the memory 304, the I / O device 306, and the network interface 308 via wired or wireless communication of the second electronic device 104.

[0068] The circuit 302 may include appropriate logic, circuits, and interfaces that may be configured to execute program instructions associated with different operations to be performed by the second electronic device 104. The circuit 302 may include one or more processing units that may be implemented as an integrated processor or processor cluster that collectively performs the functions of one or more dedicated processing units. The circuit 302 may be implemented based on a variety of processor technologies known in the art. Examples of implementations of the circuit 302 may be x86-based processors, GPUs, RISC processors, ASIC processors, CISC processors, microcontrollers, CPUs, and / or other computing circuits.

[0069] The memory 304 may include appropriate logic, circuitry, and / or interfaces that may be configured to store instructions that may be executed by the circuit 302. The memory 304 may be configured to store an operating system and associated applications. The memory 304 may also be configured to store a 3D point cloud sequence, reference frame data, decoded coordinate information associated with 3D points of a current 3D point cloud frame to be decoded, a predicted third set of features, a decompressed set of residual features, and a decoded 3D point cloud frame 130. In at least one embodiment, the second PCC encoder 116, the octree-based decoder 118, the second neural network predictor 120, and the PCC decoder 122 included in the memory 304 are implemented as a combination of programmable instructions stored in the memory 304 or logic units (i.e., programmable logic units) on the hardware circuit of the second electronic device 104. Examples of implementations of the memory 304 may include, but are not limited to, RAM, ROM, EEPROM, HDD, SSD, CPU cache, and / or SD card.

[0070] The I / O device 306 may include appropriate logic, circuitry, interfaces, and / or code that may be configured to receive user input that may trigger the receipt of a bitstream of a 3D point cloud sequence and encoded point cloud data 128. The I / O device 306 may also be configured to provide output in response to the user input. The I / O device 306 may include various input and output devices that may be configured to communicate with the circuit 302. Examples of input devices may include, but are not limited to, a touch screen, a keyboard, a mouse, a joystick, and / or a microphone. Examples of output devices may include a display device 310.

[0071] The display device 310 may include appropriate logic, circuits, interfaces and / or codes that can be configured to render each reference 3D point cloud frame in a set of reference 3D point cloud frames and the decoded 3D point cloud frame 130 onto a display screen of the display device 310. According to an embodiment, the display device 310 may include a touch screen that receives user input. The display device 310 may be implemented by several known technologies, such as but not limited to LCD display, LED display, plasma display, and / or OLED display technology, and / or other display technology. According to an embodiment, the display device 310 may refer to a display screen, a 3D display, a perspective display, a projection-based display, an electrochromic display and / or a transparent display of a smart glasses device.

[0072] The network interface 308 may include suitable logic, circuitry, interfaces and / or code that may be configured to establish communication between the first electronic device 102, the second electronic device 104 and the server 106 via the communication network 108. The network interface 308 may be implemented using various known technologies to support wired or wireless communication between the second electronic device 104 and the communication network 108. The network interface 308 may include, but is not limited to, an antenna, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a CODEC chipset, a SIM card and / or a local buffer.

[0073] The network interface 308 can communicate with a network, such as the Internet, an intranet, and / or a wireless network such as a cellular telephone network, a wireless LAN, and / or a MAN, via wireless communication. The wireless communication can use any of a variety of communication standards, protocols, and technologies, such as GSM, EDGE, W-CDMA, LTE, 5G NR, CDMA, TDMA, Bluetooth, Wi-Fi (such as IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, and / or IEEE 802.11n), VoIP, Li-Fi, Wi-MAX, email protocols, instant messaging, and / or SMS.

[0074] like Figure 1 The functions or operations performed by the second electronic device 104 shown in FIG. 3 may be performed by the circuit 302. The operations performed by the circuit 302 are, for example, Figure 4 , Figure 5 , Figure 6 , Fig. 7A , Figure 7B and Figure 8 Described in detail in.

[0075] Figure 4is a diagram illustrating an exemplary architecture for multi-scale inter-frame prediction for dynamic 3D point cloud compression and 3D point cloud reconstruction according to an embodiment of the present disclosure. Figure 1 , Figure 2 and Figure 3 To illustrate the elements Figure 4 . refer to Figure 4 , an exemplary architecture 400 for dynamic 3D point cloud compression and 3D point cloud reconstruction is shown. In the architecture 400, a first PCC encoder 110, a first neural network predictor 112, an octree-based encoder 114, a second PCC encoder 116, an octree-based decoder 118, a second neural network predictor 120 and a PCC decoder 122, a subtractor 402, a quantizer 404, an autoencoder 406, an autodecoder 408, and an accumulator 410 are shown.

[0076] At any time, the first PCC encoder 110 can receive a set of reference 3D point cloud frames and a current 3D point cloud frame to be encoded, i.e., P(t). The set of reference 3D point cloud frames can include "N" 3D point cloud frames, i.e., P(t-1), P(t-2), ... and P(tN). The set of reference 3D point cloud frames, i.e., P(t-1), P(t-2), ... or P(tN) can be before or after the current 3D point cloud frame. In some cases, such frames can be referred to as previously decoded 3D point cloud frames (i.e., frames decoded before receiving the current 3D point cloud frame). The set of reference 3D point cloud frames and the current 3D point cloud frame can be received as input (by the first PCC encoder 110). The first PCC encoder 110 can generate reference frame data and current frame data as corresponding outputs. The reference frame data may include a set of features associated with 3D points of each reference 3D point cloud frame in the set of reference 3D point cloud frames. For example, the output feature set associated with the 3D points of the input reference 3D point cloud frame P(t-1) may be F(t-1). Similarly, the output feature set associated with the 3D points of the input reference 3D point cloud frame P(tN) may be F(tN). The feature set (e.g., F(t-1)) associated with the 3D points of each reference 3D point cloud frame (e.g., P(t-1)) in the set of reference 3D point cloud frames may include reference features associated with the occupancy of the 3D points in the corresponding reference 3D point cloud frame, and reference coordinate information associated with the coordinates of the 3D points in the corresponding reference 3D point cloud frame. The current frame data may be associated with the 3D points of the current 3D point cloud frame (i.e., P(t)) and may include a first set of features, i.e., F(t). The first set of features may be associated with the occupancy of the 3D points in the current 3D point cloud frame and may be generated as an output of the first PCC encoder 110. According to an embodiment, the 3D points may be points of the current 3D point cloud frame that may need to be encoded.

[0077] The first neural network predictor 112 may receive as input a feature set associated with a 3D point of each reference 3D point cloud frame in the set of reference 3D point cloud frames (i.e., F(t-1)...F(tN)). The feature set associated with the 3D point of each reference 3D point cloud frame in the set of reference 3D point cloud frames may be downsampled by at least one scaling factor. According to an embodiment, the feature set associated with the 3D point of each reference 3D point cloud frame (whose coordinates may be included in the reference coordinate information) may be downsampled by scaling factors such as 2, 3,..., and K. For example, F(t-1) may be downsampled by scaling factors of 2, 3,..., and K to generate multi-scale feature sets F, respectively. 2 (t-1), F 3 (t-1), … and F K (t-1). Similarly, F(tN) can be downsampled to generate a multi-scale feature set F 2 (tN), F 3 (tN), ... and F K (tN).

[0078] For each scaling factor (i.e., 2, 3, ... or K), the first neural network predictor 112 may generate a spatiotemporal tensor using a feature set associated with all reference 3D point cloud frames in the set of reference 3D point cloud frames. For example, for a scaling factor of "2", the spatiotemporal tensor may be generated based on F 2 (t-1), … and F 2 (tN) generates a space-time tensor. Similarly, for the scaling factor “K”, it can be based on F K (t-1), … and F K (tN) Generate a spatiotemporal tensor. According to an embodiment, the generation of a spatiotemporal tensor for a certain scaling factor may be based on the spatiotemporal concatenation of feature sets associated with all reference 3D point cloud frames in the set of reference 3D point cloud frames downsampled by the same scaling factor. For example, the spatiotemporal tensor for scaling factor "2" may be generated based on F 2 (t-1), … and F 2 Similarly, the space-time tensor for the scaling factor “K” can be constructed based on F K (t-1), … and F K (tN) is constructed by spatiotemporal concatenation of (tN). Therefore, (K-1) spatiotemporal tensors can be constructed for scaling factors 2, 3, ... and K. Each spatiotemporal tensor (of the (K-1) spatiotemporal tensors) for a certain scaling factor can represent the features of all reference 3D point cloud frames that can be downsampled according to the scaling factor.

[0079] The first neural network predictor 112 may also receive (target) coordinate information (i.e., C(t)) associated with a 3D point of a current 3D point cloud frame (i.e., P(t)). C(t) may indicate that the current 3D point cloud frame may include coordinates of 3D points whose features may need to be predicted (or encoded) based on a feature set associated with a 3D point of each reference 3D point cloud frame (whose coordinates of the 3D point may be included in the reference coordinate information). The 3D point of each reference 3D point cloud frame whose coordinates may be associated with the reference coordinate information may correspond to a 3D point of the current 3D point cloud frame whose coordinates may be included in C(t). In some embodiments, a spatiotemporal tensor for each scaling factor may be constructed based on a feature set associated with all reference 3D point cloud frames downsampled by the corresponding scaling factor and a current feature set associated with a 3D point of the current 3D point cloud frame. The coordinates of the 3D point may be included in C(t), and for all scaling factors, the features in the current feature set associated with the 3D point may be filled with "1" (because the features in the current feature set are to be predicted).

[0080] After constructing the spatiotemporal tensor, each of the (K-1) spatiotemporal tensors may be subjected to spatiotemporal analysis. According to an embodiment, the spatiotemporal analysis may include performing a sparse convolution operation or a self-attention operation on each of the spatiotemporal tensors (i.e., the (K-1) spatiotemporal tensors). The sparse convolution operation on the spatiotemporal tensor may be performed based on a filter. The filter may weight the features represented by the spatiotemporal tensor in different spatial regions with the same weight value. The sparse convolution operation may modify each feature based on a corresponding feature, a set of neighboring features of the corresponding feature, and a weight value for weighting the corresponding feature and each neighboring feature. The count of neighboring features may be based on the size of the filter. The modification of each feature may result in the generation of a modified spatiotemporal tensor. Therefore, the sparse operation on the (K-1) spatiotemporal tensors may result in the generation of (K-1) modified spatiotemporal tensors. Similarly, the self-attention operation on the spatiotemporal tensor (of the (K-1) spatiotemporal tensors) may modify the features represented by the spatiotemporal tensor. The modification may be based on a filter of a predefined size that may weight the features represented by the spatiotemporal tensor. The filter may modify each feature based on the corresponding feature, the weight of the corresponding feature, the neighboring features of the corresponding feature, and the weight of each neighboring feature. The modification of the feature may result in the generation of a modified spatiotemporal tensor. Thus, based on applying the self-attention network to the (K-1) spatiotemporal tensors, (K-1) modified spatiotemporal tensors may be generated.

[0081] The features represented by each modified spatiotemporal tensor (generated based on sparse convolution or self-attention) can be downsampled by a certain scaling factor. The scaling factor by which the modified spatiotemporal tensor may have to be downsampled can be based on the scaling factor (such as 2, 3, ... or K) of the original version for which the modified spatiotemporal tensor can be constructed, and the highest scaling factor (i.e., K) by which the feature set of each reference 3D point cloud frame in the set of reference 3D point cloud frames is downsampled. Thereafter, an initial (inception) residual network can be applied to each downsampled modified spatiotemporal tensor to generate a final spatiotemporal tensor. Therefore, (K-1) final spatiotemporal tensors can be generated as outputs of the initial residual network.

[0082] The (K-1) final spatiotemporal tensors may be concatenated to generate a multi-scale feature concatenation vector. A sparse convolution operation or a self-attention operation may be performed on the multi-scale feature concatenation vector. The first neural network predictor 112 may predict a second set of features (i.e., F ) associated with a 3D point (whose coordinates are included in C (t)) of the current 3D point cloud frame (i.e., P (t)) based on the result of the sparse convolution operation or the self-attention operation. ～ (t)). Therefore, the second set of features can be further predicted based on the coordinate information associated with the 3D point of the current 3D point cloud frame (ie, C(t)).

[0083] Subtractor 402 may receive a first set of features (ie, F(t)) and a second set of features (ie, F ～ (t) as input. A set of residual features (i.e., R(t)) may be calculated based on the first set of features and the second set of features. The set of residual features may indicate an actual set of features (i.e., F(t) generated by the first PCC encoder 110) and a predicted set of features (i.e., F(t) generated by the first neural network predictor 112). ～ (t)). This difference can be used to compensate for any errors that may have resulted in the prediction of features associated with the current 3D point cloud frame (i.e., P(t)). The predicted features can be used to reconstruct the current 3D point cloud frame (i.e., the 3D points of P(t) whose coordinates are contained in C(t)).

[0084] The quantizer 404 may receive the set of residual features (i.e., R(t)) and may quantize each of the residual features in the set of residual features at a quantization level in a set of predefined quantization levels. The quantizer 404 may generate a set of quantized residual features as output based on applying a quantization scheme to the set of residual features. The quantization scheme may be based on an entropy model. The autoencoder 406 may receive the set of quantized residual features as input and may generate a bitstream of encoded point cloud data of the current 3D point cloud frame as output. The generation may be based on applying a coding scheme to the set of quantized residual features. The coding scheme may be based on an entropy model and may compress each of the quantized residual features in the set of quantized residual features to encode the set of quantized residual features.

[0085] According to an embodiment, the octree-based encoder 114 may receive the coordinate information (i.e., C(t)) as input, and may encode the coordinate information to generate encoded coordinate information. The circuit 202 may include the encoded coordinate information in a bit stream of the generated encoded point cloud data, and may send the generated bit stream to the second electronic device 104.

[0086] Circuit 302 may receive the generated bitstream and may extract the encoded coordinate information and the encoded point cloud data from the bitstream. Octree-based decoder 118 may receive the encoded coordinate information as input and may decode the encoded coordinate information to regenerate coordinate information (i.e., C(t)). The coordinate information may be associated with a 3D point of a current 3D point cloud frame (i.e., P(t)) to be decoded. Based on the regenerated coordinate information, the target coordinates in the current 3D point cloud frame containing the 3D point to be decoded may be determined.

[0087] The auto-decoder 408 may receive the encoded point cloud data as input and may regenerate a set of residual features (i.e., R ～ (t)) as output. The regeneration may be based on applying a decoding scheme to the encoded point cloud data. The decoding scheme may be based on an entropy model and may decompress the set of quantized residual features. The set of residual features (i.e., R ～ (t)) can be used to reconstruct the 3D points (whose coordinates are contained in C(t)) of the current 3D point cloud frame (ie, P(t)).

[0088] The second PCC encoder 116 can receive a 3D point cloud sequence as input, which can include a set of reference 3D point cloud frames, namely, P(t-1), P(t-2), ... and P(tN). Based on these inputs, reference frame data can be generated as output, and the reference frame data includes a feature set associated with the 3D points of each reference 3D point cloud frame in the set of reference 3D point cloud frames. The reference frame data can be generated based on applying the second PCC encoder 116 to the set of reference 3D point cloud frames. For example, the feature set associated with the 3D points of the reference 3D point cloud frame P(t-1) can be F(t-1). Similarly, the feature set associated with the 3D points of the reference 3D point cloud frame P(tN) can be F(tN). The feature set associated with the 3D point of each reference 3D point cloud frame in the set of reference 3D point cloud frames may include a reference feature associated with the occupancy of the 3D point in the corresponding reference 3D point cloud frame in the set of reference 3D point cloud frames, and reference coordinate information associated with the 3D point of the corresponding reference 3D point cloud frame. The 3D point may correspond to a 3D point of the current 3D point cloud frame (i.e., P(t)) whose coordinates may be included in the regenerated coordinate information (i.e., C(t)).

[0089] The second neural network predictor 120 may receive as input a feature set associated with a 3D point of each reference 3D point cloud frame in the set of reference 3D point cloud frames. Thereafter, the second neural network predictor 120 may predict a third set of features (i.e., F ) associated with a 3D point of a current 3D point cloud frame (i.e., P(t)). ’ (t)) as output. The 3D points of P(t) may need to be decoded and the coordinates of the 3D points may be included in C(t). The prediction may be based on applying the second neural network predictor 120 to the reference frame data, i.e., the feature set associated with the 3D points of each reference 3D point cloud frame. A third set of features may also be predicted based on the (re)generated coordinate information (i.e., C(t)). The generation of the third set of features may be the same as the generation of the second set of features performed by the first neural network predictor 112.

[0090] The accumulator 410 may receive a third set of features (i.e., F ’ (t)) and a set of regenerated residual features (i.e., R ～ (t)) as input. The accumulator 410 can generate a fourth set of features (i.e., F" (t) associated with the 3D points of the current 3D point cloud frame (i.e., P(t)) (i.e., whose coordinates can be contained in C(t)) as output. The generation can be based on the accumulation of the third set of features and the regenerated set of residual features. The PCC decoder 122 can receive the fourth set of features as input and can reconstruct the current 3D point cloud frame (i.e., P(t)) based on applying the decoding scheme to the fourth set of features. The PCC decoder 122 can generate a 3D point cloud frame (i.e., P’ (t)) as output (which may correspond to P(t)).

[0091] Figure 5 is a block diagram illustrating exemplary operations for predicting a set of features associated with a current 3D point cloud frame based on multi-scale features associated with a set of reference 3D point cloud frames according to an embodiment of the present disclosure. Figure 1 , Figure 2 , Figure 3 and Figure 4 To illustrate the elements Figure 5 . refer to Figure 5 , an exemplary block diagram 500 for predicting a set of features associated with a current 3D point cloud frame is shown. The exemplary block diagram 500 may include a series of blocks that may represent a sequence of operations that may start at 502 and end at 508. The sequence of operations may be performed by the first neural network predictor 112 or the second neural network predictor 120. The first neural network predictor 112 or the second neural network predictor 120 may receive two feature sets, namely, F(t-1) and F(t-2), as input. F(t-1) may be associated with a 3D point of a reference 3D point cloud frame P(t-1). F(t-2) may be associated with a 3D point of a reference 3D point cloud frame P(t-2). Both P(t-2) and P(t-1) may precede the current 3D point cloud frame P(t). P(t) will be encoded or decoded based on the prediction of the features associated with the 3D points of P(t).

[0092] F(t-1) may include reference features associated with the occupancy of 3D points in P(t-1) and reference coordinate information associated with the 3D points of P(t-1). For example, F(t-1) may include reference features f associated with four 3D points of P(t-1). 11 、f 13 、f 14 and f 15 And the reference coordinates of these four 3D points. On the other hand, F(t-2) may include reference features associated with the occupancy of 3D points in P(t-2) and reference coordinate information associated with the 3D points of P(t-2). For example, F(t-2) may include reference features associated with five 3D points of P(t-2). 23 、f 24 、f 25 、f 26 and f 28 And the reference coordinates of these 5 3D points.

[0093] A feature set associated with the 3D point of P(t) may be constructed based on the (target) coordinate information C(t). C(t) may include the coordinates of the 3D point of P(t). The first neural network predictor 112 or the second neural network predictor 120 may receive C(t) as input. For example, C(t) may include the coordinates of 5 3D points. The coordinates may be c 33 、c 34 、c 35 、c 36 and c 39 The feature set may include features associated with the five 3D points of P(t). According to an embodiment, the feature set may be constructed by filling the features associated with the five 3D points with "1". The constructed feature set may be represented as F(t), i.e., [1 33 ,1 34 ,1 35 ,1 36 ,1 39 ]). The first neural network predictor 112 or the second neural network predictor 120 may predict a feature set (i.e., F(t-2), F(t-1), and F(t)) associated with five 3D points of P(t) (whose coordinates may be included in C(t)) based on downsampling of each of F(t-2), F(t-1), and F(t) by scaling factors "2" and "3". ～ (t))(For example, [f 33 ,f 34 ,f 35 ,f 36 ,f 39 ]) to encode or decode P(t).

[0094] The reference features in F(t-2) (i.e., f 11 、f 13 、f 14 and f 15 ) can be downsampled by scaling factors of "2" and "3" to generate F 2 (t-2) and F 3 (t-2). Similarly, the reference feature in F(t-1) (i.e., f 23 、f 24 、f 25 、f 26 and f 28 ) can be downsampled by scaling factors of "2" and "3" to generate F 2 (t-1) and F 3 (t-1). Since F(t) is constructed by filling the features of the 3D points of P(t) with "1", downsampling F(t) by scaling factors "2" and "3" can return F(t).

[0095] At 502A, F(t), F2 (t-1) and F 2 (t-2) can be spliced ​​in space and time. This spatiotemporal splicing can result in the generation of a first spatiotemporal tensor. The first spatiotemporal tensor can be constructed for a scaling factor of "2" and can be generated based on the features of all reference 3D point cloud frames (i.e., P(t-2) and P(t-1)) that have been downsampled by a scaling factor of "2". For example, the features represented by the first spatiotemporal tensor may include feature vectors such as [f11,0,0], [0,0,0], [f13,f23,1], [f14,f24,1], [f15,f25,1], [0,f26,1], [0,0,0], [0,f28,0], and [0,0,1]. Here, [0,0,0] indicates an empty area.

[0096] At 502B, F(t), F 3 (t-1) and F 3 (t-2) can be spliced ​​in space and time. The spatiotemporal splicing can result in the generation of a second spatiotemporal tensor. The second spatiotemporal tensor can be constructed for a scaling factor of "3" and can be generated based on the features of all reference 3D point cloud frames (i.e., P(t-2) and P(t-1)) that have been downsampled by the scaling factor "3".

[0097] At 504A, a spatiotemporal analysis can be performed on the first spatiotemporal tensor. The spatiotemporal analysis can be based on a self-attention operation and can result in modifications to features represented by the first spatiotemporal tensor. The modification of each feature can be based on one or more of a size of a filter used to perform the self-attention operation, a weight assigned to the corresponding feature, and a weight assigned to each neighboring feature of the corresponding feature. Based on the modification of each feature represented by the first spatiotemporal tensor, a modified first spatiotemporal tensor can be generated.

[0098] At 504B, the second spatiotemporal tensor may be subjected to spatiotemporal analysis. The spatiotemporal analysis may be based on a self-attention operation, which may result in a modification of each feature represented by the second spatiotemporal tensor. Based on the modification of each feature, a modified second spatiotemporal tensor may be generated.

[0099] At 506 , a multi-scale feature stitching vector may be generated based on the spatiotemporal stitching of the modified first spatiotemporal tensor and the modified second spatiotemporal tensor.

[0100] At 508, features associated with the 3D points of the current 3D point cloud frame (i.e., P(t)) may be predicted. The prediction may be based on applying a sparse convolution operation or a self-attention operation to the generated multi-scale feature concatenation vector. The coordinates of the 3D points of the current 3D point cloud frame (whose features are predicted) may be included in C(t). Thus, the prediction may be further based on the coordinate information (i.e., C(t)). The predicted features may include f 33 、f 34 、f 35 、f 36 and f 39 Based on such predicted features, the 3D point (i.e. c 33 、c 34 、c 35 、c 36 and c 39 ) for encoding or decoding.

[0101] Figure 6 is a block diagram illustrating exemplary operations for predicting a set of features associated with a current 3D point cloud frame based on multi-scale features associated with a set of reference 3D point cloud frames according to an embodiment of the present disclosure. Figure 1 , Figure 2 , Figure 3 , Figure 4 and Figure 5 To illustrate the elements Figure 6 . refer to Figure 6 , an exemplary block diagram 600 for predicting a set of features associated with a current 3D point cloud frame (i.e., P(t)) is shown. Block diagram 600 may include a series of blocks that may represent a sequence of operations that may start at 602 and end at 608. The sequence of operations may be performed by the first neural network predictor 112 or the second neural network predictor 120. The first neural network predictor 112 or the second neural network predictor 120 may receive two feature sets, i.e., F(t-1) and F(t-2), as input. The first neural network predictor 112 or the second neural network predictor 120 may predict a feature set (i.e., F) associated with a 3D point cloud frame of P(t) whose coordinates may be contained in C(t) based on downsampling of each of F(t-2) and F(t-1) by scaling factors “2” and “3”. ～ (t)). It can be predicted that F ～ (t) to encode or decode P(t).

[0102] At 602A, F 2 (t-1) and F 2(t-2) can be spliced ​​in space and time. The spatiotemporal splicing can result in the generation of a first spatiotemporal tensor. The first spatiotemporal tensor can be constructed for a scaling factor of "2" and can be generated based on the features of all reference 3D point cloud frames (i.e., P(t-2) and P(t-1)) that have been downsampled by a scaling factor of "2". For example, the features represented by the first spatiotemporal tensor may include features such as [f 11 ,0],[0,0],[f 13 ,f 23 ]、[f 14 ,f 24 ]、[f 15 ,f 25 ]、[0,f 26 ]、[0,0]、[0,f 28 ] and [0,0]. Here, [0,0] indicates an empty region.

[0103] At 602B, F 3 (t-1) and F 3 (t-2) can be spliced ​​in space and time. The spatiotemporal splicing can result in the generation of a second spatiotemporal tensor. The second spatiotemporal tensor can be constructed for a scaling factor of "3", and therefore can be generated based on the features of all reference 3D point cloud frames (i.e., P(t-2) and P(t-1)) that have been downsampled by a scaling factor of "3".

[0104] At 604A, the first spatiotemporal tensor may be subjected to spatiotemporal analysis. In an embodiment, the spatiotemporal analysis may be based on a sparse convolution operation and may result in modification of each feature represented by the first spatiotemporal tensor. The modification of each feature may be based on the size of the filter used to perform the sparse convolution operation, and one or more of the weights assigned to each corresponding feature and the neighboring features of the corresponding feature. Based on the modification of each feature represented by the first spatiotemporal tensor, a modified first spatiotemporal tensor may be generated. Thereafter, the modified first spatiotemporal tensor may be downsampled to generate a downsampled modified first spatiotemporal tensor. The downsampled first spatiotemporal tensor may be passed as input to a first initial residual network, which may generate a final spatiotemporal tensor as output.

[0105] At 604B, the second spatiotemporal tensor may be subjected to spatiotemporal analysis. The spatiotemporal analysis may be based on a sparse convolution operation and may result in a modification of each feature represented by the second spatiotemporal tensor. Based on the modification of each feature, a modified second spatiotemporal tensor may be generated. The modified second spatiotemporal tensor may be passed as an input to the first initial residual network, which may generate a final spatiotemporal tensor as an output.

[0106] At 606, a multi-scale feature concatenation vector may be generated based on the spatiotemporal concatenation of the final spatiotemporal tensor generated by the first initial residual network and the final spatiotemporal tensor generated by the second initial residual network.

[0107] At 608, features associated with 3D points of the current 3D point cloud frame (i.e., P(t)) may be predicted. The prediction may be based on applying a sparse convolution operation or a self-attention operation to the generated multi-scale feature concatenation vector. The coordinates of the 3D points of the current 3D point cloud frame (whose features are predicted) may be included in C(t). Thus, the prediction may be further based on the coordinate information (i.e., C(t)). Based on the predicted features, the 3D points of P(t) may be encoded or decoded.

[0108] Fig. 7A is a diagram illustrating an exemplary scenario for encoding or decoding a 3D point cloud frame based on a previous 3D point cloud frame according to an embodiment of the present disclosure. Figure 1 , Figure 2 , Figure 3 , Figure 4 , Figure 5 and Figure 6 To illustrate the elements Fig. 7A . refer to Fig. 7A , an exemplary scene 700A is shown. In the exemplary scene 700A, an exemplary sequence of 3D point cloud frames is shown, which may include an "I" 3D point cloud frame and a group of "P" 3D point cloud frames. The "I" 3D point cloud frame can be independently encoded / decoded, while each "P" 3D point cloud frame can be encoded / decoded based on the previous 3D point cloud frame. For example, at time t0, the "I" 3D point cloud frame can be encoded / decoded. At time t1, a "P" 3D point cloud frame in the group of "P" 3D point cloud frames can be encoded / decoded based on predicted features associated with the occupancy of the 3D points of the "I" 3D point cloud frame. At time t2, another "P" 3D point cloud frame in the group of "P" 3D point cloud frames can be encoded / decoded based on predicted features associated with the occupancy of the 3D points of the "P" 3D point cloud frame encoded / decoded at time t1. Similarly, other “P” 3D point cloud frames in the group of “P” 3D point cloud frames may be encoded / decoded.

[0109] Figure 7B is a diagram illustrating an exemplary scenario for encoding or decoding a 3D point cloud frame based on two previous 3D point cloud frames according to an embodiment of the present disclosure. Figure 1 , Figure 2 , Figure 3 , Figure 4 , Figure 5 , Figure 6 and Fig. 7A To illustrate the elements Figure 7B . refer to Figure 7B , an exemplary scene 700B is shown. In the exemplary scene 700B, an exemplary sequence of 3D point cloud frames is shown, which may include two "I" 3D point cloud frames and a group of "P" 3D point cloud frames. The "I" 3D point cloud frames can be independently encoded / decoded, while each "P" 3D point cloud frame can be encoded / decoded based on two previous 3D point cloud frames.

[0110] For example, at time t0, the first "I" 3D point cloud frame may be encoded or decoded, and at time t1, the second "I" 3D point cloud frame may be encoded or decoded. At time t2, one "P" 3D point cloud frame in the group of "P" 3D point cloud frames may be encoded / decoded based on the predicted features associated with the occupancy of the 3D points of the first "I" 3D point cloud frame and the predicted features associated with the occupancy of the 3D points of the second "I" 3D point cloud frame. At time t3, another "P" 3D point cloud frame in the group of "P" 3D point cloud frames may be encoded / decoded based on the predicted features associated with the occupancy of the 3D points of the second "I" 3D point cloud frame (encoded / decoded at time t1) and the predicted features associated with the occupancy of the 3D points of the "P" 3D point cloud frame encoded / decoded at time t2. At time t4, another "P" 3D point cloud frame in the group of "P" 3D point cloud frames may be encoded / decoded based on the predicted features associated with the occupancy of the 3D points of the "P" 3D point cloud frame encoded / decoded at time t2 and the predicted features associated with the occupancy of the 3D points of the "P" 3D point cloud frame encoded / encoded at time t2. Similarly, other "P" 3D point cloud frames in the group of "P" 3D point cloud frames may be encoded / decoded.

[0111] Figure 8 is a diagram illustrating an exemplary scenario for encoding / decoding a 3D point cloud frame based on a previous 3D point cloud frame and a subsequent 3D point cloud frame according to an embodiment of the present disclosure. Figure 1 , Figure 2 , Figure 3 , Figure 4 , Figure 5 , Figure 6 , Fig. 7A and Figure 7B To illustrate the elements Figure 8 . refer to Figure 8, an exemplary scene 800 is shown. In the exemplary scene 800, an exemplary sequence of 3D point cloud frames is shown, which may include two "I" 3D point cloud frames, three "P" 3D point cloud frames, and three "B" 3D point cloud frames. The "I" 3D point cloud frames can be independently encoded / decoded. Each "P" 3D point cloud frame can be encoded / decoded based on a previous 3D point cloud frame, such as an "I" 3D point cloud frame or a P" 3D point cloud frame. Each "B" 3D point cloud frame can be encoded / decoded based on a previous 3D point cloud frame (such as an I" 3D point cloud frame or a P" 3D point cloud frame) and a subsequent 3D point cloud frame (such as a P" 3D point cloud frame).

[0112] For example, at time t0, the first "I" 3D point cloud frame (i.e., the first frame with frame index 0) can be encoded / decoded, and at time t1, the second "I" 3D point cloud frame (i.e., the second frame with frame index 1) can be encoded / decoded. At time t2, the first "P" 3D point cloud frame (i.e., the fourth frame with frame index 3) can be encoded or decoded based on the predicted features associated with the occupancy of the 3D points of the second "I" 3D point cloud frame. Since the first "P" 3D point cloud frame is the third frame to be encoded / decoded (after the encoding / decoding of the first "I" 3D point cloud frame and the second "I" 3D point cloud frame), the encoding / decoding order of the first "P" 3D point cloud frame can be "2". At time t3, the first "B" 3D point cloud frame (i.e., the third frame with frame index 2) can be encoded or decoded based on the predicted features associated with the occupancy of the 3D points of the second "I" 3D point cloud frame (i.e., the previous frame) and the predicted features associated with the occupancy of the 3D points of the first "P" 3D point cloud frame (i.e., the subsequent frame). The encoding / decoding order of the first "B" 3D point cloud frame can be "3" (i.e., the fourth frame to be encoded or decoded). At time t4, the second "P" 3D point cloud frame (i.e., the sixth frame with frame index 5) can be encoded / decoded based on the predicted features associated with the occupancy of the 3D points of the first "P" 3D point cloud frame. The encoding / decoding order of the second "P" 3D point cloud frame can be "4" (i.e., the fifth frame to be encoded or decoded). At time t5, the second "B" 3D point cloud frame (i.e., the fifth frame with frame index 4) can be encoded or decoded based on the predicted features associated with the occupancy of the 3D points of the first "P" 3D point cloud frame (i.e., the previous frame) and the predicted features associated with the occupancy of the 3D points of the second "P" 3D point cloud frame (i.e., the subsequent frame). The encoding / decoding order of the second "B" 3D point cloud frame can be "5" (i.e., the sixth frame to be encoded or decoded). Similarly, the third "P" 3D point cloud frame (i.e., the eighth frame with frame index 7) and the third "B" 3D point cloud frame (i.e., the seventh frame with frame index 6) can be encoded / decoded.

[0113] Fig. 9 is a flowchart illustrating the operation of an exemplary method for multi-scale inter-frame prediction for dynamic 3D point cloud frame compression according to an embodiment of the present disclosure. Figure 1 , Figure 2 , Figure 3 , Figure 4 , Figure 5 , Figure 6 , Fig. 7A , Figure 7B and Figure 8 To illustrate the elements Fig. 9 . refer to Fig. 9 , a flowchart 900 is shown. The operations from 902 to 916 can be implemented by any computing system, such as the first electronic device 102 or the circuit 202 of the first electronic device 102. The operations can start at 902 and proceed to 904.

[0114] At 904, a 3D point cloud sequence may be received that may include a set of reference 3D point cloud frames and a current 3D point cloud frame to be encoded. In at least one embodiment, circuit 202 may be configured to receive a 3D point cloud sequence that may include a set of reference 3D point cloud frames and a current 3D point cloud frame to be encoded. For example, Figure 1 and Figure 4 Details of receiving a set of reference 3D point cloud frames and a current 3D point cloud frame are described in .

[0115] At 906, reference frame data may be generated, the reference frame data including a feature set associated with the 3D points of each reference 3D point cloud frame in the set of reference 3D point cloud frames. In at least one embodiment, circuit 202 may be configured to generate reference frame data including a feature set associated with the 3D points of each reference 3D point cloud frame in the set of reference 3D point cloud frames. For example, Figure 1 and Figure 4 The details of the generation of reference frame data are described in .

[0116] At 908, current frame data associated with the 3D points of the current 3D point cloud frame may be generated. In at least one embodiment, circuit 202 may be configured to generate current frame data associated with the 3D points of the current 3D point cloud frame. The current frame data may include a first set of features associated with the occupancy of the 3D points in the current 3D point cloud frame. For example, in Figure 1 and Figure 4 The details of the generation of the current frame data are described in .

[0117] At 910, a second set of features associated with the 3D points of the current 3D point cloud frame may be predicted based on applying the first neural network predictor to the reference frame data. In at least one embodiment, circuit 202 may be configured to predict a second set of features associated with the 3D points of the current 3D point cloud frame based on applying the first neural network predictor to the reference frame data. For example, Figure 1 , Figure 4 , Figure 5 and Figure 6 The details of the prediction of the second set of features are described in .

[0118] At 912, a set of residual features may be calculated based on the first set of features and the second set of features. In at least one embodiment, circuit 202 may be configured to calculate a set of residual features based on the first set of features and the second set of features. Figure 1 and Figure 4 The details of the computation of a set of residual features are described in .

[0119] At 914, a set of quantized residual features may be generated based on applying the quantization scheme to the set of residual features. In at least one embodiment, circuit 202 may be configured to generate a set of quantized residual features based on applying the quantization scheme to the set of residual features. For example, Figure 1 and Figure 4 The details of the generation of a set of quantized residual features are described in .

[0120] At 916, a bitstream of encoded point cloud data may be generated for the current 3D point cloud frame based on applying the encoding scheme to the set of quantized residual features. In at least one embodiment, circuit 202 may be configured to generate a bitstream of encoded point cloud data for the current 3D point cloud frame based on applying the encoding scheme to the set of quantized residual features. For example, Figure 1 and Figure 4 Details of the generation of the bitstream encoding the point cloud data are described in . Control may be transferred to end.

[0121] Although flowchart 900 is illustrated as discrete operations, such as 904, 906, 908, 910, 912, 914, and 916, the present disclosure is not limited thereto. Thus, in certain embodiments, such discrete operations may be further divided into additional operations, combined into fewer operations, or eliminated, depending on the implementation, without compromising the essence of the disclosed embodiments.

[0122] Fig.10 is a flowchart illustrating the operation of an exemplary method for multi-scale inter-frame prediction for 3D point cloud frame reconstruction according to an embodiment of the present disclosure. Figure 1 , Figure 2 , Figure 3 , Figure 4 , Figure 5 , Figure 6 , Fig. 7A , Figure 7B , Figure 8 and Fig. 9 elements to illustrate Fig.10 . refer to Fig.10 , a flowchart 1000 is shown. The operations from 1002 to 1014 can be implemented by any computing system, such as by the second electronic device 104 or the circuit 302 of the second electronic device 104. The operations can start at 1002 and can proceed to 1004.

[0123] At 1004, a 3D point cloud sequence may be received that may include a set of reference 3D point cloud frames. In at least one embodiment, circuit 302 may be configured to receive a 3D point cloud sequence that may include a set of reference 3D point cloud frames. For example, Figure 1 and Figure 4 The details of receiving the 3D point cloud sequence are described in.

[0124] At 1006, reference frame data may be generated, the reference frame data including a feature set associated with the 3D points of each reference 3D point cloud frame in the set of reference 3D point cloud frames. In at least one embodiment, circuit 302 may be configured to generate reference frame data including a feature set associated with the 3D points of each reference 3D point cloud frame in the set of reference 3D point cloud frames. For example, Figure 1 and Figure 4 The details of the generation of reference frame data are described in .

[0125] At 1008, a bitstream of encoded point cloud data associated with the current 3D point cloud frame to be decoded may be received. In at least one embodiment, circuit 302 may be configured to receive a bitstream of encoded point cloud data associated with the current 3D point cloud frame to be decoded. For example, Figure 1 and Figure 4 The details of receiving the bitstream are described in .

[0126] At 1010, a third set of features associated with the 3D points of the current 3D point cloud frame may be predicted based on applying the second neural network predictor to the reference frame data. In at least one embodiment, circuit 302 may be configured to predict a third set of features associated with the 3D points of the current 3D point cloud frame based on applying the second neural network predictor to the reference frame data. For example, Figure 1 , Figure 4 , Figure 5 and Figure 6 The details of the prediction of the third set of features are described in .

[0127] At 1012, a fourth set of features associated with the 3D points of the current 3D point cloud frame may be generated based on the received bitstream of the encoded point cloud data and the predicted third set of features. In at least one embodiment, circuit 302 may be configured to generate a fourth set of features associated with the 3D points of the current 3D point cloud frame based on the received bitstream of the encoded point cloud data and the predicted third set of features. For example, Figure 1 and Figure 4 The details of the generation of the fourth set of features are described in .

[0128] At 1014, the current 3D point cloud frame may be reconstructed based on applying the decoding scheme to the determined fourth set of features. In at least one embodiment, the circuit 302 may be configured to reconstruct the current 3D point cloud frame based on applying the decoding scheme to the determined fourth set of features. Figure 1 and Figure 4 The details of the reconstruction of the current 3D point cloud frame are described in. Control can be transferred to end.

[0129] Although flowchart 1000 is illustrated as discrete operations, such as 1004, 1006, 1008, 1010, 1012, and 1014, the present disclosure is not limited thereto. Thus, in certain embodiments, depending on the implementation, such discrete operations may be further divided into additional operations, combined into fewer operations, or eliminated without compromising the essence of the disclosed embodiments.

[0130] Exemplary aspects of the present disclosure may include an electronic device (such as Figure 1 The first electronic device 102 may include a circuit (e.g. Figure 2 The electronic device may be communicatively coupled to another electronic device (e.g. Figure 1 The first electronic device 102 may also include a memory (such as Figure 2 204), which may be configured to store a predictor (e.g. Figure 1 The memory 204 may be configured to store the PCC encoder (e.g. Figure 1The circuit 202 may be configured to receive a 3D point cloud sequence, which may include a set of reference 3D point cloud frames and a current 3D point cloud frame to be encoded. The set of reference 3D point cloud frames may include at least one reference 3D point cloud frame that may be before the current 3D point cloud frame, or at least one reference 3D point cloud frame that may be after the current 3D point cloud frame. The circuit 202 may also be configured to generate reference frame data, which includes a feature set associated with 3D points of each reference 3D point cloud frame in the set of reference 3D point cloud frames. The reference frame data may be generated based on applying the first PCC encoder 110 to the set of reference 3D point cloud frames. The feature set associated with the 3D points of each reference 3D point cloud frame in the set of reference 3D point cloud frames may include reference features associated with the occupancy of 3D points in corresponding reference 3D point cloud frames in the set of reference 3D point cloud frames, and reference coordinate information associated with the 3D points of the corresponding reference 3D point cloud frame. The circuit 202 may also be configured to generate current frame data associated with 3D points of the current 3D point cloud frame. The current frame data may be generated based on applying the first PCC encoder 110 to the current 3D point cloud frame. The current frame data may include a first set of features associated with the occupancy of 3D points in the current 3D point cloud frame. The circuit 202 may also be configured to predict a second set of features associated with the 3D points of the current 3D point cloud frame based on applying the first neural network predictor 112 to the reference frame data. The second set of features may also be predicted based on coordinate information associated with the 3D points of the current 3D point cloud frame. The circuit 202 may also be configured to calculate a set of residual features based on the first set of features and the second set of features. The circuit 202 may also be configured to generate a set of quantized residual features based on applying a quantization scheme to the set of residual features. The circuit 202 may also be configured to generate a bitstream of encoded point cloud data for the current 3D point cloud frame based on applying an encoding scheme to the set of quantized residual features.

[0131] According to an embodiment, the circuit 202 may be further configured to encode the coordinate information based on applying an octree-based encoder, such as the octree-based encoder 114, to the coordinate information. The bitstream encoding the point cloud data may include the encoded coordinate information.

[0132] According to an embodiment, the circuit 202 may be further configured to downsample the feature set associated with the 3D points of each reference 3D point cloud frame in the set of reference 3D point cloud frames by at least one scaling factor.

[0133] Exemplary aspects of the present disclosure may include an electronic device (such as Figure 1 The second electronic device 104 may include a circuit (such as Figure 3302), the electronic device can be communicatively coupled to another electronic device (such as Figure 1 The second electronic device 104 may also include a memory (such as Figure 3 304), which may be configured to store a predictor (e.g. Figure 1 The memory 204 may be configured to store the PCC encoder (e.g. Figure 1 The second PCC encoder 116) and the PCC decoder (such as Figure 1 PCC decoder 122 of the set of reference 3D point cloud frames). Circuit 302 may be configured to receive a 3D point cloud sequence that may include a set of reference 3D point cloud frames. Circuit 302 may also be configured to generate reference frame data, the reference frame data including a feature set associated with 3D points of each reference 3D point cloud frame in the set of reference 3D point cloud frames. The reference frame data may be generated based on applying the second PCC encoder 116 to the set of reference 3D point cloud frames. The feature set associated with the 3D points of each reference 3D point cloud frame in the set of reference 3D point cloud frames may include reference features associated with the occupancy of the 3D points in the corresponding reference 3D point cloud frame in the set of reference 3D point cloud frames, and reference coordinate information associated with the 3D points of the corresponding reference 3D point cloud frame. Circuit 302 may also be configured to receive a bitstream of encoded point cloud data associated with the current 3D point cloud frame to be decoded. The received bitstream of encoded point cloud data may also include encoded coordinate information associated with the 3D points of the current 3D point cloud frame. The circuit 302 may also be configured to predict a third set of features associated with the 3D points of the current 3D point cloud frame based on applying the second neural network predictor 120 to the reference frame data. The circuit 302 may also be configured to generate a fourth set of features associated with the 3D points of the current 3D point cloud frame based on the received bitstream of the encoded point cloud data and the predicted third set of features. The circuit 302 may also be configured to reconstruct the current 3D point cloud frame based on applying the decoding scheme to the determined fourth set of features.

[0134] According to an embodiment, the circuit 302 may be further configured to generate coordinate information associated with the 3D points of the current 3D point cloud frame based on applying an octree-based decoder (such as the octree-based decoder 118) to the encoded coordinate information. The third set of features may also be predicted based on the generated coordinate information.

[0135] The present disclosure may be implemented in hardware or in a combination of hardware and software. The present disclosure may be implemented in a centralized manner in at least one computer system, or in a distributed manner in which different elements may be distributed on several interconnected computer systems. A computer system or other device suitable for performing the methods described herein may be suitable. The combination of hardware and software may be a general-purpose computer system with a computer program that, when loaded and executed, can control the computer system so that it performs the methods described herein. The present disclosure may be implemented in hardware including a portion of an integrated circuit that also performs other functions.

[0136] The present disclosure may also be embedded in a computer program product comprising all the features enabling the implementation of the methods described herein and capable of carrying out these methods when loaded into a computer system. In this context, a computer program means any expression in any language, code or symbol of a set of instructions intended to cause a system with information processing capabilities to perform a specific function directly or after one or both of the following: a) conversion into other languages, codes or symbols; b) reproduction in a different material form.

[0137] Although the present disclosure is described with reference to certain embodiments, it will be appreciated by those skilled in the art that various changes may be made without departing from the scope of the present disclosure, and equivalents may be substituted. In addition, many modifications may be made to adapt specific circumstances or materials to the teachings of the present disclosure without departing from its scope. Therefore, the present disclosure is not intended to be limited to the disclosed embodiments, but is intended to include all embodiments falling within the scope of the appended claims.

Claims

1. A first electronic device, comprising: a memory configured to store a first neural network predictor; as well as A circuit, the circuit being configured as: receiving a three-dimensional (3D) point cloud sequence, the three-dimensional (3D) point cloud sequence comprising a set of reference 3D point cloud frames and a current 3D point cloud frame to be encoded; generating reference frame data comprising a set of features associated with 3D points of each reference 3D point cloud frame in the set of reference 3D point cloud frames; Generate current frame data associated with the 3D points of the current 3D point cloud frame, wherein the current frame data includes a first set of features associated with occupancy of 3D points in the current 3D point cloud frame; predicting a second set of features associated with 3D points of a current 3D point cloud frame based on applying a first neural network predictor to the reference frame data; Calculate a set of residual features based on the first set of features and the second set of features; generating a set of quantized residual features based on applying a quantization scheme to the set of residual features; as well as Based on applying the encoding scheme to the set of quantized residual features, a bitstream of encoded point cloud data is generated for the current 3D point cloud frame. 2 . The first electronic device according to claim 1 , wherein the second set of features is also predicted based on coordinate information associated with the 3D points of the current 3D point cloud frame.

3. The first electronic device of claim 2, wherein the circuit is further configured to encode the coordinate information based on applying an octree-based encoder to the coordinate information, The bit stream of the encoded point cloud data includes encoded coordinate information.

4. The first electronic device of claim 1, wherein the memory is further configured to store a first point cloud compression (PCC) encoder, wherein the reference frame data is generated based on applying a first PCC encoder to the set of reference 3D point cloud frames, and The current frame data is generated based on applying a first PCC encoder to a current 3D point cloud frame.

5. The first electronic device of claim 1 , wherein the feature set associated with the 3D points of each reference 3D point cloud frame in the set of reference 3D point cloud frames comprises: a reference feature associated with an occupancy of a 3D point in a corresponding reference 3D point cloud frame in the set of reference 3D point cloud frames, and Reference coordinate information associated with the 3D point of the corresponding reference 3D point cloud frame. 6 . The first electronic device according to claim 1 , wherein the set of reference 3D point cloud frames includes at least one reference 3D point cloud frame before the current 3D point cloud frame, or at least one reference 3D point cloud frame after the current 3D point cloud frame.

7. The first electronic device of claim 1, wherein the circuit is further configured to downsample a set of features associated with 3D points of each reference 3D point cloud frame in the set of reference 3D point cloud frames by at least one scaling factor.

8. A second electronic device, comprising: a memory configured to store a second neural network predictor; as well as A circuit, the circuit being configured as: receiving a three-dimensional (3D) point cloud sequence including a set of reference 3D point cloud frames; generating reference frame data comprising a set of features associated with 3D points of each reference 3D point cloud frame in the set of reference 3D point cloud frames; receiving a bitstream of encoded point cloud data associated with a current 3D point cloud frame to be decoded; predicting a third set of features associated with 3D points of a current 3D point cloud frame based on applying a second neural network predictor to the reference frame data; generating a fourth set of features associated with the 3D points of the current 3D point cloud frame based on the received bitstream of the encoded point cloud data and the predicted third set of features; as well as The current 3D point cloud frame is reconstructed based on applying the decoding scheme to the determined fourth set of features. 9 . The second electronic device of claim 8 , wherein the received bitstream of encoded point cloud data further comprises encoded coordinate information associated with the 3D points of the current 3D point cloud frame.

10. The second electronic device of claim 9, wherein the circuit is further configured to generate coordinate information associated with the 3D points of the current 3D point cloud frame based on applying an octree-based decoder to the encoded coordinate information. The second electronic device according to claim 10 , wherein a third set of features is also predicted based on the generated coordinate information.

12. The second electronic device according to claim 8, wherein the memory is further configured to store a point cloud compression (PCC) encoder, The reference frame data is generated based on applying a PCC encoder to the set of reference 3D point cloud frames.

13. The second electronic device of claim 8, wherein the feature set associated with the 3D points of each reference 3D point cloud frame in the set of reference 3D point cloud frames comprises: a reference feature associated with an occupancy of a 3D point in a corresponding reference 3D point cloud frame in the set of reference 3D point cloud frames, and Reference coordinate information associated with the 3D point of the corresponding reference 3D point cloud frame.

14. A method comprising: In the First Electronic Device: receiving a three-dimensional (3D) point cloud sequence, the three-dimensional (3D) point cloud sequence comprising a set of reference 3D point cloud frames and a current 3D point cloud frame to be encoded; generating reference frame data, the reference frame data comprising a set of features associated with 3D points of each reference 3D point cloud frame in the set of 3D point cloud reference frames; Generate current frame data associated with the 3D points of the current 3D point cloud frame, wherein the current frame data includes a first set of features associated with occupancy of 3D points in the current 3D point cloud frame; predicting a second set of features associated with 3D points of a current 3D point cloud frame based on applying a first neural network predictor to the reference frame data; Calculate a set of residual features based on the first set of features and the second set of features; generating a set of quantized residual features based on applying a quantization scheme to the set of residual features; as well as Based on applying the encoding scheme to the set of quantized residual features, a bitstream of encoded point cloud data is generated for the current 3D point cloud frame.

15. The method of claim 14, wherein the second set of features is also predicted based on coordinate information associated with the 3D points of the current 3D point cloud frame.

16. The method of claim 15, further comprising encoding the coordinate information based on applying an octree-based encoder to the coordinate information, The bit stream of the encoded point cloud data includes encoded coordinate information.

17. The method according to claim 14, wherein: The reference frame data is generated based on applying a first point cloud compression (PCC) encoder to the set of reference 3D point cloud frames, and The current frame data is generated based on applying a first PCC encoder to a current 3D point cloud frame.

18. The method of claim 14, wherein the feature set associated with the 3D points of each reference 3D point cloud frame in the set of reference 3D point cloud frames comprises: a reference feature associated with an occupancy of a 3D point in a corresponding reference 3D point cloud frame in the set of reference 3D point cloud frames, and Reference coordinate information associated with the 3D point of the corresponding reference 3D point cloud frame.

19. The method according to claim 14, wherein the set of reference 3D point cloud frames includes at least one reference 3D point cloud frame before the current 3D point cloud frame, or at least one reference 3D point cloud frame after the current 3D point cloud frame.

20. The method of claim 14, further comprising downsampling a set of features associated with 3D points of each reference 3D point cloud frame in the set of reference 3D point cloud frames by at least one scaling factor.

Images