Geometric Coordinate Scaling for AI-Based Dynamic Point Cloud Coding

Abstract

An exemplary device for decrypting point cloud data includes a memory configured to store the point cloud data and one or more processors implemented in circuitry and configured to determine the number of times to upscale a downscaled encoded representation of the point cloud geometry, decrypt the downscaled encoded representation of the point cloud geometry, upscale the downscaled representation of the point cloud geometry by this number of times to form an upscaled representation of the point cloud geometry, and reproduce the point cloud using the upscaled representation of the point cloud geometry.

Description

Technical Field

[0001] This application claims the benefit of U.S. Patent Application No. 18 / 318,498, filed May 16, 2023, which claims the benefit of U.S. Provisional Application No. 63 / 364,862, filed May 17, 2022, the entire contents of which are hereby incorporated by reference.

[0002] This disclosure relates to point cloud coding, including point cloud encoding and decoding.

Background Art

[0003] A point cloud (PC) is a 3D data representation of a three-dimensional (3D) space. PCs may be used for tasks such as extended reality (XR), augmented reality (AR), virtual reality (VR), and mixed reality (MR), autonomous driving, cultural heritage, etc. A PC is a set of points in 3D space represented by their 3D coordinates (x, y, z) called geometry. Each point may also be associated with multiple attributes such as color, normal vector, or reflectivity.

[0004] Depending on the target application and the method of PC acquisition, the PC can be classified into a point cloud scene or a point cloud object. The point cloud scene may be captured using a LiDAR sensor and is often acquired dynamically. The point cloud object can be further subdivided into a static point cloud and a dynamic point cloud. The static PC is a single object, while the dynamic PC is a time-varying PC where each instance of the dynamic PC is a static PC. The dynamic time-varying PC may be used in AR / VR, volumetric video streaming, and telepresence, and can be generated using 3D models, i.e., CGI, or captured from real-world scenarios using various methods such as multiple cameras with depth sensors surrounding the object. These PCs can be high-density photorealistic point clouds with a large number of points, especially in high-precision or large-scale capture (up to millions of points per frame at 60 frames per second (FPS)). Therefore, efficient point cloud compression (PCC) is particularly important to enable practical use in VR and MR applications.

Summary of the Invention

[0005] Generally, the present disclosure describes a flexible configuration of a system that can apply a flexible amount of downscaling rather than adopting a fixed three-time downscaling / downsampling representation. That is, the point cloud encoder and the point cloud decoder can be configured to downscale / upscale the point cloud geometry data a variable number of times. The point cloud encoder may signal in the point cloud bitstream a value indicating the number of times the point cloud geometry has been downscaled, whereby the point cloud decoder can upscale the decoded point cloud geometry data by the same factor. The flexible amount of downscaling allows for better compression depending on the use case, bit requirements, and type of point cloud. The flexible configuration can be adopted for both intra-point cloud compression for static and dynamic point clouds and inter-point cloud compression for dynamic point clouds.

[0006] In one embodiment, a method for encoding point cloud data includes determining the number of times to downscale the representation of the point cloud geometry, downscaling the representation of the point cloud geometry by this number of times to form a downscaled representation of the point cloud geometry, encoding the downscaled representation of the point cloud geometry, and outputting the encoded downscaled representation of the point cloud geometry.

[0007] In another embodiment, a device for encoding point cloud data includes a memory configured to store the point cloud data, and one or more processors implemented in a circuit and configured to determine the number of times to downscale the representation of the point cloud geometry, downscale the representation of the point cloud geometry by this number of times to form a downscaled representation of the point cloud geometry, encode the downscaled representation of the point cloud geometry, and output the encoded downscaled representation of the point cloud geometry.

[0008] In another embodiment, a device for encoding point cloud data includes means for determining the number of times to downscale the representation of the point cloud geometry, means for downscaling the representation of the point cloud geometry by this number of times to form a downscaled representation of the point cloud geometry, means for encoding the downscaled representation of the point cloud geometry, and means for outputting the encoded downscaled representation of the point cloud geometry.

[0009] In another embodiment, a method for decoding point cloud data includes determining the number of times to upscale the downscaled encoded representation of the point cloud geometry, decoding the downscaled encoded representation of the point cloud geometry, upscaling the downscaled representation of the point cloud geometry by this number of times to form an upscaled representation of the point cloud geometry, and reproducing the point cloud using the upscaled representation of the point cloud geometry.

[0010] In another embodiment, a device for decoding point cloud data includes a memory configured to store the point cloud data, and one or more processors implemented in circuitry and configured to determine the number of times to upscale a downscaled encoded representation of the point cloud geometry, decode the downscaled encoded representation of the point cloud geometry, upscale the downscaled representation of the point cloud geometry by this number of times to form an upscaled representation of the point cloud geometry, and reproduce the point cloud using the upscaled representation of the point cloud geometry.

[0011] In another embodiment, a device for decoding point cloud data includes means for determining the number of times to upscale a downscaled encoded representation of the point cloud geometry, means for decoding the downscaled encoded representation of the point cloud geometry, means for upscaling the downscaled representation of the point cloud geometry by this number of times to form an upscaled representation of the point cloud geometry, and means for reproducing the point cloud using the upscaled representation of the point cloud geometry.

[0012] Details of one or more examples are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description, drawings, and claims.

Brief Description of the Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

DETAILED DESCRIPTION OF THE INVENTION

[0014] Efficient point cloud compression is essential for applications such as virtual reality and mixed reality, autonomous driving, and cultural heritage. Point cloud compression for high-density dynamic point clouds can be performed using a deep learning network that includes an encoding unit and a decoding unit. The encoding unit can extract features from the point cloud geometry and downscale the point cloud geometry three times using the corresponding feature embedding. The three-times downscaled geometry and the corresponding features can be sent to the decoding unit. The decoding unit can then hierarchically reconstruct the original point cloud geometry from the downscaled representation using progressive resizing.

[0015] This disclosure describes a flexible configuration of a system that can apply a flexible amount of downscaling rather than adopting a three-times downscaled / downsampled representation. The flexible amount of downscaling can result in better compression depending on the use case, bit requirements, and type of point cloud. The flexible configuration can be adopted for both in-point cloud compression for static and dynamic point clouds and inter-point cloud compression for dynamic point clouds.

[0016] FIG. 1 is a block diagram showing an exemplary encoding and decoding system 100 in which the techniques of this disclosure can be implemented. The techniques of this disclosure generally relate to coding (encoding and / or decoding) point cloud data, i.e., supporting point cloud compression. Generally, point cloud data includes any data for processing a point cloud. Coding can be effective for compressing and / or decompressing point cloud data.

[0017] As shown in FIG. 1, system 100 includes a source device 102 and a destination device 116. The source device 102 provides encoded point cloud data that is decoded by the destination device 116. Specifically, in the example of FIG. 1, the source device 102 provides the point cloud data to the destination device 116 via a computer-readable medium 110. The source device 102 and the destination device 116 can include any of a wide range of devices, including desktop computers, notebook (i.e., laptop) computers, tablet computers, set-top boxes, telephone handsets such as smartphones, televisions, cameras, display devices, digital media players, video gaming consoles, video streaming devices, land or sea vehicles, spacecraft, aircraft, robots, LIDAR devices, satellites, etc. In some cases, the source device 102 and the destination device 116 may support wireless communication.

[0018] In the embodiment of FIG. 1, the source device 102 includes a data source 104, a memory 106, a point cloud encoder 200, and an output interface 108. The destination device 116 includes an input interface 122, a point cloud decoder 300, a memory 120, and a data consumer 118. Thus, the source device 102 represents an example of an encoding device, and the destination device 116 represents an example of a decoding device. In other examples, the source device 102 and the destination device 116 may include other components or arrangements. For example, the source device 102 may receive data (e.g., point cloud data) from an internal or external source. Similarly, the destination device 116 may interface with an external data consumer rather than including the data consumer within the same device.

[0019] System 100 as shown in FIG. 1 is merely an example. Source device 102 and destination device 116 are merely examples of devices that generate data coded for transmission from source device 102 to destination device 116. The present disclosure refers to a device that performs data coding (encoding and / or decoding) as a “coding” device. Thus, point cloud encoder 200 and point cloud decoder 300 represent examples of coding devices, specifically, an encoder and a decoder, respectively. In some examples, source device 102 and destination device 116 may operate substantially symmetrically such that each of source device 102 and destination device 116 includes encoding and decoding components. Therefore, system 100 may support one-way or two-way transmission between source device 102 and destination device 116, for example, for streaming, playback, broadcast, telephony, navigation, and other applications.

[0020] Generally, data source 104 represents a source of data (i.e., raw, unencoded point cloud data) and can provide a sequential series of “frames” of data to point cloud encoder 200 that encodes the frame data. The data source 104 of source device 102 can include a point cloud capture device such as various cameras or sensors, e.g., a 3D scanner or a light detection and ranging (LIDAR) device, one or more video cameras, an archive containing previously captured data, and / or a data feed interface for receiving data from a data content provider. Additionally or alternatively, the point cloud data can be computer-generated from scanner data, camera data, sensor data, or other data. For example, data source 104 can generate computer graphics-based data as source data or generate a combination of live data, archive data, and computer-generated data. In each case, point cloud encoder 200 encodes the captured data, pre-captured data, or computer-generated data. Point cloud encoder 200 can reorder the frames from the reception order (sometimes referred to as the “display order”) to the coding order for coding. Point cloud encoder 200 can generate one or more bitstreams containing the encoded data. Next, source device 102 can output the encoded data onto computer-readable medium 110 via output interface 108 for reception and / or retrieval, e.g., by input interface 122 of destination device 116.

[0021] The memory 106 of the source device 102 and the memory 120 of the destination device 116 may represent general-purpose memory. In some embodiments, the memories 106 and 120 can store raw data, e.g., raw data from the data source 104, and raw decoded data from the point cloud decoder 300. Additionally or alternatively, the memories 106 and 120 can store software instructions executable by, e.g., the point cloud encoder 200 and the point cloud decoder 300 respectively. Although the memories 106 and 120 are shown separately from the point cloud encoder 200 and the point cloud decoder 300 in this embodiment, it should be understood that the point cloud encoder 200 and the point cloud decoder 300 may also include internal memory for functionally similar or equivalent purposes. Further, the memories 106 and 120 can store encoded data that is output from, e.g., the point cloud encoder 200 and input to the point cloud decoder 300. In some examples, portions of the memories 106 and 120 can be allocated as one or more buffers for storing, e.g., raw, decoded, and / or encoded data. For example, the memories 106 and 120 can store data representing a point cloud.

[0022] Computer-readable medium 110 may represent any type of medium or device capable of transporting the encoded data from source device 102 to destination device 116. In one example, computer-readable medium 110 represents a communication medium that enables source device 102 to directly transmit the encoded data to destination device 116 in real time via, for example, a radio frequency network or a computer-based network. Output interface 108 may modulate a transmission signal including the encoded data, and input interface 122 may demodulate the received transmission signal, in accordance with a communication standard such as a wireless communication protocol. The communication medium may include any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. The communication medium may include a router, a switch, a base station, or any other device that may be useful in facilitating communication from source device 102 to destination device 116.

[0023] In some examples, source device 102 may output the encoded data from output interface 108 to storage device 112. Similarly, destination device 116 may access the encoded data from storage device 112 via input interface 122. Storage device 112 may include any of a variety of distributed or locally accessible data storage media, such as a hard drive, a Blu-ray disk, a DVD, a CD-ROM, a flash memory, a volatile or non-volatile memory, or any other suitable digital storage media for storing the encoded data.

[0024] In some examples, the source device 102 may output the encoded data to a file server 114 or another intermediate storage device that may store the encoded data generated by the source device 102. The destination device 116 may access the data stored from the file server 114 via streaming or download. The file server 114 can be any type of server device that can store the encoded data and transmit the encoded data to the destination device 116. The file server 114 may represent a web server (e.g., of a website), a File Transfer Protocol (FTP) server, a content delivery network device, or a Network Attached Storage (NAS) device. The destination device 116 may access the encoded data from the file server 114 through any standard data connection including an Internet connection. This may include a wireless channel (e.g., Wi-Fi connection), a wired connection (e.g., digital subscriber line (DSL), cable modem, etc.), or a combination of both suitable for accessing the encoded data stored in the file server 114. The file server 114 and the input interface 122 may be configured to operate according to a streaming transmission protocol, a download transmission protocol, or a combination thereof.

[0025] The output interface 108 and the input interface 122 may represent a wireless transmitter / receiver, a modem, a wired network component (e.g., an Ethernet card), a wireless communication component operating according to any of various IEEE 802.11 standards, or other physical components. In examples where the output interface 108 and the input interface 122 include wireless components, the output interface 108 and the input interface 122 may be configured to transfer data such as encoded data according to cellular communication standards such as 4G, 4G-LTE (Long-Term Evolution), LTE Advanced, 5G, etc. In some examples where the output interface 108 includes a wireless transmitter, the output interface 108 and the input interface 122 may be configured to transfer data such as encoded data according to other wireless standards such as the IEEE 802.11 specification, the IEEE 802.15 specification (e.g., ZigBee (trademark)), the Bluetooth (trademark) standard, etc. In some examples, the source device 102 and / or the destination device 116 may each include a system-on-a-chip (SoC) device. For example, the source device 102 can include an SoC device for performing functions attributed to the point cloud encoder 200 and / or the output interface 108, and the destination device 116 can include an SoC device for performing functions attributed to the point cloud decoder 300 and / or the input interface 122.

[0026] The techniques of the present disclosure may be applied to encoding and decoding that support any of various applications such as communication between autonomous vehicles, communication between scanners, cameras, sensors and a processing device such as a local or remote server, geographical mapping, or other applications.

[0027] The input interface 122 of the destination device 116 receives an encoded bitstream from a computer-readable medium 110 (e.g., a communication medium, a storage device 112, a file server 114, etc.). The encoded bitstream can include signaling information defined by a point cloud encoder 200 that is also used by a point cloud decoder 300, such as syntax elements having values that describe the characteristics and / or processing of units to be coded (e.g., slices, pictures, groups of pictures, sequences, etc.). A data consumer 118 uses the decoded data. For example, the data consumer 118 may use the decoded data to determine the location of a physical object. In some examples, the data consumer 118 may include a display for displaying an image based on the point cloud.

[0028] The point cloud encoder 200 and the point cloud decoder 300 can each be implemented as any of a variety of suitable encoder and / or decoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. When the technique is implemented partially in software, the device can store instructions for the software in a suitable non-transitory computer-readable medium and execute the instructions in hardware using one or more processors to implement the techniques of the present disclosure. Each of the point cloud encoder 200 and the point cloud decoder 300 may be included within one or more encoders or decoders, and any of them may be integrated as part of a combined encoder / decoder (CODEC) in their respective devices. Devices including the point cloud encoder 200 and / or the point cloud decoder 300 can comprise one or more integrated circuits, microprocessors, and / or other types of devices.

[0029] The point cloud encoder 200 and the point cloud decoder 300 can operate according to a coding standard such as the video point cloud compression (V-PCC) standard or the geometric point cloud compression (point cloud) standard. The present disclosure may generally refer to the coding (e.g., encoding and decoding) of pictures, including the process of encoding or decoding data. The encoded bitstream generally includes a series of values of syntax elements representing coding decisions (e.g., coding modes).

[0030] This disclosure may generally refer to "signaling" some information such as a syntax element. The term "signaling" may generally refer to the communication of the value of a syntax element and / or other data used to decode encoded data. That is, the point cloud encoder 200 can signal the value of a syntax element in a bitstream. Generally, signaling refers to generating a value within a bitstream. As described above, the source device 102 may transfer the bitstream to the destination device 116 substantially in real time or non-real time, which may occur, for example, when storing syntax elements in the storage device 112 for later retrieval by the destination device 116.

[0031] ISO / IEC MPEG (JTC1 / SC29 / WG11) has investigated the potential need for standardizing point cloud coding technology with compression capabilities far exceeding those of current approaches and aims to create a standard. The group is working together in this quest within a joint effort called the 3D Graphics Team (3DG) to evaluate compression technology designs proposed by those experts in this field.

[0032] Point cloud compression activities are classified into two different methods. The first method is "Video Point Cloud Compression" (V-PCC), which segments 3D objects and projects the segments into multiple 2D planes (represented as "patches" in 2D frames), which are further coded by legacy 2D video coders such as the High Efficiency Video Coding (HEVC) (ITU-T H.265) codec. The second method is "Geometry-based Point Cloud Compression" (Point Cloud), which directly compresses 3D geometry, i.e., the positions of a set of points in 3D space and the associated attribute values (for each point associated with the 3D geometry). Point Cloud addresses the compression of point clouds in both Category 1 (static point clouds) and Category 3 (dynamically acquired point clouds). The latest draft of the Point Cloud standard is available at Point Cloud DIS, ISO / IEC JTC1 / SC29 / WG11 w19088, Brussels, Belgium, January 2020, and the codec description is available at Point Cloud Codec Description v6, ISO / IEC JTC1 / SC29 / WG11 w19091, Brussels, Belgium, January 2020.

[0033] A point cloud contains a set of points in 3D space and may have attributes associated with the points. The attributes can be color information such as R, G, B, or Y, Cb, Cr, or reflectance information, or other attributes. The point cloud may be captured by various cameras or sensors such as LIDAR sensors and 3D scanners, or may also be computer-generated. Point cloud data is used in various applications, including but not limited to architecture (modeling), graphics (3D models for visualization and animation), and the automotive industry (LIDAR sensors used to assist navigation).

[0034] The 3D space occupied by point cloud data can be enclosed by a virtual bounding box. The positions of points in the bounding box can be represented with a certain precision, and thus, the positions of one or more points can be quantized based on that precision. At the minimum level, the bounding box is divided into voxels, which are the smallest units of space represented by unit cubes. The voxels within the bounding box can be associated with zero, one, or more than one point. The bounding box can be divided into a plurality of cube / rectangular prism regions, sometimes called tiles. Each tile can be coded into one or more slices. The partitioning of the bounding box into slices and tiles can be based on the number of points in each partition, or on other considerations (e.g., a particular region may be coded as a tile). The slice regions can be further partitioned using a decision for partitioning similar to that in a video codec.

[0035] Figure 2 provides an overview of the point cloud encoder 200. Figure 3 provides an overview of the point cloud decoder 300. The illustrated modules are logical and do not necessarily correspond one-to-one to the implementation code in the reference implementation of the point cloud codec, i.e., the TMC13 test model software considered by ISO / IEC MPEG (JTC1 / SC29 / WG11).

[0036] In some cases, the compressed geometry can be represented as an octree from the root to the leaf level of individual voxels. In some cases, the compressed geometry can be represented by an octree with culling (i.e., an octree from the root to the leaf level of blocks larger than voxels), with a model added to approximate the surface within each leaf of the culled octree. In this way, various types of data can share the octree coding mechanism, while in some cases, the data can approximate the voxels within each leaf using a surface model. The surface model used is a triangulation with 1 to 10 triangles per block, resulting in a triangle soup.

[0037] At each node of the octree, for one or more of its child nodes (up to eight nodes), occupancy is signaled (when not already inferred). A plurality of neighboring nodes are specified, including (a) nodes sharing a face with the current octree node, (b) nodes sharing a face, edge, or vertex with the current octree node, etc. Within each neighboring node, the occupancy of the node and / or its child nodes can be used to predict the occupancy of the current node or its children. For points sparsely present at a particular node of the octree, the codec also supports a direct coding mode in which the 3D position of the point is directly encoded. A flag can be signaled to indicate that the direct mode is signaled. At the lowest level, the number of points associated with the octree node / leaf node can also be coded.

[0038] When the geometry is coded, the attributes corresponding to the geometry points are coded. When there are multiple attribute points corresponding to one reconstructed / decoded geometry point, an attribute value representing the reconstructed point can be derived.

[0039] There are three attribute coding methods for the point cloud, namely, Region Adaptive Hierarchical Transform (RAHT) coding, interpolation-based hierarchical nearest neighbor prediction (prediction transform), and interpolation-based hierarchical nearest neighbor prediction with update / lifting steps (lifting transform). RAHT and lifting are typically used for category 1 data, and prediction is typically used for category 3 data. However, any method can be used for any data, and the attribute coding method used to code the point cloud, as in the geometry codec for the point cloud, is specified in the bitstream.

[0040] The coding of attributes can be done at a level-of-detail (LOD), and using each LOD, a more detailed representation of the point cloud attributes can be obtained. Each LOD can be specified based on a distance metric from neighboring nodes or based on a sampling distance.

[0041] In the point cloud encoder 200, the residual obtained as the output of the attribute coding method is quantized. The residual can be obtained by subtracting the attribute value from a prediction derived based on the neighboring points of the current point and based on the attribute values of the previously encoded points. The quantized residual can be coded using context adaptive arithmetic coding.

[0042] In the example of FIG. 2, the point cloud encoder 200 includes a downscaling and encoding unit 202, a reconstruction unit 206, a color conversion unit 204, an attribute transfer unit 208, a RAHT unit 218, a LOD generation unit 220, a lifting unit 222, a coefficient quantization unit 224, and an arithmetic coding unit 226.

[0043] As shown in the example of FIG. 2, the point cloud encoder 200 can obtain a set of positions of points in the point cloud and a set of attributes. The point cloud encoder 200 can obtain a set of positions of points in the point cloud and a set of attributes from the data source 104 (FIG. 1). The positions can include the coordinates of the points in the point cloud. The attributes can include information about the points in the point cloud, such as the color associated with the points in the point cloud. The point cloud encoder 200 can generate a geometric bitstream 203 including an encoded representation of the positions of the points in the point cloud. The point cloud encoder 200 can also generate an attribute bitstream 205 including an encoded representation of the set of attributes.

[0044] The downscaling and encoding unit 202 can generally downscale the received geometric data by a certain amount along, for example, the X-axis, Y-axis, and / or Z-axis, and encode the downscaled geometric data. In some embodiments, the downscaling and encoding unit 202 can include, as a unit, a set of downscaling and encoding task units, each of which downscales and encodes the geometric data by a certain amount and then passes the downscaled and encoded geometric data to a subsequent task unit. In other embodiments, the downscaling and encoding unit 202 can include a series of downscaling stages, each of which downscales the geometric data, and a set of one or more encoding task units that then encode the downscaled geometric data.

[0045] The downscaling and encoding unit 202 can downscale the input geometric data (e.g., octree) by a factor of 8, i.e., 2 times along the X-axis, 2 times along the Y-axis, and 2 times along the Z-axis. Such downscaling can be performed by an artificial intelligence / machine learning (AI / ML) unit such as a neural network. Further details regarding such embodiments of downscaling will be described below.

[0046] The color conversion unit 204 can apply a conversion to convert the color information of the attributes to a different domain. For example, the color conversion unit 204 can convert the color information from the RGB color space to the YCbCr color space.

[0047] The reconstruction unit 206 can decode and reconstruct the geometric data. The attribute transfer unit 208 can transfer the attributes of the original points of the point cloud to the reconstructed points of the point cloud.

[0048] Furthermore, the RAHT unit 218 can apply RAHT coding to the attributes of the reconfigured points. In some examples, under RAHT, the attributes of a block of 2×2×2 point positions are taken and transformed along one direction to obtain four low-frequency nodes (L) and four high-frequency nodes (H). Then, the four low-frequency nodes (L) are transformed in a second direction to obtain two low-frequency nodes (LL) and two high-frequency nodes (LH). The two low-frequency nodes (LL) are transformed in a third direction to obtain one low-frequency node (LLL) and one high-frequency node (LLH). The low-frequency node LLL corresponds to the DC coefficient, and the high-frequency nodes H, LH, and LLH correspond to the AC coefficients. The transformation in each direction can be a 1-D transformation using two coefficient weights. The low-frequency coefficients can be regarded as the coefficients of the 2×2×2 block for the next higher level of the RAHT transform, and the AC coefficients are coded without change, and such transformations continue up to the top root node. Such transformations continue up to the top root node. The tree traversal for coding is a top-down traversal used to calculate the weights used for the coefficients, and the transformation order is bottom-up. Next, the coefficients can be quantized and coded.

[0049] Additionally or alternatively, the LOD generation unit 220 and the lifting unit 222 can apply LOD processing and lifting respectively to the attributes of the reconstructed points. LOD generation is used to divide the attributes into different levels of refinement. Each level of refinement provides refinement to the attributes of the point cloud. The first level of refinement provides a coarse approximation and contains a small number of points, and subsequent levels of refinement typically contain more points, and so on. The levels of refinement may be configured using a distance-based metric or may use one or more other classification criteria (e.g., subsampling from a specific order). Thus, all the reconstructed points can be included in the levels of refinement. Each detail level is generated by obtaining the union of all points up to a specific refinement level. For example, LOD1 is obtained based on the refinement level RL1, LOD2 is obtained based on RL1 and RL2, and LODN is obtained by the union of RL1, RL2,... RLN. In some cases, after LOD generation, a prediction scheme (such as a prediction transform) may follow in which the attributes associated with each point in the LOD are predicted from the weighted average of the preceding points, and the residuals are quantized and entropy-coded. The lifting scheme is constructed on top of a prediction transform mechanism in which an update operator is used to update the coefficients and adaptive quantization of the coefficients is performed.

[0050] The RAHT unit 218 and the lifting unit 222 can generate coefficients based on the attributes. The coefficient quantization unit 224 can quantize the coefficients generated by the RAHT unit 218 or the lifting unit 222. The arithmetic coding unit 226 can apply arithmetic coding to the syntax elements representing the quantized coefficients. The point cloud encoder 200 can output these syntax elements in the attribute bitstream 205. The attribute bitstream 205 may also include other syntax elements including non-arithmetically encoded syntax elements.

[0051] In the embodiment of FIG. 3, the point cloud decoder 300 includes a decoding and upscaling unit 302, an attribute arithmetic decoding unit 304, an inverse quantization unit 308, a RAHT unit 314, a LOD generation unit 316, an inverse lifting unit 318, an inverse coordinate transformation unit 320, and an inverse color transformation unit 322.

[0052] The point cloud decoder 300 can obtain the geometric bitstream 203 and the attribute bitstream 205. The decoding and upscaling unit 302 of the decoder 300 can decode a value indicating the amount of upscaling to be applied to the geometric data. Further, the decoding and upscaling unit 302 can perform both decoding and upscaling of the geometric data, and the amount of upscaling can correspond to the decoded value. In some embodiments, the decoding and upscaling unit 302 can include a set of units including both a decoding unit and an upscaling unit, and the number of sets is equal to the decoded value representing the amount of upscaling. In some embodiments, the decoding and upscaling unit 302 can include one or more decoding units followed by a set of upscaling units, and the number of sets is equal to the decoded value representing the amount of upscaling. In some embodiments, the decoding and upscaling unit 302 can further reconstruct the point cloud geometry using the decoded and upscaled point cloud geometry data.

[0053] The attribute arithmetic decoding unit 304 can apply arithmetic decoding to the syntax elements in the attribute bitstream 205.

[0054] The octree synthesis unit 306 can synthesize an octree based on the syntax elements parsed from the geometric bitstream 203. Starting from the root node of the octree, the occupancy of each of the eight child nodes at each octree level is signaled in the bitstream. When the signaling indicates that a child node at a particular octree level is occupied, the occupancy of the children of this child node is signaled. The signaling of the nodes at each octree level is done before proceeding to the subsequent octree level. At the final level of the octree, each node corresponds to a voxel position, and when a leaf node is occupied, one or more points can be specified to be occupied at the voxel position. In some instances, due to quantization, some branches of the octree may terminate before the final level. In such cases, the leaf node is considered an occupied node without children.

[0055] The inverse coordinate transformation unit 320 can apply an inverse transformation to the reconstructed coordinates to reverse the reconstructed coordinates (positions) of the points in the point cloud from the transform domain back to the initial domain. The positions of the points in the point cloud may be in the floating-point domain, but the point positions in the point cloud codec are coded in the integer domain. The inverse transformation can be used to transform the positions back to the original domain.

[0056] In addition, in the example of FIG. 3, the inverse quantization unit 308 can inverse-quantize the attribute values. The attribute values can be based on syntax elements (including, for example, syntax elements decoded by the attribute arithmetic decoding unit 304) obtained from the attribute bitstream 205.

[0057] Depending on how the attribute values are encoded, the RAHT unit 314 may perform RAHT coding to determine color values for the points of the point cloud based on the inverse-quantized attribute values. RAHT decoding is performed from the top to the bottom of the tree. At each level, the low-frequency and high-frequency coefficients derived from the inverse quantization process are used to derive the configuration values. At the leaf node, the derived value corresponds to the attribute value for the coefficient. The weight derivation process for the points is the same as the process used in the point cloud encoder 200. Alternatively, the LOD generation unit 316 and the inverse lifting unit 318 may determine color values for the points of the point cloud using a level-of-detail-based technique. The LOD generation unit 316 decodes each LOD to represent the attributes of the points more gradually in detail. Using prediction transformation, the LOD generation unit 316 derives a predicted value for a point from the weighted sum of points that are within the previous LOD or that have been previously reconstructed within the same LOD. The LOD generation unit 316 may add the predicted value to the residual (obtained after inverse quantization) to obtain the reconstructed value of the attribute. When the lifting scheme is used, the LOD generation unit 316 may also include an update operator for updating the coefficients used to derive the attribute values. In this case, the LOD generation unit 316 may also apply inverse adaptive quantization.

[0058] Furthermore, in the example of FIG. 3, the inverse color conversion unit 322 may apply an inverse color conversion to the color values. The inverse color conversion may be the inverse of the color conversion applied by the color conversion unit 204 of the encoder 200. For example, the color conversion unit 204 may convert color information from the RGB color space to the YCbCr color space. Accordingly, the inverse color conversion unit 322 may convert the color information from the YCbCr color space to the RGB color space.

[0059] The various units of FIGS. 2 and 3 are shown to assist in understanding the operations performed by the encoder 200 and the decoder 300. The units can be implemented as fixed-function circuits, programmable circuits, or combinations thereof. A fixed-function circuit refers to a circuit that provides a specific function, and the operations that can be performed are pre-set. A programmable circuit refers to a circuit that can be programmed to perform various tasks and provides a flexible function in the operations that can be performed. For example, a programmable circuit can execute software or firmware that operates the programmable circuit in a manner defined by software or firmware instructions. A fixed-function circuit can execute software instructions (e.g., receive or output parameters), but the type of operations performed by the fixed-function circuit is generally invariant. In some examples, one or more of the units may be different circuit blocks (fixed-function or programmable), and in some examples, one or more of the units may be integrated circuits.

[0060] FIG. 4 is a conceptual diagram showing an exemplary system model for implementing the technology of the present disclosure. The present disclosure will describe an irreversible geometric compression method that can be executed by the system 400 in FIG. 4 to predict the latent representation of the current frame of point cloud data using the previous frame (the previously decoded frame) by adopting a prediction network. The system 400 performs P-frame inter-frame point cloud encoding in which the current frame is encoded with the help of the previously decoded frame. The system 400 can be implemented using a sparse convolutional neural network (CNN) having a sparse tensor. The system 400 can adopt convolution on the target coordinates to map the latent representation of the previous frame to the downsampled coordinates of the current frame and predict the feature embedding of the current frame. The system 400 can transmit the residual between the predicted feature and the actual feature by compressing the residual between the predicted feature and the actual feature using a learned probabilistic factorization entropy model. Compared with G-PCC and V-PCC, the technology executed by FIG. 4 according to the present disclosure exhibits better compression performance for high-density point clouds with an efficient encoding / decoding runtime, as will be described in more detail below.

[0061] Conventional deep learning-based PCC solutions include intra-prediction techniques for static point clouds and do not utilize inter-prediction coding gains by predicting the current frame using the previously decoded frame. The technology of the present disclosure includes a deep learning-based framework for point cloud geometric inter-frame encoding similar to P-frame encoding in video compression. The technology of the present disclosure includes a predictor module that learns the feature embedding of the current PC frame from the previous PC frame. The network can use hierarchical multi-scale feature extraction and adopt convolution on the target coordinates to map the latent features from the previous frame to the downsampled coordinates of the current frame and learn the feature embedding of the current frame.

[0062] Point cloud geometry compression (PCGC) is an intra-frame point cloud compression method suitable for static point clouds. The inter-frame prediction method of the present disclosure uses an encoder and a decoder network together with a prediction network to predict the feature embedding for the current PC frame from the previous PC frame. According to these techniques, system 400 can calculate the residual (i.e., a series of differences) between the predicted features and the ground truth features, and then can transmit the residual features together with the coordinates downsampled three times. The coordinates downsampled three times can be reversibly encoded by the octree encoder 406 using, for example, the G-PCC TMC13 octree codec as described above with respect to FIG. 2. The residual features can be encoded in an irreversible manner using a factored entropy model to predict the probability distribution for arithmetic coding. Note that in system 400, the encoder and the prediction network are present in both the transmitter 402 and the receiver 404. The network is trained with joint reconstruction and bitrate loss to optimize the rate distortion.

[0063] The octree encoder 406 may generally correspond to the portion of the point cloud encoder 200 in FIG. 2, and the octree decoder 408 may generally correspond to the portion of the point cloud decoder 300 in FIG. 3. The transmitter 402 may generally correspond to the source device 102 in FIG. 1, and the receiver 404 may generally correspond to the destination device 116 in FIG. 1.

[0064] In the embodiment of FIG. 4, system 400 includes a transmitter 402 and a receiver 404. The transmitter 402 encodes the feature embedding of the current frame p

[0065]

Number

[0066]

Number

[0067]

Number

[0068]

Number

[0069]

Number

[0070]

Number

[0071]

Number

[0072]

Number

[0073]

Number

[0074] This disclosure uses the terms "downscaled" and "downsampled" interchangeably. The overall framework can be regarded as a transmission system, a compression pipeline, or a deep learning model. Regarding the transmission system, the elements before bitstream generation may be referred to as the formative part of the "transmitter", and the elements after bitstream generation may be referred to as the formative part of the "receiver". The system 400 in FIG. 4 can include an encoder and predictor module in the transmitter 402 and an encoder, predictor, and decoder module in the receiver 404. Regarding the compression pipeline, the system 400 can be seen as including both an encoding part and a decoding part. In FIG. 4, the predictor module is used for both frame encoding and frame decoding. Regarding the deep learning model, the encoder can be regarded as a multi-scale feature extractor, and the decoder can be regarded as a progressive upscaling network with hierarchical reconstruction of the point cloud.

[0075] FIG. 5 is a conceptual diagram showing an exemplary encoder network 502 and decoder network 504. The encoder network 502 can correspond to all or part of the point cloud encoder 200. The encoder network 502 has four different scales that capture multi-scale features at different levels of detail: P 0ds ,P 1ds ,P 2ds ,P 3ds =E(P) to obtain the PC tensor. where P idsrepresents the sparsely sampled tensor P that has been downsampled i times. The decoder network 504 can correspond to all or part of the point cloud decoder 300. In this embodiment, the decoder network 504 receives the PC tensor that has been downsampled 3 times, hierarchically upsamples the downsampled PC tensor to reconstruct the original PC tensor, and employs different reconstruction losses at each scale. The decoder network 504 can use transposed convolution to upsample the PC tensor.

[0076] That is, the encoder network 502 can generate sparse features from the original point cloud sparse tensor P at four different scales: P 0ds , P 1ds , P 2ds , P 3ds . The decoder network 504 receives the sparsely sampled tensor that has been downsampled 3 times and can hierarchically reconstruct the original point cloud using progressive rescaling. The decoder network 504 upsamples the sparse tensor, after which the pruning layer can prune the false voxels.

[0077] In this way, FIG. 5 shows an embodiment of the encoder network 502 that includes a series of sets of units, where each set of units includes a downscaling unit and an encoding unit. The number of sets in the series may be equal to the encoded value in the bitstream indicating the number of times the geometric data is downscaled. Similarly, FIG. 5 shows an embodiment of the decoder network 504 that includes a series of sets of units, where each set of units includes a decoding unit and an upscaling unit. The number of sets in the series may be equal to the value decoded from the bitstream indicating the number of times the geometric data is to be upscaled.

[0078] FIG. 6 is a block diagram showing an exemplary pruning layer 600 of a decoder according to the technology of the present disclosure. The decoder can use the pruning layer 600 to reconstruct the geometry of a PC. The pruning layer 600 can use binary classification after each upscaling to prune false voxels and extract true occupied voxels.

[0079] In the example of FIG. 6, the input sparse tensor P a has coordinates C of shape 139,244×3 a and their corresponding features of shape 139,244×64. P a is a sparse tensor P with features F of shape 139,244×1 b and is passed through a convolution with a channel size of 1 (CONV 1×3 602) to obtain P b . From F b , the top k features (k = 52612 in this example) and their corresponding coordinates are selected using binary classification. The pruning unit 604 then prunes the false coordinates and their corresponding features from P a to obtain P c . During training, the binary voxel classification loss is applied to P b to learn appropriate point cloud reconstruction.

[0080] FIG. 7 is a conceptual diagram showing an exemplary prediction network 700. The prediction network 700 can be configured to predict the latent representation of the current frame from a previously reconstructed frame. The prediction network 700 may be included in an encoder such as the point cloud encoder 200, the transmitter 402, or the encoder network 502, or in a decoder such as the point cloud decoder 300, the receiver 404, or the decoder network 504. The prediction network 700 downsamples the input three times while concatenating the downsampled input with the corresponding scale features. Finally, the prediction network 700 employs a convolution on the target coordinates to obtain the features of the current frame.

[0081] In the embodiment of FIG. 7, the prediction network 700 receives four multi-scale features from the previous frame and the downsampled coordinates of the current frame

[0082]

Number

[0083]

Number

[0084] FIG. 8 is a conceptual diagram showing an exemplary convolution on target coordinates in two dimensions (2D). The input is shown in the bottommost layer 802 shown (specifically, the shaded cells of the bottommost layer), and the corresponding output coordinates are shown as the shaded cells of the topmost layer 800 in FIG. 8.

[0085] The convolution on target coordinates is the downsampled coordinates of P 1 , that is,

[0086]

Number

[0087]

Number

[0088] The technology of this disclosure was implemented and tested. During the implementation and testing, to reduce the complexity of the calculations during training, the point cloud (PC) frames were divided into smaller chunks by applying the same kd-tree partition to two consecutive frames. During the inference time, the entire point cloud was used. A 10-bit dataset representation was used for both training and testing.

[0089] The technology of this disclosure was compared with the deep learning intra-frame technology of PCGC, the G-PCC (octree and trisoup) technology, and the V-PCC method (inter-frame and intra-frame coding). The reference implementation form TMC13-v14 of G-PCC was adopted. For V-PCC, the implementation form TMC2-v17 using the HEVC video codec was adopted. The point cloud was encoded using the V-PCC inter-frame low-latency setting, which involves P-frame coding for a fair comparison with the proposed P-frame coding method. The following table summarizes the test methods.

[0090]

Table 1

[0091]

Table 2

[0092] The following table summarizes the Bjontegaard Delta (BD)-rate gain of the technology of this disclosure compared with other conventional technologies using the D1 distortion measurement for various standardized test sequences.

[0093]

Table 3

[0094] Figures 9 and 10 are graphs representing the rate-distortion curves comparing the technology of this disclosure with other conventional technologies.

[0095] In the above tests, the Intel Core i9-11900F CPU and Nvidia GeForce GTX 3090 GPU were adopted. The G-PCC runtime calculated for the highest bitrate on the CPU. Both PCGC and the proposed method (the "present disclosure") utilize the GPU. Due to platform diversity, such as CPU vs. GPU, Python vs. C / C++, and differences in other such implementation forms, the execution time comparison serves as an intuitive criterion for representing the complexity of general calculations.

[0096] The following table shows the encoding and decoding execution times of various technologies compared with the technology of the present disclosure (the "present disclosure").

[0097]

Table 4

[0098]

Table 5

[0099] The present disclosure describes a deep learning-based inter-frame compression method for dynamic point clouds that encodes the current frame using previously decoded previous frames. The technology of the present disclosure employs an encoder to obtain multi-scale features and a decoder to hierarchically reconstruct the point cloud by progressive scaling. The prediction network module can predict the latent representation of the current frame by using convolution on the target coordinates to map the latent features of the previous frame to the downsampled coordinates of the current frame. The residuals between the predicted features and the actual features are encoded and transmitted. The experimental results show a BD-rate gain of more than 91% over G-PCC (octree), a BD-rate gain of more than 84% over G-PCC (trisoup), a BD-rate gain of more than 34% over the intra-frame network PCGC, a BD-rate improvement of more than 62% over the V-PCC intra-frame coding mode, and a BD-rate improvement of more than 52% over the V-PCC inter-frame low-latency coding mode.

[0100] The above-described inter-frame point cloud compression method is an example of P-frame point cloud encoding, where the previous frame is used to encode the current frame. Similar techniques can be implemented for I-frame-based intra-frame point cloud encoding as well as B-frame-based inter-frame point cloud encoding.

[0101] FIG. 11 is a conceptual diagram showing an exemplary system 1100 for intra (I)-frame-based intra-frame point cloud encoding. In intra-frame encoding, only the current frame is used to encode the current frame. System 1100 includes an encoder portion 1102 and a decoder portion 1104.

[0102] FIG. 12 is a conceptual diagram showing an exemplary system 1200 for bidirectional intra prediction (B) frame encoding. The system 1200 includes an encoder portion 1202 and a decoder portion 1204. In B frame encoding, both the previous frame and the next frame (in display order) can be used to encode the current frame.

[0103] The I-frame, P-frame, and B-frame methods described above use fixed downscaling parameters in the encoder to obtain a three-times downscaled representation of a point cloud having geometry and corresponding coordinates. Each time the geometry is downscaled by a factor of two, a total of eight-times geometric downscaling is achieved. The three-times (eight-times) downscaled coordinates can be transmitted in a reversible manner, while the corresponding features can be transmitted in an irreversible manner. In this way, the techniques of the present disclosure can generate a certain bitrate cost with respect to the three-times downsampled coordinates

[0104]

Number

[0105]

Number

[0106] The techniques of the present disclosure can provide a flexible configuration in a system model, and instead of having a fixed three-times downscaled representation of a point cloud, there can be different architectures that generate different downscaled representations of the point cloud. In this way, the level of downscaling required to obtain the best performance can be selected. Thus, these techniques can be applied to the downsampled coordinates

[0107]

Number

[0108]

Number

[0109] Referring to FIG. 4 again, the system 400 can use the level of the downscaled representation to indicate the operating parameters of the model, for example, the level at which the coordinates are transmitted in a reversible manner. For example, level x can indicate that the downscaled representation of the point cloud is obtained after x downscaling steps. In another embodiment, level x can indicate that the downscaled representation is obtained using downscaling by a coefficient f(x), i.e., a function of x (e.g., f(x) = 2 x ). Conversely, at the receiver, level x indicates the number of upscaling steps of the model.

[0110] The transmitter 402 can signal a syntax element in the bitstream to indicate the level of the downscaled representation. For example, the transmitter 402 can signal the syntax element ds_level_minus1, and ds_level_minus1 + 1 specifies the level of the downscaled representation used for coding. The decoder of the receiver 404 can decode this syntax element to determine the number of upscaling steps for decoding the bitstream.

[0111] In one alternative form, transmitter 402 can signal syntax elements in relation to one or more frames. Receiver 404 can use the signaled values to derive the level of the downscaled representation for one or more frames.

[0112] System 400 can be configured to transmit triply downscaled coordinates

[0113]

Number

[0114] FIG. 13 is a graph 1300 representing a rate-distortion curve for the technology of the present disclosure. As can be seen from the figure, the lowest bitrate point of the P-frame is {bpp: 0.0278, PSNR: 64.04}, and the lowest bitrate point of the I-frame is {bpp: 0.0231, PSNR: 61.60}. FIG. 13 shows that at the lowest bitrates, most of the bitrate is consumed by the coordinates

[0115]

Number

[0116]

Number

[0117]

Number

[0118] Similarly, for the highest bit rate and PSNR, the P-frame point is {bpp: 0.1818, PSNR: 73.54}, and the I-frame point is {bpp: 0.3837, PSNR: 73.83}. At these points, the coordinates

[0119]

Number

[0120]

Number

[0121] A plurality of system models are used that have the ability to generate downscaled / downsampled representations with different point clouds. This provides more flexibility in the rate-distortion curve in order to generate point clouds of better quality with fewer bits. To indicate which encoder was employed, extra bits may be transmitted from transmitter 402 in FIG. 4 to receiver 404, and thus the corresponding decoder can be used at receiver 404. Similarly, in the case of inter-frame coding, a dedicated predictor module can be employed for each level of the downscaled representation.

[0122] The techniques of the present disclosure can be employed in various use cases, such as 1) in each type of compression technique (I-frame, P-frame, B-frame), even when motion vector and optical flow techniques are employed, 2) point cloud attribute compression (going beyond mere geometric compression), and 3) including downscaled representations of the point cloud and then resizing the downscaled point cloud back to a reconstructed point cloud, for any other kind of point cloud compression or communication task.

[0123] The number of times the geometric coordinates are downscaled can be signaled and coded in the bitstream from transmitter 402 to receiver 404. In this way, receiver 404 can select the corresponding architecture and network to reconstruct and upsample the point cloud. This parameter can be signaled in one or more parameter sets, such as a sequence parameter set (SPS), a geometry parameter set (GPS), or in a header such as a slice header, frame header.

[0124] FIG. 14 is a flowchart showing an exemplary method of encoding point cloud data according to the techniques of the present disclosure. The method of FIG. 14 is described with respect to point cloud encoder 200. However, it should be understood that the method of FIG. 14 can also be performed by other devices such as transmitter 402 of FIG. 4, encoder network 502 of FIG. 5, encoder portion 1102 of FIG. 11, or encoder portion 1202 of FIG. 12.

[0125] First, point cloud encoder 200 determines the number of times to downscale the point cloud geometry of the point cloud (1400). For example, point cloud encoder 200 may retrieve configuration information indicating the number of times to downscale the point cloud geometry, for example, along the X, Y, and Z axes. In some embodiments, point cloud encoder 200 can be configured to downscale each of the X, Y, and Z axes by a factor of 2 for a total downscaling factor of 8. The downscaling and encoding unit 202 can downscale the point cloud geometry (e.g., hierarchical octree) by the determined number of times, e.g., 8 (1402). As described above, the downscaling and encoding unit 202 can be configured to perform such downscaling using an AI / ML model as described with respect to FIGS. 4, 5, 6, 7, 11, and 12. In some embodiments, point cloud encoder 200 can encode data representing the number of times the point cloud geometry has been downscaled, e.g., the value of syntax element ds_level_minus1, where ds_levelminus1+1 can indicate the number of times the point cloud geometry has been downscaled.

[0126] The point cloud encoder 200 can also encode the downscaled point cloud geometry (1404). The downscaling and encoding unit 202 can encode each intermediate set of the downscaled point cloud geometry, for example, using an AI / ML model for downscaling and encoding point cloud geometry data. In particular, in some embodiments, as described above, the downscaling and encoding unit 202 can include a series of sets of units, where each set of units includes a downscaling unit (e.g., a convolutional neural network (CNN)) and a set of encoding units. The number of sets in the series can correspond to the number of times the point cloud geometry is downscaled. The point cloud encoder 200 can also encode the features of the downscaled point cloud geometry (1406). As described above, the features can include attribute data such as color, reflectivity, luminance, surface normal, etc. The point cloud encoder 200 can then output the encoded point cloud geometry and features, for example, as a bitstream (1408).

[0127] In this way, the method of FIG. 14 represents an example of a method for encoding point cloud geometry data, including determining the number of times to downscale the representation of the point cloud geometry, downscaling the representation of the point cloud geometry this number of times to form a downscaled representation of the point cloud geometry, encoding the downscaled representation of the point cloud geometry, and outputting the encoded downscaled representation of the point cloud geometry.

[0128] FIG. 15 is a flowchart showing an exemplary method for decoding point cloud data according to the techniques of the present disclosure. The method of FIG. 15 is described with respect to the point cloud decoder 300. However, it should be understood that the method of FIG. 15 can also be performed by other devices such as the receiver 404 of FIG. 4, the decoder network 504 of FIG. 5, the decoder portion 1104 of FIG. 11, or the decoder portion 1204 of FIG. 12.

[0129] First, the point cloud decoder 300 can determine the number of times the point cloud geometry has been downscaled (1500). The number of times the point cloud geometry has been downscaled can correspond to the number of times the point cloud geometry should be upscaled. Thus, the point cloud decoder 300 can alternatively be configured to determine the number of times the point cloud geometry should be upscaled. For example, the point cloud decoder 300 can decode the value of a syntax element such as ds_level_minus1, and ds_levelminus1 + 1 can indicate the number of times the point cloud geometry has been downscaled and / or the number of times the point cloud geometry should be upscaled.

[0130] The decoding and upscaling unit 302 of the point cloud decoder 300 can then decode the downscaled point cloud geometry (1502) and can upscale the downscaled point cloud geometry (1504). For example, the decoding and upscaling unit 302 can include a series of sets of units, where each set of units includes a decoding unit and an upscaling unit. The number of sets in the series can be equal to the value of a syntax element indicating the number of times the point cloud geometry has been downscaled / should be upscaled. The point cloud decoder 300 can also decode the features of the downscaled point cloud geometry (1506). The decoding and upscaling of the point cloud geometry can be performed using an AI / ML model for decoding and upscaling point cloud data. In some embodiments, the point cloud decoder 300 can upscale the downscaled point cloud geometry and then use the upscaled point cloud geometry when decoding the features. The features can correspond to, for example, color, reflectivity, luminance, surface normal, etc. Finally, the point cloud decoder 300 can reproduce the point cloud from the upscaled point cloud geometry and features (1508), for example, by applying the features to the points of the upscaled point cloud geometry.

[0131] In this way, the method of FIG. 15 includes determining the number of times to upscale the downscaled and encoded representation of the point cloud geometry, decoding the downscaled and encoded representation of the point cloud geometry, upscaling the representation of the point cloud geometry by this number of times to form an upscaled representation of the point cloud geometry, and reproducing the point cloud using the upscaled representation of the point cloud geometry, and represents an embodiment of a method for decoding point cloud data.

[0132] Various examples of the techniques of the present disclosure are summarized in the following clauses.

[0133] Clause 1: A method for encoding point cloud data, including determining the number of times to downscale the representation of the point cloud geometry, downscaling the representation of the point cloud geometry by this number of times to form a downscaled representation of the point cloud geometry, encoding the downscaled representation of the point cloud geometry, encoding the features corresponding to the downscaled representation of the point cloud geometry, and outputting the encoded and downscaled representation of the point cloud geometry and the encoded features.

[0134] Clause 2: The method according to clause 1, wherein determining the number of times to downscale the representation includes determining the number of times to downscale the representation according to at least one of the use case, bit requirements, or type of the point cloud for the point cloud geometry.

[0135] Clause 3: The method according to clause 1 or 2, wherein encoding the downscaled representation of the point cloud geometry includes reversibly encoding the downscaled representation of the point cloud geometry.

[0136] Clause 4: The method according to any one of clauses 1 to 3, wherein encoding the features includes encoding the features using irreversible encoding.

[0137] Clause 5: The method according to any one of Clauses 1 to 4, wherein encoding the downscaled representation of point cloud geometry includes encoding the downscaled representation of point cloud geometry using one of intra-frame (I-frame) encoding, inter-frame (P-frame) encoding, or bi-directional inter-frame (B-frame) encoding.

[0138] Clause 6: The method according to any one of Clauses 1 to 5, wherein the downscaled representation of point cloud geometry includes the downscaled representation of point cloud geometry for the current frame, and encoding the features includes extracting multi-scale features from a previously coded frame of the point cloud data, forming predicted features of the current frame from the extracted multi-scale features and the downscaled representation of point cloud geometry for the current frame by a predictor network, and coding residual data representing the difference between the features of the current frame and the predicted features of the current frame.

[0139] Clause 7: The method according to any one of Clauses 1 to 6, further including coding data representing the number of times the representation of the point cloud geometry has been downscaled.

[0140] Clause 8: The method according to Clause 1, wherein determining the number of times to downscale the representation includes determining the number of times to downscale the representation according to at least one of the use case, bit requirement, or type of point cloud for the point cloud geometry.

[0141] Clause 9: The method according to Clause 1, wherein encoding the downscaled representation of point cloud geometry includes reversibly encoding the downscaled representation of point cloud geometry.

[0142] Clause 10: The method according to Clause 1, wherein encoding the features includes encoding the features using irreversible encoding.

[0143] Clause 11: The method according to Clause 1, wherein encoding a downscaled representation of point cloud geometry includes encoding the downscaled representation of point cloud geometry using one of intra-frame (I-frame) encoding, inter-frame (P-frame) encoding, or bi-directional inter-frame (B-frame) encoding.

[0144] Clause 12: The method according to Clause 1, wherein the downscaled representation of point cloud geometry includes the downscaled representation of point cloud geometry for the current frame, and encoding the features includes extracting multi-scale features from a previously coded frame of the point cloud data, forming predicted features of the current frame from the extracted multi-scale features and the downscaled representation of point cloud geometry for the current frame by a predictor network, and coding residual data representing the difference between the features of the current frame and the predicted features of the current frame.

[0145] Clause 13: The method according to Clause 1, further including coding data representing the number of times the representation of the point cloud geometry has been downscaled.

[0146] Clause 14: A method of decoding point cloud data, including determining the number of times to upscale a downscaled encoded representation of point cloud geometry, decoding the downscaled encoded representation of point cloud geometry, decoding features corresponding to the downscaled representation of point cloud geometry, upscaling the representation of point cloud geometry this number of times to form an upscaled representation of point cloud geometry, reproducing the point cloud using the upscaled representation of point cloud geometry and the features, and outputting the reproduced point cloud.

[0147] Clause 15: The method according to Clause 14, wherein determining the number of times to downscale the representation includes determining the number of times to downscale the representation according to at least one of a use case, a bit requirement, or a type of point cloud for the point cloud geometry.

[0148] Clause 16: The method according to clause 14 or 15, wherein determining the number of times to downscale the representation includes decoding data representing the number of times the representation of the point cloud geometry has been downscaled.

[0149] Clause 17: The method according to any one of clauses 14 to 16, wherein decoding the downscaled representation of the point cloud geometry includes decoding the downscaled representation of the point cloud geometry using one of intra-frame (I-frame) decoding, inter-frame (P-frame) decoding, or bi-directional inter-frame (B-frame) decoding.

[0150] Clause 18: The method according to any one of clauses 14 to 17, wherein the downscaled representation of the point cloud geometry includes the downscaled representation of the point cloud geometry for the current frame, and decoding the features includes extracting multi-scale features from a previously decoded frame of the point cloud data, forming predicted features for the current frame from the extracted multi-scale features and the downscaled representation of the point cloud geometry for the current frame by a predictor network, decoding residual data representing the difference between the original features of the current frame and the predicted features of the current frame, and applying the residual data to the predicted features to reproduce the original features.

[0151] Clause 19: The method according to clause 14, wherein determining the number of times to downscale the representation includes determining the number of times to downscale the representation according to at least one of a use case, a bit requirement, or a type of point cloud for the point cloud geometry.

[0152] Clause 20: The method according to clause 14, wherein determining the number of times to downscale the representation includes decoding data representing the number of times the representation of the point cloud geometry has been downscaled.

[0153] Clause 21: The method according to clause 14, wherein decoding the downscaled representation of the point cloud geometry includes decoding the downscaled representation of the point cloud geometry using one of intra-frame (I-frame) decoding, inter-frame (P-frame) decoding, or bi-directional inter-frame (B-frame) decoding.

[0154] Clause 22: The method according to clause 14, wherein the downscaled representation of the point cloud geometry includes the downscaled representation of the point cloud geometry for the current frame, and decoding the features includes extracting multi-scale features from previously decoded frames of the point cloud data, forming predicted features for the current frame from the extracted multi-scale features and the downscaled representation of the point cloud geometry for the current frame by a predictor network, decoding residual data representing the difference between the original features of the current frame and the predicted features of the current frame, and applying the residual data to the predicted features to reproduce the original features.

[0155] Clause 23: The method according to clause 14, wherein upscaling the representation of the point cloud geometry this number of times further includes using binary classification after each upscaling to prune false voxels and extract truly occupied voxels.

[0156] Clause 24: The method according to clause 14, further including encoding the downscaled representation and features of the point cloud geometry before decoding the downscaled representation and features of the point cloud geometry.

[0157] Clause 25: The method according to any one of clauses 14 to 18, further including encoding the downscaled representation and features of the point cloud geometry before decoding the downscaled representation and features of the point cloud geometry.

[0158] Clause 26: A device for coding point cloud data, the device comprising one or more means for performing the method according to any one of clauses 1 to 25.

[0159] Clause 27: The device according to clause 26, wherein one or more means include one or more processors implemented in a circuit.

[0160] Clause 28: The device according to clause 26 or 27, further comprising a display configured to display the decoded video data.

[0161] Clause 29: The device according to any one of clauses 26 to 28, wherein the device includes one or more of a camera, a computer, a mobile device, a broadcast receiver device, or a set-top box.

[0162] Clause 30: The device according to any one of clauses 26 to 29, further comprising a memory configured to store point cloud data.

[0163] Clause 31: A computer-readable storage medium that, when executed, stores instructions that cause a processor to execute the method according to any one of clauses 1 to 25.

[0164] Clause 32: A method for encoding point cloud data, comprising: determining the number of times to downscale the representation of the point cloud geometry; downscaling the representation of the point cloud geometry by this number of times to form a downscaled representation of the point cloud geometry; encoding the downscaled representation of the point cloud geometry; and outputting the encoded and downscaled representation of the point cloud geometry.

[0165] Clause 33: The method according to clause 32, wherein determining the number of times to downscale the representation includes determining the number of times to downscale the representation according to at least one of a use case, a bit requirement, or the type of point cloud for the point cloud geometry.

[0166] Clause 34: The method according to clause 32, wherein encoding a downscaled representation of point cloud geometry includes reversibly encoding the downscaled representation of point cloud geometry.

[0167] Clause 35: The method according to clause 32, further including encoding features corresponding to a downscaled representation of point cloud geometry using irreversible encoding.

[0168] Clause 36: The method according to clause 32, wherein encoding a downscaled representation of point cloud geometry includes encoding the downscaled representation of point cloud geometry using one of intra-frame (I-frame) encoding, inter-frame (P-frame) encoding, or bi-directional inter-frame (B-frame) encoding.

[0169] Clause 37: The method according to clause 32, wherein the downscaled representation of point cloud geometry includes the downscaled representation of point cloud geometry for the current frame, and encoding the features includes extracting multi-scale features from previously encoded frames of the point cloud data, forming predicted features for the current frame from the extracted multi-scale features and the downscaled representation of point cloud geometry for the current frame by a predictor network, and encoding residual data representing the difference between the features of the current frame and the predicted features of the current frame.

[0170] Clause 38: The method according to clause 32, further including encoding data representing the number of times the representation of point cloud geometry has been downscaled.

[0171] Clause 39: The method according to clause 32, wherein downscaling the representation of point cloud geometry and encoding the downscaled representation includes executing a neural network trained to downscale and encode the point cloud geometry in a number of stages equal to the number of times the representation of point cloud geometry is downscaled, each stage including a step of downscaling and a part of encoding.

[0172] Clause 40: Scaling down the representation of point cloud geometry and encoding the scaled-down representation includes executing a neural network trained to scale down the point cloud geometry in a number of stages equal to the number of times the representation of the point cloud geometry is scaled down, each stage including a step of scaling down and then encoding the scaled-down representation of the point cloud geometry, the method according to clause 32.

[0173] Clause 41: A device for encoding point cloud data, comprising a memory configured to store the point cloud data, and one or more processors implemented in a circuit and configured to determine the number of times to scale down the representation of the point cloud geometry, scale down the representation of the point cloud geometry this number of times to form a scaled-down representation of the point cloud geometry, encode the scaled-down representation of the point cloud geometry, and output the encoded and scaled-down representation of the point cloud geometry.

[0174] Clause 42: The device according to clause 41, wherein one or more processors are configured to determine the number of times to scale down the representation according to at least one of a use case, bit requirements, or the type of point cloud for the point cloud geometry.

[0175] Clause 43: The device according to clause 41, wherein one or more processors are configured to reversibly encode the scaled-down representation of the point cloud geometry for encoding the scaled-down representation of the point cloud geometry.

[0176] Clause 44: The device according to clause 41, wherein one or more processors are further configured to encode features corresponding to the scaled-down representation of the point cloud geometry using irreversible encoding.

[0177] Clause 45: The device according to Clause 41, wherein one or more processors are configured to encode a downscaled representation of point cloud geometry using one of intra-frame (I-frame) encoding, inter-frame (P-frame) encoding, or bi-directional inter-frame (B-frame) encoding.

[0178] Clause 46: The device according to Clause 41, wherein the downscaled representation of point cloud geometry includes the downscaled representation of point cloud geometry for the current frame, and in order to encode features, one or more processors extract multi-scale features from a previously coded frame of the point cloud data, execute a predictor network to form predicted features of the current frame from the extracted multi-scale features and the downscaled representation of point cloud geometry for the current frame, and code residual data representing the difference between the features of the current frame and the predicted features of the current frame.

[0179] Clause 47: The device according to Clause 41, wherein one or more processors are further configured to encode data representing the number of times the representation of the point cloud geometry has been downscaled.

[0180] Clause 48: The device according to Clause 41, wherein in order to downscale the representation of point cloud geometry and encode the downscaled representation, one or more processors execute a neural network trained to downscale and encode the point cloud geometry in a number of stages equal to the number of times the representation of the point cloud geometry is to be downscaled, each stage including a downscaling step and an encoding part.

[0181] Clause 49: To downscale the representation of point cloud geometry and encode the downscaled representation, one or more processors are configured to execute a neural network trained to downscale the point cloud geometry in a number of stages equal to the number of times the representation of the point cloud geometry is to be downscaled, each stage including a step of downscaling and then encoding the downscaled representation of the point cloud geometry, the device according to Clause 41.

[0182] Clause 50: A device for encoding point cloud data, comprising means for determining the number of times to downscale the representation of point cloud geometry, means for downscaling the representation of point cloud geometry this number of times to form a downscaled representation of point cloud geometry, means for encoding the downscaled representation of point cloud geometry, and means for outputting the encoded downscaled representation of point cloud geometry.

[0183] Clause 51: A method for decoding point cloud data, including determining the number of times to upscale the downscaled encoded representation of point cloud geometry, decoding the downscaled encoded representation of point cloud geometry, upscaling the downscaled representation of point cloud geometry this number of times to form an upscaled representation of point cloud geometry, and reproducing the point cloud using the upscaled representation of point cloud geometry.

[0184] Clause 52: The method according to Clause 51, wherein determining the number of times to upscale the downscaled representation includes determining the number of times to upscale the downscaled representation according to at least one of a use case, bit requirements, or the type of point cloud for the point cloud geometry.

[0185] Clause 53: The method according to Clause 51, wherein determining the number of times to upscale the downscaled representation includes decoding data representing the number of times the representation of point cloud geometry has been downscaled.

[0186] Clause 54: The method according to clause 51, wherein decoding the downscaled representation of point cloud geometry includes decoding the downscaled representation of point cloud geometry using one of intra-frame (I-frame) decoding, inter-frame (P-frame) decoding, or bi-directional inter-frame (B-frame) decoding.

[0187] Clause 55: The method according to clause 51, wherein the downscaled representation of point cloud geometry includes the downscaled representation of point cloud geometry for the current frame, and the method further includes extracting multi-scale features from previously decoded frames of point cloud data, forming predicted features for the current frame from the extracted multi-scale features and the downscaled representation of point cloud geometry for the current frame by a predictor network, decoding residual data representing the difference between the original features of the current frame and the predicted features of the current frame, and applying the residual data to the predicted features to reproduce the original features, and further includes decoding features corresponding to the downscaled representation of point cloud geometry.

[0188] Clause 56: The method according to clause 51, wherein upscaling the representation of point cloud geometry this number of times further includes using binary classification after each upscaling to prune false voxels and extract truly occupied voxels.

[0189] Clause 57: The method according to clause 51, wherein decoding the downscaled representation of point cloud geometry and upscaling the representation includes executing a neural network trained to decode and upscale point cloud geometry in a number of stages equal to the number of times the representation of point cloud geometry is upscaled, each stage including a step of decoding and a part of upscaling.

[0190] Clause 58: The method according to clause 51, comprising decrypting a downscaled representation of point cloud geometry and upscaling the downscaled representation by executing a neural network trained to decrypt a representation of point cloud geometry and then upscale the representation of point cloud geometry in a number of steps equal to the number of times the representation of point cloud geometry is upscaled.

[0191] Clause 59: A device for decrypting point cloud data, comprising a memory configured to store the point cloud data, and one or more processors implemented in a circuit and configured to determine the number of times to upscale a downscaled encoded representation of point cloud geometry, decrypt the downscaled encoded representation of point cloud geometry, upscale the representation of point cloud geometry by this number of times to form an upscaled representation of point cloud geometry, and reproduce the point cloud using the upscaled representation of point cloud geometry.

[0192] Clause 60: The device according to clause 59, wherein one or more processors are configured to determine the number of times to upscale a downscaled representation according to at least one of a use case, bit requirements, or the type of point cloud for point cloud geometry.

[0193] Clause 61: The device according to clause 59, wherein one or more processors are configured to decrypt data representing the number of times the representation of point cloud geometry has been downscaled to determine the number of times to upscale the downscaled representation.

[0194] Clause 62: To decode a downscaled representation of point cloud geometry, one or more processors are configured to decode the downscaled representation of point cloud geometry using one of intra-frame (I-frame) decoding, inter-frame (P-frame) decoding, or bi-directional inter-frame (B-frame) decoding, the device according to Clause 59.

[0195] Clause 63: The representation of point cloud geometry includes a downscaled representation of point cloud geometry for the current frame. To decode the downscaled representation of point cloud geometry, one or more processors extract multi-scale features from a previously decoded frame of point cloud data, execute a predictor network to form predicted features of the current frame from the extracted multi-scale features and the downscaled representation of point cloud geometry for the current frame, decode residual data representing the difference between the original features of the current frame and the predicted features of the current frame, and apply the residual data to the predicted features to reproduce the original features, the device according to Clause 59.

[0196] Clause 64: To upscale the representation of point cloud geometry this number of times, one or more processors are configured to prune false voxels and extract truly occupied voxels using binary classification after each upscaling, the device according to Clause 59.

[0197] Clause 65: To decode a downscaled representation of point cloud geometry and upscale the representation, one or more processors are configured to execute a neural network trained to decode and upscale point cloud geometry in a number of stages equal to the number of times the representation of point cloud geometry is upscaled, each stage including a step of decoding and a portion of upscaling, the device according to Clause 59.

[0198] Clause 66: The device according to Clause 59, wherein one or more processors are configured to execute a neural network trained to decode a downscaled representation of point cloud geometry and then upscale the point cloud geometry in a number of steps equal to the number of times the representation of the point cloud geometry is upscaled, to decode the downscaled representation of point cloud geometry and then upscale the downscaled representation of point cloud geometry.

[0199] Clause 67: A device for decoding point cloud data, comprising means for determining the number of times to upscale a downscaled encoded representation of point cloud geometry, means for decoding the downscaled encoded representation of point cloud geometry, means for upscaling the downscaled representation of point cloud geometry this number of times to form an upscaled representation of point cloud geometry, and means for reproducing the point cloud using the upscaled representation of point cloud geometry.

[0200] Clause 68: A method for encoding point cloud data, comprising determining the number of times to downscale a representation of point cloud geometry, downscaling the representation of point cloud geometry this number of times to form a downscaled representation of point cloud geometry, encoding the downscaled representation of point cloud geometry, and outputting the encoded downscaled representation of point cloud geometry.

[0201] Clause 69: The method according to Clause 68, wherein determining the number of times to downscale the representation includes determining the number of times to downscale the representation according to at least one of a use case, bit requirements, or the type of point cloud for the point cloud geometry.

[0202] Clause 70: The method according to Clause 68 or 69, wherein encoding the downscaled representation of point cloud geometry includes reversibly encoding the downscaled representation of point cloud geometry.

[0203] Clause 71: The method according to any one of Clauses 68 to 70, further comprising encoding features corresponding to a downscaled representation of point cloud geometry using irreversible quantization.

[0204] Clause 72: The method according to any one of Clauses 68 to 71, wherein encoding a downscaled representation of point cloud geometry includes encoding the downscaled representation of point cloud geometry using one of intra-frame (I-frame) encoding, inter-frame (P-frame) encoding, or bi-directional inter-frame (B-frame) encoding.

[0205] Clause 73: The method according to any one of Clauses 68 to 72, wherein the downscaled representation of point cloud geometry includes the downscaled representation of point cloud geometry for the current frame, and encoding the features includes extracting multi-scale features from a previously coded frame of the point cloud data, forming predicted features for the current frame from the extracted multi-scale features and the downscaled representation of point cloud geometry for the current frame by a predictor network, and coding residual data representing the difference between the features of the current frame and the predicted features of the current frame.

[0206] Clause 74: The method according to any one of Clauses 68 to 73, further comprising encoding data representing the number of times the representation of the point cloud geometry has been downscaled.

[0207] Clause 75: The method according to any one of Clauses 68 to 74, wherein downscaling the representation of the point cloud geometry and encoding the downscaled representation includes executing a neural network trained to downscale and encode the point cloud geometry in a number of stages equal to the number of times the representation of the point cloud geometry is downscaled, each stage including a step of downscaling and a part of encoding.

[0208] Clause 76: Downscaling the representation of point cloud geometry and encoding the downscaled representation includes executing a neural network trained to downscale the point cloud geometry in a number of stages equal to the number of times the representation of the point cloud geometry is downscaled, each stage including a step of downscaling and then encoding the downscaled representation of the point cloud geometry, according to any one of Clauses 68 to 74.

[0209] Clause 77: A device for encoding point cloud data, comprising a memory configured to store the point cloud data, and one or more processors implemented in circuitry and configured to determine the number of times to downscale the representation of the point cloud geometry, downscale the representation of the point cloud geometry that number of times to form a downscaled representation of the point cloud geometry, encode the downscaled representation of the point cloud geometry, and output the encoded downscaled representation of the point cloud geometry.

[0210] Clause 78: The device according to Clause 77, wherein one or more processors are configured to determine the number of times to downscale the representation according to at least one of a use case, bit requirements, or the type of point cloud for the point cloud geometry.

[0211] Clause 79: The device according to Clause 77 or 78, wherein one or more processors are configured to reversibly encode the downscaled representation of the point cloud geometry in order to encode the downscaled representation of the point cloud geometry.

[0212] Clause 80: The device according to any one of Clauses 77 to 79, wherein one or more processors are further configured to encode features corresponding to the downscaled representation of the point cloud geometry using irreversible encoding.

[0213] Clause 81: To encode a downscaled representation of point cloud geometry, one or more processors are configured to encode the downscaled representation of point cloud geometry using one of intra-frame (I-frame) encoding, inter-frame (P-frame) encoding, or bi-directional inter-frame (B-frame) encoding, as described in any one of Clauses 77 to 80 of the device.

[0214] Clause 82: The downscaled representation of point cloud geometry includes the downscaled representation of point cloud geometry for the current frame. To encode features, one or more processors extract multi-scale features from previously coded frames of point cloud data, execute a predictor network to form predicted features for the current frame from the extracted multi-scale features and the downscaled representation of point cloud geometry for the current frame, and code residual data representing the difference between the features of the current frame and the predicted features of the current frame, as described in any one of Clauses 77 to 81 of the device.

[0215] Clause 83: The one or more processors are further configured to encode data representing the number of times the representation of point cloud geometry has been downscaled, as described in any one of Clauses 77 to 82 of the device.

[0216] Clause 84: To downscale the representation of point cloud geometry and encode the downscaled representation, one or more processors are configured to execute a neural network trained to downscale and encode point cloud geometry in a number of stages equal to the number of times the representation of point cloud geometry is downscaled, where each stage includes a downscaling step and an encoding part, as described in any one of Clauses 77 to 83 of the device.

[0217] Clause 85: To downscale the representation of point cloud geometry and encode the downscaled representation, one or more processors are configured to execute a neural network trained to downscale the point cloud geometry in a number of stages equal to the number of times the representation of the point cloud geometry is to be downscaled, each stage including a step of downscaling and then encoding the downscaled representation of the point cloud geometry, the device according to any one of Clauses 77 to 83.

[0218] Clause 86: A device for encoding point cloud data, comprising means for determining the number of times to downscale the representation of the point cloud geometry, means for downscaling the representation of the point cloud geometry this number of times to form a downscaled representation of the point cloud geometry, means for encoding the downscaled representation of the point cloud geometry, and means for outputting the encoded downscaled representation of the point cloud geometry.

[0219] Clause 87: A method for decoding point cloud data, comprising determining the number of times to upscale the downscaled encoded representation of the point cloud geometry, decoding the downscaled encoded representation of the point cloud geometry, upscaling the downscaled representation of the point cloud geometry this number of times to form an upscaled representation of the point cloud geometry, and reproducing the point cloud using the upscaled representation of the point cloud geometry.

[0220] Clause 88: The method according to Clause 87, wherein determining the number of times to upscale the downscaled representation includes determining the number of times to upscale the downscaled representation according to at least one of a use case, bit requirements, or the type of point cloud for the point cloud geometry.

[0221] Clause 89: The method according to clause 87 or 88, wherein determining the number of times to upscale a downscaled representation includes decoding data representing the number of times the representation of the point cloud geometry has been downscaled.

[0222] Clause 90: The method according to any one of clauses 87 to 89, wherein decoding a downscaled representation of point cloud geometry includes decoding the downscaled representation of the point cloud geometry using one of intra-frame (I-frame) decoding, inter-frame (P-frame) decoding, or bi-directional inter-frame (B-frame) decoding.

[0223] Clause 91: The method according to any one of clauses 87 to 90, wherein the downscaled representation of the point cloud geometry includes the downscaled representation of the point cloud geometry for the current frame, and the method further includes extracting multi-scale features from a previously decoded frame of the point cloud data, forming predicted features of the current frame from the extracted multi-scale features and the downscaled representation of the point cloud geometry for the current frame by a predictor network, decoding residual data representing the difference between the original features of the current frame and the predicted features of the current frame, and applying the residual data to the predicted features to reproduce the original features, and further includes decoding features corresponding to the downscaled representation of the point cloud geometry.

[0224] Clause 92: The method according to any one of clauses 87 to 91, wherein upscaling the representation of the point cloud geometry this number of times further includes pruning false voxels and extracting true occupied voxels using binary classification after each upscaling.

[0225] Clause 93: Decoding a downscaled representation of point cloud geometry and upscaling the representation includes executing a neural network trained to decode and upscale the point cloud geometry in a number of stages equal to the number of times the representation of the point cloud geometry is upscaled, each stage including a step of decoding and a part of upscaling, the method according to any one of Clauses 87 to 92.

[0226] Clause 94: Decoding a downscaled representation of point cloud geometry and upscaling the downscaled representation includes executing a neural network trained to decode the representation of the point cloud geometry and then upscale the point cloud geometry in a number of stages equal to the number of times the representation of the point cloud geometry is upscaled, the method according to any one of Clauses 87 to 92.

[0227] Clause 95: A device for decoding point cloud data, comprising a memory configured to store the point cloud data, and one or more processors implemented in a circuit and configured to determine the number of times to upscale a downscaled encoded representation of the point cloud geometry, decode the downscaled encoded representation of the point cloud geometry, upscale the representation of the point cloud geometry by this number of times to form an upscaled representation of the point cloud geometry, and reproduce the point cloud using the upscaled representation of the point cloud geometry.

[0228] Clause 96: The device according to Clause 95, wherein one or more processors are configured to determine the number of times to upscale the downscaled representation according to at least one of a use case, bit requirements, or the type of point cloud for the point cloud geometry.

[0229] Clause 97: The device according to clause 95 or 96, wherein one or more processors are configured to decode data representing the number of times the representation of point cloud geometry has been downscaled in order to determine the number of times to upscale the downscaled representation.

[0230] Clause 98: The device according to any one of clauses 95 to 97, wherein one or more processors are configured to decode the downscaled representation of point cloud geometry using one of intra-frame (I-frame) decoding, inter-frame (P-frame) decoding, or bi-directional inter-frame (B-frame) decoding.

[0231] Clause 99: The representation of point cloud geometry includes the downscaled representation of point cloud geometry for the current frame. To decode the downscaled representation of point cloud geometry, one or more processors extract multi-scale features from previously decoded frames of point cloud data, execute a predictor network to form predicted features of the current frame from the extracted multi-scale features and the downscaled representation of point cloud geometry for the current frame, decode residual data representing the difference between the original features of the current frame and the predicted features of the current frame, and apply the residual data to the predicted features to reproduce the original features. The device according to any one of clauses 95 to 98.

[0232] Clause 100: The device according to any one of clauses 95 to 99, wherein one or more processors are configured to use binary classification after each upscaling to prune false voxels and extract truly occupied voxels in order to upscale the representation of point cloud geometry this number of times.

[0233] Clause 101: To decode a downscaled representation of point cloud geometry and upscale the representation, one or more processors are configured to execute a neural network trained to decode and upscale the point cloud geometry in a number of stages equal to the number of times the point cloud geometry is upscaled, each stage including a step of decoding and a portion of upscaling, the device according to any one of Clauses 95 to 100.

[0234] Clause 102: To decode a downscaled representation of point cloud geometry and upscale the downscaled representation, one or more processors are configured to execute a neural network trained to decode the point cloud geometry and then upscale the point cloud geometry in a number of stages equal to the number of times the point cloud geometry is upscaled, the device according to any one of Clauses 95 to 100.

[0235] Clause 103: A device for decoding point cloud data, comprising means for determining the number of times to upscale a downscaled encoded representation of point cloud geometry, means for decoding the downscaled encoded representation of point cloud geometry, means for upscaling the downscaled representation of point cloud geometry this number of times to form an upscaled representation of point cloud geometry, and means for reproducing the point cloud using the upscaled representation of point cloud geometry.

[0236] It should be recognized that, in some examples, some of the acts or events of any of the techniques described herein may be performed in a different order, may be added, merged, or completely excluded (e.g., not all acts or events described are necessary for the practice of the technique). Moreover, in some examples, the acts or events may not be performed sequentially, but may be performed, for example, in parallel through multi-threaded processing, interrupt processing, or multiple processors.

[0237] In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted via a computer-readable medium as one or more instructions or code and executed by a hardware-based processing unit. The computer-readable medium may include a computer-readable storage medium corresponding to a tangible medium such as a data storage medium, or a communication medium including any medium that facilitates transfer of a computer program from one place to another, for example, according to a communication protocol. Thus, the computer-readable medium generally may correspond to (1) a tangible computer-readable storage medium that is non-transitory, or (2) a communication medium such as a signal or carrier wave. The data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described in this disclosure. A computer program product may include a computer-readable medium.

[0238] By way of example and not limitation, such computer-readable storage media can include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or any other medium that can be used to store the desired program code in the form of instructions or data structures and that is accessible by a computer. Also, any connection can properly be called a computer-readable medium. For example, if the instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of the medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but instead are directed to non-transitory, tangible storage media. As used herein, disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc, where disk typically magnetically reproduces data, while disc optically reproduces data using a laser. Combinations of the above should also be included within the scope of computer-readable media.

[0239] The commands may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated logic circuit configurations or discrete logic circuit configurations. Accordingly, the terms "processor" and "processing circuit" as used herein may refer to either the foregoing structures, or any other structure suitable for implementation of the techniques described herein. Additionally, in some aspects, the functions described herein may be provided within dedicated hardware modules and / or software modules configured for encoding and decoding, or may be incorporated within a composite codec. Also, the techniques may be implemented entirely within one or more circuits or logic elements.

[0240] The techniques of the present disclosure may be implemented in a variety of devices or apparatuses, including wireless handsets, integrated circuits (ICs), or sets of ICs (e.g., chip sets). To emphasize the functional aspects of devices configured to implement the disclosed techniques, various components, modules, or units have been described in this disclosure, but they do not necessarily require realization by different hardware units. Rather, as described above, the various units may be combined in a codec hardware unit, or may be provided by a set of interoperable hardware units that include one or more of the processors described above in conjunction with suitable software and / or firmware.

[0241] Various examples have been described. These and other examples fall within the scope of the following claims.

Claims

1. A method for encoding point cloud data, Determining the amount of downscaling to apply to the representation of point cloud geometry, The representation of the point cloud geometry is downscaled by the determined amount to form a downscaled representation of the point cloud geometry. Encoding the downscaled representation of the point cloud geometry, Outputting the encoded downscaled representation of the point cloud geometry, Methods that include...

2. The method according to claim 1, wherein determining the amount of downscaling to apply to the representation includes determining the amount of downscaling to apply to the representation according to the bit requirements for the point cloud geometry.

3. Encoding the downscaled representation of the point cloud geometry is Reversibly encoding the downscaled representation of the point cloud geometry, or Encoding the downscaled representation of the point cloud geometry using one of the following: intra-frame (I-frame) coding, inter-frame (P-frame) coding, or bidirectional inter-frame (B-frame) coding. The method according to claim 1, including the method described in claim 1.

4. The method according to claim 1, further comprising encoding features corresponding to the downscaled representation of the point cloud geometry using lossy encoding.

5. The downscaled representation of the point cloud geometry includes a downscaled representation of the point cloud geometry for the current frame, and the method Extracting multiscale features from previously coded frames of point cloud data, The predictor network forms the predicted features of the current frame from the extracted multiscale features and the downscaled representation of the point cloud geometry for the current frame, Coding residual data representing the difference between the characteristics of the current frame and the predicted characteristics of the current frame, Further comprising encoding features corresponding to the downscaled representation of the point cloud geometry, including The method according to claim 1.

6. The method according to claim 1, further comprising encoding data representing the amount of downscaling to be applied to the point cloud geometry.

7. Downscaling the aforementioned representation of the point cloud geometry and encoding the downscaled representation are Downscaling and encoding the point cloud geometry in a number of steps equal to the amount of downscaling to apply to the representation of the point cloud geometry, wherein each step includes a step to downscale and a portion to encode, or Downscaling the point cloud geometry in a number of steps equal to the amount of downscaling to apply to the representation of the point cloud geometry, wherein each step includes a step of downscaling and then encoding the downscaled representation of the point cloud geometry. The method according to claim 1, comprising running a neural network trained to perform the following:

8. A device for encoding point cloud data, A memory configured to store point cloud data, Implemented in the circuit, Determine the amount of downscaling to apply to the representation of point cloud geometry, The representation of the point cloud geometry is downscaled by the determined amount to form a downscaled representation of the point cloud geometry. Encode the downscaled representation of the point cloud geometry, Outputs the encoded downscaled representation of the point cloud geometry. One or more processors configured as such, A device equipped with the following features.

9. A method for decoding point cloud data, Determining the amount of upscaling to apply to a downscaled encoded representation of point cloud geometry, Decoding the downscaled encoded representation of the point cloud geometry, The downscaled representation of the point cloud geometry is upscaled by a determined amount to form an upscaled representation of the point cloud geometry. Reproducing the point cloud using the upscaled representation of the point cloud geometry, Methods that include...

10. Determining the amount of upscaling to apply to the downscaled representation is Determining the amount of upscaling to apply to the downscaled representation according to the bit requirements for the point cloud geometry, or Decoding data representing the amount of upscaling to be applied to the aforementioned representation of the point cloud geometry. The method according to claim 9, including the method described in claim 9.

11. The method according to claim 9, wherein decoding the downscaled representation of the point cloud geometry includes decoding the downscaled representation of the point cloud geometry using one of intraframe (I-frame) decoding, interframe (P-frame) decoding, or bidirectional interframe (B-frame) decoding.

12. The downscaled representation of the point cloud geometry includes a downscaled representation of the point cloud geometry for the current frame, and the method Extracting multiscale features from previously decoded frames of point cloud data, The predictor network forms the predicted features of the current frame from the extracted multiscale features and the downscaled representation of the point cloud geometry for the current frame, Decoding residual data representing the difference between the original features of the current frame and the predicted features of the current frame, Applying the residual data to the predicted features to reproduce the original features, Further comprising decoding features corresponding to the downscaled representation of the point cloud geometry, including The method according to claim 9.

13. The method according to claim 9, further comprising upscaling the representation of the point cloud geometry by a determined amount, pruning false voxels and extracting true occupied voxels using binary classification after each upscaling.

14. Decoding the downscaled representation of the point cloud geometry and upscaling the representation are performed. Decoding and upscaling the point cloud geometry in a number of steps equal to the amount of upscaling to apply to the representation of the point cloud geometry, wherein each step includes a decoding step and a portion to be upscaled, or Decode the representation of the point cloud geometry, and then upscale the point cloud geometry in a number of steps equal to the amount of upscaling to apply to the representation of the point cloud geometry. The method according to claim 9, comprising running a neural network trained to do the following.

15. A device for decoding point cloud data, A memory configured to store point cloud data, Implemented in the circuit, Determine the amount of upscaling to apply to the downscaled encoded representation of point cloud geometry. Decode the downscaled encoded representation of the point cloud geometry, The representation of the point cloud geometry is upscaled by the determined amount to form the upscaled representation of the point cloud geometry. The point cloud is reproduced using the upscaled representation of the point cloud geometry. One or more processors configured as such, A device equipped with the following features.

Images