Geometric coordinate scaling for AI-based dynamic point cloud coding

Abstract

An example apparatus for decoding point cloud data includes a memory configured to store point cloud data; and one or more processors implemented in the circuitry and configured to: determine a number of times the reduced encoded representation of the point cloud geometry is amplified; decoding the reduced encoded representation of the point cloud geometry; magnifying the reduced representation of the point cloud geometry by the number of times to form a magnified representation of the point cloud geometry; and reproducing a point cloud using the enlarged representation of the point cloud geometry.

Description

[0001] This application claims the benefit of U.S. Patent Application No. 18 / 318,498, filed on May 16, 2023, which claims the benefit of U.S. Provisional Application No. 63 / 364,862, filed on May 17, 2022, the entire contents of which are hereby incorporated by reference into this document. Technical Field

[0002] The present disclosure relates to point cloud coding including point cloud encoding and point cloud decoding. Background Art

[0003] Point Cloud (PC) is a 3D data representation of a three-dimensional (3D) space. PC can be used for tasks such as extended reality (XR), augmented reality (AR), virtual reality (VR) and mixed reality (MR), autonomous driving or cultural heritage. PC is a set of points in 3D space, represented by their 3D coordinates (x, y, z) called geometry. Each point can also be associated with multiple attributes such as color, normal vector or reflectivity.

[0004] Depending on the target application and the PC acquisition method, PCs can be classified as point cloud scenes or point cloud objects. Point cloud scenes can be captured using LiDAR sensors and are often acquired dynamically. Point cloud objects can be further subdivided into static point clouds and dynamic point clouds. Static PCs are single objects, while dynamic PCs are time-varying PCs, where each instance of a dynamic PC is a static PC. Dynamic time-varying PCs can be used for AR / VR, volumetric video streaming, and telepresence, and can be generated using 3D models (i.e., CGI) or captured from real-world scenes using various methods (such as multiple cameras surrounding an object with a depth sensor). These PCs are dense photorealistic point clouds that can have a large number of points, especially in high-precision or large-scale captures (up to 60 frames per second (FPS), millions of points per frame). Therefore, efficient point cloud compression (PCC) is particularly important for achieving practical uses in VR and MR applications. Summary of the invention

[0005] In general, the present disclosure describes a flexible configuration of a system in which flexible reduction amounts can be applied instead of a fixed cubic reduction / downsampling representation. That is, the point cloud encoder and point cloud decoder can be configured to reduce / amplify the point cloud geometry data a variable number of times. The point cloud encoder can signal a value in the point cloud bitstream indicating the number of times the point cloud geometry was reduced, so that the point cloud decoder can amplify the decoded point cloud geometry data by the same factor. Depending on the use case, bit requirements, and point cloud type, the flexible reduction amount can achieve better compression. The flexible configuration can be used for intra-point cloud compression for static and dynamic point clouds and inter-point cloud compression for dynamic point clouds.

[0006] In one example, a method for encoding point cloud data includes: determining a number of times to reduce a representation of a point cloud geometry; reducing the representation of the point cloud geometry the number of times to form a reduced representation of the point cloud geometry; encoding the reduced representation of the point cloud geometry; and outputting the encoded reduced representation of the point cloud geometry.

[0007] In another example, a device for encoding point cloud data includes: a memory configured to store point cloud data; and one or more processors implemented in a circuit system and configured to: determine a number of times to reduce a representation of a point cloud geometry; reduce the representation of the point cloud geometry the number of times to form a reduced representation of the point cloud geometry; encode the reduced representation of the point cloud geometry; and output the encoded reduced representation of the point cloud geometry.

[0008] In another example, an apparatus for encoding point cloud data includes: a component for determining a number of times to reduce a representation of a point cloud geometry; a component for reducing the representation of the point cloud geometry the number of times to form a reduced representation of the point cloud geometry; a component for encoding the reduced representation of the point cloud geometry; and a component for outputting the encoded reduced representation of the point cloud geometry.

[0009] In another example, a method for decoding point cloud data includes: determining a number of times to magnify a reduced encoded representation of a point cloud geometry; decoding the reduced encoded representation of the point cloud geometry; magnifying the reduced representation of the point cloud geometry the number of times to form an enlarged representation of the point cloud geometry; and reproducing a point cloud using the enlarged representation of the point cloud geometry.

[0010] In another example, a device for decoding point cloud data includes: a memory configured to store the point cloud data; and one or more processors implemented in a circuit system and configured to: determine a number of times to magnify a reduced encoded representation of a point cloud geometry; decode the reduced encoded representation of the point cloud geometry; magnify the reduced representation of the point cloud geometry the number of times to form an enlarged representation of the point cloud geometry; and reproduce a point cloud using the enlarged representation of the point cloud geometry.

[0011] In another example, an apparatus for decoding point cloud data includes: means for determining a number of times to magnify a reduced encoded representation of a point cloud geometry; means for decoding the reduced encoded representation of the point cloud geometry; means for magnifying the reduced representation of the point cloud geometry the number of times to form an magnified representation of the point cloud geometry; and means for reproducing a point cloud using the magnified representation of the point cloud geometry.

[0012] The details of one or more examples are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a block diagram illustrating an example point cloud encoding and decoding system that may perform the techniques of this disclosure.

[0014] Figure 2 is a block diagram illustrating an example point cloud encoder.

[0015] Figure 3 is a block diagram illustrating an example point cloud decoder.

[0016] Figure 4 is a conceptual diagram illustrating an example system model for performing the techniques of this disclosure.

[0017] Figure 5 is a conceptual diagram illustrating example encoder and decoder networks.

[0018] Figure 6 is a block diagram illustrating example pruning layers of a decoder in accordance with techniques of this disclosure.

[0019] Figure 7 is a conceptual diagram illustrating an example prediction network.

[0020] Figure 8 is a conceptual diagram illustrating an example convolution on two-dimensional (2D) target coordinates.

[0021] Fig. 9 and Fig.10 is a graph showing rate-distortion curves comparing the technology of the present disclosure with other conventional technologies.

[0022] Fig.11 is a conceptual diagram illustrating an example system for intra point cloud encoding based on intra (I) frames.

[0023] Fig.12 is a conceptual diagram illustrating an example system for bidirectional intra-prediction (B) frame encoding.

[0024] Fig.13is a graph showing a rate-distortion curve for the disclosed technique.

[0025] Fig.14 is a flow chart illustrating an example method of encoding point cloud data according to the techniques of this disclosure.

[0026] Fig.15 is a flow chart illustrating an example method of decoding point cloud data according to techniques of this disclosure. DETAILED DESCRIPTION

[0027] Efficient point cloud compression is critical for applications such as virtual and mixed reality, autonomous driving, and cultural heritage. Point cloud compression for dense dynamic point clouds can be performed using a deep learning network including an encoding unit and a decoding unit. The encoding unit can extract features from the point cloud geometry into a cubically reduced point cloud geometry with corresponding feature embeddings. The cubically reduced geometry and the corresponding features can be sent to the decoding unit. The decoding unit can then hierarchically reconstruct the original point cloud geometry from the reduced representation using progressive rescaling.

[0028] The present disclosure describes a flexible configuration of the system where a flexible amount of reduction can be applied instead of using a cubic reduction / downsampling representation. Depending on the use case, bit requirements, and point cloud type, the flexible amount of reduction can achieve better compression. The flexible configuration can be used for intra-point cloud compression for static and dynamic point clouds and inter-point cloud compression for dynamic point clouds.

[0029] Figure 1 is a block diagram illustrating an example encoding and decoding system 100 that can perform the techniques of the present disclosure. The techniques of the present disclosure are generally related to decoding (encoding and / or decoding) point cloud data, that is, supporting point cloud compression. Generally speaking, point cloud data includes any data used to process point clouds. The decoding can be effective in compressing and / or decompressing point cloud data.

[0030] like Figure 1 As shown, the system 100 includes a source device 102 and a destination device 116. The source device 102 provides encoded point cloud data to be decoded by the destination device 116. Figure 1In the example of , source device 102 provides point cloud data to destination device 116 via computer-readable medium 110. Source device 102 and destination device 116 may include any of a wide range of devices, including desktop computers, notebook (i.e., laptop) computers, tablet computers, set-top boxes, telephone handsets (such as smartphones), televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, ground or sea vehicles, spacecraft, aircraft, robots, LIDAR devices, satellites, etc. In some cases, source device 102 and destination device 116 may be equipped for wireless communication.

[0031] exist Figure 1 In the example of , source device 102 includes data source 104, memory 106, point cloud encoder 200, and output interface 108. Destination device 116 includes input interface 122, point cloud decoder 300, memory 120, and data consumer 118. Thus, source device 102 represents an example of an encoding device, while destination device 116 represents an example of a decoding device. In other examples, source device 102 and destination device 116 may include other components or arrangements. For example, source device 102 may receive data (e.g., point cloud data) from an internal source or an external source. Similarly, destination device 116 may interface with an external data consumer without including the data consumer in the same device.

[0032] like Figure 1 The system 100 shown is only one example. The source device 102 and the destination device 116 are only examples of such devices, where the source device 102 generates decoded data for transmission to the destination device 116. The present disclosure refers to a "decoding" device as a device that performs decoding (e.g., encoding and / or decoding) of data. Thus, the point cloud encoder 200 and the point cloud decoder 300 represent examples of decoding devices, specifically, an encoder and a decoder, respectively. In some examples, the source device 102 and the destination device 116 can operate in a substantially symmetrical manner, such that each of the source device 102 and the destination device 116 includes an encoding component and a decoding component. Thus, the system 100 can support one-way or two-way transmission between the source device 102 and the destination device 116, for example, for streaming, playback, broadcasting, telephony, navigation, and other applications.

[0033] Generally speaking, the data source 104 represents a source of data (i.e., raw unencoded point cloud data) and may provide a series of sequential "frames" of data to the point cloud encoder 200, which encodes the data of these frames. The data source 104 of the source device 102 may include a point cloud capture device such as any of a variety of cameras or sensors (e.g., a 3D scanner or light detection and ranging (LIDAR) device, one or more cameras), an archive containing previously captured data, and / or a data feed interface that receives data from a data content provider. Additionally or alternatively, the point cloud data may be computer-generated from a scanner, camera, sensor, or other data. For example, the data source 104 may generate computer graphics-based data as source data, or produce a combination of real-time data, archived data, and computer-generated data. In each case, the point cloud encoder 200 encodes the captured data, pre-captured data, or computer-generated data. The point cloud encoder 200 may rearrange the frames from the order in which they are received (sometimes referred to as "display order") to a decoding order for decoding. The point cloud encoder 200 may generate one or more bitstreams including the encoded data. Source device 102 may then output the encoded data onto computer-readable medium 110 via output interface 108 for receipt and / or retrieval by, for example, input interface 122 of destination device 116 .

[0034] The memory 106 of the source device 102 and the memory 120 of the destination device 116 can represent general purpose memory. In some examples, the memory 106 and the memory 120 can store raw data, for example, raw data from the data source 104 and raw decoded data from the point cloud decoder 300. Additionally or alternatively, the memory 106 and the memory 120 can store software instructions that can be executed by, for example, the point cloud encoder 200 and the point cloud decoder 300, respectively. Although the memory 106 and the memory 120 are shown separately from the point cloud encoder 200 and the point cloud decoder 300 in this example, it should be understood that the point cloud encoder 200 and the point cloud decoder 300 can also include internal memory for functionally similar or equivalent purposes. In addition, the memory 106 and the memory 120 can store encoded data, for example, output from the point cloud encoder 200 and input to the point cloud decoder 300. In some examples, portions of the memory 106 and the memory 120 can be allocated as one or more buffers, for example, for storing raw decoded and / or encoded data. For example, memory 106 and memory 120 may store data representing a point cloud.

[0035] The computer-readable medium 110 may represent any type of medium or device capable of transmitting the encoded data from the source device 102 to the destination device 116. In one example, the computer-readable medium 110 represents a communication medium for enabling the source device 102 to transmit the encoded data directly to the destination device 116 in real time, for example, via a radio frequency network or a computer-based network. The output interface 108 may modulate a transmit signal including the encoded data, and the input interface 122 may demodulate the received transmit signal according to a communication standard, such as a wireless communication protocol. The communication medium may include any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network such as a local area network, a wide area network, or a global network such as the Internet. The communication medium may include a router, a switch, a base station, or any other equipment that may be useful for facilitating communication from the source device 102 to the destination device 116.

[0036] In some examples, source device 102 may output the encoded data from output interface 108 to storage device 112. Similarly, destination device 116 may access the encoded data from storage device 112 via input interface 122. Storage device 112 may include any of a variety of distributed or locally accessed data storage media, such as a hard drive, Blu-ray disc, DVD, CD-ROM, flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded data.

[0037] In some examples, source device 102 may output the encoded data to file server 114 or another intermediate storage device that may store the encoded data generated by source device 102. Destination device 116 may access the stored data from file server 114 via streaming or downloading. File server 114 may be any type of server device capable of storing the encoded data and sending the encoded data to destination device 116. File server 114 may represent a web server (e.g., for a website), a file transfer protocol (FTP) server, a content delivery network device, or a network attached storage (NAS) device. Destination device 116 may access the encoded data from file server 114 via any standard data connection including an Internet connection. This may include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., a digital subscriber line (DSL), a cable modem, etc.) suitable for accessing the encoded data stored on file server 114, or a combination of the two. File server 114 and input interface 122 may be configured to operate according to a streaming transmission protocol, a download transmission protocol, or a combination thereof.

[0038] Output interface 108 and input interface 122 may represent wireless transmitters / receivers, modems, wired networking components (e.g., Ethernet cards), wireless communication components that operate according to any of the various IEEE 802.11 standards, or other physical components. In examples where output interface 108 and input interface 122 include wireless components, output interface 108 and input interface 122 may be configured to communicate data (such as encoded data) according to a cellular communication standard (such as 4G, 4G-LTE (Long Term Evolution), Advanced LTE, 5G, etc.). In some examples where output interface 108 includes a wireless transmitter, output interface 108 and input interface 122 may be configured to communicate data (such as encoded data) according to other wireless standards such as the IEEE 802.11 specification, the IEEE 802.15 specification (e.g., ZigBee 5G, etc.). TM ),Bluetooth TM Standards, etc. to transfer data (such as encoded data). In some examples, source device 102 and / or destination device 116 may include corresponding system-on-chip (SoC) devices. For example, source device 102 may include a SoC device for performing functionality belonging to point cloud encoder 200 and / or output interface 108, and destination device 116 may include a SoC device for performing functionality belonging to point cloud decoder 300 and / or input interface 122.

[0039] The techniques of this disclosure may be applicable to encoding and decoding to support any of a variety of applications, such as communications between autonomous vehicles, communications between scanners, cameras, sensors and processing devices such as local or remote servers, geographic mapping, or other applications.

[0040] An input interface 122 of a destination device 116 receives an encoded bitstream from a computer-readable medium 110 (e.g., a communication medium, a storage device 112, a file server 114, etc.). The encoded bitstream may include signaling information defined by the point cloud encoder 200 and also used by the point cloud decoder 300, such as syntax elements with values ​​describing characteristics and / or processing of coded units (e.g., slices, pictures, groups of pictures, sequences, etc.). The decoded data is used by a data consumer 118. For example, the data consumer 118 may use the decoded data to determine a location of a physical object. In some examples, the data consumer 118 may include a display for presenting an image based on the point cloud.

[0041] The point cloud encoder 200 and the point cloud decoder 300 can each be implemented as any of a variety of suitable encoder and / or decoder circuit systems, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. When the technology is partially implemented in software, the device can store instructions for the software in an appropriate non-transitory computer-readable medium and execute the instructions in hardware using one or more processors to perform the technology of the present disclosure. Each of the point cloud encoder 200 and the point cloud decoder 300 can be included in one or more encoders or decoders, any of which can be integrated as part of a combined encoder / decoder (codec) in the corresponding device. A device including the point cloud encoder 200 and / or the point cloud decoder 300 may include one or more integrated circuits, microprocessors, and / or other types of devices.

[0042] The point cloud encoder 200 and the point cloud decoder 300 may operate in accordance with a coding standard such as the Video Point Cloud Compression (V-PCC) standard or the Geometry Point Cloud Compression (Point Cloud) standard. The present disclosure may generally relate to the coding (e.g., encoding and decoding) of a picture, thereby including the process of encoding or decoding data. The encoded bitstream typically includes a series of values ​​for syntax elements that represent coding decisions (e.g., decoding modes).

[0043] The present disclosure may generally involve "signaling" certain information (such as syntax elements). The term "signaling" may generally refer to the communication of values ​​for syntax elements and / or other data used to decode encoded data. That is, the point cloud encoder 200 may signal the values ​​of syntax elements in a bitstream. Generally speaking, signaling refers to generating values ​​in a bitstream. As noted above, the source device 102 may transmit the bitstream to the destination device 116 in substantially real time or in non-real time (such as may occur when storing syntax elements to the storage device 112 for later retrieval by the destination device 116).

[0044] ISO / IEC MPEG (JTC 1 / SC 29 / WG 11) is investigating the potential need for standardization of point cloud coding technology with significantly greater compression capabilities than current approaches and will work to create such a standard. The group is working together in a collaboration called the 3D Graphics Group (3DG) to evaluate compression technology designs proposed by experts in the field.

[0045] Point cloud compression activities are categorized into two different approaches. The first approach is "Video Point Cloud Compression" (V-PCC), which segments 3D objects and projects these segments into multiple 2D planes (which are represented as "patches" in 2D frames), which are further decoded by legacy 2D video codecs such as the High Efficiency Video Coding (HEVC) (ITU-T H.265) codec. The second approach is "Geometry-based Point Cloud Compression" (Point Cloud), which directly compresses 3D geometry (i.e., the position of a set of points in 3D space) and associated attribute values ​​(for each point associated with the 3D geometry). Point Cloud addresses the compression of point clouds in both category 1 (static point clouds) and category 3 (dynamically acquired point clouds). The latest draft of the Point Cloud standard is available in Point Cloud DIS (ISO / IEC JTC1 / SC29 / WG11 w19088, Brussels, Belgium, January 2020) and the specification of the codec is available in Point Cloud Codec Specification v6 (ISO / IEC JTC1 / SC29 / WG11 w19091, Brussels, Belgium, January 2020).

[0046] A point cloud contains a set of points in 3D space and may have attributes associated with the points. The attributes may be color information such as R, G, B or Y, Cb, Cr, or reflectivity information, or other attributes. Point clouds may be captured by various cameras or sensors (such as LIDAR sensors and 3D scanners), and may also be computer generated. Point cloud data is used in a variety of applications, including but not limited to architecture (modeling), graphics (3D models for visualization and animation), and the automotive industry (LIDAR sensors to aid navigation).

[0047] The 3D space occupied by the point cloud data can be surrounded by a virtual bounding box. The position of a point in the bounding box can be represented by a certain precision; therefore, the position of one or more points can be quantized based on the precision. At the smallest level, the bounding box is divided into voxels, which are the smallest spatial units represented by a unit cube. A voxel in a bounding box can be associated with zero, one, or more than one point. The bounding box can be divided into multiple cubic / cuboid regions, which can be referred to as tiles. Each tile can be decoded into one or more slices. The division of the bounding box into slices and tiles can be based on the number of points in each partition, or based on other considerations (for example, a particular area can be decoded as a tile). The slice area can be further divided using split decisions similar to those in video codecs.

[0048] Figure 2 An overview of the point cloud encoder 200 is provided. Figure 3An overview of the point cloud decoder 300 is provided. The modules shown are logical and do not necessarily correspond one-to-one with the implemented code in the reference implementation of the point cloud codec (ie, the TMC13 test model software developed by ISO / IECMPEG (JTC 1 / SC 29 / WG 11)).

[0049] In some cases, the compressed geometry can be represented as an octree from the root all the way down to the leaf level of individual voxels. In some cases, the compressed geometry can be represented by a pruned octree (i.e., an octree from the root to the leaf level of blocks larger than a voxel) plus a model that approximates the surface inside each leaf of the pruned octree. In this way, various types of data can share the octree decoding mechanism, and in some cases, the data can utilize a surface model to approximate the voxels within each leaf. The surface model used is a triangulation that includes 1-10 triangles per block, forming a triangle soup.

[0050] At each node of the octree, occupancy is signaled for one or more of its child nodes (up to eight nodes) (when not inferred). Multiple neighborhoods are specified, including (a) nodes that share faces with the current octree node, (b) nodes that share faces, edges, or vertices with the current octree node, etc. Within each neighborhood, the occupancy of the node and / or its child nodes can be used to predict the occupancy of the current node or its child nodes. For points that are sparsely populated in certain nodes of the octree, the codec also supports a direct decoding mode in which the 3D position of the point is encoded directly. A flag can be signaled to indicate that the direct mode is signaled. At the lowest level, the number of points associated with the octree node / leaf node can also be decoded.

[0051] Once the geometry is decoded, the attributes corresponding to the geometry point are decoded. When there are multiple attribute points corresponding to one reconstructed / decoded geometry point, the attribute value representing the reconstructed point can be derived.

[0052] There are three methods for attribute decoding in point clouds: Region Adaptive Hierarchical Transform (RAHT) decoding, interpolation-based hierarchical nearest neighbor prediction (prediction transform), and interpolation-based hierarchical nearest neighbor prediction with an update / lifting step (lifting transform). RAHT and lifting are typically used for class 1 data, while prediction is typically used for class 3 data. However, either method can be used for any data, and just like the geometry codec in point clouds, the attribute decoding method used to decode the point cloud is specified in the bitstream.

[0053] The decoding of attributes can be performed in levels of detail (LOD), where each level of detail can obtain a more refined representation of the point cloud attributes. Each level of detail can be specified based on a distance metric to neighboring nodes or based on a sampling distance.

[0054] At the point cloud encoder 200, the residual obtained as the output of the attribute coding method can be quantized. The residual can be obtained by subtracting the attribute value from a prediction derived based on points in the neighborhood of the current point and based on the attribute value of the previously encoded point. The quantized residual can be decoded using context adaptive arithmetic decoding.

[0055] exist Figure 2 In the example, the point cloud encoder 200 includes a reduction and encoding unit 202, a reconstruction unit 206, a color transformation unit 204, an attribute transfer unit 208, a RAHT unit 218, an LOD generation unit 220, a lifting unit 222, a coefficient quantization unit 224 and an arithmetic coding unit 226.

[0056] like Figure 2 As shown in the example of , the point cloud encoder 200 can obtain a location set and an attribute set of points in the point cloud. The point cloud encoder 200 can obtain the location set and the attribute set of points in the point cloud from the data source 104 ( Figure 1 ) obtains a set of positions and a set of attributes of points in the point cloud. The positions may include coordinates of the points in the point cloud. The attributes may include information about the points in the point cloud, such as a color associated with the points in the point cloud. The point cloud encoder 200 may generate a geometry bitstream 203 including encoded representations of the positions of the points in the point cloud. The point cloud encoder 200 may also generate an attribute bitstream 205 including encoded representations of the attribute sets.

[0057] The reduction and encoding unit 202 may generally reduce the received geometry data by a certain amount, for example, along the X-axis, the Y-axis, and / or the Z-axis, and encode the reduced geometry data. In some examples, the reduction and encoding unit 202 may include a series of reduction and encoding task unit sets, each of which, as a unit, reduces the geometry data by a certain amount and encodes the geometry data, and then passes the reduced and encoded geometry data to a subsequent task unit. In other examples, the reduction and encoding unit 202 may include a series of reduction stages, each of which shortens the geometry data, and includes a set of one or more encoding task units that encode the reduced geometry data.

[0058] The reduction and encoding unit 202 may reduce the input geometry data (e.g., octree) by a factor of 8: along the X-axis by a factor of 2, along the Y-axis by a factor of 2, and along the Z-axis by a factor of 2. Such reduction may be performed by an artificial intelligence / machine learning (AI / ML) unit, such as a neural network. Additional details regarding examples of such reduction are discussed below.

[0059] The color transformation unit 204 may apply a transformation to transform the color information of the attribute to a different domain. For example, the color transformation unit 204 may transform the color information from an RGB color space to a YCbCr color space.

[0060] The reconstruction unit 206 may decode the geometric structure data and reconstruct the geometric structure data. The attribute transfer unit 208 may transfer the attributes of the original points in the point cloud to the reconstructed points in the point cloud.

[0061] In addition, the RAHT unit 218 may apply RAHT decoding to the attributes of the reconstruction point. In some examples, according to RAHT, the attributes of the block of 2x2x2 point positions are obtained and transformed in one direction to obtain four low-frequency nodes (L) and four high-frequency nodes (H). Subsequently, the four low-frequency nodes (L) are transformed in the second direction to obtain two low-frequency nodes (LL) and two high-frequency nodes (LH). The two low-frequency nodes (LL) are transformed in the third direction to obtain one low-frequency node (LLL) and one high-frequency node (LLH). The low-frequency node LLL corresponds to the DC coefficient, and the high-frequency nodes H, LH and LLH correspond to the AC coefficient. The transformation in each direction may be a 1-D transformation with two coefficient weights. The low-frequency coefficients may be obtained as coefficients for the next higher-level 2x2x2 block for the RAHT transform, and the AC coefficients may be encoded without change; such transformations continue until the top root node. The weights to be used for these coefficients are calculated using a tree traversal for encoding from top to bottom; the transformation order is from bottom to top. Then, these coefficients may be quantized and decoded.

[0062] Additionally or alternatively, the LOD generation unit 220 and the lifting unit 222 may apply LOD processing and lifting to the attributes of the reconstructed points, respectively. LOD generation is used to split the attributes into different refinement levels. Each refinement level provides a refinement of the attributes of the point cloud. The first refinement level provides a rough approximation and contains few points; subsequent refinement levels typically contain more points, and so on. The refinement level can be constructed using a distance-based metric, or one or more other classification criteria (e.g., subsampling from a specific order) can also be used. Therefore, all reconstructed points can be included in the refinement level. Each level of detail is generated by taking the union of all points up to a specific refinement level: for example, LOD1 is obtained based on the refinement level RL1, LOD2 is obtained based on RL1 and RL2, ..., and LODN is obtained by the union of RL1, RL2, ..., RLN. In some cases, LOD generation can be followed by a prediction scheme (e.g., a predictive transform) in which the attributes associated with each point in the LOD are predicted based on a weighted average of previous points and the residual is quantized and entropy decoded. The lifting scheme is built on top of the predictive transform mechanism, in which update operators are used to update coefficients and adaptive quantization of coefficients is performed.

[0063] RAHT unit 218 and lifting unit 222 may generate coefficients based on these attributes. Coefficient quantization unit 224 may quantize the coefficients generated by RAHT unit 218 or lifting unit 222. Arithmetic coding unit 226 may apply arithmetic coding to syntax elements representing the quantized coefficients. Point cloud encoder 200 may output these syntax elements in attribute bitstream 205. Attribute bitstream 205 may also include other syntax elements, including syntax elements that are not arithmetically coded.

[0064] exist Figure 3 In the example of , the point cloud decoder 300 includes a decoding and upscaling unit 302, an attribute arithmetic decoding unit 304, an inverse quantization unit 308, a RAHT unit 314, a LoD generation unit 316, an inverse lifting unit 318, an inverse transformation coordinate unit 320 and an inverse transformation color unit 322.

[0065] The point cloud decoder 300 may obtain the geometry bitstream 203 and the attribute bitstream 205. The decoding and amplification unit 302 of the decoder 300 may decode a value indicating an amount of amplification to be applied to the geometry data. In addition, the decoding and amplification unit 302 may decode the geometry data and amplify the geometry data, wherein the amplification may correspond to the decoded value. In some examples, the decoding and amplification unit 302 may include a sequence of unit sets, the sequence of unit sets including both decoding units and amplification units, wherein the number of sets is equal to the decoded value representing the amplification. In some examples, the decoding and amplification unit 302 may include one or more decoding units, and then include a sequence of amplification unit sets, wherein the number of sets is equal to the decoded value representing the amplification. In some examples, the decoding and amplification unit 302 may further use the decoded and amplified point cloud geometry data to reconstruct the point cloud geometry.

[0066] The attribute arithmetic decoding unit 304 may apply arithmetic decoding to the syntax elements in the attribute bitstream 205 .

[0067] The octree synthesis unit 306 can synthesize an octree based on the syntax elements parsed from the geometry bitstream 203. Starting from the root node of the octree, the occupancy of each of the eight child nodes at each octree level is signaled in the bitstream. When the signaling indicates that a child node at a specific octree level is occupied, the occupancy of the child nodes of the child node is signaled. Before advancing to the subsequent octree level, the signaling of the node at each octree level is signaled. At the last level of the octree, each node corresponds to a voxel position; when a leaf node is occupied, one or more points can be designated as occupied at the voxel position. In some instances, due to quantization, some branches of the octree may terminate earlier than the final level. In such cases, a leaf node is considered to be an occupied node that has no child nodes.

[0068] The inverse transform coordinate unit 320 can apply an inverse transform to the reconstructed coordinates to convert the reconstructed coordinates (positions) of the points in the point cloud from the transformed domain back to the original domain. The positions of the points in the point cloud may be in the floating point domain, but the point positions in the point cloud codec are decoded in the integer domain. An inverse transform can be used to convert these positions back to the original domain.

[0069] Additionally, in Figure 3 In the example of , the inverse quantization unit 308 may inverse quantize the property value. The property value may be based on syntax elements obtained from the property bitstream 205 (eg, including syntax elements decoded by the property arithmetic decoding unit 304).

[0070] Depending on how the attribute value is encoded, the RAHT unit 314 may perform RAHT decoding to determine the color value of a point in the point cloud based on the inverse quantized attribute value. RAHT decoding is performed from the top to the bottom of the tree. At each level, the component values ​​are derived using the low-frequency coefficients and high-frequency coefficients derived from the inverse quantization process. At the leaf node, the derived value corresponds to the attribute value of the coefficient. The weight derivation process of the point is similar to the process used at the point cloud encoder 200. Alternatively, the LOD generation unit 316 and the inverse lifting unit 318 may use a detail level-based technique to determine the color value of a point in the point cloud. The LOD generation unit 316 decodes each LOD, thereby giving a progressively finer representation of the attribute of the point. In the case of a predictive transform, the LOD generation unit 316 derives a prediction of the point from the weighted sum of the points previously reconstructed in the same LOD or previously in the previous LOD. The LOD generation unit 316 may add the prediction to the residual (which is obtained after inverse quantization) to obtain the reconstructed value of the attribute. When a lifting scheme is used, the LOD generation unit 316 may also include an update operator to update the coefficients used to derive the property value. In this case, the LOD generation unit 316 may also apply inverse adaptive quantization.

[0071] In addition, Figure 3 In the example of , the inverse color transform unit 322 can apply an inverse color transform to the color value. The inverse color transform can be the inverse of the color transform applied by the color transform unit 204 of the encoder 200. For example, the color transform unit 204 can transform the color information from the RGB color space to the YCbCr color space. Accordingly, the inverse color transform unit 322 can transform the color information from the YCbCr color space to the RGB color space.

[0072] Example Figure 2 and Figure 3 Various units are provided to assist in understanding the operations performed by the encoder 200 and the decoder 300. The units may be implemented as fixed-function circuits, programmable circuits, or a combination thereof. Fixed-function circuits refer to circuits that provide specific functionality and are pre-set for the operations that can be performed. Programmable circuits refer to circuits that can be programmed to perform various tasks and provide flexible functionality in the operations that can be performed. For example, a programmable circuit may execute software or firmware that causes the programmable circuit to operate in a manner defined by the instructions of the software or firmware. Fixed-function circuits may execute software instructions (e.g., to receive parameters or output parameters), but the types of operations performed by fixed-function circuits are generally immutable. In some examples, one or more of the units may be different circuit blocks (fixed-function or programmable), and in some examples, one or more of the units may be integrated circuits.

[0073] Figure 4is a conceptual diagram illustrating an example system model for performing the techniques of the present disclosure. The present disclosure describes a lossy geometry compression scheme that can be Figure 4 The system 400 is performed to predict the potential representation of the current point cloud data frame using the previous frame (previously decoded frame) by adopting a prediction network. The system 400 performs P-frame inter-point cloud coding, where the current frame is encoded with the help of the previously decoded frame. The system 400 can be implemented using a sparse convolutional neural network (CNN) with a sparse tensor. The system 400 can use convolution on the target coordinates to map the potential representation of the previous frame to the downsampled coordinates of the current frame, thereby predicting the feature embedding of the current frame. The system 400 can send the residuals of the predicted features and the actual features by compressing the predicted features and the actual features using the learned probability factorization entropy model. Compared with G-PCC and V-PCC, the system according to the present disclosure Figure 4 The implemented techniques demonstrate better compression performance on dense point clouds with efficient encoding / decoding runtime, as discussed in more detail below.

[0074] Conventional deep learning based PCC solutions include intra prediction techniques for static point clouds and do not exploit inter prediction coding gain by using previously decoded frames to predict the current frame. The disclosed technology includes a deep learning based framework for inter coding of point cloud geometry (similar to P frame coding in video compression). The disclosed technology includes a predictor module that learns the feature embedding of the current PC frame from the previous PC frame. The network can use hierarchical multi-scale feature extraction and can employ convolution on the target coordinates to map the latent features of the previous frame to the downsampled coordinates of the current frame to learn the feature embedding of the current frame.

[0075] Point Cloud Geometry Compression (PCGC) is an intra-frame point cloud compression scheme suitable for static point clouds. The inter-frame prediction scheme of the present disclosure uses an encoder and decoder network and a prediction network to predict the feature embedding of the current PC frame from the previous PC frame. According to these techniques, the system 400 can calculate the residual (that is, a series of differences) between the predicted features and the ground truth features, and then the residual features can be sent along with the three-fold downsampled coordinates. The three-fold downsampled coordinates can be losslessly encoded by the octree encoder 406 using the G-PCC TMC13 octree codec, for example, as described above with respect to Figure 2 As discussed. The residual features can be encoded in a lossy manner using a factorized entropy model to predict the probability distribution for arithmetic coding. It should be noted that in the system 400, the encoder and prediction network are present at both the transmitter 402 and the receiver 404. The network is trained using joint reconstruction and bit rate loss to optimize rate distortion.

[0076] The octree encoder 406 may generally correspond to Figure 2 The various parts of the point cloud encoder 200, and the octree decoder 408 can generally correspond to Figure 3 The transmitter 402 may generally correspond to Figure 1 The source device 102, and the receiver 404 may generally correspond to Figure 1 Destination device 116.

[0077] exist Figure 4 In the example of FIG. 4 , system 400 includes a transmitter 402 and a receiver 404. Transmitter 402 uses a previously decoded frame For the current frame p 2 The predictor network processes frames from previously decoded The multi-scale features and 2 The three-fold downsampling coordinates To get feature embedding The transmitter 402 uses an octree encoder to send the three downsampled coordinates to the receiver 404 in a lossless manner The transmitter 402 may subtract the predicted downsampled features and the original downsampled features To calculate the residual features Then, the transmitter 402 can use the learned entropy model to send the residual features in a lossy manner

[0078] This disclosure uses the terms "reduction" and "downsampling" interchangeably. The entire framework can be viewed as a transmission system, a compression pipeline, or a deep learning model. In terms of the transmission system, the elements before the bitstream generation can be referred to as forming part of the "transmitter", while the elements after the bitstream can be referred to as forming part of the "receiver". Figure 4 The system 400 may include an encoder and predictor module at a transmitter 402, and an encoder, predictor, and decoder module at a receiver 404. In terms of a compression pipeline, the system 400 may be viewed as including both an encoding portion and a decoding portion. Figure 4 In

[15] , a predictor module is employed in both the encoding of a frame as well as the decoding of a frame. In terms of a deep learning model, the encoder can be viewed as a multi-scale feature extractor, while the decoder can be viewed as a progressive upscaling network with hierarchical reconstruction of the point cloud.

[0079] Figure 5 is a conceptual diagram illustrating an example encoder network 502 and a decoder network 504. The encoder network 502 may correspond to all or part of the point cloud encoder 200. The encoder network 502 may obtain PC tensors of four different scales, thereby capturing multi-scale features at different levels of detail:

[0080] P 0ds ,P 1ds ,P 2ds ,P 3ds =E(P)

[0081] Where P ids Denotes a sparse tensor P that has been downsampled i times. The decoder network 504 may correspond to all or part of the point cloud decoder 300. In this example, the decoder network 504 receives a three-times downsampled PC tensor and hierarchically upsamples the downsampled PC tensor to reconstruct the original PC tensor, using a different reconstruction loss at each scale. The decoder network 504 may use a transposed convolution to upsample the PC tensor.

[0082] That is, the encoder network 502 can create sparse features from the original point cloud sparse tensor P at four different scales: P 0ds , P 1ds , P 2ds and P 3ds The decoder network 504 may receive the three-times downsampled sparse tensor and layer-wise reconstruct the original point cloud using progressive rescaling. The decoder network 504 may upsample the sparse tensor and then perform a pruning layer to prune away spurious voxels.

[0083] so, Figure 5 An example of an encoder network 502 comprising a series of unit sets, each unit set comprising a reduction unit and an encoding unit, is depicted. The number of sets in the series may be equal to a value encoded in the bitstream indicating the number of times the geometry data is reduced. Likewise, Figure 5 An example of a decoder network 504 comprising a series of unit sets, each unit set comprising a decoding unit and an upscaling unit is depicted. The number of sets in the series may be equal to a value decoded from the bitstream indicating the number of times the geometry data is to be upscaled.

[0084] Figure 6 600 is a block diagram illustrating an example pruning layer 600 of a decoder according to the techniques of this disclosure. The decoder can use the pruning layer 600 to reconstruct the geometry of the PC. The pruning layer 600 can prune false voxels and use binary classification after each magnification to extract real occupied voxels.

[0085] exist Figure 6 In the example, the input sparse tensor P a Coordinate C with shape 139,244×3 a and its corresponding features of shape 139,244×64. a Through the convolution of channel size 1 (CONV 1x3602), to obtain the feature F with shape 139,244×1b The sparse tensor P b According to F b , use binary classification to select top k features (in this example, k = 52,612) and their corresponding coordinates. Then, the pruning unit 604 calculates the number of features according to P a Prune the spurious coordinates and their corresponding features to obtain P c During training, P b A binary voxel classification loss is applied to obtain a proper point cloud reconstruction.

[0086] Figure 7 is a conceptual diagram illustrating an example prediction network 700. The prediction network 700 can be configured to predict a potential representation of a current frame based on a previously reconstructed frame. The prediction network 700 can be included in an encoder such as the point cloud encoder 200, the transmitter 402, or the encoder network 502, or in a decoder such as the point cloud decoder 300, the receiver 404, or the decoder network 504. The prediction network 700 reduces the input three times while concatenating the reduced input with the corresponding scaled features. Finally, the prediction network 700 applies convolution to the target coordinates to obtain features of the current frame.

[0087] exist Figure 7 In the example of , the prediction network 700 receives four multi-scale features from the previous frame and three downsampled coordinates of the current frame To get the feature embedding of the current frame

[0088] Figure 8 802 (specifically, the shaded cells of the bottom layer), and the corresponding output coordinates are shown in Figure 8 The shaded cells in the top layer 800 are shown.

[0089] The convolution on the target coordinate helps to transform the latent features from P 1 The downsampled coordinates (i.e., ) is converted to P 2 The downsampled coordinates (i.e., ). Convolution at target coordinates can be viewed as a convolution with arbitrary input and output coordinates, where features from the input coordinates are convolved with the kernel and the output remains only at the output coordinates.

[0090] The technology of the present disclosure was implemented and tested. During implementation and testing, in order to reduce the computational complexity during training, the point cloud (PC) frame was divided into smaller blocks by applying the same kd-tree partitioning to two consecutive frames. During inference time, the entire point cloud was used. A 10-bit dataset representation was used for training and testing.

[0091] The disclosed techniques are compared with PCGC, G-PCC (octree and triangle soup) techniques, and deep learning intra techniques of the V-PCC approach (inter- and intra-coding). The reference implementation TMC13-v14 of G-PCC is used, while for V-PCC, the implementation TMC2-v17 using the HEVC video codec is used. The point cloud is encoded using the V-PCC inter-frame low-latency setting, which involves P-frame encoding for fair comparison with the proposed P-frame encoding scheme. The following table summarizes the test methods:

[0092]

[0093]

[0094]

[0095] The following table summarizes the Bjontegaard Delta (BD) rate gain of the disclosed technique compared to other conventional techniques using D1 distortion measurement for various standardized test sequences:

[0096]

[0097] Fig. 9 and Fig.10 is a graph showing rate-distortion curves comparing the technology of the present disclosure with other conventional technologies.

[0098] In the above test, Intel Core i9-11900F CPU and Nvidia GeForce GTX3090 GPU were used. G-PCC runtime is calculated for the highest bit rate on CPU. Both PCGC and the proposed method ("the disclosed method") utilize GPU. Due to the diversity of platforms (e.g., CPU vs. GPU, Python vs. C / C++ and other such specific implementation differences), runtime comparison serves as an intuitive reference to represent general computational complexity.

[0099] The following table depicts the encoding and decoding runtimes of various techniques compared to the techniques of the present disclosure (“the present disclosure”):

[0100] G-PCC(O) G-PCC(T) PCGC The Disclosure Code(s) 1.13 6.15 0.258 0.364 Decode(s) 0.44 5.01 0.537 0.714

[0101]

[0102] The present disclosure describes a deep learning-based inter-frame compression scheme for dynamic point clouds, which uses previously decoded previous frames to encode the current frame. The technology of the present disclosure adopts an encoder to obtain multi-scale features, and a decoder to hierarchically reconstruct the point cloud through progressive scaling. The prediction network module can predict the potential representation of the current frame by mapping the potential features of the previous frame to the downsampled coordinates of the current frame using convolution on the target coordinates. The residual of the predicted features and the actual features is encoded and sent. Experimental results show that compared with G-PCC (octree), the BD rate gain exceeds 91%; compared with G-PCC (triangle soup), the BD rate gain exceeds 84%; compared with the intra-frame network PCGC, the BD rate gain exceeds 34%; compared with the V-PCC intra-frame coding mode, the BD rate improvement exceeds 62%; compared with the V-PCC inter-frame low-delay coding mode, the BD rate improvement exceeds 52%.

[0103] The inter-frame point cloud compression scheme discussed above is an example of P-frame point cloud coding, where the current frame is encoded using the previous frame. Similar techniques can be implemented for intra-frame point cloud coding based on I frames and inter-frame point cloud coding based on B frames.

[0104] Fig.11 1 is a conceptual diagram illustrating an example system 1100 for intra-frame point cloud coding based on intra (I) frames. In intra-frame coding, only the current frame is used to encode the current frame. The system 1100 includes an encoder portion 1102 and a decoder portion 1104.

[0105] Fig.12 is a conceptual diagram illustrating an example system 1200 for bidirectional intra-prediction (B) frame encoding. System 1200 includes an encoder portion 1202 and a decoder portion 1204. In B frame encoding, both the previous frame and the next frame (in display order) may be used to encode the current frame.

[0106] The above I-frame, P-frame, and B-frame methods use fixed reduction parameters in the encoder to obtain a three-fold reduced representation of the point cloud with geometric structures and corresponding coordinates. The geometric structure is reduced by a factor of 2 each time, resulting in a total of 8 times reduction in the geometric structure. The three-fold (8 times) reduced coordinates can be sent in a lossless manner, while the corresponding features can be sent in a lossy manner. In this way, the technology of the present disclosure can be based on the three-fold downsampled coordinates. To produce a constant bit rate cost. The three downsampling features Different bit rates can be used for encoding in order to obtain different levels of distortion in the reconstructed point cloud.

[0107] The techniques of the present disclosure can provide a flexible configuration in the system model, where there can be different architectures that produce different reduced representations of the point cloud, rather than having a fixed cubic reduced representation of the point cloud. In this way, the reduction level required to achieve the best performance can be selected. Thus, these techniques can allow the reduction of coordinates ( where i is the number of times the coordinates are reduced) and the corresponding features Therefore, these techniques can allow the system to have greater flexibility and can produce better compression performance.

[0108] Reference again Figure 4 , the system 400 can use the level of reduced representation to indicate the operating parameters of the model, for example, the level at which coordinates are sent in a lossless manner. For example, level x can indicate that a reduced representation of the point cloud is obtained after x reduction steps. In another example, level x can indicate that a reduced representation of the point cloud is obtained using a factor f(x) (i.e., a function of x (e.g., f(x)=2 x )) to obtain the reduced representation. Vice versa, at the receiver, the level x indicates the number of upscaling steps of the model.

[0109] The transmitter 402 may signal a syntax element in the bitstream to indicate the level of the reduced representation. For example, the transmitter 402 may signal a syntax element ds_level_minus1, where ds_level_minus1+1 specifies the level of the reduced representation for decoding. The decoder of the receiver 404 may decode the syntax element to determine the number of upscaling steps used to decode the bitstream.

[0110] In an alternative, the sender 402 may signal syntax elements associated with one or more frames. The receiver 404 may use the signaled values ​​to derive the level of the reduced representation of the one or more frames.

[0111] System 400 may be configured to send three times reduced coordinates This incurs a constant bitrate cost (e.g., 0.023bpp on some point clouds). Having a constant bitrate on the coordinates provides limited flexibility, as the quality of the reconstructed point cloud can only be changed by changing the characteristic bitrate. At lower bitrates, the bitrate is lower bounded by the geometry bits, while at higher bitrates, the peak signal-to-noise ratio (PSNR) is also upper bounded by the geometry bits. Examples of bottlenecks caused by constant coordinate bitrate are discussed below.

[0112] Fig.1313 is a graph showing the rate-distortion curve of the disclosed technique. It can be seen that the lowest bit rate point for P-frames is {bpp: 0.0278, PSNR: 64.04}, while the lowest bit rate point for I-frames is {bpp: 0.0231, PSNR: 61.60}. Fig.13 It shows that at the lowest bit rate, most of the bit rate is consumption, the coordinate is about 0.023bpp, and the characteristic The bit rate is close to zero. In this scenario, reducing the point cloud representation to more than a cubic reduction will reduce the coordinate bit rate, which will allow some information to be transmitted through the features. This will provide more flexibility and better reconstruction quality.

[0113] Similarly, for the highest bitrate and PSNR, the P-frame point is {bpp: 0.1818, PSNR: 73.54}, and the I-frame point is {bpp: 0.3837, PSNR: 73.83}. At these points, the coordinates The constant bitrate consumed is about 0.023bpp, which is negligible when compared to high total bitrates like 0.3837bpp or 0.1818bpp. In this scenario, the features are consuming most of the bitrate. Point cloud representations with less than three reduction versions can increase the coordinates bitrate, which can allow reducing the feature bitrate, resulting in better quality of the reconstructed point cloud.

[0114] Employ multiple system models that can generate different reduced / downsampled representations of the point cloud. This will provide more flexibility in the rate-distortion curve to generate better quality point clouds with fewer bits. Figure 4 The transmitter 402 transmits additional bits to the receiver 404 to indicate which encoder is used so that the corresponding decoder can be used at the receiver 404. Similarly, for inter-frame decoding, a dedicated predictor module can be used for each level of the reduced representation.

[0115] The techniques disclosed herein can be used for a variety of use cases, such as: 1) in every type of compression technique (I-frames, P-frames, B-frames), even when motion vector and optical flow techniques are employed; 2) point cloud attribute compression (not just geometry compression); and 3) any other kind of point cloud compression or communication task that involves a reduced representation of a point cloud and then rescaling the reduced point cloud back to a reconstructed point cloud.

[0116] The number of times the geometry coordinates are reduced can be signaled and decoded by the transmitter 402 in the bitstream to the receiver 404. In this way, the receiver 404 can select the corresponding architecture and network to reconstruct the point cloud and upsample the point cloud. The parameters can be signaled in one or more parameter sets such as a sequence parameter set (SPS), a geometry parameter set (GPS), or a header such as a slice header, a frame header, etc.

[0117] Fig.14 is a flowchart illustrating an example method for encoding point cloud data according to the techniques of the present disclosure. Fig.14 However, it should be understood that Fig.14 The method may also be performed by other devices, such as Figure 4 The transmitter 402, Figure 5 The encoder network 502, Fig.11 The encoder portion 1102 or Fig.12 Encoder portion 1202.

[0118] Initially, the point cloud encoder 200 determines a number of times to reduce the point cloud geometry of the point cloud (1400). For example, the point cloud encoder 200 may retrieve configuration information indicating a number of times to reduce the point cloud geometry, for example, along an X-axis, a Y-axis, and a Z-axis. In some examples, the point cloud encoder 200 may be configured to reduce each of the X-axis, the Y-axis, and the Z-axis by a factor of 2, for a total reduction by a factor of 8. The reduction and encoding unit 202 may reduce the point cloud geometry (e.g., a hierarchical octree) the determined number of times (1402), for example, 8 times. As discussed above, the reduction and encoding unit 202 may be configured to perform such reduction using an AI / ML model, such as with respect to Figure 4 , Figure 5 , Figure 6 , Figure 7 , Fig.11 and Fig.12 In some examples, the point cloud encoder 200 may encode a value indicating the number of times the point cloud geometry is reduced (eg, the value of the syntax element ds_level_minus1), where ds_levelminus1+1 may indicate the number of times the point cloud geometry is reduced.

[0119] The point cloud encoder 200 may also encode the reduced point cloud geometry (1404). For example, the reduction and encoding unit 202 may use an AI / ML model for reducing and encoding point cloud geometry data to encode the corresponding intermediate set of reduced point cloud geometry. Specifically, in some examples, as explained above, the reduction and encoding unit 202 may include a series of unit sets, each of which includes a reduction unit (e.g., a convolutional neural network (CNN)) and an encoding unit set. The number of sets in the series may correspond to the number of times the point cloud geometry is reduced. The point cloud encoder 200 may also encode features of the reduced point cloud geometry (1406). As discussed above, features may include attribute data such as color, reflectivity, brightness, surface normals, etc. The point cloud encoder 200 may then output the encoded point cloud geometry and features (1408), for example as a bitstream.

[0120] so, Fig.14 The method represents an example of a method for encoding point cloud geometry data, the method comprising: determining a number of times to reduce a representation of a point cloud geometry; reducing the representation of the point cloud geometry by the number of times to form a reduced representation of the point cloud geometry; encoding the reduced representation of the point cloud geometry; and outputting the encoded reduced representation of the point cloud geometry.

[0121] Fig.15 is a flowchart illustrating an example method for decoding point cloud data according to the techniques of the present disclosure. Fig.15 However, it should be understood that Fig.15 The method may also be performed by other devices, such as Figure 4 Receiver 404, Figure 5 The decoder network 504, Fig.11 The decoder portion 1104 or Fig.12 Decoder portion 1204.

[0122] Initially, the point cloud decoder 300 may determine a number of times the point cloud geometry is to be reduced (1500). The number of times the point cloud geometry is to be reduced may correspond to the number of times the point cloud geometry is to be enlarged. Thus, additionally or alternatively, the point cloud decoder 300 may be configured to determine a number of times the point cloud geometry is to be enlarged. For example, the point cloud decoder 300 may decode a value of a syntax element such as ds_level_minus1, where ds_levelminus1+1 may indicate a number of times the point cloud geometry is to be reduced and / or a number of times the point cloud geometry is to be enlarged.

[0123] Then, the decoding and amplification unit 302 of the point cloud decoder 300 can decode the reduced point cloud geometry (1502), and amplify the reduced point cloud geometry (1504). For example, the decoding and amplification unit 302 may include a series of unit sets, each unit set including a decoding unit and an amplification unit. The number of sets in the series may be equal to the value of the syntax element indicating the number of times the point cloud geometry is reduced / to be amplified. The point cloud decoder 300 may also decode the features of the reduced point cloud geometry (1506). The point cloud geometry may be decoded and amplified using an AI / ML model for decoding and amplifying point cloud data. In some examples, the point cloud decoder 300 may amplify the reduced point cloud geometry and then use the amplified point cloud geometry when decoding the features. These features may correspond to, for example, color, reflectivity, brightness, surface normals, etc. Finally, the point cloud decoder 300 may reproduce the point cloud from the amplified point cloud geometry and features (1508), for example, applying the features to the points of the amplified point cloud geometry.

[0124] so, Fig.15 The method represents an example of a method for decoding point cloud data, the method comprising: determining a number of times to magnify a reduced encoded representation of a point cloud geometry; decoding the reduced encoded representation of the point cloud geometry; magnifying the representation of the point cloud geometry the number of times to form an enlarged representation of the point cloud geometry; and reproducing a point cloud using the enlarged representation of the point cloud geometry.

[0125] Various examples of the techniques of this disclosure are summarized in the following clauses:

[0126] Item 1: A method for encoding point cloud data, the method comprising: determining a number of times to reduce a representation of a point cloud geometry; reducing the representation of the point cloud geometry by the number of times to form a reduced representation of the point cloud geometry; encoding the reduced representation of the point cloud geometry; encoding features corresponding to the reduced representation of the point cloud geometry; and outputting the encoded reduced representation of the point cloud geometry and the encoded features.

[0127] Clause 2: The method of clause 1, wherein determining the number of times the representation is reduced comprises determining the number of times the representation is reduced based on at least one of a use case, a bit requirement, or a point cloud type of the point cloud geometry.

[0128] Clause 3: The method of any one of clauses 1 and 2, wherein encoding the reduced representation of the point cloud geometry comprises losslessly encoding the reduced representation of the point cloud geometry.

[0129] Clause 4: The method of any one of clauses 1 to 3, wherein encoding the feature comprises encoding the feature using a lossy encoding.

[0130] Clause 5: A method according to any one of clauses 1 to 4, wherein encoding the reduced representation of the point cloud geometry comprises encoding the reduced representation of the point cloud geometry using one of intra-frame (I-frame) coding, inter-frame (P-frame) coding or bidirectional inter-frame (B-frame) coding.

[0131] Clause 6: A method according to any one of clauses 1 to 5, wherein the reduced representation of the point cloud geometry comprises a reduced representation of the point cloud geometry of a current frame, and wherein encoding the features comprises: extracting multi-scale features from a previously decoded point cloud data frame; forming, by a predictor network, predicted features of the current frame based on the extracted multi-scale features and the reduced representation of the point cloud geometry of the current frame; and decoding residual data representing the difference between the features of the current frame and the predicted features of the current frame.

[0132] Clause 7: A method according to any one of clauses 1 to 6, further comprising: encoding data representing the number of times the representation of the point cloud geometry is reduced.

[0133] Clause 8: The method of clause 1, wherein determining the number of times to reduce the representation comprises determining the number of times to reduce the representation based on at least one of a use case, a bit requirement, or a point cloud type of the point cloud geometry.

[0134] Clause 9: The method of clause 1, wherein encoding the reduced representation of the point cloud geometry comprises losslessly encoding the reduced representation of the point cloud geometry.

[0135] Clause 10: The method of clause 1, wherein encoding the feature comprises encoding the feature using a lossy encoding.

[0136] Clause 11: A method according to clause 1, wherein encoding the reduced representation of the point cloud geometry structure includes: encoding the reduced representation of the point cloud geometry structure using one of intra-frame (I frame) coding, inter-frame (P frame) coding or bidirectional inter-frame (B frame) coding.

[0137] Clause 12: A method according to clause 1, wherein the reduced representation of the point cloud geometry comprises a reduced representation of the point cloud geometry of a current frame, and wherein encoding the features comprises: extracting multi-scale features from a previously decoded point cloud data frame; forming, by a predictor network, predicted features of the current frame based on the extracted multi-scale features and the reduced representation of the point cloud geometry of the current frame; and decoding residual data representing the difference between the features of the current frame and the predicted features of the current frame.

[0138] Clause 13: The method of clause 1, further comprising: encoding data representing the number of times the representation of the point cloud geometry is reduced.

[0139] Item 14: A method for decoding point cloud data, the method comprising: determining a number of times to magnify a reduced encoded representation of a point cloud geometry; decoding the reduced encoded representation of the point cloud geometry; decoding a feature corresponding to the reduced representation of the point cloud geometry; magnifying the representation of the point cloud geometry a number of times to form an enlarged representation of the point cloud geometry; reproducing a point cloud using the enlarged representation of the point cloud geometry and the feature; and outputting the reproduced point cloud.

[0140] Clause 15: The method of clause 14, wherein determining the number of times to reduce the representation comprises determining the number of times to reduce the representation based on at least one of a use case, a bit requirement, or a point cloud type of the point cloud geometry.

[0141] Clause 16: The method of any of clauses 14 and 15, wherein determining the number of times the representation is downscaled comprises decoding data representing the number of times the representation of the point cloud geometry is downscaled.

[0142] Clause 17: A method according to any one of clauses 14 to 16, wherein decoding the reduced representation of the point cloud geometry comprises decoding the reduced representation of the point cloud geometry using one of intra-frame (I frame) decoding, inter-frame (P frame) decoding or bidirectional inter-frame (B frame) decoding.

[0143] Clause 18: A method according to any one of clauses 14 to 17, wherein the reduced representation of the point cloud geometry comprises a reduced representation of the point cloud geometry of a current frame, and wherein decoding the features comprises: extracting multi-scale features from a previously decoded point cloud data frame; forming, by a predictor network, predicted features of the current frame based on the extracted multi-scale features and the reduced representation of the point cloud geometry of the current frame; decoding residual data representing the difference between the original features of the current frame and the predicted features of the current frame; and applying the residual data to the predicted features to reproduce the original features.

[0144] Clause 19: The method of clause 14, wherein determining the number of times to reduce the representation comprises determining the number of times to reduce the representation based on at least one of a use case, a bit requirement, or a point cloud type of the point cloud geometry.

[0145] Clause 20: The method of clause 14, wherein determining the number of times the representation is downscaled comprises decoding data representing the number of times the representation of the point cloud geometry is downscaled.

[0146] Clause 21: A method according to clause 14, wherein decoding the reduced representation of the point cloud geometry structure includes: decoding the reduced representation of the point cloud geometry structure using one of intra-frame (I frame) decoding, inter-frame (P frame) decoding, or bidirectional inter-frame (B frame) decoding.

[0147] Clause 22: A method according to clause 14, wherein the reduced representation of the point cloud geometry comprises a reduced representation of the point cloud geometry of a current frame, and wherein decoding the features comprises: extracting multi-scale features from a previously decoded point cloud data frame; forming, by a predictor network, predicted features of the current frame based on the extracted multi-scale features and the reduced representation of the point cloud geometry of the current frame; decoding residual data representing the difference between the original features of the current frame and the predicted features of the current frame; and applying the residual data to the predicted features to reproduce the original features.

[0148] Clause 23: The method of clause 14, wherein upscaling the representation of the point cloud geometry the number of times further comprises: pruning spurious voxels after each upscaling and using binary classification to extract true occupied voxels.

[0149] Clause 24: The method of clause 14, further comprising encoding the reduced representation of the point cloud geometry and the features prior to decoding the reduced representation of the point cloud geometry and the features.

[0150] Clause 25: A method according to any one of clauses 14 to 18, further comprising: encoding the reduced representation of the point cloud geometry and the features before decoding the reduced representation of the point cloud geometry and the features.

[0151] Clause 26: An apparatus for decoding point cloud data, the apparatus comprising one or more means for performing the method according to any one of clauses 1 to 25.

[0152] Clause 27: The apparatus of clause 26, wherein the one or more components comprise one or more processors implemented in circuitry.

[0153] Clause 28: The apparatus of any of clauses 26 and 27, further comprising a display configured to display the decoded video data.

[0154] Clause 29: The device of any of clauses 26 to 28, wherein the device comprises one or more of a camera, a computer, a mobile device, a broadcast receiver device, or a set-top box.

[0155] Clause 30: The apparatus of clauses 26 to 29, further comprising a memory configured to store the point cloud data.

[0156] Clause 31: A computer readable storage medium having stored thereon instructions which, when executed, cause a processor to perform the method of any one of clauses 1 to 25.

[0157] Item 32: A method for encoding point cloud data, the method comprising: determining a number of times to reduce a representation of a point cloud geometry; reducing the representation of the point cloud geometry by the number of times to form a reduced representation of the point cloud geometry; encoding the reduced representation of the point cloud geometry; and outputting the encoded reduced representation of the point cloud geometry.

[0158] Clause 33: The method of clause 32, wherein determining the number of times the representation is reduced comprises determining the number of times the representation is reduced based on at least one of a use case, a bit requirement, or a point cloud type of the point cloud geometry.

[0159] Clause 34: The method of clause 32, wherein encoding the reduced representation of the point cloud geometry comprises losslessly encoding the reduced representation of the point cloud geometry.

[0160] Clause 35: The method of clause 32, further comprising encoding features corresponding to the reduced representation of the point cloud geometry using a lossy encoding.

[0161] Clause 36: A method according to clause 32, wherein encoding the reduced representation of the point cloud geometry structure includes: encoding the reduced representation of the point cloud geometry structure using one of intra-frame (I frame) coding, inter-frame (P frame) coding or bidirectional inter-frame (B frame) coding.

[0162] Clause 37: A method according to clause 32, wherein the reduced representation of the point cloud geometry comprises a reduced representation of the point cloud geometry of the current frame, and wherein encoding the features comprises: extracting multi-scale features from a previously decoded point cloud data frame; forming, by a predictor network, predicted features of the current frame based on the extracted multi-scale features and the reduced representation of the point cloud geometry of the current frame; and decoding residual data representing the difference between the features of the current frame and the predicted features of the current frame.

[0163] Clause 38: The method of clause 32, further comprising: encoding data representing the number of times the representation of the point cloud geometry is reduced.

[0164] Clause 39: A method according to clause 32, wherein reducing the representation of the point cloud geometry and encoding the reduced representation comprises: executing a neural network trained to reduce the point cloud geometry and encode the point cloud geometry in a number of stages, the number of stages being equal to the number of times the representation of the point cloud geometry is reduced, each stage comprising a reduction step and an encoding portion.

[0165] Clause 40: A method according to clause 32, wherein reducing the representation of the point cloud geometry and encoding the reduced representation comprises: executing a neural network, the neural network being trained to reduce the point cloud geometry in a number of stages and being trained to subsequently encode the reduced representation of the point cloud geometry, the number of stages being equal to the number of times the representation of the point cloud geometry is reduced, each stage comprising a reduction step.

[0166] Item 41: A device for encoding point cloud data, the device comprising: a memory configured to store point cloud data; and one or more processors implemented in a circuit system and configured to: determine a number of times to reduce a representation of a point cloud geometry; reduce the representation of the point cloud geometry the number of times to form a reduced representation of the point cloud geometry; encode the reduced representation of the point cloud geometry; and output the encoded reduced representation of the point cloud geometry.

[0167] Clause 42: A device according to clause 41, wherein in order to determine the number of times the representation is reduced, the one or more processors are configured to: determine the number of times the representation is reduced based on at least one of a use case, a bit requirement, or a point cloud type of the point cloud geometry.

[0168] Clause 43: The apparatus of clause 41, wherein to encode the reduced representation of the point cloud geometry, the one or more processors are configured to: losslessly encode the reduced representation of the point cloud geometry.

[0169] Clause 44: The apparatus of clause 41, wherein the one or more processors are further configured to: encode features corresponding to the reduced representation of the point cloud geometry using lossy encoding.

[0170] Clause 45: A device according to clause 41, wherein in order to encode the reduced representation of the point cloud geometry structure, the one or more processors are configured to: encode the reduced representation of the point cloud geometry structure using one of intra-frame (I frame) coding, inter-frame (P frame) coding or bidirectional inter-frame (B frame) coding.

[0171] Clause 46: A device according to clause 41, wherein the reduced representation of the point cloud geometry includes a reduced representation of the point cloud geometry of the current frame, and wherein in order to encode the features, the one or more processors are configured to: extract multi-scale features from a previously decoded point cloud data frame; execute a predictor network to form predicted features for the current frame based on the extracted multi-scale features and the reduced representation of the point cloud geometry of the current frame; and decode residual data representing the difference between the features of the current frame and the predicted features of the current frame.

[0172] Clause 47: The apparatus of clause 41, wherein the one or more processors are further configured to: encode data representing the number of times the representation of the point cloud geometry is reduced.

[0173] Clause 48: An apparatus according to clause 41, wherein in order to reduce the representation of the point cloud geometry and encode the reduced representation, the one or more processors are configured to: execute a neural network, which is trained to reduce the point cloud geometry and encode the point cloud geometry in a number of stages, the number of stages being equal to the number of times the representation of the point cloud geometry is reduced, each stage comprising a reduction step and an encoding part.

[0174] Clause 49: An apparatus according to clause 41, wherein in order to reduce the representation of the point cloud geometry and encode the reduced representation, the one or more processors are configured to: execute a neural network, which is trained to reduce the point cloud geometry in a number of stages and is trained to subsequently encode the reduced representation of the point cloud geometry, wherein the number of stages is equal to the number of times the representation of the point cloud geometry is reduced, each stage comprising a reduction step.

[0175] Item 50: A device for encoding point cloud data, the device comprising: a component for determining a number of times to reduce a representation of a point cloud geometry; a component for reducing the representation of the point cloud geometry a number of times to form a reduced representation of the point cloud geometry; a component for encoding the reduced representation of the point cloud geometry; and a component for outputting the encoded reduced representation of the point cloud geometry.

[0176] Item 51: A method for decoding point cloud data, the method comprising: determining a number of times to magnify a reduced encoded representation of a point cloud geometry; decoding the reduced encoded representation of the point cloud geometry; magnifying the reduced representation of the point cloud geometry the number of times to form an enlarged representation of the point cloud geometry; and reproducing a point cloud using the enlarged representation of the point cloud geometry.

[0177] Clause 52: The method of clause 51, wherein determining the number of times the reduced representation is upscaled comprises determining the number of times the reduced representation is upscaled based on at least one of a use case, a bit requirement, or a point cloud type of the point cloud geometry.

[0178] Clause 53: The method of clause 51, wherein determining the number of times the reduced representation is upscaled comprises decoding data representing the number of times the representation of the point cloud geometry is downscaled.

[0179] Clause 54: A method according to clause 51, wherein decoding the reduced representation of the point cloud geometry structure includes: decoding the reduced representation of the point cloud geometry structure using one of intra-frame (I frame) decoding, inter-frame (P frame) decoding, or bidirectional inter-frame (B frame) decoding.

[0180] Item 55: A method according to Item 51, wherein the reduced representation of the point cloud geometry includes a reduced representation of the point cloud geometry of the current frame, and the method further includes: decoding features corresponding to the reduced representation of the point cloud geometry, including: extracting multi-scale features from a previously decoded point cloud data frame; forming predicted features of the current frame by a predictor network based on the extracted multi-scale features and the reduced representation of the point cloud geometry of the current frame; decoding residual data representing the difference between the original features of the current frame and the predicted features of the current frame; and applying the residual data to the predicted features to reproduce the original features.

[0181] Clause 56: The method of clause 51, wherein upscaling the representation of the point cloud geometry the number of times further comprises: pruning spurious voxels after each upscaling and using binary classification to extract true occupied voxels.

[0182] Clause 57: A method according to clause 51, wherein decoding the reduced representation of the point cloud geometry and upscaling the representation comprises: executing a neural network trained to decode the point cloud geometry and upscaling the point cloud geometry in a number of stages, the number of stages being equal to the number of times the representation of the point cloud geometry is upscaled, each stage comprising a decoding step and an upscaling portion.

[0183] Clause 58: A method according to clause 51, wherein decoding the reduced representation of the point cloud geometry and upscaling the reduced representation comprises: executing a neural network trained to decode the representation of the point cloud geometry in a number of stages and then upscaling the point cloud geometry, the number of stages being equal to the number of times the representation of the point cloud geometry is upscaled.

[0184] Item 59: A device for decoding point cloud data, the device comprising: a memory configured to store point cloud data; and one or more processors implemented in a circuit system and configured to: determine a number of times to magnify a reduced encoded representation of a point cloud geometry; decode the reduced encoded representation of the point cloud geometry; magnify the representation of the point cloud geometry the number of times to form an enlarged representation of the point cloud geometry; and reproduce a point cloud using the enlarged representation of the point cloud geometry.

[0185] Clause 60: An apparatus according to clause 59, wherein in order to determine the number of times the reduced representation is enlarged, the one or more processors are configured to: determine the number of times the reduced representation is enlarged based on at least one of a use case, a bit requirement, or a point cloud type of the point cloud geometry.

[0186] Clause 61: The apparatus of clause 59, wherein to determine the number of times the reduced representation is upscaled, the one or more processors are configured to: decode data representing the number of times the representation of the point cloud geometry is downscaled.

[0187] Clause 62: A device according to clause 59, wherein in order to decode the reduced representation of the point cloud geometry structure, the one or more processors are configured to: decode the reduced representation of the point cloud geometry structure using one of intra-frame (I frame) decoding, inter-frame (P frame) decoding, or bidirectional inter-frame (B frame) decoding.

[0188] Clause 63: A device according to clause 59, wherein the representation of the point cloud geometry comprises a reduced representation of the point cloud geometry of a current frame, and wherein in order to decode the reduced representation of the point cloud geometry, the one or more processors are configured to: extract multi-scale features from a previously decoded point cloud data frame; execute a predictor network to form predicted features of the current frame based on the extracted multi-scale features and the reduced representation of the point cloud geometry of the current frame; decode residual data representing the difference between the original features of the current frame and the predicted features of the current frame; and apply the residual data to the predicted features to reproduce the original features.

[0189] Clause 64: The apparatus of clause 59, wherein to upscale the representation of the point cloud geometry the number of times, the one or more processors are configured to: prune spurious voxels after each upscale and use binary classification to extract truly occupied voxels.

[0190] Clause 65: A device according to clause 59, wherein in order to decode the reduced representation of the point cloud geometry and upscale the representation, the one or more processors are configured to: execute a neural network, which is trained to decode the point cloud geometry and upscale the point cloud geometry in a number of stages, the number of stages being equal to the number of times the representation of the point cloud geometry is upscaled, each stage comprising a decoding step and an upscaling portion.

[0191] Clause 66: A device according to clause 59, wherein in order to decode the reduced representation of the point cloud geometry and upscale the reduced representation, the one or more processors are configured to: execute a neural network, which is trained to decode the representation of the point cloud geometry in a number of stages and then upscale the point cloud geometry, wherein the number of stages is equal to the number of times the representation of the point cloud geometry is upscaled.

[0192] Item 67: A device for decoding point cloud data, the device comprising: a component for determining a number of times to magnify a reduced encoded representation of a point cloud geometry; a component for decoding the reduced encoded representation of the point cloud geometry; a component for magnifying the reduced representation of the point cloud geometry the number of times to form an enlarged representation of the point cloud geometry; and a component for reproducing a point cloud using the enlarged representation of the point cloud geometry.

[0193] Item 68: A method for encoding point cloud data, the method comprising: determining a number of times to reduce a representation of a point cloud geometry; reducing the representation of the point cloud geometry by the number of times to form a reduced representation of the point cloud geometry; encoding the reduced representation of the point cloud geometry; and outputting the encoded reduced representation of the point cloud geometry.

[0194] Clause 69: The method of clause 68, wherein determining the number of times to reduce the representation comprises determining the number of times to reduce the representation based on at least one of a use case, a bit requirement, or a point cloud type of the point cloud geometry.

[0195] Clause 70: The method of any of clauses 68 and 69, wherein encoding the reduced representation of the point cloud geometry comprises losslessly encoding the reduced representation of the point cloud geometry.

[0196] Clause 71: The method of any one of clauses 68 to 70, further comprising encoding features corresponding to the reduced representation of the point cloud geometry using a lossy encoding.

[0197] Clause 72: A method according to any one of clauses 68 to 71, wherein encoding the reduced representation of the point cloud geometry comprises encoding the reduced representation of the point cloud geometry using one of intra-frame (I-frame) coding, inter-frame (P-frame) coding or bidirectional inter-frame (B-frame) coding.

[0198] Clause 73: A method according to any one of clauses 68 to 72, wherein the reduced representation of the point cloud geometry comprises a reduced representation of the point cloud geometry of a current frame, and wherein encoding the features comprises: extracting multi-scale features from a previously decoded point cloud data frame; forming, by a predictor network, predicted features of the current frame based on the extracted multi-scale features and the reduced representation of the point cloud geometry of the current frame; and decoding residual data representing the difference between the features of the current frame and the predicted features of the current frame.

[0199] Clause 74: The method of any one of clauses 68 to 73, further comprising encoding data representing the number of times the representation of the point cloud geometry is reduced.

[0200] Clause 75: A method according to any one of clauses 68 to 74, wherein reducing the representation of the point cloud geometry and encoding the reduced representation comprises: executing a neural network trained to reduce the point cloud geometry and encode the point cloud geometry in a number of stages, the number of stages being equal to the number of times the representation of the point cloud geometry is reduced, each stage comprising a reduction step and an encoding part.

[0201] Clause 76: A method according to any one of clauses 68 to 74, wherein reducing the representation of the point cloud geometry and encoding the reduced representation comprises: executing a neural network trained to reduce the point cloud geometry in a number of stages and trained to subsequently encode the reduced representation of the point cloud geometry, the number of stages being equal to the number of times the representation of the point cloud geometry is reduced, each stage comprising a reduction step.

[0202] Item 77: A device for encoding point cloud data, the device comprising: a memory configured to store point cloud data; and one or more processors implemented in a circuit system and configured to: determine a number of times to reduce a representation of a point cloud geometry; reduce the representation of the point cloud geometry the number of times to form a reduced representation of the point cloud geometry; encode the reduced representation of the point cloud geometry; and output the encoded reduced representation of the point cloud geometry.

[0203] Clause 78: A device according to clause 77, wherein in order to determine the number of times the representation is reduced, the one or more processors are configured to: determine the number of times the representation is reduced based on at least one of a use case, a bit requirement, or a point cloud type of the point cloud geometry.

[0204] Clause 79: The apparatus of any of clauses 77 and 78, wherein to encode the reduced representation of the point cloud geometry, the one or more processors are configured to: losslessly encode the reduced representation of the point cloud geometry.

[0205] Clause 80: The apparatus of any of clauses 77 to 79, wherein the one or more processors are further configured to: encode features corresponding to the reduced representation of the point cloud geometry using lossy encoding.

[0206] Clause 81: A device according to any one of clauses 77 to 80, wherein in order to encode the reduced representation of the point cloud geometry structure, the one or more processors are configured to: encode the reduced representation of the point cloud geometry structure using one of intra-frame (I frame) coding, inter-frame (P frame) coding or bidirectional inter-frame (B frame) coding.

[0207] Clause 82: An apparatus according to any one of clauses 77 to 81, wherein the reduced representation of the point cloud geometry comprises a reduced representation of the point cloud geometry of a current frame, and wherein, in order to encode the features, the one or more processors are configured to: extract multi-scale features from a previously decoded point cloud data frame; execute a predictor network to form predicted features for the current frame based on the extracted multi-scale features and the reduced representation of the point cloud geometry of the current frame; and decode residual data representing the difference between the features of the current frame and the predicted features of the current frame.

[0208] Clause 83: The apparatus of any of clauses 77 to 82, wherein the one or more processors are further configured to: encode data representing the number of times the representation of the point cloud geometry is reduced.

[0209] Clause 84: An apparatus according to any one of clauses 77 to 83, wherein in order to reduce the representation of the point cloud geometry and encode the reduced representation, the one or more processors are configured to: execute a neural network, which is trained to reduce the point cloud geometry and encode the point cloud geometry in a number of stages, the number of stages being equal to the number of times the representation of the point cloud geometry is reduced, each stage comprising a reduction step and an encoding part.

[0210] Clause 85: An apparatus according to any one of clauses 77 to 83, wherein in order to reduce the representation of the point cloud geometry and encode the reduced representation, the one or more processors are configured to: execute a neural network, which is trained to reduce the point cloud geometry in a number of stages and is trained to subsequently encode the reduced representation of the point cloud geometry, the number of stages being equal to the number of times the representation of the point cloud geometry is reduced, each stage comprising a reduction step.

[0211] Item 86: A device for encoding point cloud data, the device comprising: a component for determining a number of times to reduce a representation of a point cloud geometry; a component for reducing the representation of the point cloud geometry a number of times to form a reduced representation of the point cloud geometry; a component for encoding the reduced representation of the point cloud geometry; and a component for outputting the encoded reduced representation of the point cloud geometry.

[0212] Clause 87: A method for decoding point cloud data, the method comprising: determining a number of times to magnify a reduced encoded representation of a point cloud geometry; decoding the reduced encoded representation of the point cloud geometry; magnifying the reduced representation of the point cloud geometry the number of times to form an enlarged representation of the point cloud geometry; and reproducing a point cloud using the enlarged representation of the point cloud geometry.

[0213] Clause 88: The method of clause 87, wherein determining the number of times the reduced representation is upscaled comprises determining the number of times the reduced representation is upscaled based on at least one of a use case, a bit requirement, or a point cloud type of the point cloud geometry.

[0214] Clause 89: The method of any of clauses 87 and 88, wherein determining the number of times the reduced representation is upscaled comprises decoding data representing the number of times the representation of the point cloud geometry is downscaled.

[0215] Clause 90: A method according to any one of clauses 87 to 89, wherein decoding the reduced representation of the point cloud geometry comprises decoding the reduced representation of the point cloud geometry using one of intra-frame (I frame) decoding, inter-frame (P frame) decoding, or bidirectional inter-frame (B frame) decoding.

[0216] Clause 91: A method according to any one of clauses 87 to 90, wherein the reduced representation of the point cloud geometry comprises a reduced representation of the point cloud geometry of a current frame, the method further comprising: decoding features corresponding to the reduced representation of the point cloud geometry, comprising: extracting multi-scale features from a previously decoded point cloud data frame; forming predicted features of the current frame by a predictor network based on the extracted multi-scale features and the reduced representation of the point cloud geometry of the current frame; decoding residual data representing the difference between the original features of the current frame and the predicted features of the current frame; and applying the residual data to the predicted features to reproduce the original features.

[0217] Clause 92: The method of any of clauses 87 to 91, wherein upscaling the representation of the point cloud geometry the number of times further comprises: pruning spurious voxels after each upscaling and using binary classification to extract truly occupied voxels.

[0218] Clause 93: A method according to any one of clauses 87 to 92, wherein decoding the reduced representation of the point cloud geometry and upscaling the representation comprises: executing a neural network trained to decode the point cloud geometry and upscaling the point cloud geometry in a number of stages, the number of stages being equal to the number of times the representation of the point cloud geometry is upscaled, each stage comprising a decoding step and an upscaling portion.

[0219] Clause 94: A method according to any one of clauses 87 to 92, wherein decoding the reduced representation of the point cloud geometry and upscaling the reduced representation comprises: executing a neural network trained to decode the representation of the point cloud geometry in a number of stages and then upscaling the point cloud geometry, the number of stages being equal to the number of times the representation of the point cloud geometry is upscaled.

[0220] Item 95: A device for decoding point cloud data, the device comprising: a memory configured to store point cloud data; and one or more processors implemented in a circuit system and configured to: determine a number of times to magnify a reduced encoded representation of a point cloud geometry; decode the reduced encoded representation of the point cloud geometry; magnify the representation of the point cloud geometry the number of times to form an enlarged representation of the point cloud geometry; and reproduce a point cloud using the enlarged representation of the point cloud geometry.

[0221] Clause 96: An apparatus according to clause 95, wherein in order to determine the number of times the reduced representation is enlarged, the one or more processors are configured to: determine the number of times the reduced representation is enlarged based on at least one of a use case, a bit requirement, or a point cloud type of the point cloud geometry.

[0222] Clause 97: A device according to any of clauses 95 and 96, wherein in order to determine the number of times the reduced representation is enlarged, the one or more processors are configured to: decode data representing the number of times the representation of the point cloud geometry is reduced.

[0223] Clause 98: A device according to any one of clauses 95 to 97, wherein in order to decode the reduced representation of the point cloud geometry structure, the one or more processors are configured to: decode the reduced representation of the point cloud geometry structure using one of intra-frame (I frame) decoding, inter-frame (P frame) decoding, or bidirectional inter-frame (B frame) decoding.

[0224] Clause 99: An apparatus according to any one of clauses 95 to 98, wherein the representation of the point cloud geometry comprises a reduced representation of the point cloud geometry of a current frame, and wherein in order to decode the reduced representation of the point cloud geometry, the one or more processors are configured to: extract multi-scale features from a previously decoded point cloud data frame; execute a predictor network to form predicted features of the current frame based on the extracted multi-scale features and the reduced representation of the point cloud geometry of the current frame; decode residual data representing the difference between the original features of the current frame and the predicted features of the current frame; and apply the residual data to the predicted features to reproduce the original features.

[0225] Clause 100: The apparatus of any one of clauses 95 to 99, wherein to magnify the representation of the point cloud geometry the number of times, the one or more processors are configured to: prune spurious voxels after each magnification and extract truly occupied voxels using binary classification.

[0226] Clause 101: An apparatus according to any one of clauses 95 to 100, wherein in order to decode the reduced representation of the point cloud geometry and upscale the representation, the one or more processors are configured to: execute a neural network, which is trained to decode the point cloud geometry and upscale the point cloud geometry in a number of stages, the number of stages being equal to the number of times the representation of the point cloud geometry is upscaled, each stage comprising a decoding step and an upscaling part.

[0227] Clause 102: An apparatus according to any one of clauses 95 to 100, wherein in order to decode the reduced representation of the point cloud geometry and upscale the reduced representation, the one or more processors are configured to: execute a neural network that is trained to decode the representation of the point cloud geometry and then upscale the point cloud geometry in a number of stages, wherein the number of stages is equal to the number of times the representation of the point cloud geometry is upscaled.

[0228] Item 103: A device for decoding point cloud data, the device comprising: a component for determining a number of times to magnify a reduced encoded representation of a point cloud geometry; a component for decoding the reduced encoded representation of the point cloud geometry; a component for magnifying the reduced representation of the point cloud geometry the number of times to form an enlarged representation of the point cloud geometry; and a component for reproducing a point cloud using the enlarged representation of the point cloud geometry.

[0229] It should be appreciated that, according to examples, certain actions or events of any of the techniques described herein may be performed in a different order, may be added, combined, or omitted entirely (e.g., not all actions or events described are necessary to implement the technique). In addition, in some examples, actions or events may be performed concurrently (e.g., by multithreading, interrupt handling, or multiple processors) rather than sequentially.

[0230] In one or more examples, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functionality may be stored as one or more instructions or codes on a computer-readable medium or sent via a computer-readable medium and executed by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium (which corresponds to a tangible medium such as a data storage medium) or a communication medium, which includes, for example, any medium that facilitates the transfer of a computer program from one place to another according to a communication protocol. Thus, a computer-readable medium may generally correspond to (1) a non-transient tangible computer-readable storage medium, or (2) a communication medium such as a signal or carrier wave. A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures for implementing the techniques described in the present disclosure. A computer program product may include a computer-readable medium.

[0231] By way of example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory, or any other medium that can be used to store the desired program code in the form of instructions or data structures and can be accessed by a computer. In addition, any connection is appropriately referred to as a computer-readable medium. For example, if the instruction is sent from a website, server or other remote source using a coaxial cable, optical fiber cable, twisted pair, digital subscriber line (DSL) or wireless technology (such as infrared, radio and microwave), the coaxial cable, optical fiber cable, twisted pair, DSL or wireless technology (such as infrared, radio and microwave) is included in the definition of the medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals or other transient media, but are instead directed to non-transient tangible storage media. As used herein, disks and optical disks include compact disks (CDs), laser optical disks, optical optical disks, digital versatile disks (DVDs), floppy disks and blue-ray disks, wherein disks usually reproduce data magnetically, while optical disks reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0232] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuit systems. Therefore, the terms "processor" and "processing circuit system" as used herein may refer to any of the aforementioned structures or any other structure suitable for implementing the techniques described herein. In addition, in some aspects, the functionality described herein may be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into a combined codec. In addition, these techniques may be fully implemented in one or more circuits or logic elements.

[0233] The techniques of the present disclosure may be implemented in a variety of devices or apparatuses, including a wireless handset, an integrated circuit (IC), or a set of ICs (e.g., a chipset). Various components, modules, or units are described in the present disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require implementation by different hardware units. Specifically, as described above, the various units may be combined in a codec hardware unit, or the various units may be provided by a collection of interoperable hardware units (including one or more processors as described above) in combination with appropriate software and / or firmware.

[0234] Various examples have been described. These and other examples are within the scope of the following claims.

Claims

1. A method for encoding point cloud data, the method comprising: determining a number of times to reduce the representation of the point cloud geometry; reducing the representation of the point cloud geometry the number of times to form a reduced representation of the point cloud geometry; encoding the reduced representation of the point cloud geometry; as well as The encoded reduced representation of the point cloud geometry is output.

2. The method of claim 1 , wherein determining the number of times the representation is reduced comprises: The number of times the representation is reduced is determined based on at least one of a use case, a bit requirement, or a point cloud type of the point cloud geometry.

3. The method of claim 1 , wherein encoding the reduced representation of the point cloud geometry comprises: The reduced representation of the point cloud geometry is losslessly encoded.

4. The method according to claim 1, further comprising: Features corresponding to the reduced representation of the point cloud geometry are encoded using a lossy encoding.

5. The method of claim 1 , wherein encoding the reduced representation of the point cloud geometry comprises: The reduced representation of the point cloud geometry is encoded using one of intra-frame (I-frame) coding, inter-frame (P-frame) coding, or bidirectional inter-frame (B-frame) coding.

6. The method of claim 1 , wherein the reduced representation of the point cloud geometry comprises a reduced representation of the point cloud geometry of a current frame, the method further comprising: Encoding features corresponding to the reduced representation of the point cloud geometry comprises: Extracting multi-scale features from previously decoded point cloud data frames; forming, by a predictor network, predicted features for the current frame based on the extracted multi-scale features and the reduced representation of the point cloud geometry of the current frame; and Residual data representing a difference between a feature of the current frame and the predicted feature of the current frame is decoded.

7. The method according to claim 1, further comprising: Data representing the number of times the representation of the point cloud geometry is reduced is encoded.

8. The method of claim 1 , wherein reducing the representation of the point cloud geometry and encoding the reduced representation comprises: A neural network is executed, the neural network being trained to reduce the point cloud geometry and encode the point cloud geometry in a number of stages equal to the number of times the representation of the point cloud geometry is reduced, each stage comprising a reduction step and an encoding portion.

9. The method of claim 1 , wherein reducing the representation of the point cloud geometry and encoding the reduced representation comprises: A neural network is executed, the neural network being trained to reduce the point cloud geometry in a number of stages equal to the number of times the representation of the point cloud geometry is reduced, each stage comprising a reduction step, and being trained to subsequently encode the reduced representation of the point cloud geometry.

10. A device for encoding point cloud data, the device comprising: a memory configured to store point cloud data; and One or more processors implemented in the circuit system and configured to: determining a number of times to reduce the representation of the point cloud geometry; reducing the representation of the point cloud geometry the number of times to form a reduced representation of the point cloud geometry; encoding the reduced representation of the point cloud geometry; as well as The encoded reduced representation of the point cloud geometry is output.

11. The apparatus of claim 10, wherein, to determine the number of times the representation is reduced, the one or more processors are configured to: determine the number of times the representation is reduced based on at least one of a use case, a bit requirement, or a point cloud type of the point cloud geometry. 12 . The apparatus of claim 10 , wherein to encode the reduced representation of the point cloud geometry, the one or more processors are configured to: losslessly encode the reduced representation of the point cloud geometry.

13. The apparatus of claim 10, wherein the one or more processors are further configured to encode features corresponding to the reduced representation of the point cloud geometry using lossy encoding.

14. A device according to claim 10, wherein, in order to encode the reduced representation of the point cloud geometry structure, the one or more processors are configured to: encode the reduced representation of the point cloud geometry structure using one of intra-frame (I frame) coding, inter-frame (P frame) coding, or bidirectional inter-frame (B frame) coding.

15. The apparatus of claim 10, wherein the reduced representation of the point cloud geometry comprises a reduced representation of the point cloud geometry of a current frame, and wherein the one or more processors are further configured to: encode features corresponding to the point cloud geometry, wherein to encode the features, the one or more processors are configured to: Extracting multi-scale features from previously decoded point cloud data frames; executing a predictor network to form predicted features for the current frame based on the extracted multi-scale features and the reduced representation of the point cloud geometry of the current frame; and Residual data representing a difference between a feature of the current frame and the predicted feature of the current frame is decoded.

16. The device of claim 10, wherein the one or more processors are further configured to encode data representing the number of times the representation of the point cloud geometry is reduced.

17. The apparatus of claim 10, wherein to reduce the representation of the point cloud geometry and encode the reduced representation, the one or more processors are configured to: execute a neural network trained to reduce the point cloud geometry and encode the point cloud geometry in a number of stages equal to the number of times the representation of the point cloud geometry is reduced, each stage comprising a reduction step and an encoding portion.

18. The apparatus of claim 10, wherein to reduce the representation of the point cloud geometry and encode the reduced representation, the one or more processors are configured to: execute a neural network trained to reduce the point cloud geometry in a number of stages and trained to subsequently encode the reduced representation of the point cloud geometry, the number of stages being equal to the number of times the representation of the point cloud geometry is reduced, each stage comprising a reduction step.

19. A device for encoding point cloud data, the device comprising: means for determining a number of times to reduce a representation of a point cloud geometry; means for reducing the representation of the point cloud geometry the number of times to form a reduced representation of the point cloud geometry; means for encoding the reduced representation of the point cloud geometry; and Means for outputting the encoded reduced representation of the point cloud geometry.

20. A method for decoding point cloud data, the method comprising: determining a number of times to upscale the reduced encoded representation of the point cloud geometry; decoding the reduced encoded representation of the point cloud geometry; enlarging the reduced representation of the point cloud geometry the number of times to form an enlarged representation of the point cloud geometry; as well as A point cloud is reproduced using the magnified representation of the point cloud geometry.

21. The method of claim 20, wherein determining the number of times to upscale the reduced representation comprises: The number of times the reduced representation is upscaled is determined based on at least one of a use case, a bit requirement, or a point cloud type of the point cloud geometry.

22. The method of claim 20, wherein determining the number of times to upscale the reduced representation comprises: Data representing the number of times the representation of the point cloud geometry is reduced is decoded.

23. The method of claim 20, wherein decoding the reduced representation of the point cloud geometry comprises: The reduced representation of the point cloud geometry is decoded using one of intra-frame (I-frame) decoding, inter-frame (P-frame) decoding, or bidirectional inter-frame (B-frame) decoding.

24. The method of claim 20, wherein the reduced representation of the point cloud geometry comprises a reduced representation of the point cloud geometry of a current frame, the method further comprising: Decoding features corresponding to the reduced representation of the point cloud geometry, comprising: Extracting multi-scale features from a previously decoded point cloud data frame; forming, by a predictor network, predicted features for the current frame based on the extracted multi-scale features and the reduced representation of the point cloud geometry of the current frame; decoding residual data representing a difference between the original features of the current frame and the predicted features of the current frame; and The residual data is applied to the predicted features to reproduce the original features.

25. The method of claim 20, wherein magnifying the representation of the point cloud geometry the number of times further comprises: Spurious voxels were pruned after each upscaling and true occupied voxels were extracted using binary classification.

26. The method of claim 20, wherein decoding the reduced representation of the point cloud geometry and upscaling the representation comprises: A neural network is executed, the neural network being trained to decode the point cloud geometry and upscale the point cloud geometry in a number of stages equal to the number of times the representation of the point cloud geometry is upscaled, each stage comprising a decoding step and an upscaling portion.

27. The method of claim 20, wherein decoding the reduced representation of the point cloud geometry and upscaling the reduced representation comprises: A neural network is executed that is trained to decode the representation of the point cloud geometry in a number of stages equal to the number of times the representation of the point cloud geometry is upscaled and then upscale the point cloud geometry.

28. A device for decoding point cloud data, the device comprising: a memory configured to store point cloud data; and One or more processors implemented in the circuit system and configured to: determining a number of times to upscale the reduced encoded representation of the point cloud geometry; decoding the reduced encoded representation of the point cloud geometry; enlarging the representation of the point cloud geometry the number of times to form an enlarged representation of the point cloud geometry; as well as A point cloud is reproduced using the magnified representation of the point cloud geometry.

29. A device according to claim 28, wherein in order to determine the number of times the reduced representation is enlarged, the one or more processors are configured to: determine the number of times the reduced representation is enlarged based on at least one of a use case, a bit requirement, or a point cloud type of the point cloud geometry.

30. The apparatus of claim 28, wherein to determine the number of times the reduced representation is upscaled, the one or more processors are configured to decode data representing the number of times the representation of the point cloud geometry is downscaled.

31. A device according to claim 28, wherein in order to decode the reduced representation of the point cloud geometry structure, the one or more processors are configured to: decode the reduced representation of the point cloud geometry structure using one of intra-frame (I frame) decoding, inter-frame (P frame) decoding, or bidirectional inter-frame (B frame) decoding.

32. The apparatus of claim 28, wherein the representation of the point cloud geometry comprises a reduced representation of the point cloud geometry of a current frame, and wherein to decode the reduced representation of the point cloud geometry, the one or more processors are configured to: Extracting multi-scale features from a previously decoded point cloud data frame; executing a predictor network to form predicted features for the current frame based on the extracted multi-scale features and the reduced representation of the point cloud geometry of the current frame; decoding residual data representing a difference between the original features of the current frame and the predicted features of the current frame; and The residual data is applied to the predicted features to reproduce the original features.

33. The apparatus of claim 28, wherein to upscale the representation of the point cloud geometry the number of times, the one or more processors are configured to: prune spurious voxels after each upscale and extract true occupied voxels using binary classification.

34. The apparatus of claim 28, wherein to decode the reduced representation of the point cloud geometry and upscale the representation, the one or more processors are configured to: execute a neural network trained to decode the point cloud geometry and upscale the point cloud geometry in a number of stages equal to the number of times the representation of the point cloud geometry is upscaled, each stage comprising a decoding step and an upscaling portion.

35. The apparatus of claim 28, wherein to decode the reduced representation of the point cloud geometry and upscale the reduced representation, the one or more processors are configured to: execute a neural network trained to decode the representation of the point cloud geometry and then upscale the point cloud geometry in a number of stages equal to the number of times the representation of the point cloud geometry is upscaled.

36. A device for decoding point cloud data, the device comprising: means for determining a number of times to upscale the reduced encoded representation of the point cloud geometry; means for decoding the reduced encoded representation of the point cloud geometry; means for enlarging the reduced representation of the point cloud geometry the number of times to form an enlarged representation of the point cloud geometry; and Means for reproducing a point cloud using the enlarged representation of the point cloud geometry.

Images