Intra prediction in point cloud attribute compression

Abstract

To encode and decode a point cloud, the point cloud is represented by an octree of nodes including a list of spatial locations and attribute values, such as color, reflectivity, normal vectors, etc. The invention discloses an early termination decision workflow for disabling transform domain prediction. According to the present principles, when intra prediction needs to be performed is predicted using a distance from a tree root, i.e., a hierarchy of nodes in an octree, as a parameter, and outlier nodes are removed prior to calculating the prediction. In the embodiment, the memory byte consumption is reduced by four times of the point number.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of European application No. 23305017.8, filed on January 6, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present principles generally relate to the field of compression of point clouds including attribute data. This document may also be understood in the context of encoding, formatting, and decoding data representing point clouds, for example, for rendering volumetric content on end-user devices such as mobile devices or head-mounted displays (HMDs). In particular, this document relates to compression of attributes associated with points of a point cloud, such as color, reflectivity, or normal vectors. Background Art

[0004] This section is intended to introduce the reader to various technical aspects that may be related to various aspects of the present principles described and / or claimed below. It is believed that this discussion will help provide the reader with background information to better understand the various aspects of the present principles. Therefore, it should be understood that these statements should be read in this context and not as admissions of prior art.

[0005] Advances in 3D (three-dimensional) capture and rendering technologies are driving new applications and services in autonomous driving, cultural heritage archiving, immersive telepresence, and virtual / augmented reality. Point clouds have become one of the primary 3D scene representations used for these applications. A point cloud frame consists of a collection of 3D points, each represented by its 3D position and potentially multiple attributes such as color, transparency, reflectivity, and normal vector.

[0006] The geometry of a point cloud (i.e., the location of the points in 3D space) can be represented by an octree, which is a tree structure that divides the space into eight voxels at each level of the tree. If a voxel contains at least one point, its corresponding node contains 1, otherwise it contains 0. The number of levels of the tree can be adjusted to manage the resolution at which you want to represent the data. For example, the attribute of a voxel is the average of the attributes of the points contained in that voxel (i.e., a node of the octree). If the value of a node can be predicted from the values of its parent node, uncle node, and / or sibling node in the octree, the amount of data required to encode the attribute is reduced.

[0007] Standardization activities for point cloud compression are being conducted by the ISO / IEC JTC1 / SC29 / WG7 "MPEG 3D Graphics and Haptic Coding" group (GPCC). The Region Adaptive Hierarchical Transform (RAHT) is one of the two transforms used for attribute coding in G-PCC. RAHT is performed on an occupancy tree (octree) representation of the point cloud, starting from the leaf nodes (highest level) of the octree and working backwards until the root node (lowest level) is reached. At each node, the transform is performed in three steps (in each of the x, y, and z directions) to generate low-pass and high-pass coefficients. In G-PCC, upsampled transform domain prediction is introduced to improve RAHT coding efficiency. For each block, the transformed upconverted attribute sum of the current level d, calculated from the decoded attribute sum of the lower level d-1, is used as a prediction for the transformed attribute sum of level d, generating a high-pass residual that can be further quantized and entropy coded. The upconversion process is performed by weighted averaging of neighboring nodes.

[0008] RAHT prediction isn't always helpful. In fact, for example, when a given node has very few neighbors in the previous level, the prediction may be unsatisfactory and simply increase computation time without significant gain. In such cases, early termination of the decision process for transform-domain prediction is advisable. Prediction can also be disabled when the current node has only a single occupied child, as no transform will be performed in this case, leading to early termination of the decision process.

[0009] Figure 1 The current early termination decision process for disabling RAHT transform domain prediction is complex and requires manual tuning. There is no method to simplify this process to achieve lower complexity without affecting coding performance. Summary of the Invention

[0010] The following is a simplified summary of the present principles to provide a basic understanding of certain aspects of the present principles. This summary is not an extensive overview of the present principles. It is not intended to identify the key or critical elements of the present principles. The following summary merely presents certain aspects of the present principles in a simplified form as a prelude to the more detailed description below.

[0011] The present principles relate to a method comprising obtaining a node tree linking nodes representing a point cloud, and predicting an attribute value for a node under consideration in the node tree only if the node under consideration has a level below a first threshold in the node tree, a number of sibling nodes of the node under consideration is greater than zero, and a number of valid neighbor nodes of the node under consideration is less than a second threshold.

[0012] The present principles relate to an apparatus comprising a memory associated with a processor, the processor being configured to implement the above-described method. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The present disclosure will be better understood and other specific features and advantages will emerge from reading the following description with reference to the accompanying drawings:

[0014] Figure 1 illustrates the early termination decision workflow for disabling transform domain prediction;

[0015] Figure 2 illustrates an early termination decision workflow for disabling transform domain prediction according to the present principles;

[0016] Figure 3 An exemplary architecture of a device that can be configured to implement the method for encoding or decoding a point cloud according to the present principles is shown;

[0017] Figure 4 An example of an embodiment of a stream syntax when transmitting data over a packet-based transport protocol is shown. DETAILED DESCRIPTION

[0018] The present principles will be described more fully below with reference to the accompanying drawings, in which examples of the present principles are shown. However, the present principles can be embodied in many alternative forms and should not be construed as limited to the examples set forth herein. Therefore, while the present principles are susceptible to various modifications and alternative forms, specific examples thereof are shown by way of example in the accompanying drawings and will be described in detail herein. However, it should be understood that it is not intended to limit the present principles to the particular forms disclosed, but rather, the present disclosure is intended to cover all modifications, equivalents, and alternatives within the spirit and scope of the present principles as defined by the claims.

[0019] The terms used herein are for the purpose of describing specific examples only and are not intended to limit the present principles. As used herein, the singular forms "a", "an" and "the" are intended to also include the plural forms, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises", "comprising", "includes" and / or "including" when used in this specification indicate the presence of the features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof. In addition, when an element is referred to as "responsive to" or "connected to" another element, it can be directly responsive to or connected to the other element, or there can be intervening elements. In contrast, when an element is referred to as "directly responsive to" or "directly connected to" another element, there are no intervening elements. As used herein, the term "and / or" includes any and all combinations of one or more of the relevant listed items and can be abbreviated as " / ".

[0020] It should be understood that although the terms first, second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element without departing from the teachings of the present principles.

[0021] Although some illustrations include arrows on communication paths to show a primary direction of communication, it is understood that communication can occur in the opposite direction of the depicted arrows.

[0022] Some examples are described with reference to block diagrams and operational flow charts, where each block represents a circuit element, module, or code portion that includes one or more executable instructions for implementing the specified logical function. It should also be noted that in other implementations, the functions indicated in the blocks may occur in a different order than indicated. For example, two blocks shown in succession may actually be executed substantially simultaneously, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved.

[0023] Reference herein to "according to an example" or "in an example" means that a particular feature, structure, or characteristic described in connection with the example may be included in at least one implementation of the present principles. Multiple appearances of the phrase "according to an example" or "in an example" in the specification are not necessarily all referring to the same example, nor are separate or alternative examples necessarily mutually exclusive of other examples.

[0024] Reference numerals appearing in the claims are for illustration only and shall have no limiting effect on the scope of the claims.The examples and variations may be used in any combination or subcombination even if not explicitly described.

[0025] Figure 1The diagram illustrates an early termination decision workflow 10 for disabling transform domain prediction. A node is said to be valid when it contains at least one point. This information is provided by the occupancy octree. In step 11, the number of valid sibling nodes is calculated. If the node under consideration is the only child of its parent node, then step 12 is performed and prediction is disabled in step 13. In step 12, the number of parent nodes is recorded in a memory, where the node under consideration is associated with this number. In fact, in this case, the node inherits the properties of its parent node without any modification. If the node under consideration has sibling nodes, then step 14 is performed to calculate the number of valid "grandfather nodes" (i.e., valid parent nodes of the parent node of the node under consideration, which are used to predict the attribute value of the parent node). According to existing methods, when this number is less than a first threshold TH1 (e.g., TH1=2 or 3 or 9), prediction is disabled in step 13. Otherwise, in steps 15 and 16, the number of valid neighbors of the node under consideration is calculated. A neighbor is a parent node or a valid uncle node (i.e., a valid sibling node of the parent node of the node under consideration). When the number of valid neighbors is less than a second threshold TH2 (e.g. TH2=4 or 6 or 9), prediction is disabled in step 13. Otherwise, prediction is performed in step 17. In both cases, as in step 12, the parent node number is recorded in a memory, with the node under consideration being associated with this number.

[0026] Figure 1 The early termination decision flow shown relies on the number of grandparent nodes to decide on early termination (NumValidGP ≥ Th1). Since a single child node (i.e., a node with no sibling nodes, i.e., a node whose parent has only one child) can still be used for prediction in the next layer, it is necessary to calculate the number of grandparent nodes that may be used in the next layer. However, this approach may still enable prediction even when the number of grandparent nodes indicates that prediction is useless.

[0027] Figure 2 The early termination decision workflow 20 for disabling transform domain prediction according to the present principles is illustrated. In this embodiment, the early termination workflow for disabling transform domain prediction is decoupled from the number of grandparent nodes. The distance from the root of the tree (i.e. the level of the node) is used as a parameter to predict when intra prediction needs to be performed, and outlier nodes are removed before calculating the prediction. According to the present method, in step 21, the level of the considered node is considered. If this level is below a threshold TH1, prediction is disabled in step 22. In fact, if the level is low, its resolution is also low (voxels are larger and contain many points), so predicting the attribute values of its child nodes does not bring accurate new information. The present principles eliminate Figure 1The dependence of the disabled transform domain prediction on the number of grandparent nodes is shown. In fact, the number of (neighboring) grandparent nodes increases as the node under consideration is far away from the root of the tree. According to the present principles, it is useful to use the distance from the root of the tree instead of the number of (neighboring) grandparent nodes to decide when to predict the attributes of a node. This approach allows to reduce memory usage since there is no need to cache the number of grandparent nodes. The dynamic memory byte consumption is reduced by a factor of one. For example, if integers are represented with four bytes, the memory reduction factor is eight (one integer for the parent node and another integer for the grandparent node).

[0028] If the level of the node under consideration is higher than the threshold TH1, step 23 is executed. Step 23 is equivalent to Figure 1 Step 11. If the number of child nodes of the considered node is equal to one, no prediction is required and step 22 is executed, i.e. the attribute prediction process is discarded. Otherwise, step 24, which is equivalent to step 15, is executed to find the neighbors of the considered node. Then, in step 25, the outlier neighbors are removed. In fact, if the attributes of some neighbor nodes (color, brightness, reflectivity, normal vector, etc.) are higher or lower than the limits determined by the attribute values of the parent node of the considered node, these neighbor nodes can be ignored during the prediction. For example, if the attribute value of a neighbor node is higher than 250% (or 200% or 300%) or lower than 20% (or 15% or 35%) of the attribute value of the parent node of the considered node, the neighbor node is regarded as an outlier node and is removed from the neighbor node list. In the equivalent Figure 1 In step 26 of step 16, the number of valid neighbor nodes is calculated, and if this number is greater than a given threshold TH2, the prediction is discarded in step 22. Otherwise, in step 27, intra prediction of the considered node attributes is performed.

[0029] This principle applies to both the decoding and encoding processes. In practice, the method used to retain or discard the attribute value prediction for a node under consideration has a direct impact on the encoding or prediction residual value of the attribute value. The early termination decision workflow for disabling transform-domain prediction according to this principle results in different encoded values.

[0030] Figure 3 An exemplary architecture of a device 30 is shown, which may be configured to implement a method for encoding or decoding a point cloud according to the present principles. The encoder and / or decoder may be implemented in conjunction with Figure 2 Alternatively, each circuit of the encoder and / or decoder may be based on Figure 3 The device 30 is linked together by its bus 31 and / or I / O interface 36. The device 30 includes the following elements linked together by the data and address bus 31:

[0031] a microprocessor 32 (or CPU), such as a DSP (or digital signal processor);

[0032] ROM (or read-only memory) 33;

[0033] RAM (or random access memory) 34;

[0034] Storage interface 35;

[0035] I / O interface 36 for receiving data to be transmitted from an application program; and

[0036] A power source, such as a battery.

[0037] According to one example, the power supply is external to the device. In each memory mentioned, the term "register" as used in this specification can correspond to a small area (a few bits) or a very large area (e.g., an entire program or a large amount of received or decoded data). ROM 33 includes at least one program and parameters. ROM 33 can store algorithms and instructions for executing the technology according to the present principles. When turned on, CPU 32 uploads the program to RAM and executes the corresponding instructions.

[0038] The RAM 34 contains in its registers the program executed and uploaded by the CPU 32 after the device 30 is turned on, input data in registers, intermediate data in registers at different stages of the method and other variables in registers used to execute the method.

[0039] The embodiments described herein can be implemented, for example, in a method or process, an apparatus, a computer program product, a data stream, or a signal. Even if discussed only in the context of a single implementation form (e.g., discussed only as a method or device), the implementation of the features discussed can also be implemented in other forms (e.g., programs). The apparatus can be implemented, for example, in appropriate hardware, software, and firmware. The method can be implemented, for example, in an apparatus such as a processor, which generally refers to a processing device, such as a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, such as computers, mobile phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate information communication between end users.

[0040] According to an example, device 30 belongs to a set including the following devices:

[0041] mobile device;

[0042] communications equipment;

[0043] Gaming equipment;

[0044] Tablet computer (or tablet computer);

[0045] laptops;

[0046] Still picture camera;

[0047] Camera;

[0048] Encoding chip;

[0049] Server (such as a broadcast server, video-on-demand server, or web server).

[0050] Figure 4 An example of an embodiment of the syntax of a stream when data is transmitted through a packet-based transport protocol is shown. Figure 4 An example of the structure 4 of a stream encoding a point cloud according to the present principles is shown. The structure resides in a container that organizes the stream into independent syntax elements. The structure may include a header portion 41, which is a collection of data common to each syntax element in the stream. For example, the header portion contains some metadata about the syntax elements, describing the characteristics and role of each element. The structure includes a payload, which contains syntax elements 42 and at least one syntax element 43 (for example, there may be a syntax element 43 for each type of attribute data, such as one for color, one for reflectivity, one for normal vector, etc.). Syntax element 42 contains data representing the geometric structure of the point cloud, i.e., a series of bits representing an octree. Syntax element 43 is part of the data stream payload and may contain data encoding the attribute values represented in the point cloud or as residual data for a prediction step.

[0051] The embodiments described herein can be implemented, for example, in a method or process, an apparatus, a computer program product, a data stream, or a signal. Even if discussed only in the context of a single implementation form (e.g., discussed only as a method or device), the implementation of the features discussed can also be implemented in other forms (e.g., programs). The apparatus can be implemented, for example, in appropriate hardware, software, and firmware. The method can be implemented, for example, in an apparatus such as a processor, which generally refers to a processing device, such as a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, such as computers, mobile phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate information communication between end users.

[0052] The implementation of the various processes and features described herein can be embodied in a variety of different devices or applications, particularly, for example, devices or applications associated with data encoding, data decoding, view generation, texture processing, and other processing of images and related texture information and / or depth information. Examples of such devices include encoders, decoders, post-processors for processing decoder outputs, pre-processors for providing inputs to encoders, video encoders, video decoders, video codecs, web servers, set-top boxes, laptops, personal computers, mobile phones, PDAs, and other communication devices. It should be clear that the device can be mobile and can even be installed in a mobile vehicle.

[0053] Furthermore, the methods may be implemented by instructions executed by a processor, and such instructions (and / or data values resulting from the implementation) may be stored on a processor-readable medium, such as an integrated circuit, a software carrier, or other storage device, such as a hard disk, a compact disc ("CD"), an optical disc (e.g., a DVD, commonly known as a digital versatile disc or digital video disc), a random access memory ("RAM"), or a read-only memory ("ROM"). The instructions may constitute an application program tangibly embodied on a processor-readable medium. The instructions may, for example, be hardware, firmware, software, or a combination thereof. The instructions may, for example, be found in an operating system, a separate application program, or a combination of both. Thus, a processor may, for example, be characterized as a device configured to perform a process and a device including a processor-readable medium (such as a storage device) having instructions for performing the process. Furthermore, the processor-readable medium may store data values resulting from the implementation in addition to or in lieu of the instructions.

[0054] As will be apparent to those skilled in the art, an implementation may generate various signals formatted to carry information that may, for example, be stored or transmitted. The information may, for example, include instructions for performing a method, or data generated by one of the implementations. For example, a signal may be formatted to carry as data the rules for writing or reading the grammar of the embodiment, or to carry as data the actual grammar values written by the embodiment. Such a signal may be formatted, for example, as an electromagnetic wave (e.g., using the radio frequency portion of the spectrum) or as a baseband signal. The formatting may, for example, include encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may, for example, be analog or digital information. The signal may be transmitted over a variety of different wired or wireless links, as is known. The signal may be stored on a processor-readable medium.

[0055] Many implementations have been described. However, it should be understood that various modifications are possible. For example, elements of different implementations may be combined, supplemented, modified, or removed to produce other implementations. Furthermore, one of ordinary skill in the art will appreciate that other structures and processes may be substituted for the disclosed structures and processes, and that the resulting implementations will perform at least substantially the same functions in at least substantially the same manner to achieve at least substantially the same results as the disclosed implementations. Therefore, this application contemplates these and other implementations.

Claims

1. A method comprising: Get a node tree that links nodes representing point clouds; as well as For a considered node in the node tree, if the level of the considered node in the node tree is lower than a first threshold, the number of sibling nodes of the considered node is greater than zero, and the number of valid neighbor nodes of the considered node is less than a second threshold, then predict an attribute value for the considered node. 2 . The method of claim 1 , wherein a neighbor node is valid if the difference between the value of the attribute of the neighbor node and the value of the same attribute of the parent node of the considered node is lower than a first percentage or higher than a second percentage. The method according to claim 1 , wherein the node tree is an octree. 4 . The method according to claim 1 , wherein a sibling node of a node is another node having the same parent node as the node, and a neighbor node is a parent node of the node or a valid sibling node of the parent node of the node.

5. A device comprising a memory associated with a processor, the processor configured to: Obtaining a node tree linking nodes representing the point cloud; and For a considered node in the node tree, if the level of the considered node in the node tree is lower than a first threshold, the number of sibling nodes of the considered node is greater than zero, and the number of valid neighbor nodes of the considered node is less than a second threshold, then predict an attribute value for the considered node. 6 . The apparatus of claim 5 , wherein a neighbor node is valid if the difference between the value of the attribute of the neighbor node and the value of the same attribute of the parent node of the considered node is lower than a first percentage or higher than a second percentage.

7. The apparatus according to claim 5 or 6, wherein the node tree is an octree.

8. The apparatus according to any one of claims 5 to 7, wherein a sibling node of a node is another node having the same parent node as the node, and a neighbor node is a parent node of the node or a valid sibling node of the parent node of the node.

Images