Method and apparatus for intra or interhybrid compression schemes of point cloud attributes

Abstract

Methods and apparatus for encoding and decoding high-density point cloud sequences are disclosed. The compression scheme for high-density point cloud sequences described herein leverages the temporal redundancy between consecutive point cloud frames in dynamic point cloud sequences to improve the compression performance of point cloud rendering attributes. Furthermore, in dynamic point clouds, not all object parts can be predicted from previous point cloud frames. Some parts are more efficiently predicted in intra-mode, i.e., from already decoded parts of the current point cloud frame (parts at coarse resolution or parts from the vicinity of causal decoding). This principle allows for the establishment of hybrid intra- or inter-compression schemes.

Description

[Technical Field]

[0001] This principle generally relates to the domain of point cloud compression, including attribute data. This specification is also understood in the context of encoding, formatting, and decoding data representing point clouds for rendering volumetric content in end-user devices such as mobile devices and head-mounted displays (HMDs). In particular, this specification relates to hybrid intra or inter-compression schemes for point cloud attribute values. [Background technology]

[0002] This application claims priority to European Patent Application No. 23305539.1, filed on 12 April 2023, the entirety of which is incorporated herein by reference.

[0003] This section aims to introduce to the reader various aspects of art that may be related to the various aspects of the Principle described and / or asserted below. This discussion is intended to provide background information that will help the reader better understand the various aspects of the Principle. Therefore, it should be understood that these descriptions should be read from this perspective, not as an endorsement of prior art.

[0004] Advances in 3D (three-dimensional) capture and rendering technologies have enabled new applications and services in the fields of autonomous driving, cultural heritage archiving, immersive remote presence, virtual reality, and augmented reality. Point cloud data has emerged as one of the primary 3D scene representations in these applications. A point cloud frame consists of a collection of 3D points, each represented by its 3D position and multiple rendering attributes such as color, transparency, reflectivity, and normal vector. Point clouds can be dynamic. A temporally continuous sequence of point clouds is captured. Each point cloud becomes a frame in the sequence and is rendered based on the timestamp within the sequence. Compressing dynamic point clouds is a technical challenge due to the enormous amount of data. Combinations of multiple tools, such as the Octry + Triangle Soup combination for geometry coding, motion-compensated interframe prediction of geometry (Octry and Triangle Soup), context-adaptive arithmetic coding of Octry and Triangle Soup, and Region-Adaptive Hierarchical Transform (RAHT) for in-frame color coding, can be used to perform point cloud compression sequences.

[0005] Point cloud encoding schemes combining such tools leverage spatial (intra-frame) and temporal (inter-frame) redundancy. However, for rendering attributes, only intra-frame redundancy is removed. To achieve better compression ratios, it has been suggested that temporal prediction of rendering attributes, which takes into account the motion between two consecutive point cloud frames, needs to be added. Combining intra-frame and inter-frame predictions in a point cloud rendering attribute compression scheme realizes a hybrid compression scheme based on a paradigm similar to motion-compensated hybrid 2D video codecs. [Brief explanation of the drawing]

[0006] This disclosure will be better understood by reading the following description, which will reveal other specific features and benefits, and this description will refer to the attached drawings, which include the following:

[0007] [Figure 1] This figure illustrates a compression scheme for rendering attributes of dynamic point clouds, showing a combination of motion-compensated interframe and intraframe predictions associated with hierarchical transformations, in accordance with this principle. [Figure 2] This figure shows a decoder scheme based on this principle, corresponding to the encoding scheme in Figure 1. [Figure 3] This figure shows an exemplary architecture of a device configured to implement a method for encoding or decoding point clouds based on this principle. [Figure 4] This figure illustrates an exemplary embodiment of the syntax of a stream when data is transmitted via a packet-based transmission protocol. [Figure 5] This figure shows the binary tree used to notify the decoder of the prediction mode based on this principle. [Figure 6] This diagram shows the octree (as a binary tree), and the top two levels (highest resolution) are based on estimations and predictions derived from this principle. [Modes for carrying out the invention]

[0008] The following is a simplified overview to provide a basic understanding of some aspects of this principle. This overview is not a comprehensive overview of the principle. It is not intended to identify the main or important elements of the principle. The following overview merely presents some aspects of the principle in a simplified form as an introduction to the content that will be detailed later.

[0009] This principle relates to a method for encoding a point cloud sequence. The point cloud of a sequence includes geometry and rendering attributes. The method involves obtaining a temporal prediction of the current point cloud by motion-compensating for the geometry and rendering attributes of the previous frame of the sequence for the current frame. The geometry of the current point cloud is compressed, and the compressed geometry is encoded into a data stream. Next, the temporal point cloud is obtained by restoring the compressed geometry, and the rendering attributes of the current point cloud are mapped to the geometry of the temporal point cloud. For each block of the geometry of the temporal point cloud, a prediction mode for the rendering attributes of the block is selected by comparing the rendering attributes of the temporal prediction of the current point cloud with the rendering attributes of the temporal point cloud. The prediction of the rendering attributes of the block is calculated according to the selected prediction mode, and the residual is obtained by subtracting the prediction from the rendering attributes of the temporal point cloud. The residual and prediction mode are encoded into a data stream.

[0010] In one embodiment, the method further includes calculating an intraframe prediction of the current point cloud. The selection of a prediction mode for a block is further based on this intraframe prediction. In another embodiment, the sequence point cloud and the temporal point cloud are structured as an octree, where blocks of geometry in the temporal point cloud become nodes in the octree, and the selection of a prediction mode is performed for each layer of the octree. In yet another embodiment, the prediction mode is encoded using biner adaptive arithmetic coding.

[0011] This principle also relates to a device including memory associated with a processor on which the above method is implemented.

[0012] This principle also relates to a method for decoding point cloud sequences. The point cloud in a sequence includes geometry and rendering attributes. This method involves decoding the compressed geometry from the data stream and decompressing the compressed geometry to obtain a temporal point cloud. Next, a temporal prediction of the temporal point cloud is obtained by motion-compensating the geometry and rendering attributes of the previously decoded point clouds in the sequence. For each block of geometry in the temporal point cloud, this method involves decoding the prediction mode and residual of the rendering attributes of that block from the bitstream. The prediction of the rendering attributes of that block is calculated based on the temporal prediction and the decoded prediction mode. The rendering attributes of that block are set by adding the residual to the prediction value.

[0013] In one embodiment, the rendering attributes of a block are further calculated based on an intra-frame prediction of the temporal point cloud. In another embodiment, the current point cloud is structured as an octree, with blocks of geometry in the current point cloud becoming nodes in the octree, and the selection of the prediction mode is performed for each layer of the octree. In yet another embodiment, the prediction mode is decoded using binary adaptive arithmetic coding.

[0014] This principle also relates to a device including memory associated with a processor configured to implement the above method.

[0015] The principle is described in more detail later with reference to the accompanying drawings, which illustrate the principle. However, the principle can be embodied in many alternative forms and should not be construed as being limited to the examples described herein. Therefore, various modifications and alternative forms of the principle are possible, specific examples of which are illustrated in the drawings and described in detail herein. However, it should be understood that there is no intention to limit the principle to the specific forms disclosed, but rather that it encompasses all modifications, equivalents, and alternatives that fall within the spirit and scope of the principle as defined in the claims.

[0016] The terms used in this specification are for the purpose of describing specific examples and are not intended to limit the principle. In this specification, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. Further, the terms "comprising", "comprises", "including", and / or "includes" as used in this specification identify the presence of the described features, elements, steps, operations, components, and / or components, but do not preclude the presence or addition of one or more other features, elements, steps, operations, components, components, and / or groups thereof. Further, when an element is described as "responding to" or "connected to" another element, it may be directly responsive or connected, or there may be intervening elements. In contrast, when an element is described as "directly responding to" or "directly connecting to" another element, there are no intervening elements. The term "and / or" as used in this specification includes any and all combinations of one or more of the associated listed items and may be abbreviated as " / ".

[0017] In this specification, terms such as "first", "second", etc. may be used to describe various elements, but it should be understood that these elements should not be limited by these terms. These terms are merely used to distinguish one element from another. For example, the first element may be referred to as the second element, and similarly, the second element may be referred to as the first element, and this may be done without departing from the teachings of the principle.

[0018] Some figures include arrows in the communication path to indicate the main direction of communication, but it should be understood that communication can also occur in the opposite direction to the illustrated arrows.

[0019] Several examples related to block diagrams and operation flowcharts are described, and each block represents a circuit element, module, or part of code that includes one or more executable instructions for implementing the specified logical function. It should also be noted that in other embodiments, the functions described in the blocks may be executed in an order different from the described order. For example, two consecutively shown blocks may actually be executed substantially in parallel, or depending on the functionality involved, the blocks may be executed in reverse order.

[0020] The expressions "by way of example" or "in an example" in this specification mean that the specific function, structure, or feature described in relation to the example can be included in at least one embodiment of the present principle. The expressions "by way of example" or "in an example" appearing in various places in the specification do not necessarily all refer to the same example, and separate examples or alternative examples are not necessarily mutually exclusive with other examples.

[0021] The reference numbers described in the claims are merely illustrative and do not limit the scope of the claims. Even if not explicitly described, the present embodiments and variations can be used in any combination or partial combination.

[0022] In the compression scheme of a high-density point cloud sequence, usually, the point cloud geometry (3D coordinates of points) is compressed prior to the rendering attributes. On the encoder side, then, the rendering attribute transfer (also called "re-coloring") is executed, and the colors of the original points are assigned to the points of the decoded geometry. In fact, in the case of irreversible geometry compression, the positions of the original points may be displaced in the decoded point cloud. Next, independently of the geometry encoding method, the rendering attributes are encoded on the decoded geometry. On the decoder side, first the geometry is decoded, and then the "reconstructed geometry" is available for the decoding of the rendering attributes in the second stage.

[0023] This principle describes a compression scheme for high-density point cloud sequences that leverages temporal redundancy between consecutive point cloud frames in dynamic point cloud sequences to improve the compression performance of point cloud rendering attributes. Furthermore, in dynamic point clouds, not all object parts can be predicted from previous point cloud frames. For example, interframe prediction may be difficult or impossible for non-rigid or self-shielding object parts, especially if the motion model used for temporal prediction is too simple (e.g., pure translational motion only). Some parts can be predicted more efficiently in intra-mode, i.e., from already decoded parts of the current point cloud frame, at a coarser resolution, or from causally related decoding neighborhoods. Based on this principle, a hybrid intra- or inter-compression scheme is established.

[0024] Figure 1 shows a compression scheme for rendering attributes of a dynamic point cloud, employing motion-compensated interframe prediction combined with interframe prediction associated with hierarchical transformation, in accordance with this principle. The encoder includes different modules configured according to this principle.

[0025] Block-based motion estimation 61 is performed between two consecutive point cloud frames (the current "target" frame 51 to be encoded (PC t ), and previously decoded "reference" frame 52 (PC t-1It is executed by )). To implement this module, different motion estimation algorithms can be embodied. This module only assumes a block-based output motion format and provides a motion vector for each occupied cube (``node'' in the octree representation) in the target point cloud frame, indicating the corresponding position in the reference point cloud frame. The geometry encoder compresses the geometry and generates the reconstructed (decoded) geometry. The geometry encoder utilizes 3D motion estimation and removes temporal redundancy to improve the geometry compression rate. Any geometry compression algorithm can embody this module. In the case of irreversible compression, the reconstructed geometry is different from the input geometry.

[0026] Motion-compensated inter-frame prediction 62 of the rendering attributes of the target point cloud frame is performed: The points of the decoded geometry 53 are displaced in the direction of the estimated motion vector, and the corresponding decoded rendering attribute A ^ t-1 performs a temporal prediction of the rendering attribute A of the current frame, which is mapped to the temporal prediction of the geometry G p t of. p t

[0027] The rendering attribute transfer processing block registers the input rendering attribute on top of the current decoded geometry G ^^ t . In a variant, the temporally predicted rendering attribute is also registered on top of the reconstructed geometry. In this case, the input rendering attribute and the temporally predicted rendering attribute share the same geometry. In that case, there is always a temporal rendering attribute predictor for the input rendering attribute.

[0028] The input frame is decomposed into resolution layers according to the octree representation of the decoded geometry. From the root node with a single averaged attribute for the entire frame to the leaf nodes with the attributes of each point.​

[0029] For each resolution layer, multiple prediction modes are evaluated for each node in the octry. Multiple interframe predictors are computed, using rendering attributes already coded in the target point cloud in causally related encoding neighborhoods in coarser resolution layers or the current resolution layer. Interframe predictors are derived from interframe predicted point cloud frames. Additional predictors may also mix already computed intraframe and interframe values. Ultimately, no prediction is made (directly coding the current rendering attribute values). The encoder decision module 66 selects the optimal prediction mode for the current node that simultaneously minimizes the prediction error and bitrate using rate-distortion optimization (RDO) criteria. The coding mode is encoded using context-based arithmetic coding.

[0030] The rendering attributes of the current node's child nodes (up to 8 if the node is fully occupied) are predicted in the selected coding mode. Both the predicted rendering attribute values ​​of the child nodes and the input values ​​are input to the hierarchical transformation used (e.g., the RAHT transformation), the difference between the predicted and input transformation coefficients is quantized, and then entropy encoded in a binary arithmetic encoder.

[0031] Figure 2 shows a decoder scheme based on this principle, corresponding to the encoding scheme in Figure 1. The same module is used.

[0032] As a non-exhaustive list of selectable prediction modes for each node, the following 28 types are proposed herein.

[0033] [Table 1]

[0034] (Intraframe prediction mode) In this type of prediction mode, child attributes are predicted by a weighted average of the neighboring node attributes at previous levels ("parent" and "uncle") and the rendered attributes of neighboring child nodes ("cousins") already encoded or decoded at the same level. The weights are optimized during the encoding stage for a given point cloud. Multiple weight sets can be pre-calculated, hardcoded, and transmitted to the decoder. In the non-exclusive list above, four sets are proposed for intra-frame prediction (modes 2-5).

[0035] (Interframe prediction mode) In the list above, one such mode (mode 6) is proposed. The predictor uses motion-compensated time-predicted point cloud frame A. p t This is the rendering attribute of the corresponding node in the context.

[0036] (Hybrid Intra-Inter Predictive Mode) The list above proposes four modes of this type (modes 7-10). In the intra-frame prediction mode, not all adjacent nodes at the same level are used for weighted prediction; only nodes that have already been decoded in the causally related decoding neighborhood are used. This hybrid intra-interframe mode replaces missing "cousin nodes" with corresponding nodes at the same location in the time-predicted point cloud frame.

[0037] (Interframe prediction mode with motion vector refinement) 3D motion vectors are optimized for geometry prediction and used within hybrid inter- or intra-geometry encoding processes, but they do not always provide optimal predictions of rendering attributes (due to limitations of the motion model in non-rigid deformations, object occlusion, lighting effects, and light changes). To overcome the limitations of the inter-frame prediction mode (mode 6), ±Δ variations around points in the temporal prediction point cloud frame are used to provide alternative temporal predictions of rendering attributes. In the non-exhaustive list above, 18 increments (modes 11-28) are tested. The value of Δ is equal to 1 / 2 or 1 / 4 of the node size along the x, y, or z axis. Thus, the value of Δ changes from level to level, but remains constant for all nodes within the same level. This pattern has room for improvement. For example, allowing Δ increments along the x, y, and z axes simultaneously would yield eight additional prediction modes.

[0038] The encoder selects a prediction mode for each octree node. First, for each available prediction mode, the difference between the prediction conversion coefficient and the conversion coefficient of the rendering attribute currently being encoded is calculated. This difference is quantized, and the number of bits required to encode that coefficient is estimated.

[0039] Next, the rendering attributes are reconstructed by dequantizing the difference in quantized coefficients, re-adding it to the predicted value, and performing an inverse transform. The root mean square error (MSE) of the reconstructed rendering attributes based on each prediction mode is calculated.

[0040] A prediction mode is selected that jointly minimizes the prediction error and transmission rate. In this process, the cost function MSE+λRate is used, where Rate represents the average number of bits per child node required to encode the difference in quantization coefficients and to notify the prediction mode.

[0041] The weighting coefficient λ is determined by (Equation 1).

[0042]

number

[0043] Here, μ MSE μ is the mean squared error of the null prediction. Rate is the average number of bits per child node for encoding the null prediction. These average values ​​are learned during encoding by (Equation 2) and (Equation 3), where i is the index of the current encoding node.

[0044]

number

[0045] The value is initialized by (Equation 5).

[0046]

number

[0047] Figure 5 shows the binary tree used to notify the decoder of the prediction mode according to this principle. Each leaf of this tree corresponds to one of the 28 prediction modes in the non-exhaustive list above. Meanwhile, the encoder traverses the tree until it reaches the leaf corresponding to the selected prediction mode. At each node it passes through, a binary flag indicates the direction to proceed, and this flag is entropy-coded into a bitstream using an arithmetic coder. Meanwhile, the decoder traverses the tree, reads the flag from the bitstream at each node, decodes the direction to proceed, and continues this process until it reaches the leaf of the tree corresponding to the decoded prediction mode.

[0048] Flags are encoded using binary arithmetic coding that specifies the probability of the flag being true or false. This probability is adaptive and depends on two contexts, ctxMode and ctxLevel, which are integers starting from 0 and represent different scenarios. This probability also depends on the node in the tree.

[0049] Node 1 uses the probability isNull[ctxMode]. The flag is encoded as true if the selected prediction is null with probability (1-isNull[ctxMode]), and false otherwise with probability isNull[ctxMode].

[0050] Node 2 is used in probability isIntra[ctxMode]. If the selected prediction is in one of the intra modes (Intra 0-3) (probability 1-isIntra[ctxMode]), the flag true is encoded; otherwise, the flag false is encoded in probability isIntra[ctxMode].

[0051] Node 3 is used in probability isInter[ctxMode]. If the selected prediction is one of the intermodes with probability (1-isInter[ctxMode]), the flag true is encoded; otherwise, the flag false is encoded in probability isInter[ctxMode].

[0052] Node 4 is used in the probability motionCompensation[ctxLevel][0]. If the selected prediction is inter and no motion compensation refinement is performed, the flag is encoded as true in the probability (1-motionCompensation[ctxLevel][0]), otherwise the flag is encoded as false in the probability motionCompensation[ctxLevel][0].

[0053] Nodes 5 and 6 are used with the probability intraChoice[ctxLevel][0]. If the selected prediction is Intra 0 or Hybrid 0, the flag true is encoded with the probability (1-intraChoice[ctxLevel][0]); otherwise, the flag false is encoded with the probability intraChoice[ctxLevel][0].

[0054] Nodes 7 and 8 are used in the probability intraChoice[ctxLevel][1]. If the selected prediction is Intra 1 or Hybrid 1, the flag true is encoded in the probability (1-intraChoice[ctxLevel][1]), otherwise the flag false is encoded in the probability intraChoice[ctxLevel][1].

[0055] Nodes 9 and 10 are used in the probability intraChoice[ctxLevel][2]. If the selected prediction is Intra 2 or Hybrid 2, the flag true is encoded in the probability (1-intraChoice[ctxLevel][2]), otherwise the flag false is encoded in the probability intraChoice[ctxLevel][2].

[0056] Node 11 is used in the probability motionCompensation[ctxLevel][1]. If the selected prediction is an inter with motion compensation refinement (+x, +x+y, +xy, -x, -xy, or x+y), the flag true is encoded in the probability (1-motionCompensation[ctxLevel][1]), otherwise the flag false is encoded in the probability motionCompensation[ctxLevel][1].

[0057] Node 12 is used in the probability motionCompensation[ctxLevel][2]. If the selected prediction is an inter with motion compensation refinement (+y, +y+z, +yz, -y, -yz, or y+z), the flag true is encoded in probability (1-motionCompensation[ctxLevel][2])), otherwise the flag false is encoded in probability motionCompensation[ctxLevel][3].

[0058] Node 13 is used in the probability motionCompensation[ctxLevel][3]. If the selected prediction is an inter with motion compensation refinement (-x, -xy, or x+y), the flag true is encoded in the probability (1-motionCompensation[ctxLevel][3]); otherwise, the flag false is encoded in the probability motionCompensation[ctxLevel][3].

[0059] Node 14 is used in the probability motionCompensation[ctxLevel][4]. If the selected prediction is an inter with motion compensation refinement (-y, -yz, or y+z), the flag true is encoded in the probability (1-motionCompensation[ctxLevel][4]), otherwise the flag false is encoded in the probability motionCompensation[ctxLevel][4].

[0060] Node 15 is used in the probability motionCompensation[ctxLevel][5]. If the selected prediction is an inter with motion compensation refinement (-z, -zx, or z+x), the flag true is encoded in the probability (1-motionCompensation[ctxLevel][5]), otherwise the flag false is encoded in the probability motionCompensation[ctxLevel][5].

[0061] Node 16 is used in the probability motionCompensation[ctxLevel][6]. If the selected prediction is an inter with motion compensation refinement (+x), the flag true is encoded in the probability (1-motionCompensation[ctxLevel][6]), otherwise the flag false is encoded in the probability motionCompensation[ctxLevel][6].

[0062] Node 17 is used in the probability motionCompensation[ctxLevel][7]. If the selected prediction is an inter with motion compensation refinement (+y), the flag true is encoded in the probability (1-motionCompensation[ctxLevel][7]), otherwise the flag false is encoded in the probability motionCompensation[ctxLevel][7].

[0063] Node 18 is used in the probability motionCompensation[ctxLevel][8]. If the selected prediction is an inter with motion compensation refinement (+z), the flag true is encoded in the probability (1-motionCompensation[ctxLevel][8]), otherwise the flag false is encoded in the probability motionCompensation[ctxLevel][8].

[0064] Node 19 is used in the probability motionCompensation[ctxLevel][9]. If the selected prediction is an inter with motion compensation refinement (-x), the flag true is encoded in the probability (1-motionCompensation[ctxLevel][9]), otherwise the flag false is encoded in the probability motionCompensation[ctxLevel][9].

[0065] Node 20 is used in the probability motionCompensation[ctxLevel]

[10] . If the selected prediction is an inter with motion compensation refinement (-y), the flag true is encoded in the probability (1-motionCompensation[ctxLevel]

[10] ), otherwise the flag false is encoded in the probability motionCompensation[ctxLevel]

[10] .

[0066] Node 21 is used in the probability motionCompensation[ctxLevel]

[11] . If the selected prediction is an inter with motion compensation refinement (-z), the flag true is encoded in the probability (1-motionCompensation[ctxLevel]

[11] ), otherwise the flag false is encoded in the probability motionCompensation[ctxLevel]

[11] .

[0067] Node 22 is used in the probability motionCompensation[ctxLevel]

[12] . If the selected prediction is an inter with motion compensation refinement (+x+y), the flag true is encoded in the probability (1-motionCompensation[ctxLevel]

[12] ), otherwise the flag false is encoded in the probability motionCompensation[ctxLevel]

[12] .

[0068] Node 23 is used in the probability motionCompensation[ctxLevel]

[13] . If the selected prediction is an inter with motion compensation refinement (-xy), the flag true is encoded in the probability (1-motionCompensation[ctxLevel]

[13] ), otherwise the flag false is encoded in the probability motionCompensation[ctxLevel]

[13] .

[0069] Node 24 is used in the probability motionCompensation[ctxLevel]

[14] . If the selected prediction is an inter with motion compensation refinement (+y+z), the flag true is encoded in the probability (1-motionCompensation[ctxLevel]

[14] ), otherwise the flag false is encoded in the probability motionCompensation[ctxLevel]

[14] .

[0070] Node 25 is used in the probability motionCompensation[ctxLevel]

[15] . If the selected prediction is an inter with motion compensation refinement (-yz), the flag true is encoded in the probability (1-motionCompensation[ctxLevel]

[15] ), otherwise the flag false is encoded in the probability motionCompensation[ctxLevel]

[15] .

[0071] Node 26 is used in the probability motionCompensation[ctxLevel]

[16] . If the selected prediction is an inter with motion compensation refinement (+z+x), the flag true is encoded in the probability (1-motionCompensation[ctxLevel]

[16] ), otherwise the flag false is encoded in the probability motionCompensation[ctxLevel]

[16] .

[0072] Node 27 is used in the probability motionCompensation[ctxLevel]

[17] . If the selected prediction is an inter with motion compensation refinement (-zx), the flag true is encoded in the probability (1-motionCompensation[ctxLevel]

[17] ), otherwise the flag false is encoded in the probability motionCompensation[ctxLevel]

[17] .

[0073] The probability is updated periodically during encoding each time it is used to encode a flag. If the probability is used to encode a true flag, the probability decreases. If it is used to encode a false flag, the probability increases.

[0074] The probabilities to be maintained are 3 × NbCtxtMode + 21 × NbCtxLevel, which are associated with the node flag and are context-dependent, where NbCtxMode is the number of contexts of type 1 and NbCtxLevel is the number of contexts of type 2, which are detailed below.

[0075] Depending on the control flags below, notifications for some prediction modes can be removed from the notification tree in Figure 5.

[0076] • "enableInter" is set to false when an intra-frame is encoded (a frame encoded using only intra-frame prediction without referencing other frames in the sequence) because inter-frame prediction is never used.

[0077] • "enableHybrid" is set to false if intraframes are encoded, or if the user chooses not to use hybrid mode at all.

[0078] • "enableIntra" is set to false if the RAHT conversion-specific decision flow predicts that intra-frame prediction will result in a decrease in encoding performance.

[0079] • "enableIntraChoice" is set by the user and is used to enable or disable the selection of different weights in intra-prediction and hybrid prediction.

[0080] "enableInterMC" allows the user to configure and enable or disable the refined motion-compensated prediction mode.

[0081] When some of these flags are set to false, the decision tree shrinks. When removing modes from the signaling tree, some nodes may enter a degenerate state. If there is only one traceable path from a node, the flags for that node are not encoded or decoded.

[0082] The context ctxMode combines a set of context information available to the encoder and decoder. There are 216 such contexts, which are indexed by (Equation 6).

[0083]

number

[0084] Here, [Enable prediction flags]: If the enableIntra flag defined in the previous section is false, it is predicted that performance will be worse when intra-frame prediction is used than when it is not used. A second interCompleteness flag related to inter-frame prediction is defined and set to true when inter-frame prediction provides predictions for all child nodes of the current node. This flag is always false for intra-frames. These two flags are merged into one as follows.

[0085] If both are true, then [Enable prediction flags]=2.

[0086] Otherwise, if enableIntra is true, then [Enable prediction flags]=1.

[0087] Otherwise, [Enable prediction flags]=0.

[0088] [Parent prediction mode]: Depends on the prediction mode selected in parent mode.

[0089] If the parent node uses one of the interpretation modes, then [Parent prediction mode]=2.

[0090] Otherwise, if the parent node is using either intra or hybrid prediction mode, [Parent prediction mode]=1.

[0091] Otherwise, [Parent prediction mode]=0.

[0092] [Neighbours prediction mode]: A voting process is performed between neighboring uncle nodes and already decoded cousin nodes to determine the most frequently used prediction mode among them. There are four categories: null, intra (including all intra modes), hybrid (including all hybrid modes), and inter (including all inter modes). A vote counter is created for each category and incremented by the votes of each uncle node and already decoded cousin node. The vote of an uncle node has a weight of 1, and the vote of an already decoded cousin node has a weight of 3.

[0093] Subsequently, if the counter for the inter category is at its highest value, then [Neighbours prediction mode] = 3.

[0094] If the intra-category counter is at its highest value, then [Neighbours prediction mode] = 2.

[0095] If the counter for the null category is at its highest value, then [Neighbours prediction mode]=1.

[0096] Otherwise, [Neighbours prediction mode]=0.

[0097] [Number of children]: If the node has 5 or more children, [Number of children] = 2.

[0098] Otherwise, if the node has 3 or more children, [Number of children] = 1.

[0099] Otherwise, [Number of children] = 0.

[0100] [MSE of the inter-predicted DC value]: The DC value of the node's predicted rendering attribute (without motion refinement) is compared to the parent node's DC value because it is information available to both the encoder and decoder. The mean squared error (MSE) between the predicted DC value and the node's DC value is calculated.

[0101] If the MSE exceeds a threshold (a value known to both the encoder and decoder, e.g., a fixed value or a value set at the coding stage and transmitted in the high-level syntax of the bitstream), then [MSE of the inter prediction DC value] = 1.

[0102] Otherwise, [MSE of the inter-reproduction DC value] = 0.

[0103] Figure 6 schematically shows an octree (as a binary tree), where the top two levels (highest resolution) consist of estimated predictions based on this principle. The octree is a multi-resolution representation of geometry, organized by level. The root node is located at level 0, and its child nodes are located at level 1. This structure continues down to the leaf nodes of MAX_TREE_LVL. These level values ​​are used to generate the ctxLevel context by (Equation 7).

[0104]

number

[0105] Here, NBR_INFER_LVL is a user-defined value (and is transmitted in the bitstream's high-level syntax). The value of ctxLevel is capped by MAX_CTX_LVL.

[0106] If ctxLevel is a negative value, the encoder does not notify the selected prediction mode, and the estimation mode is used. The prediction mode for a node is not notified (because the cost would be enormous at high resolution levels and with a large number of nodes), but it is estimated from contextual information available on both the encoder and decoder sides. In the first embodiment, the estimation of the prediction mode is performed as follows:

[0107] -If the interCompleteness flag is true (a single inter-prediction value for all child nodes) and the parent node was predicted using one of the inter-prediction modes, then the inter-prediction (in this case, without MC refinement) is used.

[0108] - Otherwise, if the enableIntra flag is true, the intra 0 prediction mode will be used.

[0109] - Otherwise, null prediction mode is used.

[0110] In the second embodiment, the prediction mode is estimated as follows (interpretation is not used):

[0111] If the -enableIntra flag is true, intra 0 prediction mode is used.

[0112] - Otherwise, null prediction mode is used.

[0113] Figure 3 shows an exemplary architecture of a device 30 that may be configured to implement a method for encoding or decoding a point cloud sequence according to the present principle. The encoder may implement the method described in relation to Figure 1, and the decoder may implement the method described in relation to Figure 2. Alternatively, the encoder and / or decoder circuits may be interconnected via, for example, a bus 31 and / or an input / output interface 36, in a device based on the architecture of Figure 3. The device 30 consists of the following elements interconnected by a data and address bus 31:

[0114] - For example, a DSP (Digital Signal Processor), such as a microprocessor 32 (or CPU).

[0115] -ROM (Read-Only Memory) 33.

[0116] -RAM (Random Access Memory) 34.

[0117] -Storage interface 35.

[0118] - An input / output interface 36 for receiving data sent from the application.

[0119] - Power source (e.g., battery).

[0120] For example, the power supply is located outside the device. In each memory described herein, the term “register” as used herein can refer to a small capacity area (a few bits) to a very large capacity area (e.g., an entire program, or a large amount of received or decoded data). ROM33 contains at least the program and parameters. ROM33 may store algorithms and instructions for performing the technology based on this principle. The CPU32 loads the program into RAM at startup and executes the corresponding instructions.

[0121] RAM34 contains, in its registers, a program to be executed by the CPU32 after the device 30 is powered on, input data in the registers, intermediate data in different states of the method in the registers, and other variables used to execute the method.

[0122] The embodiments described herein may be implemented, for example, as methods or processes, apparatus, computer program products, data streams, or signals. Even when discussed in the context of a single embodiment (for example, when discussed only as a method or apparatus), embodiments of the features discussed may be implemented in other forms (for example, programs). Apparatus may be implemented, for example, with appropriate hardware, software, and firmware. These methods may be implemented with apparatus such as a processor, where processor refers to a general processing device such as a computer, microprocessor, integrated circuit, or programmable logic device. Processors also include communication devices such as computers, mobile phones, portable or personal digital assistants (PDAs), and other communication devices that enable information communication between end users.

[0123] For example, device 30 belongs to a set that includes the following:

[0124] - Mobile devices.

[0125] -Communication device.

[0126] - Gaming device.

[0127] - Tablet (or tablet computer).

[0128] -Laptop computer.

[0129] -Still image camera.

[0130] - Video camera.

[0131] - Encoding chip.

[0132] - A server (for example, a broadcast server, a video-on-demand server, or a web server).

[0133] Figure 4 shows an example of the syntax of a stream when data is transmitted via a packet-based transmission protocol. Figure 4 shows an exemplary structure 4 of a stream encoding a sequence of point clouds according to this principle. This structure consists of a container that organizes the stream into independent syntactic elements. The structure may include a header portion 41, which is a set of data common to all syntactic elements in the stream. For example, the header portion includes some metadata about the syntactic elements, describing the nature and role of each element. The structure includes a payload containing syntactic elements 42 and at least one syntactic element 43 (there may be one syntactic element 43 for each type of attribute data, e.g., for color, reflectance, normal vector, etc.). Syntactic element 42 contains a set of bits representing the data that represents the geometry of the point cloud, i.e., the octry. Syntactic element 43 is part of the payload of the data stream and may contain encoded data of attribute values ​​represented in the point cloud, or residual data for the prediction step (called residual values).

[0134] The embodiments described herein may be implemented, for example, as methods or processes, apparatus, computer program products, data streams, or signals. Even when discussed in the context of a single embodiment (for example, when discussed only as a method or apparatus), embodiments of the features discussed may be implemented in other forms (for example, programs). Apparatus may be implemented, for example, with appropriate hardware, software, and firmware. These methods may be implemented with apparatus such as a processor, where processor refers to a general processing device such as a computer, microprocessor, integrated circuit, or programmable logic device. Processors also include communication devices such as smartphones, tablets, computers, mobile phones, portable or personal digital assistants (PDAs), and other devices that facilitate information communication between end users.

[0135] The various processing and functional embodiments described herein can be embodied in various different devices or applications, particularly those relating to data encoding, data decoding, view generation, texture processing, and other processing of images and associated texture information and / or depth information. Examples of such devices include encoders, decoders, post-processing devices for processing the output from decoders, pre-processing devices for providing input to encoders, video coders, video decoders, video codecs, web servers, set-top boxes, laptops, personal computers, mobile phones, PDAs, and other communication devices. As is obvious, the devices are portable and may be mounted on mobile vehicles.

[0136] Furthermore, these methods can be implemented by instructions executed by the processor, and such instructions (and / or data values ​​generated by the implementation) may be stored in processor-readable media such as integrated circuits, software carriers, or hard disks, compact discs (CDs), optical discs (e.g., DVDs (also known as digital versatile discs or digital video discs)), random access memory (RAM), or read-only memory (ROM). These instructions can form application programs embodied and recorded on the processor-readable media. Instructions may exist, for example, as hardware, firmware, software, or a combination thereof, and they may exist in, for example, operating systems, standalone applications, or a combination of both. Thus, a processor can be characterized, for example, as both a device configured to execute processes and a device equipped with processor-readable media (such as a storage device) containing instructions for executing those processes. Furthermore, the processor-readable media may store data values ​​generated by the implementation in addition to, or instead of, instructions.

[0137] As those skilled in the art will understand, implementations can generate various signals formatted to transmit, for example, information that can be stored or transmitted. This information may include, for example, instructions for performing a method or data generated by either of the described implementations. For example, a signal may be formatted to transmit rules for writing or reading the syntax of a described embodiment as data, or to transmit syntax values ​​actually written by a described embodiment as data. Such signals may be formatted, for example, as electromagnetic waves (e.g., using the radio frequency portion of the spectrum) or as baseband signals. This formatting may include, for example, encoding of data streams or modulation of transmission waves by encoded data streams. The information carried by the signals may be, for example, analog or digital information. Signals may be transmitted over various wired or wireless links, as is well known. Signals may be stored in processor-readable media.

[0138] Various embodiments have been described. However, it will be understood that various modifications are possible. For example, other embodiments can be generated by combining, complementing, modifying, or removing elements of different embodiments. Furthermore, those skilled in the art will understand that there are other structures and processes that can replace the disclosed structures and processes, and that the resulting embodiments can perform at least substantially the same functions in at least substantially the same way to achieve at least substantially the same results as the disclosed embodiments. Accordingly, this application encompasses these and other embodiments.

Claims

1. A method for encoding a sequence of point clouds in a data stream, wherein the point cloud of the sequence comprises geometry and rendering attributes, and the method, for the current frame, By performing motion compensation on the geometry and rendering attributes of previous frames in the sequence, a temporal prediction of the current point cloud is obtained. The process involves compressing the geometry of the current point cloud, encoding the compressed geometry in the data stream, and restoring the compressed geometry to obtain a temporal point cloud. Mapping the rendering attributes of the current point cloud to the geometry of the time point cloud, For each block of the geometry of the temporal point cloud, a prediction mode for the rendering attributes of the block is selected by comparing the rendering attributes of the current point cloud's temporal prediction with the rendering attributes of the temporal point cloud; the prediction of the rendering attributes of the block is calculated according to the selected prediction mode; the residual is obtained by subtracting the prediction from the rendering attributes of the temporal point cloud; and the residual and the prediction mode are encoded in the data stream. A method that includes [a certain feature].

2. The method according to claim 1, comprising calculating an intraframe prediction of the current point cloud, wherein the selection of a prediction mode for a block is further based on the intraframe prediction.

3. The method according to claim 1 or 2, wherein the sequence point cloud and the temporal point cloud are structured as an octree, the blocks of geometry of the temporal point cloud are nodes of the octree, and the selection of the prediction mode is performed for each layer of the octree.

4. The method according to any one of claims 1 to 3, wherein the prediction mode is encoded using binary adaptive arithmetic coding.

5. A device for encoding a sequence of point clouds in a data stream, wherein the point cloud of the sequence comprises geometry and rendering attributes, and the device comprises a processor associated with memory, the processor for the current frame, By motion-compensating for the geometry and rendering attributes of previous frames in the sequence, a temporal prediction of the current point cloud is obtained. A time-lapse point cloud is obtained by compressing the geometry of the current point cloud, encoding the compressed geometry in the data stream, and restoring the compressed geometry. The rendering attributes of the current point cloud are mapped to the geometry of the time point cloud. For each block of the geometry of the temporal point cloud, a prediction mode for the rendering attributes of the block is selected by comparing the rendering attributes of the current point cloud's temporal prediction with the rendering attributes of the temporal point cloud; the prediction of the rendering attributes of the block is calculated according to the selected prediction mode; the residual is obtained by subtracting the prediction from the rendering attributes of the temporal point cloud; and the residual and the prediction mode are encoded in the data stream. A device configured in such a way.

6. The apparatus according to claim 5, wherein the processor is further configured to calculate an intraframe prediction of the current point cloud and to select a prediction mode for a block, based on the intraframe prediction.

7. The apparatus according to claim 5 or 6, wherein the sequence point cloud and the temporal point cloud are structured as an octree, the blocks of geometry of the temporal point cloud are nodes of the octree, and the selection of the prediction mode is performed for each layer of the octree.

8. The apparatus according to any one of claims 5 to 7, wherein the prediction mode is encoded using binary adaptive arithmetic coding.

9. A method for decoding a sequence of point clouds, wherein the point cloud of the sequence comprises geometry and rendering attributes, and the method is The process involves decoding the compressed geometry from the aforementioned data stream and restoring the compressed geometry to obtain a time-based point cloud, By performing motion compensation on the geometry and rendering attributes of the previously decoded point cloud of the sequence, a temporal prediction of the temporal point cloud is obtained. For each block of the geometry of the temporal point cloud, the prediction mode and residuals for the rendering attributes of the block are decoded from the bitstream, the prediction of the rendering attributes of the block is calculated from the temporal prediction according to the decoded prediction mode, and the rendering attributes of the block are set by adding the residuals to the prediction. A method that includes [a certain feature].

10. The method according to claim 9, wherein the prediction of the rendering attributes of the block is further calculated from the intraframe prediction of the temporal point cloud.

11. The method according to claim 9 or 10, wherein the current point cloud is structured as an octree, the blocks of geometry of the current point cloud are nodes of the octree, and the selection of the prediction mode is performed for each layer of the octree.

12. The method according to any one of claims 9 to 11, wherein the prediction mode is decoded using binary adaptive arithmetic coding.

13. A device for decoding a sequence of point clouds, wherein the point cloud of the sequence comprises geometry and rendering attributes, the device comprises a processor associated with memory, and the processor is A time point cloud is obtained by decoding the compressed geometry from the aforementioned data stream and restoring the compressed geometry. By motion-compensating the geometry and rendering attributes of the previously decoded point cloud of the sequence, a temporal prediction of the temporal point cloud is obtained. For each block of the geometry of the temporal point cloud, the prediction mode and residuals for the rendering attributes of the block are decoded from the bitstream, the prediction of the rendering attributes of the block is calculated from the temporal prediction according to the decoded prediction mode, and the rendering attributes of the block are set by adding the residuals to the prediction. A device configured in such a way.

14. The apparatus according to claim 13, wherein the prediction of the rendering attributes of the block is further calculated from the intraframe prediction of the temporal point cloud.

15. The apparatus according to claim 13 or 14, wherein the current point cloud is structured as an octree, the blocks of geometry of the current point cloud are nodes of the octree, and the selection of the prediction mode is performed for each layer of the octree.

16. The apparatus according to any one of claims 13 to 15, wherein the prediction mode is decoded using binary adaptive arithmetic coding.

Images