A method for compressing point cloud attributes using implicit inter-frame prediction.

Abstract

A method and device for encoding and decoding a high-density point cloud sequence are disclosed. Encoding involves obtaining a temporal prediction of the point cloud, compressing it, and decompressing it. According to this principle, rendering attributes are mapped to the temporal prediction on the one hand and to the decompressed point cloud on the other. The first is subtracted from the second to obtain an encoded residual along with the compressed geometry. In decoding, the reverse steps are performed.

Description

Technical Field

[0001] This principle generally relates to the area of compression of point clouds containing attribute data. This document is also understood in the context of encoding, formatting, and decoding data representing point clouds for rendering volumetric content on an end-user device such as, for example, a mobile device or a head-mounted display (HMD). In particular, this document relates to a method for compressing point cloud attributes by implicit inter-frame prediction.

Background Art

[0002] This section is intended to introduce the reader to various aspects of the art, which may be related to various aspects of the present disclosure described and / or claimed below. This discussion is thought to be useful in providing the reader with background information to facilitate a better understanding of the various aspects of the principle. Accordingly, it should be understood that these descriptions should be read from this perspective and should not be read as an admission of prior art.

[0003] Advances in 3D (three-dimensional) capture and rendering technologies are enabling new applications and services in the fields of autonomous driving, cultural heritage archiving, immersive telepresence, and virtual / augmented reality. Point clouds have emerged as one of the primary 3D scene representations for such applications. A point cloud frame consists of a set of 3D points, each represented by its 3D position and, if applicable, several rendering attributes such as color, transparency, reflectivity, and normal vectors. Point clouds can be dynamic. A temporally continuous sequence of point clouds is captured. Each point cloud is a frame in the sequence and is rendered according to the timestamp within the sequence. Compressing dynamic point clouds is a technical challenge due to the enormous amount of data. Compression of point cloud sequences can be performed using a combination of tools, e.g., octotree + triangle soup for geometry coding, motion-compensated inter-frame prediction of geometry (octotree and triangle soup), context-adaptive arithmetic coding of octotree and triangle soup, and region-adaptive hierarchical transformation (RAHT) for in-frame color coding.

[0004] Point cloud coding schemes that combine such tools leverage spatial (intra-frame) and temporal (inter-frame) redundancy for geometry compression. However, for rendering attributes, only intra-frame redundancy is removed. To achieve higher compression ratios, solutions have recently been proposed that add temporal prediction of color attributes from previously coded point cloud frames, taking into account movement between two consecutive point cloud frames. However, such compression schemes achieve high performance at the expense of encoder complexity. In fact, all prediction modes must be computed, their performance measured at each node, and the bitrate for signaling the selected prediction mode to the decoder increases with the depth of the octotree representing the 3D point cloud. Therefore, there is a lack of solutions that provide a lightweight compression scheme for compressing point cloud attributes without the need to compute, select, and signal various prediction modes, while still leveraging temporal redundancy. [Overview of the project]

[0005] Below is a simplified overview of the Principle to provide a basic understanding of some aspects of it. This overview is not a comprehensive summary of the Principle. It is not intended to identify the main or important elements of the Principle. The following overview merely presents some aspects of the Principle in a simplified form as a prelude to the more detailed explanation provided below.

[0006] This principle relates to a method for encoding a sequence of point clouds into a data stream. The point cloud of the sequence includes geometry and rendering attributes. The method includes obtaining a temporal prediction of the current point cloud by motion compensating for the geometry and rendering attributes of the previously decoded point cloud of the sequence for the current frame. Next, the geometry of the current point cloud is compressed, the compressed geometry is encoded into a data stream, and the reconstructed point cloud is obtained by decompressing the compressed geometry. In this step, dual transfer is performed. The mapped current point cloud is obtained by mapping the rendering attributes of the current point cloud to the geometry of the reconstructed point cloud, and the mapped temporal prediction is obtained by mapping the rendering attributes of the temporal prediction to the geometry of the reconstructed point cloud. The residual point cloud is then obtained by subtracting the attributes of the mapped temporal prediction from the attributes of the mapped current point cloud. The rendering attributes of the resulting residual point cloud are encoded in-frame into a data stream. In one embodiment, low-pass filtering is applied to the rendering attributes of the mapped temporal prediction.

[0007] This principle also applies to devices having memory associated with a processor configured to implement the method described above.

[0008] This principle also relates to a method for decoding a sequence of point clouds, where the point cloud of the sequence includes geometry and rendering attributes. The method includes decoding the compressed geometry from a data stream and obtaining a reconstructed point cloud by decompressing the compressed geometry. The temporal prediction of the reconstructed point cloud is obtained by motion compensating for the geometry and rendering attributes of the previously decoded point cloud of the sequence. The mapped temporal prediction is obtained by mapping the rendering attributes of the temporal prediction to the geometry of the reconstructed point cloud. Next, in-frame decoding of the rendering attributes obtained from the data stream is performed, and the attributes of the reconstructed point cloud are set by adding the decoded rendering attributes to the attributes of the temporal prediction. In one embodiment, low-pass filtering is applied to the rendering attributes of the mapped temporal prediction.

[0009] This principle also applies to devices having memory associated with a processor configured to implement the method described above. [Brief explanation of the drawing]

[0010] Reading the following description will help to better understand this disclosure and reveal other specific features and advantages, and this description refers to the attached drawings. [Figure 1] The point cloud attribute coding method based on this principle is illustrated diagrammatically. [Figure 2] The decoder scheme corresponding to the encoding scheme in Figure 1, based on this principle, is shown below. [Figure 3] An exemplary architecture of a device that may be configured to implement a method for encoding or decoding point clouds based on this principle is shown. [Figure 4] This shows an example of one embodiment of the syntax of a stream when data is transmitted via a packet-based transmission protocol. [Figure 5] A two-speed encoder for point cloud attributes based on this principle is graphically shown. [Modes for carrying out the invention]

[0011] The principle is described more fully below with reference to the accompanying drawings illustrating examples of the principle. However, the principle may be embodied in many alternative forms and should not be construed as being limited to the examples expressed herein. Thus, the principle is open to various modifications and alternative forms, specific examples of which are shown as examples in the drawings and described in detail herein. However, there is no intention to limit the principle to the specific forms disclosed, but rather this disclosure should be understood to encompass all modifications, equivalents, and alternatives that fall within the spirit and scope of the principle as defined in the claims.

[0012] The terminology used herein is intended solely to illustrate specific examples and is not intended to limit the principles. Where used herein, the singular forms “a,” “an,” and “the” are intended to include the plural form unless otherwise specified in the context. Where used herein, the terms “equip,” “equip,” “contain,” and / or “contain” identify the presence of the described feature, integer, step, action, element, and / or component, but do not exclude the presence or addition of one or more other features, integers, steps, actions, elements, components, and / or groups thereof. Furthermore, where an element is referred to as “responding to” or “connecting to” another element, it may directly respond to or be connected to the other element, or there may be an intervening element. In contrast, where an element is referred to as “directly responding to” or “directly connected to” another element, there is no intervening element. Where used herein, the term “and / or” includes any combination of one or more of the enumerated items relating to it and may be abbreviated as “ / .”

[0013] In this specification, terms such as "first" and "second" may be used to describe various elements, but it should be understood that these elements should not be limited by these terms. These terms are used solely to distinguish one element from another. For example, without deviating from the teachings of this principle, the first element may be called the second element, and similarly, the second element may be called the first element.

[0014] Some diagrams include arrows on the communication path to indicate the primary direction of communication, but please understand that communication may occur in the opposite direction to the depicted arrow.

[0015] Some examples are illustrated with block diagrams and operation flowcharts, where each block represents a circuit element, module, or portion of code containing one or more executable instructions to implement a specified logical function. Note that in other implementations, the functions described in a block may occur in a different order than those listed. For example, two blocks shown consecutively may actually be executed substantially simultaneously, or blocks may sometimes be executed in reverse order depending on the functions they involve.

[0016] Any reference in this specification to “by example” or “in one example” means that a particular feature, structure, or characteristic described in relation to the example may be included in at least one implementation of the principle. The occurrence of the phrase “by example” or “in one example” in various places in this specification does not necessarily refer to the same example, nor do separate or alternative examples necessarily exclude each other.

[0017] Reference numerals appearing in the claims are illustrative only and shall not limit the scope of the claims. Although not expressly described, these examples and modifications may be used in any combination or partial combination.

[0018] In compression methods for high-density point cloud sequences, it is common practice to first compress the point cloud geometry (the 3D coordinates of the points) before rendering the attributes. On the encoder side, the transfer of rendering attributes (also called "recoloring") is then performed, assigning the original point colors to the points in the decoded geometry. In fact, in the case of lossy geometry compression, the positions of the original points may be displaced within the decoded point cloud. Subsequently, rendering attributes are encoded on top of the decoded geometry, independently of how the geometry is encoded. On the decoder side, the geometry is first decoded, and the reconstructed geometry is available for rendering the decoded attributes in the second step.

[0019] This principle describes a compression method for compressing / decompressing attributes of dynamic point clouds, using motion-compensated inter-frame prediction combined with intra-frame prediction associated with hierarchical transformation.

[0020] Figure 1 schematically illustrates the point cloud attribute coding scheme based on this principle. 3D motion estimation 11 is performed between two consecutive point clouds from a sequence of point clouds (called point cloud frames), namely, the current target frame PCt to be encoded and the previously decoded reference frame PCt-1. Any motion estimation algorithm can be used in this method. This assumes only a block-based output motion format, providing motion vectors for each occupied cube (or "node" in an octotree-based representation) in the target point cloud frame, pointing to a corresponding location in the reference point cloud frame. The geometry encoder compresses the geometry. The compressed geometry is encoded into a data stream. The compressed geometry is then decompressed to obtain the reconstructed geometry. The geometry encoder utilizes 3D motion estimation to remove temporal redundancy and achieve the best geometry compression ratio. In the case of lossy compression, the reconstructed geometry is different from the input geometry.

[0021] Motion compensation interframe prediction of the attributes of the target point cloud frame is performed by module 12 and the decoded geometry

[0022]

Number

[0023]

Number

[0024]

Number

[0025]

Number

[0026]

Number

[0027]

Number

[0028]

Number

[0029] Figure 2 shows a decoder scheme corresponding to the encoding scheme in Figure 1, based on this principle. The same module is used.

[0030] In the encoding and decoding methods based on this principle, the attributes of the temporal point cloud prediction are low-pass filtered by module 14 before being subtracted from the attributes of the current input frame, removing noise and outlier attribute values. Such low-pass filtering of the point cloud attributes is performed in two steps. First, the point cloud is divided into fixed-size N×N×N non-overlapping 3D blocks, where N is equal to, for example, 4, 8, or 10. Next, all occupied voxels within a block (i.e., voxels containing at least one point) are assigned the same attribute value A, which is the average of the attribute values ​​in adjacent blocks, weighted by the number of points in each such block. For example, the block weights are W1 for the block itself, W2 for six adjacent blocks that share a face with the block, and W3 for twelve adjacent nodes that share an edge, and equation Eq1 is generated.

[0031]

number

[0032]

number

[0033]

number

[0034] In one embodiment, the attribute transformation performed by module 16 is an RAHT transformation, as implemented within an MPEG G-PCC encoder, where transformation region predictions are added to the basic RAHT transformation to generate quantizable and entropy-encodeable high-pass residuals. These predictions are obtained by averaging predictions from nodes already decoded at lower resolutions or predictions near causal decoding at the same resolution level. Such intra-frame predictions can capture inter-layer and intra-layer redundancy by averaging predictions from nodes already decoded at lower resolutions or predictions near causal decoding at the same resolution level. By inputting temporal prediction residuals according to this principle, both temporal and spatial redundancy are utilized.

[0035] Figure 5 schematically illustrates a two-speed encoder for point cloud attributes based on this principle. Low complexity (LC) path 51 and high complexity (HC) path 52 are shown. Several processing blocks are common to both paths. This method is particularly suitable for low-bitrate operating points where explicit signaling of the prediction mode is no longer negligible compared to the overall bitrate, as coarse quantization makes it easier to encode input residual attributes. Simulation results demonstrate competitive performance at low bitrates compared to "all-intra" (i.e., no temporal prediction) methods or more complex intra / inter-hybrid methods with prediction mode signaling. However, at medium to high bitrates, the input temporal prediction residuals become more difficult to encode, and performance degrades. Furthermore, for sequences where motion compensation provides an estimated point cloud very similar to the current point cloud, especially for sequences with little motion, the method based on this principle offers competitive performance even at high bitrates.

[0036] Therefore, as shown in Figure 5, a two-speed coding scheme is established, having a low-complexity encoder incorporated within a more complex hybrid encoder according to the principles of the present invention. The low-complexity coding workflow is activated depending on the target bitrate or input sequence characteristics of the encoder, and this determination is signaled to the decoder in the data stream.

[0037] Figure 3 shows an exemplary architecture of a device 30 that may be configured to implement a method for encoding or decoding a sequence of point clouds according to the principles of the present invention. The encoder may implement the method described in relation to Figure 1, and the decoder may implement the method described in relation to Figure 2. Alternatively, the encoder and / or decoder circuits may be linked together, for example, via their bus 31 and / or via the I / O interface 36, in a device according to the architecture of Figure 3. The device 30 comprises the following elements linked together by a data bus and an address bus 31: - For example, a microprocessor 32 (or CPU), which is a DSP (or digital signal processor), -ROM (or read-only memory) 33, -RAM (or random access memory) 34, -Storage interface 35, - I / O interface 36 for receiving data sent from the application, - A power source, such as a battery.

[0038] For example, the power supply is external to the device. In each of the above-mentioned memories, the word "register" as used herein may correspond to a small area (a few bits) or a very large area (e.g., an entire program or a large amount of received or decoded data). ROM33 contains at least a program and parameters. ROM33 may store algorithms and instructions for performing the technology according to this principle. When switched on, CPU32 uploads the program into RAM and executes the corresponding instructions.

[0039] RAM34 contains registers for a program that is executed by CPU32 and uploaded after device30 is switched on, input data, intermediate data for different states of the method, and other variables used to execute the method.

[0040] The implementations described herein may be implemented, for example, in methods or processes, apparatus, computer program products, data streams, or signals. Even when considered only in the context of a single implementation (for example, only as a method or device), the implementation of the features considered may also be implemented in other forms (for example, programs). Apparatus may be implemented, for example, in appropriate hardware, software, and firmware. These methods may be implemented in apparatus, for example, and generally refer to processing devices, such as processors, including computers, microprocessors, integrated circuits, or programmable logic devices. Processors also include communication devices, such as computers, mobile phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate the communication of information between end users.

[0041] For example, device 30 is - Mobile devices, -Communication devices, - Game devices, - Tablet (or tablet computer), -Laptop, -Still camera, -Video camera, - Encoding chip, - Belongs to a set that includes a server (e.g., a broadcast server, a video-on-demand server, or a web server).

[0042] Figure 4 shows an example of one embodiment of the syntax of a stream when data is transmitted via a packet-based transmission protocol. Figure 4 shows an exemplary structure 4 of a stream encoding a sequence of point clouds according to the present principle. The structure resides within a container that organizes the stream into independent syntax elements. This structure may include a header section 41, which is a set of data common to all syntax elements of the stream. For example, the header section includes some metadata about the syntax elements, describing the properties and roles of each of them. This structure includes a payload containing elements of syntax 42 and at least one element of syntax 43 (there may be a syntax element for each type of attribute data, e.g., one for color, one for reflectance, one for normal vector, etc.). The syntax element 42 comprises data representing the geometry of the point cloud, which is a set of bits representing an octotree. The elements of syntax 43 are part of the payload of the data stream and may contain data that encodes attribute values, such as represented as a point cloud or as residual data (called residuals) for the prediction step.

[0043] The implementations described herein may be implemented, for example, in methods or processes, apparatus, computer program products, data streams, or signals. Even when considered only in the context of a single implementation (for example, only as a method or device), the implementation of the features considered may also be implemented in other forms (for example, programs). Apparatus may be implemented, for example, in appropriate hardware, software, and firmware. These methods may be implemented in apparatus, for example, and generally refer to processing devices, such as processors, including computers, microprocessors, integrated circuits, or programmable logic devices. Processors also include communication devices such as smartphones, tablets, computers, mobile phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate the transmission of information between end users.

[0044] The various processes and features described herein may be embodied in a variety of different devices or applications, specifically, for example, in devices or applications associated with data encoding, data decoding, view generation, texture processing, and other image processing, as well as related texture information and / or depth information. Examples of such devices include encoders, decoders, post-processors that process the output from decoders, pre-processors that supply inputs to encoders, video coders, video decoders, video codecs, web servers, set-top boxes, laptops, personal computers, mobile phones, PDAs, and other communication devices. As should be obvious, the devices may be portable and may even be mounted on mobile vehicles.

[0045] Furthermore, the method may be implemented by instructions executed by the processor, and such instructions (and / or data values ​​generated by the implementation) may be stored in a processor-readable medium such as an integrated circuit, a software carrier, or other storage devices such as a hard disk, a compact diskette ("CD"), an optical disc (such as a DVD, often referred to as a digital multipurpose disc or digital video disc), random access memory ("RAM"), or read-only memory ("ROM"). The instructions may form an application program that is tangibly embodied in the processor-readable medium. The instructions may be in hardware, firmware, software, or a combination of both. The instructions may be found in an operating system, a separate application, or a combination of the two. Thus, a processor may be characterized as both, for example, a device configured to perform processing and a device including a processor-readable medium (such as a storage device) having instructions for performing processing. Furthermore, the processor-readable medium may store data values ​​generated by the implementation in addition to, or instead of, instructions.

[0046] As will be apparent to those skilled in the art, the implementation can generate a variety of signals formatted to carry information, which can, for example, be stored or transmitted. The information may include, for example, instructions to carry out a method, or data generated by one of the implementations described. For example, a signal may be formatted as data to convey rules for writing or reading the syntax of the embodiment described, or as data to convey actual syntax values ​​described from the embodiment described. Such a signal may be formatted, for example, as an electromagnetic wave (e.g., using the radio frequency portion of the spectrum) or as a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier wave with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. The signal may be transmitted over a wide variety of different wired or wireless links, as is known. The signal may be stored in a processor-readable medium.

[0047] Several implementations have been described. Nevertheless, it will be understood that various modifications are possible. For example, elements of different implementations may be combined, supplemented, modified, or deleted to produce other implementations. Furthermore, those skilled in the art will understand that other structures and processes may be substituted for the disclosed structures and processes, and that the resulting implementations may perform at least substantially the same functions(s) in at least substantially the same manner(s) as the disclosed implementations to achieve at least substantially the same results(s). Accordingly, these and other implementations are conceived in this application.

Claims

1. A method for encoding a sequence of point clouds into a data stream, wherein the point cloud of the sequence includes geometry and rendering attributes, and the method, for the current frame, - Obtain a temporal prediction of the current point cloud by motion compensating for the geometry and rendering attributes of the previously decoded point cloud of the sequence, - To obtain a reconstructed point cloud by compressing the geometry of the current point cloud, encoding the compressed geometry into the data stream, and decompressing the compressed geometry, - Obtaining the mapped current point cloud by mapping the rendering attributes of the current point cloud to the geometry of the reconstructed point cloud, - To obtain a mapped temporal prediction by mapping the rendering attributes of the temporal prediction to the geometry of the reconstructed point cloud, - Obtaining a residual point cloud by subtracting the rendering attributes of the mapped temporal prediction from the rendering attributes of the mapped current point cloud, - Encoding the rendering attributes of the residual point group into the data stream within the frame, Methods that include...

2. The method according to claim 1, further comprising low-pass filtering the rendering attributes of the mapped temporal prediction.

3. The method according to claim 1 or 2, further comprising encoding data in the data stream indicating that the method according to claim 1 or 2 is being used.

4. A device for encoding a sequence of point clouds into a data stream, wherein the point cloud of the sequence includes geometry and rendering attributes, and the device includes a processor associated with memory, the processor is - By motion-compensating for the geometry and rendering attributes of the previously decoded point cloud of the sequence, a temporal prediction of the current point cloud is obtained. - Compress the geometry of the current point cloud, encode the compressed geometry into the data stream, and obtain the reconstructed point cloud by decompressing the compressed geometry. -By mapping the rendering attributes of the current point cloud to the geometry of the reconstructed point cloud, the mapped current point cloud is obtained. - By mapping the rendering attributes of the temporal prediction to the geometry of the reconstructed point cloud, the mapped temporal prediction is obtained. - Obtain the residual point cloud by subtracting the attributes of the mapped temporal prediction from the attributes of the mapped current point cloud, and - Encode the rendering attributes of the residual point group into the data stream within the frame. A device configured in such a way.

5. The device according to claim 4, wherein the processor is further configured to perform low-pass filtering on the rendering attributes of the mapped temporal prediction.

6. The device according to claim 4 or 5, wherein the processor is further configured to encode data in the data stream indicating that the method described in claim 1 or 2 is being used.

7. A method for decoding a sequence of points, wherein the point cloud of the sequence includes geometry and rendering attributes, and the method is - Decoding the compressed geometry from the data stream and obtaining a reconstructed point cloud by decompressing the compressed geometry, - Obtain a temporal prediction of the reconstructed point cloud by motion compensating for the geometry and rendering attributes of the previously decoded point cloud of the sequence, - To obtain a mapped temporal prediction by mapping the rendering attributes of the temporal prediction to the geometry of the reconstructed point cloud, - Decoding the rendering attributes obtained from the aforementioned data stream within the frame, - Setting the attributes of the reconstructed point cloud by adding the decoded rendering attributes to the attributes of the temporal prediction, Methods that include...

8. The method according to claim 7, further comprising low-pass filtering the rendering attributes of the mapped temporal prediction.

9. The method according to claim 7 or 8, comprising first obtaining data from the data stream indicating whether the method described in claim 1 or 2 is being used, applying the decoding method described in claim 9 or 10 if it is being used, and applying the method indicated by the data otherwise.

10. A device for decoding a sequence of points, wherein the point cloud of the sequence includes geometry and rendering attributes, the device comprises a processor associated with memory, and the processor - Decode the compressed geometry from the data stream, and obtain a point cloud reconstructed by decompressing the compressed geometry. - By motion-compensating for the geometry and rendering attributes of the previously decoded point cloud of the sequence, a temporal prediction of the reconstructed point cloud is obtained. - By mapping the rendering attributes of the temporal prediction to the geometry of the reconstructed point cloud, the mapped temporal prediction is obtained. - Decode the rendering attributes obtained from the aforementioned data stream within the frame. - The system is configured to set the attributes of the reconstructed point cloud by adding the decoded rendering attributes to the attributes of the temporal prediction, device.

11. The device according to claim 10, wherein the processor is further configured to perform low-pass filtering on the rendering attributes of the mapped temporal prediction.

12. The device according to claim 10 or 11, wherein data indicating whether the method described in claim 1 or 2 is being used is first obtained from the data stream, and if it is being used, the method described in claim 9 or 10 is applied, and otherwise the method indicated by the data is applied.

Images