Encoding point cloud attributes by passing function parameters

Abstract

A point cloud, comprising points in a space, with points of the point cloud having attributes, is compressed and encoded. The space is first divided into blocks. For each block comprising at least one point, the geometry of the points is compressed and a transfer function is determined that associates coordinates within the block with attribute values from the at least one point. The compressed geometry and the transfer function are then encoded in a data stream.

Description

Technical Field

[0001] This principle generally relates to the field of encoding, transmitting, and decoding point clouds with attributes. Specifically, this principle relates to formatting the attributes of a point cloud independently of how geometry is encoded or decoded, and in a manner that allows for the parallel encoding and decoding of the geometry and attributes of the point cloud. Background Technology

[0002] This section aims to introduce the reader to various aspects of this art that may relate to the aspects of this principle described and / or claimed below. It is believed that this discussion will help provide the reader with background information to better understand the various aspects of this principle. Therefore, it should be understood that these statements should be read in this context, rather than as an admission of prior art.

[0003] Advances in 3D capture and rendering technologies have enabled new applications and services in fields such as autonomous driving, cultural heritage archiving, immersive telepresence, and virtual / augmented reality. Point clouds have emerged as one of the primary 3D scene representations for such applications. A point cloud frame consists of a set of 3D points, each represented by its 3D position and one or more attributes, such as color, reflectivity, normal vector, or transparency. Dense dynamic point cloud compression based on geometric methods requires a large amount of information. 3D-to-2D projection can leverage existing 2D video codecs to encode 3D point clouds or 3D meshes. For example, the Geometric Point Cloud Compression (G-PCC) encoder for dense dynamic point clouds incorporates several tool modules, including a pruned octagonal tree (octree) plus triangular soup for geometric encoding, a Region Adaptive Hierarchical Transform (RAHT) for color attribute encoding, motion-compensated inter-frame prediction, and context-adaptive arithmetic coding.

[0004] In such a system, geometry and attributes are compressed sequentially. The geometry is compressed first, and then the input attributes are transferred to the reconstructed geometry (after decompression) and compressed again. Therefore, at the decoder end, the reconstructed geometry must be available (fully decompressed) when decoding attributes. For real-time applications, there is a lack of parallelizable encoding / decoding schemes. Summary of the Invention

[0005] The following is a simplified overview of the principle to provide a basic understanding of some aspects of it. This overview is not a comprehensive summary of the principles of the invention. It is not intended to identify key or essential elements of the principle. The following overview presents only some aspects of the principle in a simplified form as an introduction to the more detailed description provided below.

[0006] This principle relates to a method for encoding point clouds contained in space. The points in the point cloud have attributes. First, the space is divided into blocks. For each block containing at least one point, the geometry of the points in the block is compressed. Simultaneously, based on the attribute values ​​of the points in the block, a transfer function is determined that associates the coordinates within the block with the attribute values. Then, the compressed geometry and the transfer function are encoded in a data stream.

[0007] This principle also relates to a device including a processor and memory associated with the processor, the device being configured to implement the above-described method.

[0008] This principle also relates to a method for decoding point clouds encoded according to the principles described above. A data stream is received. This data represents a point cloud contained in a space divided into blocks. The points in the point cloud have attributes. An encoded block includes at least one point and a transfer function that associates coordinates within the block with attribute values. For each encoded block, the geometry of the points in the block is decompressed, and attribute values ​​are set to the decompressed points using the block's transfer function.

[0009] This principle also relates to a device including a processor and memory associated with the processor, the device being configured to implement the above-described method.

[0010] This principle also relates to a data stream representing a point cloud contained in a space divided into blocks, where the points in the point cloud have attributes, and the encoded block includes at least one point and a transfer function that associates the coordinates within the block with the attribute values. Attached Figure Description

[0011] This disclosure will be better understood by reading the following description, which refers to the accompanying drawings, in which: - Figure 1 The diagram illustrates the G-PCC scheme, in which geometry and properties are sequentially compressed; - Figure 2 The diagram illustrates how to divide the 3D space of the 3D point cloud to be encoded; - Figure 3 An example architecture of device 30 is shown, which can be configured to implement encoding and / or decoding methods according to embodiments of this principle; - Figure 4 An example of an embodiment of the syntax of a stream is shown when data is transmitted via a packet-based transport protocol; - Figure 5 The figure illustrates the encoding and decoding scheme for 3D point cloud blocks based on this principle, where geometry and attributes are encoded / decoded in block parallel. - Figure 6 Five examples of predefined graphical configurations are shown. Detailed Implementation

[0012] The principle will now be described more fully with reference to the accompanying drawings, in which examples of the principle are shown. However, the principle can be implemented in many alternative forms and should not be construed as limited to the examples set forth herein. Thus, while the principle is readily adaptable to various modifications and substitutions, specific examples are shown by way of example in the accompanying drawings and will be described in detail herein. However, it should be understood that the principle is not intended to be limited to the specific forms disclosed; rather, this disclosure will cover all modifications, equivalents, and substitutions falling within the spirit and scope of the principle as defined by the claims.

[0013] The terminology used herein is for the purpose of describing particular examples only and is not intended to limit the principles herein. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the terms “comprising,” “including,” “containing,” and / or “include” as used in this specification specify the presence of the stated feature, integer, step, operation, element, and / or component, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. Furthermore, when an element is referred to as “responding” or “connected” to another element, it may directly respond to or be connected to the other element, or there may be intermediate elements present. Conversely, when an element is referred to as “directly responding” or “directly connected” to another element, there are no intermediate elements present. As used herein, the term “and / or” includes any and all combinations of one or more of the related listed items and may be abbreviated to “ / .”

[0014] It should be understood that although the terms first, second, etc., may be used here to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, without departing from the teachings of this principle, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element.

[0015] Although some diagrams include arrows along the communication path to indicate the main direction of communication, it should be understood that communication can occur in the opposite direction to the arrows depicted.

[0016] The reference block diagrams and operation flowcharts illustrate examples where each block represents a circuit element, module, or code section comprising one or more executable instructions for implementing one or more specified logical functions. It should also be noted that in other implementations, the functions(s) marked in the blocks may not occur in the order indicated. For example, two blocks shown consecutively may actually execute substantially simultaneously, or these blocks may sometimes execute in reverse order, depending on the functions involved.

[0017] The phrase "according to the example" or "in the example" as used herein means that a particular feature, structure, or characteristic described in connection with the example may be included in at least one implementation of this principle. The phrases "according to the example" or "in the example" appearing in different places in the specification do not necessarily refer to the same example, nor are they necessarily separate or alternative examples that are mutually exclusive with other examples.

[0018] The reference numerals appearing in the claims are for illustrative purposes only and do not limit the scope of the claims. Although not explicitly described, these examples and variations can be used in any combination or subcombination.

[0019] Figure 1 The diagram illustrates the G-PCC scheme, where geometry and attributes are compressed sequentially. In encoder 11, a point cloud 12 with attributes (e.g., color or transparency) is obtained from the source. Geometry is first compressed, for example, by partitioning the 3D space of the 3D point cloud into voxels and representing it as an octree. Then, the geometry is decompressed, and attributes are transferred to the decompressed geometry for compression, for example, using RAHT. Currently, RAHT is not a parallelizable transformation because it encodes attributes of the entire frame together. Furthermore, the encoding of attributes depends on the result of the geometry encoder, which introduces errors if the geometry encoding is lossy. Therefore, geometry encoding cannot be performed in parallel with attribute encoding because attributes need to be transferred to the decoded geometry before the point cloud attributes are encoded. The same problem occurs at decoder 13, where geometry must be fully decompressed before attributes are decompressed on the decompressed geometry to obtain a decoded point cloud 14 with attributes.

[0020] Based on this principle, a scheme is proposed to encode / decode the properties of point clouds independently of how geometry is encoded / decoded.

[0021] Figure 2 The diagram illustrates how the 3D space of the 3D point cloud to be encoded is divided. The point cloud 3D space is divided into areas of size [size missing]. 3D blocks, such as Figure 2 The 2D example is shown in the figure, where , ,and These are predetermined parameters, such as 2, 4, 8, 16, 32, or 64. According to this principle, geometry and attribute encoding / decoding is performed separately for each block, and only occupied blocks are processed (empty blocks are skipped). Signaling indicating which block is occupied can be transmitted to the decoder, for example, via an octree. In one embodiment, the 3D coordinates of points within a block are represented relative to the block origin.

[0022] Figure 5The figure illustrates an encoding and decoding scheme for 3D point cloud blocks according to the principles of the present invention, wherein geometry and attributes are encoded / decoded in block parallel. Let... It is a pass-through function that returns the value relative to the block origin. The attribute corresponding to the point The function returns a value determined by the parameter vector. Control. In point clouds, attributes can be YUV color, RGB color, YCbCr color, reflectivity, etc., and It can be adopted Equivalent value, each attribute has one parameter, for example and From the input point cloud patch, parameters describing how attributes change with position are obtained. These parameters can be lossless or lossy representations of the attributes; that is, the points in the point cloud... value This can be an exact value or an approximate value for the attribute at that location. Parameter They are then quantized and entropy-encoded to be added to the bitstream. Furthermore, the parameters can be downsampled to further reduce the amount of data to be encoded. On the decoder side, the parameters are entropy-decoded, dequantized, and can be upsampled to obtain the reconstruction parameters for each block. The geometry decoder generates a list of reconstructed points. For the k-th reconstructed point... Reconstructed attributes Sure.

[0023] In the first embodiment, a system called [missing information] is constructed based on the input point cloud blocks. of An image representing the attributes of each possible location within the block. If the input point cloud block is at location If there is only one single point, then The value is equal to the attribute of the point at this location. Value. If the input point cloud block is at location If there is more than one point, it will retrieve the properties of all points at that location. The average value. This occurs when the input point cloud block has juxtaposed points. In other cases, if the location... If there is no dot, then The value is equal to the interpolation attribute based on the average of the nearest points. .parameter These are the transform coefficients of a 3D transform, such as 3D DCT (Discrete Cosine Transform), 3D DST (Discrete Sine Transform), or 3D DFT / FFT (Discrete Fourier Transform, Fast Fourier Transform). The transform to be used can be the same for all blocks, or it can be determined for each block and signaled to the decoder. For each attribute channel, the parameters can be adjusted independently. Perform downsampling. This can be done by reducing the image size. The image can be downsampled directly at a specific resolution, or the coefficients can be transformed by discarding high-frequency coefficients. The transformed coefficients are quantized by dividing them by the quantization matrix and rounding them to the nearest integer. A 3D zigzag path is used to rasterize the coefficients into a vector, with low-pass coefficients at the beginning and high-pass coefficients at the end.

[0024] In this first embodiment, at the decoder, after entropy decoding of the parameters, these values ​​are dequantized by multiplying them by the same quantization matrix used at the encoder. If the parameters are downsampled at the encoder, they are upsampled at the decoder. If downsampling is applied to the transformed coefficients, upsampling can be performed by zero-padding the positions of discarded high-frequency coefficients. If the 3D image has already been processed... By applying downsampling, the image can be reconstructed from the transformed dequantized coefficients. Upsampling is performed by interpolating the image to its original resolution. Finally, the function utilizes the reconstructed geometry of the blocks. To Perform an inverse transform on the given coefficients, and from the reconstructed... Reconstructed attributes are picked up at the corresponding locations in the image.

[0025] In the second embodiment, the parameter vector These are the coefficients of the GFT (Graphical Fourier Transform). In the first step, a graph configuration for connecting nodes is selected from a list of predefined graph configurations within the block, where the nodes are sampling locations. Figure 6 Five examples of predefined graph configurations are shown. The average attribute value of the nearest points is assigned to each node of the graph configuration. The graph configuration can be passed to the decoder via the index of the predefined configuration. These parameters... These are the transform coefficients of the GFT. They are quantized and entropy encoded.

[0026] After entropy decoding and dequantization coefficients, the decoder applies an inverse transform to obtain the reconstructed properties of each node. Finally, using the geometry of the reconstructed blocks, the function... The attributes are interpolated by averaging the attribute values ​​of nodes at 3D positions closest to the reconstruction point.

[0027] In the third embodiment, the function It's a neural network, and it learns parameters. .therefore, This can be a pre-trained network with a large point cloud dataset, a network trained on a scene-by-scene basis (overfitting), or a fine-tuned network initialized using a pre-trained network. In variations, It can also be independent of the point. A general-purpose network can be trained, or it can be trained locally for each block. In the second case, a smaller set of neural networks is trained, and the corresponding parameters are quantized and entropy-encoded in parallel. This speeds up processing.

[0028] In all three embodiments, blocks can be encoded and decoded in parallel. Attribute decoding is performed block by block, thus eliminating the need to wait for the entire geometry to be decoded. Since parameters describe how attributes change based on point positions within a 3D block, they also include an attribute transfer phase. Parameters can be estimated directly from the input point cloud in parallel with geometry encoding. Dividing the point cloud into blocks allows for parallel processing of these blocks, thereby accelerating the encoding / decoding process for real-time applications.

[0029] Figure 3 An example architecture of device 30 is shown, which can be configured to implement encoding and / or decoding methods according to embodiments of this principle. The device is linked to other devices via its bus 31 and / or via I / O interface 36.

[0030] Device 30 includes the following components linked together by data and address buses 31:D: - Processor 32 (or CPU), which is, for example, a DSP (or digital signal processor); - ROM (or read-only memory) 33; - RAM (or random access memory) 34; - Storage interface 35; - I / O interface 36, used to receive data to be transmitted from the application; and - power supply( Figure 2 (not shown in the image), for example, a battery.

[0031] According to one example, the power supply is external to the device. In each of the mentioned memories, the term "register" used in the specification can correspond to a small area (a few bits) or a very large area (e.g., the entire program or a large amount of received or decoded data). ROM 33 includes at least the program and parameters. ROM 33 can store algorithms and instructions to execute techniques according to these principles. When powered on, CPU 32 uploads the program to RAM and executes the corresponding instructions.

[0032] RAM 34 includes in registers the program executed by CPU 32 and uploaded after device 30 is turned on, input data in registers, intermediate data of different states of the method in registers, and other variables in registers used to execute the method.

[0033] The implementations described herein can be implemented, for example, in a method or process, apparatus, computer program product, data stream, or signal. Even if discussed only in the context of a single implementation form (e.g., only as a method or apparatus), the features under discussion can be implemented in other forms (e.g., a program). Apparatus can be implemented, for example, in suitable hardware, software, and firmware. The method can be implemented in an apparatus such as a processor, which generally refers to a processing device, including, for example, a computer, microprocessor, integrated circuit, or programmable logic device. Processors also include communication devices, such as, for example, computers, cellular phones, portable / personal digital assistants (“PDAs”), and other devices that facilitate information communication between end users.

[0034] Device 30 is linked, for example, via bus 31 to a set of sensors 37 and a set of presentation devices 38. Sensors 37 may be, for example, a camera, microphone, temperature sensor, inertial measurement unit, GPS, humidity sensor, IR or UV light sensor, or wind sensor. Presentation devices 38 may be, for example, a display, speaker, vibrator, heater, fan, etc.

[0035] According to the example, device 30 is configured to implement a method based on this principle that encodes, decodes, and renders a 3D point cloud with attributes, and belongs to a set including: - mobile device; - Communication equipment; - Gaming equipment; - Tablet (or tablet computer); - Laptop; - Still image camera; - Camera.

[0036] Figure 4 An example of an embodiment of the syntax of a stream is shown when data is transmitted via a packet-based transport protocol. Figure 4Example structure 4 of a stream of encoded point clouds according to this principle is shown. This structure contains containers that organize the stream into individual syntax elements. The structure may include a header section 41, which is a set of data common to each syntax element of the stream. For example, the header section includes metadata about the syntax elements, describing the properties and roles of each syntax element. The structure includes a payload, which includes syntax elements 42 and at least one syntax element 43 (there may be syntax element 43 for each type of attribute data, such as one for color, one for reflectivity, one for normal vector, etc.). Syntax element 42 includes data representing the geometry of the point cloud, i.e., a series of bits representing 3D blocks, for example, represented as an octree.

[0037] The implementations described herein can be implemented, for example, as methods or processes, apparatuses, software programs, data streams, or signals. Even if discussed only in the context of a single implementation (e.g., only as a method or apparatus), the features under discussion can be implemented in other forms (e.g., programs). Apparatuses can be implemented, for example, in appropriate hardware, software, and firmware. Methods can be implemented in apparatuses such as, for example, processors, which generally refer to processing devices, including, for example, computers, microprocessors, integrated circuits, or programmable logic devices. Processors also include communication devices, such as, for example, smartphones, tablets, computers, cellular phones, portable / personal digital assistants (“PDAs”), and other devices that facilitate communication of information between end users.

[0038] The various processes and features described herein can be implemented in a wide variety of devices or applications, particularly those associated with data encoding, data decoding, view generation, texture processing, and other processing of images and related texture and / or depth information. Examples of such devices include encoders, decoders, post-processors that process the output from decoders, pre-processors that provide input to encoders, video encoders, video decoders, video codecs, web servers, set-top boxes, laptops, personal computers, cellular phones, PDAs, and other communication devices. It should be clear that the device can be mobile, even installed in a mobile vehicle.

[0039] Furthermore, the method can be implemented by instructions executed by a processor, and such instructions (and / or data values ​​generated by the implementation) can be stored on a processor-readable medium, such as, for example, an integrated circuit, a software carrier, or other storage device, such as, for example, a hard disk, an optical disc (“CD”), an optical disc (such as, for example, a DVD, often referred to as a digital universal disc or digital video disc), random access memory (“RAM”), or read-only memory (“ROM”). These instructions can form an application program tangibly embodied on the processor-readable medium. The instructions can be, for example, in hardware, firmware, software, or a combination thereof. The instructions can be found, for example, in an operating system, a separate application program, or a combination of both. Thus, a processor can be characterized, for example, as a device configured to execute a process and a device comprising a processor-readable medium (e.g., a storage device) having instructions for executing the process. Furthermore, in addition to or instead of instructions, the processor-readable medium can store data values ​​generated by the implementation.

[0040] As will be apparent to those skilled in the art, the implementation can generate various signals, which are formatted to carry information, for example, that can be stored or transmitted. This information may include, for example, instructions for performing a method, or data generated by one of the implementations. For example, a signal may be formatted to carry as data rules the syntax for writing or reading the described embodiments, or as data actual syntax values ​​written by the described embodiments. Such a signal may be formatted as, for example, electromagnetic waves (e.g., using the radio frequency portion of the spectrum) or as a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier wave with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is known, the signal can be transmitted via a variety of different wired or wireless links. The signal may be stored on a processor-readable medium.

[0041] Many implementations have been described. However, it should be understood that various modifications can be made. For example, elements of different implementations can be combined, supplemented, modified, or removed to produce other implementations. Furthermore, those skilled in the art will understand that other structures and processes can replace those disclosed, and the resulting implementations will perform at least substantially the same functions in at least substantially the same manner to achieve at least substantially the same results as the disclosed implementations. Therefore, these and other implementations are contemplated in this application.

Claims

1. A method for encoding a point cloud contained in a space, wherein the points of the point cloud have attributes, the method comprising: - Divide the space into blocks; - For each block that includes at least one point: ● Compress the geometry of at least one point; ● Determine the transfer function that associates the coordinates within the block with the attribute value based on the attribute value of at least one point; and - Encode compressed geometry and transfer functions in the data stream.

2. An apparatus for encoding a point cloud contained in space, the points of the point cloud having attributes, the apparatus including a memory associated with a processor configured to: - Divide the space into blocks; - For each block that includes at least one point: ● Compress the geometry of at least one point; ● Determine the transfer function that associates the coordinates within the block with the attribute values, based on the attribute values ​​of at least one point; and - Encode compressed geometry and transfer functions in the data stream.

3. A method comprising: - Obtain data from the data stream representing a point cloud contained in a space divided into blocks, where the points in the point cloud have attributes, and the encoded block includes at least one point and a transfer function that associates the coordinates within the block with the attribute values; - For each encoded block: ● Decompress the geometry of at least one point; ● For each of the at least one point, its attribute value is set by using the transfer function of the block.

4. An apparatus including memory associated with a processor, the processor being configured to: - Obtain points of a point cloud from a data stream representing a point cloud contained in a space divided into blocks, wherein the points of the point cloud have attributes, and the encoded blocks include at least one point and a transfer function that associates coordinates within the block with attribute values. - For each encoded block: ● Decompress the geometry of at least one point; ● For each of the at least one point, its attribute value is set by using the transfer function of the block.

5. A data stream representing a point cloud contained in a space divided into blocks, wherein the points of the point cloud have attributes, and the encoded blocks include at least one point and a transfer function that associates coordinates within the block with attribute values.

Images