Coding method and apparatus, decoding method and apparatus, and coder, decoder, attribute code stream, device and storage medium

By employing a progressive downsampling and neural network transformation encoding method, the limitations of point cloud data storage and transmission bandwidth were solved, achieving more efficient point cloud compression and saving storage space and transmission bandwidth.

WO2024229609A1PCT designated stage expired Publication Date: 2024-11-14GUANGDONG OPPO MOBILE TELECOMMUNICATIONS CORP LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2023/092345
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-05
Publication Date
2024-11-14

AI Technical Summary

Technical Problem

In existing technologies, the limitations of storage space and transmission bandwidth of point cloud data make it difficult to process massive amounts of point cloud data, requiring more efficient compression methods to save storage space and reduce transmission traffic.

Method used

An encoding method using progressive downsampling and neural network transformation is adopted. Attribute sample values ​​are generated by progressive downsampling of the point cloud, and the trained neural network is used for transformation and entropy encoding to generate an attribute code stream.

Benefits of technology

It achieves a more efficient point cloud compression rate, saves bitrate, and improves compression performance, saving approximately 25% bitrate compared to traditional methods.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2023092345_14112024_PF_FP_ABST
    Figure CN2023092345_14112024_PF_FP_ABST
Patent Text Reader

Abstract

Disclosed in the present application are a coding method and apparatus, a decoding method and apparatus, and a coder, a decoder, an attribute code stream, a device and a storage medium. The coding method comprises: performing resolution-by-resolution down-sampling on a first point cloud, so as to obtain attribute sampling values of a plurality of point clouds with different resolutions, wherein the plurality of point clouds with different resolutions at least comprise a point cloud with an (i-1)th resolution and a point cloud with an ith resolution; and according to a coded attribute reconstruction value of the point cloud with the (i-1)th resolution, executing a first coding process on an uncoded attribute sampling value of the point cloud with the ith resolution, so as to obtain an attribute code stream of the point cloud with the ith resolution, wherein the first coding process comprises: according to the attribute reconstruction value of the point cloud with the (i-1)th resolution and the attribute sampling value of the point cloud with the ith resolution, determining a first attribute residual value of the point cloud with the ith resolution; transforming the first attribute residual value of the point cloud with the ith resolution by using a trained first neural network having the ith resolution, so as to obtain a second attribute residual value of the point cloud with the ith resolution; and generating the attribute code stream according to the second attribute residual value of the point cloud with the ith resolution.
Need to check novelty before this filing date? Find Prior Art

Description

Coding and decoding method and device, codec, attribute code stream, device, storage medium Technical Field

[0001] The embodiments of the present application relate to point cloud compression technology, including but not limited to encoding and decoding methods and devices, codecs, attribute code streams, devices, and storage media. Background Art

[0002] A point cloud is a type of three-dimensional data, representing a collection of vectors in a three-dimensional coordinate system. These vectors are typically represented as (x, y, z) geometric coordinates and also include attribute information such as color, material, and / or reflection intensity. Point clouds are generally used to represent three-dimensional objects or scenes. With the advancement of hardware processing power and the rapid development of computer vision, three-dimensional point clouds have become the next generation of immersive multimedia, following audio, images, and video. They are widely used in applications such as virtual reality, augmented reality, autonomous driving, and environmental modeling.

[0003] However, as application demand grows, the processing of massive point cloud data is facing bottlenecks in storage space and transmission bandwidth. To better manage data, conserve server storage space, and reduce transmission traffic and time between servers and clients, point cloud compression has become a key issue in promoting the development of the point cloud industry.

[0004] Summary of the Invention

[0005] In view of this, the encoding and decoding method and apparatus, codec, attribute code stream, device, and storage medium provided in the embodiments of the present application can improve the compression rate of the point cloud, thereby saving the bit rate. The encoding and decoding method and apparatus, codec, attribute code stream, device, and storage medium provided in the embodiments of the present application are implemented as follows:

[0006] According to one aspect of an embodiment of the present application, a coding method is provided, which is applied to an encoder, and the method comprises: performing resolution-by-resolution downsampling on a first point cloud to obtain attribute sampling values ​​of a plurality of point clouds of different resolutions; wherein the plurality of point clouds of different resolutions include at least a point cloud of an i-1th resolution and a point cloud of an i-th resolution, and the point cloud of the i-1th resolution is obtained by performing a voxel downsampling on the point cloud of the i-th resolution; and reconstructing the attribute of the uncoded point cloud of the i-th resolution according to the attribute reconstruction value of the encoded point cloud of the i-1th resolution. The first encoding process is performed to obtain the attribute code stream of the point cloud of the i-th resolution; wherein the first encoding process includes: determining the first attribute residual value of the point cloud of the i-th resolution according to the attribute reconstruction value of the point cloud of the i-1-th resolution and the attribute sampling value of the point cloud of the i-th resolution; using the trained first neural network of the i-th resolution, transforming the first attribute residual value of the point cloud of the i-th resolution to obtain the second attribute residual value of the point cloud of the i-th resolution; generating the attribute code stream according to the second attribute residual value of the point cloud of the i-th resolution.

[0007] According to one aspect of an embodiment of the present application, a decoding method is provided, which is applied to a decoder, and the method includes: parsing an attribute code stream to determine a second attribute residual reconstruction value of a point cloud of an i-th resolution; performing a first decoding process on the second attribute residual reconstruction value of the point cloud of the i-th resolution according to the attribute reconstruction value of the point cloud of the i-th resolution to obtain the attribute reconstruction value of the point cloud of the i-th resolution; wherein the first decoding process includes: using a trained sixth neural network of the i-th resolution to perform an inverse transformation on the second attribute residual reconstruction value of the point cloud of the i-th resolution to obtain the first attribute residual reconstruction value of the point cloud of the i-th resolution; performing voxel upsampling on the attribute reconstruction value of the point cloud of the i-1th resolution to obtain the attribute prediction value of the point cloud of the i-th resolution; wherein the point cloud of the i-1th resolution is obtained by performing a voxel downsampling on the point cloud of the i-th resolution; and determining the attribute reconstruction value of the point cloud of the i-th resolution according to the attribute prediction value of the point cloud of the i-th resolution and the first attribute reconstruction residual value of the point cloud of the i-th resolution.

[0008] According to one aspect of an embodiment of the present application, there is provided an encoding device, which is applied to an encoder, and the device comprises: a downsampling module, configured to perform resolution-by-resolution downsampling on a first point cloud to obtain attribute sampling values ​​of a plurality of point clouds of different resolutions; wherein the plurality of point clouds of different resolutions include at least a point cloud of an i-1th resolution and a point cloud of an i-th resolution, and the point cloud of the i-1th resolution is obtained by performing a voxel downsampling on the point cloud of the i-th resolution; an encoding module, configured to reconstruct the attribute of the encoded i-1th resolution point cloud according to the reconstructed attribute value of the encoded i-1th resolution point cloud, The attribute sampling values ​​of the point cloud are subjected to a first encoding process to obtain the attribute code stream of the point cloud of the i-th resolution; wherein the first encoding process includes: determining the first attribute residual value of the point cloud of the i-th resolution according to the attribute reconstruction value of the point cloud of the i-1-th resolution and the attribute sampling value of the point cloud of the i-th resolution; using the trained first neural network of the i-th resolution to transform the first attribute residual value of the point cloud of the i-th resolution to obtain the second attribute residual value of the point cloud of the i-th resolution; generating the attribute code stream according to the second attribute residual value of the point cloud of the i-th resolution.

[0009] According to one aspect of an embodiment of the present application, an encoder is provided, comprising a first memory and a first processor; wherein the first memory is used to store a computer program that can be run on the first processor; and the first processor is used to execute the encoding method as described in the embodiment of the present application when running the computer program.

[0010] According to one aspect of an embodiment of the present application, a code stream is provided, where the code stream is obtained using the encoding method described in the embodiment of the present application.

[0011] According to one aspect of an embodiment of the present application, a decoding device is provided, which is applied to a decoder, and includes: A parsing module is configured to parse the attribute code stream and determine the second attribute residual reconstruction value of the point cloud of the i-th resolution; a reconstruction module is configured to perform a first decoding process on the second attribute residual reconstruction value of the point cloud of the i-th resolution according to the attribute reconstruction value of the point cloud of the i-th resolution, so as to obtain the attribute reconstruction value of the point cloud of the i-th resolution; wherein the first decoding process includes: using the trained sixth neural network of the i-th resolution to inversely transform the second attribute residual reconstruction value of the point cloud of the i-th resolution to obtain the first attribute residual reconstruction value of the point cloud of the i-th resolution; voxel upsampling the attribute reconstruction value of the point cloud of the i-th resolution to obtain the attribute prediction value of the point cloud of the i-th resolution; wherein the point cloud of the i-1th resolution is obtained by performing a voxel downsampling on the point cloud of the i-th resolution; and determining the attribute reconstruction value of the point cloud of the i-th resolution according to the attribute prediction value of the point cloud of the i-th resolution and the first attribute reconstruction residual value of the point cloud of the i-th resolution.

[0012] According to one aspect of an embodiment of the present application, a decoder is provided, comprising a second memory and a second processor; wherein the second memory is used to store a computer program that can be run on the second processor; and the second processor is used to execute the decoding method as described in the embodiment of the present application when running the computer program.

[0013] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising: a processor adapted to execute a computer program; and a computer-readable storage medium storing a computer program, wherein when the computer program is executed by the processor, the encoding method as described in the embodiment of the present application is implemented, or when the computer program is executed by the processor, the decoding method as described in the embodiment of the present application is implemented.

[0014] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed, the encoding method as described in the embodiment of the present application is implemented, or when the computer program is executed, the decoding method as described in the embodiment of the present application is implemented.

[0015] In an embodiment of the present application, for a layered point cloud attribute lossy compression coding method, at least when transforming the first attribute residual value of the point cloud of the i-th resolution, the trained first neural network of the i-th resolution is used; this is beneficial to saving the bit rate overhead brought by attribute coding. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings herein are incorporated into and constitute a part of this specification. These drawings illustrate embodiments consistent with the present application and, together with the specification, serve to illustrate the technical solutions of the present application. Obviously, the drawings described below are merely some embodiments of the present application. Those skilled in the art can, without inventive effort, derive other drawings from these drawings.

[0017] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0018] FIG1 is a schematic diagram of an implementation flow of an encoding method provided in an embodiment of the present application;

[0019] FIG2 is an example diagram of performing resolution-by-resolution downsampling of a point cloud according to an embodiment of the present application;

[0020] FIG3 is an example diagram of voxel upsampling and voxel downsampling provided by an embodiment of the present application;

[0021] FIG4A is a schematic diagram of the structure of a first neural network provided in an embodiment of the present application;

[0022] FIG4B is a schematic diagram of the structure of the ResNet in the neural network provided in an embodiment of the present application;

[0023] FIG5 is a schematic diagram of the structure of a third neural network provided in an embodiment of the present application;

[0024] FIG6 is a schematic diagram of a probability distribution parameter of a third attribute residual value determined by a third neural network according to an embodiment of the present application;

[0025] FIG7 is a schematic diagram of the structure of a fifth neural network provided in an embodiment of the present application;

[0026] FIG8 is a schematic diagram of the structure of a sixth neural network provided in an embodiment of the present application;

[0027] FIG9 is a schematic diagram of the structure of a seventh neural network provided in an embodiment of the present application;

[0028] FIG10 is a schematic diagram of the structure of an eighth neural network provided in an embodiment of the present application;

[0029] FIG11 is a schematic diagram of an implementation flow of a decoding method provided in an embodiment of the present application;

[0030] FIG12A is a schematic diagram of an implementation flow of an encoding method provided in an embodiment of the present application;

[0031] FIG12B is a schematic diagram of an encoding implementation process using a certain intermediate layer i as an example provided in an embodiment of the present application;

[0032] FIG13 is a schematic diagram showing a comparison of compression performance of the encoding method provided in an embodiment of the present application compared to the RAHT algorithm in MPEG G-PCC;

[0033] FIG14 is a schematic diagram of the structure of an encoding device provided in an embodiment of the present application;

[0034] FIG15 is a schematic structural diagram of a decoding device provided in an embodiment of the present application;

[0035] FIG16 is a schematic diagram of the structure of an encoder provided in an embodiment of the present application;

[0036] FIG17 is a schematic diagram of the structure of the decoder provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] To make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the specific technical solutions of the present application will be further described in detail below in conjunction with the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application but are not intended to limit the scope of the present application.

[0038] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0039] In the following description, references to “some embodiments,” “this embodiment,” “embodiments of the present application,” and examples, etc., describe a subset of all possible embodiments. However, it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict.

[0040] The descriptions such as "first, second, third" appearing in the embodiments of this application are only for illustration and distinction of the described objects. There is no order, nor does it indicate any special limitation on the number of devices in the embodiments of this application, and cannot constitute any limitation on the embodiments of this application.

[0041] The codec architecture and service scenarios described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that, with the evolution of codec architectures and the emergence of new service scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0042] Point cloud is a three-dimensional data format that uses a large collection of three-dimensional coordinate points (x, y, z) to represent the three-dimensional information of objects and scenes. Its huge data volume requires high-performance compression algorithms. Representative point cloud compression algorithms are two technical solutions developed by the Moving Picture Experts Group (MPEG): Video-based Point Cloud Compression (V-PCC) and Geometry-based Point Cloud Compression (G-PCC). Geometric compression in G-PCC is mainly achieved through octree models and / or triangular surface models. V-PCC is mainly achieved through 3D to 2D projection and video compression.

[0043] According to the different compression objects, point cloud compression algorithms are often divided into geometric compression and attribute compression. The object of geometric compression is the coordinates of the point, and the object of attribute compression is the attribute information such as the color and / or reflectivity of the point.

[0044] An embodiment of the present application provides an encoding method, which is applied to an encoder. FIG1 is a schematic diagram of an implementation flow of the encoding method provided in an embodiment of the present application. As shown in FIG1 , the encoding method includes the following steps 101 to 102:

[0045] Step 101: downsampling a first point cloud resolution by resolution to obtain attribute sampling values ​​of a plurality of point clouds of different resolutions; wherein the plurality of point clouds of different resolutions include at least a point cloud of an i-1th resolution and a point cloud of an i-th resolution, and the point cloud of the i-1th resolution is obtained by performing a single voxel downsampling on the point cloud of the i-th resolution;

[0046] Step 102: performing a first encoding process on the unencoded attribute sampling values ​​of the point cloud at the i-th resolution according to the encoded attribute reconstruction values ​​of the point cloud at the i-th resolution, to obtain an attribute code stream of the point cloud at the i-th resolution;

[0047] The first encoding process includes the following steps 1021 to 1023:

[0048] Step 1021 , determining a first attribute residual value of the point cloud at the i-th resolution based on the attribute reconstruction value of the point cloud at the i-th resolution and the attribute sampling value of the point cloud at the i-th resolution;

[0049] Step 1022: using the trained first neural network of the i-th resolution, transform the first attribute residual value of the point cloud of the i-th resolution to obtain the second attribute residual value of the point cloud of the i-th resolution;

[0050] Step 1023 : Generate an attribute code stream according to the second attribute residual value of the point cloud at the i-th resolution.

[0051] In an embodiment of the present application, for a layered point cloud attribute lossy compression coding method, at least when transforming the first attribute residual value of the point cloud of the i-th resolution, the trained first neural network of the i-th resolution is used; this is beneficial to saving the bit rate overhead brought by attribute coding.

[0052] In some embodiments, the first encoding process further includes the following steps 201 to 203:

[0053] Step 201: Dequantize the third attribute residual value of the point cloud of the i-th resolution to obtain the second attribute residual reconstruction value of the point cloud of the i-th resolution; wherein the third attribute residual value is obtained by quantizing the second attribute residual value of the point cloud of the i-th resolution by the encoder.

[0054] Step 202 , performing an inverse transformation on the second attribute residual reconstruction value of the point cloud of the i-th resolution to obtain the first attribute residual reconstruction value of the point cloud of the i-th resolution;

[0055] Step 203 : determining the attribute reconstruction value of the point cloud at the i-th resolution according to the attribute prediction value of the point cloud at the i-th resolution and the first attribute residual reconstruction value of the point cloud at the i-th resolution.

[0056] The following describes further optional implementations and related terms of each of the above steps.

[0057] In step 101, a first point cloud is downsampled resolution by resolution to obtain attribute sampling values ​​of a plurality of point clouds of different resolutions; wherein the plurality of point clouds of different resolutions include at least a point cloud of an i-1th resolution and a point cloud of an i-th resolution, and the point cloud of the i-1th resolution is obtained by performing a single voxel downsampling on the point cloud of the i-th resolution;

[0058] It can be understood that the so-called resolution-by-resolution downsampling refers to voxel downsampling of the point cloud of the previous resolution. For example, as shown in Figure 2, a cube represents a voxel in the point cloud, and the encoder performs voxel downsampling on the point cloud of the i+1th resolution at a ratio of 2×2×2 voxels, that is, all voxels in each 2×2×2 space are merged into one voxel, and its attribute sampling value is equal to the attribute average value of all voxels in the 2×2×2 space or any attribute value therein (such as the maximum value or minimum value or the attribute value of the voxel at a specific position), thereby obtaining a point cloud of the i-th resolution; then the point cloud of the i-th resolution is voxel downsampled at a ratio of 2×2×2 voxels to obtain a point cloud of the i-1th resolution; and so on.

[0059] As the voxel-by-voxel downsampling proceeds, the amount of voxel data in the point cloud decreases exponentially. Therefore, the encoder may not downsample the first point cloud resolution by resolution until it cannot be downsampled. Instead, in some embodiments, the encoder downsamples the first point cloud resolution by resolution until a downsampling threshold is reached, or until the number of voxels in the point cloud obtained by the current downsampling is less than or equal to a voxel number threshold, to obtain attribute sampling values ​​of point clouds of multiple different resolutions; wherein the downsampling threshold is greater than 1, and the voxel number threshold is greater than or equal to 1; in this way, the computational overhead caused by downsampling can be saved.

[0060] Furthermore, in some embodiments, the encoder may perform voxel downsampling on the point cloud of the i+1th resolution according to a first sampling ratio to obtain the point cloud of the i-th resolution; and perform voxel downsampling on the point cloud of the i-th resolution according to a second sampling ratio to obtain the point cloud of the i-1th resolution; wherein the first sampling ratio is the same as or different from the second sampling ratio.

[0061] In the embodiment of the present application, the first sampling ratio and the second sampling ratio are not limited and can be any ratio, such as a ratio of 2×2×2 voxels, a ratio of 2×1×1 voxels, or a ratio of 3×2×2 voxels, etc. The encoder can use the average value of the attributes of all voxels in the sampling ratio as the attribute sampling value of the corresponding voxel in the point cloud of the next resolution, or use the maximum attribute value or the minimum attribute value of all voxels in the sampling ratio as the attribute sampling value of the corresponding voxel in the point cloud of the next resolution, or use the attribute value of the voxel at a specific position among all voxels in the sampling ratio as the attribute sampling value of the corresponding voxel in the point cloud of the next resolution, etc.

[0062] It can be understood that a point cloud with a higher resolution contains more voxels, i.e., a point cloud with a higher point density. A point cloud with the i-th resolution can also be understood as a point cloud with the i-th scale, i.e., "resolution" can also be understood as "scale."

[0063] It should be noted that the point cloud of the i-th resolution generally refers to any point cloud of a non-minimum resolution among the multiple point clouds of different resolutions. In the embodiment of the present application, the encoder performs attribute encoding on point clouds of all resolutions except the minimum resolution point cloud using steps 1021 to 1023. For the minimum resolution point cloud, the encoder can use lossless compression encoding to encode the attribute values ​​of the point cloud, thereby generating an attribute code stream. For the minimum resolution point cloud, the attribute reconstruction value of the point cloud is equal to the attribute sampling value of the point cloud at that resolution.

[0064] In step 1021, a first attribute residual value of the point cloud at the i-th resolution is determined based on the attribute reconstruction value of the point cloud at the i-th resolution and the attribute sampling value of the point cloud at the i-th resolution;

[0065] In some embodiments, voxel upsampling is performed on the attribute reconstructed values ​​of the point cloud at the i-1th resolution to obtain attribute prediction values ​​of the point cloud at the i-th resolution; and a first attribute residual value of the point cloud at the i-th resolution is determined based on the attribute prediction values ​​of the point cloud at the i-th resolution and the attribute sampling values ​​of the point cloud at the i-th resolution. For example, the first attribute residual value of the point cloud at the i-th resolution is equal to the difference between the attribute sampling values ​​of the point cloud at the i-th resolution and the attribute prediction value of the corresponding voxel.

[0066] In the embodiment of the present application, voxel upsampling and voxel downsampling correspond to each other. For example, as shown in FIG3 , the point cloud of the i-1th resolution is the result of voxel downsampling of the point cloud of the i-th resolution. Based on this, the encoder can perform voxel upsampling on the attribute reconstruction value of the point cloud of the i-1th resolution, thereby obtaining the attribute prediction value of the point cloud of the i-th resolution. It should be noted that FIG3 is only a schematic illustration of how the attribute prediction value of the point cloud of the i-th resolution is obtained, and does not constitute a limitation on the scope of protection of the embodiment of the present application. The attribute reconstruction value of the point cloud of the i-1th resolution shown in FIG3 is equal to the attribute prediction value of the point cloud of the i-1th resolution, which is only an example.

[0067] In embodiments of the present application, voxel upsampling can be implemented according to voxel upsampling rules or through a neural network. For example, in some embodiments, the encoder utilizes a trained second neural network at the i-th resolution to perform voxel upsampling on the attribute reconstruction values ​​of the point cloud at the i-1-th resolution to obtain attribute prediction values ​​of the point cloud at the i-th resolution.

[0068] In some embodiments, when the point cloud at the i-1th resolution is the point cloud with the smallest resolution among the multiple point clouds with different resolutions, the attribute reconstruction value of the point cloud at the i-1th resolution is equal to the attribute sampling value of the i-1th resolution; in this case, for the point cloud at the i-1th resolution, the encoder encodes the attributes of the point cloud at this resolution based on the attribute reconstruction value of the point cloud at the i-1th resolution, thereby generating an attribute code stream. Exemplarily, in some embodiments, the encoder uses lossless compression coding to encode the attributes of the point cloud at the smallest resolution.

[0069] In step 1022 , the first attribute residual value of the point cloud of the i-th resolution is transformed using the trained first neural network of the i-th resolution to obtain the second attribute residual value of the point cloud of the i-th resolution.

[0070] In some embodiments, the trained first neural network at the i-th resolution is a first neural network trained using a rate-distortion cost as a loss function. The iterative training of the first neural network may terminate when the rate-distortion cost converges to a rate-distortion cost threshold or when the number of iterative training iterations reaches a threshold.

[0071] In the embodiment of the present application, the structure of the first neural network is not limited, and can be a variety of models that can achieve the transformation of the first attribute residual value. For example, the structure of the first neural network is shown in FIG4A , where The first attribute residual value of the point cloud at the i-th resolution is represented by Latent, and the second attribute residual value of the point cloud at the i-th resolution is represented by Latent. The structure of the ResNet shown in Figure 4A is shown in Figure 4B. For the neural networks mentioned below, the structure of the ResNet can also refer to the structure shown in Figure 4B.

[0072] Step 1023 : Generate an attribute code stream according to the second attribute residual value of the point cloud at the i-th resolution.

[0073] In some embodiments, the encoder can quantize the second attribute residual value of the point cloud of the i-th resolution to obtain the third attribute residual value of the point cloud of the i-th resolution; then, according to the trained third neural network of the i-th resolution, the third attribute residual value of the point cloud of the i-th resolution is entropy encoded to obtain the attribute code stream of the point cloud of the i-th resolution; wherein, the trained third neural network of the i-th resolution is also trained with rate-distortion cost as the loss function.

[0074] Furthermore, in some embodiments, the encoder at least inputs the attribute prediction value of the point cloud of the i-th resolution into the trained third neural network of the i-th resolution to obtain the probability distribution parameters of the third attribute residual value of the point cloud of the i-th resolution; and performs entropy encoding on the third attribute residual value of the i-th resolution according to the probability distribution parameters to obtain the attribute code stream of the point cloud of the i-th resolution.

[0075] In the embodiment of the present application, the structure of the third neural network is not limited. In short, it is sufficient to obtain the probability distribution parameters of the third attribute residual value of the point cloud at the i-th resolution. For example, the structure of the third neural network is shown in Figure 5, where: represents the attribute prediction value of the point cloud at the i-th resolution, and {μ, σ} represents the probability distribution parameter of the third attribute residual value of the point cloud at the i-th resolution.

[0076] In an embodiment of the present application, the third neural network can determine the probability distribution parameters of the third attribute residual value based on one parameter or multiple different parameters, that is, in some embodiments, such as shown in FIG5 , the encoder inputs the attribute prediction value of the point cloud of the i-th resolution into the trained third neural network of the i-th resolution to obtain the probability distribution parameters of the third attribute residual value of the point cloud of the i-th resolution. Alternatively, in other embodiments, the encoder performs voxel upsampling on the first attribute residual reconstruction value of the point cloud of the i-th resolution to obtain the attribute residual prediction value of the point cloud of the i-th resolution; performs voxel upsampling on the attribute feature reconstruction value of the point cloud of the i-th resolution to obtain the attribute feature prediction value of the point cloud of the i-th resolution; inputs the attribute prediction value, attribute residual prediction value and attribute feature prediction value of the point cloud of the i-th resolution into the trained third neural network of the i-th resolution to obtain the probability distribution parameters of the third attribute residual value of the point cloud of the i-th resolution, such as shown in FIG6 , wherein, Represents the attribute residual prediction value of the point cloud at the i-th resolution, Represents the attribute feature prediction value of the point cloud at the i-th resolution.

[0077] In an embodiment of the present application, there is no limitation on the voxel upsampling method used to obtain the attribute residual prediction value of the point cloud of the i-th resolution. It can be based on a preset voxel upsampling rule (for example, the voxel upsampling rule shown in Figure 3). In other embodiments, the first attribute residual reconstruction value of the point cloud of the i-1-th resolution can also be input into the trained fourth neural network of the i-th resolution to obtain the attribute residual prediction value of the point cloud of the i-th resolution.

[0078] For example, in some embodiments, the encoder inputs the attribute feature reconstruction value of the point cloud at the i-1th resolution into the trained fifth neural network at the i-th resolution to obtain the attribute feature prediction value of the point cloud at the i-th resolution. In the embodiment of the present application, the structure of the fifth neural network is not limited. The structure of the fifth neural network is shown in FIG7 , where: Represents the attribute feature reconstruction value of the point cloud at the i-1th resolution, Represents the attribute feature prediction value of the point cloud at the i-th resolution.

[0079] In step 201, the third attribute residual value of the point cloud of the i-th resolution is dequantized to obtain a second attribute residual reconstruction value of the point cloud of the i-th resolution;

[0080] In step 202, the second attribute residual reconstruction value of the point cloud of the i-th resolution is inversely transformed to obtain the first attribute residual reconstruction value of the point cloud of the i-th resolution;

[0081] In some embodiments, the trained sixth neural network of the i-th resolution is used to perform an inverse transformation on the second attribute residual reconstruction value of the point cloud of the i-th resolution to obtain the first attribute residual reconstruction value of the point cloud of the i-th resolution.

[0082] For example, the structure of the sixth neural network is shown in FIG8 , wherein Latent represents the second attribute residual reconstruction value, Represents the residual reconstruction value of the first attribute of the point cloud at the i-th resolution. It can be understood that the structure of the sixth neural network is symmetrical to that of the first neural network, and is composed of a stack of multiple layers of residual network blocks and convolutional layers.

[0083] In step 203 , the attribute reconstruction value of the point cloud at the i-th resolution is determined according to the attribute prediction value of the point cloud at the i-th resolution and the first attribute residual reconstruction value of the point cloud at the i-th resolution.

[0084] In some embodiments, the encoder may implement step 203 as follows: determining the attribute feature reconstruction value of the point cloud of the i-th resolution based on the sum of the attribute feature prediction value of the point cloud of the i-th resolution and the first attribute residual reconstruction value; determining the attribute reconstruction value of the point cloud of the i-th resolution based on the attribute feature reconstruction value of the point cloud of the i-th resolution.

[0085] Furthermore, in some embodiments, the encoder may determine the attribute feature reconstruction value of the point cloud of the i-th resolution as follows: input the sum of the attribute feature prediction value of the point cloud of the i-th resolution and the first attribute residual reconstruction value into the trained seventh neural network of the i-th resolution to obtain the attribute feature reconstruction value of the point cloud of the i-th resolution.

[0086] For example, in some embodiments, the structure of the seventh neural network is shown in FIG9 , wherein: represents the first attribute residual reconstruction value, Represents the attribute feature prediction value of the point cloud at the i-th resolution, Represents the attribute feature reconstruction value of the point cloud at the i-th resolution.

[0087] Furthermore, in some embodiments, the encoder may determine the attribute reconstruction value of the point cloud of the i-th resolution as follows: input the attribute feature reconstruction value of the point cloud of the i-th resolution into the trained eighth neural network of the i-th resolution to obtain the third attribute residual reconstruction value of the point cloud of the i-th resolution; determine the attribute reconstruction value of the point cloud of the i-th resolution based on the third attribute residual reconstruction value of the point cloud of the i-th resolution and the attribute prediction value of the point cloud of the i-th resolution.

[0088] Exemplarily, the structure of the eighth neural network is shown in FIG10 , wherein: Represents the attribute feature reconstruction value of the point cloud at the i-th resolution, Represents the third attribute residual reconstruction value of the point cloud at the i-th resolution.

[0089] Exemplarily, the attribute reconstruction value of the point cloud at the i-th resolution is equal to the sum of the third attribute residual reconstruction value and the attribute prediction value of the point cloud at the i-th resolution.

[0090] It should be noted that, in the embodiment of the present application, the network parameters of the trained first neural network to the eighth neural network are all obtained based on rate-distortion cost training.

[0091] Specifically, in some embodiments, the training process of a neural network at any resolution except the minimum resolution among the multiple different resolutions includes: iteratively training the neural network at any resolution using multiple second point clouds at the any resolution to obtain a trained neural network at the any resolution; wherein the neural network at any resolution refers to a neural network used to implement encoding and decoding of the first point cloud at the any resolution; and the second point cloud is a sample point cloud different from the first point cloud;

[0092] Among them, each iterative process of the iterative training includes: executing the first encoding process on the attribute sampling values ​​of the second point cloud of any resolution to obtain the attribute reconstruction value and the third attribute residual value of the second point cloud of any resolution; determining the rate-distortion cost of the second point cloud of any resolution based on the attribute sampling values ​​of the second point cloud of any resolution, the attribute reconstruction value and the third attribute residual value of the second point cloud of any resolution obtained in the current iteration; using the rate-distortion cost as the loss function, updating the network parameters of the neural network of any resolution obtained in the current iteration.

[0093] Furthermore, in some embodiments, the rate-distortion cost of the second point cloud of any resolution is determined based on the attribute sampling values ​​of the second point cloud of any resolution, the attribute reconstruction values ​​of the second point cloud of any resolution obtained in the current iteration, and the third attribute residual value, including: determining the reconstruction error of the attribute reconstruction values ​​of the second point cloud of any resolution based on the attribute sampling values ​​of the second point cloud of any resolution and the attribute reconstruction values ​​of the second point cloud of any resolution obtained in the current iteration; determining the bit rate based on the third attribute residual value of the second point cloud of any resolution obtained in the current iteration; and determining the rate-distortion cost of the second point cloud of any resolution based on the reconstruction error and the bit rate.

[0094] The present application further provides a decoding method, which is applied to a decoder. FIG11 is a schematic diagram of an implementation flow of the decoding method provided in the present application embodiment. As shown in FIG11 , the decoding method includes the following steps 111 to 112:

[0095] Step 111: parse the attribute code stream to determine the second attribute residual reconstruction value of the point cloud at the i-th resolution;

[0096] In some embodiments, the decoding method further includes: parsing the attribute code stream to obtain the attribute reconstruction value of the point cloud with the i-1th resolution; wherein the point cloud with the i-1th resolution is the point cloud with the smallest resolution among the multiple point clouds with different resolutions.

[0097] In some embodiments, step 111 can be implemented as follows: parsing the attribute code stream to obtain the third attribute residual value of the point cloud of the i-th resolution; dequantizing the third attribute residual value of the point cloud of the i-th resolution to obtain the second attribute residual reconstruction value of the point cloud of the i-th resolution.

[0098] Step 112: performing a first decoding process on the second attribute residual reconstruction value of the point cloud at the i-th resolution according to the attribute reconstruction value of the point cloud at the i-th resolution to obtain the attribute reconstruction value of the point cloud at the i-th resolution;

[0099] The first decoding process includes the following steps 1121 to 1123:

[0100] Step 1121: using the trained sixth neural network of the i-th resolution, perform an inverse transformation on the second attribute residual reconstruction value of the point cloud of the i-th resolution to obtain the first attribute residual reconstruction value of the point cloud of the i-th resolution;

[0101] In some embodiments, the trained sixth neural network of the i-th resolution is a sixth neural network trained using rate-distortion cost as a loss function.

[0102] Step 1122: Perform voxel upsampling on the attribute reconstruction values ​​of the point cloud at the (i-1)th resolution to obtain attribute prediction values ​​of the point cloud at the (i)th resolution; wherein the point cloud at the (i-1)th resolution is obtained by performing one voxel downsampling on the point cloud at the (i)th resolution;

[0103] In some embodiments, a second neural network with a trained i-th resolution may be used to perform voxel upsampling on the attribute reconstruction values ​​of the point cloud with the i-1-th resolution to obtain attribute prediction values ​​of the point cloud with the i-th resolution.

[0104] Step 1123 : Determine the attribute reconstruction value of the point cloud at the i-th resolution according to the attribute prediction value of the point cloud at the i-th resolution and the first attribute reconstruction residual value of the point cloud at the i-th resolution.

[0105] In some embodiments, step 1123 can be implemented as follows: the attribute feature reconstruction value of the point cloud with the i-1th resolution is voxel-up sampled to obtain the attribute feature prediction value of the point cloud with the i-th resolution; the attribute feature reconstruction value of the point cloud with the i-th resolution is determined based on the sum of the attribute feature prediction value of the point cloud with the i-th resolution and the first attribute residual reconstruction value; the attribute reconstruction value of the point cloud with the i-th resolution is determined based on the attribute feature reconstruction value of the point cloud with the i-th resolution.

[0106] Furthermore, in some embodiments, determining the attribute feature reconstruction value of the point cloud with the i-th resolution based on the sum of the attribute feature prediction value of the point cloud with the i-th resolution and the first attribute residual reconstruction value includes: inputting the sum of the attribute feature prediction value of the point cloud with the i-th resolution and the attribute residual reconstruction value into the trained seventh neural network of the i-th resolution to obtain the attribute feature reconstruction value of the point cloud with the i-th resolution.

[0107] Furthermore, in some embodiments, the attribute reconstruction value of the point cloud of the i-th resolution is determined based on the attribute feature reconstruction value of the point cloud of the i-th resolution, including: inputting the attribute feature reconstruction value of the point cloud of the i-th resolution into the trained eighth neural network of the i-th resolution to obtain the third attribute residual reconstruction value of the point cloud of the i-th resolution; and determining the attribute reconstruction value of the point cloud of the i-th resolution based on the third attribute residual reconstruction value of the point cloud of the i-th resolution and the attribute prediction value of the point cloud of the i-th resolution.

[0108] The following describes an exemplary application of the embodiments of the present application in a practical application scenario.

[0109] In an embodiment of the present application, a neural network-based method for lossy compression of layered attributes of point clouds is provided. This method uses voxel downsampling to decompose the point cloud into multi-layer representations and utilizes a neural network for inter-layer prediction, transformation, and entropy coding, thereby achieving efficient lossy attribute compression. This embodiment of the present application can achieve high compression performance and high practicality.

[0110] It is understandable that the original point cloud consists of a large number of disordered points and lacks organization and structure. Through layered operations, its spatial structure can be efficiently characterized and further compressed by leveraging inter-layer correlation. In an embodiment of the present application, a layered attribute lossy compression algorithm based on a neural network is provided. This method decomposes the original point cloud into multiple layers for layer-by-layer encoding and decoding. Combined with a neural network, it can fully utilize spatial correlation for inter-layer residual coding and entropy coding. Experimental results show that this method has good compression performance.

[0111] The method comprises the following steps S1 and S2:

[0112] S1 decomposes the original point cloud into multiple layers of components through scale-by-scale downsampling and pooling.

[0113] S2, starting from the lowest layer, upsamples layer by layer and encodes the residual of each layer and the current layer. The encoding process of each layer can be divided into the following four steps S2.1 to S2.4:

[0114] S2.1, Pre-processing and input data: The reconstruction result of the previous layer is sampled to the current layer through the pre-processing network. The residual is calculated with the true value of the current layer as the input data.

[0115] S2.2, encoding and decoding transformation: The input residual is transformed into a hidden layer representation through the encoding network, the hidden layer representation is quantized and entropy encoded, and then the quantized and encoded hidden layer representation is decoded through the decoding transformation to obtain the decoding information.

[0116] S2.3, Entropy model: For the quantized hidden layer representation in the previous step, use the entropy model network to estimate its probability distribution. The entropy model takes the reconstructed information of the previous layer after pre-processing as input and outputs the probability distribution of the hidden layer representation.

[0117] S2.4, post-processing and output data: The decoded signal of this layer and the reconstructed signal of the previous layer are fused and added through the post-processing network to form the final reconstruction result of this layer. This is then used to assist in the compression of the next layer.

[0118] The implementation method of the above solution will be further described in detail below with reference to the accompanying drawings.

[0119] The overall structure of the layered attribute compression of this embodiment is shown in FIG12A , which can be roughly divided into a layer-by-layer downsampling and a layer-by-layer upsampling process:

[0120] S1, layer-by-layer downsampling: voxel downsampling can be used for layering. As shown in Figure 2, the first voxelized point cloud X gt Downsampling is done at a ratio of 2×2×2, that is, all voxels in each 2×2×2 space are merged into one voxel, and its attribute is the average value of all voxels in the 2×2×2 space. Sampling layer by layer according to this rule, we get multi-layer data. Until all voxels are merged into one voxel in, Represents the attribute sampling value of the point cloud at the i-th resolution, Represents the attribute sampling value of the point cloud at the i+1th resolution, Indicates the attribute sampling value of the point cloud at the i-1th resolution. However, it can be stopped midway because the amount of voxel data decreases exponentially as the sampling progresses. For example, if the number of points is downsampled once, it is about 1 / 4 of the original number, and if it is downsampled twice, it is 1 / 4 of the original number. 2,..., after sampling 5 times, the number of points is only about 1 / 1000 of the original, and there is no need for compression, so sampling can be stopped and the remaining data can be directly stored.

[0121] S2, layer-by-layer upsampling: After the original point cloud is downsampled layer by layer, the attribute data of the lowest layer is very small and is directly losslessly encoded and stored. The attributes of the other layers are then compressed during the layer-by-layer upsampling process. The compression process for each layer is the same: the previous layer is used to predict the current layer and calculate the residual. This residual is then lossily encoded, and the residual and the predicted value are added together to obtain the reconstructed value of the current layer.

[0122] Next, the detailed compression process of each layer is described by taking a certain middle layer i as an example, as shown in FIG12B .

[0123] S2.1: Preprocessing and input data: For the current layer i to be compressed, the reconstruction information from the previous layer i-1 is known, including the reconstruction attributes of the previous layer (i.e., the attribute reconstruction value of the point cloud at the i-1th resolution), features (i.e., the attribute feature reconstruction value of the point cloud at the i-1th resolution) and the residual decoding value of the previous layer (i.e., the first attribute residual reconstruction value of the point cloud at the i-1th resolution). When i=1, since the previous layer is directly losslessly encoded, its reconstruction value is For the known information of the previous layer, it is necessary to transfer it to the current layer through upsampling operation to serve as the initial prediction value of the current layer. and Upsampling is performed by a simple unpooling operation, that is, the value of each voxel in the current layer (i.e., the attribute prediction value of the point cloud at the i-th resolution) and (i.e., the attribute residual prediction value of the point cloud at the i-th resolution) is directly equal to the value of the corresponding parent voxel in the previous layer. (i.e., the attribute feature reconstruction value of the point cloud at the i-1th resolution) is upsampled to the current layer through a transposed convolutional network TconvNet. Its network structure is shown in Figure 7, which mainly includes a transposed convolution with a step size of 2×2×2 and multiple residual network blocks. Then the original attribute (i.e., the attribute sampling value of the point cloud at the i-th resolution) and the predicted attribute Subtract to obtain the attribute residual information to be encoded (i.e., the first attribute residual value of the point cloud at the i-th resolution).

[0124] S2.2, encoding and decoding transformation: In order to Efficient encoding is performed and transformed into latent variables Latent (i.e., the second attribute residual reconstruction value) through an encoding network Encoder (i.e., the first neural network). The encoding network consists of a multi-layer residual network block and a convolutional layer stack. The network structure is shown in Figure 4A. The quantized latent variable Qlatent (the third attribute residual value of the point cloud at the i-th resolution) is transformed into feature vectors through a decoding network Decoder (i.e., the sixth neural network). The decoder network and encoder network are symmetrical in structure, consisting of several layers of residual network blocks and convolutional layers.

[0125] S2.3, Entropy Model: The latent variables are quantized and then losslessly entropy coded. An entropy model (i.e., the third neural network) is used to estimate its probability distribution. The entropy model consists of a multi-layer convolutional network, as shown in Figure 5. Its input is obtained in the preprocessing stage. (i.e., the predicted attribute features of the point cloud at the i-th resolution) and (i.e., the attribute residual prediction value of the point cloud at the i-th resolution), the output is the probability distribution parameter {μ, σ} of each value of the latent variable, from which the probability value is calculated for entropy coding. The probability calculation formula is: where x t Represents each element in the quantized latent variable Qlatent, Indicates that the mean and variance are μ t , σ t Laplace distribution. express arrive The average distribution of . Through the integration operation of the two distributions, we can obtain its probability p(x t ).

[0126] S2.4, Post-processing and output data: For decoded data (i.e., the first attribute residual reconstruction value of the point cloud at the i-th resolution), and Add and process through the linear network LinearNet1 (the seventh neural network) to obtain the output features of the current layer (i.e., the attribute feature reconstruction value of the point cloud at the i-th resolution). After being processed by the linear network LinearNet2 (the eighth neural network), the reconstructed attribute residual is output (i.e. the third attribute residual reconstruction value of the point cloud at the i-th resolution), and compare it with the input attribute Add to get the output attributes of the current layer The network structures of LinearNet1 and LinearNet2 are shown in Figures 9 and 10, where Linear128*128 means that the input and output channels are 128 and 128 respectively.

[0127] In an embodiment of the present application, a neural network and a hierarchical coding framework are combined, and compression tools such as prediction, transformation and / or entropy coding are composed of a neural network, which can obtain optimal network parameters through end-to-end training to achieve better compression performance.

[0128] In the embodiments of the present application, the neural network parameters are not limited to the examples in the specification. Increasing or decreasing the number of convolutional layers, adjusting the convolution kernel size, the number of feature channels and other parameters, and using other convolutional network models are also within the scope of protection of the embodiments of the present application.

[0129] The encoding method provided by the embodiment of the present application has significant gains compared to traditional point cloud attribute lossy compression algorithms. As shown in Figure 13, compared with the RAHT algorithm in MPEG G-PCC, the encoding method provided by the embodiment of the present application can save about 25% of the bit rate.

[0130] It should be noted that in the embodiments of the present application, for the same operations used in the decoding method and the encoding method, for technical details not disclosed in the decoding method embodiments, please refer to the description of the encoding method embodiments of the present application for understanding.

[0131] It should be noted that although the steps of the method of the present application are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all steps must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps; or steps in different embodiments may be combined to form a new technical solution.

[0132] Based on the above embodiments, an embodiment of the present application provides an encoding device, which is applied to an encoder. FIG14 is a schematic structural diagram of the encoding device provided in an embodiment of the present application. As shown in FIG14 , the encoding device 14 includes:

[0133] The downsampling module 141 is configured to perform resolution-by-resolution downsampling on the first point cloud to obtain attribute sampling values ​​of a plurality of point clouds of different resolutions; wherein the plurality of point clouds of different resolutions include at least a point cloud of an i-1th resolution and a point cloud of an i-th resolution, and the point cloud of the i-1th resolution is obtained by performing a single voxel downsampling on the point cloud of the i-th resolution;

[0134] The encoding module 142 is configured to perform a first encoding process on the attribute sampling values ​​of the unencoded point cloud with the i-th resolution based on the attribute reconstruction values ​​of the encoded point cloud with the i-1th resolution to obtain the attribute code stream of the point cloud with the i-th resolution; wherein the first encoding process includes: determining the first attribute residual value of the point cloud with the i-th resolution based on the attribute reconstruction value of the point cloud with the i-1th resolution and the attribute sampling value of the point cloud with the i-th resolution; using the trained first neural network of the i-th resolution to transform the first attribute residual value of the point cloud with the i-th resolution to obtain the second attribute residual value of the point cloud with the i-th resolution; and generating the attribute code stream based on the second attribute residual value of the point cloud with the i-th resolution.

[0135] In some embodiments, when the point cloud with the i-1th resolution is the point cloud with the smallest resolution among the multiple point clouds with different resolutions, the attribute reconstruction value of the point cloud with the i-1th resolution is equal to the attribute sampling value of the point cloud with the i-1th resolution.

[0136] In some embodiments, the encoding module 142 is further configured to generate an attribute code stream according to the attribute reconstruction value of the point cloud at the (i-1)th resolution.

[0137] In some embodiments, the trained first neural network at the i-th resolution is a first neural network trained using rate-distortion cost as a loss function.

[0138] In some embodiments, determining the first attribute residual value of the point cloud with the i-th resolution based on the attribute reconstruction value of the point cloud with the i-1th resolution and the attribute sampling value of the point cloud with the i-th resolution includes: performing voxel upsampling on the attribute reconstruction value of the point cloud with the i-1th resolution to obtain the attribute prediction value of the point cloud with the i-th resolution; determining the first attribute residual value of the point cloud with the i-th resolution based on the attribute prediction value of the point cloud with the i-th resolution and the attribute sampling value of the point cloud with the i-th resolution.

[0139] In some embodiments, the voxel upsampling of the attribute reconstruction values ​​of the point cloud with the i-1th resolution to obtain the attribute prediction values ​​of the point cloud with the i-th resolution includes: using a trained second neural network with the i-th resolution to perform voxel upsampling of the attribute reconstruction values ​​of the point cloud with the i-1th resolution to obtain the attribute prediction values ​​of the point cloud with the i-th resolution.

[0140] In some embodiments, generating an attribute code stream based on the second attribute residual value of the point cloud of the i-th resolution includes: quantizing the second attribute residual value of the point cloud of the i-th resolution to obtain a third attribute residual value of the point cloud of the i-th resolution; and entropy encoding the third attribute residual value of the point cloud of the i-th resolution based on a trained third neural network of the i-th resolution to obtain an attribute code stream of the point cloud of the i-th resolution.

[0141] In some embodiments, the entropy encoding of the third attribute residual value of the point cloud of the i-th resolution is performed based on the trained third neural network of the i-th resolution to obtain the attribute code stream of the point cloud of the i-th resolution, including: inputting at least the attribute prediction value of the point cloud of the i-th resolution into the trained third neural network of the i-th resolution to obtain the probability distribution parameters of the third attribute residual value of the point cloud of the i-th resolution; and entropy encoding of the third attribute residual value of the i-th resolution according to the probability distribution parameters to obtain the attribute code stream of the point cloud of the i-th resolution.

[0142] In some embodiments, at least the attribute prediction value of the point cloud of the i-th resolution is input into the trained third neural network of the i-th resolution to obtain the probability distribution parameters of the third attribute residual value of the point cloud of the i-th resolution, including: performing voxel upsampling on the first attribute residual reconstruction value of the point cloud of the i-1-th resolution to obtain the attribute residual prediction value of the point cloud of the i-th resolution; performing voxel upsampling on the attribute feature reconstruction value of the point cloud of the i-1-th resolution to obtain the attribute feature prediction value of the point cloud of the i-th resolution; inputting the attribute prediction value, attribute residual prediction value and attribute feature prediction value of the point cloud of the i-th resolution into the trained third neural network of the i-th resolution to obtain the probability distribution parameters of the third attribute residual value of the point cloud of the i-th resolution.

[0143] In some embodiments, the voxel upsampling of the first attribute residual reconstruction value of the point cloud with the i-1th resolution to obtain the attribute residual prediction value of the point cloud with the i-th resolution includes: inputting the first attribute residual reconstruction value of the point cloud with the i-1th resolution into the trained fourth neural network of the i-th resolution to obtain the attribute residual prediction value of the point cloud with the i-th resolution.

[0144] In some embodiments, the voxel up-sampling of the attribute feature reconstruction values ​​of the point cloud with the i-1th resolution to obtain the attribute feature prediction values ​​of the point cloud with the i-th resolution includes: inputting the attribute feature reconstruction values ​​of the point cloud with the i-1th resolution into the trained fifth neural network of the i-th resolution to obtain the attribute feature prediction values ​​of the point cloud with the i-th resolution.

[0145] In some embodiments, when the point cloud at the i-th resolution is not the first point cloud, the first encoding process further includes: dequantizing the third attribute residual value of the point cloud at the i-th resolution to obtain the second attribute residual reconstruction value of the point cloud at the i-th resolution; inversely transforming the second attribute residual reconstruction value of the point cloud at the i-th resolution to obtain the first attribute residual reconstruction value of the point cloud at the i-th resolution; and determining the attribute reconstruction value of the point cloud at the i-th resolution based on the attribute prediction value of the point cloud at the i-th resolution and the first attribute residual reconstruction value of the point cloud at the i-th resolution.

[0146] In some embodiments, the inverse transformation of the second attribute residual reconstruction value of the point cloud of the i-th resolution to obtain the first attribute residual reconstruction value of the point cloud of the i-th resolution includes: using the trained sixth neural network of the i-th resolution to inverse transform the second attribute residual reconstruction value of the point cloud of the i-th resolution to obtain the first attribute residual reconstruction value of the point cloud of the i-th resolution.

[0147] In some embodiments, the attribute reconstruction value of the point cloud with the i-th resolution is determined based on the attribute prediction value of the point cloud with the i-th resolution and the first attribute residual reconstruction value of the point cloud with the i-th resolution, including: determining the attribute feature reconstruction value of the point cloud with the i-th resolution based on the sum of the attribute feature prediction value of the point cloud with the i-th resolution and the first attribute residual reconstruction value; determining the attribute reconstruction value of the point cloud with the i-th resolution based on the attribute feature reconstruction value of the point cloud with the i-th resolution.

[0148] In some embodiments, determining the attribute feature reconstruction value of the point cloud with the i-th resolution based on the sum of the attribute feature prediction value of the point cloud with the i-th resolution and the first attribute residual reconstruction value includes: inputting the sum of the attribute feature prediction value of the point cloud with the i-th resolution and the first attribute residual reconstruction value into the trained seventh neural network of the i-th resolution to obtain the attribute feature reconstruction value of the point cloud with the i-th resolution.

[0149] In some embodiments, determining the attribute reconstruction value of the point cloud with the i-th resolution based on the attribute feature reconstruction value of the point cloud with the i-th resolution includes: inputting the attribute feature reconstruction value of the point cloud with the i-th resolution into the trained eighth neural network of the i-th resolution to obtain the third attribute residual reconstruction value of the point cloud with the i-th resolution; determining the attribute reconstruction value of the point cloud with the i-th resolution based on the third attribute residual reconstruction value of the point cloud with the i-th resolution and the attribute prediction value of the point cloud with the i-th resolution.

[0150] In some embodiments, the encoding module 142 is further configured to: perform the first encoding process according to the attribute reconstruction value of the point cloud of the i-th resolution and the attribute sampling value of the point cloud of the i+1-th resolution to obtain the attribute code stream of the point cloud of the i+1-th resolution and the attribute reconstruction value of the point cloud of the i+1-th resolution; iteratively perform the first encoding process until the attribute code stream of the first point cloud is obtained; wherein, the point cloud of the i-th resolution is obtained by performing a voxel downsampling on the point cloud of the i+1-th resolution; the point cloud of the i-th resolution is a point cloud of any intermediate resolution among the multiple point clouds of different resolutions.

[0151] In some embodiments, the downsampling module 141 is configured to: downsample the first point cloud resolution by resolution until a downsampling times threshold is reached, or until the number of voxels of the point cloud obtained by the current downsampling is less than or equal to a voxel number threshold, to obtain attribute sampling values ​​of multiple point clouds of different resolutions; wherein the downsampling times threshold is greater than 1, and the voxel number threshold is greater than or equal to 1.

[0152] In some embodiments, resolution-by-resolution downsampling includes: performing voxel downsampling on the point cloud of the i+1th resolution according to a first sampling ratio to obtain the point cloud of the i-th resolution; performing voxel downsampling on the point cloud of the i-th resolution according to a second sampling ratio to obtain the point cloud of the i-1th resolution; wherein the first sampling ratio is the same as or different from the second sampling ratio.

[0153] In some embodiments, the training process of the neural network of any resolution except the minimum resolution among the multiple different resolutions includes: using multiple second point clouds of any resolution to iteratively train the neural network of any resolution to obtain the trained neural network of any resolution; wherein the neural network of any resolution refers to the neural network used to implement the encoding and decoding of the first point cloud of any resolution; wherein each iteration process of the iterative training includes: performing the first encoding process on the attribute sampling value of the second point cloud of any resolution to obtain the attribute reconstruction value and the third attribute residual value of the second point cloud of any resolution; determining the rate-distortion cost of the second point cloud of any resolution based on the attribute sampling value of the second point cloud of any resolution, the attribute reconstruction value and the third attribute residual value of the second point cloud of any resolution obtained in the current iteration; and updating the network parameters of the neural network of any resolution obtained in the current iteration using the rate-distortion cost as the loss function.

[0154] In some embodiments, the rate-distortion cost of the second point cloud of any resolution is determined based on the attribute sampling values ​​of the second point cloud of any resolution, the attribute reconstruction values ​​of the second point cloud of any resolution obtained in the current iteration, and the third attribute residual value, including: determining the reconstruction error of the attribute reconstruction values ​​of the second point cloud of any resolution based on the attribute sampling values ​​of the second point cloud of any resolution and the attribute reconstruction values ​​of the second point cloud of any resolution obtained in the current iteration; determining the bit rate based on the third attribute residual value of the second point cloud of any resolution obtained in the current iteration; determining the rate-distortion cost of the second point cloud of any resolution based on the reconstruction error and the bit rate.

[0155] The description of the above encoding device embodiment is similar to the description of the above encoding method embodiment, and has similar beneficial effects as the encoding method embodiment. For technical details not disclosed in the encoding device embodiment of this application, please refer to the description of the encoding method embodiment of this application for understanding.

[0156] The present application embodiment provides a decoding device, which is applied to a decoder. FIG15 is a schematic structural diagram of the decoding device provided in the present application embodiment. As shown in FIG15 , the decoding device 15 includes:

[0157] The parsing module 151 is configured to parse the attribute code stream and determine a second attribute residual reconstruction value of the point cloud at the i-th resolution;

[0158] The reconstruction module 152 is configured to perform a first decoding process on the second attribute residual reconstruction value of the point cloud at the i-th resolution according to the attribute reconstruction value of the point cloud at the i-th resolution, to obtain the attribute reconstruction value of the point cloud at the i-th resolution; wherein,

[0159] The first decoding process includes:

[0160] Using the trained sixth neural network of the i-th resolution, inversely transform the second attribute residual reconstruction value of the point cloud of the i-th resolution to obtain the first attribute residual reconstruction value of the point cloud of the i-th resolution;

[0161] Performing voxel upsampling on the attribute reconstruction value of the point cloud with the i-1th resolution to obtain the attribute prediction value of the point cloud with the i-th resolution; wherein the point cloud with the i-1th resolution is obtained by performing one voxel downsampling on the point cloud with the i-th resolution;

[0162] An attribute reconstruction value of the point cloud at the i-th resolution is determined according to the attribute prediction value of the point cloud at the i-th resolution and the first attribute reconstruction residual value of the point cloud at the i-th resolution.

[0163] Parse the attribute code stream to obtain attribute reconstruction values ​​of the point cloud with the i-1th resolution; wherein the point cloud with the i-1th resolution is a point cloud with the minimum resolution among the multiple point clouds with different resolutions.

[0164] In some embodiments, the trained sixth neural network of the i-th resolution is a sixth neural network trained using rate-distortion cost as a loss function.

[0165] In some embodiments, the parsing module 151 is configured to parse the attribute code stream to obtain the third attribute residual value of the point cloud of the i-th resolution; and dequantize the third attribute residual value of the point cloud of the i-th resolution to obtain the second attribute residual reconstruction value of the point cloud of the i-th resolution.

[0166] In some embodiments, the voxel upsampling of the attribute reconstruction values ​​of the point cloud with the i-1th resolution to obtain the attribute prediction values ​​of the point cloud with the i-th resolution includes: using a trained second neural network with the i-th resolution to perform voxel upsampling of the attribute reconstruction values ​​of the point cloud with the i-1th resolution to obtain the attribute prediction values ​​of the point cloud with the i-th resolution.

[0167] In some embodiments, the attribute reconstruction value of the point cloud with the i-th resolution is determined based on the attribute prediction value of the point cloud with the i-th resolution and the first attribute reconstruction residual value of the point cloud with the i-th resolution, including: voxel up-sampling the attribute feature reconstruction value of the point cloud with the i-1-th resolution to obtain the attribute feature prediction value of the point cloud with the i-th resolution; determining the attribute feature reconstruction value of the point cloud with the i-th resolution based on the sum of the attribute feature prediction value of the point cloud with the i-th resolution and the first attribute residual reconstruction value; determining the attribute reconstruction value of the point cloud with the i-th resolution based on the attribute feature reconstruction value of the point cloud with the i-th resolution.

[0168] In some embodiments, determining the attribute feature reconstruction value of the point cloud of the i-th resolution based on the sum of the attribute feature prediction value of the point cloud of the i-th resolution and the first attribute residual reconstruction value includes: inputting the sum of the attribute feature prediction value of the point cloud of the i-th resolution and the attribute residual reconstruction value into the trained seventh neural network of the i-th resolution to obtain the attribute feature reconstruction value of the point cloud of the i-th resolution.

[0169] In some embodiments, determining the attribute reconstruction value of the point cloud with the i-th resolution based on the attribute feature reconstruction value of the point cloud with the i-th resolution includes: inputting the attribute feature reconstruction value of the point cloud with the i-th resolution into the trained eighth neural network of the i-th resolution to obtain the third attribute residual reconstruction value of the point cloud with the i-th resolution; determining the attribute reconstruction value of the point cloud with the i-th resolution based on the third attribute residual reconstruction value of the point cloud with the i-th resolution and the attribute prediction value of the point cloud with the i-th resolution.

[0170] The description of the above decoding device embodiment is similar to the description of the above encoding / decoding method embodiment, and has similar beneficial effects as the encoding / decoding method embodiment. For technical details not disclosed in the decoding device embodiment of this application, please refer to the description of the encoding / decoding method embodiment of this application for understanding.

[0171] It should be noted that the division of modules in the encoding / decoding device described in the embodiments of the present application is schematic and is merely a logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units. They may also be implemented in a combination of software and hardware.

[0172] It should be noted that, in the embodiment of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling an electronic device to execute all or part of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0173] An embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed, the encoding method or the decoding method as described in the embodiment of the present application is implemented.

[0174] An embodiment of the present application provides an encoder, as shown in FIG16 , the encoder 16 includes: a first communication interface 161, a first memory 162, and a first processor 163; each component is coupled together via a first bus system 164. It is understood that the first bus system 164 is used to achieve connection and communication between these components. In addition to the data bus, the first bus system 164 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, in FIG16 , various buses are labeled as the first bus system 164. Among them,

[0175] The first communication interface 161 is used for receiving and sending signals during the process of sending and receiving information between other external network elements;

[0176] A first memory 162 is used to store computer programs that can be run on the first processor 163;

[0177] The first processor 163 is configured to execute the encoding method described in the embodiment of the present application when running the computer program.

[0178] It is understood that the first memory 162 in the embodiment of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DRRAM). The first memory 162 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0179] The first processor 163 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits or software instructions in the first processor 163. The above-mentioned first processor 163 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the first memory 162 , and the first processor 163 reads the information in the first memory 162 and completes the steps of the above method in combination with its hardware.

[0180] It is to be understood that these embodiments described in the present application can be implemented with hardware, software, firmware, middleware, microcode or its combination.For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (Application Specific Integrated Circuits, ASIC), digital signal processor (Digital Signal Processing, DSP), digital signal processing equipment (DSP Device, DSPD), programmable logic device (Programmable Logic Device, PLD), field programmable gate array (Field-Programmable Gate Array, FPGA), general-purpose processor, controller, microcontroller, microprocessor, other electronic units for performing functions described in the present application or its combination.For software implementation, the technology described in the present application can be realized by the module (such as process, function etc.) that performs functions described in the present application. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0181] Optionally, as another embodiment, the first processor 163 is further configured to execute any of the aforementioned encoding method embodiments when running the computer program.

[0182] The present application provides a decoder, as shown in FIG17 , the decoder 17 includes: a second communication interface 171, a second memory 172, and a second processor 173; each component is coupled together via a second bus system 174. It is understood that the second bus system 174 is used to achieve connection and communication between these components. In addition to the data bus, the second bus system 174 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, in FIG17 , various buses are labeled as the second bus system 174. Among them,

[0183] The second communication interface 171 is used for receiving and sending signals during the process of sending and receiving information between other external network elements;

[0184] The second memory 172 is used to store computer programs that can be run on the second processor 173;

[0185] The second processor 173 is configured to execute the decoding method described in the embodiment of the present application when running the computer program.

[0186] It can be understood that the hardware functions of the second memory 172 and the first memory 162 are similar, and the hardware functions of the second processor 173 and the first processor 163 are similar; they will not be described in detail here.

[0187] The embodiment of the present application further provides a code stream, which is obtained by using the aforementioned encoding method.

[0188] An embodiment of the present application provides an electronic device, comprising: a processor adapted to execute a computer program; and a computer-readable storage medium storing the computer program, wherein when the computer program is executed by the processor, the encoding method and / or decoding method described in the embodiment of the present application are implemented. The electronic device can be any type of device capable of video encoding and / or video decoding, such as a mobile phone, tablet computer, laptop computer, personal computer, television, projection device, or monitoring device.

[0189] It should be noted that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium, storage medium, and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0190] It should be understood that "one embodiment" or "an embodiment" or "some embodiments" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" or "in some embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments. The above description of the various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other. For the sake of brevity, they will not be repeated here.

[0191] The term "and / or" in this article is only a description of the association relationship between associated objects, indicating that there can be three relationships. For example, object A and / or object B can mean: object A exists alone, object A and object B exist at the same time, and object B exists alone.

[0192] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0193] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The embodiments described above are merely illustrative. For example, the division of the modules is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or modules can be electrical, mechanical or other forms.

[0194] The modules described above as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules; they may be located in one place or distributed across multiple network units; some or all of the modules may be selected according to actual needs to achieve the purpose of this embodiment.

[0195] In addition, all functional modules in the embodiments of the present application can be integrated into one processing unit, or each module can be a separate unit, or two or more modules can be integrated into one unit; the above-mentioned integrated modules can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0196] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, and other media that can store program codes.

[0197] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling an electronic device to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks or optical disks.

[0198] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined, if they do not conflict, to obtain new method embodiments. The features disclosed in the several product embodiments provided in this application can be arbitrarily combined, if they do not conflict, to obtain new product embodiments. The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined, if they do not conflict, to obtain new method embodiments or device embodiments.

[0199] The above is merely an embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A coding method, the method being applied to an encoder, the method comprising: Downsampling the first point cloud resolution by resolution to obtain attribute sampling values ​​of a plurality of point clouds with different resolutions; wherein the plurality of point clouds with different resolutions include at least a point cloud with an i-1th resolution and a point cloud with an i-th resolution, and the point cloud with the i-1th resolution is obtained by performing a voxel downsampling on the point cloud with the i-th resolution; According to the encoded attribute reconstruction value of the point cloud with the i-1th resolution, a first encoding process is performed on the unencoded attribute sampling value of the point cloud with the i-th resolution to obtain an attribute code stream of the point cloud with the i-th resolution; The first encoding process includes: Determine a first attribute residual value of the point cloud of the i-th resolution according to the attribute reconstruction value of the point cloud of the i-th resolution and the attribute sampling value of the point cloud of the i-th resolution; Using a trained first neural network of the i-th resolution, transforming a first attribute residual value of the point cloud of the i-th resolution to obtain a second attribute residual value of the point cloud of the i-th resolution; An attribute code stream is generated according to the second attribute residual value of the point cloud of the i-th resolution.

2. The method according to claim 1, wherein: When the point cloud with the i-1th resolution is the point cloud with the smallest resolution among the multiple point clouds with different resolutions, the attribute reconstruction value of the point cloud with the i-1th resolution is equal to the attribute sampling value of the point cloud with the i-1th resolution.

3. The method according to claim 2, wherein: The method further includes: generating an attribute code stream according to the attribute reconstruction value of the point cloud at the i-1th resolution.

4. The method according to claim 1, wherein: The trained first neural network with the i-th resolution is a first neural network trained with rate-distortion cost as a loss function.

5. The method according to claim 1, wherein: The determining, according to the attribute reconstruction value of the point cloud at the i-1th resolution and the attribute sampling value of the point cloud at the i-th resolution, the first attribute residual value of the point cloud at the i-th resolution comprises: Performing voxel upsampling on the attribute reconstruction value of the point cloud with the i-1th resolution to obtain the attribute prediction value of the point cloud with the i-th resolution; A first attribute residual value of the point cloud at the i-th resolution is determined according to the attribute prediction value of the point cloud at the i-th resolution and the attribute sampling value of the point cloud at the i-th resolution.

6. The method according to claim 5, wherein: The step of performing voxel upsampling on the attribute reconstruction value of the point cloud of the i-1th resolution to obtain the attribute prediction value of the point cloud of the i-th resolution includes: The attribute reconstruction values ​​of the point cloud of the i-th resolution are upsampled by voxel by using the trained second neural network of the i-th resolution to obtain the attribute prediction values ​​of the point cloud of the i-th resolution.

7. The method according to claim 1, wherein: The generating an attribute code stream according to the second attribute residual value of the point cloud of the i-th resolution includes: quantizing the second attribute residual value of the point cloud at the i-th resolution to obtain a third attribute residual value of the point cloud at the i-th resolution; According to the trained third neural network of the i-th resolution, entropy coding is performed on the third attribute residual value of the point cloud of the i-th resolution to obtain an attribute code stream of the point cloud of the i-th resolution.

8. The method according to claim 7, wherein: The method of performing entropy coding on the third attribute residual value of the point cloud of the i-th resolution according to the trained third neural network of the i-th resolution to obtain the attribute code stream of the point cloud of the i-th resolution includes: Inputting at least the attribute prediction value of the point cloud of the i-th resolution into the trained third neural network of the i-th resolution to obtain the probability distribution parameter of the third attribute residual value of the point cloud of the i-th resolution; According to the probability distribution parameter, entropy coding is performed on the third attribute residual value of the i-th resolution to obtain an attribute code stream of the point cloud of the i-th resolution.

9. The method according to claim 8, wherein: The step of inputting at least the attribute prediction value of the point cloud of the i-th resolution into the trained third neural network of the i-th resolution to obtain the probability distribution parameter of the third attribute residual value of the point cloud of the i-th resolution includes: Performing voxel upsampling on the first attribute residual reconstruction value of the point cloud of the i-1th resolution to obtain the attribute residual prediction value of the point cloud of the i-th resolution; Performing voxel upsampling on the attribute feature reconstruction value of the point cloud with the i-1th resolution to obtain the attribute feature prediction value of the point cloud with the i-th resolution; The attribute prediction value, attribute residual prediction value and attribute feature prediction value of the point cloud of the i-th resolution are input into the trained third neural network of the i-th resolution to obtain the probability distribution parameters of the third attribute residual value of the point cloud of the i-th resolution.

10. The method according to claim 9, wherein: The step of performing voxel upsampling on the first attribute residual reconstruction value of the point cloud of the i-1th resolution to obtain the attribute residual prediction value of the point cloud of the i-th resolution includes: The first attribute residual reconstruction value of the point cloud with the i-1th resolution is input into the trained fourth neural network with the i-th resolution to obtain the attribute residual prediction value of the point cloud with the i-th resolution.

11. The method according to claim 9, wherein: The step of performing voxel upsampling on the attribute feature reconstruction value of the point cloud at the i-1th resolution to obtain the attribute feature prediction value of the point cloud at the i-th resolution includes: The attribute feature reconstruction value of the point cloud with the i-1th resolution is input into the trained fifth neural network with the i-th resolution to obtain the attribute feature prediction value of the point cloud with the i-th resolution.

12. The method according to any one of claims 7 to 11, wherein: In the case where the point cloud at the i-th resolution is not the first point cloud, the first encoding process further includes: Dequantizing the third attribute residual value of the point cloud of the i-th resolution to obtain a second attribute residual reconstruction value of the point cloud of the i-th resolution; Performing an inverse transformation on the second attribute residual reconstruction value of the point cloud of the i-th resolution to obtain the first attribute residual reconstruction value of the point cloud of the i-th resolution; An attribute reconstruction value of the point cloud at the i-th resolution is determined according to the attribute prediction value of the point cloud at the i-th resolution and the first attribute residual reconstruction value of the point cloud at the i-th resolution.

13. The method according to claim 12, wherein: The inverse transformation of the second attribute residual reconstruction value of the point cloud of the i-th resolution to obtain the first attribute residual reconstruction value of the point cloud of the i-th resolution includes: The trained sixth neural network of the i-th resolution is used to perform an inverse transformation on the second attribute residual reconstruction value of the point cloud of the i-th resolution to obtain the first attribute residual reconstruction value of the point cloud of the i-th resolution.

14. The method according to claim 12, wherein: The determining the attribute reconstruction value of the point cloud of the i-th resolution according to the attribute prediction value of the point cloud of the i-th resolution and the first attribute residual reconstruction value of the point cloud of the i-th resolution comprises: Determining an attribute feature reconstruction value of the point cloud at the i-th resolution according to a sum of the attribute feature prediction value of the point cloud at the i-th resolution and the first attribute residual reconstruction value; According to the attribute feature reconstruction value of the point cloud at the i-th resolution, the attribute reconstruction value of the point cloud at the i-th resolution is determined.

15. The method according to claim 14, wherein: The step of determining the attribute feature reconstruction value of the point cloud at the i-th resolution according to the sum of the attribute feature prediction value of the point cloud at the i-th resolution and the first attribute residual reconstruction value comprises: The sum of the attribute feature prediction value of the point cloud of the i-th resolution and the first attribute residual reconstruction value is input into the trained seventh neural network of the i-th resolution to obtain the attribute feature reconstruction value of the point cloud of the i-th resolution.

16. The method according to claim 14, wherein: The step of determining the attribute reconstruction value of the point cloud at the i-th resolution according to the attribute feature reconstruction value of the point cloud at the i-th resolution comprises: Inputting the attribute feature reconstruction value of the point cloud of the i-th resolution into the trained eighth neural network of the i-th resolution to obtain the third attribute residual reconstruction value of the point cloud of the i-th resolution; The attribute reconstruction value of the point cloud with the i-th resolution is determined according to the third attribute residual reconstruction value of the point cloud with the i-th resolution and the attribute prediction value of the point cloud with the i-th resolution.

17. The method according to any one of claims 12 to 16, wherein: The method further comprises: The first encoding process is performed according to the attribute reconstruction value of the point cloud with the i-th resolution and the attribute sampling value of the point cloud with the i+1-th resolution to obtain the attribute code stream of the point cloud with the i+1-th resolution and the attribute reconstruction value of the point cloud with the i+1-th resolution; Iteratively executing the first encoding process until an attribute code stream of the first point cloud is obtained; The point cloud of the i-th resolution is obtained by performing a voxel downsampling on the point cloud of the (i+1)-th resolution; the point cloud of the i-th resolution is a point cloud of any intermediate resolution among the multiple point clouds of different resolutions.

18. The method according to claim 1, wherein: The step of downsampling the first point cloud resolution by resolution to obtain attribute sampling values ​​of point clouds of multiple different resolutions includes: Downsampling the first point cloud resolution by resolution until a downsampling number threshold is reached, or until the number of voxels of the point cloud obtained by the current downsampling is less than or equal to a voxel number threshold, to obtain attribute sampling values ​​of point clouds of multiple different resolutions; The downsampling times threshold is greater than 1, and the voxel number threshold is greater than or equal to 1.

19. The method according to claim 18, wherein: Per-resolution downsampling, including: According to a first sampling ratio, performing a voxel down-sampling on the point cloud of the i+1th resolution to obtain the point cloud of the i-th resolution; According to a second sampling ratio, the point cloud of the i-th resolution is downsampled once by voxels to obtain the point cloud of the i-th resolution; wherein the first sampling ratio is the same as or different from the second sampling ratio.

20. The method according to any one of claims 1 to 19, wherein: The training process of the neural network of any resolution except the minimum resolution among the multiple different resolutions includes: Iteratively training the neural network of any resolution using the plurality of second point clouds of any resolution to obtain a trained neural network of any resolution; wherein the neural network of any resolution refers to a neural network used to implement encoding and decoding of the first point cloud of any resolution; Wherein, each iteration process of the iterative training includes: Performing the first encoding process on the attribute sampling values ​​of the second point cloud of any resolution to obtain the attribute reconstruction value and the third attribute residual value of the second point cloud of any resolution; Determine a rate-distortion cost of the second point cloud of any resolution according to the attribute sampling value of the second point cloud of any resolution, the attribute reconstruction value of the second point cloud of any resolution obtained in the current iteration, and the third attribute residual value; The network parameters of the neural network of any resolution obtained in the current iteration are updated using the rate distortion cost as the loss function.

21. The method according to claim 20, wherein: The determining the rate-distortion cost of the second point cloud of any resolution according to the attribute sampling value of the second point cloud of any resolution, the attribute reconstruction value of the second point cloud of any resolution obtained in the current iteration, and the third attribute residual value comprises: Determining a reconstruction error of the attribute reconstruction value of the second point cloud of any resolution according to the attribute sampling value of the second point cloud of any resolution and the attribute reconstruction value of the second point cloud of any resolution obtained in the current iteration; Determining a bit rate according to a third attribute residual value of the second point cloud of any resolution obtained in a current iteration; A rate-distortion cost of the second point cloud of any resolution is determined according to the reconstruction error and the bit rate.

22. A decoding method, the decoding method being applied to a decoder, the method comprising: Parse the attribute code stream to determine the second attribute residual reconstruction value of the point cloud at the i-th resolution; According to the attribute reconstruction value of the point cloud with the i-1th resolution, a first decoding process is performed on the second attribute residual reconstruction value of the point cloud with the i-th resolution to obtain the attribute reconstruction value of the point cloud with the i-th resolution; The first decoding process includes: Using the trained sixth neural network of the i-th resolution, inversely transform the second attribute residual reconstruction value of the point cloud of the i-th resolution to obtain the first attribute residual reconstruction value of the point cloud of the i-th resolution; Performing voxel upsampling on the attribute reconstruction value of the point cloud with the i-1th resolution to obtain the attribute prediction value of the point cloud with the i-th resolution; wherein the point cloud with the i-1th resolution is obtained by performing voxel downsampling on the point cloud with the i-th resolution once; An attribute reconstruction value of the point cloud at the i-th resolution is determined according to the attribute prediction value of the point cloud at the i-th resolution and the first attribute reconstruction residual value of the point cloud at the i-th resolution.

23. The method according to claim 22, wherein: The method further comprises: Parse the attribute code stream to obtain the attribute reconstruction value of the point cloud with the i-1th resolution; wherein the point cloud with the i-1th resolution is a point cloud with the smallest resolution among the multiple point clouds with different resolutions.

24. The method according to claim 22, wherein: The trained sixth neural network of the i-th resolution is a sixth neural network trained with rate distortion cost as a loss function.

25. The method according to claim 22, wherein: The parsing attribute code stream to determine the second attribute residual reconstruction value of the point cloud at the i-th resolution includes: Parsing the attribute code stream to obtain a third attribute residual value of the point cloud at the i-th resolution; The third attribute residual value of the point cloud of the i-th resolution is inversely quantized to obtain a second attribute residual reconstruction value of the point cloud of the i-th resolution.

26. The method of claim 22, wherein: The step of performing voxel upsampling on the attribute reconstruction value of the point cloud at the i-1th resolution to obtain the attribute prediction value of the point cloud at the i-th resolution includes: The attribute reconstruction values ​​of the point cloud of the i-th resolution are upsampled by voxel by using the trained second neural network of the i-th resolution to obtain the attribute prediction values ​​of the point cloud of the i-th resolution.

27. The method of claim 22, wherein: The determining the attribute reconstruction value of the point cloud of the i-th resolution according to the attribute prediction value of the point cloud of the i-th resolution and the first attribute reconstruction residual value of the point cloud of the i-th resolution comprises: Performing voxel upsampling on the attribute feature reconstruction value of the point cloud with the i-1th resolution to obtain the attribute feature prediction value of the point cloud with the i-th resolution; Determining an attribute feature reconstruction value of the point cloud at the i-th resolution according to a sum of the attribute feature prediction value of the point cloud at the i-th resolution and the first attribute residual reconstruction value; According to the attribute feature reconstruction value of the point cloud at the i-th resolution, the attribute reconstruction value of the point cloud at the i-th resolution is determined.

28. The method according to claim 27, wherein: The step of determining the attribute feature reconstruction value of the point cloud at the i-th resolution according to the sum of the attribute feature prediction value of the point cloud at the i-th resolution and the first attribute residual reconstruction value comprises: The sum of the attribute feature prediction value and the attribute residual reconstruction value of the point cloud of the i-th resolution is input into the trained seventh neural network of the i-th resolution to obtain the attribute feature reconstruction value of the point cloud of the i-th resolution.

29. The method according to claim 27, wherein: The step of determining the attribute reconstruction value of the point cloud at the i-th resolution according to the attribute feature reconstruction value of the point cloud at the i-th resolution comprises: Inputting the attribute feature reconstruction value of the point cloud of the i-th resolution into the trained eighth neural network of the i-th resolution to obtain the third attribute residual reconstruction value of the point cloud of the i-th resolution; The attribute reconstruction value of the point cloud with the i-th resolution is determined according to the third attribute residual reconstruction value of the point cloud with the i-th resolution and the attribute prediction value of the point cloud with the i-th resolution.

30. An encoding device, applied to an encoder, the device comprising: A downsampling module is configured to perform resolution-by-resolution downsampling on the first point cloud to obtain attribute sampling values ​​of a plurality of point clouds with different resolutions; wherein the plurality of point clouds with different resolutions at least include a point cloud with an i-1th resolution and a point cloud with an i-th resolution, and the point cloud with the i-1th resolution is obtained by performing a voxel downsampling on the point cloud with the i-th resolution; The encoding module is configured to perform a first encoding process on the unencoded attribute sampling values ​​of the point cloud of the i-th resolution according to the encoded attribute reconstruction values ​​of the point cloud of the i-th resolution, so as to obtain the attribute code stream of the point cloud of the i-th resolution; wherein, The first encoding process includes: determining a first attribute residual value of the point cloud of the i-th resolution based on the attribute reconstruction value of the point cloud of the i-1th resolution and the attribute sampling value of the point cloud of the i-th resolution; using a trained first neural network of the i-th resolution to transform the first attribute residual value of the point cloud of the i-th resolution to obtain a second attribute residual value of the point cloud of the i-th resolution; and generating an attribute code stream based on the second attribute residual value of the point cloud of the i-th resolution.

31. An encoder comprising a first memory and a first processor; wherein: The first memory is used to store a computer program that can be run on the first processor; The first processor is configured to execute the method according to any one of claims 1 to 21 when running the computer program.

32. An attribute code stream, wherein the attribute code stream is obtained by using the encoding method according to any one of claims 1 to 21.

33. A decoding device, the device being applied to a decoder, the device comprising: A parsing module configured to parse the attribute code stream to determine a second attribute residual reconstruction value of the point cloud at an i-th resolution; A reconstruction module is configured to perform a first decoding process on the second attribute residual reconstruction value of the point cloud of the i-th resolution according to the attribute reconstruction value of the point cloud of the i-th resolution, so as to obtain the attribute reconstruction value of the point cloud of the i-th resolution; wherein, The first decoding process comprises: Using the trained sixth neural network of the i-th resolution, inversely transform the second attribute residual reconstruction value of the point cloud of the i-th resolution to obtain the first attribute residual reconstruction value of the point cloud of the i-th resolution; Performing voxel upsampling on the attribute reconstruction value of the point cloud with the i-1th resolution to obtain the attribute prediction value of the point cloud with the i-th resolution; wherein the point cloud with the i-1th resolution is obtained by performing voxel downsampling on the point cloud with the i-th resolution once; An attribute reconstruction value of the point cloud at the i-th resolution is determined according to the attribute prediction value of the point cloud at the i-th resolution and the first attribute reconstruction residual value of the point cloud at the i-th resolution.

34. A decoder comprising a second memory and a second processor; wherein: The second memory is used to store a computer program that can be run on the second processor; The second processor is configured to execute the method according to any one of claims 22 to 29 when running the computer program.

35. An electronic device comprising: a processor adapted to execute a computer program; A computer-readable storage medium having a computer program stored therein, wherein when the computer program is executed by the processor, the method according to any one of claims 1 to 21 is implemented, or when the computer program is executed by the processor, the method according to any one of claims 22 to 29 is implemented.

36. A computer-readable storage medium, wherein: The computer-readable storage medium stores a computer program, and when the computer program is executed, the method according to any one of claims 1 to 21 is implemented, or when the computer program is executed, the method according to any one of claims 22 to 29 is implemented.

Citation Information

Patent Citations

  • Point cloud attribute lossy compression device and method based on neural network

    CN115239563A

  • Resolution adaptive point cloud geometric lossy coding method and device based on deep residual compression and sparse representation, and medium

    CN115883850A

  • Techniques and apparatus for scalable lifting for point-cloud attribute coding

    US20210006837A1

  • Techniques and apparatus for inter-channel prediction and transform for point-cloud attribute coding

    US20210112276A1