Encoding and decoding method, encoder, decoder and storage medium

Abstract

The embodiment of the invention provides a coding and decoding method, which comprises the following steps of: inputting geometric information of an ith layer of nodes into a first embedded network, and determining embedded characteristics of the ith layer of nodes; inputting the embedding characteristics of the i-th layer of nodes into a coding network, and determining the latent variables of the i-th layer of nodes; determining a reconstruction value of a residual error of the i-th layer node according to the latent variable of the i-th layer node, and determining a reconstruction value of the latent variable of the i-th layer node according to the reconstruction value of the residual error of the i-th layer node; and inputting the reconstruction value of the latent variable of the ith-layer node into the first decoding network to generate a code stream of the geometric information of the ith-layer node. Decoding the code stream, and determining the reconstruction value of the residual error of the ith layer node; determining a reconstruction value of a latent variable of the i-th layer node according to the reconstruction value of the residual error of the i-th layer node; and inputting the reconstruction value of the latent variable of the ith-layer node into a second decoding network, and determining the geometric information of the ith-layer node.

Description

Coding and decoding method, encoder, decoder and storage medium Technical Field

[0001] The embodiments of the present application relate to the field of point cloud compression technology, and in particular to a coding and decoding method, an encoder, a decoder, and a storage medium. Background Art

[0002] Currently, OctSqueeze is a commonly used entropy model for laser radar (LIDAR) point clouds. OctSqueeze is an octree entropy model for LIDAR point clouds based on ancestral nodes. OctAttention is an extension of OctSqueeze technology and is an octree entropy model for LIDAR point clouds based on ancestral sibling nodes.

[0003] However, the OctSqueeze scheme assumes conditional independence of neighboring nodes given their known parent nodes, a generally incorrect assumption that can affect encoding and decoding efficiency. In the OctAttention scheme, since the context window continuously slides during decoding, the model must be continuously run to update the semantic information within the context window. This process lacks parallelism and therefore has extremely high time complexity.

[0004] Summary of the Invention

[0005] The embodiments of the present application provide a coding and decoding method, an encoder, a decoder, and a storage medium, which can improve the efficiency of point cloud coding and decoding while improving compression performance.

[0006] The technical solution of the embodiment of the present application can be implemented as follows:

[0007] In a first aspect, an embodiment of the present application provides an encoding method applied to an encoder, wherein the encoder includes a first embedding network, an encoding network, and a first decoding network, and the method includes:

[0008] Inputting the geometric information of the i-th layer node obtained by dividing the point cloud according to the octree into the first embedding network to determine the embedding features of the i-th layer node; wherein i is greater than 2;

[0009] Inputting the embedded features of the i-th layer nodes into the encoding network to determine the latent variables of the i-th layer nodes;

[0010] Determining a reconstructed value of the residual of the i-th layer node according to the latent variable of the i-th layer node, and determining a reconstructed value of the latent variable of the i-th layer node according to the reconstructed value of the residual of the i-th layer node;

[0011] The reconstructed value of the latent variable of the i-th layer node is input into the first decoding network to generate a code stream of the geometric information of the i-th layer node.

[0012] In a second aspect, an embodiment of the present application provides a decoding method, applied to a decoder, wherein the decoder includes a second decoding network, and the method includes:

[0013] Decode the code stream and determine the reconstructed value of the residual of the i-th layer node of the point cloud; where i is greater than 2;

[0014] Determining a reconstructed value of the latent variable of the i-th layer node according to the reconstructed value of the residual of the i-th layer node;

[0015] The reconstructed value of the latent variable of the i-th layer node is input into the second decoding network to determine the geometric information of the i-th layer node.

[0016] In a third aspect, an embodiment of the present application provides an encoder, comprising a first determining unit, an encoding unit, and a generating unit; wherein,

[0017] The first determining unit is configured to input the geometric information of the i-th layer node obtained by dividing the point cloud according to the octree into a first embedding network to determine the embedding features of the i-th layer node; wherein i is greater than 2; input the embedding features of the i-th layer node into an encoding network to determine the latent variables of the i-th layer node; determine the reconstructed value of the residual of the i-th layer node according to the latent variables of the i-th layer node, and determine the reconstructed value of the latent variable of the i-th layer node according to the reconstructed value of the residual of the i-th layer node;

[0018] The encoding unit is configured to write the reconstructed value of the residual of the i-th layer node into the bitstream;

[0019] The generating unit is configured to input the reconstructed value of the latent variable of the i-th layer node into the first decoding network to generate a code stream of the geometric information of the i-th layer node.

[0020] In a fourth aspect, an embodiment of the present application provides an encoder, comprising a first memory and a first processor; wherein,

[0021] The first memory is used to store a computer program that can be run on the first processor;

[0022] The first processor is configured to execute the encoding method described above when running the computer program.

[0023] In a fifth aspect, an embodiment of the present application provides a decoder, comprising a decoding unit and a second determining unit; wherein,

[0024] The decoding unit is configured to decode the code stream;

[0025] The second determination unit is configured to determine the reconstructed value of the residual of the i-th layer node obtained by dividing the point cloud according to the octree; wherein i is greater than 2; determine the reconstructed value of the latent variable of the i-th layer node based on the reconstructed value of the residual of the i-th layer node; input the reconstructed value of the latent variable of the i-th layer node into the second decoding network to determine the geometric information of the i-th layer node.

[0026] In a sixth aspect, an embodiment of the present application provides a decoder, comprising a second memory and a second processor; wherein:

[0027] The second memory is used to store a computer program that can be run on the second processor;

[0028] The second processor is configured to execute the above-mentioned decoding method when running the computer program.

[0029] In a seventh aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed, it implements the decoding method described above, or implements the encoding method described above.

[0030] An embodiment of the present application provides a coding and decoding method, an encoder, a decoder, and a storage medium. At the encoding end, the encoder includes a first embedding network, an encoding network, and a first decoding network. The encoder inputs the geometric information of the i-th layer node obtained by dividing the point cloud according to the octree into the first embedding network to determine the embedding features of the i-th layer node; wherein i is greater than 2; the embedding features of the i-th layer node are input into the encoding network to determine the latent variables of the i-th layer node; the reconstructed value of the residual of the i-th layer node is determined according to the latent variable of the i-th layer node, and the reconstructed value of the latent variable of the i-th layer node is determined according to the reconstructed value of the residual of the i-th layer node; the reconstructed value of the latent variable of the i-th layer node is input into the first decoding network to generate a code stream of the geometric information of the i-th layer node. At the decoding end, the decoder includes a second decoding network, which decodes the code stream and determines the reconstructed value of the residual of the i-th layer node of the point cloud; wherein i is greater than 2; determines the reconstructed value of the latent variable of the i-th layer node based on the reconstructed value of the residual of the i-th layer node; and inputs the reconstructed value of the latent variable of the i-th layer node into the second decoding network to determine the geometric information of the i-th layer node. It can be seen that in the embodiment of the present application, when encoding and decoding the point cloud, based on the constructed octree structure, latent variables can be used to characterize the spatial correlation between nodes of the same level in the octree, and ultimately higher compression performance can be obtained; at the same time, based on the constructed network, parallel encoding and decoding processing is performed on the nodes of the same level of the octree, thereby improving the encoding and decoding efficiency and further improving the compression performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] FIG1 is a schematic diagram of a network architecture of a point cloud encoding and decoding method provided in an embodiment of the present application;

[0032] FIG2 is a schematic diagram of a first implementation flow of the encoding method proposed in an embodiment of the present application;

[0033] FIG3 is a schematic diagram of a network structure embedded in a network;

[0034] FIG4 is a schematic diagram of the implementation of the embedded network;

[0035] FIG5 is a schematic diagram of the network structure of the coding network;

[0036] FIG6 is a schematic diagram of an implementation of a coding network;

[0037] FIG7 is a schematic diagram of the network structure of the subtraction network;

[0038] FIG8 is a schematic diagram of the implementation of the subtraction network;

[0039] FIG9 is a schematic diagram of the network structure of the addition network;

[0040] FIG10 is a schematic diagram of the implementation of the addition network;

[0041] FIG11 is a schematic diagram of a network structure of a decoding network;

[0042] FIG12 is a first schematic diagram of a decoding network implementation;

[0043] FIG13 is a second schematic diagram of the implementation flow of the encoding method proposed in an embodiment of the present application;

[0044] FIG14 is a second schematic diagram of the network structure of the decoding network;

[0045] FIG15 is a second schematic diagram of the implementation of the decoding network;

[0046] FIG16 is a schematic diagram of the overall framework for executing the encoding method;

[0047] FIG17 is a first schematic diagram of an implementation flow of a decoding method proposed in an embodiment of the present application;

[0048] FIG18 is a second schematic diagram of the implementation flow of the decoding method proposed in an embodiment of the present application;

[0049] FIG19 is a schematic diagram of the overall framework for executing the decoding method;

[0050] FIG20 is a schematic diagram of the structure of the encoder;

[0051] Figure 21 is a schematic diagram of the second structure of the encoder;

[0052] FIG22 is a schematic diagram of the structure of the decoder;

[0053] FIG23 is a second schematic diagram of the decoder structure. DETAILED DESCRIPTION

[0054] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below with reference to the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present application.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0056] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0057] It should also be pointed out that the terms "first\second\third" involved in the embodiments of the present application are only used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0058] Point cloud is a three-dimensional representation of the surface of an object. Point cloud (data) of the surface of an object can be collected through acquisition equipment such as photoelectric radar, lidar, laser scanner, and multi-view camera.

[0059] A point cloud is a set of irregularly distributed discrete points in space that express the spatial structure and surface properties of a three-dimensional object or scene. Figure 1A shows a three-dimensional point cloud image and Figure 1B shows a partially enlarged view of the three-dimensional point cloud image. It can be seen that the point cloud surface is composed of densely distributed points.

[0060] In a two-dimensional image, each pixel contains information and is distributed regularly, so there's no need to record its location. However, the distribution of points in a point cloud in three-dimensional space is random and irregular, so recording the location of each point in space is necessary to fully represent the point cloud. Similar to a two-dimensional image, each location in the acquisition process has corresponding attribute information, typically an RGB color value, which reflects the object's color. For a point cloud, in addition to color information, each point's corresponding attribute information often includes a reflectance value, which reflects the surface texture of the object. Therefore, a point in a point cloud can include both its location information and its attribute information. For example, the location information of a point can be its three-dimensional coordinates (x, y, z). The location information of a point can also be referred to as its geometric information. For example, the attribute information of a point can include color information (three-dimensional color information) and / or reflectance (one-dimensional reflectance information r). For example, the color information can be information in any color space. For example, the color information can be RGB information, where R represents red (Red), G represents green (Green), and B represents blue (Blue). For another example, the color information may be luminance and chrominance (YCbCr, YUV) information, where Y represents brightness (Luma), Cb (U) represents blue color difference, and Cr (V) represents red color difference.

[0061] For example, a point cloud generated using laser measurement principles can include both its 3D coordinate information and its reflectivity. For another example, a point cloud generated using photogrammetry principles can include both its 3D coordinate information and its 3D color information. For another example, a point cloud generated using a combination of laser measurement and photogrammetry principles can include both its 3D coordinate information, its reflectivity value, and its 3D color information.

[0062] Point clouds can be divided into the following categories according to the acquisition method:

[0063] Static point cloud: the object is stationary and the device that acquires the point cloud is also stationary;

[0064] Dynamic point cloud: The object is moving, but the device that obtains the point cloud is stationary;

[0065] Dynamic point cloud acquisition: The device used to acquire the point cloud is in motion.

[0066] For example, point clouds can be divided into two categories according to their usage:

[0067] Category 1: Machine perception point cloud, which can be used in scenarios such as autonomous navigation systems, real-time inspection systems, geographic information systems, visual sorting robots, and disaster relief robots;

[0068] Category 2: Human eye perception point cloud, which can be used in point cloud application scenarios such as digital cultural heritage, free viewpoint broadcasting, 3D immersive communication, and 3D immersive interaction.

[0069] Point clouds can flexibly and conveniently express the spatial structure and surface properties of three-dimensional objects or scenes. Moreover, since point clouds are obtained by directly sampling real objects, they can provide a strong sense of reality while ensuring accuracy. Therefore, they are widely used, including virtual reality games, computer-aided design, geographic information systems, automatic navigation systems, digital cultural heritage, free viewpoint broadcasting, three-dimensional immersive remote presentation, and three-dimensional reconstruction of biological tissues and organs.

[0070] Point clouds are primarily collected through computer generation, 3D laser scanning, and 3D photogrammetry. Computers can generate point clouds of virtual 3D objects and scenes; 3D laser scanning can obtain point clouds of static real-world 3D objects or scenes, generating millions of point clouds per second; and 3D photogrammetry can obtain point clouds of dynamic real-world 3D objects or scenes, generating tens of millions of point clouds per second. These technologies reduce the cost and time required to acquire point cloud data while improving data accuracy. While changes in point cloud data acquisition methods have made it possible to acquire large amounts of point cloud data, the processing of this massive amount of 3D point cloud data is facing bottlenecks due to storage space and transmission bandwidth constraints, as application demands grow.

[0071] For example, taking a point cloud video with a frame rate of 30 frames per second (fps), each frame contains 700,000 points, and each point has coordinate information (xyz, float) and color information (RGB, uchar). The data volume of a 10-second point cloud video is approximately 0.7 million × (4 bytes × 3 + 1 byte × 3) × 30 fps × 10 seconds = 3.15 GB. A 1280 × 720 2D video with a frame rate of 24 fps has a data volume of approximately 1280 × 720 × 12 bits × 24 fps × 10 seconds = 0.33 GB. A 10-second two-view 3D video has a data volume of approximately 0.33 × 2 = 0.66 GB. This shows that the data volume of a point cloud video far exceeds that of 2D and 3D videos of the same length. Therefore, in order to better realize data management, save server storage space, and reduce the transmission traffic and transmission time between the server and the client, point cloud compression has become a key issue in promoting the development of the point cloud industry.

[0072] That is to say, since the point cloud is a collection of massive points, storing the point cloud not only consumes a lot of memory, but is also not conducive to transmission. There is also not enough bandwidth to support direct transmission of the point cloud at the network layer without compression. Therefore, the point cloud needs to be compressed.

[0073] Currently, the point cloud coding framework that can compress point clouds can be the geometry-based Point Cloud Compression (G-PCC) codec framework or the video-based Point Cloud Compression (V-PCC) codec framework provided by the Moving Picture Experts Group (MPEG), or the AVS-PCC codec framework provided by AVS. The G-PCC codec framework can be used to compress the first type of static point clouds and the third type of dynamically acquired point clouds, and the V-PCC codec framework can be used to compress the second type of dynamic point clouds. The G-PCC codec framework is also called the point cloud codec TMC13, and the V-PCC codec framework is also called the point cloud codec TMC2.

[0074] An embodiment of the present application provides a network architecture of a point cloud encoding and decoding system including a decoding method and an encoding method. FIG1 is a schematic diagram of a network architecture of a point cloud encoding and decoding system provided by an embodiment of the present application. As shown in FIG1 , the network architecture includes one or more electronic devices 13 to 1N and a communication network 01, wherein the electronic devices 13 to 1N can perform video interaction through the communication network 01. During the implementation process, the electronic device can be various types of devices with point cloud encoding and decoding functions. For example, the electronic device can include a mobile phone, a tablet computer, a personal computer, a personal digital assistant, a navigator, a digital phone, a video phone, a television, a sensor device, a server, etc., which is not limited by the embodiment of the present application. Among them, the decoder or encoder in the embodiment of the present application can be the above-mentioned electronic device.

[0075] Among them, the electronic device in the embodiment of the present application has a point cloud encoding and decoding function, generally including a point cloud encoder (ie, encoder) and a point cloud decoder (ie, decoder).

[0076] In the embodiments of the present application, OctSqueeze is a LIDAR point cloud octree entropy model based on ancestor nodes. This technology is applied to the LIDAR point cloud entropy model and is mainly composed of the following parts:

[0077] (1) Encoder

[0078] The encoder first constructs an octree for the point cloud. For each node in the octree, its depth, parent node occupancy code, and coordinates are selected as context information, and its corresponding features are obtained through a multilayer perceptron network. This is then repeated k times. At the kth iteration, the features of the current node are concatenated with those of its parent node, and the features of the current node at the kth iteration are obtained through the kth multilayer perceptron. For each node's features after the kth iteration, a 256-dimensional softmax layer is used to obtain the probability of each occupancy code. The current occupancy code is losslessly encoded using arithmetic coding and the probability of the current occupancy code.

[0079] (2) Code stream

[0080] The code stream of the OctSqueeze technology is composed of the code stream corresponding to the occupied code of each layer of nodes.

[0081] (3) Decoder

[0082] The decoder decodes the octree in layers, from low to high layers. The decoding process is exactly the same as the encoding process except for the order of nodes.

[0083] (4) Loss function

[0084] The loss function is the cross entropy between the predicted node probability and the actual occupancy code, as follows:

[0085]

[0086] Among them, N is the number of octree nodes, x i represents the i-th node, j is the dimension in 256 dimensions, p ij is the probability of the corresponding occupied code.

[0087] In the embodiments of this application, OctAttention is a LIDAR point cloud octree entropy model based on ancestor sibling nodes. This technology is applied to the LIDAR point cloud entropy model and is an extension of the OctSqueeze technology. It mainly consists of the following parts:

[0088] (1) Encoder

[0089] This technique encodes and decodes point clouds in breadth-first order, using a context window to store already encoded and decoded nodes. For the current node to be encoded, an attention-based network is used to extract information from the context window, and a softmax function is used to determine the probability of each occupied code at the current node. Arithmetic coding is used to losslessly encode the current node and add it to the context window.

[0090] (2) Code stream

[0091] The code stream of this technology consists of the code streams of each node arranged in breadth-first order.

[0092] (3) Decoder

[0093] The decoding process is the same as the encoding process.

[0094] (4) Loss function

[0095] The loss function is the cross entropy between the predicted node probability and the actual occupancy code, as follows:

[0096]

[0097] Among them, N is the number of octree nodes, x i represents the i-th node, j is the dimension in 256 dimensions, p ij is the probability of the corresponding occupied code.

[0098] It's understandable that the aforementioned OctSqueeze scheme assumes conditional independence of neighboring nodes given their known parent nodes, a generally incorrect assumption that leads to suboptimal coding efficiency. Furthermore, in the aforementioned OctAttention scheme, since the context window continuously slides during decoding, the model must be continuously run to update the semantic information within the context window. This process lacks parallelism and, therefore, has extremely high time complexity.

[0099] It can be seen that the common point cloud encoding and decoding methods have the defect of high complexity, which affects the compression performance and reduces the encoding and decoding efficiency to a certain extent.

[0100] In order to solve the above problems, an embodiment of the present application provides a coding and decoding method, an encoder, a decoder and a storage medium. At the encoding end, the encoder includes a first embedding network, an encoding network, and a first decoding network. The encoder inputs the geometric information of the i-th layer node obtained by dividing the point cloud according to the octree into the first embedding network to determine the embedding features of the i-th layer node; wherein i is greater than 2; the embedding features of the i-th layer node are input into the encoding network to determine the latent variables of the i-th layer node; the reconstructed value of the residual of the i-th layer node is determined according to the latent variable of the i-th layer node, and the reconstructed value of the latent variable of the i-th layer node is determined according to the reconstructed value of the residual of the i-th layer node; the reconstructed value of the latent variable of the i-th layer node is input into the first decoding network to generate a code stream of the geometric information of the i-th layer node. At the decoding end, the decoder includes a second decoding network. The decoder decodes the code stream and determines the reconstructed value of the residual of the i-th layer node obtained by dividing the point cloud according to the octree; wherein i is greater than 2; the reconstructed value of the latent variable of the i-th layer node is determined based on the reconstructed value of the residual of the i-th layer node; the reconstructed value of the latent variable of the i-th layer node is input into the second decoding network to determine the geometric information of the i-th layer node. It can be seen that in the embodiment of the present application, when encoding and decoding the point cloud, based on the constructed octree structure, latent variables can be used to characterize the spatial correlation between nodes of the same level in the octree, and ultimately higher compression performance can be obtained; at the same time, based on the constructed network, parallel encoding and decoding processing is performed on the nodes of the same level of the octree, thereby improving the encoding and decoding efficiency and further improving the compression performance.

[0101] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0102] An embodiment of the present application proposes a point cloud encoding method, which can be applied to an encoder, wherein the encoder may include a first embedding network, an encoding network, and a first decoding network.

[0103] It should be noted that, in the embodiment of the present application, the encoder may be provided with a multi-layer network, wherein at least one layer of the multi-layer network may include a first embedding network, an encoding network, and a first decoding network.

[0104] Exemplarily, in an embodiment of the present application, the encoder may be provided with a 5-layer network, wherein one layer or multiple layers of the 5-layer network may include a first embedding network, an encoding network, and a first decoding network.

[0105] FIG2 is a schematic diagram of a first implementation flow of the encoding method proposed in an embodiment of the present application. As shown in FIG2 , the encoding method performed by the encoder may include the following steps:

[0106] Step 101: Input the geometric information of the i-th layer nodes obtained by dividing the point cloud according to the octree into the first embedding network to determine the embedding features of the i-th layer nodes; where i is greater than 2.

[0107] In an embodiment of the present application, the encoder may first input the geometric information of the i-th layer nodes obtained by dividing the point cloud according to the octree into the first embedding network, so as to determine the embedding features of the i-th layer nodes.

[0108] It should be noted that, in the embodiments of the present application, i can be an integer greater than 2.

[0109] It can be understood that, in the embodiment of the present application, the i-th layer network in the multi-layer network can perform parallel encoding processing on the geometric information of the i-th layer nodes.

[0110] Furthermore, in an embodiment of the present application, when geometric coding is performed on a point cloud, an octree may be constructed first. The octree structure is used to recursively partition the point cloud space into eight sub-blocks of equal size, and the occupancy codeword of each sub-block is determined. If the sub-block does not contain a point, it is recorded as empty; otherwise, it is recorded as non-empty. At the last level of the recursive partitioning, the occupancy codeword information of all blocks is recorded, and geometric coding is performed. The geometric information expressed by the octree structure can, on the one hand, further form a geometric code stream, and on the other hand, the reconstructed geometric information can be used as additional information for attribute coding during geometric reconstruction.

[0111] It can be understood that in the embodiment of the present application, for the constructed octree, the parent node layer can be defined as the high layer and the child node layer can be defined as the low layer; or the parent node layer can be defined as the low layer and the child node layer can be defined as the high layer.

[0112] The encoding order of the encoder may be from the root node to the leaf node of the constructed octree. Correspondingly, on the decoding side, the decoding order may be from the leaf node to the root node of the constructed octree. For example, if the root node layer of the constructed octree is the i-th layer, and the leaf node layers are from the i-1th layer to the first layer, then the encoding order is from the i-th layer to the first layer, while the decoding order is the opposite of the encoding order, that is, from the first layer to the i-th layer.

[0113] It can be understood that in the embodiments of the present application, the geometric information of the i-th layer node of the point cloud can represent the geometric information of the i-th layer node, such as the position coordinates of the node, and can also represent the occupation codeword situation of the i-th layer node, that is, the occupancy situation (placeholder information).

[0114] That is to say, in the embodiment of the present application, the geometric information of the i-th layer node includes both the position coordinate information of the node and the occupancy information of the node.

[0115] It can be understood that in an embodiment of the present application, at the encoding end, the input of the first embedding network is the geometric information of the i-th layer node, wherein the geometric information of the node as the input of the first embedding network can simultaneously include the node's position coordinate information and the node's placeholder information; and the code stream of the geometric information output by the first decoding network mainly includes the code stream of the node's placeholder information, and may include the code stream of the node's position coordinate information, or may not include the code stream of the node's position coordinate information.

[0116] It can be seen that in the embodiment of the present application, the embedding network and the encoding network need to refer to the placeholder information and position coordinate information of the node at the same time, while the decoding network focuses on generating the code stream of the placeholder information.

[0117] Furthermore, in an embodiment of the present application, the first embedding network may be an occupancy embedding network, wherein the first embedding network may be used to map octree nodes from 256-dimensional one-hot vectors to a 16-dimensional continuous feature space, so that occupancy conditions that are similar in space are also similar in the feature space.

[0118] It should be noted that, in an embodiment of the present application, the first embedding network may include a sparse convolutional network.

[0119] For example, in an embodiment of the present application, the network structure of the first embedded network can be composed of a multi-layer sparse convolutional network. For example, FIG3 is a schematic diagram of the network structure of the embedded network. As shown in FIG3, the network structure of the first embedded network can be a three-layer sparse convolutional network.

[0120] Furthermore, in an embodiment of the present application, after inputting the geometric information of the i-th layer node in the point cloud into the first embedding network, the encoder can output the embedded features of the i-th layer node. The embedded features of the i-th layer node can determine the occupancy of the i-th layer node after being mapped to the feature space.

[0121] That is to say, in the embodiment of the present application, the embedded features of the i-th layer nodes can determine the occupancy of the i-th layer nodes.

[0122] For example, in an embodiment of the present application, FIG4 is a schematic diagram of an implementation of an embedded network. As shown in FIG4, x (l) Represents the geometric information of the l-th layer node, e (l) Represents the embedding features of the l-th layer node output by the first embedding network. For the l-th layer node, the encoder can convert the geometric information x of the l-th layer node into(l) Input into the first embedding network, so that the embedding feature e of the corresponding l-th layer node can be obtained through the first embedding network (l) .

[0123] Step 102: Input the embedded features of the i-th layer nodes into the encoding network to determine the latent variables of the i-th layer nodes.

[0124] In an embodiment of the present application, after the encoder inputs the geometric information of the i-th layer node in the point cloud into the first embedding network and outputs the embedding features of the i-th layer node, it can further input the embedding features of the i-th layer node into the encoding network to determine the latent variables of the i-th layer node.

[0125] Furthermore, in an embodiment of the present application, the encoding network may be an encoder network. The encoding network may be used to determine the spatial correlation between the latent variables of the nodes in the previous layer and the embedded features of the nodes in the current layer, thereby obtaining the latent variables of the nodes in the current layer. That is, through the encoding network, the latent variables of the nodes in the i-th layer can be determined based on the embedded features of the nodes in the i-th layer output by the first embedding network.

[0126] It should be noted that, in the embodiment of the present application, the latent variable of the i-th layer node can represent the correlation between the sibling nodes of the i-th layer. In other words, in the present application, the encoder can make full use of the hierarchical latent variables to capture the correlation of the LIDAR point cloud.

[0127] It should be noted that, in the embodiments of the present application, the encoding network can be implemented by sparse convolution, which can specifically include a sparse convolutional network, a ReLU activation function, an initial residual network (Inception Resnet, IRN), and a feature fusion network Concatenate.

[0128] For example, in an embodiment of the present application, the network structure of the encoding network can be composed of a sparse convolutional network, a ReLU activation function, an initial residual network, and a concatenate network. For example, Figure 5 is a schematic diagram of the network structure of the encoding network. As shown in Figure 5, the network structure of the encoding network can be a sparse convolutional layer with a convolution kernel size of 2 and a stride of 2, followed by a ReLU activation function, and then connected to three layers of IRN, and finally connected to a sparse convolutional network after passing through the concatenate network.

[0129] Furthermore, in an embodiment of the present application, after the first embedding network outputs the embedding features of the i-th layer nodes, the encoder can continue to input the embedding features of the i-th layer nodes into the encoding network, and then output the latent variables of the i-th layer nodes through the encoding network.

[0130] It should be noted that in the embodiments of the present application, when the encoding network uses the embedded features of the current layer nodes to determine the latent variables of the current layer nodes, it needs to combine the latent variables of the previous layer nodes. The previous layer of the current layer can be understood as the parent node layer of the current node.

[0131] It can be understood that in an embodiment of the present application, for the layer above the current layer, after the encoding network in the previous layer determines the corresponding latent variable, the latent variable of the previous layer node can be input into the encoding network of the current layer, so that the encoding network of the current layer can determine the spatial correlation between the latent variable of the previous layer node and the embedded feature of the current layer node, thereby obtaining the latent variable of the current layer node.

[0132] That is, in an embodiment of the present application, the input of the encoding network of the i-th layer network may include the latent variables of the i+1-th layer nodes output by the encoding network of the i+1-th layer network.

[0133] Correspondingly, in an embodiment of the present application, after determining the latent variables of the i-th layer nodes, the encoding network of the i-th layer can also input the latent variables of the i-th layer nodes into the encoding network of the i-1-th layer network, that is, the input of the encoding network of the i-1-th layer network can include the latent variables of the i-th layer nodes output by the encoding network of the i-th layer network.

[0134] That is, in an embodiment of the present application, the encoder can input the embedded features of the i-th layer node and the latent variables of the i+1-th layer node into the encoding network of the i-th layer network, thereby determining the latent variables of the i-th layer node. The embedded features of the i-th layer node are output by the first embedding unit of the i-th layer network, and the latent variables of the i+1-th layer node are output by the encoding network of the i+1-th layer network.

[0135] For example, in an embodiment of the present application, FIG6 is a schematic diagram of an implementation of a coding network. As shown in FIG6 , e (l) represents the embedding features of the l-th layer node output by the first embedding network, f (l) represents the latent variable of the l-th layer node, f (l+1) Represents the latent variable of the l+1th layer node. For the lth layer node, the encoder can explore f (l+1) With e (l) The spatial correlation of the latent variable f is outputted at the lower layer (the lth layer). (l) For example, sparse tensors can be used to represent octrees, where the coordinate matrix of the sparse tensor represents the node coordinates of the octree; the attribute matrix of the sparse tensor represents the node occupancy code of the octree. The encoding network can be implemented using sparse convolution, where f (l)The downsampling of is achieved by a sparse convolution layer with a kernel size of 2 and a stride of 2, followed by a ReLU activation function. After that, the features of the sibling nodes are aggregated using the initial residual network and output to the Concatenate network to be combined with e (l) Perform feature fusion and finally, use a sparse convolution layer to fuse e (l) With the downsampled f (l+1) , get the latent variable of the low-level node, that is, the latent variable f of the l+1th layer node (l) .

[0136] Step 103: Determine the reconstructed value of the residual of the i-th layer node according to the latent variable of the i-th layer node, and determine the reconstructed value of the latent variable of the i-th layer node according to the reconstructed value of the residual of the i-th layer node.

[0137] In an embodiment of the present application, after the encoder inputs the embedded features of the i-th layer node into the encoding network and determines the latent variable of the i-th layer node, it can further determine the reconstructed value of the residual of the i-th layer node based on the latent variable of the i-th layer node, and then determine the reconstructed value of the latent variable of the i-th layer node based on the reconstructed value of the residual of the i-th layer node. At the same time, the reconstructed value of the residual of the i-th layer node can be written into the code stream.

[0138] It should be noted that, in an embodiment of the present application, when the encoder determines the reconstructed value of the residual of the i-th layer node based on the latent variable of the i-th layer node, it can first determine the residual of the i-th layer node based on the predicted value of the latent variable of the i-th layer node and the latent variable of the i-th layer node; then the residual of the i-th layer node can be quantized, and then the reconstructed value of the residual of the i-th layer node can be determined.

[0139] It is understood that in the embodiment of the present application, the encoder can write the reconstructed value of the residual of the i-th layer node into the code stream and transmit it to the decoding end. In other words, the encoder can decompose and compress the latent variable by using residual coding.

[0140] Furthermore, in an embodiment of the present application, the predicted value of the latent variable of the i-th layer node can be obtained first. The predicted value of the latent variable of the i-th layer node can be obtained by prediction by the first decoding network of the next layer network of the current layer, that is, the predicted value of the latent variable of the i-th layer node can be output by the first decoding network in the i-1th layer network.

[0141] It should be noted that, in an embodiment of the present application, the encoder may further include a subtraction network, wherein the subtraction network may be used to determine the residual of the latent variable.

[0142] That is, in the embodiment of the present application, at least one layer of the multi-layer network provided by the encoder may further include a subtraction network, wherein the subtraction network is a soft subtraction network that can be used to replace the common hard subtraction process.

[0143] For example, in an embodiment of the present application, based on a subtraction network, the residual can be determined by the following formula:

[0144]

[0145] Among them, g s Represents the operation after splicing through a sparse convolution layer, r (l) Represents the residual of the l-th layer node, f (l) represents the latent variable of the l-th layer node, Represents the predicted value of the latent variable of the l-th layer node.

[0146] Exemplarily, in an embodiment of the present application, when determining the residual of the i-th layer node, the encoder can input the predicted value of the latent variable of the i-th layer node and the latent variable of the i-th layer node into the subtraction network, and then obtain the residual of the i-th layer node.

[0147] It should be noted that, in the embodiments of the present application, the subtraction network may include a sparse convolutional network and a feature fusion network Concatenate.

[0148] Exemplarily, in an embodiment of the present application, Figure 7 is a schematic diagram of the network structure of the subtraction network. As shown in Figure 7, the network structure of the subtraction network can be a sparse convolution layer with a convolution kernel size of 3 and a step size of 1, followed by a Concatenate network.

[0149] For example, in an embodiment of the present application, FIG8 is a schematic diagram of an implementation of a subtraction network. As shown in FIG8 , f (l) represents the latent variable of the l-th layer node, Represents the predicted value of the latent variable of the l-th layer node, r (l) Represents the residual of the l-th layer node. Among them, f (l) and After the fusion of the Concatenate network, the residual r of the l-th layer node can be output through the operation of the sparse convolution layer. (l) .

[0150] Furthermore, in an embodiment of the present application, after the encoder quantizes the obtained residual of the i-th layer node and determines the reconstructed value of the residual of the i-th layer node, it can further determine the reconstructed value of the latent variable of the i-th layer node based on the reconstructed value of the residual of the i-th layer node and the predicted value of the latent variable of the i-th layer node. The predicted value of the latent variable of the i-th layer node can be output by the first decoding network in the i-1th layer.

[0151] It should be noted that, in an embodiment of the present application, the encoder may further include an addition network, wherein the addition network may be used to determine the reconstructed value of the latent variable.

[0152] That is, in the embodiment of the present application, at least one layer of the multi-layer network provided by the encoder may further include an addition network, wherein the addition network is a soft addition network that can be used to replace the common hard addition process.

[0153] For example, in an embodiment of the present application, based on the additive network, the predicted value of the latent variable can be determined by the following formula:

[0154]

[0155] Among them, g a Represents the operation of passing through a sparse convolution layer after splicing, Represents the reconstructed value of the residual of the l-th layer node, represents the reconstructed value of the latent variable of the l-th layer node, Represents the predicted value of the latent variable of the l-th layer node.

[0156] Exemplarily, in an embodiment of the present application, when the encoder determines the reconstructed value of the latent variable of the i-th layer node, it can first input the reconstructed value of the residual of the i-th layer node and the predicted value of the latent variable of the i-th layer node into the additive network, and then obtain the reconstructed value of the latent variable of the i-th layer node.

[0157] It should be noted that, in the embodiments of the present application, the addition network may include a sparse convolutional network and a feature fusion network Concatenate.

[0158] Exemplarily, in an embodiment of the present application, Figure 9 is a schematic diagram of the network structure of the additive network. As shown in Figure 9, the network structure of the additive network can be a sparse convolution layer with a convolution kernel size of 3 and a step size of 1, followed by a concatenate network.

[0159] For example, in an embodiment of the present application, FIG10 is a schematic diagram of an implementation of an addition network. As shown in FIG10 , Represents the reconstructed value of the residual of the l-th layer node, represents the reconstructed value of the latent variable of the l-th layer node, Represents the predicted value of the latent variable of the l-th layer node. and After the fusion of the Concatenate network, the reconstruction value of the latent variable of the l-th layer node can be output through the operation of the sparse convolution layer.

[0160] It can be understood that, in the embodiments of the present application, a soft addition / subtraction network (addition network and subtraction network) is used to replace the original hard addition and subtraction, which greatly improves the network flexibility.

[0161] Step 104: Input the reconstructed value of the latent variable of the i-th layer node into the first decoding network to generate a code stream of the geometric information of the i-th layer node.

[0162] In an embodiment of the present application, after the encoder determines the reconstructed value of the latent variable of the i-th layer node based on the latent variable of the i-th layer node, the encoder can further input the reconstructed value of the latent variable of the i-th layer node into the first decoding network, thereby generating a code stream of the geometric information of the i-th layer node.

[0163] It should be noted that in an embodiment of the present application, before the code stream of the geometric information of the i-th layer node is generated through the first decoding network, the first embedding network from the i-th layer network to the first layer network has completed the output of the corresponding embedded features, the encoding network from the i-th layer network to the first layer network has completed the output of the corresponding latent variables, and at the same time, the i-1-th layer network to the first layer network has completed the output of the reconstructed values ​​of the corresponding latent variables.

[0164] Furthermore, in an embodiment of the present application, the input of the first decoding network of the i-th layer network may include: the embedding features of the i-2th layer nodes output by the first embedding network of the i-2th layer network, the embedding features of the i-1th layer nodes output by the first embedding network of the i-1th layer network, and the reconstructed values ​​of the latent variables of the i-1th layer nodes obtained by the i-1th layer network; the input of the first decoding network of the i-th layer network may also include the reconstructed values ​​of the latent variables of the i-th layer nodes.

[0165] It should be noted that, in an embodiment of the present application, when the encoder generates a code stream of the geometric information of the i-th layer node based on the reconstructed value of the latent variable of the i-th layer node, the reconstructed value of the latent variable of the i-th layer node, the embedded feature of the i-2 layer node, the embedded feature of the i-1 layer node, and the reconstructed value of the latent variable of the i-1 layer node are first input into the first decoding network of the i-th layer network, thereby generating a code stream of the geometric information of the i-th layer node.

[0166] It can be understood that in the embodiment of the present application, based on the first decoding network, the probability parameters of the i-th layer nodes can be determined first; then the code stream of the geometric information of the i-th layer nodes can be further generated according to the i-th layer probability parameters.

[0167] That is to say, in an embodiment of the present application, the embedding features of the i-2 layer nodes, the embedding features of the i-1 layer nodes, and the reconstruction values ​​of the latent variables of the i-1 layer nodes can be obtained first. Among them, the embedding features of the i-2 layer nodes can be obtained by the first embedding network of the two layers of networks below the current layer, that is, the embedding features of the i-2 layer nodes can be output by the first embedding network in the i-2 layer network; the embedding features of the i-1 layer nodes can be obtained by the first embedding network of the layer below the current layer, that is, the embedding features of the i-1 layer nodes can be output by the first embedding network in the i-1 layer network; the reconstruction values ​​of the latent variables of the i-1 layer nodes can be obtained by the network of the layer below the current layer.

[0168] It should be noted that, in an embodiment of the present application, the first decoding network may include a sparse convolutional network, a deconvolution layer, an initial residual network, a feature fusion network, and a ReLU activation function.

[0169] Exemplarily, in an embodiment of the present application, Figure 11 is a schematic diagram of the network structure of the decoding network. As shown in Figure 11, the network structure of the first decoding network can be a concatenate network followed by two layers of sparse convolution layers with a convolution kernel size of 3 and a stride of 1, followed by an autoencoder, such as Binary AE.

[0170] For example, in an embodiment of the present application, FIG12 is a schematic diagram of a first implementation of a decoding network. As shown in FIG12 , represents the reconstructed value of the latent variable of the l-th layer node, represents the reconstructed value of the latent variable of the l-1th layer node, e (l-1) represents the embedding features of the l-1th layer node output by the first embedding network, e (l-2) Represents the embedding features of the l-2th layer nodes output by the first embedding network. e (l-1) , e (l-2) After that, you can use the Concatenate network to e (l-1) , e (l-2) After splicing, a two-layer sparse convolutional network is used to predict the probability p of each node in the lth layer. (l) , and finally generate x through arithmetic encoder, such as Binary AE, entropy coding (l) The corresponding binary code stream.

[0171] Furthermore, in an embodiment of the present application, FIG13 is a second schematic diagram of an implementation flow of the encoding method proposed in an embodiment of the present application. As shown in FIG13 , after determining the reconstructed value of the latent variable of the i-th layer node based on the latent variable of the i-th layer node, that is, after step 103, the encoding method of the encoder may further include the following steps:

[0172] Step 105: Input the reconstructed value of the latent variable of the i-th layer node into the first decoding network to determine the predicted value of the latent variable of the i+1-th layer node.

[0173] In an embodiment of the present application, after the encoder determines the reconstructed value of the latent variable of the i-th layer node based on the latent variable of the i-th layer node, it can further input the reconstructed value of the latent variable of the i-th layer node into the first decoding network, so as to determine the predicted value of the latent variable of the i+1-th layer node.

[0174] It should be noted that, in an embodiment of the present application, before the predicted value of the latent variable of the i+1th layer node is generated by the first decoding network, the first embedding network from the i-th layer network to the first layer network has completed the output of the corresponding embedded features, the encoding network from the i-th layer network to the first layer network has completed the output of the corresponding latent variables, and at the same time, the i-1th layer network to the first layer network has completed the output of the reconstructed value of the corresponding latent variable.

[0175] Furthermore, in an embodiment of the present application, the input of the first decoding network of the i-th layer network may include: the embedding features of the i-1th layer nodes output by the first embedding network of the i-1th layer network, and the reconstructed values ​​of the latent variables of the i-1th layer nodes obtained by the i-1th layer network; the input of the first decoding network of the i-th layer network may also include the reconstructed values ​​of the latent variables of the i-th layer nodes and the i-th embedding features.

[0176] It should be noted that, in an embodiment of the present application, when the encoder determines the predicted value of the latent variable of the i+1th layer node based on the reconstructed value of the latent variable of the i-th layer node, the reconstructed value of the latent variable of the i-th layer node, the embedded feature of the i-th layer node, the embedded feature of the i-1th layer node, and the reconstructed value of the latent variable of the i-1th layer node can be input into the first decoding network of the i-th layer network, so as to determine the predicted value of the latent variable of the i+1th layer node.

[0177] That is, in an embodiment of the present application, the embedding features of the i-1th layer node and the reconstructed values ​​of the latent variables of the i-1th layer node can be obtained first. The embedding features of the i-1th layer node can be obtained by the first embedding network of the next layer network of the current layer, that is, the embedding features of the i-1th layer node can be output by the first embedding network in the i-1th layer network; the reconstructed values ​​of the latent variables of the i-1th layer node can be obtained by the network of the next layer of the current layer.

[0178] It should be noted that, in an embodiment of the present application, the first decoding network may include a sparse convolutional network, a deconvolution layer, an initial residual network, a feature fusion network, and a ReLU activation function.

[0179] Exemplarily, in an embodiment of the present application, Figure 14 is a second schematic diagram of the network structure of the decoding network. As shown in Figure 14, the network structure of the first decoding network can be a sparse convolution layer with a convolution kernel size of 2 and a step size of 2 followed by a Concatenate network, followed by three initial residual networks, and followed by a sparse convolution layer with a convolution kernel size of 3 and a step size of 1.

[0180] For example, in an embodiment of the present application, FIG15 is a second schematic diagram of an implementation of a decoding network. As shown in FIG15 , represents the reconstructed value of the latent variable of the l-th layer node, represents the reconstructed value of the latent variable of the l-1th layer node, e (l) represents the embedding features of the lth layer node output by the first embedding network, e (l-1) Represents the embedding features of the l-1th layer node output by the first embedding network. e (l) , e (l-1) After that, you can use the Concatenate network to e (l-1) , e (l-2) After splicing, it passes through a deconvolution layer with a convolution kernel size of 2 and a step size of 2, and then passes through the initial residual network to obtain the predicted value of the latent variable of the l+1 layer node

[0181] It can be seen that in an embodiment of the present application, for the first decoding network, two different branches can be output based on different inputs, one branch is the code stream corresponding to the geometric information of the node of the i-th layer of the octree, and the other branch is the predicted value of the latent variable of the node of the i+1-th layer of the octree.

[0182] It can be understood that in an embodiment of the present application, the code stream corresponding to the geometric information of the i-th layer node of the octree can be transmitted to the decoding end, and the predicted value of the latent variable of the i+1-th layer node of the octree can be input into the i+1-th layer network for determining the residual of the i+1-th layer node and the reconstructed value of the latent variable of the i+1-th layer node.

[0183] Furthermore, in an embodiment of the present application, for the first layer, the encoder can first quantize the latent variables of the first-layer nodes output by the encoding unit of the first-layer network to determine the reconstructed values ​​of the latent variables of the first-layer nodes; then the reconstructed values ​​of the latent variables of the first-layer nodes and the preset embedded features can be output to the first decoding unit of the first-layer network, and then the code stream of the geometric information of the first-layer nodes and the predicted values ​​of the latent variables of the second-layer nodes can be output respectively.

[0184] For example, in an embodiment of the present application, the first layer may be a layer in the octree where all leaf nodes are empty. That is, when constructing an octree for a point cloud, the last layer of the octree that cannot be further divided is the first layer.

[0185] For example, in the embodiments of the present application, the first layer may also be a layer in the octree that is divided into the smallest unit, such as a 1x1x1 smallest unit block. In other words, when constructing an octree for a point cloud, the last layer of the octree that is divided into the smallest unit block is the first layer.

[0186] Furthermore, in an embodiment of the present application, the encoder may adopt an entropy encoder in the style of a factorized variational auto-encoder to perform residual r (l) compression.

[0187] For example, in the embodiment of the present application, for the variable y to be compressed, since the quantization process is not differentiable, uniform noise U(-0.5, 0.5) is used instead of quantization in the training phase. The quantized variable is recorded as Then the following formula:

[0188]

[0189] Among them, p(x) is defined as the probability distribution of x, and the probability hypothesis The independence of each dimension can be expressed as follows:

[0190]

[0191] Among them, the cumulative distribution c(y) fitted by the neural network is based on c(y) should have the following properties:

[0192] y∈(-∞,∞)

[0193] c(y) increases

[0194] c(-∞)=0,c(∞)=1

[0195] Therefore, the following neural network is used to fit c(y):

[0196]

[0197] f k (x) = g k (H (k) x+b (k) ),1≤k≤K

[0198] f K (x) = sigmoid(H (K) x+b (K) )

[0199] Calculated by p(y) The method is as follows:

[0200]

[0201] get After the probability distribution, use the arithmetic encoder to Perform entropy encoding and decoding.

[0202] Furthermore, in the embodiment of the present application, at the encoding end, the final output code stream can be composed of the code streams corresponding to the output of at least one layer of the multi-layer network. For one layer of the network, such as the lth layer network, the output code stream includes the lth layer node generated by the factored variational autoencoder. The code stream, the geometric information x of the l-th layer node generated by the arithmetic encoder (l) Code stream and structure information.

[0203] In summary, the encoding method proposed in steps 101 to 105 above can rely on the octree structure, use the constructed embedding network, encoding network and decoding network, introduce latent variables to capture the correlation between brother nodes in the same layer of the octree, and obtain higher compression performance; at the same time, parallel encoding and decoding processing is performed in units of octree levels, which can effectively shorten the encoding and decoding time, greatly improve the encoding and decoding efficiency, and further improve the compression performance.

[0204] The embodiment of the present application provides an encoding method. At the encoding end, the encoder includes a first embedding network, an encoding network, and a first decoding network. The encoder inputs the geometric information of the i-th layer node obtained by dividing the point cloud according to the octree into the first embedding network to determine the embedding features of the i-th layer node; wherein i is greater than 2; the embedding features of the i-th layer node are input into the encoding network to determine the latent variables of the i-th layer node; the reconstructed value of the residual of the i-th layer node is determined according to the latent variables of the i-th layer node, and the reconstructed value of the latent variable of the i-th layer node is determined according to the reconstructed value of the residual of the i-th layer node; the reconstructed value of the latent variable of the i-th layer node is input into the first decoding network to generate a code stream of the geometric information of the i-th layer node. It can be seen that in the embodiment of the present application, when encoding and decoding the point cloud, based on the constructed octree structure, the latent variables can be used to characterize the spatial correlation between nodes of the same level in the octree, and ultimately higher compression performance can be obtained; at the same time, based on the constructed network, parallel encoding and decoding processing is performed on the nodes of the same level of the octree, thereby improving the encoding and decoding efficiency and further improving the compression performance.

[0205] Based on the above embodiments, the encoding method proposed in the embodiments of this application can be understood as an end-to-end LIDAR point cloud deep entropy model, or as an end-to-end self-supervised dynamic point cloud compression technology based on deep learning. By relying on an octree structure, introducing latent variables to capture the correlation between octree sibling nodes, and constructing a network using sparse convolution, the method achieves optimal results in a lossless entropy model for LIDAR point clouds.

[0206] At the same time, the encoding method proposed in the embodiment of the present application can improve the encoding and decoding efficiency of LIDAR point clouds. Specifically, the embodiment of the present application has a lower encoding and decoding time. This is because the encoding and decoding of each node in the embodiment of the present application is processed in parallel on a layer-by-layer basis and the network is constructed using efficient sparse convolution. Therefore, the encoding and decoding time is much lower than that of common encoding and decoding solutions.

[0207] It should be noted that the network for executing the encoding method proposed in the embodiment of the present application can make full use of hierarchical latent variables to capture the correlation of LIDAR point clouds, and can also use residual coding to decompose and compress latent variables. Furthermore, a soft addition / subtraction network can be used to replace the original hard addition and subtraction, thereby improving the flexibility of the network.

[0208] Furthermore, in an embodiment of the application, FIG16 is a schematic diagram of the overall framework for executing the encoding method. As shown in FIG16 , the encoder may be provided with a multi-layer network, wherein at least one layer of the multi-layer network may include a first embedding network (embedding), an encoding network (encoder), and a first decoding network (decoder). For example, each layer is composed of an encoding network (encoder), a decoding network (decoder), and an embedding network (embedding).

[0209] It can be understood that, in the embodiment of the present application, the lth layer network in the multi-layer network can perform parallel encoding processing on the geometric information of the lth layer nodes.

[0210] As shown in the figure, x (l) Represents the geometric information of the l-th layer node, e (l) represents the embedding feature of the lth layer node after occupying the embedding network, f (l) represents the latent variable of the l-th layer node, represents the reconstructed value of the l-th layer residual, represents the reconstructed value of the latent variable of the l-th layer node, Represents the predicted value of the latent variable of the l-th layer node. In the figure, the + / - symbol represents the soft addition / subtraction network (addition network / subtraction network), and Q represents the quantization operation.

[0211] For example, in the embodiment of the present application, as shown in the figure, the network set in the encoder can be composed of a 5-layer network. Among them, for the l-th layer network, the geometric information x of the l-th layer node (l) By embedding the network, we get the occupancy embedding e (l) (Embedding features). The encoding network captures (l) and the latent variable f of the previous layer (the l+1 layer) (l+1) The spatial correlation of the node and output the latent variable f of the current layer (l) The decoding network is divided into two paths, generating the current layer x of the octree respectively. (l) The code stream and the predicted value of the latent variable of the previous layer node Among them, the predicted value of the latent variable of the l-th layer node is and the true value f (l) The residual of the l-th layer node is r (l) , which is the reconstructed value of the residual of the l-th layer node after quantization Lossless entropy coding by factorized variational autoencoders.

[0212] It should be noted that, in the embodiment of the present application, for the first layer (the 1st to 5th layers as shown in the figure), the encoder may first perform the latent variable f of the first layer node output by the encoding unit of the first layer network. (l-5)Perform quantization to determine the reconstruction value of the latent variable of the first layer node Then the reconstructed value of the latent variable of the first layer node and the preset embedding feature e (l-6) Output to the first decoding unit of the first layer network, and then output the code stream of the geometric information of the first layer nodes and the predicted value of the latent variable of the second layer nodes respectively

[0213] Furthermore, in an embodiment of the present application, as shown in the figure, the input of the encoding network of the l-1 layer network may include the latent variables of the l-1 layer nodes output by the encoding network of the l-1 layer network. That is, in an embodiment of the present application, the encoder may input the embedded features of the l-1 layer nodes and the latent variables of the l-1 layer nodes into the encoding network of the l-1 layer network, thereby determining the latent variables of the l-1 layer nodes. Among them, the embedded features of the l-1 layer nodes are output by the first embedding unit of the l-1 layer network, and the latent variables of the l-1 layer nodes are output by the encoding network of the l-1 layer network.

[0214] Furthermore, in an embodiment of the present application, as shown in the figure, the input of the first decoding network of the l-1 layer network may include: the embedded features of the l-3 layer nodes output by the first embedding network of the l-3 layer network, the embedded features of the l-2 layer nodes output by the first embedding network of the l-2 layer network, and the reconstructed values ​​of the latent variables of the l-2 layer nodes obtained by the l-2 layer network; the input of the first decoding network of the l-1 layer network may also include the reconstructed values ​​of the latent variables of the l-1 layer nodes. Accordingly, when the encoder generates the code stream of the geometric information of the l-1 layer nodes, it can first input the reconstructed values ​​of the latent variables of the l-1 layer nodes, the embedded features of the l-3 layer nodes, the embedded features of the l-2 layer nodes, and the reconstructed values ​​of the latent variables of the l-2 layer nodes into the first decoding network of the l-1 layer network, thereby generating the code stream of the geometric information of the l-1 layer nodes.

[0215] It should be noted that, in an embodiment of the present application, when the encoder determines the predicted value of the latent variable of the l-th layer node, the reconstructed value of the latent variable of the l-1th layer node, the embedded features of the l-1th layer node, the embedded features of the l-2th layer node, and the reconstructed value of the latent variable of the l-2th layer node can be input into the first decoding network of the l-1th layer network, so as to determine the predicted value of the latent variable of the l-th layer node.

[0216] Furthermore, in an embodiment of the present application, as shown in the figure, after the first decoding network of the l-1 layer network outputs the predicted value of the latent variable of the l-layer node, the predicted value of the latent variable of the l-layer node can be used in the subsequent encoding processing of the l-layer, and the addition network and the subtraction network can be used to respectively determine the residual of the l-layer node and the reconstructed value of the latent variable of the l-layer node.

[0217] For example, in an embodiment of the present application, an encoder network can be used to explore e (l) and the latent variable f of the previous layer (the l+1 layer) (l+1) The spatial correlation of the node and output the latent variable f of the current layer (l) For example, sparse tensors can be used to represent octrees, where the coordinate matrix of the sparse tensor represents the node coordinates of the octree; the attribute matrix of the sparse tensor represents the node occupancy code of the octree. The encoding network can be implemented using sparse convolution, where f (l) The downsampling of is achieved by a sparse convolution layer with a kernel size of 2 and a stride of 2, followed by a ReLU activation function. After that, the features of the sibling nodes are aggregated using the initial residual network and output to the Concatenate network to be combined with e (l) Perform feature fusion and finally, use a sparse convolution layer to fuse e (l) With the downsampled f (l+1) , get the latent variable of the low-level node, that is, the latent variable f of the l+1th layer node (l) .

[0218] Exemplarily, in an embodiment of the present application, the decoding network (decoder) is used to generate the current layer x of the octree (l) The code stream and the predicted value of the latent variable of the previous layer node Among them, one branch includes: obtaining e (l-1) , e (l-2) After that, you can use the Concatenate network to e (l-1) , e (l-2) After splicing, a two-layer sparse convolutional network is used to predict the probability p of each node in the lth layer. (l) , and finally generate x through arithmetic encoder and entropy coding (l) The corresponding binary code stream. Another branch includes: e (l) , e (l-1) After that, you can use the Concatenate network to e (l-1) , e (l-2)After splicing, it passes through a deconvolution layer with a convolution kernel size of 2 and a step size of 2, and then passes through the initial residual network to obtain the predicted value of the latent variable of the l+1 layer node

[0219] Exemplarily, in an embodiment of the present application, an embedding network can be used to map octree nodes from 256-dimensional one-hot vectors to a 16-dimensional continuous feature space, so that occupancy conditions that are similar in space are also similar in the feature space.

[0220] For example, in an embodiment of the present application, at least one layer of the multi-layer network provided by the encoder may further include a subtraction network and an addition network. Using a soft addition / subtraction network (addition network and subtraction network) instead of the original hard addition and subtraction greatly improves network flexibility.

[0221] To sum up, the encoding method proposed in the embodiment of the present application can rely on the octree structure, use the constructed embedding network, encoding network and decoding network, introduce latent variables to capture the correlation between brother nodes in the same layer of the octree, and obtain higher compression performance; at the same time, parallel encoding and decoding processing is performed in units of octree levels, which can effectively shorten the encoding and decoding time, greatly improve the encoding and decoding efficiency, and further improve the compression performance.

[0222] The embodiment of the present application provides an encoding method. At the encoding end, the encoder includes a first embedding network, an encoding network, and a first decoding network. The encoder inputs the geometric information of the i-th layer node obtained by dividing the point cloud according to the octree into the first embedding network to determine the embedding features of the i-th layer node; wherein i is greater than 2; the embedding features of the i-th layer node are input into the encoding network to determine the latent variables of the i-th layer node; the reconstructed value of the residual of the i-th layer node is determined according to the latent variables of the i-th layer node, and the reconstructed value of the latent variable of the i-th layer node is determined according to the reconstructed value of the residual of the i-th layer node; the reconstructed value of the latent variable of the i-th layer node is input into the first decoding network to generate a code stream of the geometric information of the i-th layer node. It can be seen that in the embodiment of the present application, when encoding and decoding the point cloud, based on the constructed octree structure, the latent variables can be used to characterize the spatial correlation between nodes of the same level in the octree, and ultimately higher compression performance can be obtained; at the same time, based on the constructed network, parallel encoding and decoding processing is performed on the nodes of the same level of the octree, thereby improving the encoding and decoding efficiency and further improving the compression performance.

[0223] An embodiment of the present application proposes a point cloud decoding method, which can be applied to a decoder, wherein the decoder may include a second embedding network and a second decoding network.

[0224] It should be noted that, in the embodiment of the present application, the decoder may be provided with a multi-layer network, wherein at least one layer of the multi-layer network may include a second embedding network and a second decoding network.

[0225] Exemplarily, in an embodiment of the present application, the decoder may be provided with a five-layer network, wherein one layer or multiple layers of the five-layer network may include a second embedding network and a second decoding network.

[0226] FIG17 is a first schematic diagram of a decoding method according to an embodiment of the present application. As shown in FIG17 , the decoding method may include the following steps:

[0227] Step 201: Decode the bitstream and determine the reconstructed value of the residual of the i-th layer node obtained by partitioning the point cloud according to the octree; where i is greater than 2.

[0228] In an embodiment of the present application, the decoder can determine the reconstructed value of the residual of the i-th layer node obtained by dividing the point cloud according to the octree by decoding the code stream.

[0229] It should be noted that, in the embodiments of the present application, i can be an integer greater than 2.

[0230] It can be understood that, in the embodiment of the present application, the i-th layer network in the multi-layer network can perform parallel decoding processing on the geometric information of the i-th layer nodes.

[0231] Furthermore, in an embodiment of the present application, when geometric coding is performed on a point cloud, an octree may be constructed first. The octree structure is used to recursively partition the point cloud space into eight sub-blocks of equal size, and the occupancy codeword of each sub-block is determined. If the sub-block does not contain a point, it is recorded as empty; otherwise, it is recorded as non-empty. At the last level of the recursive partitioning, the occupancy codeword information of all blocks is recorded, and geometric coding is performed. The geometric information expressed by the octree structure can, on the one hand, further form a geometric code stream, and on the other hand, the reconstructed geometric information can be used as additional information for attribute coding during geometric reconstruction.

[0232] Accordingly, in the embodiment of the present application, by parsing the geometric code stream transmitted by the encoding side, the reconstructed value of the residual of the i-th layer node of the point cloud can be determined.

[0233] It can be understood that in the embodiment of the present application, for the constructed octree, the parent node layer can be defined as the high layer and the child node layer can be defined as the low layer; or the parent node layer can be defined as the low layer and the child node layer can be defined as the high layer.

[0234] The encoding order of the encoder may be from the root node to the leaf node of the constructed octree. Correspondingly, on the decoding side, the decoding order may be from the leaf node to the root node of the constructed octree. For example, if the root node layer of the constructed octree is the i-th layer, and the leaf node layers are from the i-1th layer to the first layer, then the encoding order is from the i-th layer to the first layer, while the decoding order is the opposite of the encoding order, that is, from the first layer to the i-th layer.

[0235] It should be noted that in an embodiment of the present application, at the encoding end, the residual of the i-th layer node can be determined based on the predicted value of the latent variable of the i-th layer node and the latent variable of the i-th layer node; then the residual of the i-th layer node can be quantized, and then the reconstructed value of the residual of the i-th layer node can be determined.

[0236] It is understood that in the embodiment of the present application, the encoder can write the reconstructed value of the residual of the i-th layer node into the code stream and transmit it to the decoding end. In other words, the encoder can decompose and compress the latent variable by using residual coding.

[0237] Step 202: Determine the reconstructed value of the latent variable of the i-th layer node according to the reconstructed value of the residual of the i-th layer node.

[0238] In an embodiment of the present application, after the decoder determines the reconstructed value of the residual of the i-th layer node by decoding the code stream, it can determine the reconstructed value of the latent variable of the i-th layer node based on the reconstructed value of the residual of the i-th layer node.

[0239] Furthermore, in an embodiment of the present application, the predicted value of the latent variable of the i-th layer node can be obtained first. The predicted value of the latent variable of the i-th layer node can be obtained by prediction by the second decoding network of the next layer network of the current layer, that is, the predicted value of the latent variable of the i-th layer node can be output by the second decoding network in the i-1th layer network.

[0240] Furthermore, in an embodiment of the present application, after determining the reconstructed value of the residual of the i-th layer node, the decoder may further determine the reconstructed value of the latent variable of the i-th layer node based on the reconstructed value of the residual of the i-th layer node and the predicted value of the latent variable of the i-th layer node. The predicted value of the latent variable of the i-th layer node may be output by the second decoding network in the i-1th layer.

[0241] It should be noted that, in an embodiment of the present application, the decoder may further include an addition network, wherein the addition network may be used to determine the reconstructed value of the latent variable.

[0242] That is, in the embodiment of the present application, at least one layer of the multi-layer network provided by the decoder may further include an addition network, wherein the addition network is a soft addition network that can be used to replace the common hard addition process.

[0243] Illustratively, in an embodiment of the present application, based on an addition network, the predicted value of the latent variable can be determined by the above formula (2).

[0244] Exemplarily, in an embodiment of the present application, when the decoder determines the reconstructed value of the latent variable of the i-th layer node, it can first input the reconstructed value of the residual of the i-th layer node and the predicted value of the latent variable of the i-th layer node into the additive network, and then obtain the reconstructed value of the latent variable of the i-th layer node.

[0245] It should be noted that, in the embodiments of the present application, the addition network may include a sparse convolutional network and a feature fusion network Concatenate.

[0246] Exemplarily, in an embodiment of the present application, as shown in FIG9 , the network structure of the additive network can be a sparse convolution layer with a convolution kernel size of 3 and a stride of 1, followed by a concatenate network.

[0247] For example, in the embodiment of the present application, as shown in FIG10 , Represents the reconstructed value of the residual of the l-th layer node, represents the reconstructed value of the latent variable of the l-th layer node, Represents the predicted value of the latent variable of the l-th layer node. and After the fusion of the Concatenate network, the reconstruction value of the latent variable of the l-th layer node can be output through the operation of the sparse convolution layer.

[0248] It can be understood that, in the embodiments of the present application, a soft addition network (addition network) is used to replace the original hard addition, which greatly improves the network flexibility.

[0249] It should be noted that, in the embodiments of the present application, the latent variable of the i-th layer node can represent the correlation between the sibling nodes of the i-th layer. In other words, in the present application, the decoder can fully utilize the hierarchical latent variables to capture the correlation of the LIDAR point cloud.

[0250] Step 203: Input the reconstructed value of the latent variable of the i-th layer node into the second decoding network to determine the geometric information of the i-th layer node.

[0251] In an embodiment of the present application, after the decoder determines the reconstructed value of the latent variable of the i-th layer node based on the reconstructed value of the residual of the i-th layer node, the reconstructed value of the latent variable of the i-th layer node can be input into the second decoding network, so as to determine the geometric information of the i-th layer node.

[0252] It can be understood that in the embodiments of the present application, the geometric information of the i-th layer node of the point cloud can represent the geometric information of the i-th layer node, such as the position coordinates of the node, and can also represent the occupation codeword situation of the i-th layer node, that is, the occupancy situation (placeholder information).

[0253] That is to say, in the embodiment of the present application, the geometric information of the i-th layer node includes both the position coordinate information of the node and the occupancy information of the node.

[0254] It can be understood that in an embodiment of the present application, at the decoding end, the input of the second embedding network is the geometric information of the i-th layer node, wherein the geometric information of the node as the input of the second embedding network can simultaneously include the node's position coordinate information and the node's placeholder information; and the geometric information output by the second decoding network mainly includes the node's placeholder information, which may include the node's position coordinate information or may not include the node's position coordinate information.

[0255] It can be seen that in the embodiment of the present application, the embedding network needs to refer to both the placeholder information and the position coordinate information of the node, while the decoding network focuses on generating the placeholder information.

[0256] It should be noted that in an embodiment of the present application, before the geometric information of the i-th layer node is determined by the second decoding network, the second embedding network from the i-1th layer network to the first layer network has completed the output of the corresponding embedded features. At the same time, the i-1th layer network to the first layer network has completed the output of the reconstructed values ​​of the corresponding latent variables.

[0257] Furthermore, in an embodiment of the present application, the input of the second decoding network of the i-th layer network may include: the embedding features of the i-2th layer nodes output by the second embedding network of the i-2th layer network, the embedding features of the i-1th layer nodes output by the second embedding network of the i-1th layer network, and the reconstructed values ​​of the latent variables of the i-1th layer nodes obtained by the i-1th layer network; the input of the second decoding network of the i-th layer network may also include the reconstructed values ​​of the latent variables of the i-th layer nodes.

[0258] It should be noted that, in an embodiment of the present application, when the decoder generates the geometric information of the i-th layer node based on the reconstructed value of the latent variable of the i-th layer node, it can first input the reconstructed value of the latent variable of the i-th layer node, the embedded feature of the i-2 layer node, the embedded feature of the i-1 layer node, and the reconstructed value of the latent variable of the i-1 layer node into the second decoding network of the i-th layer network, so as to generate the geometric information of the i-th layer node.

[0259] It can be understood that in the embodiment of the present application, based on the second decoding network, the probability parameters of the i-th layer nodes can be determined first; then the geometric information of the i-th layer nodes can be further generated based on the i-th layer probability parameters.

[0260] That is to say, in an embodiment of the present application, the embedding features of the i-2th layer nodes, the embedding features of the i-1th layer nodes, and the reconstruction values ​​of the latent variables of the i-1th layer nodes can be obtained first. Among them, the embedding features of the i-2th layer nodes can be obtained by the second embedding network of the two layers of networks below the current layer, that is, the embedding features of the i-2th layer nodes can be output by the second embedding network in the i-2th layer network; the embedding features of the i-1th layer nodes can be obtained by the second embedding network of the layer below the current layer, that is, the embedding features of the i-1th layer nodes can be output by the second embedding network in the i-1th layer network; the reconstruction values ​​of the latent variables of the i-1th layer nodes can be obtained by the network of the layer below the current layer.

[0261] It should be noted that, in an embodiment of the present application, the second decoding network may include a sparse convolutional network, a deconvolution layer, an initial residual network, a feature fusion network, and a ReLU activation function.

[0262] Exemplarily, in an embodiment of the present application, as shown in Figure 11, the network structure of the second decoding network can be a concatenate network followed by two sparse convolution layers with a convolution kernel size of 3 and a stride of 1, followed by a self-decoder, such as Binary AE.

[0263] For example, in an embodiment of the present application, as shown in FIG12 , represents the reconstructed value of the latent variable of the l-th layer node, represents the reconstructed value of the latent variable of the l-1th layer node, e (l-1) represents the embedding features of the l-1th layer node output by the second embedding network, e (l-2) Represents the embedding features of the l-2th layer nodes output by the second embedding network. e (l-1) , e (l-2) After that, you can use the Concatenate network to e (l-1), e (l-2) After splicing, a two-layer sparse convolutional network is used to predict the probability p of each node in the lth layer. (l) , and finally generate x through arithmetic encoder, such as Binary AE, entropy decoding (l) .

[0264] Furthermore, in an embodiment of the present application, after determining the geometric information of the i-th layer node, the decoder can continue to input the geometric information of the i-th layer node into the second embedding network, so as to determine the embedding features of the i-th layer node.

[0265] Furthermore, in an embodiment of the present application, the second embedding network may be an occupancy embedding network. The second embedding network may be used to map octree nodes from 256-dimensional one-hot vectors to a 16-dimensional continuous feature space, such that similar occupancy conditions in the space are also similar in the feature space.

[0266] It should be noted that, in an embodiment of the present application, the second embedding network may include a sparse convolutional network.

[0267] For example, in an embodiment of the present application, the network structure of the second embedding network can be composed of a multi-layer sparse convolutional network. For example, as shown in FIG3 , the network structure of the second embedding network can be a three-layer sparse convolutional network.

[0268] Furthermore, in an embodiment of the present application, after the decoder inputs the geometric information of the i-th layer node in the point cloud into the second embedding network, it can output the embedded features of the i-th layer node. The embedded features of the i-th layer node can determine the occupancy of the i-th layer node after being mapped to the feature space.

[0269] That is to say, in the embodiment of the present application, the embedded features of the i-th layer nodes can determine the occupancy of the i-th layer nodes.

[0270] For example, in the embodiment of the present application, as shown in FIG4 , x (l) Represents the geometric information of the l-th layer node, e (l) Represents the embedding features of the l-th layer node output by the second embedding network. For the l-th layer node, the decoder can convert the geometric information x of the l-th layer node into (l) Input into the second embedding network, so that the embedding feature e of the corresponding l-th layer node can be obtained through the second embedding network (l) .

[0271] Furthermore, in an embodiment of the present application, FIG18 is a second schematic diagram of an implementation flow of the decoding method proposed in an embodiment of the present application. As shown in FIG18 , after determining the reconstructed value of the latent variable of the i-th layer node based on the reconstructed value of the residual of the i-th layer node, that is, after step 202, the method for the decoder to perform decoding processing may further include the following steps:

[0272] Step 204: Input the reconstructed value of the latent variable of the i-th layer node into the second decoding network to determine the predicted value of the latent variable of the i+1-th layer node.

[0273] In an embodiment of the present application, after the decoder determines the reconstructed value of the latent variable of the i-th layer node based on the reconstructed value of the residual of the i-th layer node, it can further input the reconstructed value of the latent variable of the i-th layer node into the second decoding network, so as to determine the predicted value of the latent variable of the i+1-th layer node.

[0274] It should be noted that in an embodiment of the present application, before the predicted value of the latent variable of the i+1th layer node is generated by the second decoding network, the second embedding network from the i-th layer network to the first layer network has completed the output of the corresponding embedded features, and at the same time, the i-1th layer network to the first layer network has completed the output of the reconstructed value of the corresponding latent variable.

[0275] Furthermore, in an embodiment of the present application, the input of the second decoding network of the i-th layer network may include: the embedding features of the i-1th layer nodes output by the second embedding network of the i-1th layer network, and the reconstructed values ​​of the latent variables of the i-1th layer nodes obtained by the i-1th layer network; the input of the second decoding network of the i-th layer network may also include the reconstructed values ​​of the latent variables of the i-th layer nodes and the i-th embedding features.

[0276] It should be noted that, in an embodiment of the present application, when the decoder determines the predicted value of the latent variable of the i+1th layer node based on the reconstructed value of the latent variable of the i-th layer node, the reconstructed value of the latent variable of the i-th layer node, the embedded feature of the i-th layer node, the embedded feature of the i-1th layer node, and the reconstructed value of the latent variable of the i-1th layer node can be input into the second decoding network of the i-th layer network, so as to determine the predicted value of the latent variable of the i+1th layer node.

[0277] That is, in an embodiment of the present application, the embedded features of the i-1th layer node and the reconstructed values ​​of the latent variables of the i-1th layer node can be obtained first. The embedded features of the i-1th layer node can be obtained by the second embedding network of the next layer network of the current layer, that is, the embedded features of the i-1th layer node can be output by the second embedding network in the i-1th layer network; the reconstructed values ​​of the latent variables of the i-1th layer node can be obtained by the network of the next layer of the current layer.

[0278] It should be noted that, in an embodiment of the present application, the second decoding network may include a sparse convolutional network, a deconvolution layer, an initial residual network, a feature fusion network, and a ReLU activation function.

[0279] Exemplarily, in an embodiment of the present application, as shown in Figure 14, the network structure of the second decoding network can be a concatenate network followed by a sparse convolution layer with a convolution kernel size of 2 and a stride of 2, followed by three initial residual networks, and then a sparse convolution layer with a convolution kernel size of 3 and a stride of 1.

[0280] For example, in an embodiment of the present application, as shown in FIG15 , represents the reconstructed value of the latent variable of the l-th layer node, represents the reconstructed value of the latent variable of the l-1th layer node, e (l) represents the embedding features of the lth layer node output by the second embedding network, e (l-1) Represents the embedding features of the l-1th layer node output by the second embedding network. e (l) , e (l-1) After that, you can use the Concatenate network to e (l-1) , e (l-2) After splicing, it passes through a deconvolution layer with a convolution kernel size of 2 and a step size of 2, and then passes through the initial residual network to obtain the predicted value of the latent variable of the l+1 layer node

[0281] It can be seen that in an embodiment of the present application, for the second decoding network, two different branches can be output based on different inputs, one branch is the geometric information of the i-th layer node of the octree, and the other branch is the predicted value of the latent variable of the i+1-th layer node of the octree.

[0282] It can be understood that in an embodiment of the present application, the predicted value of the latent variable of the i+1th layer node of the octree can be input into the i+1th layer network to determine the reconstructed value of the latent variable of the i+1th layer node.

[0283] Furthermore, in an embodiment of the present application, for the first layer, after the decoder determines the reconstructed value of the latent variable of the first layer node using the reconstructed value of the residual of the first layer node obtained by decoding the code stream; the reconstructed value of the latent variable of the first layer node and the preset embedded feature can be output to the second decoding unit of the first layer network, and then the geometric information of the first layer node and the predicted value of the latent variable of the second layer node can be output respectively.

[0284] For example, in an embodiment of the present application, the first layer may be a layer in the octree where all leaf nodes are empty. That is, when constructing an octree for a point cloud, the last layer of the octree that cannot be further divided is the first layer.

[0285] For example, in the embodiments of the present application, the first layer may also be a layer in the octree that is divided into the smallest unit, such as a 1x1x1 smallest unit block. In other words, when constructing an octree for a point cloud, the last layer of the octree that is divided into the smallest unit block is the first layer.

[0286] Furthermore, in an embodiment of the present application, the decoder may adopt an entropy codec in the style of a factorized variational auto-encoder to perform residual r (l) compression.

[0287] Furthermore, in the embodiment of the present application, at the decoding end, the code stream finally received can be composed of the code stream corresponding to at least one layer of the multi-layer network. For one layer of the network, such as the lth layer network, the corresponding code stream includes the lth layer node generated by the factored variational autoencoder. The code stream, the geometric information x of the l-th layer node generated by the arithmetic encoder (l) Code stream and structure information.

[0288] In summary, the decoding method proposed in the above steps 201 to 204 can rely on the octree structure, use the constructed embedding network and decoding network, introduce latent variables to capture the correlation between sibling nodes in the same layer of the octree, and obtain higher compression performance; at the same time, parallel encoding and decoding processing is performed in units of octree levels, which can effectively shorten the encoding and decoding time, greatly improve the encoding and decoding efficiency, and further improve the compression performance.

[0289] The embodiment of the present application provides a decoding method. At the decoding end, the decoder includes a second decoding network. The decoder decodes the code stream and determines the reconstructed value of the residual of the i-th layer node obtained by dividing the point cloud according to the octree; wherein i is greater than 2; the reconstructed value of the latent variable of the i-th layer node is determined based on the reconstructed value of the residual of the i-th layer node; the reconstructed value of the latent variable of the i-th layer node is input into the second decoding network to determine the geometric information of the i-th layer node. It can be seen that in the embodiment of the present application, when encoding and decoding the point cloud, based on the constructed octree structure, latent variables can be used to characterize the spatial correlation between nodes of the same level in the octree, and ultimately higher compression performance can be obtained; at the same time, based on the constructed network, parallel encoding and decoding processing is performed on the nodes of the same level of the octree, thereby improving the encoding and decoding efficiency and further improving the compression performance.

[0290] Based on the above embodiments, the decoding method proposed in the embodiments of this application can be understood as an end-to-end LIDAR point cloud deep entropy model, or as an end-to-end self-supervised dynamic point cloud compression technology based on deep learning. By relying on an octree structure, introducing latent variables to capture the correlation between octree sibling nodes, and constructing a network using sparse convolution, the method achieves optimal results in a lossless entropy model for LIDAR point clouds.

[0291] At the same time, the decoding method proposed in the embodiment of the present application can improve the encoding and decoding efficiency of LIDAR point clouds. Among them, the embodiment of the present application has a lower encoding and decoding time. This is because the encoding and decoding of each node in the embodiment of the present application is processed in parallel on a layer-by-layer basis and the network is constructed using efficient sparse convolution. Therefore, the encoding and decoding time is much lower than that of common encoding and decoding solutions.

[0292] It should be noted that the network for executing the decoding method proposed in the embodiment of the present application can make full use of hierarchical latent variables to capture the correlation of LIDAR point clouds, and can also use residual coding to decompose and compress latent variables. Furthermore, a soft addition / subtraction network can be used to replace the original hard addition and subtraction, thereby improving the flexibility of the network.

[0293] Furthermore, in an embodiment of the application, FIG19 is a schematic diagram of the overall framework for executing the decoding method. As shown in FIG19 , the decoder can be provided with a multi-layer network, wherein at least one layer of the multi-layer network can include a second embedding network (embedding) and a second decoding network (decoder). For example, each layer is composed of a decoding network (decoder) and an embedding network (embedding).

[0294] It can be understood that, in the embodiment of the present application, the lth layer network in the multi-layer network can perform parallel decoding processing on the geometric information of the lth layer nodes.

[0295] As shown in the figure, x (l) Represents the geometric information of the l-th layer node, e (l) represents the embedding feature of the lth layer node after occupying the embedding network, f (l) represents the latent variable of the l-th layer node, represents the reconstructed value of the l-th layer residual, represents the reconstructed value of the latent variable of the l-th layer node, Represents the predicted value of the latent variable of the l-th layer node. The + symbol in the figure represents the soft additive network (additive network).

[0296] For example, in an embodiment of the present application, as shown in the figure, the network set in the decoder can be composed of a 5-layer network. Among them, for the l-th layer network, the geometric information x of the l-th layer node (l) By embedding the network, we get the occupancy embedding e (l) (Embedded features). The decoding network is divided into two paths, generating the current layer x of the octree respectively. (l) , and the predicted value of the latent variable of the previous layer node

[0297] It should be noted that, in the embodiment of the present application, for the first layer (the 1st to 5th layers as shown in the figure), the decoder may first determine the reconstructed value of the latent variable of the first layer node Then the reconstructed value of the latent variable of the first layer node and the preset embedding feature e (l-6) Output to the second decoding unit of the first layer network, and then output the geometric information of the first layer nodes and the predicted value of the latent variable of the second layer nodes respectively

[0298] Furthermore, in an embodiment of the present application, as shown in the figure, the input of the second decoding network of the l-1 layer network may include: the embedded features of the l-3 layer nodes output by the second embedding network of the l-3 layer network, the embedded features of the l-2 layer nodes output by the second embedding network of the l-2 layer network, and the reconstructed values ​​of the latent variables of the l-2 layer nodes obtained by the l-2 layer network; the input of the second decoding network of the l-1 layer network may also include the reconstructed values ​​of the latent variables of the l-1 layer nodes. Accordingly, when the decoder generates the geometric information of the l-1 layer nodes, it can first input the reconstructed values ​​of the latent variables of the l-1 layer nodes, the embedded features of the l-3 layer nodes, the embedded features of the l-2 layer nodes, and the reconstructed values ​​of the latent variables of the l-2 layer nodes into the second decoding network of the l-1 layer network, thereby generating the geometric information of the l-1 layer nodes.

[0299] It should be noted that, in an embodiment of the present application, when the decoder determines the predicted value of the latent variable of the l-th layer node, it can input the reconstructed value of the latent variable of the l-1th layer node, the embedded features of the l-1th layer node, the embedded features of the l-2th layer node, and the reconstructed value of the latent variable of the l-2th layer node into the second decoding network of the l-1th layer network, so as to determine the predicted value of the latent variable of the l-th layer node.

[0300] Furthermore, in an embodiment of the present application, as shown in the figure, after the second decoding network of the l-1 layer network outputs the predicted value of the latent variable of the l-layer node, the predicted value of the latent variable of the l-layer node can be used in the subsequent decoding processing of the l-layer node, and the addition network can be used to determine the reconstructed value of the latent variable of the l-layer node.

[0301] Exemplarily, in an embodiment of the present application, the decoding network (decoder) is used to generate the current layer x of the octree (l) , and the predicted value of the latent variable of the previous layer node Among them, one branch includes: obtaining e (l-1) , e (l-2) After that, you can use the Concatenate network to e (l-1) , e (l-2) After splicing, a two-layer sparse convolutional network is used to predict the probability p of each node in the lth layer. (l) , and finally generate x through arithmetic encoder and entropy decoding (l) Another branch includes: obtaining e (l) , e (l-1) After that, you can use the Concatenate network to e (l-1) , e (l-2) After splicing, it passes through a deconvolution layer with a convolution kernel size of 2 and a step size of 2, and then passes through the initial residual network to obtain the predicted value of the latent variable of the l+1 layer node

[0302] Exemplarily, in an embodiment of the present application, an embedding network can be used to map octree nodes from 256-dimensional one-hot vectors to a 16-dimensional continuous feature space, so that occupancy conditions that are similar in space are also similar in the feature space.

[0303] For example, in an embodiment of the present application, at least one layer of the multi-layer network provided by the decoder may further include an addition network, wherein a soft addition network (addition network) is used instead of the original hard addition network, which greatly improves the network flexibility.

[0304] To sum up, the decoding method proposed in the embodiment of the present application can rely on the octree structure, use the constructed embedding network, encoding network and decoding network, introduce latent variables to capture the correlation between brother nodes in the same layer of the octree, and obtain higher compression performance; at the same time, parallel encoding and decoding processing is performed in units of octree levels, which can effectively shorten the encoding and decoding time, greatly improve the encoding and decoding efficiency, and further improve the compression performance.

[0305] The embodiment of the present application provides a decoding method. At the decoding end, the decoder includes a second decoding network. The decoder decodes the code stream and determines the reconstructed value of the residual of the i-th layer node obtained by dividing the point cloud according to the octree; wherein i is greater than 2; the reconstructed value of the latent variable of the i-th layer node is determined based on the reconstructed value of the residual of the i-th layer node; the reconstructed value of the latent variable of the i-th layer node is input into the second decoding network to determine the geometric information of the i-th layer node. It can be seen that in the embodiment of the present application, when encoding and decoding the point cloud, based on the constructed octree structure, latent variables can be used to characterize the spatial correlation between nodes of the same level in the octree, and ultimately higher compression performance can be obtained; at the same time, based on the constructed network, parallel encoding and decoding processing is performed on the nodes of the same level of the octree, thereby improving the encoding and decoding efficiency and further improving the compression performance.

[0306] Based on the above embodiment, in another embodiment of the present application, based on the same inventive concept as the above embodiment, FIG20 is a schematic diagram of the composition structure of the encoder. As shown in FIG20 , the encoder 10 may include: a first determining unit 11, an encoding unit 12, and a generating unit 13; wherein,

[0307] The first determining unit 11 is configured to input the geometric information of the i-th layer node obtained by dividing the point cloud according to the octree into a first embedding network to determine the embedding features of the i-th layer node; wherein i is greater than 2; input the embedding features of the i-th layer node into an encoding network to determine the latent variables of the i-th layer node; determine the reconstructed value of the residual of the i-th layer node according to the latent variables of the i-th layer node, and determine the reconstructed value of the latent variable of the i-th layer node according to the reconstructed value of the residual of the i-th layer node;

[0308] The encoding unit 12 is configured to write the reconstructed value of the residual of the i-th layer node into the bitstream;

[0309] The generating unit 13 is configured to input the reconstructed value of the latent variable of the i-th layer node into the first decoding network to generate a code stream of the geometric information of the i-th layer node.

[0310] It is understood that in this embodiment, a "unit" can be a portion of a circuit, a portion of a processor, a portion of a program or software, etc., and can also be a module or a non-modular system. Furthermore, the various components in this embodiment can be integrated into a single processing unit, or each unit can exist physically separately, or two or more units can be integrated into a single unit. The aforementioned integrated units can be implemented in the form of hardware or software functional modules.

[0311] If the integrated unit is implemented as a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, or the portion that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method described in this embodiment. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0312] Therefore, an embodiment of the present application provides a computer-readable storage medium, which is applied to the encoder 10. The computer-readable storage medium stores a computer program, and when the computer program is executed by the first processor, it implements the method described in any one of the aforementioned embodiments.

[0313] Based on the composition of the above-mentioned encoder 10 and the computer-readable storage medium, Figure 21 is a second schematic diagram of the composition structure of the encoder. As shown in Figure 21, the encoder 10 may include: a first memory 14 and a first processor 15, a first communication interface 16 and a first bus system 17. The first memory 14, the first processor 15, and the first communication interface 16 are coupled together through the first bus system 17. It can be understood that the first bus system 17 is used to achieve connection and communication between these components. In addition to the data bus, the first bus system 17 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, various buses are labeled as the first bus system 17 in Figure 9. Among them,

[0314] The first communication interface 16 is used for receiving and sending signals during the process of sending and receiving information with other external network elements;

[0315] The first memory 14 is used to store a computer program that can be run on the first processor;

[0316] The first processor 15 is used to input the geometric information of the i-th layer node obtained by dividing the point cloud according to the octree into the first embedding network when running the computer program, and determine the embedding features of the i-th layer node; wherein i is greater than 2; input the embedding features of the i-th layer node into the encoding network to determine the latent variables of the i-th layer node; determine the reconstructed value of the residual of the i-th layer node according to the latent variables of the i-th layer node, and determine the reconstructed value of the latent variable of the i-th layer node according to the reconstructed value of the residual of the i-th layer node; input the reconstructed value of the latent variable of the i-th layer node into the first decoding network to generate a code stream of the geometric information of the i-th layer node.

[0317] It is understood that the first memory 14 in the embodiment of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DRRAM). The first memory 14 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0318] The first processor 15 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits or software instructions in the first processor 15. The above-mentioned first processor 15 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the first memory 14. The first processor 15 reads the information in the first memory 14 and, in conjunction with its hardware, completes the steps of the above method.

[0319] It is to be understood that these embodiments described in the present application can be implemented with hardware, software, firmware, middleware, microcode or its combination.For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (Application Specific Integrated Circuits, ASIC), digital signal processor (Digital Signal Processing, DSP), digital signal processing equipment (DSP Device, DSPD), programmable logic device (Programmable Logic Device, PLD), field programmable gate array (Field-Programmable Gate Array, FPGA), general-purpose processor, controller, microcontroller, microprocessor, other electronic units for performing functions described in the present application or its combination.For software implementation, the technology described in the present application can be realized by the module (such as process, function etc.) that performs functions described in the present application. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0320] Optionally, as another embodiment, the first processor 15 is further configured to execute any one of the methods described in the foregoing embodiments when running the computer program.

[0321] FIG22 is a schematic diagram of the first structure of a decoder. As shown in FIG22 , the decoder 20 may include: a decoding unit 21 and a second determining unit 22; wherein,

[0322] The decoding unit 21 is configured to decode the code stream;

[0323] The second determination unit 22 is configured to determine the reconstructed value of the residual of the i-th layer node obtained by dividing the point cloud according to the octree; wherein i is greater than 2; determine the reconstructed value of the latent variable of the i-th layer node based on the reconstructed value of the residual of the i-th layer node; input the reconstructed value of the latent variable of the i-th layer node into the second decoding network to determine the geometric information of the i-th layer node.

[0324] It is understood that in this embodiment, a "unit" can be a portion of a circuit, a portion of a processor, a portion of a program or software, etc., and can also be a module or a non-modular system. Furthermore, the various components in this embodiment can be integrated into a single processing unit, or each unit can exist physically separately, or two or more units can be integrated into a single unit. The aforementioned integrated units can be implemented in the form of hardware or software functional modules.

[0325] If the integrated unit is implemented as a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, or the portion that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method described in this embodiment. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0326] Therefore, an embodiment of the present application provides a computer-readable storage medium, which is applied to the decoder 20. The computer-readable storage medium stores a computer program, and when the computer program is executed by the first processor, it implements the method described in any one of the aforementioned embodiments.

[0327] Based on the composition of the above-mentioned decoder 20 and the computer-readable storage medium, Figure 23 is a second schematic diagram of the composition structure of the decoder. As shown in Figure 23, the decoder 20 may include: a second memory 23 and a second processor 24, a second communication interface 25 and a second bus system 26. The second memory 23 and the second processor 24, and the second communication interface 25 are coupled together through the second bus system 26. It can be understood that the second bus system 26 is used to realize the connection and communication between these components. In addition to the data bus, the second bus system 26 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, various buses are labeled as the second bus system 26 in Figure 11. Among them,

[0328] The second communication interface 25 is used for receiving and sending signals during the process of sending and receiving information with other external network elements;

[0329] The second memory 23 is used to store a computer program that can be run on the second processor;

[0330] The second processor 24 is used to decode the code stream when running the computer program, determine the reconstructed value of the residual of the i-th layer node obtained by dividing the point cloud according to the octree; wherein i is greater than 2; determine the reconstructed value of the latent variable of the i-th layer node based on the reconstructed value of the residual of the i-th layer node; input the reconstructed value of the latent variable of the i-th layer node into the second decoding network to determine the geometric information of the i-th layer node.

[0331] It is understood that the second memory 23 in the embodiment of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DRRAM). The second memory 23 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0332] The second processor 24 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits or software instructions in the second processor 24. The second processor 24 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the second memory 23. The second processor 24 reads the information in the second memory 23 and, in conjunction with its hardware, completes the steps of the above method.

[0333] It is to be understood that these embodiments described in the present application can be implemented with hardware, software, firmware, middleware, microcode or its combination.For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (Application Specific Integrated Circuits, ASIC), digital signal processor (Digital Signal Processing, DSP), digital signal processing equipment (DSP Device, DSPD), programmable logic device (Programmable Logic Device, PLD), field programmable gate array (Field-Programmable Gate Array, FPGA), general-purpose processor, controller, microcontroller, microprocessor, other electronic units for performing functions described in the present application or its combination.For software implementation, the technology described in the present application can be realized by the module (such as process, function etc.) that performs functions described in the present application. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0334] The embodiments of the present application provide an encoder and a decoder. When encoding and decoding a point cloud, based on the constructed octree structure, latent variables can be used to characterize the spatial correlation between nodes of the same level in the octree, thereby ultimately achieving higher compression performance. At the same time, parallel encoding and decoding processing is performed on nodes of the same level of the octree based on the constructed network, thereby achieving improved encoding and decoding efficiency and further enhancing compression performance.

[0335] It should be noted that, in the embodiments of the present application, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0336] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0337] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0338] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0339] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0340] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims. Industrial Applicability

[0341] An embodiment of the present application provides a coding and decoding method, an encoder, a decoder, and a storage medium. At the encoding end, the encoder includes a first embedding network, an encoding network, and a first decoding network. The encoder inputs the geometric information of the i-th layer node obtained by dividing the point cloud according to the octree into the first embedding network to determine the embedding features of the i-th layer node; wherein i is greater than 2; the embedding features of the i-th layer node are input into the encoding network to determine the latent variables of the i-th layer node; the reconstructed value of the residual of the i-th layer node is determined according to the latent variable of the i-th layer node, and the reconstructed value of the latent variable of the i-th layer node is determined according to the reconstructed value of the residual of the i-th layer node; the reconstructed value of the latent variable of the i-th layer node is input into the first decoding network to generate a code stream of the geometric information of the i-th layer node. At the decoding end, the decoder includes a second decoding network. The decoder decodes the code stream and determines the reconstructed value of the residual of the i-th layer node obtained by dividing the point cloud according to the octree; wherein i is greater than 2; the reconstructed value of the latent variable of the i-th layer node is determined based on the reconstructed value of the residual of the i-th layer node; the reconstructed value of the latent variable of the i-th layer node is input into the second decoding network to determine the geometric information of the i-th layer node. It can be seen that in the embodiment of the present application, when encoding and decoding the point cloud, based on the constructed octree structure, latent variables can be used to characterize the spatial correlation between nodes of the same level in the octree, and ultimately higher compression performance can be obtained; at the same time, based on the constructed network, parallel encoding and decoding processing is performed on the nodes of the same level of the octree, thereby improving the encoding and decoding efficiency and further improving the compression performance.

Claims

1. A coding method, applied to an encoder, wherein the encoder includes a first embedding network, an encoding network, and a first decoding network, the method comprising: Inputting the geometric information of the i-th layer node obtained by dividing the point cloud according to the octree into the first embedding network to determine the embedding features of the i-th layer node; wherein i is greater than 2; Inputting the embedded features of the i-th layer nodes into the encoding network to determine the latent variables of the i-th layer nodes; Determining a reconstructed value of the residual of the i-th layer node according to the latent variable of the i-th layer node, and determining a reconstructed value of the latent variable of the i-th layer node according to the reconstructed value of the residual of the i-th layer node; The reconstructed value of the latent variable of the i-th layer node is input into the first decoding network to generate a code stream of the geometric information of the i-th layer node.

2. The method according to claim 1, wherein The method further comprises: The encoder is provided with a multi-layer network, wherein at least one layer of the multi-layer network includes the first embedding network, the encoding network, and the first decoding network.

3. The method according to claim 2, wherein: The i-th layer network in the multi-layer network performs parallel encoding processing on the geometric information of the i-th layer nodes.

4. The method according to claim 3, wherein: The method further comprises: The input of the encoding network of the i-th layer network includes the latent variables of the i+1-th layer nodes output by the encoding network of the i+1-th layer network.

5. The method according to claim 4, wherein The method further comprises: The embedded features of the i-th layer nodes and the latent variables of the i+1-th layer nodes are input into the encoding network of the i-th layer network to determine the latent variables of the i-th layer nodes.

6. The method according to any one of claims 3 to 4, wherein: The method further comprises: Determining the residual of the i-th layer node according to the predicted value of the latent variable of the i-th layer node and the latent variable of the i-th layer node; The residual of the i-th layer node is quantized to determine a reconstructed value of the residual of the i-th layer node.

7. The method according to claim 6, wherein: The method further comprises: The reconstructed value of the latent variable of the i-th layer node is determined according to the reconstructed value of the residual of the i-th layer node and the predicted value of the latent variable of the i-th layer node.

8. The method according to claim 6, wherein: The encoder further includes a subtraction network, and the method further includes: The predicted value of the latent variable of the i-th layer node and the latent variable of the i-th layer node are input into the subtraction network to determine the residual of the i-th layer node.

9. The method according to claim 7, wherein: The encoder further includes a summing network, and the method further includes: The reconstructed value of the residual of the i-th layer node and the predicted value of the latent variable of the i-th layer node are input into the additive network to determine the reconstructed value of the latent variable of the i-th layer node.

10. The method according to claim 3, wherein: The method further comprises: The input of the first decoding network of the i-th layer network includes: the embedding features of the i-2th layer node output by the first embedding network of the i-2th layer network, the embedding features of the i-1th layer node output by the first embedding network of the i-1th layer network, and the reconstructed value of the latent variable of the i-1th layer node obtained by the i-1th layer network.

11. The method according to claim 10, wherein: The method further comprises: The reconstructed value of the latent variable of the i-th layer node is input into the first decoding network of the i-th layer network to determine the predicted value of the latent variable of the i+1-th layer node.

12. The method according to claim 11, wherein The method further comprises: The reconstructed value of the latent variable of the i-th layer node, the embedded feature of the i-th layer node, the embedded feature of the i-1-th layer node, and the reconstructed value of the latent variable of the i-1-th layer node are input into the first decoding network of the i-th layer network to determine the predicted value of the latent variable of the i+1-th layer node.

13. The method according to claim 10, wherein: The method further comprises: The reconstructed values ​​of the latent variables of the i-th layer nodes, the embedded features of the i-2 layer nodes, the embedded features of the i-1 layer nodes, and the reconstructed values ​​of the latent variables of the i-1 layer nodes are input into the first decoding network of the i-th layer network to generate a code stream of the geometric information of the i-th layer nodes.

14. The method according to claim 13, wherein The method further comprises: Based on the first decoding network, probability parameters of the i-th layer nodes are determined, and a code stream of geometric information of the i-th layer nodes is generated according to the probability parameters of the i-th layer nodes.

15. The method according to claim 3, wherein The method further comprises: For the first-layer nodes obtained by the octree division, quantizing the latent variables of the first-layer nodes output by the encoding network of the first-layer network to determine the reconstructed values ​​of the latent variables of the first-layer nodes; The reconstructed values ​​of the latent variables of the first-layer nodes and the preset embedded features are output to the first decoding network of the first-layer network, and the code stream of the geometric information of the first-layer nodes and the predicted values ​​of the latent variables of the second-layer nodes are output.

16. The method according to claim 3, wherein The encoding network includes a sparse convolutional network, a ReLU activation function, an initial residual network, and a feature fusion network.

17. The method according to claim 3, wherein: The first embedding network includes a sparse convolutional network.

18. The method according to claim 3, wherein The first decoding network includes a sparse convolutional network, a deconvolution layer, an initial residual network, a feature fusion network, a ReLU activation function, and a deconvolution layer.

19. A decoding method, applied to a decoder, wherein the decoder includes a second decoding network, the method comprising: Decode the bitstream and determine the reconstructed value of the residual of the i-th layer node obtained by dividing the point cloud according to the octree; where i is greater than 2; Determining a reconstructed value of the latent variable of the i-th layer node according to the reconstructed value of the residual of the i-th layer node; The reconstructed value of the latent variable of the i-th layer node is input into the second decoding network to determine the geometric information of the i-th layer node.

20. The method according to claim 19, wherein The decoder further includes a second embedding network, and the method further includes: The geometric information of the i-th layer nodes is input into the second embedding network to determine the embedding features of the i-th layer nodes.

21. The method according to claim 20, wherein The method further comprises: The decoder is provided with a multi-layer network, wherein at least one layer of the multi-layer network includes the second embedding network and the second decoding network.

22. The method according to claim 21, wherein The i-th layer network in the multi-layer network performs decoding processing on the i-th layer node.

23. The method according to claim 22, wherein The method further comprises: The reconstructed value of the latent variable of the i-th layer node is determined according to the reconstructed value of the residual of the i-th layer node and the predicted value of the latent variable of the i-th layer node.

24. The method according to claim 23, wherein The decoder further includes an adding network, and the method further includes: The reconstructed value of the residual of the i-th layer node and the predicted value of the latent variable of the i-th layer node are input into the additive network to determine the reconstructed value of the latent variable of the i-th layer node.

25. The method according to claim 22, wherein The method further comprises: The input of the second decoding network of the i-th layer network includes: the embedding features of the i-2th layer node output by the second embedding network of the i-2th layer network, the embedding features of the i-1th layer node output by the second embedding network of the i-1th layer network, and the reconstructed value of the latent variable of the i-1th layer node obtained by the i-1th layer network.

26. The method according to claim 25, wherein The method further comprises: The reconstructed values ​​of the latent variables of the i-th layer nodes, the embedded features of the i-2 layer nodes, the embedded features of the i-1 layer nodes, and the reconstructed values ​​of the latent variables of the i-1 layer nodes are input into the second decoding network of the i-th layer network to determine the geometric information of the i-th layer nodes.

27. The method according to claim 26, wherein The method further comprises: Based on the second decoding network, probability parameters of the i-th layer nodes are determined, and geometric information of the i-th layer nodes is generated according to the probability parameters of the i-th layer nodes.

28. The method according to claim 25, wherein The method further comprises: The reconstructed value of the latent variable of the i-th layer node is input into the second decoding network to determine the predicted value of the latent variable of the i+1-th layer node.

29. The method according to claim 28, wherein The method further comprises: The reconstructed value of the latent variable of the i-th layer node, the embedded feature of the i-th layer node, the embedded feature of the i-1-th layer node, and the reconstructed value of the latent variable of the i-1-th layer node are input into the second decoding network of the i-th layer network to determine the predicted value of the latent variable of the i+1-th layer node.

30. The method of claim 22, wherein: The method further comprises: For the first-layer nodes obtained by octree division, the reconstructed values ​​of the latent variables of the first-layer nodes and the preset embedded features are output to the second decoding network of the first-layer network, and the geometric information of the first-layer nodes and the predicted values ​​of the latent variables of the second-layer nodes are output.

31. The method according to claim 22, wherein The second embedding network includes a sparse convolutional network.

32. The method of claim 22, wherein: The second decoding network includes a sparse convolutional network, a deconvolution layer, an initial residual network, a feature fusion network, a ReLU activation function, and a deconvolution layer.

33. An encoder, comprising a first determining unit, an encoding unit, and a generating unit; wherein: The first determining unit is configured to input the geometric information of the i-th layer node obtained by dividing the point cloud according to the octree into a first embedding network to determine the embedding features of the i-th layer node; wherein i is greater than 2; input the embedding features of the i-th layer node into an encoding network to determine the latent variables of the i-th layer node; determine the reconstructed value of the residual of the i-th layer node according to the latent variables of the i-th layer node, and determine the reconstructed value of the latent variable of the i-th layer node according to the reconstructed value of the residual of the i-th layer node; The encoding unit is configured to write the reconstructed value of the residual of the i-th layer node into the bitstream; The generating unit is configured to input the reconstructed value of the latent variable of the i-th layer node into the first decoding network to generate a code stream of the geometric information of the i-th layer node.

34. An encoder comprising a first memory and a first processor; wherein: The first memory is used to store a computer program that can be run on the first processor; The first processor is configured to execute the method according to any one of claims 1 to 18 when running the computer program.

35. A decoder comprising a decoding unit and a second determining unit; wherein: The decoding unit is configured to decode the code stream; The second determination unit is configured to determine the reconstructed value of the residual of the i-th layer node obtained by dividing the point cloud according to the octree; wherein i is greater than 2; determine the reconstructed value of the latent variable of the i-th layer node based on the reconstructed value of the residual of the i-th layer node; input the reconstructed value of the latent variable of the i-th layer node into the second decoding network to determine the geometric information of the i-th layer node.

36. A decoder comprising a second memory and a second processor; wherein: The second memory is used to store a computer program that can be run on the second processor; The second processor is configured to execute the method according to any one of claims 19 to 32 when running the computer program.

37. A computer-readable storage medium, wherein: The computer-readable storage medium stores a computer program, and when the computer program is executed, the method according to any one of claims 1 to 18 or the method according to any one of claims 19 to 32 is implemented.

Images