Decoding method, encoding method, training method, decoder and encoder

Abstract

Provided in the embodiments of the present application are a decoding method, an encoding method, a training method, a decoder and an encoder. The decoding method comprises: decoding a code stream of a point cloud, and determining first tensor data, wherein the first tensor data comprises a feature value, on at least one first channel, of attribute information of the point cloud, which attribute information has been subjected to feature extraction; on the basis of the first tensor data, determining second tensor data by using a first neural network, wherein the first neural network and a second neural network that is used by an encoder are used for forming a generator of a generative adversarial network, the second tensor data comprises a feature value, on at least one second channel, of reconstructed attribute information of the point cloud, and the at least one first channel is different from the at least one second channel; and on the basis of the second tensor data, determining the reconstructed attribute information of the point cloud. By means of the decoding method, when both the calculation complexity and the memory occupation of a decoder are taken into account, point cloud attribute information of a good quality can be reconstructed, i.e., the decoding performance of the decoder can be improved.

Description

Decoding methods, encoding methods, training methods, decoders, and encoders Technical Field

[0001] This application relates to the field of encoding and decoding technology, and more specifically, to decoding methods, encoding methods, training methods, decoders, and encoders. Background Technology

[0002] Digital video compression technology mainly compresses massive amounts of digital video data to facilitate transmission and storage.

[0003] However, with the surge in internet videos and people's increasing demands for video clarity, although existing digital video compression standards can achieve video decompression, there is still a need to pursue better digital video decompression technologies to reduce the bandwidth and traffic pressure of digital video transmission, thereby improving the decoding performance of decoders.

[0004] Summary of the Invention

[0005] This application provides a decoding method, an encoding method, a training method, a decoder, and an encoder, which can improve the decoding performance of the decoder.

[0006] In a first aspect, embodiments of this application provide a decoding method, including:

[0007] Decode the bitstream of the point cloud to determine the first tensor data; the first tensor data includes the feature values ​​of the point cloud's attribute information in at least one first channel after feature extraction;

[0008] Based on the first tensor data, a second tensor data is determined using a first neural network; the first neural network and the second neural network used by the encoder are used to form a generator for a generative adversarial network; the second tensor data includes feature values ​​of the reconstructed attribute information of the point cloud in at least one second channel; the at least one first channel is different from the at least one second channel;

[0009] Based on the second tensor data, the reconstruction attribute information of the point cloud is determined.

[0010] Secondly, embodiments of this application provide an encoding method, including:

[0011] Based on the attribute information of the point cloud, the third tensor data of the point cloud is determined; the third tensor data includes the feature values ​​of the attribute information of the point cloud in at least one second channel.

[0012] Based on the third tensor data, the first tensor data is determined using a second neural network; the second neural network and the first neural network used by the decoder are used to form a generator for a generative adversarial network; the first tensor data includes the feature values ​​of the point cloud's attribute information after feature extraction in at least one first channel; the at least one first channel is different from the at least one second channel;

[0013] Based on the first tensor data, the bitstream of the point cloud is determined.

[0014] Thirdly, embodiments of this application provide a method for training a generative adversarial network, including:

[0015] Based on the attribute information of the point cloud, the third tensor data of the point cloud is determined; the third tensor data includes the feature values ​​of the attribute information of the point cloud in at least one second channel.

[0016] Based on the third tensor data, the first tensor data is determined using the second neural network included in the generator of the generative adversarial network; the first tensor data includes the feature values ​​of the point cloud's attribute information after feature extraction in at least one first channel; the at least one first channel is different from the at least one second channel;

[0017] Based on the first tensor data, the generator uses a first neural network to determine the second tensor data; the second tensor data includes the feature values ​​of the reconstructed attribute information of the point cloud in at least one second channel;

[0018] Based on the second tensor data, the reconstruction attribute information of the point cloud is determined;

[0019] Based on the attribute information of the point cloud and the reconstructed attribute information of the point cloud, the generative adversarial network is trained using the discriminator in the generative adversarial network.

[0020] Fourthly, embodiments of this application provide a decoding method, including:

[0021] Decode the bitstream of the point cloud to determine the first tensor data; the first tensor data includes the feature values ​​of the point cloud's attribute information in at least one first channel after feature extraction;

[0022] Based on the first tensor data, a second tensor data is determined using a first neural network; the first neural network and the second neural network used by the encoder are used to form the generator of the generative adversarial network; the second tensor data includes the feature values ​​of the reconstructed attribute information of the point cloud in at least one second channel; the at least one first channel is different from the at least one second channel; the generative adversarial network is trained according to the training method involved in the third aspect mentioned above;

[0023] Based on the second tensor data, the reconstruction attribute information of the point cloud is determined.

[0024] Fifthly, embodiments of this application provide an encoding method, including:

[0025] Based on the attribute information of the point cloud, the third tensor data of the point cloud is determined; the third tensor data includes the feature values ​​of the attribute information of the point cloud in at least one second channel.

[0026] Based on the third tensor data, the first tensor data is determined using a second neural network; the second neural network and the first neural network used by the decoder are used to form the generator of the generative adversarial network; the first tensor data includes the feature values ​​of the point cloud's attribute information after feature extraction in at least one first channel; the at least one first channel is different from the at least one second channel; the generative adversarial network is trained according to the training method involved in the third aspect mentioned above;

[0027] Based on the first tensor data, the bitstream of the point cloud is determined.

[0028] Sixthly, embodiments of this application provide a decoder, including:

[0029] A decoding unit is used to decode the bitstream of the point cloud and determine the first tensor data; the first tensor data includes the feature values ​​of the attribute information of the point cloud in at least one first channel after feature extraction;

[0030] A first determining unit is configured to determine second tensor data based on the first tensor data using a first neural network; the first neural network and the second neural network used by the encoder are used to form a generator of a generative adversarial network; the second tensor data includes feature values ​​of the reconstructed attribute information of the point cloud in at least one second channel; the at least one first channel is different from the at least one second channel;

[0031] The second determining unit is used to determine the reconstruction attribute information of the point cloud based on the second tensor data.

[0032] In a seventh aspect, embodiments of this application provide an encoder, including:

[0033] The first determining unit is used to determine the third tensor data of the point cloud based on the attribute information of the point cloud; the third tensor data includes the feature values ​​of the attribute information of the point cloud in at least one second channel.

[0034] The second determining unit is used to determine the first tensor data based on the third tensor data using a second neural network; the second neural network and the first neural network used by the decoder are used to form a generator for a generative adversarial network; the first tensor data includes the feature values ​​of the point cloud's attribute information after feature extraction in at least one first channel; the at least one first channel is different from the at least one second channel;

[0035] The third determining unit is used to determine the bitstream of the point cloud based on the first tensor data.

[0036] Eighthly, embodiments of this application provide a decoder, including:

[0037] Processor, adapted to implement computer instructions; and,

[0038] A computer-readable storage medium storing computer instructions adapted for loading by a processor and performing the methods described in any of the first and fourth aspects or their implementations mentioned above.

[0039] In one implementation, there are one or more processors and one or more memories.

[0040] In one implementation, the computer-readable storage medium may be integrated with the processor, or the computer-readable storage medium may be disposed separately from the processor.

[0041] Ninthly, embodiments of this application provide an encoder, including:

[0042] Processor, adapted to implement computer instructions; and,

[0043] A computer-readable storage medium storing computer instructions adapted for loading by a processor and performing the methods described in the third aspect or its implementations above.

[0044] In one implementation, there are one or more processors and one or more memories.

[0045] In one implementation, the computer-readable storage medium may be integrated with the processor, or the computer-readable storage medium may be disposed separately from the processor.

[0046] In a tenth aspect, embodiments of this application provide an encoder, including:

[0047] Processor, adapted to implement computer instructions; and,

[0048] A computer-readable storage medium storing computer instructions adapted for loading by a processor and performing the methods described in any of the second and fifth aspects or their implementations mentioned above.

[0049] In one implementation, there are one or more processors and one or more memories.

[0050] In one implementation, the computer-readable storage medium may be integrated with the processor, or the computer-readable storage medium may be disposed separately from the processor.

[0051] Eleventhly, embodiments of this application provide a computer-readable storage medium storing computer instructions that, when read and executed by a processor of a computer device, cause the computer device to perform any one of the first to fifth aspects or their respective implementations described above.

[0052] In a twelfth aspect, embodiments of this application provide a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform any of the first to fifth aspects or their respective implementations described above.

[0053] Ninthly, embodiments of this application provide a bitstream, which is the bitstream described in any of the first and fourth aspects mentioned above or in their implementations, or the bitstream generated by the method described in any of the second and fifth aspects mentioned above or in their implementations.

[0054] Based on the above technical solutions, this application uses the first neural network used by the decoder and the second neural network used by the encoder to form a generator of a generative adversarial network. The first neural network is used to convert the first tensor data determined by decoding the bitstream of the point cloud into second tensor data for reconstructing attribute information. This fully combines the advantages of autoencoders and generative adversarial networks, which can not only control the computational complexity and memory usage of the decoder, but also reconstruct high-quality point cloud attribute information. That is, the decoding method provided by the embodiments of this application can reconstruct high-quality point cloud attribute information while taking into account the computational complexity and memory usage of the decoder, thus improving the decoding performance of the decoder. Attached Figure Description

[0055] Figure 1 is an example of a point cloud image provided in an embodiment of this application.

[0056] Figure 2 is a magnified view of a portion of the point cloud image shown in Figure 1.

[0057] Figure 3 is an example of a point cloud image with six viewing angles provided in an embodiment of this application.

[0058] Figure 4 is an example of the data storage format of point cloud images provided in the embodiments of this application.

[0059] Figure 5 is an example of the PCAC encoding and decoding process provided in the embodiments of this application.

[0060] Figure 6 is a schematic flowchart of the decoding method provided in the embodiments of this application.

[0061] Figure 7 is a schematic structural diagram of the generative adversarial network provided in an embodiment of this application.

[0062] Figure 8 is a schematic flowchart of the encoding method provided in the embodiments of this application.

[0063] Figure 9 is a schematic diagram of voxelization provided in an embodiment of this application.

[0064] Figure 10 is a schematic flowchart of the training method for generative adversarial networks provided in an embodiment of this application.

[0065] Figure 11 is a schematic block diagram of the decoder provided in an embodiment of this application.

[0066] Figure 12 is a schematic block diagram of the encoder provided in an embodiment of this application.

[0067] Figure 13 is a schematic block diagram of a training apparatus for a generative adversarial network provided in an embodiment of this application.

[0068] Figure 14 is a schematic block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0069] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0070] A point cloud is a set of discrete points in space that are randomly distributed and represent the spatial structure and surface properties of a three-dimensional object or scene. Figures 1 and 2 show a three-dimensional point cloud image and a magnified view, respectively, showing that the surface of the point cloud is composed of densely distributed points.

[0071] Two-dimensional images convey information at each pixel, thus eliminating the need to record its position. However, the distribution of points in a point cloud in three-dimensional space is random and irregular, requiring the recording of each point's position in space to fully represent a point cloud. Similar to two-dimensional images, each point in a point cloud has corresponding attribute information, typically RGB color values ​​reflecting the object's color. For point clouds, in addition to color, the attribute information for each point can also be a reflectance value, reflecting the object's surface material. Each point in a point cloud can include geometric and attribute information. Geometric information refers to the point's Cartesian three-dimensional coordinates. Attribute information for each point can include, but is not limited to, at least one of the following: color information, material information, and laser reflection intensity information. Color information can be from any color space. For example, color information can be Red, Green, Blue (RGB) information. Alternatively, color information can be luminance / chromaticity (YCbCr, YUV) information. Where Y represents luminance (Luma), Cb(U) represents the blue chromaticity component, and Cr(V) represents the red chromaticity component. Each point in a point cloud has the same number of attribute information. For example, each point in a point cloud has two attribute information: color and laser reflection intensity. Alternatively, each point in a point cloud may have three attribute information: color, material, and laser reflection intensity.

[0072] Point cloud images can have multiple viewing angles; for example, as shown in Figure 3, a point cloud image can have six viewing angles.

[0073] The data storage format of a point cloud image consists of a header and a data portion. The header includes the data format, data representation type, total number of points in the point cloud, and the content represented by the point cloud. For example, as shown in Figure 4, the header of the data storage format of a point cloud image may include the following: ".ply" format, represented by ASCII code, with a total of 207242 points, and each point has three-dimensional position information (xyz) and three-dimensional color information (rgb).

[0074] Point clouds can flexibly and conveniently express the spatial structure and surface properties of three-dimensional objects or scenes. Since point clouds are obtained by directly sampling real objects, they can provide a strong sense of realism while ensuring accuracy. Therefore, they are widely used, including virtual reality games, computer-aided design, geographic information systems, automatic navigation systems, digital cultural heritage, free-viewpoint broadcasting, three-dimensional immersive remote presentation, and three-dimensional reconstruction of biological tissues and organs.

[0075] Point clouds can be broadly categorized into two types based on application scenarios: machine-perceived point clouds and human-perceived point clouds. Applications of machine-perceived point clouds include, but are not limited to, autonomous navigation systems, real-time inspection systems, geographic information systems, visual sorting robots, and disaster relief robots. Applications of human-perceived point clouds include, but are not limited to, digital cultural heritage, free-viewpoint broadcasting, 3D immersive communication, and 3D immersive interaction. Correspondingly, point clouds can be classified into dense and sparse point clouds based on their acquisition method; they can also be classified into static and dynamic point clouds based on their acquisition approach. More specifically, they can be divided into three types: static point clouds, dynamic point clouds, and dynamically acquired point clouds. For static point clouds, the object is stationary, and the device acquiring the point cloud is also stationary. For dynamic point clouds, the object is moving, but the device acquiring the point cloud is stationary. For dynamically acquired point clouds, the device acquiring the point cloud is moving.

[0076] Point cloud acquisition methods include, but are not limited to, computer generation, 3D laser scanning, and 3D photogrammetry. Computers can generate point clouds of virtual 3D objects and scenes; 3D laser scanning can obtain point clouds of static real-world 3D objects or scenes, acquiring millions of point clouds per second; 3D photogrammetry can obtain point clouds of dynamic real-world 3D objects or scenes, acquiring tens of millions of point clouds per second. Specifically, point clouds of object surfaces can be acquired using acquisition devices such as photoelectric radar, lidar, laser scanners, and multi-view cameras. Point clouds obtained based on laser measurement principles can include the 3D coordinate information and laser reflection intensity of points. Point clouds obtained based on photogrammetry principles can include the 3D coordinate information and color information of points. Point clouds obtained by combining laser measurement and photogrammetry principles can include the 3D coordinate information, laser reflection intensity, and color information of points. These technologies reduce the cost and time required for point cloud data acquisition and improve data accuracy. For example, in the medical field, point clouds of biological tissues and organs can be obtained using magnetic resonance imaging (MRI), computed tomography (CT), and electromagnetic positioning information. These technologies have reduced the cost and time required to acquire point clouds and improved data accuracy. This revolution in point cloud data acquisition has made it possible to acquire massive amounts of point cloud data. However, with the growth of application demands, the processing of massive 3D point cloud data is encountering bottlenecks due to limitations in storage space and transmission bandwidth.

[0077] Taking a point cloud video with a frame rate (FPS) of 30 as an example, each frame contains 700,000 points. Each point in each frame has coordinate information (xyz, float) and color information (RGB, uchar). Therefore, the data size of a 10-second point cloud video is approximately 0.7 million × (4 bytes × 3 + 1 byte × 3) × 30 fps × 10s = 3.15 GB. In contrast, a 10-second 2D video with a YUV sampling format of 4:2:0, a frame rate (FPS) of 24, and a resolution of 1280 × 720 has a data size of approximately 1280 × 720 × 12 bits × 24 frames × 10s ≈ 0.33 GB. A 10-second two-view 3D video has a data size of approximately 0.33 × 2 = 0.66 GB. It is clear that the data size of a point cloud video far exceeds that of 2D and 3D videos of the same length. Therefore, in order to better achieve data management, save server storage space, and reduce the transmission traffic and transmission time between the server and the client, point cloud compression has become a key issue in promoting the development of the point cloud industry.

[0078] Point cloud compression typically employs separate compression of point cloud geometric information and attribute information. At the encoding end, the geometric encoder first encodes the point cloud geometric information, and then the reconstructed geometric information is input as supplementary information into the attribute encoder to assist in point cloud attribute compression. At the decoding end, the geometric decoder first decodes the point cloud geometric information, and then the decoded geometric information is input as supplementary information into the attribute decoder to assist in point cloud attribute compression. The entire encoder-decoder system consists of preprocessing / post-processing, geometric encoding / decoding, and attribute encoding / decoding.

[0079] Point clouds can be encoded and decoded using various types of encoding and decoding frameworks. As examples, the encoding / decoding framework can be the Geometry Point Cloud Compression (G-PCC) or Video Point Cloud Compression (V-PCC) framework provided by the Moving Picture Experts Group (MPEG), or the AVS-PCC or Point Cloud Compression Reference Platform (PCRM) framework provided by the Audio Video Standard (AVS) task force. The G-PCC framework can be used to compress first-type static point clouds and third-type dynamically acquired point clouds, while the V-PCC framework can be used to compress second-type dynamic point clouds. The G-PCC framework is also known as TMC13, and the V-PCC framework is also known as TMC2. Both G-PCC and AVS-PCC can be used to compress static sparse point clouds, and their encoding frameworks are largely the same.

[0080] Of course, other encoding and decoding frameworks can also be used to compress point clouds, and this application does not make specific limitations on this.

[0081] The following section introduces the encoding and decoding framework for Point Cloud Attribute Compression (PCAC).

[0082] Figure 5 is an example of the PCAC encoding and decoding process provided in the embodiments of this application.

[0083] As shown in Figure 5, the main encoder extracts features from the attribute information of the original input point cloud to obtain the latent representation of the attribute information, and then restores it into a reconstructed point cloud (PC) in the main decoder. The main encoder can be a variational autoencoder (VAE) framework.

[0084] The master encoder and master decoder are the core components of PCAC's encoding and decoding. Both the master encoder and decoder use convolutional layers and activation layers as analysis transformations. The convolutional layers can be sparse convolution (SCONv) layers, and the activation layers can be rectified linear units (ReLU).

[0085] The specific encoding process of the main encoder is as follows:

[0086] (1) Input voxelized point cloud data, and the main encoder uses convolutional layers for feature extraction. The main encoder consists of 6 convolutional layers and 2 activation layers. Optionally, all convolutional layers use 3×3×3 convolutional kernels, and except for the first convolutional layer which has 64 channels, the remaining convolutional layers have 128 channels.

[0087] (2) After each convolutional layer, downsampling is performed using an activation layer. The activation layer shares parameters with the convolutional layer; for example, the maximum value of all points within each voxel can be calculated and used as the feature representation of that voxel.

[0088] (3) Repeat steps (1) and (2) until a very small feature map is obtained.

[0089] (4) The last layer outputs a feature map, which is then mapped to the latent space through a fully connected layer. This latent space is the compressed point cloud data.

[0090] After the main encoder completes the encoding process, the main decoder's specific decoding process is as follows:

[0091] (1) Input the compressed point cloud data into the transposed convolutional layer for deconvolution, and finally map it back to the last feature map.

[0092] (2) Upsample the deconvolutioned feature map to obtain a larger feature map. All deconvolution layers share parameters with the encoder in the encoding process.

[0093] (3) Repeat steps (1) and (2) until a feature map of the same size as the original input point cloud is obtained.

[0094] Specifically, the encoding and decoding processes of the main encoder and main decoder are as follows:

[0095] First, the master encoder extracts features from the attribute information of the original input point cloud to obtain a latent representation of the attribute information. Then, it quantizes (Q) the latent representation of the attribute information output by the master encoder to obtain a quantized latent representation of the attribute information. This quantized latent representation is then input into an arithmetic encoder (AE) for arithmetic encoding to obtain a bitstream. After transmission, the arithmetic decoder (AD), connected to the master decoder, receives the bitstream output by the AE. The AD performs arithmetic decoding on the input bitstream and uses the latent representation obtained from the AD decoding by the master decoder to decode the latent representation, outputting a reconstructed point cloud.

[0096] Furthermore, for sparse PCAC, a context model determined by the super-prior encoder and super-prior decoder can be introduced. This context model is used to assist the AE or AD in entropy encoding the quantized latent representation of the main encoder output.

[0097] The super-prior encoder and super-prior decoder are auxiliary components of the PCAC encoding and decoding. Similar to the main encoder and main decoder, both the super-prior encoder and super-prior decoder use convolutional layers and activation layers as analysis transforms. The convolutional layers can be SCONv layers, and the activation layers can be ReLU layers. The encoding and decoding process of the super-prior encoder and super-prior decoder is as follows:

[0098] (1) After the main encoder maps the input point cloud data to the latent space, feature extraction is performed on the latent representation output by the main encoder to obtain a high-dimensional vector as the super prior latent representation.

[0099] (2) When the main decoder maps the latent representation extracted by the main encoder back to the original input point cloud size and shape, the super prior decoder is used to decode the generated super prior latent representation to obtain the context model; and the context model is coupled with the autoregressive neighborhood; the autoregressive neighborhood is used to determine the probability distribution of the current point and the information related to the points that have been encoded and decoded.

[0100] Specifically, the super-prior encoder extracts features again from the latent representation of the attribute information output by the main encoder, and uses the extracted latent representation as input to the super-prior encoder. The super-prior latent representation of the attribute information output by the super-prior encoder is quantized to obtain the quantized super-prior latent representation of the attribute information. This quantized super-prior latent representation is then input to the AE (Advanced Aspect-Oriented Array) for arithmetic encoding to obtain the super-prior bitstream. After transmission, the AD (Advanced Digital Array) connected to the super-prior decoder receives the super-prior bitstream, performs arithmetic decoding on the input super-prior bitstream, and uses the super-prior decoder to decode the super-prior latent representation obtained from the AD decoding to output the context model. Then, an activation layer is used to process the quantized latent representation output by the main encoder, and the resulting neighborhood information is coupled with the context model encoded by the super-prior encoder to obtain coupled features. Next, an activation layer and two convolutional layers are used to process the coupled features to obtain parameters for assisting the arithmetic encoding (AE) and the arithmetic decoding (AD). For example, the parameters for assisting the AE and AD may include u and σ. u represents the probability model of the point cloud, and σ represents the parameters used to couple the context model and neighborhood information output by the super-prior decoder. The parameters used for the AE (arithmetic encoding of the super-prior latent representation) and the AD (arithmetic decoding of the super-prior bitstream) can be the parameters output by the Factorized Entropy Model.

[0101] Therefore, it can be seen that because the arithmetic encoding and decoding parts of the master encoder and master decoder combine the super-prior context model and neighborhood information, they can accurately estimate the conditional entropy probability. Specifically, at each time t, the symbol sequence encoded at time t-1 is used as neighborhood information, and the high-dimensional vector generated by the super-prior encoder is used as conditional information. Then, the Laplace distribution is used to model the conditional probability distribution, and the parameters of the Laplace distribution are estimated by the maximum likelihood estimation method. Finally, these parameters are used for arithmetic encoding to encode the latent features output by the master encoder into a bitstream.

[0102] However, in the above process, because it requires the use of neighborhood information, PCAC makes strong assumptions about the geometry and structure of the point cloud data, and has high requirements for the resolution and density of the input point cloud. This leads to a decrease in compression performance with sparse point cloud data. Furthermore, because PCAC requires significant computational resources and storage space to train the model and store latent representations, it faces issues such as memory limitations and computation time constraints. Finally, PCAC was tested using only a small number of point cloud datasets in the experiments, and its performance and generalizability require further verification and improvement.

[0103] In view of this, this application provides a decoding method, an encoding method, a decoder, and an encoder that utilize a Generative Adversarial Network (GAN) to decode a bitstream. Specifically, the GAN mainly consists of a generator and a discriminator. The generator uses convolutional layers to compress the input voxelized point cloud data into a bitstream, and then uses transposed convolutions to decode the bitstream to obtain a reconstructed point cloud. The discriminator can return a discrimination result based on the reconstructed point cloud and the input point cloud. The discrimination result is used to continuously train the generator until the reconstructed point cloud is as consistent as possible with the input point cloud.

[0104] Figure 6 is a schematic flowchart of the decoding method 100 provided in an embodiment of this application.

[0105] As shown in Figure 6, the decoding method 100 may include:

[0106] S110, the decoder decodes the bitstream of the point cloud to determine the first tensor data; the first tensor data includes the feature values ​​of the point cloud's attribute information in at least one first channel after feature extraction.

[0107] For example, the dimension of the first tensor data may include at least one of the following: the dimension in which the at least one first channel is located, the dimension in which the voxel index in the point cloud is located, and the dimension in which the attribute information of the voxel in the point cloud is located.

[0108] Tensors are the natural representation of high-dimensional data. A tensor represents a multidimensional array, and its dimension is called its order. Generally, a 0th-order tensor represents a scalar, such as s = 123; a 1st-order tensor represents a vector, such as v = [1, 2, 3]; a 2nd-order tensor represents a matrix, such as m = [[1, 2, 3], [4, 5, 6]]; and an nth-order tensor represents a multidimensional array, typically resulting in [...].

[0109] For example, the attribute information of the point cloud can be the attribute information of the voxels in the point cloud. That is, the attribute information of the voxels obtained after the encoder divides the point cloud into voxels.

[0110] It is worth noting that the attribute information of the point cloud can be any kind of attribute information.

[0111] For example, the attribute information of the point cloud includes, but is not limited to, at least one of the following: color information, material information, laser reflection intensity information, and normal vector information.

[0112] Color information can be from any color space. For example, color information can be Red Green Blue (RGB) information. Alternatively, color information can be luminance / chrominance (YCbCr, YUV) information, where Y represents luminance (Luma), Cb(U) represents the blue chrominance component, and Cr(V) represents the red chrominance component. Each point in the point cloud has the same number of attribute information. For example, each point in the point cloud has two attribute information: color and laser reflection intensity. Or, each point in the point cloud has three attribute information: color, material, and laser reflection intensity.

[0113] It is worth noting that the number of the at least one first channel can be any integer, and this application does not impose any specific limitation on it.

[0114] For example, the number of the at least one first channel can be 64, 128 or other values.

[0115] For example, the number of the at least one first channel can be the number of channels used by the last convolutional layer in the second neural network used by the encoder. In this case, it can also be understood that the at least one first channel can be the channel used by the last convolutional layer in the second neural network.

[0116] Assuming the last convolutional layer in the second neural network uses 128 channels, then the number of the at least one first channel is 128. In this case, if the number of voxels in the point cloud is M, then the first data tensor can be implemented as a data tensor of the following form:

[0117] 128×M; M represents the number of voxels in the point cloud.

[0118] Wherein, the eigenvalue x in the first data tensor ij This represents the value of the attribute information of the i-th voxel of the point cloud in the j-th channel after feature extraction.

[0119] It should be understood that the first tensor data is a data format of the attribute information of the point cloud after feature extraction. Its data content is essentially still the value obtained after feature extraction of the original attribute information of the point cloud. The difference between it and the attribute information of the point cloud is that when the original attribute information of the point cloud is converted into the tensor data format, its corresponding channels are different from (usually less than) the at least one first channel.

[0120] S120, the decoder determines second tensor data based on the first tensor data using a first neural network; the first neural network and the second neural network used by the encoder are used to form a generator for a generative adversarial network; the second tensor data includes feature values ​​of the reconstructed attribute information of the point cloud in at least one second channel; the at least one first channel is different from the at least one second channel.

[0121] In some embodiments, the at least one second channel corresponds to at least one component of the attribute information of the point cloud.

[0122] For example, the at least one component may include at least one of the following: Y component, U component, and V component.

[0123] For example, the at least one component may include at least one of the following: R component, G component, and B component.

[0124] Of course, in other alternative embodiments, the at least one component may also be other types of components, and this application will not specifically limit them.

[0125] Assuming that the at least one second channel includes channels corresponding to the Y component, U component, and V component respectively, then the number of the at least one second channel is 3. In this case, if the number of voxels in the point cloud is M, then the second data tensor can be implemented as a data tensor of the following form:

[0126] 3×M; M represents the number of voxels in the point cloud.

[0127] Among them, the eigenvalues ​​y in the second data tensor ij This represents the value of the reconstructed attribute information of the i-th voxel of the point cloud in the j-th channel (e.g., Y component, U component, or V component).

[0128] It should be understood that the second tensor data is a data format of the reconstructed attribute information of the point cloud, and its data content is essentially still the reconstructed attribute information of the point cloud. That is to say, after the decoder obtains the second tensor data, it can directly convert the second tensor data into the reconstructed attribute information of the point cloud.

[0129] S130, the decoder determines the reconstruction attribute information of the point cloud based on the second tensor data.

[0130] In this embodiment, the first neural network used by the decoder and the second neural network used by the encoder are used to form a generator of a generative adversarial network. This allows the decoder to use the first neural network to convert the first tensor data determined by decoding the bitstream of the point cloud into second tensor data for reconstructing attribute information. This fully combines the advantages of autoencoders and generative adversarial networks, which not only controls the computational complexity and memory usage of the decoder but also reconstructs high-quality point cloud attribute information. In other words, the decoding method provided in this embodiment can reconstruct high-quality point cloud attribute information while taking into account the computational complexity and memory usage of the decoder, thus improving the decoding performance of the decoder.

[0131] The results of testing the scheme provided in this application are explained below with reference to Table 1.

[0132] The technical solution provided in this application uses the MPEG standard test set to evaluate its performance, and additionally uses the sparser point cloud dataset from MVUB for comparative testing with MPEG G-PCC (i.e., TMC13v6 and TMC13v19) and SparsePCAC. Bit Distortion Bit Rate (BD-BR) and Bit Distortion Peak Signal to Noise Ratio (BD-PSNR) are indicators of algorithm or encoding performance. BD-BR represents the change in bit rate of the encoding algorithm provided in this application compared to the original encoding algorithm; an overall negative value indicates improved performance. BD-PSNR represents the change in PSNR of the encoding algorithm provided in this application compared to the original encoding algorithm; an overall negative value indicates improved performance. Further, Y represents luminance (Luma), U represents the blue chroma component, and V represents the red chroma component.

[0133] Table 1

[0134] As shown in Table 1, the "Point cloud" column represents different point cloud categories. BD-BR and BD-PSNR are metrics used to evaluate distortion performance, indicating the performance difference between the proposed method and comparative methods. A negative value for BD-BR represents a performance improvement, while a positive value for BD-PSNR represents a performance improvement. It is evident that, for different point cloud categories, although the decoding method provided in this application exhibits a performance decrease in Y-components and YUV compared to TMC13v19, it shows performance improvements in both Y-components and YUV compared to SparsePCAC and TMC13v6. Furthermore, it is worth noting that, compared to SparsePCAC, the decoding method provided in this application does not require the introduction of a super-prior encoder and super-prior decoder, i.e., it does not need to consider domain information. This not only reduces the requirements for point cloud resolution and density but also reduces the computational complexity and memory usage of the decoder. In other words, it can reconstruct high-quality point cloud attribute information while considering the computational complexity and memory usage of the decoder, thus improving the decoding performance.

[0135] It should be understood that the first tensor data and the second tensor data are essentially representations of the attribute information of the point cloud in different tensor formats. Therefore, in other alternative embodiments, the solution of this application can be equivalently replaced by the following description without introducing tensor data and channels:

[0136] The code stream of the point cloud is decoded to determine the first attribute information of the point cloud; based on the first attribute information, a second attribute information is determined using a first neural network; the first neural network and a second neural network used by the encoder are used to form a generator of a generative adversarial network; based on the second attribute information, the reconstructed attribute information of the point cloud is determined. For example, the second attribute information is the reconstructed attribute information of the point cloud in tensor format. Alternatively, the first attribute information can be the first tensor data, and the second attribute information can be the second tensor data.

[0137] In some embodiments, the first tensor data includes feature values ​​of the attribute information of voxels in the point cloud after feature extraction on at least one first channel, and the second tensor data includes feature values ​​of the reconstructed attribute information of the voxels on at least one second channel.

[0138] The voxels in the point cloud can be coding units obtained by the encoder dividing the point cloud into voxels.

[0139] Assuming the last convolutional layer in the second neural network uses 128 channels, then the number of the at least one first channel is 128. In this case, if the number of voxels in the point cloud is M, then the first data tensor can be implemented as a data tensor of the following form:

[0140] 128×M; M represents the number of voxels in the point cloud.

[0141] Wherein, the eigenvalue x in the first data tensor ij This represents the value of the attribute information of the i-th voxel of the point cloud in the j-th channel after feature extraction.

[0142] Assuming that the at least one second channel includes channels corresponding to the Y component, U component, and V component respectively, then the number of the at least one second channel is 3. In this case, if the number of voxels in the point cloud is M, then the second data tensor can be implemented as a data tensor of the following form:

[0143] 3×M; M represents the number of voxels in the point cloud.

[0144] Among them, the eigenvalues ​​y in the second data tensor ij This represents the value of the reconstructed attribute information of the i-th voxel of the point cloud in the j-th channel (e.g., Y component, U component, or V component).

[0145] For example, the voxel resolutions of different voxels in the point cloud can be different or the same.

[0146] For example, the voxels of the point cloud may include voxels corresponding to multiple point cloud blocks. When the multiple point cloud blocks are of different types, the voxel resolutions of the voxels corresponding to the multiple point cloud blocks are different.

[0147] In one possible implementation, when the plurality of point cloud blocks include dense blocks and sparse blocks, the voxel resolution corresponding to the dense block is greater than or less than the voxel resolution corresponding to the sparse block. When the voxel resolution corresponding to the dense block is less than the voxel resolution corresponding to the sparse block, the attribute information of the sparse block can be preserved as much as possible, thus improving the reconstruction quality of the sparse block. When the voxel resolution corresponding to the dense block is greater than the voxel resolution corresponding to the sparse block, the attribute information of the dense block can be preserved as much as possible, thus improving the reconstruction quality of the sparse block. For example, the voxel resolution corresponding to the dense block can be 16×16×16, and the voxel resolution corresponding to the sparse block can be 8×8×8.

[0148] It is worth noting that the voxels of the point cloud may include voxels corresponding to multiple point cloud blocks. The sizes of the multiple point cloud blocks may be the same or different, and this application does not make specific limitations in this regard.

[0149] Of course, in other alternative embodiments, the voxels of the point cloud may include voxels corresponding to multiple point cloud blocks, and the voxels corresponding to the multiple point cloud blocks have different sizes when the types of the multiple point cloud blocks are different. Alternatively, the types of the multiple point cloud blocks may also include multiple levels determined based on density, which is not specifically limited in this application.

[0150] In one possible implementation, when the plurality of point cloud blocks include dense blocks and sparse blocks, the size of the voxel corresponding to the dense block is larger than or smaller than the size of the voxel corresponding to the sparse block. When the size of the voxel corresponding to the dense block is larger than the size of the voxel corresponding to the sparse block, the attribute information of the sparse block can be preserved as much as possible, thus improving the reconstruction quality of the sparse block. For example, the size of the voxel corresponding to the dense block can be 16×16×16, while the size of the voxel corresponding to the sparse block can be 8×8×8.

[0151] Of course, in other alternative embodiments, the first tensor data includes the feature values ​​of the attribute information of the points in the point cloud after feature extraction on the at least one first channel, and the second tensor data includes the feature values ​​of the reconstructed attribute information of the points in the point cloud on the at least one second channel.

[0152] In some embodiments, S110 may include:

[0153] The decoder dequantizes the tensor data obtained from decoding the bitstream of the point cloud to obtain the first tensor data.

[0154] For example, since the bitstream is obtained by feature extraction, quantization and arithmetic encoding by an encoder, the first tensor data includes the feature values ​​of the attribute information of the voxels in the point cloud after feature extraction, quantization, arithmetic encoding, arithmetic decoding and dequantization on the at least one first channel, and the second tensor data includes the feature values ​​of the reconstructed attribute information of the voxels on the at least one second channel.

[0155] Of course, in other alternative embodiments, the encoder may not need to voxelize the point cloud; that is, the encoder can directly extract features, quantize, and arithmetically encode the attribute information of the points to form a bitstream of the point cloud. In this case, for the decoder, the first tensor data includes the feature values ​​of the attribute information of the points in the point cloud after feature extraction on at least one first channel, and the second tensor data includes the feature values ​​of the reconstructed attribute information of the points in the point cloud on at least one second channel. This application does not impose specific limitations on this.

[0156] In some embodiments, the first neural network includes at least one feature extraction layer, the last feature extraction layer of the at least one feature extraction layer includes a deconvolution layer, and the feature extraction layers other than the last feature extraction layer of the at least one feature extraction layer include deconvolution layers and activation layers.

[0157] Among them, the deconvolution layer, also known as the transposed convolution layer, is used to upsample the input.

[0158] For example, the first neural network includes multiple feature extraction layers. The last feature extraction layer includes a deconvolution layer, and the feature extraction layers other than the last feature extraction layer include multiple deconvolution layers and activation layers. For example, the multiple feature extraction layers can be 3 or other values, and the multiple deconvolution layers can be 2 or other values. This application does not specifically limit them.

[0159] In some embodiments, the position of the deconvolution layer in the first neural network is the same as the position of the convolution layer in the second neural network, and the position of the activation layer in the first neural network is the same as the position of the activation layer in the second neural network.

[0160] For example, the position of the layer can be identified by the layer number (e.g., the number according to the direction of data transmission).

[0161] For example, the deconvolutional layer in the first neural network has the same index as the convolutional layer in the second neural network, and the activation layer in the first neural network has the same index as the activation layer in the second neural network. In other words, if the i-th layer in the first neural network is a deconvolutional layer, then the i-th layer in the second neural network is a convolutional layer; if the i-th layer in the first neural network is an activation layer, then the i-th layer in the second neural network is an activation layer. Or, to put it another way...

[0162] For example, if the deconvolutional layers in the first neural network include layers numbered 1, 2, 4, 5, 7, and 8, and the activation layers in the first neural network include layers numbered 3 and 6, then the convolutional layers in the second neural network include layers numbered 1, 2, 4, 5, 7, and 8, and the activation layers in the second neural network include layers numbered 3 and 6.

[0163] In some embodiments, the parameters of the deconvolution layer in the first neural network are the same as the parameters of the convolution layer in the second neural network.

[0164] For example, the parameters of the i-th deconvolution layer in the first neural network are the same as the parameters of the (k+1-i)-th convolution layer in the second neural network, where k represents the number of deconvolution layers in the first neural network or the number of convolution layers in the second neural network.

[0165] For example, assuming k=6, the parameters of the first deconvolution layer in the first neural network are the same as the parameters of the sixth convolution layer in the second neural network, the parameters of the second deconvolution layer in the first neural network are the same as the parameters of the fifth convolution layer in the second neural network, and so on. The parameters of the sixth deconvolution layer in the first neural network are the same as the parameters of the first convolution layer in the second neural network.

[0166] For example, the parameters of the deconvolution layer in the first neural network may include the convolution kernel and / or the number of channels.

[0167] For example, the convolution kernel of the i-th deconvolution layer in the first neural network is the same as the convolution kernel of the (k+1-i)-th convolution layer in the second neural network. Similarly, the number of channels in the i-th deconvolution layer of the first neural network is the same as the number of channels in the (k+1-i)-th convolution layer of the second neural network.

[0168] Figure 7 is a schematic structural diagram of the generative adversarial network provided in an embodiment of this application.

[0169] As shown in Figure 7, the Generative Adversarial Network (GAN) includes a generator and a discriminator. The generator comprises a second neural network used by the encoder and a first neural network used by the decoder. The generator's input includes attribute information of a point cloud (e.g., a voxelized point cloud), and its output is the reconstructed attribute information of the point cloud (e.g., a reconstructed voxelized point cloud). Specifically, the attribute information of the point cloud is first represented using tensor data. This tensor data is then compressed into a bitstream by the second neural network. The decoder decodes the bitstream to determine the first tensor data, and then uses the first neural network to convert this first tensor data into second tensor data. This second tensor data is used to determine the reconstructed attribute information of the point cloud. The discriminator compares the attribute information of the point cloud with the reconstructed attribute information and returns the result to assist in training the generator until the reconstructed attribute information of the point cloud is as consistent as possible with the attribute information of the point cloud.

[0170] The encoder employs a second neural network consisting of three convolutional layers and two activation layers. The convolutional kernels in the convolutional layers are 9×9×9, 5×5×5, and 5×5×5, and contain 128 channels. The decoder employs a first neural network consisting of three transposed convolutional layers and two activation layers, with the parameters of the second neural network shared with the first neural network. The discriminator is essentially a classification network that distinguishes whether the reconstructed attribute information of the point cloud matches the attribute information of the point cloud. The discriminator's network structure can be consistent with the second neural network, using sparse convolutions to extract features and finally using fully connected layers to output one-dimensional data to represent the true / false state of the reconstructed point cloud data.

[0171] It is worth noting that Figure 7 is merely an example of this application and should not be construed as a limitation of this application.

[0172] For example, since the accompanying diagram is an example diagram given from the perspective of training a generative adversarial network, only the second neural network used by the encoder and the first neural network used by the decoder are shown as examples. However, in the actual encoding and decoding process, the encoder also includes modules for quantizing and arithmetic encoding the second neural network, and the decoder also includes modules for arithmetic decoding and dequantization of the bitstream, and may even include filtering or other modules.

[0173] The decoding method according to the embodiments of this application has been described in detail above from the perspective of the decoder. The encoding method according to the embodiments of this application will be described below from the perspective of the encoder, with reference to FIG8.

[0174] Figure 8 is a schematic flowchart of the encoding method 200 provided in an embodiment of this application. It should be understood that the encoding method 200 can be executed by an encoder.

[0175] As shown in Figure 8, the encoding method 200 may include:

[0176] S210, the encoder determines the third tensor data of the point cloud based on the attribute information of the point cloud; the third tensor data includes the feature values ​​of the attribute information of the point cloud in at least one second channel.

[0177] S220, the encoder determines the first tensor data based on the third tensor data using a second neural network; the second neural network and the first neural network used by the decoder are used to form a generator for a generative adversarial network; the first tensor data includes the feature values ​​of the point cloud's attribute information after feature extraction in at least one first channel; the at least one first channel is different from the at least one second channel.

[0178] S230, the encoder determines the bitstream of the point cloud based on the first tensor data.

[0179] In some embodiments, the at least one second channel corresponds to at least one component of the attribute information of the point cloud.

[0180] In some embodiments, the third tensor data includes feature values ​​of the attribute information of voxels in the point cloud on at least one second channel, and the first tensor data includes feature values ​​of the attribute information of voxels on at least one first channel after feature extraction.

[0181] In some embodiments, S210 may include:

[0182] The point cloud is divided into at least one point cloud block;

[0183] Determine the type of the point cloud block;

[0184] The point cloud block is voxelized based on the resolution corresponding to the type of the point cloud block, and the attribute information of the voxels in the point cloud is determined.

[0185] The third tensor data is determined based on the attribute information of the voxels.

[0186] In this embodiment, the encoder voxels the point cloud blocks based on the resolution corresponding to the type of the point cloud blocks, which can preserve point cloud information as much as possible and reduce the computational complexity of the encoder, thereby improving the encoding performance of the encoder.

[0187] The encoder can pre-train a voxelization module for adaptive voxel resolution. This module can voxelize the input raw point cloud using multiple voxel resolutions (e.g., high and low resolutions) based on the density of the point cloud, resulting in a voxelized point cloud. During the adaptive voxel resolution voxelization process, the input raw point cloud is divided into blocks. For each block, its corresponding label is predicted to indicate which points belong to relatively sparse blocks and which belong to relatively dense blocks. For sparse blocks, voxelization A (e.g., 8×8×8 resolution) can be used; for dense blocks, voxelization B (e.g., 16×16×16 resolution) can be used. As shown in Figure 9, this voxelization module can be implemented as a neural network, consisting of multiple 3D convolutional layers and activation layers. The last layer uses a classification function (e.g., the sigmoid function) to label the output blocks as sparse or dense blocks.

[0188] The loss function of the voxelization module includes a loss for supervising the labeling process of sparse blocks and a loss for supervising the labeling process of dense blocks. The mathematical formula for the standard binary cross entropy (BCE) loss is: Where y represents the real label. The predicted labels generated for the model. Furthermore, combining the above formula with the voxelization module provided in this application, it can be seen that the loss for supervising the labeling process of sparse blocks and the loss for supervising the labeling process of dense blocks can be expressed as: loss sparse =BCE(M sparse GT sparse loss dense =BCE(M dense GT dense ).

[0189] The total loss function is: total loss =α·loss sparse +(1-α)·loss dense .

[0190] Where, loss sparse The loss represents the loss used to supervise the labeling process of sparse blocks. dense The total represents the loss used to monitor the labeling process of dense blocks. loss M represents the loss of the voxelization module (i.e., voxelization loss). sparse GT represents the true label of a sparse block. sparse M represents the predicted label of a sparse block. dense GT represents the true label of dense blocks. dense The predicted label represents the dense block, and α represents the loss. sparse The weight.

[0191] In some embodiments, S230 may include:

[0192] The first tensor data is quantized and encoded to obtain the bitstream.

[0193] In some embodiments, the second neural network includes at least one feature extraction layer, the last feature extraction layer of the at least one feature extraction layer includes a convolutional layer, and the feature extraction layers other than the last feature extraction layer of the at least one feature extraction layer include convolutional layers and activation layers.

[0194] In some embodiments, the position of the convolutional layer in the second neural network is the same as the position of the deconvolutional layer in the first neural network, and the position of the activation layer in the second neural network is the same as the position of the activation layer in the first neural network.

[0195] In some embodiments, the parameters of the convolutional layers in the second neural network are the same as the parameters of the deconvolutional layers in the first neural network.

[0196] It should be understood that the encoding method can be understood as the reverse process of the decoding method. Therefore, the specific scheme of the encoding method 200 can be found in the relevant content of the decoding method 100. For ease of description, this application will not elaborate further.

[0197] The above text describes in detail the encoding and decoding method according to the embodiments of this application from the perspective of decoder and encoder. The following text, with reference to Figure 10, describes the training method of generative adversarial network provided according to the embodiments of this application from the perspective of generative adversarial network.

[0198] Figure 10 is a schematic flowchart of a generative adversarial network training method 300 provided in an embodiment of this application.

[0199] It should be understood that this training method 300 can be executed by any electronic device with data processing capabilities.

[0200] Furthermore, the training method 300 provided in this application can be incorporated into the decoding method 100 or the encoding method 200. For example, the training device uses the training method 300 to train a generative adversarial network; the generative adversarial network includes a generator and a discriminator, the generator includes a first neural network and a second neural network, the decoder can use the first neural network to convert the first tensor data determined by decoding the bitstream of the point cloud into second tensor data for reconstructing attribute information, and correspondingly, the encoder can use the second neural network to convert the third tensor data determined based on the attribute information of the point cloud into first tensor data for determining the bitstream of the point cloud.

[0201] Furthermore, this application does not specifically limit the dataset used for training the generative adversarial network. For example, the training of the generative adversarial network can use the ShapeNet and COCO datasets. Specifically, points are densely sampled on the ShapeNet grid, and their coordinates are randomly rotated and quantized to 8-bit integers. For the color attribute, random images from COCO are projected onto these points, generating a total sample set (e.g., 8500 samples). This set is divided into two parts, one part (e.g., 6000 samples) for training and the other part (e.g., 2500 samples) for testing. To prevent overfitting, additional samples from the ModelNet40 dataset (e.g., 11000 samples) can be added, and used for training and testing in the same proportion as described above.

[0202] As shown in Figure 10, the training method 300 may include:

[0203] S310, Based on the attribute information of the point cloud, determine the third tensor data of the point cloud; the third tensor data includes the feature values ​​of the attribute information of the point cloud in at least one second channel;

[0204] S320, based on the third tensor data, the first tensor data is determined using the second neural network included in the generator of the generative adversarial network; the first tensor data includes the feature values ​​of the attribute information of the point cloud after feature extraction in at least one first channel; the at least one first channel is different from the at least one second channel;

[0205] S330, based on the first tensor data, a second tensor data is determined using a first neural network included in the generator; the second tensor data includes feature values ​​of the reconstructed attribute information of the point cloud in at least one second channel;

[0206] S340, Based on the second tensor data, determine the reconstruction attribute information of the point cloud;

[0207] S350, Based on the attribute information of the point cloud and the reconstructed attribute information of the point cloud, the generative adversarial network is trained using the discriminator in the generative adversarial network.

[0208] In some embodiments, S350 may include:

[0209] Based on the attribute information of the point cloud and the reconstructed attribute information of the point cloud, the distortion value between the attribute information of the point cloud and the reconstructed attribute information of the point cloud is determined.

[0210] Based on the bit rate of the bitstream of the point cloud and the distortion value, the loss value of the generative adversarial network is determined; the bitstream of the point cloud is the bitstream obtained by arithmetic encoding and quantization of the first tensor data.

[0211] The generative adversarial network is trained based on its loss value.

[0212] In some embodiments, the loss value of the generator is determined based on the voxel occupancy information in the point cloud; the loss value of the discriminator is determined based on the attribute information and the reconstructed attribute information of the point cloud; and the distortion value is determined based on the loss value of the generator and the loss value of the discriminator.

[0213] In some embodiments, the loss value of the generator is determined according to the following formula:

[0214] in, This represents the loss value of the generator. In the point cloud, the first... The weight of individual elements, Indicates the first The occupancy information of individual elements, where ξ represents the focusing parameter.

[0215] For example, Indicates the first The occupancy information of the individual element; in other words, the first... Individual occupancy information can be used to determine The value of . For example, the first The occupancy information of individual elements indicates the occupancy time. The value of is 1, the first The occupancy information of individual elements indicates when they are not occupied. The value of is 0. For example, the first... The occupancy information of individual elements indicates the occupancy time. The value of is 0, the first The occupancy information of individual elements indicates when they are not occupied. The value of is 1.

[0216] For example, Used to control the proportion of positive samples in the training set, its value ranges from [0,1]. The appropriate value can generally be selected through cross-validation.

[0217] For example, the value of ξ ranges from [0, +∞), with the aim of reducing the weight of easily classified samples, thereby making the generator focus more on difficult samples during training. When ξ = 0, the loss value of the generator degenerates into cross-entropy loss; the larger ξ is, the greater the penalty for easily classified samples.

[0218] In some embodiments, the loss value of the discriminator is determined according to the following formula:

[0219] in, Let E[·] represent the loss value of the discriminator, E[·] represent the expected value operation, and m represent the attribute information of the point cloud. The probability distribution of the attribute information of the point cloud is represented by n, where n represents the reconstructed attribute information of the point cloud. This represents the probability distribution of the reconstructed attribute information of the point cloud.

[0220] In some embodiments, the distortion value is determined according to the following formula:

[0221] in, This represents the distortion value. This represents the loss value of the discriminator. The weights represent the loss values ​​of the discriminator. This represents the loss value of the generator. The weights represent the loss values ​​of the generator.

[0222] In some embodiments, the loss value of the generative adversarial network is determined according to the following formula:

[0223] in, This represents the loss value of the generative adversarial network. Let λ represent the distortion value, and λ represent the distortion weight. This indicates the bit rate.

[0224] The preferred embodiments of this application have been described in detail above with reference to the accompanying drawings. However, this application is not limited to the specific details of the embodiments described above. Within the scope of the technical concept of this application, various simple modifications can be made to the technical solutions of this application, and these simple modifications all fall within the protection scope of this application. For example, the various specific technical features described in the specific embodiments described above can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, this application will not describe the various possible combinations separately. Furthermore, various different embodiments of this application can also be arbitrarily combined, as long as they do not violate the spirit of this application, and they should also be considered as the content disclosed in this application. It should also be understood that in the various method embodiments of this application, the sequence number of each process mentioned above does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0225] The method embodiments of this application have been described in detail above. The apparatus embodiments of this application will be described in detail below with reference to Figures 11 to 14.

[0226] Figure 11 is a schematic block diagram of the decoder 400 provided in an embodiment of this application.

[0227] As shown in Figure 11, the decoder 400 may include:

[0228] Decoding unit 410 is used to decode the bitstream of the point cloud and determine the first tensor data; the first tensor data includes the feature values ​​of the attribute information of the point cloud in at least one first channel after feature extraction.

[0229] The first determining unit 420 is used to determine second tensor data based on the first tensor data using a first neural network; the first neural network and the second neural network used by the encoder are used to form a generator of a generative adversarial network; the second tensor data includes feature values ​​of the reconstructed attribute information of the point cloud in at least one second channel; the at least one first channel is different from the at least one second channel;

[0230] The second determining unit 430 is used to determine the reconstruction attribute information of the point cloud based on the second tensor data.

[0231] In some embodiments, the at least one second channel corresponds to at least one component of the attribute information of the point cloud.

[0232] In some embodiments, the first tensor data includes feature values ​​of the attribute information of voxels in the point cloud after feature extraction on at least one first channel, and the second tensor data includes feature values ​​of the reconstructed attribute information of the voxels on at least one second channel.

[0233] In some embodiments, the decoding unit 410 is specifically used for:

[0234] The tensor data obtained by decoding the bitstream of the point cloud is dequantized to obtain the first tensor data.

[0235] In some embodiments, the first neural network includes at least one feature extraction layer, the last feature extraction layer of the at least one feature extraction layer includes a deconvolution layer, and the feature extraction layers other than the last feature extraction layer of the at least one feature extraction layer include deconvolution layers and activation layers.

[0236] In some embodiments, the position of the deconvolution layer in the first neural network is the same as the position of the convolution layer in the second neural network, and the position of the activation layer in the first neural network is the same as the position of the activation layer in the second neural network.

[0237] In some embodiments, the parameters of the deconvolution layer in the first neural network are the same as the parameters of the convolution layer in the second neural network.

[0238] Figure 12 is a schematic block diagram of the encoder 500 provided in an embodiment of this application.

[0239] As shown in Figure 12, the encoder 500 may include:

[0240] The first determining unit 510 is used to determine the third tensor data of the point cloud based on the attribute information of the point cloud; the third tensor data includes the feature values ​​of the attribute information of the point cloud in at least one second channel.

[0241] The second determining unit 520 is used to determine the first tensor data based on the third tensor data using a second neural network; the second neural network and the first neural network used by the decoder are used to form a generator of a generative adversarial network; the first tensor data includes the feature values ​​of the point cloud's attribute information after feature extraction in at least one first channel; the at least one first channel is different from the at least one second channel;

[0242] The third determining unit 530 is used to determine the bitstream of the point cloud based on the first tensor data.

[0243] In some embodiments, the at least one second channel corresponds to at least one component of the attribute information of the point cloud.

[0244] In some embodiments, the third tensor data includes feature values ​​of the attribute information of voxels in the point cloud on at least one second channel, and the first tensor data includes feature values ​​of the attribute information of voxels on at least one first channel after feature extraction.

[0245] In some embodiments, the first determining unit 510 is specifically used for:

[0246] The point cloud is divided into at least one point cloud block;

[0247] Determine the type of the point cloud block;

[0248] The point cloud block is voxelized based on the resolution corresponding to the type of the point cloud block, and the attribute information of the voxels in the point cloud is determined.

[0249] The third tensor data is determined based on the attribute information of the voxels.

[0250] In some embodiments, the third determining unit 530 is specifically used for:

[0251] The first tensor data is quantized and encoded to obtain the bitstream.

[0252] In some embodiments, the second neural network includes at least one feature extraction layer, the last feature extraction layer of the at least one feature extraction layer includes a convolutional layer, and the feature extraction layers other than the last feature extraction layer of the at least one feature extraction layer include convolutional layers and activation layers.

[0253] In some embodiments, the position of the convolutional layer in the second neural network is the same as the position of the deconvolutional layer in the first neural network, and the position of the activation layer in the second neural network is the same as the position of the activation layer in the first neural network.

[0254] In some embodiments, the parameters of the convolutional layers in the second neural network are the same as the parameters of the deconvolutional layers in the first neural network.

[0255] Figure 13 is a schematic block diagram of a training device 600 for a generative adversarial network provided in an embodiment of this application.

[0256] As shown in Figure 13, the training device 600 may include:

[0257] The first determining unit 610 is used to determine the third tensor data of the point cloud based on the attribute information of the point cloud; the third tensor data includes the feature values ​​of the attribute information of the point cloud in at least one second channel.

[0258] The second determining unit 620 is used to determine the first tensor data based on the third tensor data and using the second neural network included in the generator of the adversarial generative network; the first tensor data includes the feature values ​​of the attribute information of the point cloud after feature extraction in at least one first channel; the at least one first channel is different from the at least one second channel;

[0259] The third determining unit 630 is used to determine second tensor data based on the first tensor data and using the first neural network included in the generator; the second tensor data includes feature values ​​of the reconstructed attribute information of the point cloud in at least one second channel;

[0260] The fourth determining unit 640 is used to determine the reconstruction attribute information of the point cloud based on the second tensor data;

[0261] Training unit 650 is used to train the generative adversarial network (GAN) based on the attribute information of the point cloud and the reconstructed attribute information of the point cloud, using the discriminator in the GAN.

[0262] In some embodiments, the training unit 650 is specifically used for:

[0263] Based on the attribute information of the point cloud and the reconstructed attribute information of the point cloud, the distortion value between the attribute information of the point cloud and the reconstructed attribute information of the point cloud is determined.

[0264] Based on the bit rate of the bitstream of the point cloud and the distortion value, the loss value of the generative adversarial network is determined; the bitstream of the point cloud is the bitstream obtained by arithmetic encoding and quantization of the first tensor data.

[0265] The generative adversarial network is trained based on its loss value.

[0266] In some embodiments, the training unit 650 is specifically used for:

[0267] Based on the voxel occupancy information in the point cloud, the loss value of the generator is determined;

[0268] Based on the attribute information of the point cloud and the reconstructed attribute information of the point cloud, the loss value of the discriminator is determined;

[0269] The distortion value is determined based on the loss value of the generator and the loss value of the discriminator.

[0270] In some embodiments, the training unit 650 is specifically used for:

[0271] The loss value of the generator is determined according to the following formula:

[0272] in, This represents the loss value of the generator. In the point cloud, the first... The weight of individual elements, Indicates the first The occupancy information of individual elements, where ξ represents the focusing parameter.

[0273] In some embodiments, the training unit 650 is specifically used for:

[0274] The loss value of the discriminator is determined according to the following formula:

[0275] in, Let E[·] represent the loss value of the discriminator, E[·] represent the expected value operation, and m represent the attribute information of the point cloud. The probability distribution of the attribute information of the point cloud is represented by n, where n represents the reconstructed attribute information of the point cloud. This represents the probability distribution of the reconstructed attribute information of the point cloud.

[0276] In some embodiments, the training unit 650 is specifically used for:

[0277] The distortion value is determined according to the following formula:

[0278] in, This represents the distortion value. This represents the loss value of the discriminator. The weights represent the loss values ​​of the discriminator. This represents the loss value of the generator. The weights represent the loss values ​​of the generator.

[0279] In some embodiments, the training unit 650 is specifically used for:

[0280] The loss value of the generative adversarial network is determined according to the following formula:

[0281] in, This represents the loss value of the generative adversarial network. Let λ represent the distortion value, and λ represent the distortion weight. This indicates the bit rate.

[0282] It should be understood that the device embodiments and method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, further details are omitted here. Specifically, the decoder 400 shown in FIG11 can correspond to the corresponding subject in executing method 100 of the embodiments of this application, and the aforementioned and other operations and / or functions of each unit in the decoder 400 are respectively for implementing the corresponding processes in method 100 and other methods. The encoder 500 shown in FIG12 can correspond to the corresponding subject in executing method 200 of the embodiments of this application, that is, the aforementioned and other operations and / or functions of each unit in the encoder 500 are respectively for implementing the corresponding processes in method 200 and other methods. The training device 600 shown in FIG13 can correspond to the corresponding subject in executing method 300 of the embodiments of this application, that is, the aforementioned and other operations and / or functions of each unit in the training device 600 are respectively for implementing the corresponding processes in method 300 and other methods.

[0283] It should also be understood that the various units in the decoder 400, encoder 500, or training device 600 involved in the embodiments of this application can be individually or entirely merged into one or more other units, or some of the units can be further divided into multiple functionally smaller units. This can achieve the same operation without affecting the technical effect of the embodiments of this application. The units mentioned above are based on logical function division. In practical applications, the function of one unit can also be implemented by multiple units, or the function of multiple units can be implemented by one unit. In other embodiments of this application, the decoder 400, encoder 500, or training device 600 may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by multiple units. According to another embodiment of this application, the decoder 400, encoder 500, or training device 600 involved in the embodiments of this application can be constructed and the encoding method, decoding method, or training method of the embodiments of this application can be implemented by running a computer program (including program code) capable of performing the steps involved in the corresponding method on a general-purpose computing device including processing elements and storage elements such as a central processing unit (CPU), random access storage medium (RAM), and read-only storage medium (ROM). The computer program can be recorded on, for example, a computer-readable storage medium, loaded into an electronic device through the computer-readable storage medium, and run therein to implement the corresponding method of the embodiments of this application.

[0284] In other words, the units mentioned above can be implemented in hardware, in software instructions, or in a combination of hardware and software. Specifically, the steps of the method embodiments in this application can be completed by the integrated logic circuits in the processor's hardware and / or by software instructions. The steps of the method disclosed in the embodiments of this application can be directly manifested as being executed by a hardware decoding processor, or executed by a combination of hardware and software in the decoding processor. Optionally, the software can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. This storage medium is located in memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps in the method embodiments mentioned above.

[0285] Figure 14 is a schematic structural diagram of the electronic device 700 provided in an embodiment of this application.

[0286] As shown in Figure 14, the electronic device 700 includes at least a processor 710 and a computer-readable storage medium 720. The processor 710 and the computer-readable storage medium 720 can be connected via a bus or other means. The computer-readable storage medium 720 stores a computer program 721, which includes computer instructions. The processor 710 executes the computer instructions stored in the computer-readable storage medium 720. The processor 710 is the computing and control core of the electronic device 700, and is suitable for implementing one or more computer instructions, specifically for loading and executing one or more computer instructions to achieve a corresponding method flow or function.

[0287] For example, processor 710 may also be referred to as a central processing unit (CPU). Processor 710 may include, but is not limited to: general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, discrete hardware components, etc.

[0288] Exemplarily, the computer-readable storage medium 720 may be a high-speed RAM memory or a non-volatile memory, such as at least one disk storage device; optionally, it may also be at least one computer-readable storage medium located remotely from the aforementioned processor 710. Specifically, the computer-readable storage medium 720 includes, but is not limited to, volatile memory and / or non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0289] For example, the electronic device 700 may be an encoder or encoding framework involved in the embodiments of this application; the computer-readable storage medium 720 stores first computer instructions; the processor 710 loads and executes the first computer instructions stored in the computer-readable storage medium 720 to implement the corresponding steps in the encoding method provided in the embodiments of this application; in other words, the first computer instructions in the computer-readable storage medium 720 are loaded and executed by the processor 710 to implement the corresponding steps, which will not be described again here to avoid repetition.

[0290] For example, the electronic device 700 may be a decoder or decoding framework involved in the embodiments of this application; the computer-readable storage medium 720 stores second computer instructions; the processor 710 loads and executes the second computer instructions stored in the computer-readable storage medium 720 to implement the corresponding steps in the decoding method provided in the embodiments of this application; in other words, the second computer instructions in the computer-readable storage medium 720 are loaded and executed by the processor 710 to implement the corresponding steps, which will not be described again here to avoid repetition.

[0291] For example, the electronic device 700 may be a training device involved in the embodiments of this application; the computer-readable storage medium 720 stores third computer instructions; the processor 710 loads and executes the third computer instructions stored in the computer-readable storage medium 720 to implement the corresponding steps in the training method provided in the embodiments of this application; in other words, the third computer instructions in the computer-readable storage medium 720 are loaded and executed by the processor 710 to implement the corresponding steps, which will not be described again here to avoid repetition.

[0292] According to another aspect of this application, this application also provides an encoding and decoding system, including the encoder and decoder mentioned above.

[0293] According to another aspect of this application, a computer-readable storage medium (Memory) is also provided. This computer-readable storage medium is a memory device in the electronic device 700 for storing programs and data. For example, a computer-readable storage medium 720. It is understood that the computer-readable storage medium 720 here may include both the built-in storage medium in the electronic device 700 and extended storage media supported by the electronic device 700. The computer-readable storage medium provides storage space that stores the operating system of the electronic device 700. Furthermore, this storage space also stores one or more computer instructions suitable for loading and execution by the processor 710. These computer instructions may be one or more computer programs 721 (including program code).

[0294] According to another aspect of this application, this application also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. For example, computer program 721. In this case, the data processing device 700 may be a computer, and the processor 710 reads the computer instructions from the computer-readable storage medium 720. The processor 710 executes the computer instructions, causing the computer to perform the encoding method, decoding method, or training method provided in the various alternative methods described above.

[0295] In other words, when implemented using software, it can be implemented entirely or partially in the form of a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes of the embodiments of this application are run or the functions of the embodiments of this application are implemented. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0296] Those skilled in the art will recognize that the units and process steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0297] Finally, it should be noted that the above content is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A decoding method, characterized in that, include: Decode the bitstream of the point cloud to determine the first tensor data; The first tensor data includes the feature values ​​of the point cloud's attribute information in at least one first channel after feature extraction; Based on the first tensor data, a second tensor data is determined using a first neural network; the first neural network and the second neural network used by the encoder are used to form a generator for a generative adversarial network; the second tensor data includes the feature values ​​of the reconstructed attribute information of the point cloud in at least one second channel; The at least one first channel is different from the at least one second channel; Based on the second tensor data, the reconstruction attribute information of the point cloud is determined.

2. The method according to claim 1, characterized in that, The at least one second channel corresponds to at least one component of the attribute information of the point cloud.

3. The method according to claim 1 or 2, characterized in that, The first tensor data includes the feature values ​​of the voxel attribute information in the point cloud after feature extraction on at least one first channel, and the second tensor data includes the feature values ​​of the reconstructed attribute information of the voxel on at least one second channel.

4. The method according to any one of claims 1 to 3, characterized in that, The decoded point cloud bitstream determines the first tensor data, including: The tensor data obtained by decoding the bitstream of the point cloud is dequantized to obtain the first tensor data.

5. The method according to claim 4, characterized in that, The first neural network includes at least one feature extraction layer, the last feature extraction layer of the at least one feature extraction layer includes a deconvolution layer, and the feature extraction layers other than the last feature extraction layer of the at least one feature extraction layer include deconvolution layers and activation layers.

6. The method according to claim 4, characterized in that, The deconvolutional layer in the first neural network is located at the same position as the convolutional layer in the second neural network, and the activation layer in the first neural network is located at the same position as the activation layer in the second neural network.

7. The method according to claim 4, characterized in that, The parameters of the deconvolution layer in the first neural network are the same as the parameters of the convolution layer in the second neural network.

8. An encoding method, characterized in that, include: Based on the attribute information of the point cloud, the third tensor data of the point cloud is determined; The third tensor data includes the feature values ​​of the point cloud's attribute information in at least one second channel; Based on the third tensor data, the first tensor data is determined using the second neural network; the second neural network and the first neural network used by the decoder are used to form the generator of the generative adversarial network; the first tensor data includes the feature values ​​of the point cloud's attribute information in at least one first channel after feature extraction. The at least one first channel is different from the at least one second channel; Based on the first tensor data, the bitstream of the point cloud is determined.

9. The method according to claim 8, characterized in that, The at least one second channel corresponds to at least one component of the attribute information of the point cloud.

10. The method according to claim 8 or 9, characterized in that, The third tensor data includes the feature values ​​of the attribute information of the voxels in the point cloud on at least one second channel, and the first tensor data includes the feature values ​​of the attribute information of the voxels on at least one first channel after feature extraction.

11. The method according to claim 10, characterized in that, The determination of the third tensor data of the point cloud based on the attribute information of the point cloud includes: The point cloud is divided into at least one point cloud block; Determine the type of the point cloud block; The point cloud block is voxelized based on the resolution corresponding to the type of the point cloud block, and the attribute information of the voxels in the point cloud is determined. The third tensor data is determined based on the attribute information of the voxels.

12. The method according to any one of claims 8 to 11, characterized in that, Determining the bitstream of the point cloud based on the first tensor data includes: The first tensor data is quantized and encoded to obtain the bitstream.

13. The method according to claim 12, characterized in that, The second neural network includes at least one feature extraction layer, the last feature extraction layer of the at least one feature extraction layer includes a convolutional layer, and the feature extraction layers other than the last feature extraction layer of the at least one feature extraction layer include convolutional layers and activation layers.

14. The method according to claim 12, characterized in that, The convolutional layers in the second neural network are located at the same positions as the deconvolutional layers in the first neural network, and the activation layers in the second neural network are located at the same positions as the activation layers in the first neural network.

15. The method according to claim 12, characterized in that, The parameters of the convolutional layers in the second neural network are the same as the parameters of the deconvolutional layers in the first neural network.

16. A training method for generative adversarial networks, characterized in that, include: Based on the attribute information of the point cloud, the third tensor data of the point cloud is determined; The third tensor data includes the feature values ​​of the point cloud's attribute information in at least one second channel; Based on the third tensor data, the first tensor data is determined using the second neural network included in the generator of the adversarial generative network; the first tensor data includes the feature values ​​of the point cloud's attribute information in at least one first channel after feature extraction. The at least one first channel is different from the at least one second channel; Based on the first tensor data, the generator uses a first neural network to determine the second tensor data; the second tensor data includes the feature values ​​of the reconstructed attribute information of the point cloud in at least one second channel; Based on the second tensor data, the reconstruction attribute information of the point cloud is determined; Based on the attribute information of the point cloud and the reconstructed attribute information of the point cloud, the generative adversarial network is trained using the discriminator in the generative adversarial network.

17. The method according to claim 16, characterized in that, The step of training the generative adversarial network (GAN) based on the attribute information of the point cloud and the reconstructed attribute information of the point cloud, using the discriminator in the GAN, includes: Based on the attribute information of the point cloud and the reconstructed attribute information of the point cloud, the distortion value between the attribute information of the point cloud and the reconstructed attribute information of the point cloud is determined. Based on the bit rate of the bitstream of the point cloud and the distortion value, the loss value of the generative adversarial network is determined; the bitstream of the point cloud is the bitstream obtained by arithmetic encoding and quantization of the first tensor data. The generative adversarial network is trained based on its loss value.

18. The method according to claim 17, characterized in that, The step of determining the distortion value between the attribute information of the point cloud and the reconstructed attribute information of the point cloud based on the attribute information of the point cloud includes: Based on the voxel occupancy information in the point cloud, the loss value of the generator is determined; Based on the attribute information of the point cloud and the reconstructed attribute information of the point cloud, the loss value of the discriminator is determined; The distortion value is determined based on the loss value of the generator and the loss value of the discriminator.

19. The method according to claim 18, characterized in that, Determining the generator's loss value based on the voxel occupancy information in the point cloud includes: The loss value of the generator is determined according to the following formula: in, This represents the loss value of the generator. In the point cloud, the first... The weight of individual elements, Indicates the first The occupancy information of individual elements, where ξ represents the focusing parameter.

20. The method according to claim 18, characterized in that, The step of determining the loss value of the discriminator based on the attribute information of the point cloud and the reconstructed attribute information of the point cloud includes: The loss value of the discriminator is determined according to the following formula: in, Let E[·] represent the loss value of the discriminator, E[·] represent the expected value operation, and m represent the attribute information of the point cloud. The probability distribution of the attribute information of the point cloud is represented by n, where n represents the reconstructed attribute information of the point cloud. This represents the probability distribution of the reconstructed attribute information of the point cloud.

21. The method according to claim 18, characterized in that, Determining the distortion value based on the loss value of the generator and the loss value of the discriminator includes: The distortion value is determined according to the following formula: Where x represents the distortion value, This represents the loss value of the discriminator. The weights represent the loss values ​​of the discriminator. This represents the loss value of the generator. The weights represent the loss values ​​of the generator.

22. The method according to any one of claims 17 to 21, characterized in that, Determining the loss value of the generative adversarial network based on the bit rate of the bitstream from the point cloud and the distortion value includes: The loss value of the generative adversarial network is determined according to the following formula: in, Let x represent the loss value of the generative adversarial network, λ represent the distortion value, λ represent the distortion weight, and y represent the ratio. Special rate.

23. A decoding method, characterized in that, include: Decode the bitstream of the point cloud to determine the first tensor data; The first tensor data includes the feature values ​​of the point cloud's attribute information in at least one first channel after feature extraction; Based on the first tensor data, a second tensor data is determined using a first neural network; the first neural network and the second neural network used by the encoder are used to form a generator for a generative adversarial network; the second tensor data includes feature values ​​of the reconstructed attribute information of the point cloud in at least one second channel; the at least one first channel is different from the at least one second channel; the generative adversarial network is trained by the method according to any one of claims 16 to 22; Based on the second tensor data, the reconstruction attribute information of the point cloud is determined.

24. An encoding method, characterized in that, include: Based on the attribute information of the point cloud, the third tensor data of the point cloud is determined; The third tensor data includes the feature values ​​of the point cloud's attribute information in at least one second channel; Based on the third tensor data, the first tensor data is determined using a second neural network; the second neural network and the first neural network used by the decoder are used to form the generator of the generative adversarial network; the first tensor data includes the feature values ​​of the point cloud's attribute information after feature extraction in at least one first channel; the at least one first channel is different from the at least one second channel; the generative adversarial network is trained by the method according to any one of claims 16 to 22; Based on the first tensor data, the bitstream of the point cloud is determined.

25. A decoder, characterized in that, include: The decoding unit is used to decode the bitstream of the point cloud and determine the first tensor data; The first tensor data includes the feature values ​​of the point cloud's attribute information in at least one first channel after feature extraction; The first determining unit is used to determine the second tensor data based on the first tensor data using a first neural network; the first neural network and the second neural network used by the encoder are used to form a generator of a generative adversarial network; the second tensor data includes the feature values ​​of the reconstructed attribute information of the point cloud in at least one second channel; The at least one first channel is different from the at least one second channel; The second determining unit is used to determine the reconstruction attribute information of the point cloud based on the second tensor data.

26. An encoder, characterized in that, include: The first determining unit is used to determine the third tensor data of the point cloud based on the attribute information of the point cloud; The third tensor data includes the feature values ​​of the point cloud's attribute information in at least one second channel; The second determining unit is used to determine the first tensor data based on the third tensor data using a second neural network; the second neural network and the first neural network used by the decoder are used to form a generator of a generative adversarial network; the first tensor data includes the feature values ​​of the point cloud's attribute information in at least one first channel after feature extraction; The at least one first channel is different from the at least one second channel; The third determining unit is used to determine the bitstream of the point cloud based on the first tensor data.

27. A training device for generative adversarial networks, characterized in that, include: The first determining unit is used to determine the third tensor data of the point cloud based on the attribute information of the point cloud; The third tensor data includes the feature values ​​of the point cloud's attribute information in at least one second channel; The second determining unit is used to determine the first tensor data based on the third tensor data and using the second neural network included in the generator of the adversarial generative network; the first tensor data includes the feature values ​​of the attribute information of the point cloud in at least one first channel after feature extraction; The at least one first channel is different from the at least one second channel; The third determining unit is used to determine second tensor data based on the first tensor data and using the first neural network included in the generator; the second tensor data includes feature values ​​of the reconstructed attribute information of the point cloud in at least one second channel; The fourth determining unit is used to determine the reconstruction attribute information of the point cloud based on the second tensor data; The training unit is used to train the generative adversarial network (GAN) based on the attribute information of the point cloud and the reconstructed attribute information of the point cloud, using the discriminator in the GAN.

28. An electronic device, characterized in that, include: A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program that, when executed by the processor, performs at least one of the following methods: the method according to any one of claims 1 to 7, the method according to any one of claims 8 to 15, the method according to any one of claims 16 to 22, the method according to claim 23, and the method according to claim 24.

29. A computer-readable storage medium, characterized in that, Used to store a computer program that, when the computer program is run on a computer, causes the computer to perform at least one of the following methods: the method according to any one of claims 1 to 7, the method according to any one of claims 8 to 15, the method according to any one of claims 16 to 22, the method according to claim 23, and the method according to claim 24.

30. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement at least one of the following methods: the method according to any one of claims 1 to 7, the method according to any one of claims 8 to 15, the method according to any one of claims 16 to 22, the method according to claim 23, and the method according to claim 24.

31. A bitstream, characterized in that, The bitstream is a bitstream decoded by the method according to any one of claims 1 to 7 or the method according to claim 23, or the bitstream is a bitstream generated by the method according to any one of claims 8 to 15 or the method according to claim 24.

Images