Decoding method, encoding method, training method, decoder, and encoder

Abstract

The embodiment of the invention provides a decoding method, an encoding method, a training method, a decoder and an encoder. The decoding method comprises the following steps: decoding a code stream of a point cloud, and determining first tensor data; the first tensor data comprises feature values of the attribute information of the point cloud on at least one first channel after feature extraction; determining second tensor data by using a first neural network based on the first tensor data; the first neural network and a second neural network adopted by the encoder are used for forming a generator of the generative adversarial network; the second tensor data comprises feature values of reconstruction attribute information of the point cloud on at least one second channel; the at least one first channel is different from the at least one second channel; and determining reconstruction attribute information of the point cloud based on the second tensor data. According to the decoding method, the point cloud attribute information with good quality can be reconstructed under the condition of considering the calculation complexity and memory occupation of the decoder, namely, the decoding performance of the decoder can be improved.

Description

Decoding method, encoding method, training method, decoder and encoder Technical Field

[0001] The embodiments of the present application relate to the field of coding and decoding technology, and more specifically, to a decoding method, an encoding method, a training method, a decoder, and an encoder. Background Art

[0002] Digital video compression technology mainly compresses huge digital image video data for easy transmission and storage.

[0003] However, with the surge in Internet videos and people's increasing requirements for video clarity, although the existing digital video compression standards can implement video decompression technology, there is still a need to pursue better digital video decompression technology to reduce the bandwidth and traffic pressure of digital video transmission, thereby improving the decoding performance of the decoder.

[0004] Summary of the Invention

[0005] The embodiments of the present application provide a decoding method, an encoding method, a training method, a decoder, and an encoder, which can improve the decoding performance of the decoder.

[0006] In a first aspect, an embodiment of the present application provides a decoding method, comprising:

[0007] Decoding a bit stream of the point cloud to determine first tensor data; the first tensor data includes feature values ​​of attribute information of the point cloud on at least one first channel after feature extraction;

[0008] Determining second tensor data using a first neural network based on the first tensor data; the first neural network and a second neural network used by the encoder are used to form a generator of a generative adversarial network; the second tensor data includes feature values ​​of reconstructed attribute information of the point cloud on at least one second channel; the at least one first channel is different from the at least one second channel;

[0009] Based on the second tensor data, reconstructed attribute information of the point cloud is determined.

[0010] In a second aspect, an embodiment of the present application provides an encoding method, including:

[0011] Determining third tensor data of the point cloud based on the attribute information of the point cloud; the third tensor data includes a feature value of the attribute information of the point cloud on at least one second channel;

[0012] Based on the third tensor data, first tensor data is determined using a second neural network; the second neural network and the first neural network used by the decoder are used to form a generator of a generative adversarial network; the first tensor data includes feature values ​​of at least one first channel after feature extraction of attribute information of the point cloud; the at least one first channel is different from the at least one second channel;

[0013] Based on the first tensor data, a code stream of the point cloud is determined.

[0014] In a third aspect, an embodiment of the present application provides a training method for a generative adversarial network, including:

[0015] Determining third tensor data of the point cloud based on the attribute information of the point cloud; the third tensor data includes a feature value of the attribute information of the point cloud on at least one second channel;

[0016] Based on the third tensor data, first tensor data is determined using a second neural network included in a generator in a generative adversarial network; the first tensor data includes feature values ​​of at least one first channel after feature extraction of attribute information of the point cloud; the at least one first channel is different from the at least one second channel;

[0017] Based on the first tensor data, second tensor data is determined using a first neural network included in a generator; the second tensor data includes feature values ​​of the reconstructed attribute information of the point cloud on at least one second channel;

[0018] determining reconstruction attribute information of the point cloud based on the second tensor data;

[0019] Based on the attribute information of the point cloud and the reconstructed attribute information of the point cloud, the generative adversarial network is trained using a discriminator in the generative adversarial network.

[0020] In a fourth aspect, an embodiment of the present application provides a decoding method, comprising:

[0021] Decoding a bit stream of the point cloud to determine first tensor data; the first tensor data includes feature values ​​of attribute information of the point cloud on at least one first channel after feature extraction;

[0022] Based on the first tensor data, second tensor data is determined using a first neural network; the first neural network and a second neural network used by the encoder are used to form a generator of a generative adversarial network; the second tensor data includes feature values ​​of reconstructed attribute information of the point cloud on at least one second channel; the at least one first channel is different from the at least one second channel; the generative adversarial network is trained according to the training method of the third aspect mentioned above;

[0023] Based on the second tensor data, reconstructed attribute information of the point cloud is determined.

[0024] In a fifth aspect, an embodiment of the present application provides an encoding method, including:

[0025] Determining third tensor data of the point cloud based on the attribute information of the point cloud; the third tensor data includes a feature value of the attribute information of the point cloud on at least one second channel;

[0026] Based on the third tensor data, first tensor data is determined using a second neural network; the second neural network and the first neural network used by the decoder are used to form a generator of a generative adversarial network; the first tensor data includes feature values ​​of at least one first channel after feature extraction of attribute information of the point cloud; the at least one first channel is different from the at least one second channel; the generative adversarial network is trained according to the training method of the third aspect mentioned above;

[0027] Based on the first tensor data, a code stream of the point cloud is determined.

[0028] In a sixth aspect, an embodiment of the present application provides a decoder, including:

[0029] A decoding unit, configured to decode a code stream of the point cloud and determine first tensor data; the first tensor data includes feature values ​​of attribute information of the point cloud on at least one first channel after feature extraction;

[0030] a first determining unit configured to determine second tensor data using a first neural network based on the first tensor data; the first neural network and a second neural network employed by the encoder are configured to form a generator of a generative adversarial network; the second tensor data comprising feature values ​​of reconstructed attribute information of the point cloud on at least one second channel; the at least one first channel being different from the at least one second channel;

[0031] The second determining unit is configured to determine the reconstruction attribute information of the point cloud based on the second tensor data.

[0032] In a seventh aspect, an embodiment of the present application provides an encoder, including:

[0033] A first determining unit is configured to determine third tensor data of the point cloud based on the attribute information of the point cloud; the third tensor data includes a feature value of the attribute information of the point cloud on at least one second channel;

[0034] a second determining unit configured to determine first tensor data using a second neural network based on the third tensor data; the second neural network and the first neural network used by the decoder are used to form a generator of a generative adversarial network; the first tensor data includes feature values ​​of at least one first channel after feature extraction of attribute information of the point cloud; the at least one first channel is different from the at least one second channel;

[0035] A third determining unit is configured to determine a code stream of the point cloud based on the first tensor data.

[0036] In an eighth aspect, an embodiment of the present application provides a decoder, including:

[0037] a processor adapted to implement computer instructions; and,

[0038] Computer-readable storage medium, the computer-readable storage medium stores computer instructions, and the computer instructions are suitable for being loaded by a processor and executing the method described in any aspect of the first and fourth aspects mentioned above or their respective implementations.

[0039] In one implementation, there are one or more processors and one or more memories.

[0040] In one implementation, the computer-readable storage medium may be integrated with the processor, or the computer-readable storage medium may be provided separately from the processor.

[0041] In a ninth aspect, an embodiment of the present application provides an encoder, including:

[0042] a processor adapted to implement computer instructions; and,

[0043] Computer-readable storage medium, the computer-readable storage medium stores computer instructions, and the computer instructions are suitable for being loaded by a processor and executing the method described in the third aspect or its various implementations involved above.

[0044] In one implementation, there are one or more processors and one or more memories.

[0045] In one implementation, the computer-readable storage medium may be integrated with the processor, or the computer-readable storage medium may be provided separately from the processor.

[0046] In a tenth aspect, an embodiment of the present application provides an encoder, including:

[0047] a processor adapted to implement computer instructions; and,

[0048] Computer-readable storage medium, the computer-readable storage medium stores computer instructions, and the computer instructions are suitable for being loaded by a processor and executing the method described in any aspect of the second and fifth aspects or their respective implementations involved above.

[0049] In one implementation, there are one or more processors and one or more memories.

[0050] In one implementation, the computer-readable storage medium may be integrated with the processor, or the computer-readable storage medium may be provided separately from the processor.

[0051] In the eleventh aspect, an embodiment of the present application provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are read and executed by the processor of a computer device, the computer device executes the method described in any one of the first to fifth aspects mentioned above or its various implementation methods.

[0052] In a twelfth aspect, embodiments of the present application provide a computer program product or computer program, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method described in any one of the first to fifth aspects above, or in their respective implementations.

[0053] In a ninth aspect, an embodiment of the present application provides a code stream, which is a code stream involved in the method described in any aspect of the first and fourth aspects mentioned above or their implementations, or a code stream generated by the method described in any aspect of the second and fifth aspects mentioned above or their implementations.

[0054] Based on the above technical solution, the present application uses the first neural network adopted by the decoder and the second neural network adopted by the encoder to form a generator of a generative adversarial network, and uses the first neural network to convert the first tensor data determined by decoding the point cloud code stream into the second tensor data for reconstructing attribute information. It fully combines the advantages of the autoencoder and the generative adversarial network, and can not only control the computational complexity and memory occupancy of the decoder, but also reconstruct high-quality point cloud attribute information. That is, the decoding method provided in the embodiment of the present application can reconstruct high-quality point cloud attribute information while taking into account the computational complexity and memory occupancy of the decoder, that is, it can improve the decoding performance of the decoder. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] FIG1 is an example of a point cloud image provided by an embodiment of the present application.

[0056] FIG2 is a partially enlarged view of the point cloud image shown in FIG1 .

[0057] FIG3 is an example of a point cloud image with six viewing angles provided by an embodiment of the present application.

[0058] FIG4 is an example of a data storage format of a point cloud image provided in an embodiment of the present application.

[0059] FIG5 is an example of the encoding and decoding process of PCAC provided in an embodiment of the present application.

[0060] FIG6 is a schematic flowchart of a decoding method provided in an embodiment of the present application.

[0061] FIG7 is a schematic structural diagram of a generative adversarial network provided in an embodiment of the present application.

[0062] FIG8 is a schematic flowchart of the encoding method provided in an embodiment of the present application.

[0063] FIG9 is a schematic diagram of the voxelization principle provided in an embodiment of the present application.

[0064] Figure 10 is a schematic flowchart of the training method of the generative adversarial network provided in an embodiment of the present application.

[0065] FIG11 is a schematic block diagram of a decoder provided in an embodiment of the present application.

[0066] FIG12 is a schematic block diagram of an encoder provided in an embodiment of the present application.

[0067] Figure 13 is a schematic block diagram of a training device for a generative adversarial network provided in an embodiment of the present application.

[0068] FIG14 is a schematic block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0069] The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings.

[0070] A point cloud is a collection of irregularly distributed discrete points in space that represent the spatial structure and surface properties of a three-dimensional object or scene. Figures 1 and 2 show a 3D point cloud image and a zoomed-in view, respectively. The point cloud surface is composed of densely distributed points.

[0071] A two-dimensional image contains information at every pixel, so there's no need to record its location information separately. However, the distribution of points in a point cloud in three-dimensional space is random and irregular, so it's necessary to record the location of each point in space to fully represent a point cloud. Similar to a two-dimensional image, each point in a point cloud has corresponding attribute information, typically an RGB color value, which reflects the color of the object. For a point cloud, in addition to color, the attribute information corresponding to each point can also be a reflectance value, which reflects the surface material of the object. Each point in a point cloud can include both geometric and attribute information. The geometric information of each point in a point cloud refers to the Cartesian three-dimensional coordinate data of that point, and the attribute information of each point in a point cloud can include, but is not limited to, at least one of the following: color information, material information, and laser reflection intensity information. Color information can be information in any color space. For example, color information can be red, green, and blue (RGB) information. For another example, color information can also be luminance and chrominance (YCbCr, YUV) information. Here, Y represents brightness (Luma), Cb (U) represents the blue chromaticity component, and Cr (V) represents the red chromaticity component. Each point in a point cloud has the same amount of attribute information. For example, each point in a point cloud has two types of attribute information: color information and laser reflection intensity. Another example is that each point in a point cloud has three types of attribute information: color information, material information, and laser reflection intensity information.

[0072] The point cloud image may have multiple viewing angles. For example, the point cloud image shown in FIG. 3 may have six viewing angles.

[0073] The data storage format for a point cloud image consists of a file header and a data portion. The header information includes the data format, data representation type, the total number of points in the point cloud, and the content represented by the point cloud. For example, as shown in Figure 4, the header information for the data storage format for a point cloud image may include the following: ".ply" format, represented by ASCII code, with a total number of 207,242 points, and each point having 3D position information (xyz) and 3D color information (rgb).

[0074] Point clouds can flexibly and conveniently express the spatial structure and surface properties of three-dimensional objects or scenes. Moreover, since point clouds are obtained by directly sampling real objects, they can provide a strong sense of reality while ensuring accuracy. Therefore, they are widely used, including virtual reality games, computer-aided design, geographic information systems, automatic navigation systems, digital cultural heritage, free viewpoint broadcasting, three-dimensional immersive remote presentation, and three-dimensional reconstruction of biological tissues and organs.

[0075] Point clouds can be divided into two categories based on application scenarios: machine-perceived point clouds and human-perceived point clouds. Applications of machine-perceived point clouds include, but are not limited to, autonomous navigation systems, real-time inspection systems, geographic information systems, visual sorting robots, and disaster relief robots. Applications of human-perceived point clouds include, but are not limited to, digital cultural heritage, free-viewpoint broadcasting, 3D immersive communication, and 3D immersive interaction. Accordingly, point clouds can be divided into dense and sparse point clouds based on their acquisition method. They can also be divided into static and dynamic point clouds based on their acquisition approach. More specifically, point clouds can be divided into three types: static point clouds (type 1), dynamic point clouds (type 2), and dynamically acquired point clouds (type 3). For static point clouds, the object is stationary, and the device acquiring the point cloud is also stationary. For dynamic point clouds, the object is moving, but the device acquiring the point cloud is stationary. For dynamic point clouds, the device acquiring the point cloud is moving.

[0076] Point cloud acquisition methods include, but are not limited to, computer generation, 3D laser scanning, and 3D photogrammetry. Computers can generate point clouds of virtual three-dimensional objects and scenes; 3D laser scanning can obtain point clouds of static real-world three-dimensional objects or scenes, acquiring millions of point clouds per second; and 3D photogrammetry can obtain point clouds of dynamic real-world three-dimensional objects or scenes, acquiring tens of millions of point clouds per second. Specifically, point clouds of object surfaces can be acquired using acquisition devices such as photoelectric radars, lidars, laser scanners, and multi-view cameras. Point clouds obtained using laser measurement principles can include the three-dimensional coordinate information of points and the laser reflection intensity (reflectance) of points. Point clouds obtained using photogrammetry principles can include the three-dimensional coordinate information of points and the color information of points. Combining laser measurement and photogrammetry principles to obtain point clouds can include the three-dimensional coordinate information of points, the laser reflection intensity (reflectance) of points, and the color information of points. These technologies reduce the cost and time required to acquire point cloud data and improve data accuracy. For example, in the medical field, magnetic resonance imaging (MRI), computed tomography (CT), and electromagnetic positioning information can be used to generate point clouds of biological tissues and organs. These technologies reduce the cost and time required to acquire point clouds, while improving data accuracy. While changes in point cloud data acquisition methods have made it possible to acquire large amounts of point cloud data, the processing of massive amounts of 3D point cloud data is facing bottlenecks due to storage space and transmission bandwidth limitations, as application demands grow.

[0077] Taking a point cloud video with a frame rate of 30 as an example, each frame contains 700,000 points. Each point in each point cloud frame has coordinate information (xyz, float) and color information (RGB, uchar). Therefore, the data volume of a 10-second point cloud video is approximately 0.7 million × (4 bytes × 3 + 1 byte × 3) × 30 fps × 10 seconds = 3.15 GB. For a 2D video with a resolution of 1280 × 720, a YUV sampling format of 4:2:0, and a frame rate of 24, the data volume per 10 seconds is approximately 1280 × 720 × 12 bits × 24 frames × 10 seconds, which is approximately 0.33 GB. A 10-second two-view 3D video has a data volume of approximately 0.33 × 2 = 0.66 GB. This shows that the data volume of a point cloud video far exceeds that of 2D and 3D videos of the same length. Therefore, in order to better realize data management, save server storage space, and reduce the transmission traffic and transmission time between the server and the client, point cloud compression has become a key issue in promoting the development of the point cloud industry.

[0078] Point cloud compression generally compresses point cloud geometry and attribute information separately. On the encoding side, the point cloud geometry is first encoded in the geometry encoder, and the reconstructed geometry is then fed into the attribute encoder as additional information to assist in attribute compression. On the decoding side, the point cloud geometry is first decoded in the geometry decoder, and the decoded geometry is then fed into the attribute decoder as additional information to assist in attribute compression. The entire codec consists of pre-processing / post-processing, geometry encoding / decoding, and attribute encoding / decoding.

[0079] Point clouds can be encoded and decoded respectively through various types of encoding frameworks and decoding frameworks. As an example, the encoding and decoding framework can be the Geometry Point Cloud Compression (G-PCC) encoding and decoding framework or the Video Point Cloud Compression (V-PCC) encoding and decoding framework provided by the Moving Picture Experts Group (MPEG), or it can be the AVS-PCC encoding and decoding framework or the Point Cloud Compression Reference Platform (PCRM) framework provided by the Audio Video Standard (AVS) Task Force. The G-PCC encoding and decoding framework can be used to compress the first static point cloud and the third type of dynamically acquired point cloud, and the V-PCC encoding and decoding framework can be used to compress the second type of dynamic point cloud. The G-PCC encoding and decoding framework is also called TMC13, and the V-PCC encoding and decoding framework is also called TMC2. Both G-PCC and AVS-PCC can be used to compress static sparse point clouds, and their encoding frameworks are roughly the same.

[0080] Of course, other encoding and decoding frameworks can also be used to compress the point cloud, and this application does not make specific limitations on this.

[0081] The following introduces the encoding and decoding framework of point cloud attribute compression (PCAC).

[0082] FIG5 is an example of the encoding and decoding process of PCAC provided in an embodiment of the present application.

[0083] As shown in Figure 5, the main encoder extracts the attribute information of the original input point cloud to obtain a latent representation of the attribute information, and then restores it to a reconstructed point cloud (PC) in the main decoder. The main encoder can be a variational autoencoder (VAE) framework.

[0084] The main encoder and main decoder are the core components of PCAC encoding and decoding. Both the main encoder and decoder use convolutional layers and activation layers as analysis transforms. The convolutional layers can be sparse convolution (SCONv) layers, and the activation layers can be rectified linear units (ReLU).

[0085] The specific encoding process of the main encoder is as follows:

[0086] (1) Input voxelized point cloud data, and the main encoder uses convolutional layers for feature extraction. The main encoder consists of 6 convolutional layers and 2 activation layers. Optionally, all convolutional layers use 3×3×3 convolution kernels, and the number of channels in the first convolutional layer is 64, while the number of channels in the remaining convolutional layers is 128.

[0087] (2) After each convolutional layer, an activation layer is used for downsampling. The activation layer shares parameters with the convolutional layer. For example, the maximum value of all points inside each voxel can be calculated and used as the feature representation of the voxel.

[0088] (3) Repeat steps (1) and (2) until a very small feature map is obtained.

[0089] (4) The last layer outputs a feature map, which is mapped to the latent space through the fully connected layer. This latent space is the compressed point cloud data.

[0090] After the main encoder completes the encoding process, the main decoder's specific decoding process is as follows:

[0091] (1) The compressed point cloud data is input into the transposed convolution layer for deconvolution, and finally mapped back to the last feature map.

[0092] (2) Upsample the deconvolution feature map to obtain a larger feature map. All deconvolution layers share parameters with the encoder in the encoding process.

[0093] (3) Repeat steps (1) and (2) until a feature map of the same size as the original input point cloud is obtained.

[0094] Specifically, the encoding and decoding process of the main encoder and main decoder is as follows:

[0095] First, the primary encoder extracts the attribute information of the original input point cloud to obtain a latent representation of the attribute information. The latent representation of the attribute information output by the primary encoder is then quantized (Q) to obtain a quantized latent representation of the attribute information. This quantized latent representation of the attribute information is then input into the arithmetic encoder (AE) for arithmetic encoding to produce a bitstream. After the bitstream is transmitted, the arithmetic decoder (AD) connected to the primary decoder receives the bitstream output by the AE. The AD performs arithmetic decoding on the input bitstream and uses the primary decoder to decode the latent representation obtained by the AD to output a reconstructed point cloud.

[0096] In addition, for sparse PCAC, a context model determined by a super-prior encoder and a super-prior decoder can be introduced. The context model is used to assist AE or AD in entropy encoding the quantized latent representation output by the main encoder.

[0097] The super-a priori encoder and super-a priori decoder are auxiliary components of PCAC encoding and decoding. Similar to the main encoder and main decoder, the super-a priori encoder and super-a priori decoder both use convolutional layers and activation layers as analysis transformations. The convolutional layers can be SCONv layers, and the activation layers can be ReLU. The encoding and decoding process of the super-a priori encoder and super-a priori decoder is as follows:

[0098] (1) After the main encoder maps the input point cloud data to the latent space, feature extraction is performed on the latent representation output by the main encoder to obtain a high-dimensional vector as the super-prior latent representation.

[0099] (2) When the main decoder maps the latent representation extracted by the main encoder back to the size and shape of the original input point cloud, the super-prior decoder is used to decode the generated super-prior latent representation to obtain a context model; and the context model is coupled with the autoregressive neighborhood; the autoregressive neighborhood is used to determine the probability distribution of the current point and the information related to the already encoded and decoded points.

[0100] Specifically, the super-a priori encoder further extracts features from the latent representation of the attribute information output by the main encoder and uses the extracted latent representation as the input to the super-a priori encoder. The super-a priori latent representation of the attribute information output by the super-a priori encoder is quantized to obtain a quantized super-a priori latent representation of the attribute information. This quantized super-a priori latent representation of the attribute information is then input into the AE for arithmetic coding to obtain a super-a priori bitstream. After the super-a priori bitstream is transmitted, the AD connected to the super-a priori decoder receives the super-a priori bitstream, performs arithmetic decoding on the input super-a priori bitstream, and uses the super-a priori decoder to decode the super-a priori latent representation obtained by the AD decoding to output the context model. Then, an activation layer is used to process the quantized latent representation output by the main encoder, and the neighborhood information obtained by the processing is coupled with the context model of the encoding output of the super-prior encoder to obtain the coupled features; then, one activation layer and two convolutional layers are used to process the coupled features to obtain parameters for assisting AE in arithmetic coding and for assisting AD in arithmetic decoding. For example, the parameters for assisting AE in arithmetic coding and for assisting AD in arithmetic decoding may include u and σ. u represents the probability model of the point cloud, and σ represents the parameters for coupling the context model and neighborhood information output by the super-prior decoder. Among them, the parameters used by the AE for arithmetically encoding the super-prior latent representation and the AD for arithmetically decoding the super-prior code stream may be the parameters output by the factorized entropy model.

[0101] This shows that because the arithmetic coding and decoding parts of the main encoder and main decoder combine the super-prior context model and neighborhood information, they can accurately estimate the conditional entropy probability. Specifically, at each time t, the symbol sequence encoded at time t-1 is used as neighborhood information, and the high-dimensional vector generated by the super-prior encoder is used as conditional information. Then, the Laplace distribution is used to model the conditional probability distribution, and the parameters of the Laplace distribution are estimated through maximum likelihood estimation. Finally, these parameters are used for arithmetic coding to encode the latent features output by the main encoder into a bitstream.

[0102] However, due to the need to utilize neighborhood information in this process, PCAC makes strong assumptions about the geometry and structure of point cloud data, placing high demands on the resolution and density of the input point cloud. This can lead to reduced compression performance for sparse point cloud data. Furthermore, because PCAC requires significant computational resources and storage space to train the model and store information such as the latent representation, it faces challenges such as memory limitations and computational time. Finally, PCAC was tested using only a small number of point cloud datasets in its experiments, and its performance and versatility require further verification and improvement.

[0103] In view of this, the present application provides a decoding method, an encoding method, a decoder, and an encoder, which use a generative adversarial network (GAN) to decode the bitstream. Specifically, the GAN mainly consists of a generator and a discriminator, wherein the generator uses a convolution layer to compress the input voxelized point cloud data into a bitstream, and then uses a transposed convolution to decode the bitstream to obtain a reconstructed point cloud. The discriminator can return a judgment result based on the reconstructed point cloud and the input point cloud. The judgment result is used to continuously train the generator until the reconstructed point cloud is as consistent as possible with the input point cloud.

[0104] FIG6 is a schematic flowchart of a decoding method 100 provided in an embodiment of the present application.

[0105] As shown in FIG6 , the decoding method 100 may include:

[0106] S110, the decoder decodes the code stream of the point cloud to determine first tensor data; the first tensor data includes feature values ​​of the attribute information of the point cloud on at least one first channel after feature extraction.

[0107] Exemplarily, the dimension of the first tensor data may include at least one of the following: the dimension where the at least one first channel is located, the dimension where the index of the voxel in the point cloud is located, and the dimension where the attribute information of the voxel in the point cloud is located.

[0108] Tensors are a natural representation of high-dimensional data. A tensor represents a multidimensional array, and the number of dimensions of a tensor is called its rank. Generally speaking, a rank-0 tensor represents a scalar, such as s = 123; a rank-1 tensor represents a vector, such as v = [1, 2, 3]; a rank-2 tensor represents a matrix, such as m = [[1, 2, 3], [4, 5, 6]]; and a rank-n tensor represents a multidimensional array, typically [[[...]]].

[0109] Exemplarily, the attribute information of the point cloud may be attribute information of voxels in the point cloud, that is, attribute information of voxels obtained after the encoder divides the point cloud into voxels.

[0110] It is worth noting that the attribute information of the point cloud can be any kind of attribute information.

[0111] For example, the attribute information of the point cloud includes, but is not limited to, at least one of the following: color information, material information, laser reflection intensity information, and normal vector information.

[0112] The color information can be information in any color space. For example, the color information can be red, green, and blue (RGB) information. For another example, the color information can also be luminance and chrominance (YCbCr, YUV) information. Among them, Y represents brightness (Luma), Cb (U) represents the blue chrominance component, and Cr (V) represents the red chrominance component. Each point in the point cloud has the same amount of attribute information. For example, each point in the point cloud has two types of attribute information: color information and laser reflection intensity. For another example, each point in the point cloud has three types of attribute information: color information, material information, and laser reflection intensity information.

[0113] It is worth noting that the number of the at least one first channel can be any integer, and this application does not impose any specific limitation on this.

[0114] For example, the number of the at least one first channel may be 64, 128, or other values.

[0115] For another example, the number of the at least one first channel may be the number of channels used by the last convolutional layer in the second neural network adopted by the encoder. In this case, it can also be understood that the at least one first channel may be the channel used by the last convolutional layer in the second neural network.

[0116] Assuming that the number of channels used by the last convolutional layer in the second neural network is 128, the number of the at least one first channel is 128. In this case, if the number of voxels in the point cloud is M, the first data tensor can be implemented as a data tensor in the following form:

[0117] 128×M; M represents the number of voxels in the point cloud.

[0118] Among them, the eigenvalue x in the first data tensor ij It represents the value of the attribute information of the i-th voxel in the point cloud on the j-th channel after feature extraction.

[0119] It should be understood that the first tensor data is a data format of the attribute information of the point cloud after feature extraction, and its data content is essentially the value obtained after feature extraction of the original attribute information of the point cloud. The difference between it and the attribute information of the point cloud is that when the original attribute information of the point cloud is converted into the format of tensor data, its corresponding channel is different from (usually less than) the at least one first channel.

[0120] S120, the decoder determines second tensor data using a first neural network based on the first tensor data; the first neural network and the second neural network used by the encoder are used to form a generator of a generative adversarial network; the second tensor data includes eigenvalues ​​of the reconstructed attribute information of the point cloud on at least one second channel; the at least one first channel is different from the at least one second channel.

[0121] In some embodiments, the at least one second channel corresponds to at least one component of attribute information of the point cloud.

[0122] Exemplarily, the at least one component may include at least one of the following: a Y component, a U component, and a V component.

[0123] Exemplarily, the at least one component may include at least one of the following: an R component, a G component, and a B component.

[0124] Of course, in other alternative embodiments, the at least one component may also be other types of components, and this application will not make any specific limitations on this.

[0125] Assuming that the at least one second channel includes channels corresponding to the Y component, the U component, and the V component, respectively, the number of the at least one second channel is 3. In this case, if the number of voxels in the point cloud is M, the second data tensor can be implemented as a data tensor in the following form:

[0126] 3×M; M represents the number of voxels in the point cloud.

[0127] Among them, the eigenvalue y in the second data tensor ij Represents the value of the reconstructed attribute information of the i-th voxel of the point cloud on the j-th channel (for example, the Y component, the U component, or the V component).

[0128] It should be understood that the second tensor data is a data format of the reconstructed attribute information of the point cloud, and its data content is essentially the reconstructed attribute information of the point cloud. In other words, after obtaining the second tensor data, the decoder can directly convert the second tensor data into the reconstructed attribute information of the point cloud.

[0129] S130: The decoder determines the reconstructed attribute information of the point cloud based on the second tensor data.

[0130] In this embodiment, the first neural network adopted by the decoder and the second neural network adopted by the encoder are used to form a generator of a generative adversarial network, so that the decoder can use the first neural network to convert the first tensor data determined by decoding the point cloud code stream into the second tensor data for reconstructing attribute information. It fully combines the advantages of the autoencoder and the generative adversarial network, not only can it control the computational complexity and memory occupancy of the decoder, but also can reconstruct high-quality point cloud attribute information. That is, the decoding method provided in the embodiment of the present application can reconstruct high-quality point cloud attribute information while taking into account the computational complexity and memory occupancy of the decoder, that is, it can improve the decoding performance of the decoder.

[0131] The results obtained by testing the solution provided in this application are described below in conjunction with Table 1.

[0132] For the technical solution provided in this application, the MPEG standard test set was used to evaluate its performance, and the sparser point cloud data set of MVUB was additionally used to conduct comparative tests with MPEG G-PCC (i.e., TMC13v6 and TMC13v19) and SparsePCAC. Among them, bit rate distortion (Bit Distortion Bit Rate, BD-BR) and peak signal-to-noise ratio distortion (Bit Distortion Peak Signal to Noise Ratio, BD-PSNR) are indicators for measuring algorithm performance or coding performance. BD-BR represents the change in bit rate of the coding algorithm provided in this application relative to the original coding algorithm. A negative value as a whole indicates that the performance has improved. BD-PSNR represents the change in PSNR of the coding algorithm provided in this application relative to the original coding algorithm. A negative value as a whole indicates that the performance has improved. Furthermore, Y represents brightness (Luma), U represents the blue chrominance component, and V represents the red chrominance component.

[0133] Table 1

[0134] As shown in Table 1, the point cloud column represents different point cloud categories. BD-BR and BD-PSNR are indicators used to evaluate distortion performance, indicating the performance advantages and disadvantages of the proposed method and the comparative method, where a negative value for the former represents an improvement in performance, and a positive value for the latter represents an improvement in performance. It can be seen that for different categories of point clouds, although the decoding method provided in this application has a performance decline in the Y component and YUV relative to TMC13v19, it has performance improvements in the Y component and YUV relative to SparsePCAC and TMC13v6. In addition, it is worth noting that, relative to SparsePCAC, the decoding method provided in this application does not need to introduce a super-prior encoder and a super-prior decoder, that is, it does not need to consider domain information, which not only reduces the requirements of the decoding method for the resolution and density of the point cloud, but also reduces the computational complexity and memory occupancy of the decoder, that is, it can reconstruct good-quality point cloud attribute information while taking into account the computational complexity and memory occupancy of the decoder, thereby improving the decoding performance of the decoder.

[0135] It should be understood that the first tensor data and the second tensor data are essentially representations of the attribute information of the point cloud in different tensor formats. Therefore, in other alternative embodiments, the solution of the present application can also be equivalently replaced by the following description without introducing tensor data and channels:

[0136] Decoding a bitstream of a point cloud to determine first attribute information of the point cloud; determining second attribute information using a first neural network based on the first attribute information; using the first neural network and a second neural network used by the encoder to form a generator of a generative adversarial network; and determining reconstructed attribute information of the point cloud based on the second attribute information. For example, the second attribute information may be the reconstructed attribute information of the point cloud in a tensor format. For another example, the first attribute information may be the first tensor data, and the second attribute information may be the second tensor data.

[0137] In some embodiments, the first tensor data includes feature values ​​of the attribute information of the voxels in the point cloud on the at least one first channel after feature extraction, and the second tensor data includes feature values ​​of the reconstructed attribute information of the voxels on the at least one second channel.

[0138] The voxels in the point cloud may be encoding units obtained by performing voxel division on the point cloud by an encoder.

[0139] Assuming that the number of channels used by the last convolutional layer in the second neural network is 128, the number of the at least one first channel is 128. In this case, if the number of voxels in the point cloud is M, the first data tensor can be implemented as a data tensor in the following form:

[0140] 128×M; M represents the number of voxels in the point cloud.

[0141] Among them, the eigenvalue x in the first data tensor ij It represents the value of the attribute information of the i-th voxel in the point cloud on the j-th channel after feature extraction.

[0142] Assuming that the at least one second channel includes channels corresponding to the Y component, the U component, and the V component, respectively, the number of the at least one second channel is 3. In this case, if the number of voxels in the point cloud is M, the second data tensor can be implemented as a data tensor in the following form:

[0143] 3×M; M represents the number of voxels in the point cloud.

[0144] Among them, the eigenvalue y in the second data tensor ij Represents the value of the reconstructed attribute information of the i-th voxel of the point cloud on the j-th channel (for example, the Y component, the U component, or the V component).

[0145] Illustratively, the voxel resolutions of different voxels in the point cloud may be different or the same.

[0146] For example, the voxels of the point cloud may include voxels corresponding to multiple point cloud blocks. When the types of the multiple point cloud blocks are different, the voxel resolutions of the voxels corresponding to the multiple point cloud blocks are different.

[0147] In one possible implementation, when the multiple point cloud blocks include dense blocks and sparse blocks, the voxel resolution corresponding to the dense blocks is greater than or less than the voxel resolution corresponding to the sparse blocks. When the voxel resolution corresponding to the dense blocks is less than the voxel resolution corresponding to the sparse blocks, the attribute information of the sparse blocks can be retained as much as possible, and the reconstruction quality of the sparse blocks can be improved. When the voxel resolution corresponding to the dense blocks is greater than the voxel resolution corresponding to the sparse blocks, the attribute information of the dense blocks can be retained as much as possible, and the reconstruction quality of the sparse blocks can be improved. For example, the voxel resolution corresponding to the dense blocks can be a resolution of 16×16×16, and the voxel resolution corresponding to the sparse blocks can be a resolution of 8×8×8.

[0148] It is worth noting that the voxels of the point cloud may include voxels corresponding to multiple point cloud blocks, and the sizes of the multiple point cloud blocks may be the same or different, and this application does not make any specific limitations on this.

[0149] Of course, in other alternative embodiments, the voxels of the point cloud may include voxels corresponding to multiple point cloud blocks. When the types of the multiple point cloud blocks are different, the sizes of the voxels corresponding to the multiple point cloud blocks are different. Alternatively, the types of the multiple point cloud blocks may include multiple levels determined based on density, which is not specifically limited in this application.

[0150] In one possible implementation, when the multiple point cloud blocks include dense blocks and sparse blocks, the size of the voxels corresponding to the dense blocks is larger or smaller than the size of the voxels corresponding to the sparse blocks. When the size of the voxels corresponding to the dense blocks is larger than the size of the voxels corresponding to the sparse blocks, the attribute information of the sparse blocks can be retained as much as possible, and the reconstruction quality of the sparse blocks can be improved. When the size corresponding to the dense blocks is larger than the size of the voxels corresponding to the sparse blocks, the attribute information of the dense blocks can be retained as much as possible, and the reconstruction quality of the sparse blocks can be improved. For example, the size of the voxels corresponding to the dense blocks may be 16×16×16, and the size of the voxels corresponding to the sparse blocks may be 8×8×8.

[0151] Of course, in other alternative embodiments, the first tensor data includes the feature values ​​of the attribute information of the points in the point cloud on the at least one first channel after feature extraction, and the second tensor data includes the feature values ​​of the reconstructed attribute information of the points in the point cloud on the at least one second channel.

[0152] In some embodiments, the S110 may include:

[0153] The decoder dequantizes the tensor data obtained by decoding the code stream of the point cloud to obtain the first tensor data.

[0154] Exemplarily, since the code stream is obtained by the encoder through feature extraction, quantization and arithmetic coding, the first tensor data includes the eigenvalues ​​of the attribute information of the voxels in the point cloud on the at least one first channel after feature extraction, quantization, arithmetic coding, arithmetic decoding and inverse quantization, and the second tensor data includes the eigenvalues ​​of the reconstructed attribute information of the voxels on the at least one second channel.

[0155] Of course, in other alternative embodiments, the encoder may not voxelize the point cloud, that is, the encoder may directly perform feature extraction, quantization, and arithmetic coding on the attribute information of the points to form a bitstream of the point cloud. In this case, for the decoder, the first tensor data includes the feature values ​​of the attribute information of the points in the point cloud on the at least one first channel after feature extraction, and the second tensor data includes the feature values ​​of the reconstructed attribute information of the points in the point cloud on the at least one second channel. This application does not make specific limitations on this.

[0156] In some embodiments, the first neural network includes at least one feature extraction layer, the last feature extraction layer in the at least one feature extraction layer includes a deconvolution layer, and the feature extraction layers in the at least one feature extraction layer other than the last feature extraction layer include a deconvolution layer and an activation layer.

[0157] Among them, the deconvolution layer can also be called a transposed convolution layer, which is used to upsample the input.

[0158] Exemplarily, the first neural network includes multiple feature extraction layers, the last feature extraction layer among the multiple feature extraction layers includes a deconvolution layer, and the feature extraction layers other than the last feature extraction layer among the multiple feature extraction layers include multiple deconvolution layers and activation layers. For example, the number of feature extraction layers can be 3 or other values, and the number of deconvolution layers can be 2 or other values, which is not specifically limited in this application.

[0159] In some embodiments, the position of the deconvolution layer in the first neural network is the same as the position of the convolution layer in the second neural network, and the position of the activation layer in the first neural network is the same as the position of the activation layer in the second neural network.

[0160] Exemplarily, the position of the layer may be identified by a sequence number of the layer (eg, a sequence number according to a direction of data transmission).

[0161] Exemplarily, the sequence number of the deconvolution layer in the first neural network is the same as the sequence number of the convolution layer in the second neural network, and the sequence number of the activation layer in the first neural network is the same as the sequence number of the activation layer in the second neural network. In other words, if the i-th layer in the first neural network is a deconvolution layer, then the i-th layer in the second neural network is a convolution layer, and if the i-th layer in the first neural network is an activation layer, then the i-th layer in the second neural network is an activation layer. In other words,

[0162] For example, if the serial numbers of the deconvolution layers in the first neural network include layers numbered 1, 2, 4, 5, 7, and 8, and the serial numbers of the activation layers in the first neural network include layers numbered 3 and 6, then the serial numbers of the convolution layers in the second neural network include layers numbered 1, 2, 4, 5, 7, and 8, and the serial numbers of the activation layers in the second neural network include layers numbered 3 and 6.

[0163] In some embodiments, parameters of the deconvolution layer in the first neural network are the same as parameters of the convolution layer in the second neural network.

[0164] Exemplarily, the parameters of the i-th deconvolution layer in the first neural network are the same as the parameters of the k+1-i-th convolution layer in the second neural network, where k represents the number of deconvolution layers in the first neural network, or k represents the number of convolution layers in the second neural network.

[0165] For example, assuming k=6, the parameters of the first deconvolution layer in the first neural network are the same as the parameters of the sixth convolution layer in the second neural network, the parameters of the second deconvolution layer in the first neural network are the same as the parameters of the fifth convolution layer in the second neural network, and so on. The parameters of the sixth deconvolution layer in the first neural network are the same as the parameters of the first convolution layer in the second neural network.

[0166] Exemplarily, the parameters of the deconvolution layer in the first neural network may include a convolution kernel and / or the number of channels.

[0167] For example, the convolution kernel of the i-th deconvolution layer in the first neural network is the same as the convolution kernel of the k+1-i-th convolution layer in the second neural network. For another example, the number of channels of the i-th deconvolution layer in the first neural network is the same as the number of channels of the k+1-i-th convolution layer in the second neural network.

[0168] FIG7 is a schematic structural diagram of a generative adversarial network provided in an embodiment of the present application.

[0169] As shown in Figure 7, the generative adversarial network includes a generator and a discriminator, wherein the generator includes a second neural network used by the encoder and a first neural network used by the decoder. The input of the generator contains attribute information of the point cloud (such as the voxelized point cloud), and the output is the reconstructed attribute information of the point cloud (such as the reconstructed voxelized point cloud). Specifically, the attribute information of the point cloud is first represented by tensor data, which is then compressed into a bit stream through the second neural network. The decoder decodes the bit stream to determine the first tensor data, and then uses the first neural network to convert the first tensor data into second tensor data. The second tensor data is used to determine the reconstructed attribute information of the point cloud. The discriminator compares the attribute information of the point cloud with the reconstructed attribute information of the point cloud and returns the result to assist in training the generator until the reconstructed attribute information of the point cloud is as consistent as possible with the attribute information of the point cloud.

[0170] The second neural network used by the encoder can have three convolutional layers and two activation layers, with kernels of 9×9×9, 5×5×5, and 5×5×5, respectively, and contain 128 channels. The first neural network used by the decoder can have three transposed convolutional layers and two activation layers, with the parameters of the second neural network shared with those of the first. The discriminator is essentially a classification network that distinguishes whether the reconstructed attribute information of the point cloud is consistent with the attribute information of the point cloud. The discriminator's network structure can be consistent with the second neural network, using sparse convolution to extract features and finally using a fully connected layer to output one-dimensional data to represent the real / fake status of the reconstructed point cloud data.

[0171] It is worth noting that FIG7 is merely an example of the present application and should not be construed as a limitation to the present application.

[0172] For example, since the accompanying diagrams are examples from the perspective of training a generative adversarial network, only the second neural network used by the encoder and the first neural network used by the decoder are shown as examples. However, in the actual encoding and decoding process, the encoder also includes modules for quantizing and arithmetic encoding the second neural network, and the decoder also includes modules for arithmetic decoding and inverse quantization of the bitstream, and may even include filtering or other modules.

[0173] The decoding method according to the embodiment of the present application is described in detail above from the perspective of a decoder. The encoding method according to the embodiment of the present application will be described below from the perspective of an encoder in conjunction with FIG. 8 .

[0174] FIG8 is a schematic flow chart of an encoding method 200 provided in an embodiment of the present application. It should be understood that the encoding method 200 can be executed by an encoder.

[0175] As shown in FIG8 , the encoding method 200 may include:

[0176] S210, the encoder determines third tensor data of the point cloud based on the attribute information of the point cloud; the third tensor data includes feature values ​​of the attribute information of the point cloud on at least one second channel.

[0177] At S220, the encoder uses a second neural network to determine first tensor data based on the third tensor data; the second neural network and the first neural network used by the decoder are used to form a generator of a generative adversarial network; the first tensor data includes feature values ​​of at least one first channel of the point cloud attribute information after feature extraction; the at least one first channel is different from the at least one second channel.

[0178] S230: The encoder determines a code stream of the point cloud based on the first tensor data.

[0179] In some embodiments, the at least one second channel corresponds to at least one component of attribute information of the point cloud.

[0180] In some embodiments, the third tensor data includes the eigenvalues ​​of the attribute information of the voxels in the point cloud on the at least one second channel, and the first tensor data includes the eigenvalues ​​of the attribute information of the voxels on the at least one first channel after feature extraction.

[0181] In some embodiments, the S210 may include:

[0182] Dividing the point cloud into at least one point cloud block;

[0183] Determining the type of the point cloud block;

[0184] voxelize the point cloud block based on a resolution corresponding to the type of the point cloud block, and determine attribute information of the voxels in the point cloud;

[0185] The third tensor data is determined based on the attribute information of the voxel.

[0186] In this embodiment, the encoder voxelizes the point cloud block based on the resolution corresponding to the type of the point cloud block, which can preserve the point cloud information as much as possible and reduce the computational complexity of the encoder, thereby improving the encoding performance of the encoder.

[0187] The encoder can pre-train a voxelization module for adaptive voxel resolution. This voxelization module can voxelize the input raw point cloud using multiple voxel resolutions (e.g., high and low resolutions) depending on the density of the point cloud, resulting in a voxelized point cloud. During the adaptive voxelization process, the input raw point cloud is divided into multiple blocks. For each block, a label is predicted to indicate which points belong to a relatively sparse block and which to a relatively dense block. For sparse blocks, voxelization A (e.g., 8×8×8 resolution) can be used for voxelization; for dense blocks, voxelization B (e.g., 16×16×16 resolution) can be used for voxelization. As shown in Figure 9, this voxelization module can be implemented as a neural network consisting of multiple layers of 3D convolutional layers and activation layers. The final layer uses a classification function (e.g., a sigmoid function) to label the output blocks as sparse or dense.

[0188] The loss function of the voxelization module includes the loss for supervising the labeling process of sparse blocks and the loss for supervising the labeling process of dense blocks. The mathematical formula of the standard binary cross entropy (BCE) loss is: Among them, y is the true label, The predicted label generated by the model. Further, combining the above formula with the voxelization module provided in this application, it can be seen that the loss of the labeling process for supervising sparse blocks and the loss of the labeling process for supervising dense blocks can be expressed as: loss sparse =BCE(M sparse ,GT sparse ); loss dense =BCE(M dense ,GT dense ).

[0189] The total loss function is: loss =α·loss sparse +(1-α)·loss dense .

[0190] Among them, loss sparse represents the loss of the labeling process used to supervise the sparse blocks, loss dense represents the loss of the labeling process used to supervise the dense block, total loss represents the loss of the voxelization module (i.e., voxelization loss), M sparse Represents the true label of the sparse block, GT sparse represents the predicted label of the sparse block, M dense Represents the true label of the dense block, GT dense represents the predicted label of the dense block, and α represents the loss sparse The weight of .

[0191] In some embodiments, the S230 may include:

[0192] The first tensor data is quantized and encoded to obtain the code stream.

[0193] In some embodiments, the second neural network includes at least one feature extraction layer, the last feature extraction layer in the at least one feature extraction layer includes a convolutional layer, and the feature extraction layers in the at least one feature extraction layer other than the last feature extraction layer include a convolutional layer and an activation layer.

[0194] In some embodiments, the position of the convolution layer in the second neural network is the same as the position of the deconvolution layer in the first neural network, and the position of the activation layer in the second neural network is the same as the position of the activation layer in the first neural network.

[0195] In some embodiments, parameters of the convolutional layers in the second neural network are the same as parameters of the deconvolutional layers in the first neural network.

[0196] It should be understood that the encoding method can be understood as the inverse process of the decoding method. Therefore, the specific scheme of the encoding method 200 can refer to the relevant content of the decoding method 100. For the convenience of description, this application will not go into details.

[0197] The above describes in detail the encoding and decoding method according to the embodiment of the present application from the perspective of the decoder and the encoder. Now, in combination with Figure 10, the training method of the generative adversarial network provided in the embodiment of the present application will be described from the perspective of the generative adversarial network.

[0198] FIG10 is a schematic flowchart of a training method 300 for a generative adversarial network provided in an embodiment of the present application.

[0199] It should be understood that the training method 300 can be executed by any electronic device with data processing capabilities.

[0200] In addition, the training method 300 provided in this application can be incorporated into the decoding method 100 or the encoding method 200. For example, the training device uses the training method 300 to train a generative adversarial network; the generative adversarial network includes a generator and a discriminator, and the generator includes a first neural network and a second neural network. The decoder can use the first neural network to convert the first tensor data determined by decoding the point cloud code stream into the second tensor data used to reconstruct the attribute information. Correspondingly, the encoder can use the second neural network to convert the third tensor data determined based on the attribute information of the point cloud into the first tensor data used to determine the point cloud code stream.

[0201] In addition, when the generative adversarial network is trained, the present application does not specifically limit the data set adopted by it. For example, the training of the generative adversarial network can use ShapeNet and COCO data sets. Specifically, dense points are collected on the ShapeNet grid, and their coordinates are randomly rotated and quantized to 8-bit integers. For color attributes, random images in COCO are projected onto these points, and a total of sample sets (e.g., 8500 samples) are generated. Here, it is divided into two parts, one of which (e.g., 6000 samples) is used for training, and the other part (e.g., 2500 samples) is used for testing. In order to prevent overfitting, samples in the ModelNet40 data set (e.g., 11000 samples) can also be added, and the same ratio as above is used for training and testing.

[0202] As shown in FIG10 , the training method 300 may include:

[0203] S310, determining third tensor data of the point cloud based on attribute information of the point cloud; the third tensor data includes feature values ​​of the attribute information of the point cloud on at least one second channel;

[0204] S320, based on the third tensor data, using a second neural network included in a generator in a generative adversarial network to determine first tensor data; the first tensor data includes feature values ​​of at least one first channel after feature extraction of attribute information of the point cloud; the at least one first channel is different from the at least one second channel;

[0205] S330: Determine second tensor data based on the first tensor data using a first neural network included in a generator; the second tensor data includes feature values ​​of the reconstructed attribute information of the point cloud on at least one second channel;

[0206] S340, determining reconstruction attribute information of the point cloud based on the second tensor data;

[0207] S350 , based on the attribute information of the point cloud and the reconstructed attribute information of the point cloud, using a discriminator in the generative adversarial network, train the generative adversarial network.

[0208] In some embodiments, the S350 may include:

[0209] determining a distortion value between the attribute information of the point cloud and the reconstructed attribute information of the point cloud based on the attribute information of the point cloud and the reconstructed attribute information of the point cloud;

[0210] Determining a loss value of the generative adversarial network based on a bit rate of a code stream of the point cloud and the distortion value; the code stream of the point cloud is a code stream obtained by arithmetic coding and quantization of the first tensor data;

[0211] The generative adversarial network is trained based on the loss value of the generative adversarial network.

[0212] In some embodiments, the loss value of the generator is determined based on the occupancy information of the voxels in the point cloud; the loss value of the discriminator is determined based on the attribute information of the point cloud and the reconstruction attribute information of the point cloud; and the distortion value is determined based on the loss value of the generator and the loss value of the discriminator.

[0213] In some embodiments, the loss value of the generator is determined according to the following formula:

[0214] in, represents the loss value of the generator, Indicates the first The weight of the voxel, Indicates the The occupancy information of the voxel, ξ represents the focusing parameter.

[0215] For example, Indicates the The occupancy information of the voxel; in other words, the The occupancy information of the voxels can be used to determine For example, the value of The occupancy information of the voxel indicates the occupancy time The value of is 1, the When the occupancy information of a voxel indicates that it is not occupied The value of is 0. For example, The occupancy information of the voxel indicates the occupancy time The value of is 0, the When the occupancy information of a voxel indicates that it is not occupied The value of is 1.

[0216] For example, It is used to control the proportion of positive samples in the training set, and its value range is [0,1]. The appropriate value of can generally be selected through cross-validation.

[0217] For example, ξ ranges from [0 to +∞). The goal is to reduce the weight of easy-to-classify examples, thereby allowing the generator to focus more on difficult examples during training. When ξ = 0, the generator's loss value degenerates into cross-entropy loss. The larger ξ is, the greater the penalty for easy-to-classify examples.

[0218] In some embodiments, the loss value of the discriminator is determined according to the following formula:

[0219] in, represents the loss value of the discriminator, E[·] represents the expected value operation, m represents the attribute information of the point cloud, represents the probability distribution of the attribute information of the point cloud, n represents the reconstructed attribute information of the point cloud, Represents the probability distribution of the reconstructed attribute information of the point cloud.

[0220] In some embodiments, the distortion value is determined according to the following formula:

[0221] in, represents the distortion value, represents the loss value of the discriminator, represents the weight of the loss value of the discriminator, represents the loss value of the generator, represents the weight representing the loss value of the generator.

[0222] In some embodiments, the loss value of the generative adversarial network is determined according to the following formula:

[0223] in, represents the loss value of the generative adversarial network, represents the distortion value, λ represents the distortion weight, Indicates the bit rate.

[0224] The preferred embodiments of the present application are described in detail above in conjunction with the accompanying drawings. However, the present application is not limited to the specific details in the embodiments mentioned above. Within the technical concept of the present application, the technical solution of the present application can be subjected to a variety of simple modifications, and these simple modifications all fall within the scope of protection of the present application. For example, the various specific technical features described in the specific embodiments mentioned above can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present application will no longer describe the various possible combinations separately. For another example, the various different embodiments of the present application can also be arbitrarily combined, as long as they do not violate the ideas of the present application, they should also be regarded as the contents disclosed in the present application. It should also be understood that in the various method embodiments of the present application, the size of the sequence numbers of the processes mentioned above does not mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0225] The method embodiment of the present application is described in detail above. The following describes in detail an apparatus embodiment of the present application in conjunction with Figures 11 to 14.

[0226] FIG11 is a schematic block diagram of a decoder 400 provided in an embodiment of the present application.

[0227] As shown in FIG11 , the decoder 400 may include:

[0228] The decoding unit 410 is configured to decode a bit stream of the point cloud and determine first tensor data; the first tensor data includes feature values ​​of at least one first channel of attribute information of the point cloud after feature extraction;

[0229] A first determining unit 420 is configured to determine second tensor data using a first neural network based on the first tensor data; the first neural network and a second neural network used by the encoder are used to form a generator of a generative adversarial network; the second tensor data includes feature values ​​of the reconstructed attribute information of the point cloud on at least one second channel; the at least one first channel is different from the at least one second channel;

[0230] The second determining unit 430 is configured to determine the reconstructed attribute information of the point cloud based on the second tensor data.

[0231] In some embodiments, the at least one second channel corresponds to at least one component of attribute information of the point cloud.

[0232] In some embodiments, the first tensor data includes feature values ​​of the attribute information of the voxels in the point cloud on the at least one first channel after feature extraction, and the second tensor data includes feature values ​​of the reconstructed attribute information of the voxels on the at least one second channel.

[0233] In some embodiments, the decoding unit 410 is specifically configured to:

[0234] Dequantize the tensor data obtained by decoding the code stream of the point cloud to obtain the first tensor data.

[0235] In some embodiments, the first neural network includes at least one feature extraction layer, the last feature extraction layer in the at least one feature extraction layer includes a deconvolution layer, and the feature extraction layers in the at least one feature extraction layer other than the last feature extraction layer include a deconvolution layer and an activation layer.

[0236] In some embodiments, the position of the deconvolution layer in the first neural network is the same as the position of the convolution layer in the second neural network, and the position of the activation layer in the first neural network is the same as the position of the activation layer in the second neural network.

[0237] In some embodiments, parameters of the deconvolution layer in the first neural network are the same as parameters of the convolution layer in the second neural network.

[0238] FIG12 is a schematic block diagram of an encoder 500 provided in an embodiment of the present application.

[0239] As shown in FIG12 , the encoder 500 may include:

[0240] A first determining unit 510 is configured to determine third tensor data of the point cloud based on the attribute information of the point cloud; the third tensor data includes a feature value of the attribute information of the point cloud in at least one second channel;

[0241] A second determining unit 520 is configured to determine first tensor data using a second neural network based on the third tensor data; the second neural network and the first neural network used by the decoder are used to form a generator of a generative adversarial network; the first tensor data includes feature values ​​of at least one first channel after feature extraction of attribute information of the point cloud; the at least one first channel is different from the at least one second channel;

[0242] The third determining unit 530 is configured to determine a code stream of the point cloud based on the first tensor data.

[0243] In some embodiments, the at least one second channel corresponds to at least one component of attribute information of the point cloud.

[0244] In some embodiments, the third tensor data includes the eigenvalues ​​of the attribute information of the voxels in the point cloud on the at least one second channel, and the first tensor data includes the eigenvalues ​​of the attribute information of the voxels on the at least one first channel after feature extraction.

[0245] In some embodiments, the first determining unit 510 is specifically configured to:

[0246] Dividing the point cloud into at least one point cloud block;

[0247] Determining the type of the point cloud block;

[0248] voxelize the point cloud block based on a resolution corresponding to the type of the point cloud block, and determine attribute information of the voxels in the point cloud;

[0249] The third tensor data is determined based on the attribute information of the voxel.

[0250] In some embodiments, the third determining unit 530 is specifically configured to:

[0251] The first tensor data is quantized and encoded to obtain the code stream.

[0252] In some embodiments, the second neural network includes at least one feature extraction layer, the last feature extraction layer in the at least one feature extraction layer includes a convolutional layer, and the feature extraction layers in the at least one feature extraction layer other than the last feature extraction layer include a convolutional layer and an activation layer.

[0253] In some embodiments, the position of the convolution layer in the second neural network is the same as the position of the deconvolution layer in the first neural network, and the position of the activation layer in the second neural network is the same as the position of the activation layer in the first neural network.

[0254] In some embodiments, parameters of the convolutional layers in the second neural network are the same as parameters of the deconvolutional layers in the first neural network.

[0255] Figure 13 is a schematic block diagram of a training device 600 for a generative adversarial network provided in an embodiment of the present application.

[0256] As shown in FIG13 , the training device 600 may include:

[0257] A first determining unit 610 is configured to determine third tensor data of the point cloud based on the attribute information of the point cloud; the third tensor data includes a feature value of the attribute information of the point cloud in at least one second channel;

[0258] A second determining unit 620 is configured to determine first tensor data based on the third tensor data using a second neural network included in a generator in a generative adversarial network; the first tensor data includes feature values ​​of at least one first channel after feature extraction of attribute information of the point cloud; the at least one first channel is different from the at least one second channel;

[0259] A third determining unit 630 is configured to determine second tensor data based on the first tensor data using a first neural network included in the generator; the second tensor data includes a feature value of the reconstructed attribute information of the point cloud on at least one second channel;

[0260] a fourth determining unit 640, configured to determine reconstruction attribute information of the point cloud based on the second tensor data;

[0261] The training unit 650 is configured to train the generative adversarial network based on the attribute information of the point cloud and the reconstructed attribute information of the point cloud using the discriminator in the generative adversarial network.

[0262] In some embodiments, the training unit 650 is specifically configured to:

[0263] determining a distortion value between the attribute information of the point cloud and the reconstructed attribute information of the point cloud based on the attribute information of the point cloud and the reconstructed attribute information of the point cloud;

[0264] Determining a loss value of the generative adversarial network based on a bit rate of a code stream of the point cloud and the distortion value; the code stream of the point cloud is a code stream obtained by arithmetic coding and quantization of the first tensor data;

[0265] The generative adversarial network is trained based on the loss value of the generative adversarial network.

[0266] In some embodiments, the training unit 650 is specifically configured to:

[0267] Determining a loss value of the generator based on occupancy information of voxels in the point cloud;

[0268] determining a loss value of the discriminator based on the attribute information of the point cloud and the reconstructed attribute information of the point cloud;

[0269] The distortion value is determined based on the loss value of the generator and the loss value of the discriminator.

[0270] In some embodiments, the training unit 650 is specifically configured to:

[0271] The loss value of the generator is determined according to the following formula:

[0272] in, represents the loss value of the generator, Indicates the first The weight of the voxel, Indicates the The occupancy information of the voxel, ξ represents the focusing parameter.

[0273] In some embodiments, the training unit 650 is specifically configured to:

[0274] The loss value of the discriminator is determined according to the following formula:

[0275] in, represents the loss value of the discriminator, E[·] represents the expected value operation, m represents the attribute information of the point cloud, represents the probability distribution of the attribute information of the point cloud, n represents the reconstructed attribute information of the point cloud, Represents the probability distribution of the reconstructed attribute information of the point cloud.

[0276] In some embodiments, the training unit 650 is specifically configured to:

[0277] The distortion value is determined according to the following formula:

[0278] in, represents the distortion value, represents the loss value of the discriminator, represents the weight of the loss value of the discriminator, represents the loss value of the generator, represents the weight representing the loss value of the generator.

[0279] In some embodiments, the training unit 650 is specifically configured to:

[0280] The loss value of the generative adversarial network is determined according to the following formula:

[0281] in, represents the loss value of the generative adversarial network, represents the distortion value, λ represents the distortion weight, Indicates the bit rate.

[0282] It should be understood that the device embodiment and the method embodiment can correspond to each other, and similar descriptions can refer to the method embodiment. To avoid repetition, it will not be described here. Specifically, the decoder 400 shown in Figure 11 can correspond to the corresponding subject in the method 100 for executing the embodiment of the present application, and the aforementioned and other operations and / or functions of the various units in the decoder 400 are respectively for implementing the corresponding processes in the various methods such as method 100. The encoder 500 shown in Figure 12 can correspond to the corresponding subject in the method 200 for executing the embodiment of the present application, that is, the aforementioned and other operations and / or functions of the various units in the encoder 500 are respectively for implementing the corresponding processes in the various methods such as method 200. The training device 600 shown in Figure 13 can correspond to the corresponding subject in the method 300 for executing the embodiment of the present application, that is, the aforementioned and other operations and / or functions of the various units in the training device 600 are respectively for implementing the corresponding processes in the various methods such as method 300.

[0283] It should also be understood that the various units in the decoder 400, encoder 500 or training device 600 involved in the embodiments of the present application can be separately or completely merged into one or several other units to constitute, or one (or some) of the units can be further split into multiple functionally smaller units to constitute, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The units involved above are divided based on logical functions. In actual applications, the functions of one unit can also be implemented by multiple units, or the functions of multiple units can be implemented by one unit. In other embodiments of the present application, the decoder 400, encoder 500 or training device 600 may also include other units. In actual applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units. According to another embodiment of the present application, a computer program (including program code) capable of executing each step involved in the corresponding method can be run on a general-purpose computing device of a general-purpose computer including processing elements and storage elements such as a central processing unit (CPU), a random access storage medium (RAM), and a read-only storage medium (ROM). To construct the decoder 400, encoder 500, or training device 600 involved in the embodiment of the present application, and to implement the encoding method, decoding method, or training method of the embodiment of the present application. The computer program can be recorded on, for example, a computer-readable storage medium, and loaded into an electronic device through a computer-readable storage medium, and run therein to implement the corresponding method of the embodiment of the present application.

[0284] In other words, the units mentioned above can be implemented in the form of hardware, can be implemented by instructions in the form of software, or can be implemented in the form of a combination of hardware and software. Specifically, the steps of the method embodiments in the embodiments of the present application can be completed by the hardware integrated logic circuit and / or software instructions in the processor, and the steps of the method disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware decoding processor, or being executed by a combination of hardware and software in a decoding processor. Optionally, the software can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps in the method embodiments mentioned above in combination with its hardware.

[0285] FIG14 is a schematic structural diagram of an electronic device 700 provided in an embodiment of the present application.

[0286] As shown in Figure 14, the electronic device 700 includes at least a processor 710 and a computer-readable storage medium 720. The processor 710 and the computer-readable storage medium 720 may be connected via a bus or other means. The computer-readable storage medium 720 is used to store a computer program 721, which includes computer instructions. The processor 710 is used to execute the computer instructions stored in the computer-readable storage medium 720. The processor 710 is the computing core and control core of the electronic device 700. It is suitable for implementing one or more computer instructions, and is specifically suitable for loading and executing one or more computer instructions to implement the corresponding method flow or corresponding function.

[0287] For example, the processor 710 may also be referred to as a central processing unit (CPU). The processor 710 may include, but is not limited to, a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, discrete hardware components, and the like.

[0288] Exemplarily, the computer-readable storage medium 720 may be a high-speed RAM memory, or a non-volatile memory (Non-Volatile Memory), such as at least one disk memory; optionally, it may also be at least one computer-readable storage medium located away from the aforementioned processor 710. Specifically, the computer-readable storage medium 720 includes, but is not limited to: volatile memory and / or non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0289] Exemplarily, the electronic device 700 may be an encoder or encoding framework involved in an embodiment of the present application; a first computer instruction is stored in the computer-readable storage medium 720; the processor 710 loads and executes the first computer instruction stored in the computer-readable storage medium 720 to implement the corresponding steps in the encoding method provided in an embodiment of the present application; in other words, the first computer instruction in the computer-readable storage medium 720 is loaded by the processor 710 and the corresponding steps are executed. To avoid repetition, they are not repeated here.

[0290] Exemplarily, the electronic device 700 may be a decoder or decoding framework involved in an embodiment of the present application; a second computer instruction is stored in the computer-readable storage medium 720; the processor 710 loads and executes the second computer instruction stored in the computer-readable storage medium 720 to implement the corresponding steps in the decoding method provided in an embodiment of the present application; in other words, the second computer instruction in the computer-readable storage medium 720 is loaded by the processor 710 and the corresponding steps are executed, which will not be repeated here to avoid repetition.

[0291] Exemplarily, the electronic device 700 may be the training device involved in the embodiment of the present application; the computer-readable storage medium 720 stores a third computer instruction; the processor 710 loads and executes the third computer instruction stored in the computer-readable storage medium 720 to implement the corresponding steps in the training method provided in the embodiment of the present application; in other words, the third computer instruction in the computer-readable storage medium 720 is loaded by the processor 710 and the corresponding steps are executed, which will not be repeated here to avoid repetition.

[0292] According to another aspect of the present application, the present application also provides a coding and decoding system, including the encoder and decoder mentioned above.

[0293] According to another aspect of the present application, the present application also provides a computer-readable storage medium (Memory), which is a memory device in the electronic device 700 for storing programs and data. For example, a computer-readable storage medium 720. It can be understood that the computer-readable storage medium 720 here can include both the built-in storage medium in the electronic device 700 and, of course, the extended storage medium supported by the electronic device 700. The computer-readable storage medium provides a storage space, which stores the operating system of the electronic device 700. In addition, one or more computer instructions suitable for being loaded and executed by the processor 710 are also stored in the storage space. These computer instructions can be one or more computer programs 721 (including program codes).

[0294] According to another aspect of the present application, the present application further provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. For example, computer program 721. In this case, data processing device 700 may be a computer, processor 710 reads the computer instructions from computer-readable storage medium 720, and processor 710 executes the computer instructions, causing the computer to perform the encoding method, decoding method, or training method provided in the various optional embodiments mentioned above.

[0295] In other words, when implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process of the embodiment of the present application is run in whole or in part or the functions of the embodiment of the present application are implemented. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode.

[0296] Those skilled in the art will appreciate that the units and process steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0297] Finally, it should be noted that the above content is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A decoding method, characterized in that: include: Decode the code stream of the point cloud and determine the first tensor data; The first tensor data includes feature values ​​of the attribute information of the point cloud on at least one first channel after feature extraction; Based on the first tensor data, second tensor data is determined using a first neural network; the first neural network and a second neural network used by the encoder are used to form a generator of a generative adversarial network; the second tensor data includes a feature value of the reconstructed attribute information of the point cloud on at least one second channel; The at least one first channel is different from the at least one second channel; Based on the second tensor data, reconstructed attribute information of the point cloud is determined.

2. The method according to claim 1, characterized in that The at least one second channel corresponds to at least one component of the attribute information of the point cloud.

3. The method according to claim 1 or 2, characterized in that: The first tensor data includes feature values ​​of the attribute information of the voxels in the point cloud on the at least one first channel after feature extraction, and the second tensor data includes feature values ​​of the reconstructed attribute information of the voxels on the at least one second channel.

4. The method according to any one of claims 1 to 3, characterized in that The decoding of the point cloud code stream to determine the first tensor data includes: Dequantize the tensor data obtained by decoding the code stream of the point cloud to obtain the first tensor data.

5. The method according to claim 4, characterized in that The first neural network includes at least one feature extraction layer, the last feature extraction layer in the at least one feature extraction layer includes a deconvolution layer, and the feature extraction layers in the at least one feature extraction layer except the last feature extraction layer include a deconvolution layer and an activation layer.

6. The method according to claim 4, characterized in that The position of the deconvolution layer in the first neural network is the same as the position of the convolution layer in the second neural network, and the position of the activation layer in the first neural network is the same as the position of the activation layer in the second neural network.

7. The method according to claim 4, characterized in that The parameters of the deconvolution layer in the first neural network are the same as the parameters of the convolution layer in the second neural network.

8. A coding method, characterized in that: include: Determining third tensor data of the point cloud based on attribute information of the point cloud; The third tensor data includes a feature value of the attribute information of the point cloud on at least one second channel; Based on the third tensor data, the first tensor data is determined using a second neural network; the second neural network and the first neural network used by the decoder are used to form a generator of a generative adversarial network; the first tensor data includes a feature value of the attribute information of the point cloud on at least one first channel after feature extraction; The at least one first channel is different from the at least one second channel; Based on the first tensor data, a code stream of the point cloud is determined.

9. The method according to claim 8, characterized in that The at least one second channel corresponds to at least one component of the attribute information of the point cloud.

10. The method according to claim 8 or 9, characterized in that: The third tensor data includes feature values ​​of attribute information of voxels in the point cloud on the at least one second channel, and the first tensor data includes feature values ​​of attribute information of the voxels on the at least one first channel after feature extraction.

11. The method according to claim 10, characterized in that The determining of the third tensor data of the point cloud based on the attribute information of the point cloud includes: Dividing the point cloud into at least one point cloud block; Determine the type of the point cloud block; voxelize the point cloud block based on a resolution corresponding to the type of the point cloud block, and determine attribute information of the voxels in the point cloud; The third tensor data is determined based on the attribute information of the voxel.

12. The method according to any one of claims 8 to 11, characterized in that The determining, based on the first tensor data, a code stream of the point cloud includes: The first tensor data is quantized and encoded to obtain the code stream.

13. The method according to claim 12, characterized in that The second neural network includes at least one feature extraction layer, the last feature extraction layer in the at least one feature extraction layer includes a convolutional layer, and the feature extraction layers in the at least one feature extraction layer except the last feature extraction layer include a convolutional layer and an activation layer.

14. The method according to claim 12, characterized in that The position of the convolution layer in the second neural network is the same as the position of the deconvolution layer in the first neural network, and the position of the activation layer in the second neural network is the same as the position of the activation layer in the first neural network.

15. The method according to claim 12, characterized in that The parameters of the convolutional layer in the second neural network are the same as the parameters of the deconvolutional layer in the first neural network.

16. A training method for generating an adversarial network, characterized in that: include: Determining third tensor data of the point cloud based on attribute information of the point cloud; The third tensor data includes a feature value of the attribute information of the point cloud on at least one second channel; Based on the third tensor data, a first tensor data is determined using a second neural network included in a generator in a generative adversarial network; the first tensor data includes a feature value of the attribute information of the point cloud on at least one first channel after feature extraction; The at least one first channel is different from the at least one second channel; Based on the first tensor data, second tensor data is determined using a first neural network included in the generator; the second tensor data includes a feature value of the reconstructed attribute information of the point cloud on at least one second channel; Determining reconstruction attribute information of the point cloud based on the second tensor data; Based on the attribute information of the point cloud and the reconstructed attribute information of the point cloud, the generative adversarial network is trained using a discriminator in the generative adversarial network.

17. The method according to claim 16, characterized in that The method of training the generative adversarial network based on the attribute information of the point cloud and the reconstructed attribute information of the point cloud by using a discriminator in the generative adversarial network comprises: Determining a distortion value between the attribute information of the point cloud and the reconstructed attribute information of the point cloud based on the attribute information of the point cloud and the reconstructed attribute information of the point cloud; Determining a loss value of the generative adversarial network based on a bit rate of a code stream of the point cloud and the distortion value; the code stream of the point cloud is a code stream obtained by arithmetic coding and quantization of the first tensor data; Based on the loss value of the generative adversarial network, the generative adversarial network is trained.

18. The method according to claim 17, characterized in that The determining, based on the attribute information of the point cloud and the reconstructed attribute information of the point cloud, a distortion value between the attribute information of the point cloud and the reconstructed attribute information of the point cloud comprises: Determining a loss value of the generator based on occupancy information of voxels in the point cloud; Determining a loss value of the discriminator based on the attribute information of the point cloud and the reconstructed attribute information of the point cloud; The distortion value is determined based on the loss value of the generator and the loss value of the discriminator.

19. The method according to claim 18, characterized in that The step of determining the loss value of the generator based on the occupancy information of the voxels in the point cloud comprises: The loss value of the generator is determined according to the following formula: in, represents the loss value of the generator, Indicates the point cloud The weight of the voxel, Indicates the The occupancy information of the voxel, ξ represents the focusing parameter.

20. The method according to claim 18, characterized in that The determining the loss value of the discriminator based on the attribute information of the point cloud and the reconstructed attribute information of the point cloud includes: The loss value of the discriminator is determined according to the following formula: in, represents the loss value of the discriminator, E[·] represents the expected value operation, m represents the attribute information of the point cloud, represents the probability distribution of the attribute information of the point cloud, n represents the reconstructed attribute information of the point cloud, Represents the probability distribution of the reconstructed attribute information of the point cloud.

21. The method according to claim 18, characterized in that The determining the distortion value based on the loss value of the generator and the loss value of the discriminator includes: The distortion value is determined according to the following formula: Wherein, x represents the distortion value, represents the loss value of the discriminator, represents the weight of the loss value of the discriminator, represents the loss value of the generator, represents the weight representing the loss value of the generator.

22. The method according to any one of claims 17 to 21, characterized in that The step of determining the loss value of the generative adversarial network based on the bit rate of the code stream of the point cloud and the distortion value includes: The loss value of the generative adversarial network is determined according to the following formula: in, represents the loss value of the generative adversarial network, x represents the distortion value, λ represents the distortion weight, and y represents the ratio Special rate.

23. A decoding method, characterized in that: include: Decode the code stream of the point cloud and determine the first tensor data; The first tensor data includes feature values ​​of the attribute information of the point cloud on at least one first channel after feature extraction; Based on the first tensor data, a second tensor data is determined using a first neural network; the first neural network and a second neural network used by the encoder are used to form a generator of a generative adversarial network; the second tensor data includes a feature value of the reconstructed attribute information of the point cloud on at least one second channel; the at least one first channel is different from the at least one second channel; the generative adversarial network is trained by the method according to any one of claims 16 to 22; Based on the second tensor data, reconstructed attribute information of the point cloud is determined.

24. A coding method, characterized in that include: Determining third tensor data of the point cloud based on attribute information of the point cloud; The third tensor data includes a feature value of the attribute information of the point cloud on at least one second channel; Based on the third tensor data, the first tensor data is determined by using a second neural network; the second neural network and the first neural network used by the decoder are used to form a generator of a generative adversarial network; the first tensor data includes a feature value of the attribute information of the point cloud on at least one first channel after feature extraction; the at least one first channel is different from the at least one second channel; the generative adversarial network is trained according to the method described in any one of claims 16 to 22; Based on the first tensor data, a code stream of the point cloud is determined.

25. A decoder, characterized in that: include: A decoding unit, used for decoding a code stream of the point cloud to determine the first tensor data; The first tensor data includes feature values ​​of the attribute information of the point cloud on at least one first channel after feature extraction; A first determining unit is used to determine second tensor data using a first neural network based on the first tensor data; the first neural network and a second neural network used by the encoder are used to form a generator of a generative adversarial network; the second tensor data includes a feature value of the reconstructed attribute information of the point cloud on at least one second channel; The at least one first channel is different from the at least one second channel; The second determining unit is used to determine the reconstruction attribute information of the point cloud based on the second tensor data.

26. An encoder, characterized in that include: A first determining unit, configured to determine third tensor data of the point cloud based on attribute information of the point cloud; The third tensor data includes a feature value of the attribute information of the point cloud on at least one second channel; A second determination unit is used to determine the first tensor data using a second neural network based on the third tensor data; the second neural network and the first neural network used by the decoder are used to form a generator of a generative adversarial network; the first tensor data includes a feature value of the attribute information of the point cloud on at least one first channel after feature extraction; The at least one first channel is different from the at least one second channel; The third determining unit is used to determine a code stream of the point cloud based on the first tensor data.

27. A training device for generating an adversarial network, characterized in that: include: A first determining unit, configured to determine third tensor data of the point cloud based on attribute information of the point cloud; The third tensor data includes a feature value of the attribute information of the point cloud on at least one second channel; A second determination unit is used to determine first tensor data based on the third tensor data by using a second neural network included in the generator in the generative adversarial network; the first tensor data includes a feature value of the attribute information of the point cloud on at least one first channel after feature extraction; The at least one first channel is different from the at least one second channel; A third determining unit is used to determine second tensor data based on the first tensor data using a first neural network included in the generator; the second tensor data includes a feature value of the reconstructed attribute information of the point cloud on at least one second channel; a fourth determining unit, configured to determine reconstruction attribute information of the point cloud based on the second tensor data; A training unit is used to train the generative adversarial network based on the attribute information of the point cloud and the reconstructed attribute information of the point cloud using a discriminator in the generative adversarial network.

28. An electronic device, characterized in that: include: a processor adapted to execute a computer program; A computer-readable storage medium having a computer program stored therein, wherein when the computer program is executed by the processor, at least one of the following methods is performed: the method according to any one of claims 1 to 7, the method according to any one of claims 8 to 15, the method according to any one of claims 16 to 22, the method according to claim 23, and the method according to claim 24.

29. A computer-readable storage medium, characterized in that: Used to store a computer program, which, when executed on a computer, causes the computer to perform at least one of the following methods: the method according to any one of claims 1 to 7, the method according to any one of claims 8 to 15, the method according to any one of claims 16 to 22, the method according to claim 23, and the method according to claim 24.

30. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, at least one of the following methods is implemented: the method according to any one of claims 1 to 7, the method according to any one of claims 8 to 15, the method according to any one of claims 16 to 22, the method according to claim 23, and the method according to claim 24.

31. A code stream, characterized in that The code stream is a code stream decoded according to the method according to any one of claims 1 to 7 or the method according to claim 23, or the code stream is a code stream generated by the method according to any one of claims 8 to 15 or the method according to claim 24.

Images