Decoding method, decoder and storage medium

Abstract

The invention discloses a decoding method, a decoder and a storage medium, and the method comprises the steps: carrying out the image enhancement of a current point cloud decompression two-dimensional image (including at least one of an occupied image, a geometric image and an attribute image), obtaining an enhanced two-dimensional image, carrying out the point cloud reconstruction through employing the enhanced two-dimensional image, and obtaining a point cloud reconstruction result. The three-dimensional reconstruction point cloud quality can be improved, and the decoder performance is improved under the condition that the point cloud code rate is not changed. Furthermore, a new loss function is provided for an image enhancement module based on a neural network, so that a geometric image enhancement module and / or an attribute image enhancement module can pay attention to enhancement of quality of occupied points, neglect enhancement of quality of non-occupied points, and improve geometric reconstruction quality and / or attribute reconstruction quality of point cloud; and the three-dimensional reconstruction point cloud quality is further improved.

Description

A decoding method, decoder and storage medium Technical Field

[0001] The embodiments of the present application relate to the field of point cloud data processing technology, and in particular to a decoding method, a decoder, and a storage medium. Background Art

[0002] Video-based Point Cloud Compression (V-PCC) aims to compress the geometric shape and attribute information of dynamic point clouds using existing video codecs, such as High Efficiency Video Coding (HEVC). Specifically, at the encoding end, V-PCC first projects the three-dimensional (3D) point cloud into a set of geometric images and attribute images, and then uses HEVC to compress the two-dimensional (2D) geometric images and attribute images to form corresponding geometric streams and attribute streams. At the decoding end, V-PCC decodes the geometric stream and attribute stream, and then reconstructs the 3D point cloud based on the correspondence between the geometric information and the attribute information.

[0003] However, for lossy point cloud compression, V-PCC uses two-dimensional images to reconstruct three-dimensional point clouds. This inevitably introduces distortion in the reconstructed point cloud, affecting the quality of the reconstructed point cloud sequence. Therefore, as the demand for point cloud quality continues to increase, further improvements are needed in point cloud encoding and decoding technology.

[0004] Summary of the Invention

[0005] The embodiments of the present application provide a decoding method, a decoder, and a storage medium, which perform image enhancement on the two-dimensional image decompressed by the current point cloud, use the enhanced two-dimensional image to reconstruct the point cloud, and improve the quality of the three-dimensional reconstructed point cloud.

[0006] The technical solution of the embodiment of the present application can be implemented as follows:

[0007] In a first aspect, an embodiment of the present application provides a decoding method, applied to a decoder, the method comprising:

[0008] Decode the code stream to determine the decompressed 2D image of the current point cloud;

[0009] performing image enhancement processing on the decompressed two-dimensional image using an image enhancement module to determine an enhanced two-dimensional image;

[0010] Performing point cloud reconstruction based on the enhanced two-dimensional image to determine a reconstructed point cloud;

[0011] The two-dimensional image includes at least one of the following: an occupancy image including occupancy information of the current point cloud, a geometric image including geometric information of the current point cloud, and an attribute image including attribute information of the current point cloud.

[0012] In a second aspect, an embodiment of the present application provides a decoder, comprising: a decoding module, an image enhancement module, and a reconstruction module; wherein:

[0013] A decoding module configured to decode the code stream and determine the decompressed two-dimensional image of the current point cloud;

[0014] an image enhancement module configured to perform image enhancement processing on the decompressed two-dimensional image to determine an enhanced two-dimensional image;

[0015] a reconstruction module configured to perform point cloud reconstruction based on the enhanced two-dimensional image and determine a reconstructed point cloud;

[0016] The two-dimensional image includes at least one of the following: an occupancy image including occupancy information of the current point cloud, a geometric image including geometric information of the current point cloud, and an attribute image including attribute information of the current point cloud.

[0017] In a third aspect, the embodiments of the present application provide a decoder, a memory, and a processor; wherein:

[0018] The memory is used to store a computer program that can be run on the processor;

[0019] The processor is configured to execute the method according to the first aspect when running the computer program.

[0020] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed, the method described in the first aspect is implemented.

[0021] An embodiment of the present application provides a decoding method, a decoder, and a storage medium. During decoding, image enhancement is performed on the decompressed two-dimensional image of the current point cloud (including at least one of an occupancy image, a geometric image, and an attribute image) to obtain an enhanced two-dimensional image. The enhanced two-dimensional image is used to reconstruct the point cloud, which can improve the quality of the three-dimensional reconstructed point cloud and improve the decoder performance without changing the point cloud bit rate.

[0022] Furthermore, a new loss function is provided for the neural network-based image enhancement module, which can enable the geometric image enhancement module and / or attribute image enhancement module to focus on enhancing the quality of occupied points and ignore enhancing the quality of unoccupied points, thereby improving the point cloud geometric reconstruction quality and / or attribute reconstruction quality, and further improving the quality of three-dimensional reconstructed point cloud. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] FIG1A is a schematic diagram of a three-dimensional point cloud image;

[0024] FIG1B is a partially enlarged schematic diagram of a three-dimensional point cloud image;

[0025] FIG2A is a schematic diagram of a point cloud image at different viewing angles;

[0026] FIG2B is a schematic diagram of a data storage format corresponding to FIG2A ;

[0027] FIG3 is a schematic diagram of a network architecture for point cloud encoding and decoding;

[0028] FIG4 is a block diagram of a V-PCC decoder;

[0029] FIG5 is a schematic diagram of a flow chart of a decoding method provided in an embodiment of the present application;

[0030] FIG6 is a block diagram of a V-PCC decoder according to an embodiment of the present application;

[0031] FIG7 is a second block diagram of a V-PCC decoder according to an embodiment of the present application;

[0032] FIG8 is a third block diagram of a V-PCC decoder according to an embodiment of the present application;

[0033] FIG9 is a block diagram of a neural network-based image enhancement module according to an embodiment of the present application;

[0034] FIG10 is a schematic block diagram of the composition of the CSAR Block provided in an embodiment of the present application;

[0035] FIG11 is a block diagram of a CAB according to an embodiment of the present application;

[0036] FIG12 is a block diagram of the composition of the SAB provided in an embodiment of the present application;

[0037] FIG13 is a schematic diagram of the structure of the decoder;

[0038] FIG14 is a second schematic diagram of the structure of the decoder;

[0039] FIG15 is a schematic diagram showing the composition structure of the encoding and decoding system. DETAILED DESCRIPTION

[0040] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below with reference to the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present application.

[0041] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0042] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0043] It should also be pointed out that the terms "first\second\third" involved in the embodiments of the present application are only used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0044] Point cloud is a three-dimensional representation of the surface of an object. Point cloud (data) of the surface of an object can be collected through acquisition equipment such as photoelectric radar, lidar, laser scanner, and multi-view camera.

[0045] A point cloud is a set of irregularly distributed discrete points in space that express the spatial structure and surface properties of a three-dimensional object or scene. Figure 1A shows a three-dimensional point cloud image and Figure 1B shows a partially enlarged view of the three-dimensional point cloud image. It can be seen that the point cloud surface is composed of densely distributed points.

[0046] In a two-dimensional image, each pixel contains information and is distributed regularly, so there's no need to record its location. However, the distribution of points in a point cloud in three-dimensional space is random and irregular, so recording the location of each point in space is necessary to fully represent the point cloud. Similar to a two-dimensional image, each location in the acquisition process has corresponding attribute information, typically an RGB color value, which reflects the object's color. For a point cloud, in addition to color information, each point's attribute information often includes a reflectance value, which reflects the surface texture of the object. Therefore, a point in a point cloud can include both location information and attribute information. For example, a point's location information can be its three-dimensional coordinates (x, y, z). A point's location information can also be referred to as its geometric information. For example, a point's attribute information can include color information (three-dimensional color information) and / or reflectance (one-dimensional reflectance information r). For example, color information can be information in any color space, such as RGB information, where R represents red (Red), G represents green (Green), and B represents blue (Blue). For another example, the color information may be luminance and chrominance (YCbCr, YUV) information, where Y represents brightness (Luma), Cb (U) represents blue color difference, and Cr (V) represents red color difference.

[0047] For example, a point cloud generated using laser measurement principles can include both its 3D coordinate information and its reflectivity. For another example, a point cloud generated using photogrammetry principles can include both its 3D coordinate information and its 3D color information. For another example, a point cloud generated using a combination of laser measurement and photogrammetry principles can include both its 3D coordinate information, its reflectivity value, and its 3D color information.

[0048] Figures 2A and 2B show a point cloud image and its corresponding data storage format. Figure 2A provides six viewing angles of the point cloud image, while Figure 2B consists of a file header and data. The header includes the data format, data representation type, the total number of points in the point cloud, and the content represented by the point cloud. For example, the point cloud is in ".ply" format, represented by ASCII code, with a total of 207,242 points. Each point has 3D coordinate information (x, y, z) and 3D color information (r, g, b).

[0049] Point clouds can be divided into the following categories according to the acquisition method:

[0050] Static point cloud: the object is stationary and the device that acquires the point cloud is also stationary;

[0051] Dynamic point cloud: The object is moving, but the device that obtains the point cloud is stationary;

[0052] Dynamic point cloud acquisition: The device used to acquire the point cloud is in motion.

[0053] For example, point clouds can be divided into two categories according to their usage:

[0054] Category 1: Machine perception point cloud, which can be used in scenarios such as autonomous navigation systems, real-time inspection systems, geographic information systems, visual sorting robots, and disaster relief robots;

[0055] Category 2: Human eye perception point cloud, which can be used in point cloud application scenarios such as digital cultural heritage, free viewpoint broadcasting, 3D immersive communication, and 3D immersive interaction.

[0056] Point clouds can flexibly and conveniently express the spatial structure and surface properties of three-dimensional objects or scenes. Moreover, since point clouds are obtained by directly sampling real objects, they can provide a strong sense of reality while ensuring accuracy. Therefore, they are widely used, including virtual reality games, computer-aided design, geographic information systems, automatic navigation systems, digital cultural heritage, free viewpoint broadcasting, three-dimensional immersive remote presentation, and three-dimensional reconstruction of biological tissues and organs.

[0057] Point clouds are primarily collected through computer generation, 3D laser scanning, and 3D photogrammetry. Computers can generate point clouds of virtual 3D objects and scenes; 3D laser scanning can obtain point clouds of static real-world 3D objects or scenes, generating millions of point clouds per second; and 3D photogrammetry can obtain point clouds of dynamic real-world 3D objects or scenes, generating tens of millions of point clouds per second. These technologies reduce the cost and time required to acquire point cloud data while improving data accuracy. While changes in point cloud data acquisition methods have made it possible to acquire large amounts of point cloud data, the processing of this massive amount of 3D point cloud data is facing bottlenecks due to storage space and transmission bandwidth constraints, as application demands grow.

[0058] For example, taking a point cloud video with a frame rate of 30 frames per second (fps), each frame contains 700,000 points, and each point has coordinate information (xyz, float) and color information (RGB, uchar). The data volume of a 10-second point cloud video is approximately 0.7 million × (4 bytes × 3 + 1 byte × 3) × 30 fps × 10 seconds = 3.15 GB. Where 1 byte is 10 bits, and the YUV sampling format is 4:2:0, and the frame rate is 24 fps, the data volume of a 1280 × 720 2D video is approximately 1280 × 720 × 12 bits × 24 fps × 10 seconds ≈ 0.33 GB. The data volume of a 10-second two-view 3D video is approximately 0.33 × 2 = 0.66 GB. This shows that the data volume of a point cloud video far exceeds that of a 2D or 3D video of the same length. Therefore, in order to better realize data management, save server storage space, and reduce the transmission traffic and transmission time between the server and the client, point cloud compression has become a key issue in promoting the development of the point cloud industry.

[0059] That is to say, since the point cloud is a collection of massive points, storing the point cloud not only consumes a lot of memory, but is also not conducive to transmission. There is also not enough bandwidth to support direct transmission of the point cloud at the network layer without compression. Therefore, the point cloud needs to be compressed.

[0060] Currently, the point cloud coding framework that can compress point clouds can be the geometry-based Point Cloud Compression (G-PCC) codec framework or the video-based Point Cloud Compression (V-PCC) codec framework provided by the Moving Picture Experts Group (MPEG), or the AVS-PCC codec framework provided by AVS. The G-PCC codec framework can be used to compress the first type of static point clouds and the third type of dynamically acquired point clouds, which can be based on the Point Cloud Compression Test Platform (Test Model Compression 13, TMC13). The V-PCC codec framework can be used to compress the second type of dynamic point clouds, which can be based on the Point Cloud Compression Test Platform (Test Model Compression 2, TMC2). Therefore, the G-PCC codec framework is also called the point cloud codec TMC13, and the V-PCC codec framework is also called the point cloud codec TMC2.

[0061] It can be understood that in the point cloud encoding and decoding framework, the point cloud of the input three-dimensional image model can be divided into tiles / slices, and each tile / slice can be independently encoded.

[0062] An embodiment of the present application provides a network architecture of a point cloud encoding and decoding system including a decoding method and an encoding method. FIG3 is a schematic diagram of a network architecture of a point cloud encoding and decoding system provided by an embodiment of the present application. As shown in FIG3 , the network architecture includes one or more electronic devices 13 to 1N and a communication network 01, wherein the electronic devices 13 to 1N can perform video interaction through the communication network 01. During the implementation process, the electronic device can be various types of devices with point cloud encoding and decoding functions. For example, the electronic device can include a mobile phone, a tablet computer, a personal computer, a personal digital assistant, a navigator, a digital phone, a video phone, a television, a sensor device, a server, etc., which is not limited by the embodiment of the present application. Among them, the decoder or encoder in the embodiment of the present application can be the above-mentioned electronic device.

[0063] Among them, the electronic device in the embodiment of the present application has a point cloud encoding and decoding function, generally including a point cloud encoder (ie, encoder) and a point cloud decoder (ie, decoder).

[0064] The decoding method provided in the embodiment of the present application is applied to the V-PCC codec framework. V-PCC aims to use existing video codecs, such as HEVC, to compress the occupancy information, geometric information and attribute information of dynamic point clouds. Specifically, at the encoding end, V-PCC first projects the 3D point cloud into a set of two-dimensional occupancy images, geometric images and attribute images, and then uses HEVC to compress the occupancy images, geometric images and attribute images to form corresponding occupancy code streams, geometric code streams and attribute code streams. The additional metadata required to parse the three video sequences (such as atlas / block information) can be compressed separately to form metadata code streams. The encoder forms a compressed code stream through a multiplexer and sends the compressed code stream to the decoder. Figure 4 shows a schematic diagram of the V-PCC decoder's architecture. At the decoding end, a demultiplexer demultiplexes the compressed bitstream into a metadata substream, an occupancy substream, a geometry substream, and an attribute substream. These substreams are then decoded separately to produce decompressed metadata, an occupancy image, a geometry image, and an attribute image. 3D point clouds are then reconstructed based on the corresponding relationships among metadata, occupancy, geometry, and attribute information.

[0065] V-PCC uses decompressed 2D attribute images to reconstruct the attributes of 3D point clouds. If the attribute images contain various compression artifacts, the reconstructed point clouds will inevitably suffer from attribute distortion, resulting in severe attribute artifacts in the reconstructed point cloud sequence. These attribute artifacts not only severely impact the objective quality of the point cloud but also significantly affect the subjective visual experience.

[0066] In the case of lossy coding, to eliminate distortion in the attributes of the reconstructed point cloud and improve the subjective and objective reconstruction quality of the reconstructed point cloud, an embodiment of the present application provides a decoding method. During decoding, image enhancement is performed on the decompressed two-dimensional image of the current point cloud (including at least one of the occupancy image, the geometry image, and the attribute image) to obtain an enhanced two-dimensional image. The enhanced two-dimensional image is used to reconstruct the point cloud, which can improve the quality of the three-dimensional reconstructed point cloud and enhance the decoder performance without changing the point cloud bit rate.

[0067] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application.

[0068] The embodiment of the present application proposes a decoding method, which is applied to a codec system or a decoding system. When applied in a codec system, the decoding method can be applied to a decoder or to the two-dimensional image reconstruction part of an encoder. Figure 5 is a flow chart of a decoding method provided by an embodiment of the present application. As shown in Figure 5, in an embodiment of the present application, the method for decoding processing by a decoder may include the following steps:

[0069] Step 501: Decode the code stream to determine the decompressed two-dimensional image of the current point cloud;

[0070] The two-dimensional image includes at least one of the following: an occupancy image including occupancy information of the current point cloud, a geometric image including geometric information of the current point cloud, and an attribute image including attribute information of the current point cloud.

[0071] In some embodiments, the current point cloud may be a point cloud frame, or a point cloud block obtained by segmenting a point cloud frame. Segmentation includes projecting each point cloud frame into a two-dimensional image, and subdividing the pixels in the two-dimensional image into square blocks and / or rectangular blocks for compression. For example, in high efficiency video coding (HEVC), the frame may be first divided into coding tree units (CTUs), which are blocks of a predefined size (e.g., 64×64 pixels). These CTUs include luminance samples and chrominance samples. The coding tree can be used to divide the CTU into blocks, and then repeatedly subdivide these blocks until a configuration that supports further encoding is obtained. For example, the luminance component of the frame can be subdivided until each block includes a relatively uniform luminance value. In addition, the chrominance component of the frame can be subdivided until each block includes a relatively uniform color value. Therefore, the segmentation mechanism varies depending on the content of the frame.

[0072] In some embodiments, decoding the bitstream to determine the decompressed two-dimensional image of the current point cloud includes: decoding the bitstream to obtain a metadata substream, an occupancy substream, a geometry substream, and an attribute substream; performing metadata decoding on the metadata substream to obtain metadata of the current point cloud; performing video decoding on the occupancy substream to obtain an occupancy image decompressed by the current point cloud; performing video decoding on the geometry substream to obtain a geometry image decompressed by the current point cloud; and performing video decoding on the attribute substream to obtain an attribute image decompressed by the current point cloud.

[0073] In some embodiments, the code stream is demultiplexed to obtain a metadata sub-code stream, an occupancy sub-code stream, a geometry sub-code stream, and an attribute sub-code stream. In some embodiments, metadata entropy decoding is performed on the metadata sub-code stream to obtain metadata for the current point cloud.

[0074] It is understood that the decoding end obtains the compressed codestream formed by V-PCC compression and outputs the metadata substream, occupancy substream, geometry substream, and attribute substream through demultiplexing. Subsequently, these four substreams are decoded, and the occupancy image, geometry image, and attribute image can be decoded using a video decoder, outputting decompressed two-dimensional occupancy image, geometry image, and attribute image. In some embodiments, the decompressed two-dimensional image can also be referred to as a decoded two-dimensional image.

[0075] Step 502: using an image enhancement module to perform image enhancement processing on the decompressed two-dimensional image to determine an enhanced two-dimensional image;

[0076] In an embodiment of the present application, after obtaining the decompressed two-dimensional image, the decompressed two-dimensional image is subjected to image enhancement processing by an image enhancement module to improve the quality of the two-dimensional image, thereby improving the quality of the three-dimensional reconstructed point cloud.

[0077] In some embodiments, the image enhancement module includes at least one of the following: an occupancy image enhancement module for performing image enhancement processing on the occupancy image, a geometric image enhancement module for performing image enhancement processing on the geometric image, and an attribute image enhancement module for performing image enhancement processing on the attribute image. That is, the decoding end can set a corresponding image enhancement module for one or more of the occupancy image, the geometric image, and the attribute image according to the characteristics of the occupancy image, the geometric image, and the attribute image, for enhancing the corresponding image. In an embodiment of the present application, the enhanced two-dimensional image includes at least one of the occupancy image, the geometric image, and the attribute image, and point cloud reconstruction is performed using the enhanced two-dimensional image and the unenhanced two-dimensional image (if any). Among them, the unenhanced two-dimensional image can be understood as the decompressed two-dimensional image obtained by decoding the code stream in step 501, which directly participates in the subsequent three-dimensional point cloud reconstruction without enhancement.

[0078] In some embodiments, an image enhancement module is used to perform image enhancement processing on the decompressed two-dimensional image to determine an enhanced two-dimensional image, including at least one of the following: inputting the decompressed occupancy image into the occupancy image enhancement module to enhance the occupancy image to obtain an enhanced occupancy image; inputting the decompressed geometric image into the geometric image enhancement module to enhance the geometric image to obtain an enhanced geometric image; inputting the decompressed attribute image into the attribute image enhancement module to enhance the attribute image to obtain an enhanced attribute image.

[0079] In some embodiments, more than two-dimensional images may also be enhanced using the same image enhancement module.

[0080] Exemplarily, as shown in FIG6 , at the decoding end, a compressed bitstream is demultiplexed by a demultiplexer to obtain a metadata substream (Patch substream), an occupancy substream (Occupancy substream), a geometry substream (Geometry substream), and an attribute substream (Attribute substream). The metadata substream is decoded by a metadata decoder, and the occupancy substream, geometry substream, and attribute substream are decoded by a video decoder to obtain decompressed occupancy image, geometry image, and attribute image. An attribute image enhancement module is added after the attribute video decoder. This module takes the decompressed attribute image as input and outputs an enhanced attribute image for subsequent 3D attribute reconstruction.

[0081] As shown in Figure 7, a geometric image enhancement module is added after the geometry video decoder. This module takes the decompressed geometry image as input and outputs an enhanced geometry image for subsequent 3D geometry reconstruction. An attribute image enhancement module is added after the attribute video decoder. This module takes the decompressed attribute image as input and outputs an enhanced attribute image for subsequent 3D attribute reconstruction.

[0082] As shown in Figure 8, after the occupancy video decoder, an occupancy image enhancement module is added. This module takes the decompressed occupancy image as input and outputs an enhanced occupancy image for subsequent 3D geometry and attribute reconstruction. After the geometry video decoder, a geometry image enhancement module is added. This module takes the decompressed geometry image as input and outputs an enhanced geometry image for subsequent 3D geometry reconstruction. After the attribute video decoder, an attribute image enhancement module is added. This module takes the decompressed attribute image as input and outputs an enhanced attribute image for subsequent 3D attribute reconstruction. In some embodiments, the metadata decoder is an entropy decoder.

[0083] It should be noted that the application embodiments of the image enhancement module are not limited to those shown in Figures 6 to 8. The image enhancement module can also be set separately after the occupancy video decoder to enhance the occupancy image, or the image enhancement module can also be set separately after the geometry video decoder to enhance the geometric image, or the image enhancement module can also be set at the same time after the occupancy video decoder and the geometry video decoder to enhance the occupancy image and the geometric image, or the image enhancement can also be set at the same time after the occupancy video decoder and the attribute video decoder to enhance the occupancy image and the attribute image, etc.

[0084] In some embodiments, the image enhancement module may be a neural network-based image enhancement module or an image enhancement algorithm-based image enhancement module. It should be noted that a neural network can also be understood as a neural network algorithm for image enhancement, and in the embodiments of the present application, the image enhancement algorithm is one or more image enhancement algorithms other than a neural network algorithm.

[0085] For the neural network-based image enhancement module, a matching image enhancement module can be obtained by designing a neural network structure, selecting a loss function, and selecting training samples.

[0086] In some embodiments, the image enhancement module includes: a first input layer, at least one channel-spatial attention based residual block, a first output layer and an adder; the first input layer includes: at least one convolutional layer and an activation layer; the first output layer includes: at least one convolutional layer and an activation layer;

[0087] The image enhancement module is configured to: take the decompressed two-dimensional image as the input image of the image enhancement module, pass it through a first input layer, at least one residual block based on channel-spatial attention in series, and a first output layer, and output residual information; add the input image of the image enhancement module and the residual information output by the first output layer through an adder to obtain an enhanced two-dimensional image. In this embodiment of the application, the channel-spatial attention mechanism is introduced into the image enhancement module to enhance interesting information in the image and suppress useless information.

[0088] In some embodiments, the parameters and activation functions of the convolutional layer in the image enhancement module can be flexibly set based on actual needs. For example, the parameters of the convolutional layer include at least one of the following: convolution kernel size, number of input channels, number of output channels, etc. The activation function includes at least one of the following: ReLU function, sigmoid function, Tanh function, PReLU function, etc.

[0089] An embodiment of the present application provides a neural network structure. As shown in Figure 9, the image enhancement module includes a channel-spatial attention based residual network (CSAR Net), which is used to enhance the two-dimensional image. The input of CSAR Net is a decompressed two-dimensional image, and the output is an enhanced two-dimensional image. From the perspective of the two-dimensional image processing flow, CSAR Net includes: a first input layer, at least one channel-spatial attention based residual block (CSAR Block), a first output layer and an adder, etc. Among them, the first input layer and the first output layer are composed of at least one convolutional layer and an activation layer.

[0090] For example, as shown in Figure 9, the first input layer includes a convolutional layer Conv(3×3 / 64) and a PReLU activation layer. The first output layer includes a convolutional layer Conv(3×3 / 64), a PReLU activation layer, and a convolutional layer Conv(3×3 / 1).

[0091] It should be noted that the channel-spatial attention-based residual block extracts residual information. The input image and the residual information are added together to produce an enhanced image. In practical applications, the network structure of the channel-spatial attention residual block can be flexibly configured based on actual needs.

[0092] In some embodiments, the channel-spatial attention based residual block includes a channel attention block and a spatial attention block, and the channel attention block and the spatial attention block can be combined in series or in parallel.

[0093] In some embodiments, the channel-spatial attention based residual block includes: a second input layer, a channel attention block, a spatial attention block, and an adder, wherein the second input layer includes at least one convolutional layer and an activation layer;

[0094] The residual block based on channel-spatial attention is configured as follows: the output information of the first input layer is used as the input information of the residual block based on channel-spatial attention, and passes through the second input layer, the channel attention block and the spatial attention block in sequence; the input information of the residual block based on channel-spatial attention and the output information of the spatial attention block are added through an adder to obtain the output information of the residual block based on channel-spatial attention.

[0095] An embodiment of the present application provides a network structure of a residual block based on channel-spatial attention. As shown in Figure 10, the Channel-Spatial Attention based Residual Block (CSAR Block) includes: a second input layer (including a convolutional layer Conv(3×3 / 64), a PReLU activation layer and a convolutional layer Conv(3×3 / 64)), a channel attention block (CAB), a spatial attention block (SAB) and an adder. The channel attention block and the spatial attention block are combined in series, and the channel attention block is located before the spatial attention block.

[0096] It should be noted that the network structure of the channel attention block and spatial attention block, the parameters and activation function of the convolutional layer can be flexibly set according to actual needs.

[0097] In some embodiments, the channel attention block includes: a first global average pooling layer, a first global maximum pooling layer, a shared multilayer perceptron, an adder, a sigmoid activation layer, and a multiplier;

[0098] The channel attention block is configured as follows: the output information of the second input layer is used as the input information of the channel attention block, and is input into the first global average pooling layer and the first global maximum pooling layer at the same time; the output information of the first global average pooling layer and the output information of the first global maximum pooling layer are simultaneously input into the shared multilayer perceptron to obtain two output information; the two output information of the shared multilayer perceptron are added through the adder and then passed through the Sigmoid activation layer; the input information of the channel attention block and the output information of the Sigmoid activation layer are multiplied by the multiplier to obtain the output information of the channel attention block.

[0099] It should be noted that in addition to the input layer and the output layer, the shared multilayer perceptron (shared MLP) may have at least one hidden layer. In some embodiments, the shared multilayer perceptron includes: an input convolutional layer, a ReLU activation layer, and an output convolutional layer.

[0100] The present embodiment provides a network structure for a channel attention block. As shown in Figure 11, the channel attention block (CAB) includes: global average pooling, global maximum pooling, a shared MLP (including a convolutional layer Cnov(1×1), a ReLU activation layer, and a convolutional layer Cnov(1×1)), an adder, a sigmoid activation layer, and a multiplier.

[0101] In some embodiments, the spatial attention block includes: a second global average pooling layer, a second global maximum pooling layer, a concatenation layer, a convolution layer, a sigmoid activation layer, and a multiplier;

[0102] The spatial attention block is configured as follows: the output information of the channel attention block is used as the input information of the spatial attention block, and is input into the second global average pooling layer and the second global maximum pooling layer at the same time; the output information of the second global average pooling layer and the output information of the second global maximum pooling layer are spliced ​​through the splicing layer, and then pass through the convolution layer and the Sigmoid activation layer in sequence; the input information of the spatial attention block and the output information of the Sigmoid activation layer are multiplied by the multiplier to obtain the output information of the spatial attention block.

[0103] The present embodiment provides a network structure for a spatial attention block. As shown in Figure 12, the spatial attention block (SAB) includes: global average pooling, global maximum pooling, a concatenation layer (Concat, abbreviated as C), a convolutional layer Cnov (3×3), and a Sigmoid activation layer.

[0104] Optionally, in some embodiments, the neural network-based image enhancement module shown in Figures 9 to 12 is used to enhance the attribute image.

[0105] For a neural network-based image enhancement module, the loss function used during training can be selected based on actual needs. In some embodiments, the loss function includes at least one of the following: mean square error (MSE), mean absolute error (MAE), least square error (LSE), least absolute error (LAE), etc.

[0106] MSE is the most commonly used error in regression loss function, which is the original value x i and predicted value The mean of the sum of squared differences between the two values ​​is given by the following formula:

[0107] MAE is another commonly used regression loss function, which is the original value x i and predicted value The mean of the sum of the absolute differences represents the average error margin of the predicted value, ranging from 0 to ∞, and its formula is as follows:

[0108] LSE, also known as the L2 norm loss function, is to convert the original value x i and predicted value The sum of the absolute differences of is minimized, and the formula is as follows:

[0109] LAE, also known as L1 norm loss function and Least Absolute Deviations (LAD), is the original value x i and predicted value The sum of the squares of the differences is minimized, and the formula is as follows:

[0110] Among them, x is the original image, The original image and the predicted image have the same resolution, N is the total number of pixels in the image, i represents the i-th pixel and i∈(1,N), x i is the original value of the i-th pixel in the original image, is the predicted value of the i-th pixel in the predicted image.

[0111] In some embodiments, for V-PCC point cloud compression scenarios, the encoding and decoding characteristics of V-PCC itself are taken into account. That is, at the V-PCC encoding end, it converts the 3D point cloud into a two-dimensional image containing geometric and attribute information through multi-faceted projection, and uses the occupied image to indicate which pixels in the two-dimensional image are occupied and which pixels are not occupied. Only the attribute values ​​or geometric values ​​corresponding to the occupied pixels will affect the quality of the reconstructed 3D point cloud. Therefore, the enhancement of non-occupied points in the attribute image or geometric image is meaningless and will not improve the reconstruction quality of the point cloud. Only by enhancing the occupied point information can the quality of the point cloud be improved.

[0112] Based on this, in some embodiments, when the image enhancement module includes a geometric image enhancement module based on a neural network, the method also includes: in the process of training the geometric image enhancement module, determining the occupied points in the geometric image based on the occupied image, and determining the loss function of the geometric image enhancement module based on the geometric predicted values ​​and geometric original values ​​of the occupied points in the geometric image; and adjusting the parameters of the geometric image enhancement module based on the loss function of the geometric image enhancement module.

[0113] In some embodiments, when the image enhancement module includes an attribute image enhancement module based on a neural network, the method also includes: in the process of training the attribute image enhancement module, determining the occupied points in the geometric image based on the occupancy image, and determining the loss function of the attribute image enhancement module based on the attribute prediction value and the attribute original value of the occupied points in the attribute image; and adjusting the parameters of the attribute image enhancement module based on the loss function of the attribute image enhancement module.

[0114] In some embodiments, during the training phase, the occupancy image is a decompressed occupancy image or an enhanced occupancy image. Exemplarily, when the occupancy image enhancement module is not provided, the occupancy image is a decompressed occupancy image; when the occupancy image enhancement module is provided, the occupancy image is an enhanced occupancy image.

[0115] It should be noted that the embodiment of the present application improves the loss function used to train the network and designs a loss function based on the V-PCC occupancy image for training the 2D attribute image or geometric image enhancement module. The proposed loss function can be formulated as:

[0116] For example, the MSE loss function formula based on the V-PCC occupancy image is as follows:

[0117] The MAE formula based on the V-PCC occupancy image is as follows:

[0118] The LSE formula based on the V-PCC occupancy image is as follows:

[0119] The LAE formula based on the V-PCC occupancy image is as follows:

[0120] Among them, x is the original image, is the enhanced image, also called the predicted image. occ is the occupancy image with the same resolution as the original image and the predicted image, ∑occ represents the total number of occupied points in the occupancy image, N is the total number of pixels in the image, i represents the i-th pixel and i∈(1,N). i is the original value of the i-th pixel in the original image, is the predicted value of the i-th pixel in the predicted image, occ i is the occupancy value of the i-th pixel in the occupancy image.

[0121] In some embodiments, when the occupancy value is 1, it indicates that the pixel point is an occupied point; when the occupancy value is 0, it indicates that the pixel point is an unoccupied point.

[0122] In the embodiments of the present application, for the enhancement of attribute images or geometric images decoded by V-PCC based on a neural network, taking into account the differences between V-PCC images and traditional video images, and to make the network more adaptable to the enhancement of V-PCC attribute images or geometric images and improve the quality of point cloud reconstruction, a loss function based on the V-PCC occupancy map is designed. When applied to the V-PCC decoding scenario, it enables the image enhancement module to focus on enhancing the quality of occupied points and ignore the quality of unoccupied points, thereby improving the quality of the 3D reconstructed point cloud. This loss function can be used for training both the geometric image enhancement module and the attribute map enhancement module.

[0123] It should be noted that the loss function provided in the embodiments of the present application and the loss function based on the V-PCC occupancy image are merely illustrative examples and are not limitations on the embodiments of the present application. Other loss functions or improved loss functions based on the V-PCC occupancy image are all within the scope of protection of the embodiments of the present application.

[0124] For the image enhancement module based on the image enhancement algorithm, the image enhancement algorithm can be at least one of the following: interpolation algorithm, filtering algorithm, histogram equalization algorithm, logarithmic image enhancement algorithm, exponential image enhancement algorithm, Gamma correction algorithm, Laplace transform algorithm.

[0125] In some embodiments, the occupancy image, the geometric image, and the attribute image may all use an image enhancement module based on a neural network, or all three may use an image enhancement module based on an image enhancement algorithm, or some may use an image enhancement module based on a neural network, or some may use an image enhancement module based on an image enhancement algorithm, or some may use an image enhancement module based on a neural network and the other may use an image enhancement module based on an image enhancement algorithm.

[0126] Step 503: reconstructing a point cloud based on the enhanced two-dimensional image to determine a reconstructed point cloud.

[0127] In some embodiments, point cloud reconstruction is performed based on the enhanced two-dimensional image to determine the reconstructed point cloud, including: reconstructing geometric information and attribute information based on the enhanced two-dimensional image to obtain reconstructed geometric information and attribute information; reconstructing the point cloud based on the reconstructed geometric information and attribute information to determine the reconstructed point cloud.

[0128] In some embodiments, point cloud reconstruction is performed based on the reconstructed geometric information and attribute information to determine the reconstructed point cloud, including: post-processing the reconstructed geometric information; post-processing the reconstructed attribute information; and point cloud reconstruction is performed based on the post-processed geometric information and attribute information to determine the reconstructed point cloud.

[0129] In some embodiments, when the occupancy image, the geometric image, and the attribute image are all enhanced, the geometric information and the attribute information are reconstructed based on the enhanced two-dimensional image to obtain reconstructed geometric information and attribute information, including: reconstructing the geometric information based on the metadata of the current point cloud, the enhanced occupancy image, and the enhanced geometric image to obtain reconstructed geometric information; reconstructing the attribute information based on the reconstructed geometric information and the enhanced attribute image to obtain reconstructed attribute information.

[0130] In some embodiments, point cloud reconstruction is performed based on the enhanced two-dimensional image to determine the reconstructed point cloud, including: reconstructing geometric information based on the metadata of the current point cloud, the enhanced occupancy image, and the enhanced geometric image to obtain reconstructed geometric information; post-processing the reconstructed geometric information to obtain post-processed geometric information; reconstructing attribute information based on the post-processed geometric information and the enhanced attribute image to obtain reconstructed attribute information; post-processing the enhanced attribute image to obtain post-processed attribute information; reconstructing the point cloud based on the post-processed geometric information and the attribute information to determine the reconstructed point cloud.

[0131] In some embodiments, post-processing of the geometric information includes smoothing to obtain smooth geometric information and reduce discontinuity phenomena at block boundaries caused by compression distortion.

[0132] In some embodiments, post-processing of the attribute information includes color smoothing to obtain smooth color values ​​and reduce distortion at block boundaries.

[0133] By adopting the above technical solution, during decoding, the decompressed two-dimensional image of the current point cloud (including at least one of the occupancy image, geometric image and attribute image) is enhanced to obtain an enhanced two-dimensional image. The enhanced two-dimensional image is used to reconstruct the point cloud, which can improve the quality of the three-dimensional reconstructed point cloud and improve the decoder performance without changing the point cloud bit rate.

[0134] Furthermore, a new loss function is provided for the neural network-based image enhancement module, which can enable the geometric image enhancement module and / or attribute image enhancement module to focus on enhancing the quality of occupied points and ignore enhancing the quality of unoccupied points, thereby improving the point cloud geometric reconstruction quality and / or attribute reconstruction quality, and further improving the quality of three-dimensional reconstructed point cloud.

[0135] Based on the above embodiment, in another embodiment of the present application, based on the same inventive concept as the above embodiment, the embodiment of the present application further provides a decoder. FIG13 is a schematic diagram of the composition structure of the decoder. As shown in FIG13 , the decoder 130 may include: a decoding module 131, an image reconstruction module 132 and a reconstruction module 133, wherein:

[0136] A decoding module 131 is configured to decode the code stream and determine a decompressed two-dimensional image of the current point cloud;

[0137] an image enhancement module 132 configured to perform image enhancement processing on the decompressed two-dimensional image to determine an enhanced two-dimensional image;

[0138] a reconstruction module 133 configured to perform point cloud reconstruction based on the enhanced two-dimensional image and determine a reconstructed point cloud;

[0139] The two-dimensional image includes at least one of the following: an occupancy image including occupancy information of the current point cloud, a geometric image including geometric information of the current point cloud, and an attribute image including attribute information of the current point cloud.

[0140] In some embodiments, the image enhancement module includes at least one of the following: an occupancy image enhancement module for performing image enhancement processing on the occupancy image, a geometric image enhancement module for performing image enhancement processing on the geometric image, and an attribute image enhancement module for performing image enhancement processing on the attribute image.

[0141] In some embodiments, the image enhancement module 132 includes a neural network-based image enhancement module.

[0142] In some embodiments, the image enhancement module 132 includes: a first input layer, at least one channel-spatial attention based residual block, a first output layer, and an adder;

[0143] The first input layer includes: at least one convolutional layer and an activation layer;

[0144] The first output layer includes: at least one convolutional layer and an activation layer;

[0145] The image enhancement module is configured as follows: taking the decompressed two-dimensional image as the input image of the image enhancement module, sequentially passing it through a first input layer, at least one residual block based on channel-spatial attention connected in series, and a first output layer, and outputting residual information; adding the input image of the image enhancement module and the residual information output by the first output layer through the adder to obtain the enhanced two-dimensional image.

[0146] In some embodiments, the channel-spatial attention based residual block comprises: a second input layer, a channel attention block, a spatial attention block, and an adder, wherein the second input layer comprises at least one convolutional layer and an activation layer;

[0147] The channel-spatial attention-based residual block is configured as follows: the output information of the first input layer is used as the input information of the channel-spatial attention-based residual block, and passes through the second input layer, the channel attention block and the spatial attention block in sequence; the input information of the channel-spatial attention-based residual block and the output information of the spatial attention block are added through the adder to obtain the output information of the channel-spatial attention-based residual block.

[0148] In some embodiments, a first global average pooling layer, a first global maximum pooling layer, a shared multilayer perceptron, an adder, a sigmoid activation layer, and a multiplier;

[0149] The channel attention block is configured as follows: the output information of the second input layer is used as the input information of the channel attention block, and is simultaneously input into the first global average pooling layer and the first global maximum pooling layer; the output information of the first global average pooling layer and the output information of the first global maximum pooling layer are simultaneously input into the shared multi-layer perceptron to obtain two-way output information; the two-way output information of the shared multi-layer perceptron is added through the adder, and then passed through the Sigmoid activation layer; the input information of the channel attention block and the output information of the Sigmoid activation layer are multiplied by the multiplier to obtain the output information of the channel attention block.

[0150] In some embodiments, the shared multi-layer perceptron includes: an input convolutional layer, a ReLU activation layer, and an output convolutional layer.

[0151] In some embodiments, the spatial attention block includes: a second global average pooling layer, a second global maximum pooling layer, a splicing layer, a convolution layer, a sigmoid activation layer, and a multiplier;

[0152] The spatial attention block is configured as follows: the output information of the channel attention block is used as the input information of the spatial attention block, and is simultaneously input into the second global average pooling layer and the second global maximum pooling layer; the output information of the second global average pooling layer and the output information of the second global maximum pooling layer are spliced ​​through the splicing layer, and then pass through the convolution layer and the Sigmoid activation layer in sequence; the input information of the spatial attention block and the output information of the Sigmoid activation layer are multiplied by the multiplier to obtain the output information of the spatial attention block.

[0153] In some embodiments, when the image enhancement module 132 includes a geometric image enhancement module based on a neural network, during the training of the geometric image enhancement module, the occupied points in the geometric image are determined according to the occupied image, and the loss function of the geometric image enhancement module is determined according to the geometric predicted values ​​and geometric original values ​​of the occupied points in the geometric image; and the parameters of the geometric image enhancement module are adjusted according to the loss function of the geometric image enhancement module.

[0154] In some embodiments, when the image enhancement module 132 includes an attribute image enhancement module based on a neural network, during the training of the attribute image enhancement module, the occupied points in the geometric image are determined based on the occupied image, and the loss function of the attribute image enhancement module is determined based on the attribute prediction value and the attribute original value of the occupied points in the attribute image; and the parameters of the attribute image enhancement module are adjusted based on the loss function of the attribute image enhancement module.

[0155] In some embodiments, during the training phase, the occupancy image is a decompressed occupancy image or an enhanced occupancy image. Exemplarily, when the occupancy image enhancement module is not provided, the occupancy image is a decompressed occupancy image; when the occupancy image enhancement module is provided, the occupancy image is an enhanced occupancy image.

[0156] In some embodiments, the loss function includes at least one of the following: mean square error (MSE), mean absolute error (MAE), least square error (LSE), and least absolute error (LAE).

[0157] In some embodiments, the image enhancement module 132 includes an image enhancement module based on an image enhancement algorithm.

[0158] In some embodiments, the image enhancement algorithm includes at least one of the following: an interpolation algorithm, a filtering algorithm, a histogram equalization algorithm, a logarithmic image enhancement algorithm, an exponential image enhancement algorithm, a gamma correction algorithm, and a Laplace transform algorithm.

[0159] In some embodiments, the decoding module 131 is configured to: decode the codestream to obtain a metadata substream, an occupancy substream, a geometry substream, and an attribute substream; perform metadata decoding on the metadata substream to obtain metadata of the current point cloud; perform video decoding on the occupancy substream to obtain a decompressed occupancy image of the current point cloud; perform video decoding on the geometry substream to obtain a decompressed geometry image of the current point cloud; and perform video decoding on the attribute substream to obtain a decompressed attribute image of the current point cloud.

[0160] In some embodiments, the reconstruction module 133 is configured to reconstruct geometric information and attribute information based on the enhanced two-dimensional image to obtain reconstructed geometric information and attribute information; and reconstruct a point cloud based on the reconstructed geometric information and attribute information to determine a reconstructed point cloud.

[0161] In some embodiments, the reconstruction module 133 is configured to post-process the reconstructed geometric information; post-process the reconstructed attribute information; reconstruct the point cloud based on the post-processed geometric information and attribute information to determine the reconstructed point cloud.

[0162] In some embodiments, the reconstruction module 133 is configured to reconstruct geometric information based on the metadata of the current point cloud, the enhanced occupancy image, and the enhanced geometric image to obtain reconstructed geometric information; and reconstruct attribute information based on the reconstructed geometric information and the enhanced attribute image to obtain reconstructed attribute information.

[0163] In some embodiments, the reconstruction module 133 is configured to reconstruct geometric information based on the metadata of the current point cloud, the enhanced occupancy image, and the enhanced geometric image to obtain reconstructed geometric information; post-process the reconstructed geometric information to obtain post-processed geometric information; reconstruct attribute information based on the post-processed geometric information and the enhanced attribute image to obtain reconstructed attribute information; post-process the enhanced attribute image to obtain post-processed attribute information; reconstruct the point cloud based on the post-processed geometric information and the attribute information to determine the reconstructed point cloud.

[0164] It is understood that in this embodiment, a "module" or "unit" can be a portion of a circuit, a portion of a processor, a portion of a program or software, etc., and can also be modular or non-modular. Furthermore, the various components in this embodiment can be integrated into a single processing unit, or each unit can exist physically separately, or two or more units can be integrated into a single unit. The aforementioned integrated units can be implemented in the form of hardware or software functional modules.

[0165] If the integrated unit is implemented as a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, or the portion that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method described in this embodiment. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0166] Therefore, an embodiment of the present application provides a computer-readable storage medium, which is applied to the decoder 130. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements any one of the methods in the aforementioned embodiments.

[0167] Based on the composition of the above-mentioned decoder 130 and the computer-readable storage medium, Figure 14 is a second schematic diagram of the composition structure of the decoder. As shown in Figure 14, the decoder 130 may include: a memory 134 and a processor 135, a communication interface 136 and a bus system 137. The memory 134, the processor 135, and the communication interface 136 are coupled together through the bus system 137. It can be understood that the bus system 137 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 137 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, various buses are labeled as bus system 137 in Figure 14. Among them,

[0168] Communication interface 136, used for receiving and sending signals during the process of sending and receiving information with other external network elements;

[0169] The memory 134 is used to store computer programs that can be run on the processor;

[0170] The processor 135 is configured to, when running the computer program, execute:

[0171] Decode the code stream to determine the decompressed 2D image of the current point cloud;

[0172] performing image enhancement processing on the decompressed two-dimensional image using an image enhancement module to determine an enhanced two-dimensional image;

[0173] Performing point cloud reconstruction based on the enhanced two-dimensional image to determine a reconstructed point cloud;

[0174] The two-dimensional image includes at least one of the following: an occupancy image including occupancy information of the current point cloud, a geometric image including geometric information of the current point cloud, and an attribute image including attribute information of the current point cloud.

[0175] Optionally, as another embodiment, the processor 135 is further configured to execute the method described in any one of the aforementioned embodiments when running the computer program.

[0176] It is understood that the memory 134 in the embodiment of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDRSDRAM), enhanced synchronous DRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DRRAM). The memory 134 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0177] The processor 135 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits or software instructions in the processor 135. The above-mentioned processor 135 can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory 134 , and the processor 135 reads the information in the memory 134 and completes the steps of the above method in combination with its hardware.

[0178] It is to be understood that these embodiments described in the present application can be implemented with hardware, software, firmware, middleware, microcode or its combination.For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (Application Specific Integrated Circuits, ASIC), digital signal processor (Digital Signal Processing, DSP), digital signal processing equipment (DSP Device, DSPD), programmable logic device (Programmable Logic Device, PLD), field programmable gate array (Field-Programmable Gate Array, FPGA), general-purpose processor, controller, microcontroller, microprocessor, other electronic units for performing functions described in the present application or its combination.For software implementation, the technology described in the present application can be realized by the module (such as process, function etc.) that performs functions described in the present application. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0179] Optionally, as another embodiment, the processor 135 is further configured to execute the method described in any one of the aforementioned embodiments when running the computer program.

[0180] In yet another embodiment of the present application, referring to FIG15 , a schematic diagram of the structure of a coding and decoding system provided by an embodiment of the present application is shown. As shown in FIG15 , the coding and decoding system 150 may include an encoder 151 and a decoder 152 .

[0181] In the embodiment of the present application, the encoder 151 may be the encoder described in any one of the aforementioned embodiments, and the decoder 152 may be the decoder described in any one of the aforementioned embodiments.

[0182] It should be noted that, in the embodiments of the present application, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0183] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0184] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0185] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0186] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0187] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims. Industrial Applicability

[0188] An embodiment of the present application provides a decoding method, a decoder, and a storage medium. During decoding, image enhancement is performed on the decompressed two-dimensional image of the current point cloud (including at least one of an occupancy image, a geometric image, and an attribute image) to obtain an enhanced two-dimensional image. The enhanced two-dimensional image is used to reconstruct the point cloud, which can improve the quality of the three-dimensional reconstructed point cloud and improve the decoder performance without changing the point cloud bit rate.

[0189] Furthermore, a new loss function is provided for the neural network-based image enhancement module, which can enable the geometric image enhancement module and / or attribute image enhancement module to focus on enhancing the quality of occupied points and ignore enhancing the quality of unoccupied points, thereby improving the point cloud geometric reconstruction quality and / or attribute reconstruction quality, and further improving the quality of three-dimensional reconstructed point cloud.

Claims

1. A decoding method, applied to a decoder, wherein: The method comprises: Decode the code stream to determine the decompressed 2D image of the current point cloud; Performing image enhancement processing on the decompressed two-dimensional image using an image enhancement module to determine an enhanced two-dimensional image; Reconstructing a point cloud according to the enhanced two-dimensional image to determine a reconstructed point cloud; The two-dimensional image includes at least one of the following: an occupancy image including occupancy information of the current point cloud, a geometric image including geometric information of the current point cloud, and an attribute image including attribute information of the current point cloud.

2. The method according to claim 1, wherein: The image enhancement module includes at least one of the following: an occupancy image enhancement module for performing image enhancement processing on the occupancy image, a geometric image enhancement module for performing image enhancement processing on the geometric image, and an attribute image enhancement module for performing image enhancement processing on the attribute image.

3. The method according to claim 2, wherein: The image enhancement module includes an image enhancement module based on a neural network.

4. The method according to claim 3, wherein: The image enhancement module comprises: a first input layer, at least one residual block based on channel-spatial attention, a first output layer and an adder; The first input layer includes: at least one convolutional layer and an activation layer; The first output layer includes: at least one convolutional layer and an activation layer; The image enhancement module is configured as follows: Using the decompressed two-dimensional image as the input image of the image enhancement module, sequentially passing through a first input layer, at least one residual block based on channel-spatial attention connected in series, and a first output layer, and outputting residual information; The input image of the image enhancement module and the residual information output by the first output layer are added through the adder to obtain the enhanced two-dimensional image.

5. The method according to claim 4, wherein: The channel-spatial attention-based residual block comprises: a second input layer, a channel attention block, a spatial attention block and an adder, wherein the second input layer comprises at least one convolutional layer and an activation layer; The channel-spatial attention-based residual block is configured as: Using the output information of the first input layer as the input information of the residual block based on channel-spatial attention, and passing through the second input layer, the channel attention block and the spatial attention block in sequence; The input information of the channel-spatial attention based residual block and the output information of the spatial attention block are added through the adder to obtain the output information of the channel-spatial attention based residual block.

6. The method according to claim 5, wherein: The channel attention block includes: a first global average pooling layer, a first global maximum pooling layer, a shared multi-layer perceptron, an adder, a Sigmoid activation layer and a multiplier; The channel attention block is configured as: Using the output information of the second input layer as the input information of the channel attention block, and inputting it into the first global average pooling layer and the first global maximum pooling layer at the same time; The output information of the first global average pooling layer and the output information of the first global maximum pooling layer are simultaneously input into the shared multi-layer perceptron to obtain two output information; The two output information of the shared multilayer perceptron are added by the adder and then pass through the Sigmoid activation layer; The input information of the channel attention block and the output information of the Sigmoid activation layer are multiplied by the multiplier to obtain the output information of the channel attention block.

7. The method according to claim 6, wherein: The shared multi-layer perceptron includes: an input convolution layer, a ReLU activation layer and an output convolution layer.

8. The method according to claim 5, wherein: The spatial attention block includes: a second global average pooling layer, a second global maximum pooling layer, a splicing layer, a convolution layer, a Sigmoid activation layer and a multiplier; The spatial attention block is configured as: The output information of the channel attention block is used as the input information of the spatial attention block and is simultaneously input into the second global average pooling layer and the second global maximum pooling layer; The output information of the second global average pooling layer and the output information of the second global maximum pooling layer are spliced ​​through the splicing layer, and then sequentially pass through the convolution layer and the Sigmoid activation layer; The input information of the spatial attention block and the output information of the Sigmoid activation layer are multiplied by the multiplier to obtain the output information of the spatial attention block.

9. The method according to any one of claims 3 to 8, wherein: When the image enhancement module includes a geometric image enhancement module based on a neural network, the method further includes: In the process of training the geometric image enhancement module, the occupied points in the geometric image are determined according to the occupied image, and the loss function of the geometric image enhancement module is determined according to the geometric prediction values ​​and the geometric original values ​​of the occupied points in the geometric image; According to the loss function of the geometric image enhancement module, the parameters of the geometric image enhancement module are adjusted.

10. The method according to any one of claims 3 to 8, wherein: When the image enhancement module includes an attribute image enhancement module based on a neural network, the method further includes: In the process of training the attribute image enhancement module, the occupied points in the geometric image are determined according to the occupied image, and the loss function of the attribute image enhancement module is determined according to the attribute prediction value and the attribute original value of the occupied point in the attribute image; According to the loss function of the attribute image enhancement module, the parameters of the attribute image enhancement module are adjusted.

11. The method according to claim 9 or 10, wherein: The loss function includes at least one of the following: mean square error MSE, mean absolute error MAE, least square error LSE, and least absolute error LAE.

12. The method according to claim 2, wherein: The image enhancement module includes an image enhancement module based on an image enhancement algorithm.

13. The method according to claim 12, wherein: The image enhancement algorithm includes at least one of the following: an interpolation algorithm, a filtering algorithm, a histogram equalization algorithm, a logarithmic image enhancement algorithm, an exponential image enhancement algorithm, a Gamma correction algorithm, and a Laplace transform algorithm.

14. The method according to claim 1, wherein: The decoding code stream determines the two-dimensional image of the current point cloud decompression, including: Decode the code stream to obtain metadata sub-code stream, occupancy sub-code stream, geometry sub-code stream and attribute sub-code stream; Performing metadata decoding on the metadata sub-code stream to obtain metadata of the current point cloud; Performing video decoding on the occupancy sub-stream to obtain a decompressed occupancy image of the current point cloud; Performing video decoding on the geometric sub-code stream to obtain a decompressed geometric image of the current point cloud; The attribute sub-code stream is decoded to obtain a decompressed attribute image of the current point cloud.

15. The method according to claim 1, wherein: The step of reconstructing a point cloud according to the enhanced two-dimensional image to determine the reconstructed point cloud comprises: Reconstructing geometric information and attribute information according to the enhanced two-dimensional image to obtain reconstructed geometric information and attribute information; Point cloud reconstruction is performed according to the reconstructed geometric information and attribute information to determine a reconstructed point cloud.

16. The method according to claim 15, wherein: The step of reconstructing the point cloud according to the reconstructed geometric information and attribute information to determine the reconstructed point cloud includes: post-processing the reconstructed geometric information; post-processing the reconstructed attribute information; Point cloud reconstruction is performed based on post-processed geometric information and attribute information to determine the reconstructed point cloud.

17. A decoder, comprising: A decoding module, configured to decode the code stream and determine a two-dimensional image of the current point cloud decompression; An image enhancement module, configured to perform image enhancement processing on the decompressed two-dimensional image to determine an enhanced two-dimensional image; A reconstruction module, configured to perform point cloud reconstruction based on the enhanced two-dimensional image and determine a reconstructed point cloud; The two-dimensional image includes at least one of the following: an occupancy image including occupancy information of the current point cloud, a geometric image including geometric information of the current point cloud, and an attribute image including attribute information of the current point cloud.

18. A decoder, comprising: A memory and a processor; wherein, The memory is used to store a computer program that can be run on the second processor; The processor is configured to execute the method according to any one of claims 1 to 16 when running the computer program.

19. A computer-readable storage medium storing a computer program, wherein the computer program, when executed, implements the decoding method according to any one of claims 1 to 16.

Images