Encoding and decoding method, encoder, decoder and readable storage medium

Abstract

The embodiment of the invention discloses an encoding and decoding method, an encoder, a decoder and a readable storage medium, and the method comprises the steps: determining a reconstruction point set based on a reconstruction point cloud; wherein the reconstruction point set comprises at least one point; inputting the geometric information of the points in the reconstruction point set and the reconstruction value of the to-be-processed attribute into a preset network model, and determining a processing value of the to-be-processed attribute of the points in the reconstruction point set based on the preset network model; and determining a processed point cloud corresponding to the reconstructed point cloud according to the processing value of the to-be-processed attribute of the point in the reconstructed point set. Therefore, the quality of the attribute information is enhanced by using the preset network model, so that the quality of the point cloud can be improved, and the visual effect is improved; and the compression performance of the point cloud can be improved.

Description

Coding and decoding method, encoder, decoder and readable storage medium Technical Field

[0001] The embodiments of the present application relate to the field of point cloud data processing technology, and in particular to a coding and decoding method, an encoder, a decoder, and a readable storage medium. Background Art

[0002] A 3D point cloud is a three-dimensional data format composed of a large number of points with geometric and attribute information. Because point clouds typically have a large number of points, resulting in a large amount of data and space, research is underway on point cloud compression to improve storage, transmission, and subsequent processing. The Geometry-Based Point Cloud Compression (G-PCC) codec framework is a geometry-based point cloud compression platform proposed and continuously refined by relevant organizations.

[0003] However, in related technologies, the existing G-PCC encoding and decoding framework only performs basic reconstruction on the original point cloud. In the case of lossy attribute coding, the reconstructed point cloud may be quite different from the original point cloud after reconstruction, and the distortion may be more serious, thus affecting the quality and visual effect of the entire point cloud.

[0004] Summary of the Invention

[0005] The embodiments of the present application provide a coding and decoding method, an encoder, a decoder, and a readable storage medium, which can improve the quality of point clouds, improve visual effects, and thereby improve the compression performance of point clouds.

[0006] The technical solution of the embodiment of the present application can be implemented as follows:

[0007] In a first aspect, an embodiment of the present application provides a decoding method, the method comprising:

[0008] Determine a reconstruction point set based on the reconstructed point cloud; wherein the reconstruction point set includes at least one point;

[0009] Inputting the geometric information of the points in the reconstructed point set and the reconstructed values ​​of the attributes to be processed into a preset network model, and determining the processed values ​​of the attributes to be processed of the points in the reconstructed point set based on the preset network model;

[0010] According to the processing value of the to-be-processed attribute of the point in the reconstructed point set, a processed point cloud corresponding to the reconstructed point cloud is determined.

[0011] In a second aspect, an embodiment of the present application provides an encoding method, the method comprising:

[0012] Encoding and reconstruction processing are performed according to the original point cloud to obtain a reconstructed point cloud;

[0013] Determine a reconstruction point set based on the reconstructed point cloud; wherein the reconstruction point set includes at least one point;

[0014] Inputting the geometric information of the points in the reconstructed point set and the reconstructed values ​​of the attributes to be processed into a preset network model, and determining the processed values ​​of the attributes to be processed of the points in the reconstructed point set based on the preset network model;

[0015] According to the processing value of the to-be-processed attribute of the point in the reconstructed point set, a processed point cloud corresponding to the reconstructed point cloud is determined.

[0016] In a third aspect, an embodiment of the present application provides an encoder, which includes an encoding unit, a first extraction unit, a first model unit, and a first aggregation unit; wherein,

[0017] The encoding unit is configured to perform encoding and reconstruction processing according to the original point cloud to obtain a reconstructed point cloud;

[0018] A first extraction unit is configured to determine a reconstruction point set based on the reconstructed point cloud; wherein the reconstruction point set includes at least one point;

[0019] A first model unit is configured to input geometric information of a point in the reconstructed point set and a reconstructed value of the attribute to be processed into a preset network model, and determine a processed value of the attribute to be processed of the point in the reconstructed point set based on the preset network model;

[0020] The first aggregation unit is configured to determine a processed point cloud corresponding to the reconstructed point cloud according to a processing value of a to-be-processed attribute of a point in the reconstructed point set.

[0021] In a fourth aspect, an embodiment of the present application provides an encoder, comprising a first memory and a first processor; wherein,

[0022] a first memory for storing a computer program capable of running on the first processor;

[0023] The first processor is configured to execute the method according to the second aspect when running a computer program.

[0024] In a fifth aspect, an embodiment of the present application provides a decoder, comprising a second extraction unit, a second model unit, and a second aggregation unit; wherein,

[0025] A second extraction unit is configured to determine a reconstruction point set based on the reconstructed point cloud; wherein the reconstruction point set includes at least one point;

[0026] a second model unit configured to input geometric information of points in the reconstructed point set and reconstructed values ​​of attributes to be processed into a preset network model, and determine processed values ​​of the attributes to be processed of the points in the reconstructed point set based on the preset network model;

[0027] The second aggregation unit is configured to determine a processed point cloud corresponding to the reconstructed point cloud according to the processing value of the to-be-processed attribute of the point in the reconstructed point set.

[0028] In a sixth aspect, an embodiment of the present application provides a decoder, the decoder comprising a second memory and a second processor; wherein,

[0029] a second memory for storing a computer program capable of running on the second processor;

[0030] The second processor is configured to execute the method according to the first aspect when running a computer program.

[0031] In a seventh aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed, it implements the method described in the first aspect or the method described in the second aspect.

[0032] The embodiments of the present application provide a coding and decoding method, an encoder, a decoder, and a readable storage medium. Whether at the encoding or decoding end, a reconstructed point set is determined based on a reconstructed point cloud; the geometric information of the points in the reconstructed point set and the reconstructed values ​​of the attributes to be processed are input into a preset network model; the processed values ​​of the attributes to be processed of the points in the reconstructed point set are determined based on the preset network model; and a processed point cloud corresponding to the reconstructed point cloud is determined based on the processed values ​​of the attributes to be processed of the points in the reconstructed point set. In this way, the quality enhancement processing of the attribute information of the reconstructed point cloud based on the preset network model not only achieves end-to-end operation, but also determines the reconstructed point set from the reconstructed point cloud and implements patching operations on the reconstructed point cloud, effectively reducing resource consumption and improving the robustness of the model. Furthermore, by using the geometric information as an auxiliary input to the preset network model, when the attribute information of the reconstructed point cloud is quality enhanced through the preset network model, the texture of the processed point cloud can be made clearer and the transition more natural, effectively improving the quality and visual effect of the point cloud, thereby improving the compression performance of the point cloud. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] FIG1 is a schematic diagram of a composition framework of a G-PCC encoder;

[0034] FIG2 is a schematic diagram of a composition framework of a G-PCC decoder;

[0035] FIG3 is a schematic structural diagram of a zero-stroke encoding;

[0036] FIG4 is a schematic diagram of a flow chart of a decoding method provided in an embodiment of the present application;

[0037] FIG5 is a schematic diagram of a network structure of a preset network model provided in an embodiment of the present application;

[0038] FIG6 is a schematic diagram of a network structure of a graph attention mechanism module provided in an embodiment of the present application;

[0039] FIG7 is a schematic diagram of a detailed flow chart of a decoding method provided in an embodiment of the present application;

[0040] FIG8 is a schematic diagram of a network framework based on a preset network model provided in an embodiment of the present application;

[0041] FIG9 is a schematic diagram of a network structure of a GAPLayer module provided in an embodiment of the present application;

[0042] FIG10 is a schematic diagram of a network structure of a Single-Head GAPLayer module provided in an embodiment of the present application;

[0043] FIG11 is a schematic diagram of test results of RAHT transformation under C1 test conditions provided in an embodiment of the present application;

[0044] FIG12A and FIG12B are schematic diagrams showing comparisons of point cloud images before and after quality enhancement provided by an embodiment of the present application;

[0045] FIG13 is a schematic diagram of a flow chart of an encoding method provided in an embodiment of the present application;

[0046] FIG14 is a schematic diagram of the structure of an encoder provided in an embodiment of the present application;

[0047] FIG15 is a schematic diagram of a specific hardware structure of an encoder provided in an embodiment of the present application;

[0048] FIG16 is a schematic diagram of the structure of a decoder provided in an embodiment of the present application;

[0049] FIG17 is a schematic diagram of a specific hardware structure of a decoder provided in an embodiment of the present application;

[0050] FIG18 is a schematic diagram of the composition structure of a coding and decoding system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below with reference to the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present application.

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0054] In the following description, reference is made to "some embodiments," which describe a subset of all possible embodiments. However, it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict. It should also be noted that the terms "first, second, and third" in the embodiments of the present application are only used to distinguish similar objects and do not represent a specific ordering of the objects. It is understood that "first, second, and third" may be interchanged in a specific order or sequential order where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0055] Before further explaining the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained first. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations:

[0056] Geometry-based Point Cloud Compression (G-PCC or GPCC)

[0057] Video-based Point Cloud Compression (V-PCC or VPCC)

[0058] Point Cloud Quality Enhancement Net (PCQEN)

[0059] Octree

[0060] Bounding Box

[0061] K Nearest Neighbor (KNN)

[0062] Level of Detail (LOD)

[0063] Predicting Transform

[0064] Lifting Transform

[0065] Region Adaptive Hierarchal Transform (RAHT)

[0066] Multiple Layer Perceptron (MLP)

[0067] Farthest point sampling (FPS)

[0068] Peak Signal to Noise Ratio (PSNR)

[0069] Mean Square Error (MSE)

[0070] Concatenate, Concat / Cat

[0071] Common Test Condition (CTC)

[0072] Luminance component (Luminance, Luma or Y)

[0073] Chroma blue (Cb)

[0074] Chroma red (Cr)

[0075] Point cloud is a three-dimensional representation of the surface of an object. Point cloud (data) of the surface of an object can be collected through acquisition equipment such as photoelectric radar, lidar, laser scanner, and multi-view camera.

[0076] A point cloud refers to a collection of massive three-dimensional points. The points in the point cloud can include both their location information and their attribute information. For example, the location information of a point can be its three-dimensional coordinate information. The location information of a point can also be referred to as its geometric information. For example, the attribute information of a point can include color information and / or reflectivity, etc. For example, color information can be information in any color space. For example, color information can be RGB information, where R represents red (Red, R), G represents green (Green, G), and B represents blue (Blue, B). For another example, color information can be luminance and chrominance (YCbCr, YUV) information, where Y represents brightness, Cb (U) represents blue chrominance, and Cr (V) represents red chrominance.

[0077] For example, a point cloud obtained based on the principle of laser measurement can include the three-dimensional coordinate information of the point and the laser reflection intensity (reflectance) of the point. For another example, a point cloud obtained based on the principle of photogrammetry can include the three-dimensional coordinate information of the point and the color information of the point. For another example, a point cloud obtained by combining the principles of laser measurement and photogrammetry can include the three-dimensional coordinate information of the point, the laser reflection intensity (reflectance) of the point, and the color information of the point.

[0078] Point clouds can be divided into the following categories according to the acquisition method:

[0079] The first type of static point cloud: the object is stationary and the device used to obtain the point cloud is also stationary;

[0080] The second type of dynamic point cloud: the object is moving, but the device that obtains the point cloud is stationary;

[0081] The third type of dynamic point cloud acquisition: the device that acquires the point cloud is moving.

[0082] For example, point clouds can be divided into two categories according to their usage:

[0083] Category 1: Machine perception point cloud, which can be used in scenarios such as autonomous navigation systems, real-time inspection systems, geographic information systems, visual sorting robots, and disaster relief robots;

[0084] Category 2: Human eye perception point cloud, which can be used in point cloud application scenarios such as digital cultural heritage, free viewpoint broadcasting, 3D immersive communication, and 3D immersive interaction.

[0085] Since point clouds are a collection of massive points, storing point clouds not only consumes a lot of memory, but is also not conducive to transmission. There is also not enough bandwidth to support direct transmission of point clouds at the network layer without compression. Therefore, point clouds need to be compressed.

[0086] To date, the point cloud coding framework that can compress point clouds can be the G-PCC codec framework or the V-PCC codec framework provided by the Moving Picture Experts Group (MPEG), or the AVS-PCC codec framework provided by the Audio Video Standard (AVS). Among them, the G-PCC codec framework can be used to compress the first type of static point cloud and the third type of dynamically acquired point cloud, and the V-PCC codec framework can be used to compress the second type of dynamic point cloud. In the embodiments of the present application, the description is mainly based on the G-PCC codec framework.

[0087] In the embodiment of the present application, a three-dimensional point cloud is composed of a large number of points with information such as coordinates and colors, and is a three-dimensional data format. Since point clouds usually have a large number of points, a large amount of data and a large space occupied, in order to better store, transmit and subsequently process them, relevant organizations (for example, the International Organization for Standardization (ISO), the International Electrotechnical Commission (IEC), the Joint Technical Committee for Information Technology (JTC1) or the Seventh Generation Working Group (Work Group 7, WG7), etc.) are currently conducting research on point cloud compression. The G-PCC codec framework is a geometry-based point cloud compression platform proposed and continuously improved by these organizations.

[0088] Specifically, in the point cloud G-PCC encoding and decoding framework, after the point cloud of the input 3D image model is sliced, each slice can be encoded independently.

[0089] Figure 1 is a schematic diagram of the composition framework of a G-PCC encoder. As shown in Figure 1, the G-PCC encoder is applied to a point cloud encoder. In the G-PCC encoding framework, for the point cloud data to be encoded, the point cloud data is first divided into multiple slices through slice division. In each slice, the geometric information of the point cloud and the attribute information corresponding to each point cloud are encoded separately. In the geometric encoding process, the geometric information is transformed so that all the point clouds are contained in a Bounding Box, and then quantized. This quantization step mainly plays a scaling role. Due to the quantization rounding, the geometric information of some point clouds is the same, so whether to remove duplicate points is determined based on parameters. The process of quantization and removal of duplicate points is also called voxelization. The Bounding Box is then divided into an octree. In the octree-based geometry encoding process, the bounding box is divided into eight equal parts, and the non-empty sub-cubes (those containing points in the point cloud) are further divided into eight equal parts until the resulting leaf nodes are 1×1×1 unit cubes. The points in the leaf nodes are then arithmetically encoded to generate a binary geometry bitstream, also known as the geometry codestream. In the triangle soup (Trisoup)-based geometry encoding process, octree partitioning is also performed first. However, unlike the octree-based geometry encoding process, the Trisoup does not require the point cloud to be partitioned into 1×1×1 unit cubes. Instead, the partitioning stops when the point cloud reaches a sub-block with a side length of W. Based on the surface formed by the distribution of the point cloud in each block, up to twelve intersection points (vertices) generated by this surface and the twelve edges of the block are obtained. These vertices are then arithmetically encoded (surface fitting is performed based on the intersection points) to generate a binary geometry bitstream, also known as the geometry codestream. Vertex is also used to implement the geometric reconstruction process, and the reconstructed collection information is used when encoding the attributes of the point cloud.

[0090] During the attribute encoding process, after geometric encoding is completed and the geometric information is reconstructed, color conversion is performed to convert the color information (i.e., attribute information) from the RGB color space to the YUV color space. The reconstructed geometric information is then used to recolor the point cloud so that the unencoded attribute information corresponds to the reconstructed geometric information. Attribute encoding is primarily performed on color information. There are two main transformation methods for color information encoding: one is the distance-based lifting transformation that relies on LOD partitioning, and the other is direct RAHT transformation. Both methods convert color information from the spatial domain to the frequency domain, obtaining high-frequency and low-frequency coefficients through the transformation. Finally, the coefficients are quantized (i.e., quantized coefficients). Finally, the geometrically encoded data that has undergone octree partitioning and surface fitting is sliced ​​and synthesized with the attribute encoded data processed with the quantized coefficients. The vertex coordinates of each block are encoded in sequence (i.e., arithmetic coding) to generate a binary attribute bit stream, i.e., the attribute code stream.

[0091] Figure 2 is a schematic diagram of the composition framework of a G-PCC decoder. As shown in Figure 2, the G-PCC decoder is applied to a point cloud encoder. In the G-PCC decoding framework, for the acquired binary code stream, the geometric bit stream and attribute bit stream in the binary code stream are first decoded independently. When decoding the geometric bit stream, the geometric information of the point cloud is obtained through arithmetic decoding-octree synthesis-surface fitting-reconstructed geometry-inverse coordinate conversion; when decoding the attribute bit stream, the attribute information of the point cloud is obtained through arithmetic decoding-inverse quantization-LOD-based lifting inverse transform or RAHT-based inverse transform-inverse color conversion, and the three-dimensional image model of the point cloud data to be encoded is restored based on the geometric information and attribute information.

[0092] In the G-PCC encoder shown in FIG1 , LOD partitioning is mainly used for two methods: predicting transform and lifting transform in point cloud attribute transformation.

[0093] It should also be noted that the LOD division process occurs after the point cloud geometry is reconstructed, at which point the geometric coordinate information of the point cloud is directly accessible. The point cloud is divided into multiple LODs based on the Euclidean distance between the point cloud points. The color of each LOD point is decoded in turn, the number of zeros in the zero run encoding technique (denoted by zero_cnt) is calculated, and the residual is decoded based on the value of zero_cnt.

[0094] Among them, the decoding operation is performed according to the encoding zero-run encoding method. First, the size of the first zero_cnt in the code stream is decoded. If it is greater than 0, it means that there are consecutive zero_cnt residuals of 0; if zero_cnt is equal to 0, it means that the attribute residual of this point is not 0, and the corresponding residual value is decoded. Then, the decoded residual value is dequantized and added to the color prediction value of the current point to obtain the reconstructed value of the point. This operation is continued until all point cloud points are decoded. For example, Figure 3 is a structural diagram of a zero-run encoding. As shown in Figure 3, if the residual value is 73, 50, 32, or 15, then zero_cnt is equal to 0; if the residual value is 0 and there is only one, then zero_cnt is equal to 1; if the residual value is 0 and there are N, then zero_cnt is equal to N.

[0095] In other words, the color reconstruction value of the current point (reconstructedColor) is calculated based on the color prediction value (predictedColor) under the current prediction mode and the residual value (residual) of the color dequantization under the current prediction mode, that is, reconstructedColor = predictedColor + residual. Furthermore, the current point will be used as the nearest neighbor of the subsequent LOD midpoint, and the color reconstruction value of the current point will be used to predict the attributes of the subsequent points.

[0096] In the related technologies, most of the technologies for enhancing the quality of reconstructed point cloud attributes in the G-PCC codec framework use some classic algorithms, while there are few technologies that use deep learning methods for quality enhancement. The following are two algorithms for post-processing quality enhancement of reconstructed point clouds:

[0097] (1) Kalman filter algorithm: Kalman filter is an efficient recursive filter. It can gradually reduce the prediction error of the system and is particularly suitable for stationary random signals. Kalman filter uses the estimation of the previous state to find the optimal value of the current state. It contains three main modules: prediction module, correction module and update module. The attribute reconstruction value of the previous point is used as the measurement value, and the attribute prediction value of the current point is Kalman filtered (basic method) to correct the accumulated error in the Predicting Transform process. Then the algorithm can further take some optimizations: retain the true value of some points at equal intervals during the encoding process as the measurement value of the Kalman filter, which can improve the filtering performance and attribute prediction accuracy; disable the Kalman filter when the signal standard deviation is large; filter only the U and V components, etc.

[0098] (2) Wiener filter algorithm: The Wiener filter uses the minimum mean square error as the criterion, that is, to minimize the error between the reconstructed point cloud and the original point cloud. At the encoding end, a set of optimal coefficients is calculated through the neighborhood of each reconstructed point and each point is filtered; by judging whether the quality of the point cloud after filtering is improved or not, the coefficients are selectively written into the code stream and transmitted to the decoding end; at the decoding end, the optimal coefficients can be decoded and the reconstructed point cloud can be post-processed. Then, the algorithm can also be further optimized: optimizing the number of adjacent points; dividing the point cloud into blocks and then filtering it to reduce memory consumption when the point cloud is large, etc.

[0099] In other words, the G-PCC codec framework only performs basic reconstruction of the point cloud sequence; for attribute lossy (or near-lossless) encoding methods, no corresponding post-processing operations are performed after reconstruction to further improve the attribute quality of the reconstructed point cloud. This may cause the reconstructed point cloud to differ significantly from the original point cloud, resulting in severe distortion, which will affect the quality and visual effect of the entire point cloud.

[0100] However, some classic algorithms proposed in related technologies have relatively simple principles and single methods, making it difficult to achieve superior results. The final quality still has significant room for improvement. Deep learning offers several advantages over traditional algorithms, including: stronger learning capabilities, the ability to extract underlying, subtle features; wide coverage, adaptability, and robustness, enabling it to solve more complex problems; data-driven, with a higher upper limit; and excellent portability. Therefore, a point cloud quality enhancement technique based on a neural network was proposed.

[0101] The embodiment of the present application provides a coding and decoding method, which determines a reconstructed point set based on a reconstructed point cloud; wherein the reconstructed point set includes at least one point; the geometric information of the points in the reconstructed point set and the reconstructed values ​​of the attributes to be processed are input into a preset network model, and the processed values ​​of the attributes to be processed of the points in the reconstructed point set are determined based on the preset network model; and the processed point cloud corresponding to the reconstructed point cloud is determined according to the processed values ​​of the attributes to be processed of the points in the reconstructed point set. In this way, the quality enhancement processing of the attribute information of the reconstructed point cloud based on the preset network model not only realizes end-to-end operation, but also determines the reconstructed point set from the reconstructed point cloud and realizes the block operation of the reconstructed point cloud, effectively reducing resource consumption and improving the robustness of the model; in addition, when the geometric information is used as an auxiliary input of the preset network model and the attribute information of the reconstructed point cloud is subjected to quality enhancement processing through the preset network model, the texture of the processed point cloud can be made clearer and the transition more natural, effectively improving the quality and visual effect of the point cloud, thereby improving the compression performance of the point cloud.

[0102] The following will clearly and completely describe the embodiments of the present application in conjunction with the accompanying drawings.

[0103] In one embodiment of the present application, referring to FIG4 , a schematic flow chart of a decoding method provided by an embodiment of the present application is shown. As shown in FIG4 , the method may include:

[0104] S401: Determine a reconstruction point set based on the reconstructed point cloud; wherein the reconstruction point set includes at least one point.

[0105] S402: Inputting geometric information of points in the reconstructed point set and reconstructed values ​​of attributes to be processed into a preset network model, and determining processed values ​​of the attributes to be processed of the points in the reconstructed point set based on the preset network model.

[0106] S403: Determine a processed point cloud corresponding to the reconstructed point cloud according to the processed values ​​of the to-be-processed attributes of the points in the reconstructed point set.

[0107] It should be noted that the decoding method described in the embodiment of the present application specifically refers to a point cloud decoding method, which can be applied to a point cloud decoder (in the embodiment of the present application, it can be referred to as a "decoder").

[0108] It should also be noted that in the embodiments of the present application, the decoding method is mainly used to post-process the attribute information of the reconstructed point cloud obtained by G-PCC decoding. Specifically, a graph-based point cloud quality enhancement network (PCQEN) is proposed. In this preset network model, a graph structure is constructed for each point using geometric information and the reconstructed values ​​of the attributes to be processed. Then, graph convolution and graph attention mechanism operations are used for feature extraction. By learning the residual between the reconstructed point cloud and the original point cloud, the reconstructed point cloud can be made as close to the original point cloud as possible, thereby achieving the purpose of quality enhancement.

[0109] It can be understood that in the embodiment of the present application, for reconstructing the point cloud, for each point, it includes geometric information and attribute information; wherein, the geometric information represents the spatial position of the point, which can also be called three-dimensional geometric coordinate information, expressed by (x, y, z); the attribute information represents the attribute value of the point, such as the color component value.

[0110] Here, the attribute information may include color components, specifically color information of any color space. For example, the attribute information may be color information of RGB space, color information of YUV space, color information of YCbCr space, etc., which is not limited in the present embodiment.

[0111] In the embodiment of the present application, the color component may include at least one of the following: a first color component, a second color component, and a third color component. Thus, taking the attribute information as a color component as an example, if the color component conforms to the RGB color space, then the first color component, the second color component, and the third color component can be determined to be, in order: R component, G component, and B component; if the color component conforms to the YUV color space, then the first color component, the second color component, and the third color component can be determined to be, in order: Y component, U component, and V component; if the color component conforms to the YCbCr color space, then the first color component, the second color component, and the third color component can be determined to be, in order: Y component, Cb component, and Cr component.

[0112] It can also be understood that in the embodiment of the present application, for each point, in addition to the color component, the attribute information of the point may also include reflectivity, refractive index or other attributes, which is not specifically limited here.

[0113] Furthermore, in the embodiments of the present application, the attribute to be processed refers to the attribute information currently being enhanced. Taking color components as an example, the attribute to be processed can be one-dimensional information, such as a single first color component, a second color component, or a third color component; or two-dimensional information, such as a combination of any two of the first, second, and third color components; or even three-dimensional information consisting of the first, second, and third color components, without specific limitation herein.

[0114] That is to say, for each point in the reconstructed point cloud, the attribute information may include a three-dimensional color component. However, when using the preset network model to perform quality enhancement processing on the attributes to be processed, only one color component may be processed at a time, that is, a single color component and geometric information are used as inputs to the preset network model to achieve quality enhancement processing on the single color component (the remaining color components remain unchanged); then the same method is used for the remaining two color components, and they are sent to the corresponding preset network model for quality enhancement. Alternatively, when using the preset network model to perform quality enhancement processing on the attributes to be processed, all three color components and geometric information may be used as inputs to the preset network model, rather than processing only one color component at a time. This can reduce the time complexity, but the quality enhancement effect is slightly reduced.

[0115] Furthermore, in an embodiment of the present application, for the reconstructed point cloud, the reconstructed point cloud can be obtained from the original point cloud after attribute encoding, attribute reconstruction and geometric compensation. Among them, for a point in the original point cloud, the predicted value and residual value of the attribute information of the point can be determined first, and then the predicted value and the residual value can be used to further calculate the reconstructed value of the attribute information of the point, so as to construct a reconstructed point cloud. In some embodiments, the method may also include: parsing the code stream to determine the residual value of the to-be-processed attribute of the point in the original point cloud; performing attribute prediction on the to-be-processed attribute of the point in the original point cloud to determine the predicted value of the to-be-processed attribute of the point in the original point cloud; determining the reconstructed value of the to-be-processed attribute of the point in the original point cloud based on the residual value of the to-be-processed attribute of the point in the original point cloud and the predicted value of the to-be-processed attribute of the point in the original point cloud, and then determining the reconstructed point cloud.

[0116] Specifically, for a point in the original point cloud, when determining the predicted value of the attribute to be processed at that point, the geometric information and attribute information of the point's multiple target neighbor points can be used to predict the attribute information of the point in combination with the geometric information of the point, thereby obtaining the corresponding predicted value. Then, the residual value of the attribute to be processed at that point is added to the predicted value of the attribute to be processed at that point to obtain the reconstructed value of the attribute to be processed at that point. In this way, for a point in the original point cloud, after determining the reconstructed value of the attribute information of that point, the point can be used as the nearest neighbor of the subsequent LOD midpoint to continue to predict the attributes of subsequent points using the reconstructed value of the attribute information of that point, thus obtaining a reconstructed point cloud.

[0117] That is, in the embodiment of the present application, the original point cloud can be directly obtained through the point cloud reading function of the codec program, and the reconstructed point cloud is obtained after all encoding operations are completed. In addition, the reconstructed point cloud in the embodiment of the present application can be the reconstructed point cloud output after decoding, or it can be used as a reference for decoding subsequent point clouds. In addition, the reconstructed point cloud here can be used not only within the prediction loop, that is, as an inloop filter, and can be used as a reference for decoding subsequent point clouds, but also outside the prediction loop, that is, as a post filter, and not used as a reference for decoding subsequent point clouds. This is not specifically limited here.

[0118] It is also understandable that in the embodiments of the present application, considering the number of points included in the reconstructed point cloud, for example, for some large point clouds, the number of points may exceed 10 million; before inputting the preset network model, the reconstructed point cloud can be first extracted into patches. Here, a set of reconstructed points can be regarded as a patch, and each extracted patch contains at least one point.

[0119] In some embodiments, for S401, determining the reconstruction point set based on the reconstructed point cloud may include:

[0120] In the reconstructed point cloud, key points are identified;

[0121] The reconstructed point cloud is extracted and processed according to the key points to determine a set of reconstructed points; wherein there is a corresponding relationship between the key points and the set of reconstructed points.

[0122] In a specific embodiment, determining key points in the reconstructed point cloud may include: performing farthest point sampling processing on the reconstructed point cloud to determine key points.

[0123] In the embodiment of the present application, P key points can be obtained using the farthest point sampling (FPS) method, where P is an integer greater than 0. Here, for these P key points, each key point corresponds to a patch, that is, each key point corresponds to a set of reconstruction points.

[0124] Specifically, for each key point, a patch can be extracted separately to obtain a set of reconstruction points corresponding to each key point. Taking a key point as an example, in some embodiments, extracting and processing the reconstructed point cloud based on the key point to determine the set of reconstruction points may include:

[0125] Perform K-nearest neighbor search in the reconstructed point cloud based on the key points to determine the neighboring points corresponding to the key points;

[0126] Based on the neighboring points corresponding to the key points, the reconstruction point set is determined.

[0127] Furthermore, for K-nearest neighbor search, in a specific embodiment, performing K-nearest neighbor search in the reconstructed point cloud based on the key point to determine the neighboring points corresponding to the key point includes:

[0128] Based on the key points, a K-nearest neighbor search method is used to search for a first preset number of candidate points in the reconstructed point cloud;

[0129] Calculating the distance values ​​between the key point and a first preset number of candidate points respectively, and determining a second preset number of relatively smaller distance values ​​from the obtained first preset number of distance values;

[0130] Determine the neighboring points corresponding to the key point based on the candidate points corresponding to the second preset number of distance values.

[0131] In an embodiment of the present application, the second preset number is less than or equal to the first preset number.

[0132] It should also be noted that, taking a certain key point as an example, the K-nearest neighbor search method can be used to search for a first preset number of candidate points in the reconstructed point cloud, calculate the distance value between the key point and these candidate points, and then select a second preset number of candidate points closest to the key point from these candidate points; use this second preset number of candidate points as the neighboring points corresponding to the key point, and form a reconstructed point set corresponding to the key point based on these neighboring points.

[0133] Furthermore, in embodiments of the present application, the reconstructed point set may or may not include the key point itself. If the reconstructed point set includes the key point itself, then in some embodiments, determining the reconstructed point set based on the neighboring points corresponding to the key point may include determining the reconstructed point set based on the key point and its neighboring points.

[0134] It should also be noted that the reconstruction point set may include n points, where n is an integer greater than 0. For example, the value of n may be 2048, but this is not specifically limited here.

[0135] In one possible implementation, if the reconstructed point set includes the key point itself, the second preset number can be equal to (n-1); that is, after searching the reconstructed point cloud for the first preset number of candidate points using the K-nearest neighbor search method, the distance between the key point and these candidate points is calculated, and then the (n-1) nearest neighbor points to the key point are selected from these candidate points. The reconstructed point set can be formed based on the key point itself and these (n-1) nearest neighbor points. The (n-1) nearest neighbor points here specifically refer to the (n-1) nearest neighbor points in the reconstructed point cloud that are closest in geometric distance to the key point.

[0136] In another possible implementation, if the reconstructed point set does not include the key point itself, the second preset number can be equal to n. That is, after searching the reconstructed point cloud for the first preset number of candidate points using the K-nearest neighbor search method, the distance between the key point and these candidate points is calculated, and then the n nearest neighbor points to the key point are selected from these candidate points. The reconstructed point set can be formed based on these n nearest neighbor points. The n nearest neighbor points here specifically refer to the n nearest neighbor points in the reconstructed point cloud that are closest in geometric distance to the key point.

[0137] It should also be noted that the determination of the number of key points is associated with the number of points in the reconstructed point cloud and the number of points in the reconstructed point set. Therefore, in some embodiments, the method may further include: determining the number of points in the reconstructed point cloud; and determining the number of key points based on the number of points in the reconstructed point cloud and the number of points in the reconstructed point set.

[0138] In a specific embodiment, determining the number of key points based on the number of points in the reconstructed point cloud and the number of points in the reconstructed point set may include:

[0139] Determine the first factor;

[0140] Calculate the product of the number of points in the reconstructed point cloud and the first factor;

[0141] The number of key points is determined based on the product and the number of points in the reconstructed point set.

[0142] In the embodiment of the present application, the first factor can be represented by γ, which is called a repetition rate factor and is used to control the average number of times each point is sent to the preset network model. For example, the value of γ can be 3, but this is not specifically limited here.

[0143] In a more specific embodiment, assuming that the number of points in the reconstructed point cloud is N, the number of points in the reconstructed point set is n, and the number of key points is P, the relationship between the three is as follows:

[0144]

[0145] That is to say, for reconstructing the point cloud, we can first use the farthest point sampling method to determine P key points, and then extract patches based on each key point. Specifically, we perform a KNN search with K=n on each key point, so that we can obtain P patches of size n, that is, P reconstruction point sets, each of which includes n points.

[0146] It's also important to note that for points in the reconstructed point cloud, there may be duplication among the P reconstructed point sets. In other words, a point may appear in multiple reconstructed point sets, or it may not appear in any of the P reconstructed point sets. This is where the first factor (γ) comes in: it controls the average repetition rate of each point in the P reconstructed point sets, thereby improving the quality of the point cloud during the final patch aggregation.

[0147] Furthermore, in the embodiments of the present application, since the point cloud is usually represented by the RGB color space, when the quality enhancement processing of the attribute to be processed is performed using the preset network model, the YUV color space is usually used. Therefore, before the geometric information of the points in the reconstructed point set and the reconstructed values ​​of the attribute to be processed are input into the preset network model, the color components need to be converted into a color space. Specifically, in some embodiments, if the color components do not conform to the YUV color space, the color components of the points in the reconstructed point set are converted into a color space so that the converted color components conform to the YUV color space, for example, from the RGB color space to the YUV color space, and then the color components that need quality enhancement (such as the Y component) are extracted and combined with the geometric information to be input into the preset network model.

[0148] In some embodiments, for S402, inputting the geometric information of the points in the reconstructed point set and the reconstructed values ​​of the attributes to be processed into a preset network model, and determining the processed values ​​of the attributes to be processed of the points in the reconstructed point set based on the preset network model may include:

[0149] In the preset network model, the geometric information of the points in the reconstructed point set is used to assist in constructing a graph structure of the reconstructed values ​​of the attributes to be processed of the points in the reconstructed point set, so as to obtain the graph structure of the points in the reconstructed point set; and graph convolution and graph attention mechanism operations are performed on the graph structure of the points in the reconstructed point set to determine the processed values ​​of the attributes to be processed of the points in the reconstructed point set.

[0150] Here, the preset network model can be a neural network model based on deep learning. In the embodiment of the present application, the preset network model can also be referred to as a PCQEN model. The model includes at least a graph attention mechanism module and a graph convolution module to implement graph convolution and graph attention mechanism operations on the graph structure of the points in the reconstructed point set.

[0151] In a specific embodiment, the graph attention mechanism module may include a first graph attention mechanism module and a second graph attention mechanism module, and the graph convolution module may include a first graph convolution module, a second graph convolution module, a third graph convolution module, and a fourth graph convolution module. In addition, the preset network model may further include a first pooling module, a second pooling module, a first splicing module, a second splicing module, a third splicing module, and an addition module; wherein,

[0152] A first input end of the first graph attention mechanism module is used to receive geometric information, and a second input end of the first graph attention mechanism module is used to receive a reconstructed value of the attribute to be processed;

[0153] The first output of the first graph attention mechanism module is connected to the input of the first pooling module, the output of the first pooling module is connected to the input of the first graph convolution module, and the output of the first graph convolution module is connected to the first input of the first splicing module;

[0154] The second output of the first graph attention mechanism module is connected to the first input of the second splicing module, the second input of the second splicing module is used to receive the reconstructed value of the attribute to be processed, and the output of the second splicing module is connected to the input of the second graph convolution module;

[0155] The first input of the second graph attention mechanism module is used to receive geometric information, the second input of the second graph attention mechanism module is connected to the output of the second graph convolution module, the first output of the second graph attention mechanism module is connected to the input of the second pooling module, and the output of the second pooling module is connected to the second input of the first splicing module;

[0156] The second output of the second graph attention mechanism module is connected to the first input of the third splicing module, the second input of the third splicing module is connected to the output of the second graph convolution module, the output of the third splicing module is connected to the input of the third graph convolution module, and the output of the third graph convolution module is connected to the third input of the first splicing module; the output of the second graph convolution module is also connected to the fourth input of the first splicing module;

[0157] The output end of the first splicing module is connected to the input end of the fourth graph convolution module, the output end of the fourth graph convolution module is connected to the first input end of the addition module, the second input end of the addition module is used to receive the reconstructed value of the attribute to be processed, and the output end of the addition module is used to output the processed value of the attribute to be processed.

[0158] Referring to Figure 5, which shows a schematic diagram of the network structure of a preset network model provided in an embodiment of the present application. As shown in Figure 5, the preset network model may include: a first graph attention mechanism module 501, a second graph attention mechanism module 502, a first graph convolution module 503, a second graph convolution module 504, a third graph convolution module 505, a fourth graph convolution module 506, a first pooling module 507, a second pooling module 508, a first splicing module 509, a second splicing module 510, a third splicing module 511, and an addition module 512; and the connection relationship between these modules is specifically shown in Figure 5.

[0159] Among them, the first graph attention mechanism module 501 and the second graph attention mechanism module 502 have the same structure; the first graph convolution module 503, the second graph convolution module 504, the third graph convolution module 505 and the fourth graph convolution module 506 can each include at least one convolution layer (Convolution Layer) for feature extraction, and the convolution kernel of the convolution layer here can be 1×1; the first pooling module 507 and the second pooling module 508 can each include a maximum pooling layer (MaxPooling Layer), and the use of the maximum pooling layer can focus on the most important neighbor information; the first stitching module 509, the second stitching module 510 and the third stitching module 511 are mainly used for feature stitching (mainly channel number stitching). By splicing the existing features with the previous features multiple times, it is possible to better take into account the global and local, different fine-grained features, and establish a connection relationship between different layers; the addition module 512 mainly adds the residual value of the attribute to be processed and the reconstructed value of the attribute to be processed after obtaining the residual value of the attribute to be processed to obtain the processed value of the attribute to be processed, so that the attribute information of the processed point cloud is as close as possible to the original point cloud, thereby achieving the purpose of quality enhancement.

[0160] In addition, for the first graph convolution module 503, it can include three convolution layers, and the number of channels of the three convolution layers is 64, 64, and 64, respectively; for the second graph convolution module 504, it can include three convolution layers, and the number of channels of the three convolution layers is 128, 64, and 64, respectively; for the third graph convolution module 504, it can also include three convolution layers, and the number of channels of the three convolution layers is 256, 128, and 256, respectively; for the fourth graph convolution module 505, it can include three convolution layers, and the number of channels of the three convolution layers is 256, 128, and 1, respectively.

[0161] Furthermore, in an embodiment of the present application, a batch normalization (BatchNorm) layer and an activation layer may be added after the convolution layer to accelerate convergence and increase nonlinear characteristics. Therefore, in some embodiments, the first graph convolution module 503, the second graph convolution module 504, the third graph convolution module 505, and the fourth graph convolution module 506 each further include at least one batch normalization layer and at least one activation layer; wherein the batch normalization layer and the activation layer are connected after the convolution layer. However, it should be noted that the batch normalization layer and the activation layer may not be connected after the last convolution layer in the fourth graph convolution module 506.

[0162] In an embodiment of the present application, the activation layer may include an activation function. Here, the activation function may be a rectified linear unit (ReLU), also known as a linear rectification function, which is an activation function commonly used in artificial neural networks, and generally refers to a nonlinear function represented by a ramp function and its variants. That is to say, the activation function can also be a linear rectification function. Based on the ramp function, there are other variants that are also widely used in deep learning, such as a leaky linear rectification function (Leaky ReLU), a noisy linear rectification function (Noisy ReLU), etc. Exemplarily, a BatchNorm layer is connected after the 1×1 convolution layer except the last layer to speed up convergence and suppress overfitting, and then a LeakyReLU activation function with a slope of 0.2 is connected to add nonlinearity.

[0163] In a specific embodiment, for S402, inputting the geometric information of the points in the reconstructed point set and the reconstructed values ​​of the attributes to be processed into a preset network model, and determining the processed values ​​of the attributes to be processed of the points in the reconstructed point set based on the preset network model may include:

[0164] Performing feature extraction on the geometric information and the reconstructed value of the attribute to be processed by the first graph attention mechanism module 501 to obtain a first graph feature and a first attention feature;

[0165] Performing feature extraction on the first graph features through the first pooling module 507 and the first graph convolution module 503 to obtain second graph features;

[0166] The first attention feature and the reconstructed value of the attribute to be processed are spliced ​​by the second splicing module 510 to obtain a first spliced ​​attention feature;

[0167] Performing feature extraction on the first concatenated attention feature through the second graph convolution module 504 to obtain a second attention feature;

[0168] Performing feature extraction on the geometric information and the second attention feature through the second graph attention mechanism module 502 to obtain a third graph feature and a third attention feature;

[0169] Perform feature extraction on the third image features by the second pooling module 508 to obtain fourth image features;

[0170] splicing the third attention feature and the second attention feature by a third splicing module 511 to obtain a second spliced ​​attention feature;

[0171] Performing feature extraction on the second concatenated attention feature through the third graph convolution module 505 to obtain a fourth attention feature;

[0172] The first splicing module 509 splices the second image feature, the fourth image feature, the second attention feature, and the fourth attention feature to obtain a target feature;

[0173] Performing a convolution operation on the target feature through the fourth graph convolution module 506 to obtain a residual value of the attribute to be processed of the point in the reconstructed point set;

[0174] The addition module 512 performs an addition operation on the residual value of the attribute to be processed of the point in the reconstructed point set and the reconstructed value of the attribute to be processed to obtain the processed value of the attribute to be processed of the point in the reconstructed point set.

[0175] It should be noted that in the embodiment of the present application, the reconstructed point set (i.e., patch) is composed of n points, and the input of the preset network model is the geometric information of these n points and a single color component information. The geometric information can be represented by p, and its size is n×3; the single color component information is represented by c, and its size is n×1; using the geometric information as auxiliary input, a graph structure with a neighborhood size of k can be constructed according to the KNN search method. In this way, the first graph feature obtained by the first graph attention mechanism module 501 is represented by g1, and its size can be n×k×64; the first attention feature is represented by a1, and its size can be n×64; g1 passes through the first pooling module 507 and the first graph convolution module 503 to perform a convolution operation with the number of channels {64, 64, 64}, and the second graph feature obtained is represented by g2, and its size can be n×64; a1 is spliced ​​with the input color component c by the second splicing module 510, and then the second attention feature obtained after the second graph convolution module 504 performs a convolution operation with the number of channels {128, 64, 64} is represented by a2, and its size can be n×64; further, the third graph feature obtained by the second graph attention mechanism module 502 is represented by g3, and its size can be n×k×256 ; The third attention feature is represented by a3, and its size can be n×256; the fourth graph feature obtained by g3 after the second pooling module 508 is represented by g4, and its size can be n×256; a3 and a2 are spliced ​​together by the third splicing module 511, and then the fourth attention feature obtained after the third graph convolution module 505 performs a convolution operation with the number of channels {256, 128, 256} is represented by a4, and its size is n×256; g2, g4, a2, and a4 are spliced ​​together by the first splicing module 509, and then the residual value of the attribute to be processed is represented by r after the fourth graph convolution module 506 performs a convolution operation with the number of channels {256, 128, 1}; r is then added to the input color component c through the addition module 512 to obtain the final output processed color component, that is, the quality enhanced color component c′.

[0176] Here, in order to fully utilize the advantages of Convolutional Neural Networks (CNN), Point Cloud Network (PointNet) provides an effective method for directly learning shape features on unordered 3D point clouds and achieves good performance. However, local features that contribute to better contextual learning are not considered. At the same time, the attention mechanism can effectively capture node representations on graph-based data by focusing on neighboring nodes. Therefore, a new neural network for point clouds, called GAPNet, can be proposed to learn local geometric representations by embedding a graph attention mechanism in the MLP layer. In the embodiment of the present application, a GAPLayer module is introduced here to learn the attention features of each point by highlighting different attention weights on the neighborhood; secondly, in order to mine sufficient features, it adopts a multi-head mechanism, allowing the GAPLayer module to aggregate different features from a single head; thirdly, it is proposed to use an attention pooling layer on the adjacent network to capture local signals to enhance the robustness of the network; finally, GAPNet applies multiple layers of MLP to the attention features and graph features, which can fully extract the input attribute information to be processed.

[0177] That is to say, in the embodiment of the present application, the first graph attention mechanism module 501 and the second graph attention mechanism module 502 have the same structure. Both the first graph attention mechanism module 501 and the second graph attention mechanism module 502 can include a fourth splicing module and a preset number of graph attention mechanism sub-modules; wherein, the graph attention mechanism sub-module can be a single-head (Single-Head) GAPLayer module. In this way, the graph attention mechanism module composed of a preset number of Single-Head GAPLayer modules is a Multi-Head mechanism; that is, the Multi-Head GAPLayer (which can be referred to as the GAPLayer module) refers to the first graph attention mechanism module 501 or the second graph attention mechanism module 502.

[0178] In some embodiments, the internal connection relationship between the first graph attention mechanism module 501 and the second graph attention mechanism module 502 is described as follows:

[0179] In the first graph attention mechanism module 501, input ends of a preset number of graph attention mechanism sub-modules are used to receive geometric information and reconstructed values ​​of attributes to be processed, output ends of the preset number of graph attention mechanism sub-modules are connected to the input end of the fourth splicing module, and the output end of the fourth splicing module is used to output the first graph feature and the first attention feature;

[0180] In the second graph attention mechanism module 502, the input ends of a preset number of graph attention mechanism sub-modules are used to receive geometric information and the second attention feature, the output ends of a preset number of graph attention mechanism sub-modules are connected to the input end of the fourth splicing module, and the output end of the fourth splicing module is used to output the third graph feature and the third attention feature.

[0181] Refer to Figure 6, which shows a schematic diagram of the network structure of a graph attention mechanism module provided by an embodiment of the present application. As shown in Figure 6, the graph attention mechanism module may include: an input module 601, four graph attention mechanism sub-modules 602, and a fourth splicing module 603. Among them, the input module 601 is used to receive geometric information and input information; since the geometric information is a three-dimensional feature, the dimension of the input information (for example, a single color component or multiple color components) is represented by F, so it can be represented by n×(F+3); in addition, the output may include graph features and attention features, the size of the graph feature is represented by n×k×|4×F'|, and the size of the attention feature is represented by n×|4×F'|.

[0182] Here, in order to obtain sufficient structural information and stabilize the network, the outputs of the four graph attention mechanism submodules 602 are connected together through the fourth splicing module 603 to obtain multi-attention features and multi-graph features. Among them, when the graph attention mechanism module shown in Figure 6 is the first graph attention mechanism module 501, the input module 601 receives geometric information and the reconstructed value of the attribute to be processed, the output multi-graph feature is the first graph feature, and the multi-attention feature is the first attention feature; when the graph attention mechanism module shown in Figure 6 is the second graph attention mechanism module 502, the input module 601 receives geometric information and the second attention feature, the output multi-graph feature is the third graph feature, and the multi-attention feature is the third attention feature.

[0183] In some embodiments, taking the first graph attention mechanism module 501 as an example, the first graph attention mechanism module performs feature extraction on the geometric information and the reconstructed value of the attribute to be processed to obtain the first graph feature and the first attention feature, which may include:

[0184] The geometric information and the reconstructed values ​​of the attributes to be processed are input into the graph attention mechanism submodule to obtain the initial graph features and initial attention features;

[0185] Based on a preset number of graph attention mechanism submodules, a preset number of initial graph features and a preset number of initial attention features are obtained;

[0186] splicing a preset number of initial image features through a fourth splicing module to obtain a first image feature;

[0187] A preset number of initial attention features are spliced ​​together through a fourth splicing module to obtain a first attention feature.

[0188] In a specific embodiment, for the graph attention mechanism submodule, the graph attention mechanism submodule includes at least multiple multi-layer perceptron modules; accordingly, inputting the geometric information and the reconstructed value of the attribute to be processed into the graph attention mechanism submodule to obtain the initial graph features and the initial attention features may include:

[0189] The graph structure is constructed based on the reconstruction value of the attribute to be processed assisted by geometric information, and the graph structure of the points in the reconstructed point set is obtained;

[0190] Extract features from the graph structure through at least one multi-layer perceptron module to obtain initial graph features;

[0191] Performing feature extraction on the reconstructed value of the attribute to be processed by at least one multi-layer perceptron module to obtain first intermediate feature information;

[0192] Extracting features from the initial graph using at least one multi-layer perceptron module to obtain second intermediate feature information;

[0193] Performing feature fusion on the first intermediate feature information and the second intermediate feature information using a first preset function to obtain an attention coefficient;

[0194] The attention coefficient is normalized using the second preset function to obtain the feature weight;

[0195] According to the feature weights and the initial graph features, the initial attention features are obtained.

[0196] It should be noted that, in the embodiment of the present application, for the extraction of initial graph features, the features of the graph structure can be extracted by at least one multi-layer perceptron module. Exemplarily, the features of the graph structure can be extracted by one multi-layer perceptron module. For the extraction of first intermediate feature information, the features of the reconstructed values ​​of the attributes to be processed can be extracted by at least one multi-layer perceptron module. Exemplarily, the features of the reconstructed values ​​of the attributes to be processed can be extracted by two multi-layer perceptron modules. For the extraction of second intermediate feature information, the features of the initial graph can be extracted by at least one multi-layer perceptron module. Exemplarily, the features of the initial graph can be extracted by one multi-layer perceptron module. It should be noted that the number of multi-layer perceptron modules here is not specifically limited.

[0197] It should also be noted that in the embodiment of the present application, the first preset function is different from the second preset function. Among them, the first preset function is a nonlinear activation function, such as the LeakyReLU function; the second preset function is a normalized exponential function, such as the softmax function. Here, the softmax function can "compress" a K-dimensional vector z containing any real number into another K-dimensional real vector σ(z), so that the range of each element is between (0,1) and the sum of all elements is 1; in simple terms, the softmax function mainly performs normalization processing.

[0198] It should also be noted that the initial attention features are obtained based on the feature weights and the initial graph features. Specifically, the initial attention features can be generated by performing a linear combination operation on the feature weights and the initial graph features. Here, the initial graph features are n×k×F', the feature weights are n×1×k, and the initial attention features obtained after the linear combination operation are n×F'.

[0199] It can be understood that the embodiment of the present application is a graph-based attention mechanism module. After constructing the graph structure, the attention structure is used to give greater weight to the more important neighborhood features of each point, so as to better utilize graph convolution to extract features. In the first graph attention mechanism module, additional geometric information input is required to assist in constructing the graph structure. The first graph attention mechanism module can be composed of 4 graph attention mechanism sub-modules, so the final output is also obtained by splicing the output of each graph attention mechanism sub-module. In the graph attention mechanism sub-module, after constructing a graph structure with a neighborhood size of k using the KNN search method (for example, k=20 can be selected), the edge features in the graph structure are graph convolved to obtain one of the outputs, namely the initial graph feature (Graph Feature). On the other hand, the input features after two layers of MLP are fused with the graph features after another MLP, and after the activation function LeakyReLU, they are normalized by the softmax function to obtain the k-dimensional feature weight. After applying this weight to the k neighborhood of the current point, i.e., the graph feature, another output, i.e., the initial attention feature (Attention Feature), can be obtained.

[0200] In another specific embodiment, taking the second graph attention mechanism module 502 as an example, the feature extraction of geometric information and the second attention feature by the second graph attention mechanism module to obtain the third graph feature and the third attention feature may include: inputting the geometric information and the second attention feature into the graph attention mechanism sub-module to obtain the second initial graph feature and the second initial attention feature; based on a preset number of graph attention mechanism sub-modules, obtaining a preset number of second initial graph features and a preset number of second initial attention features; in this way, splicing the preset number of second initial graph features by the fourth splicing module to obtain the third graph feature; splicing the preset number of second initial attention features by the fourth splicing module to obtain the third attention feature.

[0201] Further, in some embodiments, for the graph attention mechanism submodule in the second graph attention mechanism module, the inputting of geometric information and the second attention feature into the graph attention mechanism submodule to obtain graph features and attention features may include: constructing a graph structure based on the geometric information to assist the second attention feature to obtain a second graph structure; extracting features of the second graph structure through at least one multi-layer perceptron module to obtain a second initial graph feature; extracting features of the second attention feature through at least one multi-layer perceptron module to obtain third intermediate feature information; extracting features of the second initial graph feature through at least one multi-layer perceptron module to obtain fourth intermediate feature information; performing feature fusion on the third intermediate feature information and the fourth intermediate feature information using a first preset function to obtain a second attention coefficient; normalizing the second attention coefficient using a second preset function to obtain a second feature weight; and obtaining a second initial attention feature based on the second feature weight and the second initial graph feature.

[0202] In this way, based on the preset network model shown in Figure 5, the input of the preset network model is the geometric information of the points in the reconstructed point set and the reconstructed values ​​of the attributes to be processed. By constructing a graph structure for each point in the reconstructed point set and using graph convolution and graph attention mechanisms to extract graph features, the residual between the reconstructed point cloud and the original point cloud is learned; finally, the output of the preset network model is the processed values ​​of the attributes to be processed of the points in the reconstructed point set.

[0203] In some embodiments, for S403, determining the processed point cloud corresponding to the reconstructed point cloud according to the processed value of the to-be-processed attribute of the point in the reconstructed point set may include:

[0204] Determine the target set corresponding to the reconstructed point set according to the processed values ​​of the to-be-processed attributes of the points in the reconstructed point set;

[0205] According to the target set, the processed point cloud is determined.

[0206] It should be noted that in the embodiments of the present application, by extracting patches from the reconstructed point cloud, one or more patches (i.e., reconstructed point sets) can be obtained. For a patch, the to-be-processed attributes of the points in the reconstructed point set are processed using a preset network model to obtain processed values ​​of the to-be-processed attributes of the points in the reconstructed point set. The processed values ​​of the to-be-processed attributes are then used to update the reconstructed values ​​of the to-be-processed attributes of the points in the reconstructed point set, thereby obtaining a target set corresponding to the reconstructed point set, thereby further determining a processed point cloud.

[0207] Furthermore, in some embodiments, determining the processed point cloud according to the target set may include:

[0208] When there are multiple key points, the reconstructed point cloud is extracted and processed according to the multiple key points to obtain multiple reconstructed point sets;

[0209] After determining the target sets corresponding to the multiple reconstruction point sets, aggregation processing is performed based on the obtained multiple target sets to determine a processed point cloud.

[0210] It should also be noted that in the embodiments of this application, one or more key points can be obtained using the farthest point sampling method, and each key point corresponds to a reconstruction point set. Thus, when there are multiple key points, multiple reconstruction point sets can be obtained. After obtaining the target set corresponding to the reconstruction point set, the target sets corresponding to each of the multiple reconstruction point sets can be obtained using the same operating steps. Then, based on the obtained multiple target sets, patch aggregation processing can be performed to determine the processed point cloud.

[0211] In a specific embodiment, performing aggregation processing based on the obtained multiple target sets to determine a processed point cloud may include:

[0212] If at least two target sets in the plurality of target sets include processed values ​​of the attribute to be processed of the first point, performing mean calculation on the obtained at least two processed values ​​to determine the processed value of the attribute to be processed of the first point in the processed point cloud;

[0213] If none of the multiple target sets include the processed value of the attribute to be processed of the first point, then determining the reconstructed value of the attribute to be processed of the first point in the reconstructed point cloud as the processed value of the attribute to be processed of the first point in the processed point cloud;

[0214] Among them, the first point is any point in the reconstructed point cloud.

[0215] It should be noted that in this embodiment of the present application, when constructing the reconstructed point set, some points in the reconstructed point cloud may not be extracted at all, while some points may be extracted multiple times, causing these points to be fed into the preset network model multiple times. Therefore, for points that were not extracted, their reconstruction values ​​can be retained, while for points that were extracted multiple times, their average value can be calculated as the final value. In this way, after all the reconstructed point sets are aggregated, a quality-enhanced point cloud can be obtained.

[0216] It should also be noted that, in the embodiment of the present application, since the point cloud is usually represented by the RGB color space, and the YUV component is difficult to visualize the point cloud using existing applications; therefore, after determining the processed point cloud corresponding to the reconstructed point cloud, the method may further include: if the color component does not conform to the RGB color space (for example, the YUV color space, the YCbCr color space, etc.), then the color component of the point in the processed point cloud is converted to a color space so that the converted color component conforms to the RGB color space. In this way, when the color component of the point in the processed point cloud conforms to the YUV color space, it is first necessary to convert the color component of the point in the processed point cloud from conforming to the YUV color space to conforming to the RGB color space, and then use the processed point cloud to update the original reconstructed point cloud.

[0217] Furthermore, the preset network model is obtained by training a preset point cloud quality enhancement network based on a deep learning method. Therefore, in some embodiments, the method may further include:

[0218] Determine a training sample set; wherein the training sample set includes at least one point cloud sequence;

[0219] Extracting and processing at least one point cloud sequence respectively to obtain multiple sample point sets;

[0220] At a preset bit rate, the geometric information of multiple sample point sets and the original values ​​of the attributes to be processed are used to train the initial model to determine the preset network model.

[0221] It should be noted that for the training sample set, the following sequences can be selected from the existing point cloud sequences: Andrew.ply, boxer_viewdep_vox12.ply, David.ply, exercise_vox11_00000040.ply, longdress_vox10_1100.ply, longdress_vox10_1200.ply, longdress_vox10_1300.ply, model_vox11_00000035.ply, Phil.ply, queen_0050.ply, queen_0150.ply, redandblack_vox10_1450.ply, redandblack_vox10_1500.ply, Ricardo.ply, Sarah.ply, thaidancer_viewdep_vox12.ply. Then, a patch (i.e., a set of sample points) is extracted from each of the above point cloud sequences. The number of points in each patch is: Where N is the number of points in the point cloud sequence. During model training, the total number of patches can be 34848. These patches are fed into the initial model for training to obtain the preset network model.

[0222] It should also be noted that in the embodiments of this application, the initial model is related to the bit rate. Different bit rates can correspond to different initial models, and different color components can also correspond to different initial models. Thus, for the six bit rates r01 to r06 and the three color components Y / U / V at each bit rate, a total of 18 initial models are trained, resulting in 18 preset network models. In other words, different bit rates and different color components correspond to different preset network models.

[0223] In addition, during the training process, you can use the Adam optimization set with a learning rate of 0.004. The learning rate is reduced to the original 0.25 after every 60 training iterations (epochs). The number of samples in each batch (batch size) is 16, and the total number of epochs is 200. Among them, when a complete data set passes through the preset network model once and returns once, this process is called an epoch, that is, an epoch is equivalent to the process of training all training samples once; and the batch size is the number of data in the batch input to the preset network model each time; for example, if there are 16 data in a batch, then the batch size is 16.

[0224] After the preset network model is trained, the network can also be tested using a test point cloud sequence. The test point cloud sequence can be: basketball_player_vox11_00000200.ply, dancer_vox11_00000001.ply, loot_vox10_1200.ply, soldier_vox10_0690.ply. The input during the test is the entire point cloud sequence; at each bit rate, each point cloud sequence is patch extracted separately, and then the patch is input into the trained preset network model, and the quality of the Y / U / V color components is enhanced respectively; finally, the processed patches are aggregated to generate a quality-enhanced point cloud. In other words, in an embodiment of the present application, a technology for post-processing the color attributes of the reconstructed point cloud obtained by G-PCC decoding is proposed, and a preset point cloud quality enhancement network is trained using deep learning, and the network model effect is tested on a test set.

[0225] Furthermore, in the embodiment of the present application, for the preset network model shown in FIG5 , in addition to inputting a single color component and geometric information, the three color components Y / U / V and geometric information can also be used as inputs to the preset network model, rather than processing only one color component at a time. This can reduce time complexity, but slightly reduce the effect.

[0226] Furthermore, in the embodiment of the present application, the decoding method can also expand the scope of application, not only for processing single-frame point clouds, but also for post-encoding and decoding processing of multi-frame / dynamic point clouds. For example, in the G-PCC framework InterEM V5.0, there is a link for inter-frame prediction of attribute information, so the quality of the next frame is largely related to the current frame. Therefore, the embodiment of the present application can use the preset network model to post-process the reflectivity attributes of the reconstructed point cloud after decoding each frame of the point cloud in the multi-frame point cloud, and replace the original reconstructed point cloud with the quality-enhanced processed point cloud for inter-frame prediction, thereby greatly improving the attribute reconstruction quality of the next frame of the point cloud.

[0227] An embodiment of the present application provides a decoding method for determining a reconstructed point set based on a reconstructed point cloud; wherein the reconstructed point set includes at least one point; the geometric information of the points in the reconstructed point set and the reconstructed values ​​of the attributes to be processed are input into a preset network model, and the processed values ​​of the attributes to be processed of the points in the reconstructed point set are determined based on the preset network model; and the processed point cloud corresponding to the reconstructed point cloud is determined according to the processed values ​​of the attributes to be processed of the points in the reconstructed point set. In this way, the preset network model is used to perform quality enhancement processing on the attribute information of the reconstructed point cloud. Not only can different network models be trained for each bit rate and each color component based on the network framework, effectively ensuring the point cloud quality enhancement effect under various conditions, but also end-to-end operation is achieved. At the same time, by extracting and aggregating patches on the reconstructed point cloud, the point cloud can be segmented, effectively reducing resource consumption, and by taking points, processing and calculating the average multiple times, the effect and robustness of the network model can also be improved. In addition, the quality enhancement processing of the attribute information of the reconstructed point cloud according to the preset network model can also make the texture of the processed point cloud clearer and the transition more natural, effectively improving the quality and visual effect of the point cloud, thereby improving the compression performance of the point cloud.

[0228] In another embodiment of the present application, based on the decoding method described in the previous embodiment, the present application proposes a graph-based point cloud quality enhancement network (which can be represented by a PCQEN model). In this model, by constructing a graph structure for each point and extracting graph features using graph convolution and graph attention mechanisms, the residual between the reconstructed point cloud and the original point cloud is learned, thereby making the reconstructed point cloud as close to the original point cloud as possible, achieving the purpose of quality enhancement.

[0229] Referring to Figure 7, a detailed flowchart of a decoding method provided by an embodiment of the present application is shown. As shown in Figure 7, the method may include:

[0230] S701: Perform patch extraction on the reconstructed point cloud to determine at least one reconstruction point set.

[0231] S702: Input the geometric information of each point in the reconstructed point set and the reconstructed value of the color component to be processed into a preset network model, and output the processed value of the color component to be processed of each point in the reconstructed point set through the preset network model.

[0232] S703: Determine a target set corresponding to each reconstruction point set according to the processed value of the to-be-processed color component of each reconstruction point set.

[0233] S704: performing patch aggregation on the obtained at least one target set to determine a processed point cloud corresponding to the reconstructed point cloud.

[0234] It should be noted that in the embodiment of the present application, the attribute information takes color components as an example. After S701, if the color components of the points in the reconstructed point set do not conform to the YUV color space, then the color components of the points in the reconstructed point set need to be converted to a color space so that the converted color components conform to the YUV color space. In this way, considering that point clouds are usually represented in the RGB color space and YUV components are difficult to visualize point clouds using existing applications; after S704, if the color components of the points in the processed point cloud do not conform to the RGB color space, then the color components of the points in the processed point cloud need to be converted to a color space so that the converted color components are in the RGB color space.

[0235] In a specific embodiment, the flowchart of the technical solution and the network framework of the preset network model are shown in Figure 8. As shown in Figure 8, the preset network model may include: two graph attention mechanism modules (801, 802), four graph convolution modules (803, 804, 805, 806), two pooling modules (807, 808), three splicing modules (809, 810, 811) and one addition module 812. Each graph convolution module may include at least three 1×1 convolution layers; each pooling module may include at least a maximum pooling layer.

[0236] In addition, in Figure 8, the size of the reconstructed point cloud is N×6, where N represents the number of points in the reconstructed point cloud, and 6 represents three-dimensional geometric information and three-dimensional attribute information (such as the three color components Y / U / V); the input of the preset network model is P×n×4, where P represents the number of extracted reconstructed point sets (i.e., patches), n represents the number of points in each patch, and 4 represents three-dimensional geometric information and one-dimensional attribute information (i.e., a single color component); the output of the preset network model is P×n×1, where 1 represents the quality-enhanced color component; finally, the output of the preset network model is patch aggregated to obtain an N×6 processed point cloud.

[0237] Specifically, in the embodiment of the present application, for the reconstructed point cloud obtained by G-PCC decoding, patches are first extracted. Each patch can contain n points, for example, n = 2048. Here, the farthest point sampling method is used to obtain P key points, where: N is the number of points in the reconstructed point cloud, and γ is the repetition factor, which controls the average number of times each point is fed into the predefined network model. For example, γ = 3 can be used. A KNN search with K = n is then performed on each keypoint, resulting in P patches of size n. Each point contains both 3D geometric information and 3D color component information. The color component information is then converted from RGB to YUV. The color component requiring quality enhancement (such as the Y component) is extracted and combined with the 3D geometric information into a predefined network model (PCQEN model). The model outputs the enhanced Y component values ​​of the n points. These values ​​replace the Y component values ​​in the original patch (leaving the other components unchanged), resulting in a single patch with enhanced color component quality. The remaining two components can also be fed into the corresponding PCQEN models for quality enhancement. Finally, the patches are aggregated to obtain the processed point cloud. It is important to note that some points in the reconstructed point cloud may not be extracted when constructing the patch, and some points may be fed into the PCQEN model multiple times. Therefore, for the points that were not extracted, their reconstructed values ​​can be retained, while for the points that were extracted multiple times, their average value can be taken as the final value. In this way, after all patches are aggregated, the quality-enhanced point cloud can be obtained.

[0238] Furthermore, for the PCQEN model, the total number of network parameters can be set to 829121, and the model size is 7.91MB. In the design of this model, a graph attention mechanism module (GAPLayer module) is involved. This module is a graph-based attention mechanism module. After constructing the graph structure, the designed attention structure gives greater weight to the more important neighborhood features of each point, so as to better utilize graph convolution to extract features. Among them, Figure 9 shows a network framework schematic diagram of a GAPLayer module provided in an embodiment of the present application, and Figure 10 shows a network framework schematic diagram of a Single-Head GAPLayer module provided in an embodiment of the present application. In the GAPLayer module, additional geometric information input is required to assist in constructing the graph structure. Here, the GAPLayer module can be composed of 4 Single-Head GAPLayer modules; the final output is also obtained by splicing the output of each part. In the Single-Head GAPLayer module, after using KNN search to construct a graph with a neighborhood size of k (for example, k=20), graph convolution is performed on the edge features to obtain one of the outputs, namely the graph feature. On the other hand, the input features after two MLP layers are added to the graph features after another MLP, and then after passing through an activation function (for example, the LeakyReLU function), they are normalized by the Softmax function to obtain k-dimensional feature weights. After applying this feature weight to the k-neighborhood of the current point, namely the graph features, the attention feature is obtained. Finally, the four single-head graph features are combined with the attention features to obtain the output of the GAPLayer module.

[0239] Thus, based on the framework shown in Figure 8, the input to the entire network model is the geometric information p (n×3) of a patch consisting of n points and the single color component c (n×1). After passing through a GAPLayer module (setting the number of output channels F′ of the Single-Head to 16), the graph feature g1 (n×k×64) and the attention feature a1 (n×64) are obtained, i.e., g1, a1 = GAPLayer1(c, p). Next, g1 passes through a max pooling layer and undergoes a 1×1 convolution operation with channels {64, 64, 64} to obtain g2 (n×64), i.e., g2 = MaxPooling(conv1(g1)). a1 is concatenated with the input color component c and then subjected to a 1×1 convolution operation with channels {128, 64, 64} to obtain a2 (n×64), i.e., a2 = conv2(concat(a1, c)). Feeding a2 and p into the second GAPLayer module (setting the number of output channels F′ to 64 for Single-Head) yields the graph feature g3 (n×k×256) and the attention feature a3 (n×256), i.e., g3, a3 = GAPLayer2(a2, p). Next, g3 is passed through a max pooling layer to yield g4 (n×256), i.e., g4 = MaxPooling(g3). After concatenating a3 and a2, a 1×1 convolution operation with channels {256, 128, 256} is performed to yield a4 (n×256), i.e., a4 = conv3(concat(a3, a2)). Finally, g2, g4, a2, and a4 are concatenated and passed through a 1×1 convolution operation with a channel number of {256, 128, 1} to obtain the residual value r, i.e., r = conv4(concat(a4, a2, g4, g2)). r is then added to the input color component c to obtain the final output, i.e., the quality-enhanced color component c′, i.e., c′ = c + r. It should also be noted that after all 1×1 convolution layers except the last one, a BatchNormalization layer is required to accelerate convergence and suppress overfitting, followed by an activation function (e.g., a LeakyReLU function with a slope of 0.2) to add nonlinearity.

[0240] In this way, the loss function of the PCQEN model can be calculated using MSE, as shown below:

[0241]

[0242] Among them, c′ i Represents the processed value of the color component c of the point in the point cloud after processing, Represents the original value of the color component c of the point in the original point cloud.

[0243] For example, under certain configuration conditions, for the PCQEN model, the training set of the model can be selected from the existing point cloud sequences as follows: Andrew.ply, boxer_viewdep_vox12.ply, David.ply, exercise_vox11_00000040.ply, longdress_vox10_1100.ply, longdress_vox10_1200.ply, longdress_vox10_1300.ply, model_vox11_00000035.ply, Phil.ply, queen_0050.ply, queen_0150.ply, redandblack_vox10_1450.ply, redandblack_vox10_1500.ply, Ricardo.ply, Sarah.ply, thaidancer_viewdep_vox12.ply. Extract patches from each of the above point cloud sequences. The number of patches can be: Where N is the number of points in the point cloud sequence. The total number of patches during training is 34,848. These patches are fed into the network, and 18 network models are trained for the Y / U / V color components at bitrates r01 to r06. The model training uses the Adam optimization set with a learning rate of 0.004, which is reduced to 0.25 every 60 epochs. The batch size is 16, and the total number of epochs is 200.

[0244] Furthermore, for the network testing of the PCQEN model, the test point cloud sequences are: basketball_player_vox11_00000200.ply, dancer_vox11_00000001.ply, loot_vox10_1200.ply, and soldier_vox10_0690.ply. The entire point cloud sequence is used as input for testing. At each bitrate, each point cloud sequence is divided into patches. These patches are then fed into the trained network model, and quality enhancement is performed on the Y, U, and V components. Patches are then aggregated to generate the enhanced point cloud.

[0245] Thus, after the technical solution of the embodiment of the present application was implemented on the G-PCC reference software TMC13V14.0, the above test sequence was tested under the CTC-C1 test conditions (RAHT attribute transformation mode). The test results are shown in Figure 11 and Table 1. Table 1 shows the test results for each test point cloud sequence (basketball-_player_vox11-_00000200.ply, dancer_vox11-_00000001.ply, loot_vox10-_1200.ply, and soldier_vox10-_0690.ply).

[0246] Table 1

[0247]

[0248]

[0249] In addition, combined with Figure 11, the C1 condition is a lossless geometry, lossy attribute coding method (lossless geometry, lossy attribute). The End-to-End BD-AttrRate in the picture represents the BD-Rate of the end-to-end attribute value for the attribute bitstream. The BD-Rate reflects the difference in the PSNR curves under two conditions (with or without the use of the PCQEN model). When the BD-Rate decreases, it means that when the PSNR is equal, the bit rate is reduced and the performance is improved; conversely, the performance decreases. In other words, the more the BD-Rate decreases, the better the compression effect. In Table 1, ΔY, ΔU, and ΔV are the PSNR improvements of the three components Y, U, and V of the point cloud after quality enhancement relative to the reconstructed point cloud.

[0250] In other words, as Figure 11 shows, the post-processing of the PCQEN model significantly improves overall compression performance, resulting in significant BD-rate savings. Table 1 details the quality improvements for each test sequence, bitrate, and component. This demonstrates the network model's good generalization performance, achieving relatively stable quality improvements across a wide range of scenarios. This is particularly evident for reconstructed point clouds at medium to high bitrates (where distortion is minimal).

[0251] For example, Figures 12A and 12B show a schematic diagram of the comparison of point cloud images before and after quality enhancement provided by an embodiment of the present application. Here, subjective quality comparison: loot_vox10_1200.ply is a schematic diagram of the comparison before and after quality enhancement at the r03 bit rate. Among them, Figure 12A is a point cloud image before quality enhancement; Figure 12B is a point cloud image after quality enhancement (that is, quality enhancement is performed using the PCQEN model). It can be seen from Figures 12A and 12B that the difference before and after quality enhancement is very obvious. The latter has clearer texture and more natural transition, giving people a better subjective feeling.

[0252] The embodiment of the present application provides a decoding method, and the specific implementation of the aforementioned embodiment is elaborated in detail through the above embodiment. It can be seen that according to the technical solution of the aforementioned embodiment, it proposes a technology for post-processing of reconstructed point cloud quality enhancement using a graph neural network. This technology is mainly implemented through the point cloud quality enhancement network (PCQEN model). In this network model, the graph attention module of GAPLayer is used to better focus on important features. At the same time, the design of this network model is specifically for the regression task of point cloud color quality enhancement; because it is the processing of attribute information, point cloud geometry information is also required as an auxiliary input when constructing the graph structure. In addition, in this network model, features are extracted through multiple 1×1 graph convolution or MLP operations, the maximum pooling layer is used to focus on the most important neighbor information, and the splicing of existing features and previous features is used multiple times to better balance global and local, different fine-grained features, and establish connection relationships between different layers; and a BatchNorm layer and an activation function LeakyReLU are added after the convolution layer, and the skip connection (Skip Connection) is used to learn the residual. Based on this network framework, a total of 18 network models were trained for each bit rate and each color component to effectively ensure the point cloud quality enhancement effect under various conditions. At the same time, this technical solution realizes end-to-end operation. By using the proposed extraction and aggregation of patches on the point cloud, it can realize the block operation of the point cloud, effectively reducing resource consumption, and improving the effect and robustness by sampling, processing and averaging multiple points. In this way, the quality enhancement processing of the attribute information of the reconstructed point cloud based on this network model can make the texture of the processed point cloud clearer and the transition more natural, which shows that this technical solution has good performance and can effectively improve the quality and visual effect of the point cloud.

[0253] In another embodiment of the present application, referring to FIG13 , a schematic flow chart of an encoding method provided by an embodiment of the present application is shown. As shown in FIG13 , the method may include:

[0254] S1301: Encoding and reconstruction processing are performed according to the original point cloud to obtain a reconstructed point cloud.

[0255] S1302: Determine a reconstruction point set based on the reconstructed point cloud; wherein the reconstruction point set includes at least one point.

[0256] S1303: Input the geometric information of the points in the reconstructed point set and the reconstructed values ​​of the attributes to be processed into a preset network model, and determine the processed values ​​of the attributes to be processed of the points in the reconstructed point set based on the preset network model.

[0257] S1304: Determine a processed point cloud corresponding to the reconstructed point cloud according to the processed values ​​of the to-be-processed attributes of the points in the reconstructed point set.

[0258] It should be noted that the encoding method described in the embodiment of the present application specifically refers to a point cloud encoding method, which can be applied to a point cloud encoder (in the embodiment of the present application, it can be simply referred to as an "encoder").

[0259] It should also be noted that in the embodiments of the present application, the encoding method is mainly used as a technology for post-processing the attribute information of the reconstructed point cloud that has been encoded by G-PCC. Specifically, a graph-based point cloud quality enhancement network, namely a preset network model, is proposed. In this preset network model, a graph structure is constructed for each point using geometric information and the reconstructed values ​​of the attributes to be processed. Graph convolution and graph attention mechanism operations are then used for feature extraction. By learning the residual between the reconstructed point cloud and the original point cloud, the reconstructed point cloud can be made as close to the original point cloud as possible, thereby achieving the purpose of quality enhancement.

[0260] It can be understood that in the embodiment of the present application, for reconstructing the point cloud, for each point, it includes geometric information and attribute information; wherein, the geometric information represents the spatial position of the point, which can also be called three-dimensional geometric coordinate information, expressed by (x, y, z); the attribute information represents the attribute value of the point, such as the color component value.

[0261] Here, the attribute information may include color components, specifically color information of any color space. For example, the attribute information may be color information of RGB space, color information of YUV space, color information of YCbCr space, etc., which is not limited in the present embodiment.

[0262] In the embodiment of the present application, the color component may include at least one of the following: a first color component, a second color component, and a third color component. Thus, taking the attribute information as a color component as an example, if the color component conforms to the RGB color space, then the first color component, the second color component, and the third color component can be determined to be, in order: R component, G component, and B component; if the color component conforms to the YUV color space, then the first color component, the second color component, and the third color component can be determined to be, in order: Y component, U component, and V component; if the color component conforms to the YCbCr color space, then the first color component, the second color component, and the third color component can be determined to be, in order: Y component, Cb component, and Cr component.

[0263] It can also be understood that in the embodiment of the present application, for each point, in addition to the color component, the attribute information of the point may also include reflectivity, refractive index or other attributes, which is not specifically limited here.

[0264] Furthermore, in the embodiments of the present application, the attribute to be processed refers to the attribute information currently to be enhanced. Taking color components as an example, the attribute to be processed can be one-dimensional information, such as a single first color component, a second color component, or a third color component; or two-dimensional information, such as a combination of any two of the first color component, the second color component, and the third color component; or even three-dimensional information composed of the first color component, the second color component, and the third color component, which is not specifically limited here.

[0265] That is to say, for each point in the reconstructed point cloud, the attribute information may include a three-dimensional color component. However, when using the preset network model to perform quality enhancement processing on the attributes to be processed, only one color component may be processed at a time, that is, a single color component and geometric information are used as inputs to the preset network model to achieve quality enhancement processing on the single color component (the remaining color components remain unchanged); then the same method is used for the remaining two color components, and they are sent to the corresponding preset network model for quality enhancement. Alternatively, when using the preset network model to perform quality enhancement processing on the attributes to be processed, all three color components and geometric information may be used as inputs to the preset network model, rather than processing only one color component at a time. This can reduce the time complexity, but the quality enhancement effect is slightly reduced.

[0266] Furthermore, in an embodiment of the present application, for the reconstructed point cloud, the reconstructed point cloud can be obtained by performing attribute encoding, attribute reconstruction and geometric compensation on the original point cloud. Among them, for a point in the original point cloud, the predicted value and residual value of the attribute to be processed of the point can be determined first, and then the predicted value and residual value can be used to further calculate the reconstructed value of the attribute to be processed of the point, so as to construct a reconstructed point cloud. Specifically, for a point in the original point cloud, when determining the predicted value of the attribute to be processed of the point, the geometric information and attribute information of multiple target neighboring points of the point can be used, and the attribute information of the point can be predicted in combination with the geometric information of the point to obtain the corresponding predicted value, and then the residual value of the attribute to be processed of the point and the predicted value of the attribute to be processed of the point are added to obtain the reconstructed value of the attribute to be processed of the point. In this way, for a point in the original point cloud, after determining the reconstructed value of the attribute information of the point, the point can be used as the nearest neighbor of the point in the subsequent LOD to continue to perform attribute prediction on the subsequent points using the reconstructed value of the attribute information of the point, so as to obtain a reconstructed point cloud.

[0267] Furthermore, in an embodiment of the present application, for a point in the original point cloud, the residual value of the attribute to be processed of the point can be determined by performing a difference calculation based on the original value of the attribute to be processed of the point in the original point cloud and the predicted value of the attribute to be processed of the point, thereby obtaining the residual value of the attribute to be processed of the point. In some embodiments, the method may further include: encoding the residual value of the attribute to be processed of the point in the original point cloud, and writing the obtained encoded bits into the bitstream. In this way, when the subsequent bitstream is transmitted to the decoding end, the decoding end can obtain the residual value of the attribute to be processed of the point by parsing the bitstream, and then use the predicted value and the residual value to determine the reconstructed value of the attribute to be processed of the point, so as to construct a reconstructed point cloud.

[0268] That is, in the embodiment of the present application, the original point cloud can be directly obtained through the point cloud reading function of the codec program, and the reconstructed point cloud is obtained after all encoding operations are completed. In addition, the reconstructed point cloud in the embodiment of the present application can be the reconstructed point cloud output after encoding, or it can be used as a reference for encoding subsequent point clouds. In addition, the reconstructed point cloud here can be used not only within the prediction loop, that is, as an inloop filter, and can be used as a reference for encoding subsequent point clouds, but also outside the prediction loop, that is, as a post filter, and not used as a reference for encoding subsequent point clouds. This is not specifically limited here.

[0269] It is also understandable that in the embodiments of the present application, considering the number of points included in the reconstructed point cloud, for example, for some large point clouds, the number of points may exceed 10 million; before inputting the preset network model, the reconstructed point cloud can be first extracted into patches. Here, a set of reconstructed points can be regarded as a patch, and each extracted patch contains at least one point.

[0270] In some embodiments, for S1302, determining the reconstruction point set based on the reconstructed point cloud may include:

[0271] In the reconstructed point cloud, key points are identified;

[0272] The reconstructed point cloud is extracted and processed according to the key points to determine a set of reconstructed points; wherein there is a corresponding relationship between the key points and the set of reconstructed points.

[0273] In a specific embodiment, determining key points in the reconstructed point cloud may include: performing farthest point sampling processing on the reconstructed point cloud to determine key points.

[0274] It should be noted that the embodiments of the present application can obtain P key points by using the farthest point sampling method, where P is an integer greater than zero. For each key point, a patch can be extracted separately, so as to obtain a set of reconstruction points corresponding to each key point. Taking a certain key point as an example, in some embodiments, the extraction and processing of the reconstructed point cloud based on the key point to determine the set of reconstruction points may include:

[0275] Perform K-nearest neighbor search in the reconstructed point cloud based on the key points to determine the neighboring points corresponding to the key points;

[0276] Based on the neighboring points corresponding to the key points, the reconstruction point set is determined.

[0277] Furthermore, for K-nearest neighbor search, in a specific embodiment, performing K-nearest neighbor search in the reconstructed point cloud based on the key point to determine the neighboring points corresponding to the key point includes:

[0278] Based on the key points, a K-nearest neighbor search method is used to search for a first preset number of candidate points in the reconstructed point cloud;

[0279] Calculating the distance values ​​between the key point and a first preset number of candidate points respectively, and determining a second preset number of relatively smaller distance values ​​from the obtained first preset number of distance values;

[0280] Determine the neighboring points corresponding to the key point based on the candidate points corresponding to the second preset number of distance values.

[0281] In an embodiment of the present application, the second preset number is less than or equal to the first preset number.

[0282] It should also be noted that, taking a certain key point as an example, the K-nearest neighbor search method can be used to search for a first preset number of candidate points in the reconstructed point cloud, calculate the distance value between the key point and these candidate points, and then select a second preset number of candidate points closest to the key point from these candidate points; use this second preset number of candidate points as the neighboring points corresponding to the key point, and form a reconstructed point set corresponding to the key point based on these neighboring points.

[0283] Furthermore, in embodiments of the present application, the reconstructed point set may or may not include the key point itself. If the reconstructed point set includes the key point itself, then in some embodiments, determining the reconstructed point set based on the neighboring points corresponding to the key point may include determining the reconstructed point set based on the key point and its neighboring points.

[0284] It should also be noted that the reconstructed point set may include n points, where n is an integer greater than zero. Exemplarily, the value of n may be 2048, but this is not specifically limited here. In the embodiments of the present application, the determination of the number of key points is correlated with the number of points in the reconstructed point cloud and the number of points in the reconstructed point set. Therefore, in some embodiments, the method may further include: determining the number of points in the reconstructed point cloud; and determining the number of key points based on the number of points in the reconstructed point cloud and the number of points in the reconstructed point set.

[0285] In a specific embodiment, determining the number of key points based on the number of points in the reconstructed point cloud and the number of points in the reconstructed point set may include:

[0286] Determine the first factor;

[0287] Calculate the product of the number of points in the reconstructed point cloud and the first factor;

[0288] The number of key points is determined based on the product and the number of points in the reconstructed point set.

[0289] In the embodiment of the present application, the first factor can be represented by γ, which is called a repetition rate factor and is used to control the average number of times each point is sent to the preset network model. For example, the value of γ can be 3, but this is not specifically limited here.

[0290] In a more specific embodiment, assuming that the number of points in the reconstructed point cloud is N, the number of points in the reconstructed point set is n, and the number of key points is P, then That is to say, for reconstructing the point cloud, we can first use the farthest point sampling method to determine P key points, and then extract patches based on each key point. Specifically, we perform a KNN search with K=n on each key point, so that we can obtain P patches of size n, that is, P reconstruction point sets, each of which includes n points.

[0291] It's also important to note that for points in the reconstructed point cloud, there may be duplication among the P reconstructed point sets. In other words, a point may appear in multiple reconstructed point sets, or it may not appear in any of the P reconstructed point sets. This is where the first factor (γ) comes in: it controls the average repetition rate of each point in the P reconstructed point sets, thereby improving the quality of the point cloud during the final patch aggregation.

[0292] Furthermore, in the embodiments of the present application, since the point cloud is usually represented by the RGB color space, when the quality enhancement processing of the attribute to be processed is performed using the preset network model, the YUV color space is usually used. Therefore, before the geometric information of the points in the reconstructed point set and the reconstructed values ​​of the attribute to be processed are input into the preset network model, the color components need to be converted into a color space. Specifically, in some embodiments, if the color components do not conform to the YUV color space, the color components of the points in the reconstructed point set are converted into a color space so that the converted color components conform to the YUV color space, for example, from the RGB color space to the YUV color space, and then the color components that need quality enhancement (such as the Y component) are extracted and combined with the geometric information to be input into the preset network model.

[0293] In some embodiments, for S1303, inputting the geometric information of the points in the reconstructed point set and the reconstructed values ​​of the attributes to be processed into a preset network model, and determining the processed values ​​of the attributes to be processed of the points in the reconstructed point set based on the preset network model may include:

[0294] In the preset network model, the geometric information of the points in the reconstructed point set is used to assist in constructing a graph structure of the reconstructed values ​​of the attributes to be processed of the points in the reconstructed point set, so as to obtain the graph structure of the points in the reconstructed point set; and graph convolution and graph attention mechanism operations are performed on the graph structure of the points in the reconstructed point set to determine the processed values ​​of the attributes to be processed of the points in the reconstructed point set.

[0295] Here, the preset network model can be a neural network model based on deep learning. In the embodiment of the present application, the preset network model can also be referred to as a PCQEN model. The model includes at least a graph attention mechanism module and a graph convolution module to implement graph convolution and graph attention mechanism operations on the graph structure of the points in the reconstructed point set.

[0296] In a specific embodiment, the graph attention mechanism module may include a first graph attention mechanism module and a second graph attention mechanism module, and the graph convolution module may include a first graph convolution module, a second graph convolution module, a third graph convolution module, and a fourth graph convolution module. In addition, the preset network model may further include a first pooling module, a second pooling module, a first splicing module, a second splicing module, a third splicing module, and an addition module; wherein,

[0297] A first input end of the first graph attention mechanism module is used to receive geometric information, and a second input end of the first graph attention mechanism module is used to receive a reconstructed value of the attribute to be processed;

[0298] The first output of the first graph attention mechanism module is connected to the input of the first pooling module, the output of the first pooling module is connected to the input of the first graph convolution module, and the output of the first graph convolution module is connected to the first input of the first splicing module;

[0299] The second output of the first graph attention mechanism module is connected to the first input of the second splicing module, the second input of the second splicing module is used to receive the reconstructed value of the attribute to be processed, and the output of the second splicing module is connected to the input of the second graph convolution module;

[0300] The first input of the second graph attention mechanism module is used to receive geometric information, the second input of the second graph attention mechanism module is connected to the output of the second graph convolution module, the first output of the second graph attention mechanism module is connected to the input of the second pooling module, and the output of the second pooling module is connected to the second input of the first splicing module;

[0301] The second output of the second graph attention mechanism module is connected to the first input of the third splicing module, the second input of the third splicing module is connected to the output of the second graph convolution module, the output of the third splicing module is connected to the input of the third graph convolution module, and the output of the third graph convolution module is connected to the third input of the first splicing module; the output of the second graph convolution module is also connected to the fourth input of the first splicing module;

[0302] The output end of the first splicing module is connected to the input end of the fourth graph convolution module, the output end of the fourth graph convolution module is connected to the first input end of the addition module, the second input end of the addition module is used to receive the reconstructed value of the attribute to be processed, and the output end of the addition module is used to output the processed value of the attribute to be processed.

[0303] Furthermore, in embodiments of the present application, a batch normalization layer and an activation layer may be added after the convolution layer to accelerate convergence and increase nonlinear characteristics. Therefore, in some embodiments, the first graph convolution module, the second graph convolution module, the third graph convolution module, and the fourth graph convolution module each further include at least one batch normalization layer and at least one activation layer; wherein the batch normalization layer and the activation layer are connected after the convolution layer. However, it should be noted that the batch normalization layer and the activation layer may not be connected after the last convolution layer in the fourth graph convolution module.

[0304] It should be noted that the activation layer may include activation functions, such as leaky ReLU, noisy ReLU, etc. For example, a BatchNorm layer is connected after each 1×1 convolutional layer except the last layer to accelerate convergence and suppress overfitting, and a LeakyReLU activation function with a slope of 0.2 is then connected to add nonlinearity.

[0305] In a specific embodiment, for S1303, inputting the geometric information of the points in the reconstructed point set and the reconstructed values ​​of the attributes to be processed into a preset network model, and determining the processed values ​​of the attributes to be processed of the points in the reconstructed point set based on the preset network model may include:

[0306] The first graph attention mechanism module is used to extract features of the geometric information and the reconstructed values ​​of the attributes to be processed to obtain the first graph features and the first attention features;

[0307] Extracting features of the first graph through the first pooling module and the first graph convolution module to obtain second graph features;

[0308] splicing the first attention feature and the reconstructed value of the attribute to be processed by a second splicing module to obtain a first spliced ​​attention feature;

[0309] The first spliced ​​attention feature is extracted through the second graph convolution module to obtain the second attention feature;

[0310] The geometric information and the second attention feature are extracted through the second image attention mechanism module to obtain the third image feature and the third attention feature;

[0311] Perform feature extraction on the third image features through the second pooling module to obtain fourth image features;

[0312] splicing the third attention feature and the second attention feature through a third splicing module to obtain a second spliced ​​attention feature;

[0313] The second concatenated attention feature is extracted through the third graph convolution module to obtain the fourth attention feature;

[0314] The second image feature, the fourth image feature, the second attention feature, and the fourth attention feature are spliced ​​together by a first splicing module to obtain a target feature;

[0315] The fourth graph convolution module performs a convolution operation on the target feature to obtain the residual value of the attribute to be processed of the point in the reconstructed point set;

[0316] The addition module performs an addition operation on the residual value of the attribute to be processed of the point in the reconstructed point set and the reconstructed value of the attribute to be processed to obtain the processed value of the attribute to be processed of the point in the reconstructed point set.

[0317] It should be noted that in order to fully utilize the advantages of the CNN network, the point cloud network (PointNet) provides an effective method for directly learning shape features on unordered three-dimensional point clouds and achieves good performance. However, local features that contribute to better contextual learning are not considered. At the same time, the attention mechanism can effectively capture node representations on graph-based data by paying attention to neighboring nodes. Therefore, the embodiment of the present application can propose a new neural network for point clouds, called GAPNet, which learns local geometric representations by embedding a graph attention mechanism in the MLP layer. In the embodiment of the present application, a GAPLayer module is introduced here to learn the attention features of each point by highlighting different attention weights on the neighborhood; secondly, in order to mine enough features, it adopts a Multi-Head mechanism, allowing the GAPLayer module to aggregate different features from a single head; thirdly, it is proposed to use an attention pooling layer on the adjacent network to capture local signals to enhance the robustness of the network; finally, GAPNet applies multi-layer MLP to attention features and graph features, which can fully extract the input attribute information to be processed.

[0318] That is to say, in the embodiment of the present application, the structure of the first graph attention mechanism module and the second graph attention mechanism module is the same. Both the first graph attention mechanism module and the second graph attention mechanism module can include a fourth splicing module and a preset number of graph attention mechanism sub-modules; wherein, the graph attention mechanism sub-module can be a Single-Head GAPLayer module. In this way, the graph attention mechanism module composed of a preset number of Single-Head GAPLayer modules is a Multi-Head mechanism; that is, the Multi-Head GAPLayer (which can be referred to as the GAPLayer module) refers to the first graph attention mechanism module or the second graph attention mechanism module in the embodiment of the present application.

[0319] In some embodiments, the internal connection relationship between the first graph attention mechanism module and the second graph attention mechanism module is described as follows:

[0320] In the first graph attention mechanism module, input ends of a preset number of graph attention mechanism submodules are used to receive geometric information and reconstructed values ​​of attributes to be processed, output ends of the preset number of graph attention mechanism submodules are connected to the input end of the fourth splicing module, and the output end of the fourth splicing module is used to output the first graph feature and the first attention feature;

[0321] In the second graph attention mechanism module, the input ends of a preset number of graph attention mechanism sub-modules are used to receive geometric information and the second attention feature, the output ends of a preset number of graph attention mechanism sub-modules are connected to the input end of the fourth splicing module, and the output end of the fourth splicing module is used to output the third graph feature and the third attention feature.

[0322] In an embodiment of the present application, in order to obtain sufficient structural information and stabilize the network, the outputs of the four graph attention mechanism submodules are connected together through a splicing module to obtain multi-attention features and multi-graph features. Taking Figure 6 as an example, when the graph attention mechanism module shown in Figure 6 is the first graph attention mechanism module, the input module receives the geometric information and the reconstructed value of the attribute to be processed, the output multi-graph feature is the first graph feature, and the multi-attention feature is the first attention feature; when the graph attention mechanism module shown in Figure 6 is the second graph attention mechanism module, the input module receives the geometric information and the second attention feature, the output multi-graph feature is the third graph feature, and the multi-attention feature is the third attention feature.

[0323] In some embodiments, taking the first graph attention mechanism module as an example, extracting features of the geometric information and the reconstructed value of the attribute to be processed by the first graph attention mechanism module to obtain the first graph feature and the first attention feature may include:

[0324] The geometric information and the reconstructed values ​​of the attributes to be processed are input into the graph attention mechanism submodule to obtain the initial graph features and initial attention features;

[0325] Based on a preset number of graph attention mechanism submodules, a preset number of initial graph features and a preset number of initial attention features are obtained;

[0326] Splicing a preset number of initial graph features through a splicing module to obtain a first graph feature;

[0327] A preset number of initial attention features are spliced ​​together through a splicing module to obtain a first attention feature.

[0328] In a specific embodiment, for the graph attention mechanism submodule, the graph attention mechanism submodule includes at least multiple multi-layer perceptron modules; accordingly, inputting the geometric information and the reconstructed value of the attribute to be processed into the graph attention mechanism submodule to obtain the initial graph features and the initial attention features may include:

[0329] The graph structure is constructed based on the reconstruction value of the attribute to be processed assisted by geometric information, and the graph structure of the points in the reconstructed point set is obtained;

[0330] Extract features from the graph structure through at least one multi-layer perceptron module to obtain initial graph features;

[0331] Performing feature extraction on the reconstructed value of the attribute to be processed by at least one multi-layer perceptron module to obtain first intermediate feature information;

[0332] Extracting features from the initial graph using at least one multi-layer perceptron module to obtain second intermediate feature information;

[0333] Performing feature fusion on the first intermediate feature information and the second intermediate feature information using a first preset function to obtain an attention coefficient;

[0334] The attention coefficient is normalized using the second preset function to obtain the feature weight;

[0335] According to the feature weights and the initial graph features, the initial attention features are obtained.

[0336] It should be noted that in the embodiment of the present application, the first preset function is different from the second preset function. Among them, the first preset function is a nonlinear activation function, such as the LeakyReLU function; the second preset function is a normalized exponential function, such as the softmax function. Here, the softmax function can "compress" a K-dimensional vector z containing arbitrary real numbers into another K-dimensional real vector σ(z), so that the range of each element is between (0,1) and the sum of all elements is 1; in short, the softmax function mainly performs normalization processing.

[0337] It should also be noted that the initial attention features are obtained based on the feature weights and the initial graph features. Specifically, the initial attention features can be generated by performing a linear combination operation on the feature weights and the initial graph features. Here, the initial graph features are n×k×F', the feature weights are n×1×k, and the initial attention features obtained after the linear combination operation are n×F'.

[0338] Specifically, the embodiment of the present application is a graph-based attention mechanism module. After constructing the graph structure, the attention structure is used to give greater weight to the more important neighborhood features of each point, so as to better utilize graph convolution to extract features. In the first graph attention mechanism module, additional geometric information input is required to assist in constructing the graph structure. The first graph attention mechanism module can be composed of 4 graph attention mechanism sub-modules, so the final output is also obtained by splicing the output of each graph attention mechanism sub-module. In the graph attention mechanism sub-module, after constructing a graph structure with a neighborhood size of k using the KNN search method (for example, k=20 can be selected), the edge features in the graph structure are graph convolved to obtain one of the outputs, namely the initial graph feature (Graph Feature). On the other hand, the input features after two layers of MLP are fused with the graph features after another MLP, and after the activation function LeakyReLU, they are normalized by the softmax function to obtain the k-dimensional feature weight. After applying this weight to the k neighborhood of the current point, namely the graph feature, another output, namely the initial attention feature (Attention Feature), can be obtained.

[0339] In this way, based on the preset network model described in the embodiment of the present application, the input of the preset network model is the geometric information of the points in the reconstructed point set and the reconstructed values ​​of the attributes to be processed. By constructing a graph structure for each point in the reconstructed point set and using graph convolution and graph attention mechanisms to extract graph features, the residual between the reconstructed point cloud and the original point cloud is learned; finally, the output of the preset network model is the processed values ​​of the attributes to be processed of the points in the reconstructed point set.

[0340] In some embodiments, for S1304, determining the processed point cloud corresponding to the reconstructed point cloud based on the processing values ​​of the to-be-processed attributes of the points in the reconstructed point set may include: determining the target set corresponding to the reconstructed point set based on the processing values ​​of the to-be-processed attributes of the points in the reconstructed point set; and determining the processed point cloud based on the target set.

[0341] It should be noted that in the embodiments of the present application, by extracting patches from the reconstructed point cloud, one or more patches (i.e., reconstructed point sets) can be obtained. For a patch, the to-be-processed attributes of the points in the reconstructed point set are processed using a preset network model to obtain processed values ​​of the to-be-processed attributes of the points in the reconstructed point set. The processed values ​​of the to-be-processed attributes are then used to update the reconstructed values ​​of the to-be-processed attributes of the points in the reconstructed point set, thereby obtaining a target set corresponding to the reconstructed point set, thereby further determining a processed point cloud.

[0342] Furthermore, in some embodiments, determining the processed point cloud based on the target set may include: when there are multiple key points, extracting and processing the reconstructed point cloud according to the multiple key points respectively to obtain multiple reconstructed point sets; after determining the target sets corresponding to the multiple reconstructed point sets, performing aggregation processing based on the obtained multiple target sets to determine the processed point cloud.

[0343] In a specific embodiment, performing aggregation processing based on the obtained multiple target sets to determine a processed point cloud may include:

[0344] If at least two target sets in the plurality of target sets include processed values ​​of the attribute to be processed of the first point, performing mean calculation on the obtained at least two processed values ​​to determine the processed value of the attribute to be processed of the first point in the processed point cloud;

[0345] If none of the multiple target sets include the processed value of the attribute to be processed of the first point, then determining the reconstructed value of the attribute to be processed of the first point in the reconstructed point cloud as the processed value of the attribute to be processed of the first point in the processed point cloud;

[0346] Among them, the first point is any point in the reconstructed point cloud.

[0347] It should be noted that in this embodiment of the present application, when constructing the reconstructed point set, some points in the reconstructed point cloud may not be extracted at all, while some points may be extracted multiple times, causing these points to be fed into the preset network model multiple times. Therefore, for points that were not extracted, their reconstruction values ​​can be retained, while for points that were extracted multiple times, their average value can be calculated as the final value. In this way, after all the reconstructed point sets are aggregated, a quality-enhanced point cloud can be obtained.

[0348] It should also be noted that, in the embodiment of the present application, since the point cloud is usually represented by the RGB color space, and the YUV component is difficult to visualize the point cloud using existing applications; therefore, after determining the processed point cloud corresponding to the reconstructed point cloud, the method may further include: if the color component does not conform to the RGB color space (for example, the YUV color space, the YCbCr color space, etc.), then the color component of the point in the processed point cloud is converted to a color space so that the converted color component conforms to the RGB color space. In this way, when the color component of the point in the processed point cloud conforms to the YUV color space, it is first necessary to convert the color component of the point in the processed point cloud from conforming to the YUV color space to conforming to the RGB color space, and then use the processed point cloud to update the original reconstructed point cloud.

[0349] Furthermore, the preset network model is obtained by training a preset point cloud quality enhancement network based on a deep learning method. Therefore, in some embodiments, the method may further include:

[0350] Determine a training sample set; wherein the training sample set includes at least one point cloud sequence;

[0351] Extracting and processing at least one point cloud sequence respectively to obtain multiple sample point sets;

[0352] At a preset bit rate, the geometric information of multiple sample point sets and the attribute information of the attributes to be processed are used to train the initial model to determine the preset network model.

[0353] It should be noted that for the training sample set, the following sequences can be selected from the existing point cloud sequences: Andrew.ply, boxer_viewdep_vox12.ply, David.ply, exercise_vox11_00000040.ply, longdress_vox10_1100.ply, longdress_vox10_1200.ply, longdress_vox10_1300.ply, model_vox11_00000035.ply, Phil.ply, queen_0050.ply, queen_0150.ply, redandblack_vox10_1450.ply, redandblack_vox10_1500.ply, Ricardo.ply, Sarah.ply, thaidancer_viewdep_vox12.ply. Then, a patch (i.e., a set of sample points) is extracted from each of the above point cloud sequences. The number of points in each patch is: Where N is the number of points in the point cloud sequence. When training the model, the total number of patches can be 34848. These patches are fed into the initial model for training.

[0354] It should also be noted that in the embodiments of this application, the initial model is related to the bit rate. Different bit rates can correspond to different initial models, and different color components can also correspond to different initial models. Thus, for the six bit rates r01 to r06 and the three color components Y / U / V at each bit rate, a total of 18 initial models are trained, resulting in 18 preset network models. In other words, different bit rates and different color components correspond to different preset network models.

[0355] After the preset network model is trained, the network can also be tested using a test point cloud sequence. The test point cloud sequence can be: basketball_player_vox11_00000200.ply, dancer_vox11_00000001.ply, loot_vox10_1200.ply, soldier_vox10_0690.ply. The input during the test is the entire point cloud sequence; at each bit rate, each point cloud sequence is patch extracted separately, and then the patch is input into the trained preset network model, and the quality of the Y / U / V color components is enhanced respectively; finally, the processed patches are aggregated to generate a quality-enhanced point cloud. In other words, the embodiment of the present application proposes a technology for post-processing the color attributes of the reconstructed point cloud obtained by G-PCC decoding, uses deep learning to train a preset point cloud quality enhancement network model, and tests the network model effect on a test set.

[0356] Furthermore, in the embodiments of the present application, in addition to inputting a single color component and geometric information, the Y / U / V color components and geometric information can also be used as input for the preset network model, rather than processing only one color component at a time. This can reduce time complexity, but slightly reduce the effect.

[0357] Furthermore, in the embodiment of the present application, the encoding method can also expand the scope of application, not only for processing single-frame point clouds, but also for post-encoding and decoding processing of multi-frame / dynamic point clouds. For example, in the G-PCC framework InterEM V5.0, there is a link for inter-frame prediction of attribute information, so the quality of the next frame is largely related to the current frame. Therefore, the embodiment of the present application can use the preset network model to post-process the reflectivity attributes of the reconstructed point cloud after encoding each frame of the point cloud in the multi-frame point cloud, and replace the original reconstructed point cloud with the quality-enhanced processed point cloud for inter-frame prediction, thereby greatly improving the attribute reconstruction quality of the next frame of the point cloud.

[0358] An embodiment of the present application provides an encoding method, which performs encoding and reconstruction processing on the original point cloud to obtain a reconstructed point cloud; determines a reconstructed point set based on the reconstructed point cloud; wherein the reconstructed point set includes at least one point; inputs the geometric information of the points in the reconstructed point set and the reconstructed values ​​of the attributes to be processed into a preset network model, and determines the processed values ​​of the attributes to be processed of the points in the reconstructed point set based on the preset network model; determines a processed point cloud corresponding to the reconstructed point cloud based on the processed values ​​of the attributes to be processed of the points in the reconstructed point set. In this way, the preset network model is used to perform quality enhancement processing on the attribute information of the reconstructed point cloud. Not only can different network models be trained for each bit rate and each color component based on the network framework, effectively ensuring the point cloud quality enhancement effect under various conditions, but also end-to-end operation is achieved. At the same time, by extracting and aggregating patches on the point cloud, the point cloud can be segmented, effectively reducing resource consumption, and by taking points, processing and calculating the average multiple times, the effect and robustness of the network model can also be improved. In addition, the quality enhancement processing of the attribute information of the reconstructed point cloud according to the preset network model can also make the texture of the processed point cloud clearer and the transition more natural, effectively improving the quality and visual effect of the point cloud, thereby improving the compression performance of the point cloud.

[0359] In another embodiment of the present application, based on the same inventive concept as the above embodiment, refer to FIG14 , which shows a schematic diagram of the composition structure of an encoder 300 provided in an embodiment of the present application. As shown in FIG14 , the encoder 300 may include: an encoding unit 3001, a first extraction unit 3002, a first model unit 3003, and a first aggregation unit 3004; wherein,

[0360] The encoding unit 3001 is configured to perform encoding and reconstruction processing based on the original point cloud to obtain a reconstructed point cloud;

[0361] The first extraction unit 3002 is configured to determine a reconstruction point set based on the reconstructed point cloud; wherein the reconstruction point set includes at least one point;

[0362] The first model unit 3003 is configured to input the geometric information of the points in the reconstructed point set and the reconstructed values ​​of the attributes to be processed into a preset network model, and determine the processed values ​​of the attributes to be processed of the points in the reconstructed point set based on the preset network model;

[0363] The first aggregation unit 3004 is configured to determine a processed point cloud corresponding to the reconstructed point cloud according to the processed values ​​of the to-be-processed attributes of the points in the reconstructed point set.

[0364] In some embodiments, referring to FIG14 , the encoder 300 may further include a first determining unit 3005 configured to determine key points in the reconstructed point cloud;

[0365] The first extraction unit 3002 is configured to extract and process the reconstructed point cloud according to the key points to determine a set of reconstructed points; wherein there is a corresponding relationship between the key points and the set of reconstructed points.

[0366] In some embodiments, the first determining unit 3005 is further configured to perform farthest point sampling processing on the reconstructed point cloud to determine key points.

[0367] In some embodiments, referring to FIG14 , the encoder 300 may further include a first search unit 3006 configured to perform a K-nearest neighbor search in the reconstructed point cloud according to the key point to determine a neighbor point corresponding to the key point;

[0368] The first determining unit 3005 is further configured to determine a set of reconstruction points based on neighboring points corresponding to the key points.

[0369] In some embodiments, the first search unit 3006 is configured to search for a first preset number of candidate points in the reconstructed point cloud based on the key point using a K-nearest neighbor search method; and calculate the distance values ​​between the key point and the first preset number of candidate points respectively, and determine a relatively smaller second preset number of distance values ​​from the obtained first preset number of distance values; and determine the neighboring points corresponding to the key point based on the candidate points corresponding to the second preset number of distance values; wherein the second preset number is less than or equal to the first preset number.

[0370] In some embodiments, the first determining unit 3005 is further configured to determine a set of reconstruction points based on the key points and the neighboring points corresponding to the key points.

[0371] In some embodiments, the first determining unit 3005 is further configured to determine the number of points in the reconstructed point cloud; and determine the number of key points according to the number of points in the reconstructed point cloud and the number of points in the reconstructed point set.

[0372] In some embodiments, the first determination unit 3005 is further configured to determine a first factor; and calculate the product of the number of points in the reconstructed point cloud and the first factor; and determine the number of key points based on the product and the number of points in the reconstructed point set.

[0373] In some embodiments, the first determining unit 3005 is further configured to determine a target set corresponding to the reconstructed point set according to the processed values ​​of the to-be-processed attributes of the points in the reconstructed point set; and determine a processed point cloud according to the target set.

[0374] In some embodiments, the first extraction unit 3002 is configured to, when there are multiple key points, extract and process the reconstructed point cloud according to the multiple key points to obtain multiple reconstructed point sets;

[0375] The first aggregation unit 3004 is configured to, after determining the target sets corresponding to the multiple reconstruction point sets, perform aggregation processing on the obtained multiple target sets to determine a processed point cloud.

[0376] In some embodiments, the first aggregation unit 3004 is further configured to perform mean calculation on the at least two processed values ​​obtained to determine the processed value of the attribute to be processed of the first point in the processed point cloud if at least two target sets among the multiple target sets include the processed value of the attribute to be processed of the first point; if the multiple target sets do not include the processed value of the attribute to be processed of the first point, then determine the reconstructed value of the attribute to be processed of the first point in the reconstructed point cloud as the processed value of the attribute to be processed of the first point in the processed point cloud; wherein the first point is any point in the reconstructed point cloud.

[0377] In some embodiments, the first model unit 3003 is configured to construct a graph structure in a preset network model to assist in reconstructing the reconstruction values ​​of the attributes to be processed of the points in the reconstructed point set based on the geometric information of the points in the reconstructed point set, so as to obtain the graph structure of the points in the reconstructed point set; and perform graph convolution and graph attention mechanism operations on the graph structure of the points in the reconstructed point set to determine the processing values ​​of the attributes to be processed of the points in the reconstructed point set.

[0378] In some embodiments, the preset network model is a neural network model based on deep learning; wherein, the preset network model includes at least a graph attention mechanism module and a graph convolution module.

[0379] In some embodiments, the graph attention mechanism module includes a first graph attention mechanism module and a second graph attention mechanism module, and the graph convolution module includes a first graph convolution module, a second graph convolution module, a third graph convolution module and a fourth graph convolution module; the preset network model also includes a first pooling module, a second pooling module, a first splicing module, a second splicing module, a third splicing module and an addition module; wherein, the first input end of the first graph attention mechanism module is used to receive geometric information, and the second input end of the first graph attention mechanism module is used to receive the reconstructed value of the attribute to be processed; the first output end of the first graph attention mechanism module is connected to the input end of the first pooling module, the output end of the first pooling module is connected to the input end of the first graph convolution module, and the output end of the first graph convolution module is connected to the first input end of the first splicing module; the second output end of the first graph attention mechanism module is connected to the first input end of the second splicing module, the second input end of the second splicing module is used to receive the reconstructed value of the attribute to be processed, and the output end of the second splicing module is connected to the input end of the second graph convolution module. The first input end of the second graph attention mechanism module is used to receive geometric information, the second input end of the second graph attention mechanism module is connected to the output end of the second graph convolution module, the first output end of the second graph attention mechanism module is connected to the input end of the second pooling module, and the output end of the second pooling module is connected to the second input end of the first splicing module; the second output end of the second graph attention mechanism module is connected to the first input end of the third splicing module, the second input end of the third splicing module is connected to the output end of the second graph convolution module, the output end of the third splicing module is connected to the input end of the third graph convolution module, and the output end of the third graph convolution module is connected to the third input end of the first splicing module; the output end of the second graph convolution module is also connected to the fourth input end of the first splicing module; the output end of the first splicing module is connected to the input end of the fourth graph convolution module, the output end of the fourth graph convolution module is connected to the first input end of the addition module, the second input end of the addition module is used to receive the reconstructed value of the attribute to be processed, and the output end of the addition module is used to output the processed value of the attribute to be processed.

[0380] In some embodiments, the first model unit 3003 is configured to perform feature extraction on the geometric information and the reconstructed value of the attribute to be processed through the first graph attention mechanism module to obtain a first graph feature and a first attention feature; and perform feature extraction on the first graph feature through the first pooling module and the first graph convolution module to obtain a second graph feature; and perform feature extraction on the first attention feature and the reconstructed value of the attribute to be processed through the second splicing module to obtain a first spliced ​​attention feature; and perform feature extraction on the first spliced ​​attention feature through the second graph convolution module to obtain a second attention feature; and perform feature extraction on the geometric information and the second attention feature through the second graph attention mechanism module to obtain a third graph feature and a third attention feature; and perform feature extraction on the first attention feature and the reconstructed value of the attribute to be processed through the second pooling module to obtain a first spliced ​​attention feature. Feature extraction is performed on the three image features to obtain a fourth image feature; and the third attention feature and the second attention feature are spliced ​​through the third splicing module to obtain a second spliced ​​attention feature; and the second spliced ​​attention feature is feature extracted through the third graph convolution module to obtain a fourth attention feature; and the second image feature, the fourth image feature, the second attention feature and the fourth attention feature are spliced ​​through the first splicing module to obtain a target feature; and a convolution operation is performed on the target feature through the fourth graph convolution module to obtain a residual value of the to-be-processed attribute of the point in the reconstructed point set; and an addition operation is performed on the residual value of the to-be-processed attribute of the point in the reconstructed point set and the reconstructed value of the to-be-processed attribute through the addition module to obtain a processed value of the to-be-processed attribute of the point in the reconstructed point set.

[0381] In some embodiments, the first graph convolution module, the second graph convolution module, the third graph convolution module, and the fourth graph convolution module each include at least one convolution layer.

[0382] In some embodiments, the first graph convolution module, the second graph convolution module, the third graph convolution module, and the fourth graph convolution module each further include at least one batch normalization layer and at least one activation layer; wherein the batch normalization layer and the activation layer are connected after the convolution layer.

[0383] In some embodiments, the last convolution layer in the fourth graph convolution module is not followed by a batch normalization layer and an activation layer.

[0384] In some embodiments, the first graph attention mechanism module and the second graph attention mechanism module both include a fourth splicing module and a preset number of graph attention mechanism sub-modules; wherein, in the first graph attention mechanism module, the input ends of the preset number of graph attention mechanism sub-modules are all used to receive geometric information and reconstructed values ​​of attributes to be processed, the output ends of the preset number of graph attention mechanism sub-modules are connected to the input end of the fourth splicing module, and the output end of the fourth splicing module is used to output the first graph feature and the first attention feature; in the second graph attention mechanism module, the input ends of the preset number of graph attention mechanism sub-modules are all used to receive geometric information and the second attention feature, the output ends of the preset number of graph attention mechanism sub-modules are connected to the input end of the fourth splicing module, and the output end of the fourth splicing module is used to output the third graph feature and the third attention feature.

[0385] In some embodiments, the graph attention mechanism submodule is a single-headed GAPLayer module.

[0386] In some embodiments, the first model unit 3003 is further configured to input the geometric information and the reconstructed value of the attribute to be processed into the graph attention mechanism sub-module to obtain the initial graph feature and the initial attention feature; based on a preset number of graph attention mechanism sub-modules, obtain a preset number of initial graph features and a preset number of initial attention features; and splice the preset number of initial graph features through the fourth splicing module to obtain the first graph feature; and splice the preset number of initial attention features through the fourth splicing module to obtain the first attention feature.

[0387] In some embodiments, the graph attention mechanism submodule includes at least multiple multi-layer perceptron modules; accordingly, the first model unit 3003 is further configured to construct a graph structure based on the reconstruction value of the attribute to be processed with the assistance of geometric information, and obtain the graph structure of the points in the reconstructed point set; and extract features from the graph structure through at least one multi-layer perceptron module to obtain initial graph features; and extract features from the reconstructed value of the attribute to be processed through at least one multi-layer perceptron module to obtain first intermediate feature information; and extract features from the initial graph features through at least one multi-layer perceptron module to obtain second intermediate feature information; and use a first preset function to perform feature fusion on the first intermediate feature information and the second intermediate feature information to obtain an attention coefficient; use a second preset function to normalize the attention coefficient to obtain a feature weight; and obtain an initial attention feature based on the feature weight and the initial graph feature.

[0388] In some embodiments, referring to FIG14 , the encoder 300 may further include a first training unit 3007 configured to determine a training sample set; wherein the training sample set includes at least one point cloud sequence; and extracting and processing the at least one point cloud sequence separately to obtain a plurality of sample point sets; and performing model training on the initial model using the geometric information of the plurality of sample point sets and the original values ​​of the attributes to be processed at a preset bit rate to determine a preset network model.

[0389] In some embodiments, the attributes to be processed include color components, and the color components include at least one of the following: a first color component, a second color component, and a third color component; accordingly, the first determination unit 3005 is also configured to, after determining the processed point cloud corresponding to the reconstructed point cloud, if the color component does not conform to the RGB color space, perform color space conversion on the color component of the point in the processed point cloud so that the converted color component conforms to the RGB color space.

[0390] It is understandable that in the embodiments of the present application, a "unit" can be a portion of a circuit, a portion of a processor, a portion of a program or software, etc., and of course it can also be a module, or it can be non-modular. Moreover, the various components in this embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into a single unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional modules.

[0391] If the integrated unit is implemented as a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, or the portion that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method described in this embodiment. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0392] Therefore, an embodiment of the present application provides a computer storage medium, which is applied to the encoder 300. The computer storage medium stores a computer program, and when the computer program is executed by the first processor, it implements the method described in any one of the aforementioned embodiments.

[0393] Based on the composition of the above-mentioned encoder 300 and the computer storage medium, refer to Figure 15, which shows a specific hardware structure diagram of the encoder 300 provided in an embodiment of the present application. As shown in Figure 15, the encoder 300 may include: a first communication interface 3101, a first memory 3102 and a first processor 3103; each component is coupled together through a first bus system 3104. It can be understood that the first bus system 3104 is used to realize the connection and communication between these components. In addition to the data bus, the first bus system 3104 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, various buses are labeled as the first bus system 3104 in Figure 15. Among them,

[0394] The first communication interface 3101 is used to receive and send signals when sending and receiving information with other external network elements;

[0395] A first memory 3102 is used to store computer programs that can be run on the first processor 3103;

[0396] The first processor 3103 is configured to, when running the computer program, execute:

[0397] Encoding and reconstruction processing are performed according to the original point cloud to obtain a reconstructed point cloud;

[0398] Determine a reconstruction point set based on the reconstructed point cloud; wherein the reconstruction point set includes at least one point;

[0399] Inputting the geometric information of the points in the reconstructed point set and the reconstructed values ​​of the attributes to be processed into a preset network model, and determining the processed values ​​of the attributes to be processed of the points in the reconstructed point set based on the preset network model;

[0400] According to the processing value of the to-be-processed attribute of the point in the reconstructed point set, a processed point cloud corresponding to the reconstructed point cloud is determined.

[0401] It is understood that the first memory 3102 in the embodiment of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DRRAM). The first memory 3102 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0402] The first processor 3103 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits or software instructions in the first processor 3103. The above-mentioned first processor 3103 can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The various methods, steps, and logic block diagrams disclosed in the embodiments of this application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly implemented as a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, registers, etc. The storage medium is located in the first memory 3102 , and the first processor 3103 reads the information in the first memory 3102 and completes the steps of the above method in combination with its hardware.

[0403] It is to be understood that these embodiments described in the present application can be implemented with hardware, software, firmware, middleware, microcode or its combination.For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (Application Specific Integrated Circuits, ASIC), digital signal processor (Digital Signal Processing, DSP), digital signal processing equipment (DSP Device, DSPD), programmable logic device (Programmable Logic Device, PLD), field programmable gate array (Field-Programmable Gate Array, FPGA), general-purpose processor, controller, microcontroller, microprocessor, other electronic units for performing functions described in the present application or its combination.For software implementation, the technology described in the present application can be realized by the module (such as process, function etc.) that performs functions described in the present application. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0404] Optionally, as another embodiment, the first processor 3103 is further configured to execute any one of the methods described in the foregoing embodiments when running the computer program.

[0405] This embodiment provides an encoder, in which, after obtaining the reconstructed point cloud, the quality enhancement processing of the attribute information of the reconstructed point cloud is performed based on a preset network model, which not only realizes end-to-end operation, but also utilizes the proposed patch extraction and aggregation of the point cloud to realize the blocking operation of the reconstructed point cloud, effectively reducing resource consumption and improving the robustness of the model; in this way, the quality enhancement processing of the attribute information of the reconstructed point cloud according to the network model can make the texture of the processed point cloud clearer and the transition more natural, which shows that the present technical solution has good performance and can effectively improve the quality and visual effect of the point cloud.

[0406] Based on the same inventive concept as the above embodiment, refer to FIG16 , which shows a schematic diagram of the structure of a decoder 320 provided in an embodiment of the present application. As shown in FIG16 , the decoder 320 may include: a second extraction unit 3201, a second model unit 3202, and a second aggregation unit 3203; wherein,

[0407] The second extraction unit 3201 is configured to determine a reconstruction point set based on the reconstructed point cloud; wherein the reconstruction point set includes at least one point;

[0408] The second model unit 3202 is configured to input the geometric information of the points in the reconstructed point set and the reconstructed values ​​of the attributes to be processed into a preset network model, and determine the processed values ​​of the attributes to be processed of the points in the reconstructed point set based on the preset network model;

[0409] The second aggregation unit 3203 is configured to determine a processed point cloud corresponding to the reconstructed point cloud according to the processed values ​​of the to-be-processed attributes of the points in the reconstructed point set.

[0410] In some embodiments, referring to FIG16 , the decoder 320 may further include a second determining unit 3204 configured to determine key points in the reconstructed point cloud;

[0411] The second extraction unit 3201 is configured to extract and process the reconstructed point cloud according to the key points to determine a set of reconstructed points; wherein there is a corresponding relationship between the key points and the set of reconstructed points.

[0412] In some embodiments, the second determining unit 3204 is further configured to perform farthest point sampling processing on the reconstructed point cloud to determine key points.

[0413] In some embodiments, referring to FIG16 , the decoder 320 may further include a second search unit 3205 configured to perform a K-nearest neighbor search in the reconstructed point cloud according to the key point to determine a neighbor point corresponding to the key point;

[0414] The first determining unit 3005 is further configured to determine a set of reconstruction points based on neighboring points corresponding to the key points.

[0415] In some embodiments, the second search unit 3205 is configured to search for a first preset number of candidate points in the reconstructed point cloud based on the key point using a K-nearest neighbor search method; and calculate the distance values ​​between the key point and the first preset number of candidate points respectively, and determine a relatively smaller second preset number of distance values ​​from the obtained first preset number of distance values; and determine the neighboring points corresponding to the key point based on the candidate points corresponding to the second preset number of distance values; wherein the second preset number is less than or equal to the first preset number.

[0416] In some embodiments, the second determining unit 3204 is further configured to determine a set of reconstruction points based on the key points and the neighboring points corresponding to the key points.

[0417] In some embodiments, the second determining unit 3204 is further configured to determine the number of points in the reconstructed point cloud; and determine the number of key points according to the number of points in the reconstructed point cloud and the number of points in the reconstructed point set.

[0418] In some embodiments, the second determination unit 3204 is further configured to determine a first factor; and calculate the product of the number of points in the reconstructed point cloud and the first factor; and determine the number of key points based on the product and the number of points in the reconstructed point set.

[0419] In some embodiments, the second determining unit 3204 is further configured to determine a target set corresponding to the reconstructed point set according to the processed values ​​of the to-be-processed attributes of the points in the reconstructed point set; and determine a processed point cloud according to the target set.

[0420] In some embodiments, the second extraction unit 3201 is configured to, when there are multiple key points, extract and process the reconstructed point cloud according to the multiple key points to obtain multiple reconstructed point sets;

[0421] The second aggregation unit 3203 is configured to, after determining the target sets corresponding to the multiple reconstruction point sets, perform aggregation processing on the obtained multiple target sets to determine a processed point cloud.

[0422] In some embodiments, the second aggregation unit 3203 is further configured to perform mean calculation on the at least two processed values ​​obtained to determine the processed value of the to-be-processed attribute of the first point in the processed point cloud if at least two target sets among the multiple target sets include the processed value of the to-be-processed attribute of the first point; if the multiple target sets do not include the processed value of the to-be-processed attribute of the first point, determine the reconstructed value of the to-be-processed attribute of the first point in the reconstructed point cloud as the processed value of the to-be-processed attribute of the first point in the processed point cloud; wherein the first point is any point in the reconstructed point cloud.

[0423] In some embodiments, the second model unit 3202 is configured to construct a graph structure in a preset network model to assist in reconstructing the reconstruction values ​​of the attributes to be processed of the points in the reconstructed point set based on the geometric information of the points in the reconstructed point set, so as to obtain the graph structure of the points in the reconstructed point set; and perform graph convolution and graph attention mechanism operations on the graph structure of the points in the reconstructed point set to determine the processing values ​​of the attributes to be processed of the points in the reconstructed point set.

[0424] In some embodiments, the preset network model is a neural network model based on deep learning; wherein, the preset network model includes at least a graph attention mechanism module and a graph convolution module.

[0425] In some embodiments, the graph attention mechanism module includes a first graph attention mechanism module and a second graph attention mechanism module, and the graph convolution module includes a first graph convolution module, a second graph convolution module, a third graph convolution module and a fourth graph convolution module; the preset network model also includes a first pooling module, a second pooling module, a first splicing module, a second splicing module, a third splicing module and an addition module; wherein, the first input end of the first graph attention mechanism module is used to receive geometric information, and the second input end of the first graph attention mechanism module is used to receive the reconstructed value of the attribute to be processed; the first output end of the first graph attention mechanism module is connected to the input end of the first pooling module, the output end of the first pooling module is connected to the input end of the first graph convolution module, and the output end of the first graph convolution module is connected to the first input end of the first splicing module; the second output end of the first graph attention mechanism module is connected to the first input end of the second splicing module, the second input end of the second splicing module is used to receive the reconstructed value of the attribute to be processed, and the output end of the second splicing module is connected to the input end of the second graph convolution module. The first input end of the second graph attention mechanism module is used to receive geometric information, the second input end of the second graph attention mechanism module is connected to the output end of the second graph convolution module, the first output end of the second graph attention mechanism module is connected to the input end of the second pooling module, and the output end of the second pooling module is connected to the second input end of the first splicing module; the second output end of the second graph attention mechanism module is connected to the first input end of the third splicing module, the second input end of the third splicing module is connected to the output end of the second graph convolution module, the output end of the third splicing module is connected to the input end of the third graph convolution module, and the output end of the third graph convolution module is connected to the third input end of the first splicing module; the output end of the second graph convolution module is also connected to the fourth input end of the first splicing module; the output end of the first splicing module is connected to the input end of the fourth graph convolution module, the output end of the fourth graph convolution module is connected to the first input end of the addition module, the second input end of the addition module is used to receive the reconstructed value of the attribute to be processed, and the output end of the addition module is used to output the processed value of the attribute to be processed.

[0426] In some embodiments, the second model unit 3202 is configured to perform feature extraction on the geometric information and the reconstructed value of the attribute to be processed through the first graph attention mechanism module to obtain a first graph feature and a first attention feature; and perform feature extraction on the first graph feature through the first pooling module and the first graph convolution module to obtain a second graph feature; and perform feature extraction on the first attention feature and the reconstructed value of the attribute to be processed through the second splicing module to obtain a first spliced ​​attention feature; and perform feature extraction on the first spliced ​​attention feature through the second graph convolution module to obtain a second attention feature; and perform feature extraction on the geometric information and the second attention feature through the second graph attention mechanism module to obtain a third graph feature and a third attention feature; and perform feature extraction on the first attention feature and the reconstructed value of the attribute to be processed through the second pooling module to obtain a first spliced ​​attention feature. Feature extraction is performed on the three image features to obtain a fourth image feature; and the third attention feature and the second attention feature are spliced ​​through the third splicing module to obtain a second spliced ​​attention feature; and the second spliced ​​attention feature is feature extracted through the third graph convolution module to obtain a fourth attention feature; and the second image feature, the fourth image feature, the second attention feature and the fourth attention feature are spliced ​​through the first splicing module to obtain a target feature; and a convolution operation is performed on the target feature through the fourth graph convolution module to obtain a residual value of the to-be-processed attribute of the point in the reconstructed point set; and an addition operation is performed on the residual value of the to-be-processed attribute of the point in the reconstructed point set and the reconstructed value of the to-be-processed attribute through the addition module to obtain a processed value of the to-be-processed attribute of the point in the reconstructed point set.

[0427] In some embodiments, the first graph convolution module, the second graph convolution module, the third graph convolution module, and the fourth graph convolution module each include at least one convolution layer.

[0428] In some embodiments, the first graph convolution module, the second graph convolution module, the third graph convolution module, and the fourth graph convolution module each further include at least one batch normalization layer and at least one activation layer; wherein the batch normalization layer and the activation layer are connected after the convolution layer.

[0429] In some embodiments, the last convolution layer in the fourth graph convolution module is not followed by a batch normalization layer and an activation layer.

[0430] In some embodiments, the first graph attention mechanism module and the second graph attention mechanism module both include a fourth splicing module and a preset number of graph attention mechanism sub-modules; wherein, in the first graph attention mechanism module, the input ends of the preset number of graph attention mechanism sub-modules are all used to receive geometric information and reconstructed values ​​of attributes to be processed, the output ends of the preset number of graph attention mechanism sub-modules are connected to the input end of the fourth splicing module, and the output end of the fourth splicing module is used to output the first graph feature and the first attention feature; in the second graph attention mechanism module, the input ends of the preset number of graph attention mechanism sub-modules are all used to receive geometric information and the second attention feature, the output ends of the preset number of graph attention mechanism sub-modules are connected to the input end of the fourth splicing module, and the output end of the fourth splicing module is used to output the third graph feature and the third attention feature.

[0431] In some embodiments, the graph attention mechanism submodule is a single-headed GAPLayer module.

[0432] In some embodiments, the second model unit 3202 is further configured to input the geometric information and the reconstructed value of the attribute to be processed into the graph attention mechanism sub-module to obtain the initial graph feature and the initial attention feature; based on a preset number of graph attention mechanism sub-modules, obtain a preset number of initial graph features and a preset number of initial attention features; and splice the preset number of initial graph features through the fourth splicing module to obtain the first graph feature; and splice the preset number of initial attention features through the fourth splicing module to obtain the first attention feature.

[0433] In some embodiments, the graph attention mechanism submodule includes at least multiple multi-layer perceptron modules; accordingly, the second model unit 3202 is further configured to construct a graph structure based on the geometric information to assist the reconstructed value of the attribute to be processed, and obtain the graph structure of the points in the reconstructed point set; and extract features from the graph structure through at least one multi-layer perceptron module to obtain initial graph features; and extract features from the reconstructed value of the attribute to be processed through at least one multi-layer perceptron module to obtain first intermediate feature information; and extract features from the initial graph features through at least one multi-layer perceptron module to obtain second intermediate feature information; and use a first preset function to perform feature fusion on the first intermediate feature information and the second intermediate feature information to obtain an attention coefficient; use a second preset function to normalize the attention coefficient to obtain a feature weight; and obtain an initial attention feature based on the feature weight and the initial graph feature.

[0434] In some embodiments, referring to FIG16 , the decoder 320 may further include a second training unit 3206 configured to determine a training sample set; wherein the training sample set includes at least one point cloud sequence; and extracting and processing the at least one point cloud sequence separately to obtain a plurality of sample point sets; and performing model training on the initial model using the geometric information of the plurality of sample point sets and the original values ​​of the attributes to be processed at a preset bit rate to determine a preset network model.

[0435] In some embodiments, the attributes to be processed include color components, and the color components include at least one of the following: a first color component, a second color component, and a third color component; accordingly, the second determination unit 3204 is also configured to, after determining the processed point cloud corresponding to the reconstructed point cloud, if the color component does not conform to the RGB color space, perform color space conversion on the color component of the point in the processed point cloud so that the converted color component conforms to the RGB color space.

[0436] It is understood that in this embodiment, a "unit" can be a portion of a circuit, a portion of a processor, a portion of a program or software, etc., and can also be a module or a non-modular system. Furthermore, the various components in this embodiment can be integrated into a single processing unit, or each unit can exist physically separately, or two or more units can be integrated into a single unit. The aforementioned integrated units can be implemented in the form of hardware or software functional modules.

[0437] If the integrated unit is implemented as a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, this embodiment provides a computer storage medium, which is applied to the decoder 320 and stores a computer program. When the computer program is executed by the second processor, it implements any of the methods in the aforementioned embodiments.

[0438] Based on the composition of the above-mentioned decoder 320 and the computer storage medium, refer to Figure 17, which shows a specific hardware structure diagram of the decoder 320 provided in an embodiment of the present application. As shown in Figure 17, the decoder 320 may include: a second communication interface 3301, a second memory 3302 and a second processor 3303; each component is coupled together through a second bus system 3304. It can be understood that the second bus system 3304 is used to achieve connection and communication between these components. In addition to the data bus, the second bus system 3304 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, various buses are labeled as the second bus system 3304 in Figure 17. Among them,

[0439] The second communication interface 3301 is used to receive and send signals when sending and receiving information with other external network elements;

[0440] The second memory 3302 is used to store computer programs that can be run on the second processor 3303;

[0441] The second processor 3303 is configured to, when running the computer program, execute:

[0442] Determine a reconstruction point set based on the reconstructed point cloud; wherein the reconstruction point set includes at least one point;

[0443] Inputting the geometric information of the points in the reconstructed point set and the reconstructed values ​​of the attributes to be processed into a preset network model, and determining the processed values ​​of the attributes to be processed of the points in the reconstructed point set based on the preset network model;

[0444] According to the processing value of the to-be-processed attribute of the point in the reconstructed point set, a processed point cloud corresponding to the reconstructed point cloud is determined.

[0445] Optionally, as another embodiment, the second processor 3303 is further configured to execute any one of the methods described in the foregoing embodiments when running the computer program.

[0446] It can be understood that the hardware functions of the second memory 3302 are similar to those of the first memory 3102, and the hardware functions of the second processor 3303 are similar to those of the first processor 3103; they will not be described in detail here.

[0447] This embodiment provides a decoder, in which, after obtaining the reconstructed point cloud, the quality enhancement processing of the attribute information of the reconstructed point cloud is performed based on a preset network model, which not only realizes end-to-end operation, but also utilizes the proposed patch extraction and aggregation of the point cloud to realize the blocking operation of the reconstructed point cloud, effectively reducing resource consumption and improving the robustness of the model; in this way, the quality enhancement processing of the attribute information of the reconstructed point cloud according to the network model can make the texture of the processed point cloud clearer and the transition more natural, which shows that the present technical solution has good performance and can effectively improve the quality and visual effect of the point cloud.

[0448] In yet another embodiment of the present application, referring to FIG18 , a schematic diagram illustrating the structure of a coding and decoding system provided in an embodiment of the present application is shown. As shown in FIG18 , the coding and decoding system 340 may include an encoder 3401 and a decoder 3402. The encoder 3401 may be any of the encoders described in the aforementioned embodiments, and the decoder 3402 may be any of the decoders described in the aforementioned embodiments.

[0449] In an embodiment of the present application, in the encoding and decoding system 340, after obtaining the reconstructed point cloud, both the encoder 3401 and the decoder 3402 can enhance the quality of the attribute information of the reconstructed point cloud through a preset network model, which not only realizes end-to-end operation, but also realizes block operation on the reconstructed point cloud, effectively reducing resource consumption and improving the robustness of the model; at the same time, it can also improve the quality and visual effect of the point cloud and improve the compression performance of the point cloud.

[0450] It should be noted that, in this application, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0451] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0452] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0453] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0454] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0455] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims. Industrial Applicability

[0456] In the embodiment of the present application, whether it is the encoding end or the decoding end, a reconstructed point set is determined based on the reconstructed point cloud; the geometric information of the points in the reconstructed point set and the reconstructed values ​​of the attributes to be processed are input into the preset network model, and the processing values ​​of the attributes to be processed of the points in the reconstructed point set are determined based on the preset network model; and the processed point cloud corresponding to the reconstructed point cloud is determined according to the processing values ​​of the attributes to be processed of the points in the reconstructed point set. In this way, the quality enhancement processing of the attribute information of the reconstructed point cloud based on the preset network model not only realizes end-to-end operation, but also determines the reconstructed point set from the reconstructed point cloud and realizes the block operation of the reconstructed point cloud, effectively reducing resource consumption and improving the robustness of the model; in addition, when the geometric information is used as an auxiliary input of the preset network model and the attribute information of the reconstructed point cloud is subjected to quality enhancement processing through the preset network model, the texture of the processed point cloud can be made clearer and the transition more natural, which effectively improves the quality and visual effect of the point cloud, thereby improving the compression performance of the point cloud.

Claims

1. A decoding method, comprising: Determine a reconstruction point set based on the reconstructed point cloud; wherein the reconstruction point set includes at least one point; Inputting the geometric information of the points in the reconstructed point set and the reconstructed values ​​of the attributes to be processed into a preset network model, and determining the processed values ​​of the attributes to be processed of the points in the reconstructed point set based on the preset network model; A processed point cloud corresponding to the reconstructed point cloud is determined according to the processed values ​​of the to-be-processed attributes of the points in the reconstructed point set.

2. The method according to claim 1, wherein The step of determining a reconstruction point set based on the reconstructed point cloud includes: In the reconstructed point cloud, determining key points; The reconstructed point cloud is extracted and processed according to the key points to determine the reconstructed point set; wherein there is a corresponding relationship between the key points and the reconstructed point set.

3. The method according to claim 2, wherein: Determining key points in the reconstructed point cloud includes: Perform farthest point sampling processing on the reconstructed point cloud to determine the key point.

4. The method according to claim 2, wherein: The extracting and processing the reconstructed point cloud according to the key points to determine the reconstructed point set includes: Performing a K-nearest neighbor search in the reconstructed point cloud based on the key point to determine the neighboring points corresponding to the key point; The reconstruction point set is determined based on neighboring points corresponding to the key points.

5. The method according to claim 4, wherein The performing a K-nearest neighbor search in the reconstructed point cloud according to the key point to determine the neighboring points corresponding to the key point includes: Based on the key points, searching for a first preset number of candidate points in the reconstructed point cloud using a K-nearest neighbor search method; respectively calculating the distance values ​​between the key point and the first preset number of candidate points, and determining a second preset number of relatively smaller distance values ​​from the obtained first preset number of distance values; Determine the neighboring points corresponding to the key point based on the candidate points corresponding to the second preset number of distance values; wherein the second preset number is less than or equal to the first preset number.

6. The method according to claim 4, wherein: The determining the set of reconstruction points based on the neighboring points corresponding to the key points includes: The reconstruction point set is determined according to the key point and the neighboring points corresponding to the key point.

7. The method according to claim 2, wherein: The method further comprises: determining the number of points in the reconstructed point cloud; The number of the key points is determined according to the number of points in the reconstructed point cloud and the number of points in the reconstructed point set.

8. The method according to claim 7, wherein: The determining the number of key points according to the number of points in the reconstructed point cloud and the number of points in the reconstructed point set includes: Determine the first factor; Calculating the product of the number of points in the reconstructed point cloud and the first factor; The number of the key points is determined according to the product and the number of points in the reconstructed point set.

9. The method according to claim 2, wherein: Determining a processed point cloud corresponding to the reconstructed point cloud according to the processed values ​​of the to-be-processed attributes of the points in the reconstructed point set includes: determining a target set corresponding to the reconstructed point set according to processed values ​​of to-be-processed attributes of points in the reconstructed point set; The processed point cloud is determined according to the target set.

10. The method according to claim 9, wherein: Determining the processed point cloud according to the target set includes: When there are multiple key points, extracting and processing the reconstructed point cloud according to the multiple key points to obtain multiple reconstructed point sets; After determining the target sets corresponding to the plurality of reconstructed point sets, aggregation processing is performed based on the obtained plurality of target sets to determine the processed point cloud.

11. The method according to claim 10, wherein: The performing aggregation processing on the obtained plurality of target sets to determine the processed point cloud includes: If at least two target sets in the plurality of target sets include processed values ​​of the attribute to be processed of the first point, performing mean calculation on the obtained at least two processed values ​​to determine the processed value of the attribute to be processed of the first point in the processed point cloud; If none of the plurality of target sets includes the processed value of the attribute to be processed of the first point, determining the reconstructed value of the attribute to be processed of the first point in the reconstructed point cloud as the processed value of the attribute to be processed of the first point in the processed point cloud; The first point is any point in the reconstructed point cloud.

12. The method according to claim 1, wherein Inputting the geometric information of the points in the reconstructed point set and the reconstructed values ​​of the attributes to be processed into a preset network model, and determining the processed values ​​of the attributes to be processed of the points in the reconstructed point set based on the preset network model, includes: In the preset network model, a graph structure is constructed to assist in reconstructing the values ​​of the attributes to be processed of the points in the reconstructed point set based on the geometric information of the points in the reconstructed point set, so as to obtain the graph structure of the points in the reconstructed point set; and graph convolution and graph attention mechanism operations are performed on the graph structure of the points in the reconstructed point set to determine the processed values ​​of the attributes to be processed of the points in the reconstructed point set.

13. The method according to claim 1, wherein The preset network model is a neural network model based on deep learning; wherein, the preset network model includes at least a graph attention mechanism module and a graph convolution module.

14. The method according to claim 13, wherein The graph attention mechanism module includes a first graph attention mechanism module and a second graph attention mechanism module, and the graph convolution module includes a first graph convolution module, a second graph convolution module, a third graph convolution module and a fourth graph convolution module; The preset network model further includes a first pooling module, a second pooling module, a first splicing module, a second splicing module, a third splicing module and an addition module; wherein, A first input end of the first graph attention mechanism module is used to receive the geometric information, and a second input end of the first graph attention mechanism module is used to receive the reconstructed value of the attribute to be processed; The first output of the first graph attention mechanism module is connected to the input of the first pooling module, the output of the first pooling module is connected to the input of the first graph convolution module, and the output of the first graph convolution module is connected to the first input of the first splicing module; The second output terminal of the first graph attention mechanism module is connected to the first input terminal of the second splicing module, the second input terminal of the second splicing module is used to receive the reconstructed value of the attribute to be processed, and the output terminal of the second splicing module is connected to the input terminal of the second graph convolution module; The first input end of the second graph attention mechanism module is used to receive the geometric information, the second input end of the second graph attention mechanism module is connected to the output end of the second graph convolution module, the first output end of the second graph attention mechanism module is connected to the input end of the second pooling module, and the output end of the second pooling module is connected to the second input end of the first splicing module; The second output of the second graph attention mechanism module is connected to the first input of the third splicing module, the second input of the third splicing module is connected to the output of the second graph convolution module, the output of the third splicing module is connected to the input of the third graph convolution module, and the output of the third graph convolution module is connected to the third input of the first splicing module; the output of the second graph convolution module is also connected to the fourth input of the first splicing module; The output end of the first splicing module is connected to the input end of the fourth graph convolution module, the output end of the fourth graph convolution module is connected to the first input end of the addition module, the second input end of the addition module is used to receive the reconstructed value of the attribute to be processed, and the output end of the addition module is used to output the processed value of the attribute to be processed.

15. The method according to claim 14, wherein Inputting the geometric information of the points in the reconstructed point set and the reconstructed values ​​of the attributes to be processed into a preset network model, and determining the processed values ​​of the attributes to be processed of the points in the reconstructed point set based on the preset network model, includes: Performing feature extraction on the geometric information and the reconstructed value of the attribute to be processed by the first graph attention mechanism module to obtain a first graph feature and a first attention feature; Performing feature extraction on the first graph feature through the first pooling module and the first graph convolution module to obtain a second graph feature; splicing the first attention feature and the reconstructed value of the attribute to be processed by the second splicing module to obtain a first spliced ​​attention feature; performing feature extraction on the first spliced ​​attention feature through the second graph convolution module to obtain a second attention feature; performing feature extraction on the geometric information and the second attention feature through the second graph attention mechanism module to obtain a third graph feature and a third attention feature; Performing feature extraction on the third image feature by the second pooling module to obtain a fourth image feature; splicing the third attention feature and the second attention feature by the third splicing module to obtain a second spliced ​​attention feature; performing feature extraction on the second spliced ​​attention feature through the third graph convolution module to obtain a fourth attention feature; splicing the second image feature, the fourth image feature, the second attention feature, and the fourth attention feature by the first splicing module to obtain a target feature; Performing a convolution operation on the target feature through the fourth graph convolution module to obtain a residual value of a to-be-processed attribute of a point in the reconstructed point set; The addition module performs an addition operation on the residual value of the attribute to be processed of the point in the reconstructed point set and the reconstructed value of the attribute to be processed to obtain the processed value of the attribute to be processed of the point in the reconstructed point set.

16. The method according to claim 14, wherein The first graph convolution module, the second graph convolution module, the third graph convolution module and the fourth graph convolution module each include at least one convolution layer.

17. The method according to claim 16, wherein The first graph convolution module, the second graph convolution module, the third graph convolution module and the fourth graph convolution module each further include at least one batch normalization layer and at least one activation layer; wherein the batch normalization layer and the activation layer are connected after the convolution layer.

18. The method according to claim 17, wherein The convolution layer of the last layer in the fourth graph convolution module is not connected to the batch normalization layer and the activation layer.

19. The method according to claim 15, wherein The first graph attention mechanism module and the second graph attention mechanism module both include a fourth splicing module and a preset number of graph attention mechanism sub-modules; wherein, In the first graph attention mechanism module, input ends of the preset number of graph attention mechanism sub-modules are used to receive the geometric information and the reconstructed value of the attribute to be processed, output ends of the preset number of graph attention mechanism sub-modules are connected to the input end of the fourth splicing module, and the output end of the fourth splicing module is used to output the first graph feature and the first attention feature; In the second graph attention mechanism module, the input ends of the preset number of graph attention mechanism sub-modules are all used to receive the geometric information and the second attention feature, the output ends of the preset number of graph attention mechanism sub-modules are connected to the input end of the fourth splicing module, and the output end of the fourth splicing module is used to output the third graph feature and the third attention feature.

20. The method according to claim 19, wherein The graph attention mechanism submodule is a single-head GAPLayer module.

21. The method according to claim 19, wherein The step of extracting features of the geometric information and the reconstructed value of the attribute to be processed by the first graph attention mechanism module to obtain a first graph feature and a first attention feature includes: Inputting the geometric information and the reconstructed value of the attribute to be processed into the graph attention mechanism submodule to obtain initial graph features and initial attention features; Based on the preset number of graph attention mechanism submodules, obtaining a preset number of initial graph features and a preset number of initial attention features; splicing the preset number of initial image features by the fourth splicing module to obtain the first image feature; The preset number of initial attention features are spliced ​​together by the fourth splicing module to obtain the first attention feature.

22. The method according to claim 21, wherein The graph attention mechanism submodule includes at least a plurality of multi-layer perceptron modules; The step of inputting the geometric information and the reconstructed value of the attribute to be processed into the graph attention mechanism submodule to obtain initial graph features and initial attention features includes: Assisting in constructing a graph structure of the reconstructed value of the attribute to be processed based on the geometric information to obtain a graph structure of the points in the reconstructed point set; Extracting features from the graph structure using at least one of the multi-layer perceptron modules to obtain the initial graph features; Performing feature extraction on the reconstructed value of the attribute to be processed by at least one of the multi-layer perceptron modules to obtain first intermediate feature information; Extracting features from the initial graph using at least one of the multi-layer perceptron modules to obtain second intermediate feature information; Performing feature fusion on the first intermediate feature information and the second intermediate feature information using a first preset function to obtain an attention coefficient; Normalizing the attention coefficient using a second preset function to obtain a feature weight; The initial attention feature is obtained according to the feature weight and the initial graph feature.

23. The method according to claim 1, wherein The method further comprises: Determine a training sample set; wherein the training sample set includes at least one point cloud sequence; Extracting and processing the at least one point cloud sequence respectively to obtain a plurality of sample point sets; At a preset bit rate, the geometric information of the plurality of sample point sets and the original values ​​of the attributes to be processed are used to perform model training on the initial model to determine the preset network model.

24. The method according to any one of claims 1 to 23, wherein The attribute to be processed includes a color component, and the color component includes at least one of the following: a first color component, a second color component, and a third color component; the method further includes: After determining the processed point cloud corresponding to the reconstructed point cloud, if the color component does not conform to the RGB color space, a color space conversion is performed on the color component of the point in the processed point cloud so that the converted color component conforms to the RGB color space.

25. A coding method, comprising: Encoding and reconstruction processing are performed according to the original point cloud to obtain a reconstructed point cloud; Determine a reconstruction point set based on the reconstructed point cloud; wherein the reconstruction point set includes at least one point; Inputting the geometric information of the points in the reconstructed point set and the reconstructed values ​​of the attributes to be processed into a preset network model, and determining the processed values ​​of the attributes to be processed of the points in the reconstructed point set based on the preset network model; A processed point cloud corresponding to the reconstructed point cloud is determined according to the processed values ​​of the to-be-processed attributes of the points in the reconstructed point set.

26. The method according to claim 25, wherein The determining of a reconstruction point set based on the reconstructed point cloud includes: In the reconstructed point cloud, determining key points; The reconstructed point cloud is extracted and processed according to the key points to determine the reconstructed point set; wherein there is a corresponding relationship between the key points and the reconstructed point set.

27. The method according to claim 26, wherein Determining key points in the reconstructed point cloud includes: Perform farthest point sampling processing on the reconstructed point cloud to determine the key point.

28. The method according to claim 26, wherein The extracting and processing the reconstructed point cloud according to the key points to determine the reconstructed point set includes: Performing a K-nearest neighbor search in the reconstructed point cloud based on the key point to determine the neighboring points corresponding to the key point; The reconstruction point set is determined based on neighboring points corresponding to the key points.

29. The method according to claim 28, wherein The performing a K-nearest neighbor search in the reconstructed point cloud according to the key point to determine the neighboring points corresponding to the key point includes: Based on the key points, searching for a first preset number of candidate points in the reconstructed point cloud using a K-nearest neighbor search method; respectively calculating the distance values ​​between the key point and the first preset number of candidate points, and determining a second preset number of relatively smaller distance values ​​from the obtained first preset number of distance values; Determine the neighboring points corresponding to the key point based on the candidate points corresponding to the second preset number of distance values; wherein the second preset number is less than or equal to the first preset number.

30. The method of claim 28, wherein The determining the set of reconstruction points based on the neighboring points corresponding to the key points includes: The reconstruction point set is determined according to the key point and the neighboring points corresponding to the key point.

31. The method of claim 26, wherein: The method further comprises: determining the number of points in the reconstructed point cloud; The number of the key points is determined according to the number of points in the reconstructed point cloud and the number of points in the reconstructed point set.

32. The method according to claim 31, wherein The determining the number of key points according to the number of points in the reconstructed point cloud and the number of points in the reconstructed point set includes: Determine the first factor; Calculating the product of the number of points in the reconstructed point cloud and the first factor; The number of the key points is determined according to the product and the number of points in the reconstructed point set.

33. The method of claim 26, wherein: Determining a processed point cloud corresponding to the reconstructed point cloud according to the processed values ​​of the to-be-processed attributes of the points in the reconstructed point set includes: determining a target set corresponding to the reconstructed point set according to processed values ​​of to-be-processed attributes of points in the reconstructed point set; The processed point cloud is determined according to the target set.

34. The method according to claim 33, wherein Determining the processed point cloud according to the target set includes: When there are multiple key points, extracting and processing the reconstructed point cloud according to the multiple key points to obtain multiple reconstructed point sets; After determining the target sets corresponding to the plurality of reconstructed point sets, aggregation processing is performed based on the obtained plurality of target sets to determine the processed point cloud.

35. The method according to claim 34, wherein The performing aggregation processing on the obtained plurality of target sets to determine the processed point cloud includes: If at least two target sets in the plurality of target sets include processed values ​​of the attribute to be processed of the first point, performing mean calculation on the obtained at least two processed values ​​to determine the processed value of the attribute to be processed of the first point in the processed point cloud; If none of the plurality of target sets includes the processed value of the attribute to be processed of the first point, determining the reconstructed value of the attribute to be processed of the first point in the reconstructed point cloud as the processed value of the attribute to be processed of the first point in the processed point cloud; The first point is any point in the reconstructed point cloud.

36. The method of claim 25, wherein: Inputting the geometric information of the points in the reconstructed point set and the reconstructed values ​​of the attributes to be processed into a preset network model, and determining the processed values ​​of the attributes to be processed of the points in the reconstructed point set based on the preset network model, includes: In the preset network model, a graph structure is constructed to assist in reconstructing the values ​​of the attributes to be processed of the points in the reconstructed point set based on the geometric information of the points in the reconstructed point set, so as to obtain the graph structure of the points in the reconstructed point set; and graph convolution and graph attention mechanism operations are performed on the graph structure of the points in the reconstructed point set to determine the processed values ​​of the attributes to be processed of the points in the reconstructed point set.

37. The method of claim 25, wherein: The preset network model is a neural network model based on deep learning; Among them, the preset network model includes at least a graph attention mechanism module and a graph convolution module.

38. The method of claim 37, wherein: The graph attention mechanism module includes a first graph attention mechanism module and a second graph attention mechanism module, and the graph convolution module includes a first graph convolution module, a second graph convolution module, a third graph convolution module and a fourth graph convolution module; The preset network model further includes a first pooling module, a second pooling module, a first splicing module, a second splicing module, a third splicing module and an addition module; wherein, A first input end of the first graph attention mechanism module is used to receive the geometric information, and a second input end of the first graph attention mechanism module is used to receive the reconstructed value of the attribute to be processed; The first output of the first graph attention mechanism module is connected to the input of the first pooling module, the output of the first pooling module is connected to the input of the first graph convolution module, and the output of the first graph convolution module is connected to the first input of the first splicing module; The second output terminal of the first graph attention mechanism module is connected to the first input terminal of the second splicing module, the second input terminal of the second splicing module is used to receive the reconstructed value of the attribute to be processed, and the output terminal of the second splicing module is connected to the input terminal of the second graph convolution module; The first input end of the second graph attention mechanism module is used to receive the geometric information, the second input end of the second graph attention mechanism module is connected to the output end of the second graph convolution module, the first output end of the second graph attention mechanism module is connected to the input end of the second pooling module, and the output end of the second pooling module is connected to the second input end of the first splicing module; The second output of the second graph attention mechanism module is connected to the first input of the third splicing module, the second input of the third splicing module is connected to the output of the second graph convolution module, the output of the third splicing module is connected to the input of the third graph convolution module, and the output of the third graph convolution module is connected to the third input of the first splicing module; the output of the second graph convolution module is also connected to the fourth input of the first splicing module; The output end of the first splicing module is connected to the input end of the fourth graph convolution module, the output end of the fourth graph convolution module is connected to the first input end of the addition module, the second input end of the addition module is used to receive the reconstructed value of the attribute to be processed, and the output end of the addition module is used to output the processed value of the attribute to be processed.

39. The method according to claim 38, wherein Inputting the geometric information of the points in the reconstructed point set and the reconstructed values ​​of the attributes to be processed into a preset network model, and determining the processed values ​​of the attributes to be processed of the points in the reconstructed point set based on the preset network model, includes: Performing feature extraction on the geometric information and the reconstructed value of the attribute to be processed by the first graph attention mechanism module to obtain a first graph feature and a first attention feature; Performing feature extraction on the first graph feature through the first pooling module and the first graph convolution module to obtain a second graph feature; splicing the first attention feature and the reconstructed value of the attribute to be processed by the second splicing module to obtain a first spliced ​​attention feature; performing feature extraction on the first spliced ​​attention feature through the second graph convolution module to obtain a second attention feature; performing feature extraction on the geometric information and the second attention feature through the second graph attention mechanism module to obtain a third graph feature and a third attention feature; Performing feature extraction on the third image feature by the second pooling module to obtain a fourth image feature; splicing the third attention feature and the second attention feature by the third splicing module to obtain a second spliced ​​attention feature; performing feature extraction on the second spliced ​​attention feature through the third graph convolution module to obtain a fourth attention feature; splicing the second image feature, the fourth image feature, the second attention feature, and the fourth attention feature by the first splicing module to obtain a target feature; Performing a convolution operation on the target feature through the fourth graph convolution module to obtain a residual value of a to-be-processed attribute of a point in the reconstructed point set; The addition module performs an addition operation on the residual value of the attribute to be processed of the point in the reconstructed point set and the reconstructed value of the attribute to be processed to obtain the processed value of the attribute to be processed of the point in the reconstructed point set.

40. The method of claim 38, wherein The first graph convolution module, the second graph convolution module, the third graph convolution module and the fourth graph convolution module each include at least one convolution layer.

41. The method according to claim 40, wherein The first graph convolution module, the second graph convolution module, the third graph convolution module and the fourth graph convolution module each further include at least one batch normalization layer and at least one activation layer; wherein the batch normalization layer and the activation layer are connected after the convolution layer.

42. The method according to claim 41, wherein The convolution layer of the last layer in the fourth graph convolution module is not connected to the batch normalization layer and the activation layer.

43. The method of claim 39, wherein: The first graph attention mechanism module and the second graph attention mechanism module both include a fourth splicing module and a preset number of graph attention mechanism sub-modules; wherein, In the first graph attention mechanism module, input ends of the preset number of graph attention mechanism sub-modules are used to receive the geometric information and the reconstructed value of the attribute to be processed, output ends of the preset number of graph attention mechanism sub-modules are connected to the input end of the fourth splicing module, and the output end of the fourth splicing module is used to output the first graph feature and the first attention feature; In the second graph attention mechanism module, the input ends of the preset number of graph attention mechanism sub-modules are all used to receive the geometric information and the second attention feature, the output ends of the preset number of graph attention mechanism sub-modules are connected to the input end of the fourth splicing module, and the output end of the fourth splicing module is used to output the third graph feature and the third attention feature.

44. The method according to claim 43, wherein The graph attention mechanism submodule is a single-head GAPLayer module.

45. The method of claim 43, wherein The step of extracting features of the geometric information and the reconstructed value of the attribute to be processed by the first graph attention mechanism module to obtain a first graph feature and a first attention feature includes: Inputting the geometric information and the reconstructed value of the attribute to be processed into the graph attention mechanism submodule to obtain initial graph features and initial attention features; Based on the preset number of graph attention mechanism submodules, obtaining a preset number of initial graph features and a preset number of initial attention features; splicing the preset number of initial image features by the fourth splicing module to obtain the first image feature; The preset number of initial attention features are spliced ​​together by the fourth splicing module to obtain the first attention feature.

46. ​​The method of claim 45, wherein The graph attention mechanism submodule includes at least a plurality of multi-layer perceptron modules; The step of inputting the geometric information and the reconstructed value of the attribute to be processed into the graph attention mechanism submodule to obtain initial graph features and initial attention features includes: Assisting in constructing a graph structure of the reconstructed value of the attribute to be processed based on the geometric information to obtain a graph structure of the points in the reconstructed point set; Extracting features from the graph structure using at least one of the multi-layer perceptron modules to obtain the initial graph features; Performing feature extraction on the reconstructed value of the attribute to be processed by at least one of the multi-layer perceptron modules to obtain first intermediate feature information; Extracting features from the initial graph using at least one of the multi-layer perceptron modules to obtain second intermediate feature information; Performing feature fusion on the first intermediate feature information and the second intermediate feature information using a first preset function to obtain an attention coefficient; Normalizing the attention coefficient using a second preset function to obtain a feature weight; The initial attention feature is obtained according to the feature weight and the initial graph feature.

47. The method of claim 25, wherein: The method further comprises: Determine a training sample set; wherein the training sample set includes at least one point cloud sequence; Extracting and processing the at least one point cloud sequence respectively to obtain a plurality of sample point sets; At a preset bit rate, the geometric information of the plurality of sample point sets and the original values ​​of the attributes to be processed are used to perform model training on the initial model to determine the preset network model.

48. The method according to any one of claims 25 to 47, wherein The attribute to be processed includes a color component, and the color component includes at least one of the following: a first color component, a second color component, and a third color component; the method further includes: After determining the processed point cloud corresponding to the reconstructed point cloud, if the color component does not conform to the RGB color space, a color space conversion is performed on the color component of the point in the processed point cloud so that the converted color component conforms to the RGB color space.

49. An encoder comprising an encoding unit, a first extraction unit, a first model unit, and a first aggregation unit; wherein, The encoding unit is configured to perform encoding and reconstruction processing according to the original point cloud to obtain a reconstructed point cloud; The first extraction unit is configured to determine a reconstruction point set based on the reconstructed point cloud; wherein the reconstruction point set includes at least one point; The first model unit is configured to input the geometric information of the points in the reconstructed point set and the reconstructed values ​​of the attributes to be processed into a preset network model, and determine the processed values ​​of the attributes to be processed of the points in the reconstructed point set based on the preset network model; The first aggregation unit is configured to determine a processed point cloud corresponding to the reconstructed point cloud according to a processing value of a to-be-processed attribute of a point in the reconstructed point set.

50. An encoder comprising a first memory and a first processor; wherein: The first memory is used to store a computer program that can be run on the first processor; The first processor is configured to execute the method according to any one of claims 25 to 48 when running the computer program.

51. A decoder comprising a second extraction unit, a second model unit, and a second aggregation unit; wherein: The second extraction unit is configured to determine a reconstruction point set based on the reconstructed point cloud; wherein the reconstruction point set includes at least one point; The second model unit is configured to input the geometric information of the points in the reconstructed point set and the reconstructed values ​​of the attributes to be processed into a preset network model, and determine the processed values ​​of the attributes to be processed of the points in the reconstructed point set based on the preset network model; The second aggregation unit is configured to determine a processed point cloud corresponding to the reconstructed point cloud according to a processing value of a to-be-processed attribute of a point in the reconstructed point set.

52. A decoder comprising a second memory and a second processor; wherein: The second memory is used to store a computer program that can be run on the second processor; The second processor is configured to execute the method according to any one of claims 1 to 24 when running the computer program.

53. A computer-readable storage medium, wherein: The computer-readable storage medium stores a computer program, and when the computer program is executed, the method according to any one of claims 1 to 24 or the method according to any one of claims 25 to 48 is implemented.

Images