Point cloud attribute coding method based on graph attention network

Abstract

The invention belongs to the technical field of point cloud, and relates to a point cloud attribute coding method based on a graph attention network, which comprises the following steps: acquiring point cloud data, and carrying out block division on the point cloud data; performing detail level generation on the point cloud data subjected to block division, predicting point cloud attributes according to a detail level generation sequence, and inputting a neighbor point set with attribute information into a multi-head image attention layer for feature extraction to obtain a prediction residual error of a current point; encoding the prediction residual error of the current point to obtain an attribute code stream; inputting the attribute code stream into a decoding module to obtain a prediction residual error; searching a neighbor point set of unpredicted points in the prediction point set, and inputting the neighbor point set into the attention layer of the multi-head image to predict the current point; combining the predicted value with the decoded predicted residual error to obtain a reconstruction attribute; according to the method, efficient compression and accurate reconstruction of the point cloud data are realized in a layer-by-layer recursive predictive coding mode, and the method can be applied to a three-dimensional scene needing efficient data transmission and storage.

Description

Technical Field

[0001] The present invention belongs to the field of point cloud technology, and specifically relates to a point cloud attribute encoding method based on a graph attention network. Background Art

[0002] With the gradual maturity of VR, AR and other presentation devices and the rise of the concept of the metaverse, immersive experience has become the development direction of future visual content. Point cloud is a collection of high-dimensional disordered points with various information such as coordinates, colors, normals, etc. As the main data form of immersive video, it has been widely used in scenes such as autonomous driving, mapping, and architectural modeling. Usually a point cloud can contain millions of points, which can represent the scene with high precision, but it also brings great challenges to encoding and transmission technology. Therefore, efficient compression technology is necessary.

[0003] In order to meet the development needs of immersive video, the Moving Picture Experts Group (MPEG) has developed two point cloud compression (PCC) coding standards, namely V-PCC and G-PCC. The TMC13 attribute coding framework, a research framework for GPCC compression coding proposed by MPEG, mainly includes several stages: point cloud preprocessing (recoloring of reconstructed point clouds), transformation, quantization, and entropy coding. Among them, for point cloud attribute coding, prediction transformation and lifting transformation schemes for lossless coding and lossy coding, as well as regional adaptive hierarchical transformation (RAHT) are designed. RAHT is a Haar wavelet-based scheme that combines the Laplace distributed entropy encoder for attribute compression. The prediction transformation is based on the level of detail partitioning (LoD). The prediction transformation combines the nearest neighbor interpolation algorithm and the K-NN algorithm to find the k nearest neighbors of the predicted point, and linearly interpolates the attribute values ​​based on the Euclidean distance between the points. The correlation in the spatial domain of the point cloud is used to remove the spatial redundancy of the point cloud.

[0004] In addition to traditional point cloud coding methods, some learning-based point cloud attribute coding technologies have also been proposed in recent years. Learning-based point cloud compression can better adapt to the complexity of point cloud structures, so it is a very promising way to efficiently transmit and store point clouds. Graph neural networks provide structure-adaptive, accurate and compact representations for point clouds, which is essential for describing the underlying topological structure of point clouds, thereby capturing more detailed features of point cloud attributes. Xihua Sheng et al. proposed MS-GAT in their paper (Attribute Artifacts Removal for Geometry-Based Point Cloud Compression), applying GNN to the task of removing point cloud attribute artifacts and extracting point cloud attribute features using Chebyshev graph convolution. The G-PCC prediction transformation only considers distance-based similarity, which cannot fully utilize the spatial and attribute similarity of point clouds, and it is difficult to further improve compression efficiency while maintaining accuracy. Summary of the invention

[0005] In order to solve the problems existing in the above prior art, the present invention proposes a point cloud attribute encoding method based on a graph attention network, which comprises: obtaining point cloud data and preprocessing the point cloud data; inputting the preprocessed point cloud data into a trained point cloud attribute prediction model to obtain a prediction result; wherein the point cloud attribute prediction model is a graph attention network;

[0006] Training the point cloud attribute prediction model includes: obtaining original point cloud data; dividing the original point cloud data into blocks; generating detail levels for the point cloud data after block division, and predicting the point cloud attributes in the order of detail level generation; for unpredicted points, searching for neighbor point sets in the predicted point set, inputting the neighbor point sets with detail information into the multi-head graph attention layer for feature extraction, and obtaining the prediction residual of the current point; encoding the current point prediction residual to obtain an attribute code stream; inputting the attribute code stream into the decoding module to obtain the inverse quantized prediction residual; searching for the neighbor point sets of unpredicted points in the predicted point set, and inputting the neighbor point sets into the multi-head graph attention layer to predict the current point; merging the predicted value with the decoded prediction residual to obtain the reconstructed attribute; calculating the loss function of the model according to the attribute prediction value, adjusting the loss function of the model, and completing the model training when the loss function converges.

[0007] Beneficial effects of the present invention:

[0008] The present invention realizes efficient compression and accurate reconstruction of point cloud data by layer-by-layer recursive predictive coding, and can be applied to three-dimensional scenes that require efficient data transmission and storage. The present invention improves prediction accuracy through a recursive attention mechanism, that is, a recursive attention model is used to predict point cloud attributes, and a multi-head self-attention mechanism is used to model neighborhood information, which significantly improves prediction accuracy and compression rate. The present invention adopts efficient decoding and high-fidelity reconstruction, that is, the point cloud data is gradually restored by the same recursive model at the decoding end, ensuring that the reconstruction accuracy is close to the original data, meeting high-precision application requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 It is the overall flow chart of the present invention;

[0010] Figure 2 This is a structural diagram of the attribute prediction module of the present invention;

[0011] Figure 3 A rate-distortion curve diagram of the soldier_vox10_0690 test sequence tested by the present invention;

[0012] Figure 4 A rate-distortion curve diagram of the present invention for testing the longdress_viewdep_vox12 test sequence;

[0013] Figure 5 This is a rate-distortion curve diagram of the present invention for testing the loot_viewdep_vox12 test sequence. DETAILED DESCRIPTION

[0014] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0015] In view of the limitation of measuring attribute similarity only by the distance between points in prediction transformation, the present invention proposes an attribute prediction method based on detail level division based on TGAT (Temporal Graph Attention). This method divides the point cloud data into multiple levels of detail levels and uses the same results at each scale to predict and encode the attributes of the points layer by layer. This method applies TGAT to point-level attribute prediction transformation coding. We can regard this encoding process as a graph that changes over time. The predicted points are newly added nodes in the graph, and they are predicted by finding their neighboring points.

[0016] A point cloud attribute encoding method based on a graph attention network, the method comprising: acquiring point cloud data and preprocessing the point cloud data; inputting the preprocessed point cloud data into a trained point cloud attribute prediction model to obtain a prediction result; wherein the point cloud attribute prediction model is a graph attention network. Training the point cloud attribute prediction model includes: obtaining original point cloud data; dividing the original point cloud data into blocks; generating detail levels for the point cloud data after block division, and predicting the point cloud attributes in the order of detail level generation; for unpredicted points, searching for neighbor point sets in the predicted point set, inputting the neighbor point sets with detail information into the multi-head graph attention layer for feature extraction, and obtaining the prediction residual of the current point; encoding the current point prediction residual to obtain an attribute code stream; inputting the attribute code stream into the decoding module to obtain the inverse quantized prediction residual; searching for the neighbor point sets of unpredicted points in the predicted point set, and inputting the neighbor point sets into the multi-head graph attention layer to predict the current point; merging the predicted value with the decoded prediction residual to obtain the reconstructed attribute; calculating the loss function of the model according to the attribute prediction value, adjusting the loss function of the model, and completing the model training when the loss function converges.

[0017] Figure 1 This is the attribute encoding and decoding flow chart of the present invention. The left part of the figure is the encoding flow chart, and the right part is the decoding flow chart. In the present invention, it is assumed that the geometric information has been encoded through a separate pipeline and the geometric decoding end is available. The decoded geometric information is passed as auxiliary information to the attribute encoding. Due to memory limitations, it is unrealistic to encode all the attributes of the complete point cloud at one time. Therefore, the present invention first divides the point cloud into blocks, processes each block separately, and then merges the blocks at the decoding end. Then the blocks are divided into detail levels, and the reconstructed values ​​of the encoded points are used to predict the attributes of the current point according to the order of detail level division. Finally, residual quantization and entropy encoding are performed to obtain the attribute code stream. The decoding end uses the opposite structure to the encoding end for decoding to finally obtain the reconstructed attribute value.

[0018] In this embodiment, the human body dataset 8i Voxelized Full Bodies (8iVFB) and dynamic human body mesh are used for training / testing of the model training data. Among them, five static point clouds, "basketball_player_vox11", "boxer_viewdep_vox12", "dancer_vox11_00000001", "thaidancer_viewdep_vox12", and "redandblack_viewdep_vox12", are used as training data, and other sequences are reserved for testing. Due to the huge amount of point cloud data, it is unrealistic to process the complete point cloud at one time. At the same time, in order to make up for the lack of available training data, a divide and conquer method is used. First, the training point cloud is divided into blocks, and each small block has n points. Considering the limitation of GPU memory, n is set to 2048. Finally, 8466 small blocks are collected in the training data set for training.

[0019] Specifically, each point cloud data is first segmented into a multi-layer structure according to different levels of detail, and multiple LODs are generated to capture the correlation of points at different scales. Specifically, the point cloud data is divided into multiple levels, each level consists of points of different fineness, and the detail information is gradually divided. For example, the point cloud data can be sampled and segmented according to specific rules to obtain point sets of coarse-grained layers and fine-grained layers. In this way, the point set of each scale can be regarded as an ordered level of detail. Then, each scale is sent to a branch with the same structure for attribute feature prediction. The input of the lth scale is LOD l Original attribute information A l ={a l,1 ,a l,2 ,…,a l,n}, the corresponding geometric information X l ={x l,1 ,x l,2 ,…,x l,n} as auxiliary information input, and output as predicted attribute information And the reconstructed attribute information In addition, LOD l-1 Reconstruction properties It is also used as the input of the lth scale.

[0020] In order to capture the correlation between the current point and its neighboring points and achieve more accurate attribute prediction, this module extracts features by improving the recursive network model TGAT. This module contains a self-attention mechanism, which uses a multi-head attention layer to calculate the spatial and attribute correlation between points to learn more accurate predictions. The point attributes at each level are encoded to obtain the predicted value. Then, by calculating the residual between the predicted value and the true value, the residual is quantized and stored for more efficient compression.

[0021] In this embodiment, the point cloud is divided into blocks, and the number of points in a complete point cloud can easily reach one million. Taking into account memory limitations, the point cloud is divided into blocks, and the entire point cloud with an arbitrary number of N points is divided into small blocks with a fixed number of n points. Since N is usually not divisible by n, the number of points in the last block is less than n. Therefore, a point filling operation is used to add a certain number of points to the last block so that N can be divided by n. After blocking, each block is used as a basic unit and is encoded and decoded independently and identically. Set n to 2048. Corresponding to the block partitioning, the restored blocks are reassembled into a complete point cloud. It should be noted that since points are added to the original point cloud for block division, these added points need to be removed.

[0022] In this embodiment, detail level generation for small blocks includes: dividing the point cloud data into multiple levels, each level is composed of points of different fineness, and gradually dividing the detail information. For example, the point cloud data can be sampled and segmented according to specific rules to obtain point sets of coarse-grained layers and fine-grained layers, so that the point sets of each scale can be regarded as an ordered level of detail. Specifically, the detail level generation adopts a downsampling operation to divide the small block into three levels of detail. First, 2048 points are downsampled with a sampling period of 2 to obtain LoD1 of 1024 points, and then LoD1 is downsampled to obtain LoD0 of 512 points.

[0023] Figure 2 is the i-th point v i Attribute prediction structure diagram. The network inputs the geometry and attribute information of the encoded point set. First, the KD tree is used to perform a neighborhood search to search its 26 neighboring points to predict the attribute value of the current point. The original attribute feature of the current node is the arithmetic mean of the feature values ​​of the neighboring points. Then, the neighboring nodes and the current node features are enhanced. The enhanced feature values ​​of the current node and the neighboring points are passed to the multi-head graph attention layer to learn and predict the attribute value of the node. The output is the predicted attribute value of the current node.

[0024] In this embodiment, processing the point cloud using a multi-head graph attention layer includes:

[0025] Step 1: Use KD tree to search neighbor point sets in the encoded point set to obtain the neighborhood point set of unpredicted points; this part specifically includes the following steps:

[0026] Step 11: Merge the points in the previous level of detail with the encoded points in the current level of detail to obtain an encoded reconstructed point set;

[0027] Step 12: construct a KD tree using the encoded point set, and use the KD tree to perform efficient nearest neighbor search;

[0028] Step 13: The number of neighbor point sets of each point is k. If the number of points in the encoded point set is less than k, all points are used as the neighbor point set of the current point.

[0029] Step 2: Input the points in the neighborhood point set into the attention network to obtain the attribute prediction value; specifically:

[0030] Step 21: construct a graph of the current node and the encoded neighboring nodes, with the attribute value of each point as the node feature and the distance between the current node and the neighboring node as the edge feature;

[0031] Step 22: Pass the node features and edge features into the multi-head graph attention layer, and use the multi-head graph attention layer to capture the attribute correlation between neighboring nodes; weight the information of the neighboring points into the encoding of the current point to obtain the attribute prediction value;

[0032] Step 23: Output the predicted value of the current node.

[0033] Step 3: Update the coded point set according to the attribute prediction value;

[0034] Step 4: Perform residual quantization on the attribute prediction value and the input point cloud;

[0035] Step 5: Encode the data after residual quantization processing;

[0036] Step 6: Repeat steps 1 to 5 until all point clouds are processed and the attribute encoding stream is obtained.

[0037] Specifically, the attribute prediction stage is mainly composed of a multi-head graph attention layer. Figure 2 As shown, the prediction module architecture of the lth level of detail. i and a i represents the i-th node and its original node features, h i Represents node v i The hidden representation output of this module is the original node feature a of the target node v0. i and reconstructed encoded node attribute information The specific steps are as follows:

[0038] Step 1: Point neighborhood calculation: Use KD tree to search for the nearest neighbor to determine the neighborhood point set of each point

[0039] The neighborhood point set is used to represent the spatial neighborhood information of the current point and provide the model with geometric and attribute information between the point and its neighborhood points.

[0040] Step 2, attention network coding: Node v0 and the reconstructed neighboring nodes construct a graph, the attribute value of each point is used as the node feature, and the distance between nodes is used as the edge feature, which is passed to the k-head graph attention layer, and the multi-head graph attention layer is used to capture the attribute correlation between neighboring nodes. Through this mechanism, the information of the neighboring points is weighted and integrated into the encoding of the current point to generate a more accurate prediction. The final output is the predicted value of the current node v0.

[0041] Step 3: Residual quantization and entropy coding: The residual signal between the real attribute of the current point and the predicted value is quantized to obtain the quantized residual, which is arithmetic-coded to obtain the attribute code stream.

[0042] Decoding the encoded attributes includes: the decoding process is the opposite of the encoding order, decoding the detail level of the point cloud layer by layer, and reconstructing the point attributes using the encoded residual signal. Starting from the coarsest level, the predicted attributes and residual signals of each detail level are superimposed level by level to achieve high-precision point cloud reconstruction.

[0043] The specific process of the decoding module decoding the encoded attributes is:

[0044] Step 1: The decoding end searches for the neighbor point set of the current node from the decoded reconstructed point set;

[0045] Step 2: Use the neural network structure consistent with the encoding end to predict the attribute value of the current point to obtain the predicted value of the current point;

[0046] Step 3: Remove the added points from the decoding result of the last small block;

[0047] Step 4: Reassemble the decoded blocks back into a complete point cloud.

[0048] In the process of training the model, the model training goal is to improve the prediction accuracy, so the attribute A of the original point cloud is g As the ground truth, calculate the predicted value The point-by-point difference between the color attribute and the ground truth. Considering that the three components (y,u,v) of the color attribute have large feature differences, three models with the same structure are trained for each component of (y,u,v) to perform independent and identical processing. Therefore, the loss function of each component is the same. Here we take the y component as an example:

[0049]

[0050] Among them, Y g ={y g1 ,y g2 ,…,Y gn} is the set of ground truth values ​​of the y components of n points, is a set of predicted values ​​of the y components of n points.

[0051] In order to evaluate the effectiveness of the present invention, the performance of the present invention is compared with that of the predictive transform from MPEG. The predictive transform has a similar encoding workflow as the framework of the present invention. Therefore, it has comparative significance as a benchmark for the present invention. Tests were conducted on the test data set provided by MPEG, which includes three sequences: "soldier_vox10_0690", "longdress_viewdep_vox12", and "loot_viewdep_vox12". Rate-distortion performance is used as an indicator to evaluate the performance of the framework. The rate is calculated in bits per point (Bits Per Point, BPP), and the distortion is evaluated by the Peak Signal-to-Noise Ratio (Peak Signal-to-Noise Ratio, PSNR) of the three color channels, where the Y channel draws the rate-distortion curve as shown Figure 3 to Figure 5 For quantitative analysis, other methods are also used to calculate the BD-BR metric of the proposed framework, as shown in Table 1.

[0052] Table 1 Comparison between the present invention and the predictive transformation BD-BR

[0053]

[0054] As can be seen from Table 1, the method proposed in the present invention achieves 3.1%, 7.7% and 6.5% BD-BR reduction on the Y, U and V components respectively.

[0055] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation modes of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A point cloud attribute encoding method based on graph attention network, characterized in that: include: Acquire point cloud data and pre-process the point cloud data; Input the preprocessed point cloud data into the trained point cloud attribute prediction model to obtain the prediction result; The point cloud attribute prediction model is a graph attention network; Training the point cloud attribute prediction model includes: obtaining original point cloud data; dividing the original point cloud data into blocks; generating detail levels for the point cloud data after block division, and predicting the point cloud attributes in the order of detail level generation; for unpredicted points, searching for neighbor point sets in the predicted point set, inputting the neighbor point sets with attribute information into the multi-head graph attention layer for feature extraction, and obtaining the prediction residual of the current point; encoding the current point prediction residual to obtain an attribute code stream; inputting the attribute code stream into the decoding module to obtain the inverse quantized prediction residual; searching for the neighbor point sets of unpredicted points in the predicted point set, and inputting the neighbor point sets into the multi-head graph attention layer to predict the current point; merging the predicted value with the decoded prediction residual to obtain the reconstructed attribute; calculating the loss function of the model according to the attribute prediction value, adjusting the loss function of the model, and completing the model training when the loss function converges.

2. According to claim 1, a point cloud attribute encoding method based on graph attention network is characterized in that: Preprocessing point cloud data includes cleaning point cloud data.

3. According to claim 1, a point cloud attribute encoding method based on graph attention network is characterized in that: The block division of the original point cloud data includes: dividing the point cloud data into small blocks with n point clouds, wherein the point cloud in the last small block is completed.

4. According to claim 1, a point cloud attribute encoding method based on graph attention network is characterized in that: Obtaining detailed information of the point cloud data after block division includes: dividing the small block of point cloud data into multiple levels, each level is composed of points of different fineness, and gradually dividing the detailed information.

5. The point cloud attribute encoding method based on graph attention network according to claim 1, characterized in that: The point cloud is processed using a multi-head graph attention layer including: Step 1: Use KD tree to search neighbor point sets in the encoded point set to obtain the neighborhood point set of unpredicted points; Step 2: Input the points in the neighborhood point set into the attention network to obtain the attribute prediction value; Step 3: Update the coded point set according to the attribute prediction value; Step 4: Perform residual quantization on the attribute prediction value and the input point cloud; Step 5: Encode the data after residual quantization processing; Step 6: Repeat steps 1 to 5 until all point clouds are processed and the attribute encoding stream is obtained.

6. A point cloud attribute encoding method based on graph attention network according to claim 5, characterized in that: Using KD tree to search for point clouds with detailed information includes: Step 1: Merge the points in the previous level of detail with the encoded points in the current level of detail to obtain an encoded reconstructed point set; Step 2: Construct a KD tree using the encoded point set and use the KD tree for efficient nearest neighbor search; Step 3: The number of neighbor point sets of each point is k. If the number of points in the encoded point set is less than k, all points are used as the neighbor point set of the current point.

7. The point cloud attribute encoding method based on graph attention network according to claim 5, characterized in that: The point cloud processing using attention network includes: Step 1: Construct a graph of the current node and the encoded neighboring nodes. The attribute value of each point is used as the node feature, and the distance between the current node and the neighboring node is used as the edge feature. Step 2: Pass the node features and edge features into the multi-head graph attention layer, and use the multi-head graph attention layer to capture the attribute correlation between neighboring nodes; weight the information of the neighboring points into the encoding of the current point to obtain the attribute prediction value; Step 3: Output the predicted value of the current node.

8. The point cloud attribute encoding method based on graph attention network according to claim 1, characterized in that: The decoding module decodes the encoded attributes including: Step 1: The decoding end searches for the neighbor point set of the current node from the decoded reconstructed point set; Step 2: Use the neural network structure consistent with the encoding end to predict the attribute value of the current point to obtain the predicted value of the current point; Step 3: Remove the added points from the decoding result of the last small block; Step 4: Reassemble the decoded blocks back into a complete point cloud.

9. The point cloud attribute encoding method based on graph attention network according to claim 1, characterized in that: The loss function of the model is: Among them, Y g ={y g1 ,y g2 ,…,Y gn } is the set of ground truth values ​​of the y components of n points, A set of predicted values ​​of the y components of n points.

Images