A point cloud attribute encoding method based on a graph attention network

Abstract

The application belongs to the technical field of point cloud and relates to a point cloud attribute coding method based on a graph attention network, which comprises the following steps: acquiring point cloud data, performing block division on the point cloud data; generating a detail level for the point cloud data after the block division, predicting the point cloud attribute according to the detail level generation sequence, inputting a neighbor point set with attribute information into a multi-head graph attention layer for feature extraction to obtain a prediction residual of a current point; encoding the prediction residual of the current point to obtain an attribute code stream; inputting the attribute code stream into a decoding module to obtain a prediction residual; searching for a neighbor point set of an un-predicted point in a prediction point set and inputting the neighbor point set into the multi-head graph attention layer to predict the current point; and merging the prediction value and the decoded prediction residual to obtain a reconstructed attribute; the application realizes efficient compression and accurate reconstruction of the point cloud data through a layer-by-layer recursive prediction coding mode and can be applied to a three-dimensional scene requiring efficient data transmission and storage.

Description

Technical Field

[0001] This invention belongs to the field of point cloud technology, specifically relating to a point cloud attribute encoding method based on graph attention networks. Background Technology

[0002] With the gradual maturation of VR and AR presentation devices and the rise of the metaverse concept, immersive experiences have become the future direction of visual content development. Point clouds, a collection of high-dimensional, unordered points containing various information such as coordinates, color, and normals, are widely used as the primary data form for immersive videos in scenarios such as autonomous driving, surveying, and architectural modeling. A single point cloud can typically contain millions of points, enabling high-precision representation of a scene, but this also presents significant challenges to encoding and transmission technologies. Therefore, efficient compression techniques are essential.

[0003] To meet the demands of immersive video development, the Moving Picture Experts Group (MPEG) has developed two point cloud compression (PCC) coding standards: V-PCC and G-PCC. The MPEG-proposed TMC13 attribute coding framework for GPCC compression coding mainly includes several stages: point cloud preprocessing (recoloring of reconstructed point clouds), transform, quantization, and entropy coding. Specifically, for point cloud attribute coding, predictive transform and lifting transform schemes for lossless and lossy coding, as well as Region Adaptive Hierarchical Transform (RAHT), are designed. RAHT is a Haar wavelet-based scheme that combines a Laplace distributed entropy encoder for attribute compression. The predictive transform is based on Level of Detail (LoD) partitioning. It combines nearest neighbor interpolation and K-NN algorithms to find the k nearest neighbors of the predicted point, performing linear interpolation prediction of attribute values ​​based on the Euclidean distance between points, and utilizing the spatial correlation of the point cloud to remove spatial redundancy.

[0004] Besides traditional point cloud encoding methods, several learning-based point cloud attribute encoding techniques have been proposed in recent years. Learning-based point cloud compression can better adapt to the complexity of point cloud structures, making it a promising approach for efficient transmission and storage of point clouds. Graph neural networks (GNNs) provide structurally adaptive, accurate, and compact representations of point clouds, which is crucial for describing the underlying topology of point clouds, thereby capturing more detailed features of point cloud attributes. Xihua Sheng et al. proposed MS-GAT in their paper (Attribute Artifacts Removal for Geometry-Based Point Cloud Compression), applying GNNs to the task of point cloud attribute artifact removal and using Chebyshev graph convolution to extract point cloud attribute features. G-PCC prediction transform only considers distance-based similarity, failing to fully utilize the spatial and attribute similarity of point clouds, making it difficult to further improve compression efficiency while maintaining accuracy. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention proposes a point cloud attribute encoding method based on a graph attention network. The method includes: acquiring point cloud data and preprocessing the point cloud data; inputting the preprocessed point cloud data into a trained point cloud attribute prediction model to obtain a prediction result; wherein the point cloud attribute prediction model is a graph attention network.

[0006] Training the point cloud attribute prediction model includes: acquiring raw point cloud data; dividing the raw point cloud data into blocks; generating detail levels for the block-divided point cloud data and predicting point cloud attributes according to the order of detail level generation; for unpredicted points, searching for neighboring point sets in the predicted point set, inputting the neighboring point sets with detailed information into the multi-head graph attention layer for feature extraction, and obtaining the prediction residual for the current point; encoding the prediction residual for the current point to obtain the attribute code stream; inputting the attribute code stream into the decoding module to obtain the inverse-quantized prediction residual; searching for neighboring point sets for unpredicted points in the predicted point set, inputting the neighboring point sets into the multi-head graph attention layer to predict the current point; merging the predicted values ​​with the decoded prediction residuals to obtain the reconstructed attributes; calculating the model's loss function based on the attribute prediction values, adjusting the model's loss function, and completing the model training when the loss function converges.

[0007] The beneficial effects of this invention are:

[0008] This invention achieves efficient compression and accurate reconstruction of point cloud data through a layer-by-layer recursive predictive coding approach, applicable to 3D scenes requiring efficient data transmission and storage. It improves prediction accuracy through a recursive attention mechanism, employing a recursive attention model to predict point cloud attributes and utilizing a multi-head self-attention mechanism to model neighborhood information, significantly enhancing prediction accuracy and compression ratio. Furthermore, it employs efficient decoding and high-fidelity reconstruction, using the same recursive model to progressively recover point cloud data at the decoding end, ensuring the reconstruction accuracy closely approximates the original data and meeting the demands of high-precision applications. Attached Figure Description

[0009] Figure 1 This is an overall flowchart of the present invention;

[0010] Figure 2 This is a structural diagram of the attribute prediction module of the present invention;

[0011] Figure 3 This is a rate-distortion curve of the test sequence of the Soldier_vox10_0690 according to the present invention;

[0012] Figure 4 This is a rate-distortion curve of the test sequence of longdress_viewdep_vox12 according to the present invention;

[0013] Figure 5 This is a rate-distortion curve of the test sequence of loot_viewdep_vox12 according to the present invention. Detailed Implementation

[0014] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0015] To address the limitation of measuring attribute similarity solely through distance between points in prediction transformations, this invention proposes an attribute prediction method based on level-of-detail partitioning, utilizing TGAT (Temporal Graph Attention). This method performs multi-level level-of-detail partitioning on point cloud data and uses the same results at each scale to predict and encode point attributes layer by layer. The method applies TGAT to point-level attribute prediction transformation encoding. This encoding process can be viewed as a graph that changes over time, where the predicted point is a newly added node in the graph, and prediction is performed by finding its neighboring points.

[0016] A point cloud attribute encoding method based on graph attention network is disclosed. The method includes: acquiring point cloud data and preprocessing the point cloud data; inputting the preprocessed point cloud data into a trained point cloud attribute prediction model to obtain the prediction result; wherein the point cloud attribute prediction model is a graph attention network. Training the point cloud attribute prediction model includes: acquiring raw point cloud data; dividing the raw point cloud data into blocks; generating detail levels for the block-divided point cloud data and predicting point cloud attributes according to the order of detail level generation; for unpredicted points, searching for neighboring point sets in the predicted point set, inputting the neighboring point sets with detailed information into the multi-head graph attention layer for feature extraction, and obtaining the prediction residual for the current point; encoding the prediction residual for the current point to obtain the attribute code stream; inputting the attribute code stream into the decoding module to obtain the inverse-quantized prediction residual; searching for neighboring point sets for unpredicted points in the predicted point set, inputting the neighboring point sets into the multi-head graph attention layer to predict the current point; merging the predicted values ​​with the decoded prediction residuals to obtain the reconstructed attributes; calculating the model's loss function based on the attribute prediction values, adjusting the model's loss function, and completing the model training when the loss function converges.

[0017] Figure 1 This is a flowchart of the attribute encoding and decoding process of this invention. The left side of the flowchart is the encoding process, and the right side is the decoding process. In this invention, it is assumed that the geometric information has been encoded through a separate pipeline, and the geometric decoding end is available. The decoded geometric information is passed as auxiliary information to the attribute encoder. Due to memory limitations, it is impractical to encode all attributes of the complete point cloud at once. Therefore, this invention first divides the point cloud into blocks, processes each block separately, and then merges the blocks at the decoding end. Then, the blocks are divided at the level of detail. According to the order of the level of detail division, the reconstructed values ​​of the encoded points are used to predict the attributes of the current point. Finally, residual quantization and entropy encoding are performed to obtain the attribute bitstream. The decoding end uses the opposite structure to the encoding end to decode and finally obtain the reconstructed attribute values.

[0018] In this embodiment, the training data for the model uses the 8i Voxelized Full Bodies (8iVFB) human body dataset and a dynamic human body mesh for training / testing. Five static point clouds, "basketball_player_vox11", "boxer_viewdep_vox12", "dancer_vox11_00000001", "thaidancer_viewdep_vox12", and "redandblack_viewdep_vox12", are used as training data, while other sequences are reserved for testing. Due to the massive amount of point cloud data, processing the entire point cloud at once is impractical. To compensate for the limited available training data, a divide-and-conquer approach is used. First, the training point cloud is divided into blocks, with each block containing n points. Considering the GPU memory limitations, n is set to 2048. Ultimately, 8466 blocks are collected from the training dataset for training.

[0019] Specifically, each point cloud dataset is first segmented into multiple layers according to different levels of detail, generating multiple Levels of Detail (LODs) to capture the correlation of points at different scales. In other words, the point cloud data is divided into multiple levels, each composed of points of varying fineness, progressively subdividing the detailed information. For example, the point cloud data can be sampled and segmented according to specific rules to obtain coarse-grained and fine-grained point sets, thus each scale's point set can be considered an ordered level of detail. Then, each scale is fed into a branch with the same structure for attribute feature prediction. The input for the l-th scale is the LOD. l Original attribute information A l ={a l,1 ,a l,2 ,…,a l,n}, the corresponding geometric information X l ={x l,1 ,x l,2 ,…,x l,n As auxiliary information input, the output is the predicted attribute information. and the reconstructed attribute information In addition, LOD l-1 Reconstruction properties It is also used as the input for the l-th scale.

[0020] To capture the correlation between the current point and its neighbors and achieve more accurate attribute prediction, this module improves the TGAT recurrent network model for feature extraction. This module incorporates a self-attention mechanism, using multi-head attention layers to compute spatial and attribute correlations between points, thereby learning more accurate predictions. Point attributes at each level are encoded to obtain predicted values. Then, the residuals between the predicted and true values ​​are calculated, quantized, and stored for more efficient compression.

[0021] In this embodiment, the point cloud is divided into blocks, meaning the number of points in a complete point cloud can easily reach millions. Considering memory limitations, the point cloud is divided into blocks, with any number of N points divided into smaller blocks of a fixed number of n points each. Since N is usually not divisible by n, the number of points in the last block is less than n. Therefore, a point-filling operation is used to add a certain number of points to the last block so that N is divisible by n. After block division, each block is treated as a basic unit and encoded and decoded independently and identically. n is set to 2048. Corresponding to the block partitioning, the restored blocks are reassembled back into a complete point cloud. It should be noted that since points were added to the original point cloud for block division, these added points need to be removed.

[0022] In this embodiment, generating detail levels for small blocks includes dividing the point cloud data into multiple levels, each composed of points of different fineness, and progressively subdividing the detailed information. For example, the point cloud data can be sampled and segmented according to specific rules to obtain point sets of coarse-grained and fine-grained levels. In this way, the point set at each scale can be regarded as an ordered level of detail. Specifically, the level of detail generation adopts a downsampling operation, dividing the small block into three levels of detail. First, 2048 points are downsampled with a sampling period of 2, resulting in LoD1 of 1024 points. Then, LoD1 is downsampled again to obtain LoD0 of 512 points.

[0023] Figure 2 It is the i-th point v i The attribute prediction structure diagram shows that the network takes the geometric and attribute information of the encoded point set as input. First, it uses a KD tree to perform a neighborhood search, searching for its 26 nearest neighbors to predict the attribute value of the current point. The original attribute features of the current node are the arithmetic mean of the feature values ​​of the nearest neighbors. Then, the features of the neighbor nodes and the current node are enhanced. The enhanced feature values ​​of the current node and the nearest neighbors are passed to the multi-head graph attention layer to learn and predict the attribute value of the current node. The output is the predicted attribute value of the current node.

[0024] In this embodiment, the point cloud processing using a multi-head graph attention layer includes:

[0025] Step 1: Use a KD-tree to search the neighbor set of the encoded points to obtain the set of neighbor points for the unpredicted points; this part specifically includes the following steps:

[0026] Step 11: Merge the points in the previous level of detail with the already encoded points in the current level of detail to obtain the set of encoded reconstructed points;

[0027] Step 12: Construct a KD tree using the encoded point set, and use the KD tree for efficient nearest neighbor search;

[0028] Step 13: The number of neighbor points for each point is k. If the number of points in the encoded point set is less than k, then all points are used as the neighbor point set for the current point.

[0029] Step 2: Input the points in the neighborhood point set into the attention network to obtain the attribute prediction values; specifically:

[0030] Step 21: Construct a graph between the current node and its encoded neighboring nodes. The attribute values ​​of each node are used as node features, and the distance between the current node and its neighboring nodes is used as edge features.

[0031] Step 22: Input the node features and edge features into the multi-head graph attention layer, and use the multi-head graph attention layer to capture the attribute correlation between neighboring nodes; integrate the information of the neighboring points into the encoding of the current point in a weighted manner to obtain the attribute prediction value;

[0032] Step 23: Output the predicted value of the current node.

[0033] Step 3: Update the encoded point set based on the attribute prediction values;

[0034] Step 4: Perform residual quantization on the attribute prediction values ​​and the input point cloud;

[0035] Step 5: Encode the data after residual quantization.

[0036] Step 6: Repeat steps 1 to 5 until all point clouds have been processed and the attribute encoding stream is obtained.

[0037] Specifically, the attribute prediction stage mainly consists of a multi-head graph attention layer. For example... Figure 2 The diagram shows the architecture of the prediction module at the l-th level of detail. Using v... i and a i Represents the i-th node and its original node characteristics, h i Represents node v i The hidden representation of the output. The input of this module is the original node features a of the target node v0. i and reconstructed encoded node attribute information The specific steps are as follows:

[0038] Step 1: Calculate the neighborhood of each point: Use a KD-tree to search for the nearest neighbor and determine the set of neighboring points for each point.

[0039] The neighborhood point set is used to represent the spatial neighborhood information of the current point, providing the model with geometric and attribute information between the point and its neighbors.

[0040] Step 2, Attention Network Encoding: A graph is constructed between node v0 and its reconstructed neighbor nodes. The attribute values ​​of each node are used as node features, and the distances between nodes are used as edge features. This graph is then fed into a k-head graph attention layer, which captures the attribute correlations between neighboring nodes. Through this mechanism, the information from neighboring points is weighted and incorporated into the encoding of the current node, generating a more accurate prediction. The final output is the predicted value for the current node v0.

[0041] Step 3: Residual Quantization and Entropy Coding. The residual signal between the actual attribute and the predicted value at the current point is quantized to obtain the quantized residual, which is then arithmetically encoded to obtain the attribute bitstream.

[0042] Decoding the encoded attributes involves reversing the encoding process, decoding the detail levels of the point cloud layer by layer, and reconstructing the point attributes using the encoded residual signals. Starting with the coarsest level, predicted attributes and residual signals from each detail level are superimposed progressively to achieve high-precision point cloud reconstruction.

[0043] The specific process of the decoding module decoding the encoded attributes is as follows:

[0044] Step 1: The decoding end searches for the set of neighboring points of the current node from the set of already decoded and reconstructed points;

[0045] Step 2: Use the same neural network structure as the encoder to predict the attribute value of the current point and obtain the predicted value of the current point;

[0046] Step 3: Remove the added points from the decoding result of the last small block;

[0047] Step 4: Reassemble the decoded blocks back into a complete point cloud.

[0048] During model training, the goal is to improve prediction accuracy; therefore, the original point cloud attribute A is... g As the ground truth, calculate the predicted value. Point-by-point differences from the ground truth. Considering the significant differences in features among the three components (y, u, v) of the color attribute, three models with identical structures are trained for each component (y, u, v), and processed independently and identically. Therefore, the loss function is the same for each component. Here, we take the y component as an example:

[0049]

[0050] Among them, Y g ={y g1 ,y g2 ,…,Y gn Let} be the set of ground truth values ​​of the y-components of n points. It is the set of predicted values ​​of the y-components of n points.

[0051] To evaluate the effectiveness of this invention, its performance was compared with that of the predictive transform from MPEG. The predictive transform has a similar encoding workflow to the framework of this invention. Therefore, it is meaningful to compare it as a benchmark. Tests were conducted on the test dataset provided by MPEG, which included three sequences: “soldier_vox10_0690”, “longdress_viewdep_vox12”, and “loot_viewdep_vox12”. Rate-distortion performance was used as the metric for evaluating framework performance. Rate was calculated in bits per point (BPP), and distortion was evaluated using the peak signal-to-noise ratio (PSNR) of the three color channels, with the Y channel rate-distortion curve plotted as follows: Figures 3-5 As shown in Table 1, other methods were also used to calculate the BD-BR metric of the proposed framework for quantitative analysis.

[0052] Table 1 Comparison of the present invention with BD-BR based on predictive transform.

[0053]

[0054] As shown in Table 1, the method proposed in this invention achieved BD-BR reductions of 3.1%, 7.7%, and 6.5% in the Y, U, and V components, respectively.

[0055] The above-described embodiments further illustrate the purpose, technical solution, and advantages of the present invention. It should be understood that the above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made to the present invention within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A point cloud attribute encoding method based on graph attention networks, characterized in that, include: Acquire point cloud data and preprocess the point cloud data; The preprocessed point cloud data is input into the trained point cloud attribute prediction model to obtain the prediction results; The point cloud attribute prediction model is a graph attention network. Training the point cloud attribute prediction model includes: acquiring raw point cloud data; dividing the raw point cloud data into blocks; generating detail levels for the block-divided point cloud data and predicting point cloud attributes according to the order of detail level generation; for unpredicted points, searching for neighboring point sets in the predicted point set, inputting the neighboring point sets with attribute information into the multi-head graph attention layer for feature extraction to obtain the attribute prediction value of the current point; encoding the attribute prediction value of the current point to obtain the attribute bitstream; inputting the attribute bitstream into the decoding module to obtain the inverse-quantized prediction residual; searching for neighboring point sets of unpredicted points in the predicted point set, inputting the neighboring point sets into the multi-head graph attention layer to predict the current point; merging the prediction value with the decoded prediction residual to obtain the reconstructed attribute; calculating the model's loss function based on the attribute prediction value, adjusting the model's loss function, and completing the model training when the loss function converges. Obtaining detailed information about point cloud data after block partitioning includes: dividing small block point cloud data into multiple levels, each level consisting of points of different fineness, and gradually dividing detailed information; Point cloud processing using multi-head graph attention layers includes: Step 1: Use a KD tree to search for the neighbor set of the predicted points to obtain the set of neighbor points of the unpredicted points. Step 11: Merge the points in the previous level of detail with the already encoded points in the current level of detail to obtain the set of encoded reconstructed points; Step 12: Construct a KD tree using the encoded set of reconstructed points, and use the KD tree for efficient nearest neighbor search; Step 13: The number of neighbor points for each point is k. If the number of points in the already encoded reconstructed point set is less than k, then all points are used as the neighbor point set for the current point. Step 2: Input the points in the neighborhood point set into the graph attention network to obtain the attribute prediction values; Step 3: Update the encoded reconstructed point set based on the attribute prediction values; Step 4: Perform residual quantization on the attribute prediction values ​​and the input point cloud; Step 5: Encode the data after residual quantization. Step 6: Repeat steps 1 to 5 until all point clouds have been processed and the attribute encoding stream is obtained.

2. The point cloud attribute encoding method based on graph attention network according to claim 1, characterized in that, Preprocessing point cloud data includes cleaning the point cloud data.

3. The point cloud attribute encoding method based on graph attention network according to claim 1, characterized in that, Dividing the original point cloud data into blocks involves dividing the point cloud data into small blocks with n point clouds, and completing the point cloud in the last small block.

4. The point cloud attribute encoding method based on graph attention network according to claim 1, characterized in that, The process of inputting points from the neighborhood point set into the graph attention network includes: Step 1: Construct a graph between the current node and its encoded neighboring nodes. The attribute values ​​of each node are used as node features, and the distance between the current node and its neighboring nodes is used as edge features. Step 2: Input the node features and edge features into the multi-head graph attention layer, and use the multi-head graph attention layer to capture the attribute correlation between neighboring nodes; integrate the information of the neighboring points into the encoding of the current point in a weighted manner to obtain the attribute prediction value; Step 3: Output the predicted attribute values ​​of the current node.

5. The point cloud attribute encoding method based on graph attention network according to claim 1, characterized in that, The decoding module decodes the encoded attributes, including: Step 1: The decoding end searches for the set of neighboring points of the current node from the set of already decoded and reconstructed points; Step 2: Use the same neural network structure as the encoder to predict the attribute value of the current point and obtain the predicted value of the current point; Step 3: Remove the added points from the decoding result of the last small block; Step 4: Reassemble the decoded blocks back into a complete point cloud.

6. The point cloud attribute encoding method based on graph attention network according to claim 1, characterized in that, The model's loss function is: ； in, The set of ground truth values ​​for the y-components of n points. It is the set of predicted values ​​for the y-components of n points.

Images