Dynamic point cloud efficient compression method and system based on feature anchor points, electronic equipment and medium

Through the dynamic point cloud compression method based on feature anchor points, deep feature information of point cloud data is extracted and processed, inter-frame motion information is predicted and efficiently compressed, and the problem of insufficient compression rate and accuracy in the prior art is solved, and the problem of insufficient compression rate and accuracy is realized. More efficient point cloud data compression and reconstruction.

CN118982555APending Publication Date: 2024-11-19ZHEJIANG GONGSHANG UNIVERSITY

Patent Information

Application Number
CN202411013311.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-26
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

When the existing dynamic point cloud compression method deals with complex geometric characteristics and time correlations, it is difficult to fully extract and utilize the deep feature information of the data, resulting in the impact of compression rate and accuracy after decompression.

Method used

Using dynamic point cloud compression method based on feature anchors, potential feature information is generated through feature extraction modules, representative feature anchor generation modules select representative feature anchors, scene flow prediction modules predict inter-frame motion information, and motion enhancement modules supplement fine-grained details , the motion compensation and residual compression modules are efficiently compressed and rebuilt.

Benefits of technology

It improves compression rate and reconstruction accuracy, reduces data storage, avoids spatial relationship damage caused by motion prediction in feature space, and significantly improves compression performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118982555A_ABST
    Figure CN118982555A_ABST
Patent Text Reader

Abstract

The invention discloses a dynamic point cloud efficient compression method and system based on feature anchor points, electronic equipment and a medium. According to the method, the feature anchor points are used for representing the motion information between the two frames, only a small number of anchor points need to be stored during coding storage, complete motion information can be recovered during decoding, complete motion data do not need to be stored, the data storage amount is reduced, and the compression rate is increased. Meanwhile, the scene flow network is adopted to calculate the coordinate change of the real space of the two frames, so that the spatial relationship damage possibly caused by motion prediction in the feature space is avoided, the accuracy of motion prediction is improved, and the reconstruction precision is further improved. Generally speaking, according to the method, the compression rate is improved, and meanwhile, the reconstruction precision is also remarkably improved, so that the compression performance is remarkably improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of three-dimensional point cloud data processing, and in particular relates to a method, system, electronic equipment and medium for efficiently compressing dynamic point clouds based on feature anchor points. Background Art

[0002] As an important achievement of 3D scanning technology, dynamic point cloud has shown broad application prospects in many fields such as film and television production, virtual reality, autonomous driving, and cultural relics protection because it can provide denser and more realistic spatial and temporal representations. However, the huge amount of dynamic point cloud data poses great challenges to its storage and transmission. Therefore, in order to more widely apply dynamic point cloud data in various industries, it is necessary to effectively compress it.

[0003] Traditional dynamic point cloud compression methods, such as the V-PCC standard developed by MPEG, effectively improve compression efficiency through projection-based methods. However, these methods are still unable to cope with complex geometric characteristics and time correlations, and it is difficult to fully extract and utilize the deep feature information of the data.

[0004] In recent years, dynamic point cloud compression methods based on deep learning have rapidly emerged, using the powerful feature extraction and expression capabilities of deep learning models to significantly improve compression performance. These methods usually use the CNN autoencoder architecture to encode and reconstruct point clouds, but directly applying CNN on large-scale point cloud data will face the problem of high time and space complexity. To this end, researchers have introduced techniques such as sparse convolution to reduce computational complexity and reduce data size through downsampling operations. At the same time, the ME / MC (motion estimation and compensation) structure is used for motion estimation and compensation, and the motion information between two adjacent frames is used to optimize the encoding to reduce the redundant information of dynamic point clouds in time series and further improve compression performance.

[0005] However, existing methods still have limitations. First, the commonly used downsampling technology is irrelevant to the point cloud data structure and compression task. The sampled points are not representative, and directly using these points to calculate motion information will generate a lot of unnecessary calculations and data. In addition, directly processing motion information in the feature space will destroy the spatial consistency of the point cloud motion in the physical space and generate erroneous information, which will be accumulated in the entire sequence through continuous inter-frame prediction. The above problems will affect the final compression rate and accuracy after decompression. Summary of the invention

[0006] In order to solve the above-mentioned problems, the present invention proposes a dynamic point cloud compression method, system, electronic device and medium based on feature anchor points.

[0007] On the one hand, the present invention provides a dynamic point cloud efficient compression system based on feature anchor points, comprising:

[0008] The feature extraction module is used to receive the point cloud data of the frame, perform feature encoding, and generate potential feature information.

[0009] The feature anchor generation module is used to receive input from the feature extraction module, extract global features, calculate the similarity score between the local features of each point and the global features, and select the point with the highest score as the feature anchor.

[0010] The scene flow prediction module uses a self-supervised learning network to predict the scene flow between anchor points to approximate the motion information between frames.

[0011] The motion enhancement module introduces reference frame feature information to supplement the fine-grained details of the scene flow.

[0012] Motion compensation and residual compression module: Use the enhanced scene flow information to perform motion compensation, generate prediction frames, and efficiently compress the residual through a deep entropy model.

[0013] Point cloud reconstruction module: The decompressed residual is merged into the predicted frame to obtain the potential reconstructed frame, and the final high-quality reconstructed point cloud frame is generated by upsampling.

[0014] On the other hand, the present invention provides a method for efficiently compressing dynamic point clouds based on feature anchor points, the method comprising the following steps:

[0015] Step 1: Given two frames of continuous point cloud data, the reference frame and the current frame, extract features from these two frames.

[0016] Step 2: Use the feature anchor point generation module based on geometric feature comparison to generate the feature anchor point (anchor point for short) that best represents the geometric characteristics for each frame, and compress and save the anchor point.

[0017] Step 3: Use the scene flow prediction network to calculate the coordinate change between the reference frame anchor point and the current frame anchor point, thereby obtaining the coordinate change information between the anchor points.

[0018] Step 4: The coordinate change information between the anchor points is input into the motion enhancement module to generate the motion vector between the reference frame and the current frame.

[0019] Step 5: Use the motion vector obtained in step 4 and the reference frame feature information extracted in step 1 to perform motion compensation to predict the current frame.

[0020] Step 6: Compare the current frame prediction obtained in step 5 with the actual data of the current frame obtained in step 1, calculate the residual, and compress and save the residual through entropy coding technology to complete the compression process.

[0021] Step 7: During decompression, first predict the current frame based on the feature information of the reference frame and the saved anchor points, then add the saved current frame residual to the predicted frame, and finally input it into the point cloud reconstruction module to reconstruct the complete current frame.

[0022] On the other hand, the present invention provides an electronic device for an efficient compression method of dynamic point clouds based on feature anchor points, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the contents of any of the above methods when executing the computer program.

[0023] On the other hand, the present invention provides a non-transitory computer-readable storage medium, comprising instructions, which, when executed on a computer, enable the computer to execute a dynamic point cloud efficient compression method based on feature anchor points as described in any one of the above.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] The present invention uses feature anchor points to represent the motion information between two frames, thereby realizing that only a small number of anchor points need to be saved during encoding and saving, and the complete motion information can be restored during decoding without saving the complete motion data, thereby reducing the amount of data storage and improving the compression rate. At the same time, the present invention uses a scene flow network to calculate the coordinate changes of the two frames in real space, avoiding the destruction of spatial relationships that may be caused by predicting motion in the feature space, thereby improving the accuracy of motion prediction and further improving the reconstruction accuracy. In general, the present invention significantly improves the reconstruction accuracy while improving the compression rate, thereby significantly improving the compression performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 : A dynamic point cloud compression framework diagram based on feature anchor points in an embodiment of the present application.

[0027] Figure 2 : Framework diagram of the anchor point generation module in the embodiment of the present application.

[0028] Figure 3 : Framework diagram of the motion enhancement module and related modules of the embodiment of the present application.

[0029] Figure 4 : Comparison diagram of the reconstruction effect visualization of the embodiment of the present application and the existing model.

[0030] Figure 5 : Comparison chart of D1-PSNR and D2-PSNR of the embodiment of the present application and the existing model under different BPPs. DETAILED DESCRIPTION

[0031] The present invention will be described in detail below with reference to the accompanying drawings and preferred examples, and the purpose and effects of the present invention will become more clear. It should be understood that the specific examples described herein are only used to explain the present invention and are not intended to limit the present invention.

[0032] like Figure 1 As shown, the present application provides a dynamic point cloud compression system based on feature anchor points, the system comprising:

[0033] Feature extraction module: This module receives the point cloud data of the frame, performs feature encoding, and generates potential feature information, aiming to enhance feature representation while improving data processing efficiency.

[0034] Feature anchor generation module: sparse convolution and global pooling techniques are used to extract global features, and the similarity score between the local features and the global features of each point is calculated. The point with the highest score is selected as the feature anchor. The specific framework is as follows: Figure 2 shown.

[0035] Scene flow prediction module: A self-supervised learning network is used to predict the scene flow between anchor points to approximate the motion information between frames.

[0036] Motion enhancement module: It introduces reference frame feature information and supplements the fine-grained details of the scene flow and improves the accuracy of motion information by combining the cross-attention and convolutional neural network methods.

[0037] Motion compensation and residual compression module: Use the enhanced scene flow information to perform motion compensation, generate prediction frames, and efficiently compress the residual through a deep entropy model to further improve the compression rate and reconstruction quality.

[0038] Point cloud reconstruction module: The decompressed residual is merged into the predicted frame to obtain the potential reconstructed frame, and the final high-quality reconstructed point cloud frame is generated through techniques such as upsampling layers.

[0039] Furthermore, the system uses sparse convolution and global pooling techniques to extract global features.

[0040] Furthermore, the system complements the fine-grained details of the scene flow by combining cross-attention and convolutional neural networks.

[0041] It should be noted that this application is applicable to all point cloud sequence data in .PLY format and .NPY format.

[0042] In a preferred embodiment, the present application uses four human point cloud sequences in the 8iVFB dataset during training, totaling 1200 frames; and uses four human sequences in the Owlii dataset during testing, taking the first 100 frames of each sequence, totaling 400 frames.

[0043] In a preferred embodiment, the equipment and parameters used in this application are specifically:

[0044] CPU: Intel 13900k; GPU: vidia RTX 4090; Memory: 64G; Hard disk: at least 200G space; Operating system: Ubuntu 22.04.

[0045] Parameter setting: The Adam optimizer is used for optimization, and the β parameter is set to (0.9, 0.999). The learning rate is reduced to 70% of its value every 15 epochs. During the training process, the batch size is set to 4, the initial learning rate is 0.0008, and the training is performed for 200 rounds.

[0046] Based on the above system, in a preferred embodiment, the present application also provides a specific process of a dynamic point cloud efficient compression method based on feature anchor points, the steps comprising:

[0047] Step 1: Given two consecutive frames of point cloud data, the previous frame is used as the reference frame. The next frame is the current frame x t , extract features from these two frames.

[0048] In particular, when preparing point cloud data that needs to be compressed, it should be noted that the data is a point cloud sequence composed of continuous point cloud files (in .PLY or .NPY format), and each point cloud file is a frame. Data preprocessing is performed on all frames. The coordinates of all frames are first standardized to ensure that the scaling ratios of all frames are consistent; then all frames are voxelized, and the voxel size is 0.03 (which can also be adjusted according to the data). During the encoding process, the point cloud sequence is encoded in sequence, and two adjacent frames are encoded each time.

[0049] Step 2: Reference Frame and the current frame x t First, it is initialized as a sparse tensor and in and A coordinate matrix representing a point cloud. and Represents the relevant feature matrix, uses an all-one vector to represent voxel occupancy, and the other network layers of the embodiment of the present application are also implemented by MinkowskiEngine, and the parameters are all sparse tensors consisting of coordinates and features.

[0050] Then, two DownSample blocks composed of MinkowskiConvolution are used to extract features and obtain the potential representation of the input frame. and

[0051] Step 3: Represent the feature information and t They are input into the feature anchor point generation module respectively. Taking the current frame as an example, in the feature anchor point generation module, the input feature information y t The first copy is copied into two identical copies. The first copy passes through three convolutional layers and a global pooling layer to obtain the global feature g of the frame. t , and the other extracts the local features of each point through a multi-layer perceptron and two edge convolutional layers ti The scorer in the feature anchor point generation module scores each point based on global features and local features.

[0052] Formula (1) is used to calculate the feature score of each point:

[0053]

[0054] In which, assume that point i is y t For a point in , the feature score of point i is defined as s i , where D represents the number of dimensions of the feature, and sig represents the sigmoid function. This function converts the local feature l of each point tij (i.e. l t The feature of the midpoint i in the jth dimension) and the corresponding global feature g tj (i.e. g t The difference between the features of the jth dimension in the jth dimension is mapped to [0, 1]. The more similar the local features of a point are to the global features, the higher the score s i The larger the current frame x is, the t After calculating the feature scores of all points, select the K points with the highest scores (K=1280 in this embodiment of the application) as the current frame x t The anchor point k t , the anchor point only contains coordinate information, and the data shape is [K,3].

[0055] After the above calculations, the anchor points of the reference frame and the current frame are obtained. and k t , save the anchor code.

[0056] Step 4: Use SCOOP to self-supervise scene flow prediction network calculation and k t The scene flow between k , f k Represents the coordinate change information between anchor points.

[0057] Step 5: Figure 3 As shown, the obtained scene flow information f kInput to the motion enhancement module and copied into two copies, the first copy and the reference frame feature information The two processed data are concatenated and then processed through a linear layer to obtain the motion vector f between the two frames. y , f y The resolution and Consistent.

[0058] Step 6: Use the motion vector f y And the feature information of the reference frame Processed by the motion compensation module, the potential features of the current frame are predicted

[0059] Step 7: The current frame potential features With the feature information y of the current frame t Subtract and get the residual r t , the residual is compressed and saved through the deep entropy model to complete the encoding process of the current frame.

[0060] Step 8: After encoding each frame, the anchor point k of each frame is t And the compressed residual r t Save, and compress the first frame of the point cloud sequence separately using the static compression method PCGCv2, thus completing the compression process of the entire point cloud sequence.

[0061] Step 9: During the decompression process, the first frame is first decompressed independently. Each subsequent frame will use the information of the previous frame, and so on, to decompress the entire point cloud sequence in sequence. For each frame, the residual decompression is first performed to obtain the decompressed residual Then the potential features of the frame are calculated through the trained network model Will and Add together to get the predicted potential frame y t ′ Next, the decoder performs upsampling through three sparse transposed convolutional layers with a stride of 2, and uses sparse convolutional layers to generate the occupancy probability of each voxel. Finally, the reconstructed frame is obtained. Complete decompression of the frame.

[0062] In particular, in a preferred embodiment, in order to verify the effectiveness of the present application, the 8iVFB dataset and the Owlii dataset were used for verification and compared with other advanced methods under the same conditions.

[0063] The 8iVFB dataset is used as training data, which contains four sequences, namely longdress, loot, redandblack and soldier. Each sequence contains 10 seconds of human movement and is captured by a camera at a rate of 30fps. The spatial resolution of each frame point cloud is 1024*1024*1024, containing points ranging from 700,000 to 1.1 million.

[0064] The Owlii dataset is used as test data, which has four sequences, namely Basketballplayer, Dancer, Exercise and Model. The spatial resolution of each frame point cloud is 1024*1024*1024, containing points ranging from 600,000 to 800,000. The first 100 frames of each sequence are taken.

[0065] The 8iVFB data set used in this preferred embodiment can be specifically http: / / plenodb.jpeg.org / pc / 8ilabs The Owlii dataset can be downloaded from the address of https: / / mpeg-pcc.org / index.php / pcc-content-database / owlii-dynamic-human-textured -mesh-sequence-dataset / .

[0066] The results of this preferred embodiment are compared with those of VPCC, D-DPCC and Akhtar's methods using the same data set. Specifically, the results are evaluated using three indicators: D1-PSNR, D2-PSNR and BD-Rate. These three indicators are widely used in the evaluation of point cloud compression.

[0067] Figure 5 The comparison of the rate-distortion curves of the preferred embodiment and other methods is shown. The horizontal axis of the curve represents the number of bits occupied by each point (Bits Per Point), i.e., BPP. The smaller this index is, the higher the compression rate is. The horizontal axis represents the peak signal-to-noise ratio (PSNR). The higher the value is, the better the reconstruction quality is.

[0068] For a comprehensive comparison, the peak signal-to-noise ratio based on point-to-point distance (D1-PSNR) and the peak signal-to-noise ratio based on point-to-plane distance (D2-PSNR) are used to measure the distortion between the original frame and the decoded frame. Different colors in the figure represent different methods.

[0069] according to Figure 5 As shown in the figure, our method outperforms the existing advanced methods in all four sequences and has higher D1-PSNR and D2-PSNR under the same BPP.

[0070] Table 1 BD rate gain table of this preferred embodiment

[0071]

[0072] Table 1 summarizes the BD rate gain of the preferred embodiment. Compared with D-DPCC, the present application has an average improvement of 11.89% (D1) and 11.85% (D2). In addition, compared with V-PCC, the present application has an average improvement of 82.25% (D1) and 79.04% (D2).

[0073] The results show that the proposed method generally outperforms existing methods, especially in datasets with intense motion. For example, in the "Dancer" sequence, the coding efficiency significantly exceeds that of DDPCC.

[0074] In addition to leading indicators, Figure 4 A visual comparison of frame reconstruction between D-DPCC and our method is also shown. Taking the 13th frame of the Basketballplayer point cloud sequence as an example, the reconstruction quality of our method is significantly better than that of D-DPCC at the same compression rate.

[0075] Those skilled in the art can understand that the above are only preferred examples of the invention and are not intended to limit the invention. Although the invention is described in detail with reference to the above examples, those skilled in the art can still modify the technical solutions recorded in the above examples or replace some of the technical features therein with equivalents. Any modification, equivalent replacement, etc. made within the spirit and principle of the invention shall be included in the protection scope of the invention.

Claims

1. An efficient compression system for dynamic point clouds based on feature anchor points. It is characterized in that include: The feature extraction module is used to receive the point cloud data of the frame, perform feature encoding, and generate potential feature information; The feature anchor point generation module is used to receive the input from the feature extraction module, extract the global feature, calculate the similarity score between the local feature of each point and the global feature, and select the point with the highest score as the feature anchor point; The scene flow prediction module uses a self-supervised learning network to predict the scene flow between anchor points to approximate the motion information between frames; The motion enhancement module introduces reference frame feature information to supplement the fine-grained details of the scene flow; Motion compensation and residual compression module: Use the enhanced scene flow information to perform motion compensation, generate prediction frames, and efficiently compress the residual through a deep entropy model; Point cloud reconstruction module: The decompressed residual is merged into the predicted frame to obtain the potential reconstructed frame, and the final high-quality reconstructed point cloud frame is generated by upsampling.

2. According to claim 1, a dynamic point cloud efficient compression system based on feature anchor points, It is characterized in that Sparse convolution and global pooling techniques are used to extract global features.

3. According to claim 1, a dynamic point cloud efficient compression system based on feature anchor points, It is characterized in that The fine-grained details of the scene flow are supplemented by combining criss-cross attention and convolutional neural networks.

4. An efficient compression method for dynamic point clouds based on feature anchor points. Features The method comprises the following steps: Step 1: Given two frames of continuous point cloud data, the reference frame and the current frame, extract features from these two frames; Step 2: Use the feature anchor point generation module based on geometric feature comparison to generate the feature anchor point that best represents the geometric characteristics for each frame, and compress and save the anchor point; Step 3: Use the scene flow prediction network to calculate the coordinate change between the reference frame anchor point and the current frame anchor point, thereby obtaining the coordinate change information between the anchor points; Step 4: Input the coordinate change information between the anchor points into the motion enhancement module to generate the motion vector between the reference frame and the current frame; Step 5: Use the motion vector obtained in step 4 and the reference frame feature information extracted in step 1 to perform motion compensation to predict the current frame; Step 6: Compare the current frame obtained in step 5 with the actual data of the current frame obtained in step 1, calculate the residual, compress and save the residual, and complete the compression process; Step 7: Predict the current frame based on the feature information of the reference frame and the saved anchor points, add the saved current frame residual to the predicted frame, and finally input it into the point cloud reconstruction module to reconstruct the complete current frame.

5. According to the method of claim 4, a dynamic point cloud efficient compression method based on feature anchor points, Features: Before step 1, data preprocessing of all frames is also included, specifically: First, standardize the coordinates of all frames to ensure that the scaling of all frames is consistent; All frames are then voxelized.

6. According to claim 4, a dynamic point cloud efficient compression method based on feature anchor points, Features: In step 2, the input feature information is copied into two identical copies; the first copy obtains the global features through three convolutional layers and one global pooling layer, and the other copy extracts the local features of each point through a multi-layer perceptron and two edge convolutional layers; the scorer in the feature anchor point generation module calculates the score for each point based on the global features and local features; after completing the feature score calculation for all points in the current frame, select several points with the highest scores as the anchor points of the current frame.

7. The method for efficiently compressing dynamic point clouds based on feature anchor points according to claim 6, Features: The scoring device calculates the score using the following formula: The feature score of point i is defined as s i , D represents the number of dimensions of the feature, sig represents the sigmoid function, which converts the local feature l of each point tij and the corresponding global feature g tj The difference between them is mapped to [0, 1], w j Represents the global feature g tj The weight of .

8. According to claim 4, a method for efficiently compressing dynamic point clouds based on feature anchor points, Features: Step 4 is as follows: The obtained scene flow information is input into the motion enhancement module and copied into two copies. The first copy is processed together with the reference frame feature information through the cross-attention layer, and the second copy is processed using the cross-layer convolution module. The two processed data are spliced ​​together and then processed through a linear layer to obtain the motion vector between the two frames.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, an efficient compression method for dynamic point clouds based on feature anchor points as described in any one of claims 4 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, an efficient compression method for dynamic point clouds based on feature anchor points as described in any one of claims 4 to 7 is implemented.

Citation Information

Patent Citations

  • Dynamic point cloud geometric compression method based on scene flow network and time entropy model

    CN114025146A

  • An apparatus and a method for performing a data driven pairwise registration of three-dimensional point clouds

    EP3693922A1

  • Method and apparatus for estimating pose of device

    EP4057226A1

Cited By

  • Dynamic point cloud compression method and device based on time sequence perception

    CN119379821A