Point cloud video stream lossy compression algorithm

Abstract

The invention discloses a lossy compression algorithm for a point cloud video stream, which relates to the technical field of point cloud compression and comprises the following steps of preprocessing, semantic segmentation and label recording, point cloud registration, fine registration, information processing and decoding. According to the method, the correlation between the front frame and the rear frame of the point cloud video stream is fully utilized, the compression effect reaches below 10%, under high graphics card hardware, the time expenditure of a coding part is within 300ms of millions of point cloud, the time expenditure of a decoding part is within 30ms, and through optimization of an algorithm, the compression efficiency can be remarkably improved while the point cloud data quality is kept, and the compression efficiency is improved. The method is suitable for compression and real-time transmission of dynamic point cloud data, redundant information and visually unimportant details in the point cloud data are removed, so that a higher compression ratio is achieved, and the restorability and visual quality of the data are kept.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of point cloud compression, and in particular to a point cloud video stream lossy compression algorithm. BACKGROUND

[0002] Active acquisition equipment represented by laser scanning is increasingly mature in terms of ease of operation, mobility, intelligence and efficiency. Using three-dimensional imaging technology to collect dense point spatial coordinates, color texture and reflectivity information, etc. can reconstruct the three-dimensional entity of the measured target with high fidelity and speed, and plays a very important role in scientific and engineering researches such as engineering surveying, biomedicine, smart city, virtual reality, augmented reality, etc. 3D point cloud has been widely used in many mobile application scenarios, including automatic driving and 3D perception on mobile devices. However, existing 3D point cloud models are often large in size, making them difficult to deploy on edge devices due to their high memory requirements and non-real-time delays. At the same time, with the increasing popularity of the metaverse, the storage and transmission of 3D related data are becoming increasingly important. Researchers now mainly start from computer graphics and digital signal processing, and optimize point cloud compression methods by implementing block operations or combining video coding techniques. The Moving Picture Experts Group issued a call for proposals for point cloud compression in 2017, and released a unified point cloud model compression and coding research framework in 2018. Google's draco compression algorithm greatly reduces the model size. The draco algorithm can be used on gltf / glb models to compress vertex positions, normals, colors, texture coordinates, etc. of the model, thereby improving the efficiency and speed of transmitting 3D content over the network. However, with the complexity of the scene and the increase in data volume, existing technologies cannot well solve the exponentially increasing point cloud data capacity. The existing technologies have the following problems: 3D-HEVC is a 3D coding standard developed by the Video Coding Experts Group, similar to traditional 2D video coding, which adds corresponding depth pictures to 2D images. This application scenario greatly expands the application range of 3D video streams, but it is difficult to handle chaotic 3D point cloud data video streams. Draco is an open source compression library developed by Google's Chrome Media team to improve the storage and transmission of 3D graphics. This algorithm organizes point cloud data through kdtree and achieves relatively good compression effect through entropy coding. However, it does not handle the inter-frame redundant information relationship in point cloud video streams well, and the compression effect is not optimal. SUMMARY

[0003] To solve the above technical problems, the technical scheme adopted by the present application is: A point cloud video stream lossy compression algorithm, comprising the following steps: S1: preprocessing; S2: semantic segmentation and label record; S3: point cloud registration; S4: fine registration; S5: information processing; S6: decoding.

[0004] The further improvement of the technical scheme of the application is that the S1 specifically comprises point cloud noise mixed filtering, statistical-based outlier stripping, edge filtering based on normal vector and Gaussian filtering, so as to obtain a preprocessed point cloud sequence.

[0005] The further improvement of the technical scheme of the application is that the S2 specifically refers to setting the first frame as a KeyFrame, performing semantic segmentation on the point cloud through a RandLA-Net network, and the network structure is shown in the following Figure 3 The first frame and the next frame of point cloud data are divided into many blocks, each point cloud block of the first frame is labeled, the number of point cloud blocks and the size of each point cloud block are recorded, and for each point cloud block Kd tree, Google Draco is used for compression encoding and decoding.

[0006] The further improvement of the technical scheme of the application is that the S3 is to perform point cloud registration between two frames, and the point cloud registration algorithm uses the improved DeepGMR algorithm of the patent as follows Figure 4 and Figure 5 As shown in the 3D Network part, the Pointfeature of the two is fused, the attention mechanism is increased, the proportion of point clouds with inconsistent shapes between the source point cloud block and the target point cloud block can be effectively reduced, the same part is better overlapped, the point cloud sequence obtained in S1 is divided into key frames and predicted frames, Google Draco encoding is performed on the key frames, and binary tree segmentation and block registration are performed according to the relative motion relationship of the reconstructed point cloud between the predicted frames and the key frames.

[0007] The further improvement of the technical scheme of the application is that the S4 refers to fine registration when the next frame point cloud block i and the last frame point cloud block j match to meet the accuracy requirement, find the corresponding point cloud pair, supplement the deleted point cloud data and the increased point cloud data of the current point cloud block, and the next matching is matched with the new point cloud block as a template, and is updated in sequence, and is updated in real time according to the point pair coding RGB difference value.

[0008] The further improvement of the technical scheme of the application is that the S5 stores the block information according to the point cloud matched by S4 in the format as shown in the following Figure 6 The point cloud not matched records the block ID and the size, and is compressed according to Draco encoding and decoding.

[0009] The further improvement of the technical scheme of the present application is that: the S6 is sequentially encoded according to the above-mentioned encoding mode, and the point cloud video stream is sequentially decoded according to the frame information and the block information.

[0010] The further improvement of the technical scheme of the present application is that: the further improvement of the technical scheme of the present application is that: The concept of key frame and setting rule: consistent with the standard video coding H264 and HEVC, a key frame is set every fixed frame number to prevent the increase and spread of inter-frame prediction errors, in addition to the fixed frame interval, the point cloud registration point cloud pair of the current and next two frames is less than 10%, and the current frame is also forced to be a key frame, which cuts off the relationship between the current frame point cloud stream and the previous frame point cloud stream, and re-encodes the current frame point cloud stream; The concept of point cloud block: the point cloud block is to divide the current frame point cloud into different blocks, which is convenient to find the consistent similar blocks of the previous and next frame point clouds, and after point cloud registration, the point cloud pair of the previous frame is replaced to save a lot of storage space, and in theory, when the current and next two frames have the same point cloud block, only one rotation and translation matrix can represent the current information, and due to the inconsistency, the current block cannot be represented by only one rotation and translation matrix, in addition to the matched point cloud pair, there are point clouds that are not matched, which may be the point clouds of the current frame or the previous frame, so the information of the current point cloud block needs to delete the point clouds of the previous frame that are not matched and add the point clouds of this frame that are not matched; Point cloud block division rule: first, the point clouds of the key frame and the next frame are semantically divided according to RandLA-Net, the key frame is divided into m blocks, and the next frame is divided into n blocks, then the m block point cloud data is encoded by Google Draco algorithm, the next frame n block point cloud searches for the matching block in the key frame, the current block that matches is recorded as the matching block ID, RT information and added and deleted point cloud data, and the point cloud block that is not matched is encoded by Google Draco algorithm, and the subsequent non-key frame is sequentially encoded and processed.

[0011] Due to the adoption of the above technical scheme, the present application has the following technical progress compared with the prior art: The present application provides a point cloud video stream lossy compression algorithm, which fully utilizes the correlation between the previous and next frames of the point cloud video stream, and the compression effect reaches below 10%, under the condition of high graphics card hardware, the time cost of the encoding part is within 300ms per million point clouds, and the decoding part is within 30ms, through optimization of the algorithm, the compression efficiency can be significantly improved while maintaining the quality of the point cloud data, which is suitable for dynamic point cloud data compression and real-time transmission, and by removing the redundant information and visually unimportant details in the point cloud data, a higher compression ratio can be achieved, and the data recoverability and visual quality are maintained. BRIEF DESCRIPTION OF DRAWINGS

[0012] Figure 1Flow chart of point cloud video stream lossy compression algorithm of the present application Figure 2 Flow chart of whole point cloud stream encoding of the present application Figure 3 Schematic diagram of network structure of the present application Figure 4 And Figure 5 Schematic diagram of improved DEEPGMR algorithm of the present application Figure 6 Flow chart of point cloud format storage method of the present application DETAILED DESCRIPTION

[0013] The present application will be further described in detail in combination with examples: Example 1

[0014] As shown in the figure, the present application provides a point cloud video stream lossy compression algorithm, which comprises the following steps: Figures 1-6 S1: preprocessing; S2: semantic segmentation and label recording; S3: point cloud registration; S4: fine registration; S5: information processing; S6: decoding. Example 2 As shown in the figure, on the basis of example 1, the present application provides a technical solution: preferably, S1 specifically comprises point cloud noise mixed filtering, statistical-based outlier stripping, normal vector-based edge filtering and Gaussian filtering to obtain a pre-processed point cloud sequence, S2 specifically refers to setting the first frame as KeyFrame, performing semantic segmentation on the point cloud through RandLA-Net network, and the network structure is as follows

[0015] Figures 1-6 The first frame and the next frame of point cloud data are divided into many blocks, each point cloud block of the first frame is labeled, the number of point cloud blocks and the size of each point cloud block are recorded, and for each point cloud block Kd tree, Google Draco is used for compression encoding and decoding; Figure 3 S3 is to perform point cloud registration between the two frames, and the point cloud registration algorithm uses the improved DeepGMR algorithm of the present application as follows Figure 4 And Figure 5 ​​As shown, in the 3D Network part, the Pointfeatures of the two are fused and an attention mechanism is added, which can effectively reduce the proportion of point clouds with inconsistent shapes between the source point cloud blocks and the target point cloud blocks, and make the same parts overlap better. The point cloud sequence obtained in S1 is divided into key frames and prediction frames. The key frames are encoded by Google Draco, and the relative motion relationship in the point cloud is reconstructed based on the prediction frames and key frames to perform binary tree segmentation and block registration. Preprocessing is performed using a point cloud noise mixing and filtering method to reduce noise and unnecessary details in the data, thereby improving the efficiency and quality of subsequent encoding and decoding. Video coding technology is used to convert 3D point cloud data into 2D images for processing. Google Draco coding is used to achieve better inter-frame prediction, improve the correlation between consecutive frames, and further optimize the compression effect. S4 refers to the process where, if the point cloud block i in the next frame matches the point cloud block j in the previous frame to meet the accuracy requirements, fine registration is performed to find the corresponding point cloud pair, and the deleted and added point cloud data of the current point cloud block are added. The next match is performed using the new point cloud block as a template, and the process is updated sequentially. At the same time, the process is updated in real time based on the RGB difference of the point pair encoding. S5 is based on the point cloud matched by S4, as follows: Figure 6 The format shown stores block information, including the ID and size of unmatched point cloud record blocks, compressed according to Draco encoding and decoding. S6 encodes the point cloud video stream sequentially according to the above encoding method, and decodes it sequentially based on frame information and block information.

[0016] In this embodiment, the correlation between consecutive frames of the point cloud video stream is fully utilized, achieving a compression effect of less than 10%. Under high-end graphics card hardware, the encoding time for one million point clouds is within 300ms, and the decoding time is within 30ms. Through algorithm optimization, compression efficiency can be significantly improved while maintaining the quality of point cloud data. It is suitable for the compression and real-time transmission of dynamic point cloud data. By removing redundant information and visually unimportant details from the point cloud data, a higher compression ratio is achieved while maintaining data recoverability and visual quality. Example 3

[0017] like Figures 1-6 As shown, based on Embodiment 1, the present invention provides a technical solution, preferably including the following concepts: The concept and setting rules of keyframes: Consistent with standard video codecs H.264 and HEVC, a keyframe is set every fixed number of frames to prevent the increase and spread of inter-frame prediction errors. In addition to the fixed frame interval, this algorithm also forces the current frame to be set as a keyframe when the point cloud pair between the current and next frames is less than 10%, cuts off the relationship between the current frame point cloud stream and the previous frame point cloud stream, and re-encodes the current frame point cloud stream. The concept of point cloud block: the point cloud block is to divide the current frame point cloud into different blocks, which is convenient to find the consistent similar blocks of the front and rear frame point clouds. After point cloud registration, the point cloud of the previous frame is replaced, which saves a lot of storage space. In theory, when the current and rear two frames have the same point cloud block, only a rotation and translation matrix can represent the current information. Since it is not completely consistent, the current block cannot be represented by only a rotation and translation matrix. In addition to the matched point cloud pairs, there are point clouds that are not matched. These point clouds may be the point clouds of the current frame or the previous frame. Therefore, the information of the current point cloud block needs to delete the point clouds of the previous frame that are not matched and add the point clouds of the current frame that are not matched. Point cloud block division rule: first, the point clouds of the key frame and the next frame are divided according to RandLA-Net. The key frame is divided into m blocks, and the next frame is divided into n blocks. Then, the m block point cloud data is encoded by Google Draco algorithm. The n block point cloud of the next frame finds the matching block in the key frame. The matching current block records the matching block ID, RT information, and added and deleted point cloud data. The unmatched point cloud block is encoded by Google Draco algorithm. The non-key frame is sequentially encoded and processed.

[0018] The above describes the present application in detail, but some modifications or improvements can be made on the basis of the present application, which is obvious to those skilled in the art. Therefore, the modifications or improvements without departing from the spirit of the present application are within the scope of the present application.

Claims

1. A lossy compression algorithm for point cloud video streams, characterized in that: Includes the following steps: S1: Preprocessing; S2: Semantic segmentation and labeling; S3: Point cloud registration; S4: Perform fine registration; S5: Information Processing; S6: Decoding.

2. The point cloud video stream lossy compression algorithm according to claim 1, characterized in that: S1 specifically includes point cloud noise mixing and filtering, statistical noise stripping, normal vector-based edge filtering, and Gaussian filtering to obtain a preprocessed point cloud sequence.

3. The point cloud video stream lossy compression algorithm according to claim 1, characterized in that: Specifically, S2 refers to setting the first frame as the KeyFrame, performing semantic segmentation of the point cloud through the RandLA-Net network. The network structure is shown in Figure 3 below. The point cloud data of the first frame and the next frame are divided into many blocks. Each point cloud block in the first frame is labeled. The number of point cloud blocks and the size of each point cloud block are recorded. For the Kd tree of each point cloud block, Google Draco is used for compression encoding and decoding.

4. The point cloud video stream lossy compression algorithm according to claim 1, characterized in that: S3 involves point cloud registration between the two frames. The point cloud registration algorithm uses the improved DeepGMR algorithm of this patent, as shown in Figures 4 and 5. In the 3D Network part, the point features of the two are fused and an attention mechanism is added, which can effectively reduce the proportion of point clouds with inconsistent shapes between the source point cloud block and the target point cloud block, and make the same parts overlap better. The point cloud sequence obtained in S1 is divided into key frames and prediction frames. The key frames are encoded with Google Draco, and the relative motion relationship in the point cloud is reconstructed based on the prediction frame and the key frame to perform binary tree segmentation and block registration.

5. The point cloud video stream lossy compression algorithm according to claim 1, characterized in that: S4 refers to the process of performing fine registration if the next frame point cloud block i and the previous frame point cloud block j match to meet the accuracy requirements, finding the corresponding point cloud pair, supplementing the deleted point cloud data and added point cloud data of the current point cloud block, and using the new point cloud block as the template for the next matching, updating sequentially, and updating in real time according to the RGB difference of the point pair encoding.

6. The point cloud video stream lossy compression algorithm according to claim 1, characterized in that: S5 stores block information based on the point cloud matched in S4 according to the format shown in Figure 6 below. The unmatched point cloud records the block ID and size, and compresses it according to Draco encoding and decoding.

7. The point cloud video stream lossy compression algorithm according to claim 1, characterized in that: S6 involves encoding the point cloud video stream sequentially according to the above encoding method, and decoding sequentially based on frame information and block information.

8. A lossy compression algorithm for point cloud video streams according to claim 1, comprising the following concepts, characterized in that: The concept and setting rules of keyframes: Consistent with standard video codecs H.264 and HEVC, a keyframe is set every fixed number of frames to prevent the increase and spread of inter-frame prediction errors. In addition to the fixed frame interval, this algorithm also forces the current frame to be set as a keyframe when the point cloud pair between the current and next frames is less than 10%, cuts off the relationship between the current frame point cloud stream and the previous frame point cloud stream, and re-encodes the current frame point cloud stream. The concept of point cloud blocks: Point cloud blocks divide the current frame's point cloud into different blocks, making it easier to find consistent similar blocks in the point clouds of previous and subsequent frames. After point cloud registration, the point cloud pairs from the previous frame are used to replace them, saving a lot of storage space. Theoretically, when there are identical point cloud blocks in the current and subsequent frames, only one rotation and translation matrix is ​​needed to represent the current information. However, since they are not completely consistent, the current block cannot be represented by only one rotation and translation matrix. In addition to the matched point cloud pairs, there are unmatched point clouds. These point clouds may be from the current frame or the previous frame. Therefore, the information of the current point cloud block needs to delete the unmatched point clouds from the previous frame and add the unmatched point clouds from this frame. Point cloud block partitioning rules: First, based on RandLA-Net, the point clouds of the keyframe and the next frame are semantically partitioned. The keyframe is divided into m blocks, and the next frame is divided into n blocks. The point cloud data of the m blocks are encoded by the Google Draco algorithm. The n blocks of the next frame search for matching blocks in the keyframe. The matching block ID, RT information, and added or deleted point cloud data are recorded for the current matching block. The unmatched point cloud blocks are encoded by the Google Draco algorithm. Subsequent non-keyframes are encoded in the same way.

Images