Dynamic point cloud fast compression method and system based on orb feature extraction and imu motion estimation

Abstract

The application discloses a dynamic point cloud fast compression method and system based on ORB feature extraction and IMU motion estimation, and comprises the following steps: acquiring and extracting ORB feature points of a 2D RGB image, mapping the ORB feature points to a 3D space, acquiring a camera relative pose estimated by an IMU, calculating a relative pose of key points of two frames of point clouds, encoding a transformation matrix into data Ppart, transforming a predicted frame to a key frame space based on the transformation matrix to obtain data Pcloud*, dividing the data Pcloud* into a plurality of macroblocks, traversing each macroblock of the Pcloud*, calculating the similarity of two point cloud macroblocks, determining whether the similarity of the two point cloud macroblocks satisfies a similarity threshold condition, encoding a Key of a key frame macroblock into the data Ppart or encoding data of a predicted frame macroblock into data Ipart. The application can effectively improve the compression efficiency of dynamic point clouds.

Description

Technical Field

[0001] The present invention relates to the field of data transmission and rendering technology, and in particular to a method and system for fast compression of dynamic point clouds based on ORB feature extraction and IMU motion estimation. Background Art

[0002] As the robot operating environment becomes more complex, emerging technologies such as augmented reality and mixed reality have emerged in teleoperation systems, allowing users to perform remote operations in an immersive environment, effectively improving user operation efficiency and safety. Among them, the implementation of mixed reality interaction systems relies on 3D point cloud data compression and transmission technology and environment reconstruction and rendering technologies.

[0003] Currently, in dynamic point cloud compression algorithms, inter-frame coordinate system alignment is basically achieved through 3D space calculations, such as directly using all the original data of two frames of point clouds, or extracting and matching features in 3D space to obtain paired points. However, the current 3D-based dynamic point cloud compression methods are computationally intensive, and the most time-consuming step is inter-frame motion vector estimation. A commonly used method for inter-frame motion vector estimation is to use a point cloud registration method, such as calculating the coordinate system transformation matrix of the current frame point cloud and the key frame point cloud based on the iterative closest point algorithm (ICP). When the correspondence between each point in the two frames of point clouds is not obtained, ICP can only be solved iteratively through nonlinear optimization methods. This solution method is computationally intensive and difficult to deploy on a robot teleoperation system.

[0004] In existing point cloud data processing, such as the method and system for rapidly determining inter-frame coding modes for dynamic 3D point cloud compression (patent publication number CN115499660A), this method determines whether geometric and texture features meet preset conditions. If these conditions are met, the remaining inter-frame prediction mode calculations are skipped, thereby reducing the computational complexity of the encoding process. However, this method can only skip the inter-frame motion estimation step in certain specific scenarios and cannot truly reduce the time consumption of inter-frame motion estimation.

[0005] For example, patent publication number CN112153381A discloses a method, device, and medium for rapidly partitioning CUs within a dynamic 3D point cloud compression frame. This method determines the frame type, extracts the target coded CU, and obtains its original luminance pixel values ​​for transformation. If the target coded CU has a width and height of 64 and meets a first pre-set condition, or a width and height of 32 and meets a second pre-set condition, or a width and height of 16 and meets a third pre-set condition, further partitioning is stopped. This method can effectively improve the CU partitioning speed of dynamic 3D point clouds, but it still does not solve the problem of long motion estimation time. Summary of the Invention

[0006] In order to overcome the defects and shortcomings of the existing technology, the present invention provides a dynamic point cloud fast compression method and system based on ORB feature extraction and IMU motion estimation. The present invention reduces the number of points involved in the calculation by extracting and mapping effective ORB feature points, and predicts the change in the robot posture through IMU as the initial value of the ICP algorithm, thereby reducing the calculation time of the ICP algorithm, and performs predictive frame encoding through macroblock matching, thereby improving the motion estimation efficiency of the point cloud sequence and reducing the temporal redundancy of the point cloud.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] The present invention provides a method for fast compression of dynamic point clouds based on ORB feature extraction and IMU motion estimation, comprising the following steps:

[0009] Obtain keyframes and predicted frames of 3D point clouds, 2D RGB images synchronized with the keyframes and predicted frame point clouds, and IMU pose estimation results;

[0010] Extract ORB feature points of I-frame and P-frame 2D RGB images;

[0011] Set a threshold condition. If the number of ORB feature points of the two frames does not meet the threshold condition, they are compressed as key frames. If the threshold condition is met, the ORB feature points are mapped to 3D space.

[0012] Get the relative pose of the camera estimated by the IMU and set it as the initial pose of the ICP algorithm, and set the convergence condition of the ICP algorithm;

[0013] The ICP algorithm is used to calculate the relative pose of the key points of the two frames of point cloud in 3D space. If the calculation result of the ICP algorithm is determined to meet the convergence conditions, the transformation matrix T of the I frame and the P frame is output. If the calculation result of the ICP algorithm is determined to not meet the convergence conditions, the predicted frame is compressed as a key frame;

[0014] The transformation matrix T of the I frame and the P frame is encoded into the data Ppart, and the predicted frame is transformed into the key frame space based on the transformation matrix T to obtain the data Pcloud*;

[0015] Divide the data Pcloud* into multiple macroblocks, traverse each macroblock of Pcloud*, and find whether the corresponding macroblock exists in the keyframe. If so, calculate the similarity of the two point cloud macroblocks;

[0016] A similarity threshold condition is set. If it is determined that the similarity between the two point cloud macroblocks meets the similarity threshold condition, the key of the keyframe macroblock is encoded into data Ppart, where the data Ppart is the data predicted using the keyframe in the predicted frame;

[0017] If it is determined that the similarity between the two point cloud macroblocks does not meet the similarity threshold condition, or there is no corresponding macroblock, the data of the predicted frame macroblock is encoded into data Ipart, and the data Ipart is data in the predicted frame that cannot be predicted using the key frame.

[0018] As a preferred technical solution, the OFSAT algorithm is used to extract ORB feature points of I-frame and P-frame 2D RGB images;

[0019] And / or, obtaining the camera relative pose estimated by the IMU based on the Kalman filter.

[0020] As a preferred technical solution, the predicted frame is transformed into the key frame space based on the transformation matrix T to obtain data Pcloud*, which is specifically expressed as:

[0021] PCloud * =T·Pcloud

[0022] Among them, Pcloud represents the predicted frame.

[0023] As a preferred technical solution, the similarity of two point cloud macroblocks is calculated by the number of point clouds and centroid deviation in the macroblocks, which can be specifically expressed as:

[0024]

[0025] Among them, N PM and N IM are the number of points in the data Pcloud* macroblock and the corresponding keyframe macroblock, C PM and C IM is the centroid of the corresponding macroblock, and ρ is the Euclidean distance.

[0026] As a preferred technical solution, an octree is used to store data Pcloud*. The macroblock position is described as an octree index, represented by Key (x; y; z), where (x; y; z) is the spatial coordinate of the octree leaf node.

[0027] The present invention also provides a dynamic point cloud fast compression system based on ORB feature extraction and IMU motion estimation, comprising: a data acquisition module, an ORB feature extraction module, a threshold condition judgment module, an IMU motion estimation module, a relative posture calculation module, an ICP convergence condition judgment module, a transformation matrix encoding module, a transformation module, a macroblock segmentation module, a macroblock traversal module, a similarity threshold condition judgment module, and an encoding module;

[0028] The data acquisition is used to obtain key frames and predicted frames of 3D point clouds, 2D RGB images synchronized with the key frames and predicted frame point clouds, and IMU pose estimation results;

[0029] The ORB feature extraction module is used to extract ORB feature points of I frame and P frame 2D RGB images;

[0030] The threshold condition judgment module is used to set a threshold condition. If it is determined that the number of ORB feature points of the two frames of image does not meet the threshold condition, the frames are compressed as key frames. If the number of ORB feature points of the two frames of image does meet the threshold condition, the ORB feature points are mapped to the 3D space.

[0031] The IMU motion estimation module is used to obtain the relative pose of the camera estimated by the IMU and set it as the initial pose of the ICP algorithm;

[0032] The relative pose calculation module is used to calculate the relative pose of the key points of the two frame point clouds using the ICP algorithm in 3D space;

[0033] The ICP convergence condition judgment module is used to set the convergence condition of the ICP algorithm. If it is determined that the calculation result of the ICP algorithm meets the convergence condition, the transformation matrix T of the I frame and the P frame is output; if it is determined that the calculation result of the ICP algorithm does not meet the convergence condition, the predicted frame is compressed as a key frame;

[0034] The transformation matrix encoding module is used to encode the transformation matrix T of the I frame and the P frame into data Ppart;

[0035] The transformation module is used to transform the predicted frame into the key frame space based on the transformation matrix T to obtain data Pcloud*;

[0036] The macroblock segmentation module is used to segment the data Pcloud* into multiple macroblocks;

[0037] The macroblock traversal module is used to traverse each macroblock of Pcloud* and find whether there is a corresponding macroblock in the keyframe. If so, the similarity between the two point cloud macroblocks is calculated;

[0038] The similarity threshold condition judgment module is used to set a similarity threshold condition. If it is determined that the similarity between two point cloud macroblocks meets the similarity threshold condition, the encoding module encodes the key of the key frame macroblock into data Ppart, and the data Ppart is the data predicted by the key frame in the prediction frame;

[0039] If it is determined that the similarity between the two point cloud macroblocks does not meet the similarity threshold condition, or there is no corresponding macroblock, the encoding module encodes the data of the predicted frame macroblock into data Ipart, and the data Ipart is the data in the predicted frame that cannot be predicted using the key frame.

[0040] As a preferred technical solution, the ORB feature extraction module uses the OFSAT algorithm to extract ORB feature points of I-frame and P-frame 2D RGB images;

[0041] And / or, the IMU motion estimation module obtains the camera relative pose estimated by the IMU based on a Kalman filter.

[0042] As a preferred technical solution, the transformation module is used to transform the predicted frame into the key frame space based on the transformation matrix T to obtain data Pcloud*, which is specifically expressed as:

[0043] PCloud * =T·Pcloud

[0044] Among them, Pcloud represents the predicted frame.

[0045] As a preferred technical solution, the similarity of two point cloud macroblocks is calculated by the number of point clouds and centroid deviation in the macroblocks, which can be specifically expressed as:

[0046]

[0047] Among them, N PM and N IM are the number of points in the data Pcloud* macroblock and the corresponding keyframe macroblock, C PM and C IM is the centroid of the corresponding macroblock, and ρ is the Euclidean distance.

[0048] As a preferred technical solution, an octree is used to store data Pcloud*. The macroblock position is described as an octree index, represented by Key (x; y; z), where (x; y; z) is the spatial coordinate of the octree leaf node.

[0049] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0050] (1) The present invention extracts and maps effective ORB feature points in 2D space, reduces the time consumption of feature extraction, and reduces the number of points involved in motion estimation through feature points, thereby significantly improving the compression efficiency of dynamic point clouds.

[0051] (2) The present invention fuses six-axis IMU data through a Kalman filter and uses the IMU to predict the change in camera posture as the initial value of the ICP algorithm, thereby effectively preventing the ICP algorithm from failing to converge and reducing the number of iterations of the ICP algorithm.

[0052] (3) This invention combines ORB feature extraction and IMU motion estimation, and uses an iterative closest point algorithm to solve the motion estimation of two frame point clouds. It can efficiently achieve point cloud registration and quickly establish the coordinate relationship between frames. This technology reduces the time complexity of dynamic point cloud compression and is expected to play an important role in real-time applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 Schematic diagram of the process of the fast compression method of dynamic point cloud based on ORB feature extraction and IMU motion estimation of the present invention;

[0054] Figure 2 Schematic diagram of the process of performing ICP calculation by combining ORB feature extraction and IMU motion estimation in the present invention;

[0055] Figure 3 This is a flow chart of a predictive frame coding algorithm using an octree structure and macroblock matching according to the present invention. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0057] Example 1

[0058] like Figure 1 As shown, this embodiment provides a method for fast compression of dynamic point clouds based on ORB feature extraction and IMU motion estimation, comprising the following steps:

[0059] S1: Obtain the key frame (Icloud) and predicted frame (Pcloud) of the 3D point cloud, the 2D RGB images (Iimage and Pimage) synchronized with these two point cloud frames, and the IMU pose estimation results running at a frequency of 1 kHz;

[0060] S2: If Figure 2 As shown, the ORB feature points of the I-frame and P-frame 2D RGB images are extracted by the ORB feature point extraction algorithm;

[0061] In this embodiment, the ORB feature point extraction algorithm adopts the OFSAT algorithm. The OFAST algorithm has high computational efficiency, and the extracted ORB feature points are rotationally invariant and scale-invariant, and have high robustness.

[0062] S3: Set a threshold condition. If the number of ORB feature points of the two frames does not meet the threshold condition (for example, the number of feature points is less than 60), it is determined that the predicted frame cannot be encoded and is directly compressed as a key frame. Otherwise, the ORB feature points of the 2D RGB image are mapped to the 3D space according to the camera parameters.

[0063] S4: Get the relative pose of the camera estimated by the IMU and set it as the initial pose of the ICP algorithm. At the same time, set the convergence conditions of the ICP algorithm, such as the maximum number of iterations, the difference between the two transformation matrices, and the mean square error.

[0064] S5: Use the ICP algorithm to calculate the relative pose of the key points of the two frames of point clouds in 3D space. If the ICP calculation result converges, the transformation matrix T of the I frame and the P frame is obtained, and the motion estimation of the two frames of point clouds is successfully performed; otherwise, the predicted frame (Pcloud) is compressed as a key frame;

[0065] In this embodiment, the ORB feature points and the relative pose estimated by the IMU are used to align the point clouds through the ICP algorithm, thereby realizing motion estimation of the two frames of point clouds. This can effectively handle different situations of motion estimation and flexibly determine whether to perform predictive frame encoding, providing robustness and reliability for point cloud processing.

[0066] S6: As Figure 3 As shown, the transformation matrix T of the I frame and the P frame is encoded into Ppart, and then the predicted frame (Pcloud) is transformed into the key frame (Icloud) space based on the transformation matrix T, which is expressed as Pcloud*, that is:

[0067] PCloud * =T·Pcloud

[0068] S7: Use an octree to store Icloud and the transformed Pcloud*. Set the octree leaf resolution to M, and the point cloud is divided into M×M×M macroblocks. Traverse each macroblock of Pcloud* to find whether there is a corresponding macroblock in Icloud. If so, check the similarity of the two point cloud macroblocks. The similarity is measured by the number of point clouds in the macroblock and the centroid deviation, as shown in the following formula:

[0069]

[0070] Among them, N PM and N IM are the number of midpoints in the Pcloud* macroblock and the corresponding Icloud macroblock, C PM and C IM is the centroid of the macroblock, and ρ is the Euclidean distance. After practice, after aligning the coordinate systems of the point clouds, this similarity evaluation method can quickly calculate the similarity and ensure that the quality of the predicted point cloud is close to the original point cloud.

[0071] If the similarity condition is met, it can be considered that the Pcloud* macroblock can be described by the position of the Icloud macroblock; the macroblock position can be described as an octree index, represented by Key (x; y; z), where (x; y; z) is the spatial coordinate of the octree leaf node; the Key of the Icloud macroblock is encoded into Ppart; if the similarity condition is not met, or there is no corresponding macroblock, the data of the Pcloud macroblock is encoded into Ipart.

[0072] In this embodiment, Ppart is the data in Pcloud that can be predicted using Icloud, and stores information compressed by the matrix T and the corresponding Macroblock position index Key; on the other hand, Ipart is the data in Pcloud that cannot be predicted using Icloud, and this part of the data is stored in the form of original point cloud.

[0073] Example 2

[0074] This embodiment provides a dynamic point cloud fast compression system based on ORB feature extraction and IMU motion estimation, including: a data acquisition module, an ORB feature extraction module, a threshold condition judgment module, an IMU motion estimation module, a relative pose calculation module, an ICP convergence condition judgment module, a transformation matrix encoding module, a transformation module, a macroblock segmentation module, a macroblock traversal module, a similarity threshold condition judgment module, and an encoding module;

[0075] In this embodiment, data acquisition is used to obtain key frames and predicted frames of 3D point clouds, 2D RGB images synchronized with the key frames and predicted frame point clouds, and IMU pose estimation results;

[0076] In this embodiment, the ORB feature extraction module is used to extract ORB feature points of I-frame and P-frame 2D RGB images;

[0077] In this embodiment, the ORB feature extraction module uses the OFSAT algorithm to extract ORB feature points from the 2D RGB images of the I frame and the P frame, and maps the 2D ORB feature points to the 3D space according to the stereo camera parameters to generate 3D feature points. This reduces the number of points involved in the 3D calculation and avoids feature extraction and matching in the 3D space, thereby reducing time consumption and further reducing the amount of calculation, thereby improving the real-time performance of the remote teleoperation system.

[0078] In this embodiment, the stereo camera consists of a monocular sensor, a depth sensor, and a six-axis IMU. It can output 2D RGB images, 3D point clouds, and three-axis gyroscope and acceleration data. Specifically, the Intel RealSense D435 depth camera is used. This camera has a large field of view and a global shutter configured on the depth sensor, providing a reliable hardware foundation for the point cloud compression algorithm.

[0079] In this embodiment, the threshold condition judgment module is used to set a threshold condition. If it is determined that the number of ORB feature points of the two frames of image does not meet the threshold condition, the frames are compressed as key frames. If the number of ORB feature points of the two frames of image does meet the threshold condition, the ORB feature points are mapped to the 3D space.

[0080] In this embodiment, the IMU motion estimation module is used to obtain the relative pose of the camera estimated by the IMU and set it as the initial pose of the ICP algorithm;

[0081] In this embodiment, the IMU motion estimation module obtains the relative position of the camera estimated by the IMU based on the Kalman filter.

[0082] In this embodiment, the IMU motion estimation module uses a Kalman filter to process the six-axis IMU data of the 3D camera to predict the change in camera posture, which is used as the initial value of the ICP algorithm to improve the convergence speed of the ICP.

[0083] In this embodiment, the six-axis IMU is a six-axis inertial measurement unit (Six-axis Inertial Measurement Unit). This device contains three accelerometers and three gyroscopes, which are used to measure the linear acceleration and angular velocity of an object respectively. The six axes are along the three spatial directions of the object: X-axis, Y-axis and Z-axis.

[0084] In this embodiment, the Kalman filter is used to fuse the acceleration and angular velocity data of the IMU to obtain a more reliable attitude, because the acceleration can only obtain a more accurate attitude at a stationary moment, while the gyroscope is only sensitive to attitude changes during rotation. Moreover, if the gyroscope itself has errors, the errors will continue to increase after continuous time integration. Therefore, it is necessary to combine the attitudes calculated by the two and use the Kalman filter for fusion.

[0085] In this embodiment, the six-axis IMU Kalman filter is modeled as follows:

[0086] Define the system state as the posture of the IMU relative to the earth coordinate system, and assume that the initial coordinate system of the IMU coincides with the earth coordinate system. Then the prior estimation equation of the Kalman filter is:

[0087]

[0088] Right now

[0089]

[0090] in, All angular velocity variables are relative to the geodetic coordinate system.

[0091] The observation equation is:

[0092]

[0093] Right now

[0094]

[0095] The covariance update equation is:

[0096]

[0097] The Kalman gain is:

[0098]

[0099] The optimal estimating equation is:

[0100]

[0101] According to the results of multiple debugging, the process noise covariance matrix Q and the measurement noise covariance matrix R are set to the initial values:

[0102]

[0103] Set the prior estimated covariance matrix P and the system initial state x as:

[0104]

[0105] Thus, the posture of the camera relative to the initial state is obtained.

[0106] In this embodiment, the relative pose calculation module is used to calculate the relative pose of key points of two frames of point clouds using the ICP algorithm in 3D space;

[0107] In this embodiment, motion estimation is performed on two frames of point clouds using an iterative closest point (ICP) algorithm.

[0108] The ICP algorithm is an iterative optimization algorithm for point cloud registration. Its core idea is to find the optimal transformation that minimizes the distance between corresponding points in two point clouds through repeated iterations, thereby achieving accurate registration of point clouds. In the initial stage of the algorithm, the initial correspondence between these points is established by finding the closest points in the two point clouds. Then, the rigid transformation matrix that minimizes the distance between corresponding points is calculated through the singular value decomposition (SVD) method. This transformation matrix is ​​applied to one of the point clouds, and the updated point cloud is used again for the next round of iteration. This process is iterative until specific convergence conditions are met, such as the change in the transformation matrix is ​​less than a certain threshold or a predetermined number of iterations is reached;

[0109] In the ICP algorithm, the corresponding points of the two frame point clouds are considered to be the points with the closest distance. However, the nearest neighbor of a point is often not the corresponding point in the other point cloud, so the matching effect of the obtained Euclidean transformation R and t will not be too good. Therefore, the ICP algorithm needs to go through multiple iterations to make the nearest neighbor selected each time gradually become more accurate, and the obtained R and t will match better and better. When using ICP for posture estimation in engineering, it is generally necessary to set parameters such as the maximum number of iterations, the difference between the two transformation matrices, and the mean square error. These parameters serve as the convergence conditions of the ICP algorithm iteration. When the conditions are met, the ICP algorithm ends the calculation;

[0110] Therefore, the ICP algorithm is quite sensitive to the initial pose when performing motion estimation. The accuracy of the initial pose directly affects the convergence of the algorithm and the final matching results. When the matching degree of the initial pose is low, the algorithm needs to perform more iterations to continuously optimize the transformation parameters so that the corresponding points between the two point clouds are more accurately aligned.

[0111] To improve the computational efficiency of ICP motion estimation, an IMU is introduced to assist in estimating the pose of the stereo camera. By using the acceleration and angular velocity information measured by the IMU, the camera's motion trajectory in space can be more accurately estimated. This relative pose estimate is used as the initial pose of the ICP algorithm, providing better initial conditions for the registration process. The pose information provided by the IMU is crucial to the accuracy of the initial pose, as the ICP algorithm is highly sensitive to the initial pose. By integrating the IMU's measurements, a relatively accurate camera pose can be obtained, thus avoiding the problem of the ICP algorithm requiring a large number of iterations when the initial pose is inaccurate.

[0112] In this embodiment, the ICP convergence condition judgment module is used to set the convergence condition of the ICP algorithm. If it is determined that the calculation result of the ICP algorithm meets the convergence condition, the transformation matrix T of the I frame and the P frame is output. If it is determined that the calculation result of the ICP algorithm does not meet the convergence condition, the predicted frame is compressed as a key frame.

[0113] In this embodiment, the transformation matrix encoding module is used to encode the transformation matrix T of the I frame and the P frame into data Ppart;

[0114] In this embodiment, the transformation module is used to transform the predicted frame into the key frame space based on the transformation matrix T to obtain data Pcloud*, which is specifically expressed as:

[0115] PCloud * =T·Pcloud

[0116] Among them, Pcloud represents the predicted frame.

[0117] In this embodiment, the macroblock segmentation module is used to segment the data Pcloud* into multiple macroblocks;

[0118] In this embodiment, the macroblock traversal module is used to traverse each macroblock of Pcloud* to find whether a corresponding macroblock exists in the keyframe. If so, the similarity between the two point cloud macroblocks is calculated;

[0119] In this embodiment, the similarity between two point cloud macroblocks is calculated by the number of point cloud data in each macroblock and the centroid deviation between the two point cloud groups. This similarity evaluation method can quickly calculate the similarity and ensure that the quality of the predicted point cloud is close to the original point cloud. Specifically, it is expressed as:

[0120]

[0121] Among them, N PM and N IM are the number of points in the data Pcloud* macroblock and the corresponding keyframe macroblock, C PM and C IM is the centroid of the corresponding macroblock, and ρ is the Euclidean distance.

[0122] In this embodiment, an octree is used to store data Pcloud*. The macroblock position is described as an octree index, represented by Key(x; y; z), where (x; y; z) is the spatial coordinate of the octree leaf node. This embodiment uses the octree structure to divide the point cloud into multiple groups of macroblocks, checks the matching degree of each group of macroblocks between the predicted frame and the key frame, and replaces the successfully matched macroblocks with the macroblock position index for compression, thereby reducing the temporal redundancy of the point cloud.

[0123] In this embodiment, the octree structure is a spatial segmentation data structure used to efficiently organize and store point cloud data. It recursively divides the three-dimensional space into eight sub-cubes, each of which can be further divided until a stopping condition is reached. In the point cloud, each octree node represents a cubic region and stores the point cloud data or other related information within that region. This structure makes searching, traversing, or performing spatial queries more efficient because it only needs to focus on the nodes in a specific region without having to process the entire point cloud.

[0124] In this embodiment, the similarity threshold condition judgment module is used to set the similarity threshold condition. If it is determined that the similarity between two point cloud macroblocks meets the similarity threshold condition, the encoding module encodes the key of the key frame macroblock into the data Ppart. The data Ppart is the data predicted using the key frame in the prediction frame.

[0125] If it is determined that the similarity between the two point cloud macroblocks does not meet the similarity threshold condition, or there is no corresponding macroblock, the encoding module encodes the data of the predicted frame macroblock into data Ipart, and the data Ipart is the data in the predicted frame that cannot be predicted using the key frame.

[0126] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A fast compression method for dynamic point clouds based on ORB feature extraction and IMU motion estimation, characterized in that: The steps include: Obtain keyframes and predicted frames of 3D point clouds, 2D RGB images synchronized with the keyframes and predicted frame point clouds, and IMU pose estimation results; Extract ORB feature points of key frames and predicted frames 2D RGB images; Set a threshold condition. If the number of ORB feature points of the two frames does not meet the threshold condition, they are compressed as key frames. If the threshold condition is met, the ORB feature points are mapped to 3D space. Get the relative pose of the camera estimated by the IMU and set it as the initial pose of the ICP algorithm, and set the convergence condition of the ICP algorithm; The ICP algorithm is used to calculate the relative pose of the key points of the two frames of point cloud in 3D space. If the calculation result of the ICP algorithm is determined to meet the convergence conditions, the transformation matrix T of the key frame and the predicted frame is output. If the calculation result of the ICP algorithm is determined to not meet the convergence conditions, the predicted frame is compressed as the key frame; The transformation matrix T of the key frame and the predicted frame is encoded into the data Ppart, and the predicted frame is transformed into the key frame space based on the transformation matrix T to obtain the data Pcloud*; Divide the data Pcloud* into multiple macroblocks, traverse each macroblock of Pcloud*, and find whether the corresponding macroblock exists in the keyframe. If so, calculate the similarity of the two point cloud macroblocks; A similarity threshold condition is set. If it is determined that the similarity between the two point cloud macroblocks meets the similarity threshold condition, the key of the keyframe macroblock is encoded into data Ppart, where the data Ppart is the data predicted using the keyframe in the predicted frame; If it is determined that the similarity between the two point cloud macroblocks does not meet the similarity threshold condition, or there is no corresponding macroblock, the data of the predicted frame macroblock is encoded into data Ipart, and the data Ipart is data in the predicted frame that cannot be predicted using the key frame.

2. The method for fast compression of dynamic point clouds based on ORB feature extraction and IMU motion estimation according to claim 1, characterized in that: The OFSAT algorithm is used to extract ORB feature points of the key frame and prediction frame 2D RGB images; And / or, obtaining the camera relative pose estimated by the IMU based on the Kalman filter.

3. The method for fast compression of dynamic point clouds based on ORB feature extraction and IMU motion estimation according to claim 1, characterized in that: Based on the transformation matrix T, the predicted frame is transformed into the key frame space to obtain the data Pcloud*, which is specifically expressed as: PCloud*=T·Pcloud Among them, Pcloud represents the predicted frame.

4. The method for fast compression of dynamic point clouds based on ORB feature extraction and IMU motion estimation according to claim 1, characterized in that: The similarity between two point cloud macroblocks is calculated by the number of point clouds and centroid deviation in the macroblocks, which can be expressed as: Among them, N PM and N IM are the number of points in the data Pcloud* macroblock and the corresponding keyframe macroblock, C PM and C IM is the centroid of the corresponding macroblock, and ρ is the Euclidean distance.

5. The method for fast compression of dynamic point clouds based on ORB feature extraction and IMU motion estimation according to claim 1, characterized in that: Octree is used to store data Pcloud*. The macroblock position is described as the index of the octree, which is represented by Key (x; y; z), where (x; y; z) is the spatial coordinate of the octree leaf node.

6. A dynamic point cloud fast compression system based on ORB feature extraction and IMU motion estimation, characterized in that: include: Data acquisition module, ORB feature extraction module, threshold condition judgment module, IMU motion estimation module, relative pose calculation module, ICP convergence condition judgment module, transformation matrix encoding module, transformation module, macroblock segmentation module, macroblock traversal module, similarity threshold condition judgment module, encoding module; The data acquisition is used to obtain key frames and predicted frames of 3D point clouds, 2D RGB images synchronized with the key frames and predicted frame point clouds, and IMU pose estimation results; The ORB feature extraction module is used to extract ORB feature points of key frames and predicted frame 2D RGB images; The threshold condition judgment module is used to set a threshold condition. If it is determined that the number of ORB feature points of the two frames of image does not meet the threshold condition, the frames are compressed as key frames. If the number of ORB feature points of the two frames of image does meet the threshold condition, the ORB feature points are mapped to the 3D space. The IMU motion estimation module is used to obtain the relative pose of the camera estimated by the IMU and set it as the initial pose of the ICP algorithm; The relative pose calculation module is used to calculate the relative pose of the key points of the two frame point clouds using the ICP algorithm in 3D space; The ICP convergence condition judgment module is used to set the convergence condition of the ICP algorithm. If it is determined that the calculation result of the ICP algorithm meets the convergence condition, the transformation matrix T of the key frame and the predicted frame is output; if it is determined that the calculation result of the ICP algorithm does not meet the convergence condition, the predicted frame is compressed as a key frame; The transformation matrix encoding module is used to encode the transformation matrix T of the key frame and the predicted frame into data Ppart; The transformation module is used to transform the predicted frame into the key frame space based on the transformation matrix T to obtain data Pcloud*; The macroblock segmentation module is used to segment the data Pcloud* into multiple macroblocks; The macroblock traversal module is used to traverse each macroblock of Pcloud* and find whether there is a corresponding macroblock in the keyframe. If so, the similarity between the two point cloud macroblocks is calculated; The similarity threshold condition judgment module is used to set a similarity threshold condition. If it is determined that the similarity between two point cloud macroblocks meets the similarity threshold condition, the encoding module encodes the key of the key frame macroblock into data Ppart, and the data Ppart is the data predicted by the key frame in the prediction frame; If it is determined that the similarity between the two point cloud macroblocks does not meet the similarity threshold condition, or there is no corresponding macroblock, the encoding module encodes the data of the predicted frame macroblock into data Ipart, and the data Ipart is the data in the predicted frame that cannot be predicted using the key frame.

7. The dynamic point cloud fast compression system based on ORB feature extraction and IMU motion estimation according to claim 6 is characterized in that: The ORB feature extraction module uses the OFSAT algorithm to extract ORB feature points of the key frame and the predicted frame 2D RGB image; And / or, the IMU motion estimation module obtains the camera relative pose estimated by the IMU based on a Kalman filter.

8. The dynamic point cloud fast compression system based on ORB feature extraction and IMU motion estimation according to claim 6 is characterized in that: The transformation module is used to transform the predicted frame into the key frame space based on the transformation matrix T to obtain data Pcloud*, which is specifically expressed as: PCloud * =T·Pcloud Among them, Pcloud represents the predicted frame.

9. The dynamic point cloud fast compression system based on ORB feature extraction and IMU motion estimation according to claim 6 is characterized in that: The similarity between two point cloud macroblocks is calculated by the number of point clouds and centroid deviation in the macroblocks, which can be expressed as: Among them, N PM and N IM are the number of points in the data Pcloud* macroblock and the corresponding keyframe macroblock, C PM and C IM is the centroid of the corresponding macroblock, and ρ is the Euclidean distance.

10. The dynamic point cloud fast compression system based on ORB feature extraction and IMU motion estimation according to claim 6, characterized in that: Octree is used to store data Pcloud*. The macroblock position is described as the index of the octree, which is represented by Key (x; y; z), where (x; y; z) is the spatial coordinate of the octree leaf node.

Images