3D point cloud registration method based on point and line feature fusion

Abstract

The application belongs to the technical field of computer vision, and discloses a three-dimensional point cloud registration method based on point-line feature fusion. The three-dimensional point cloud registration method firstly uses SAC-IA to complete static registration of three-dimensional point clouds on the surface of a target object, and establishes an initial motion transformation relationship from the three-dimensional point clouds of the 3D model of the target object to the three-dimensional point clouds on the surface of the target object. Then, in the dynamic tracking stage, point features extracted by ORB are matched with line segment features extracted by LSD-LBD, coarse registration is realized by combining an EPnPL algorithm, and the robustness of pose estimation is improved by eliminating false matching through RANSAC. Finally, in the fine registration stage, point-to-plane ICP is used to further optimize pose estimation. The method does not need to rely on artificial marking, and can complete stable three-dimensional point cloud registration based on natural features, and can be widely applied to complex scenes such as mechanical product maintenance and industrial assembly.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology and relates to a three-dimensional point cloud registration method based on point and line feature fusion. Background Technology

[0002] With the rapid adoption of Augmented Reality (AR) technology in fields such as industrial equipment maintenance, complex equipment assembly, and medical surgical navigation, the demand for environmental perception and spatial positioning based on 3D sensors is becoming increasingly significant. In these application scenarios, the system needs to acquire real-time 3D geometric information of the site and accurately track the dynamic posture of equipment or tools in space to ensure that virtual guidance information can be correctly superimposed on the surface of real objects, thereby assisting operators in completing precise operations.

[0003] 3D point cloud registration, as one of the core technologies of augmented reality systems, is a key step in achieving seamless integration of virtual information and real-world scenes. To achieve sufficient accuracy and stability in practical applications such as equipment maintenance, assembly sequence guidance, and operation process visualization, registration methods not only need to effectively identify various geometric structures (such as equipment edges, structural component contours, and hole boundaries), but also need to achieve robust pose estimation in dynamic environments.

[0004] The shortcomings of existing technologies are mainly reflected in the following aspects:

[0005] I. Limitations of Traditional ICP (Iterative Closest Point) Algorithms. Besl and McKay proposed a classic ICP registration framework in their paper "A Method for Registration of 3-D Shapes." This method employs a nearest-point iterative alignment strategy, enabling point cloud registration even in environments with missing textures and unstable lighting. However, this method is highly dependent on the initial pose; if the depth sensor moves rapidly or the initial correspondence is inaccurate, it is prone to getting trapped in local minima, leading to registration failure. Rusinkiewicz and Levoy systematically analyzed various improved ICP strategies in their paper "Efficient Variants of the ICP Algorithm," but these improvements still do not solve the problems of sensitivity to initial pose and unstable convergence in low-overlap or partially scanned scenes. Subsequently, Yang et al. proposed the Go-ICP method in the paper "Go-ICP: A Globally Optimal Solution to 3D ICP Point-Set Registration". The Go-ICP method improves the stability of the initial alignment stage by introducing global optimization. However, the algorithm has a large computational load and is difficult to meet the real-time requirements of augmented reality and other application scenarios.

[0006] II. Registration methods based solely on point or edge features have limitations. Wang et al., in "A 3D tracking and registration method based on point cloud and visual features foraugmented reality aided assembly system," proposed a 3D tracking and registration algorithm that combines point cloud data with visual features. Building upon ICP point cloud registration, this method introduces visual feature matching to provide more accurate initial pose estimation for ICP, thereby improving overall robustness during rapid movement of the depth sensor. However, because this method relies on ORB (Oriented Fast and Rotated BRIEF) feature extraction, it still has significant limitations in scenes lacking texture features. Ombari et al., in "BOLD features to detect textureless objects," extracted a 3D point cloud registration method based on edge contours, which estimates pose by identifying and locating object contour features, thus achieving tracking and registration. Damen et al., in "Real-time learning and detection of 3D textureless objects: a scalable approach," also proposed a similar method. However, such methods are difficult to achieve ideal results in situations with complex backgrounds, cluttered outlines, or occlusion.

[0007] To address the aforementioned issues, this invention proposes a three-dimensional point cloud registration method based on point-line feature fusion. Summary of the Invention

[0008] The purpose of this invention is to propose a 3D point cloud registration method based on point-line feature fusion. This 3D point cloud registration method first uses SAC-IA (Sample Consensus Initial Alignment) to complete static initialization registration, establishing the initial motion transformation relationship from the object model to the scene object point cloud. Then, in the dynamic tracking stage, it extracts and matches ORB point features with LSD-LBD (Line Segment Detector, Line Band Descriptor) line features, combined with EPnPL (Extended Perspective-n-Point and Line) to achieve coarse registration, and uses RANSAC (Random Sample Consensus) to eliminate mismatches, improving the robustness of pose estimation. Finally, in the fine registration stage, it employs point-to-surface ICP to further optimize pose estimation.

[0009] The technical solution of the present invention:

[0010] A 3D point cloud registration method based on point and line feature fusion, comprising the following steps:

[0011] Step 1: Obtain the 3D model of the target object to be tracked. Use voxel filtering downsampling to obtain the 3D point cloud of the target object's 3D model, denoted as... The process involves: acquiring real-time color and depth images of the work scene using an RGB-D camera; aligning the depth images to color images using a coordinate transformation matrix between the color and depth cameras; detecting target objects in the color images using a YOLOv8 model and obtaining their bounding boxes; and then calculating the 3D point cloud of the target object's surface, denoted as... ;

[0012] Step 2: Before performing dynamic registration, static registration of the 3D point cloud of the target object's surface is first performed; the 3D point cloud of the target object's 3D model is completed using SAC-IA. 3D point cloud of the target object surface Registration between them, estimating the position from the 3D model to the target object as observed in the camera coordinate system. Motion transformation matrix :

[0013]

[0014] in, and These represent the rotation matrix and translation vector from the position of the 3D model to the position of the target object, respectively.

[0015] Step 3: Save the color image, depth image, and motion transformation matrix from steps 1 and 2. The bounding box of the target object and the 3D point cloud of the target object's surface. This is used to indicate the state of the target object at the previous moment, and the color image at the previous moment is recorded as the previous frame; the color image, depth image, bounding box of the target object, and 3D point cloud of the target object surface at the current moment are obtained using the same method as in step 1, and the 3D point cloud of the target object surface at the current moment is recorded as... The color image at the current moment is the current frame;

[0016] Step 4: Perform the same feature extraction operation on the previous frame and the current frame, specifically including: first, extracting feature points using the ORB operator; then, detecting line segment features using LSD; further encoding the line segment features using LBD; and finally, performing bidirectional matching of feature points and line segment features using a brute-force matching algorithm, obtaining a total of [number missing] feature points. Matching pairs and Each matching line pair; all the above feature extraction operations are performed within the ROI region inside the bounding box of the target object;

[0017] Step 5: Set the maximum number of iterations to 200, and the point reprojection error threshold. Using a unit of 5 pixels, the percentage of inlier points that converge early is determined to be 80%.

[0018] Step 6: Randomly sample 12 matching point pairs and 6 matching line pairs from the matching point pairs and matching line pairs respectively; estimate the motion transformation matrix between the previous frame and the current frame using the EPnPL algorithm; initialize the four control points from the previous time step. The four control points are the x-axis, y-axis, z-axis, and origin of the Cartesian coordinate system; the endpoints of the feature points and line segment features are denoted as three-dimensional points. All three points Represented as four control points Linear combination: ;in Given the weights, based on the three points The positions of the four control points are uniquely determined; let the three-dimensional points be... 3D point cloud of the target object surface at the current moment The corresponding three-dimensional points are The projection point corresponding to the current frame is Let the three-dimensional point cloud of the target object's surface at the previous moment be... Control point 3D point cloud of the target object surface at the current moment The corresponding control point is Based on the projection relationship from 3D space to the camera plane and the linear relationship between 3D points and control points, the following point reprojection error equation is obtained:

[0019]

[0020] in, For the intrinsic parameters of the color camera; each matching point pair corresponds to two equations. Each matching point is One equation;

[0021] Step 7: Let the endpoints of the line segment features detected in the current frame be respectively , If all are homogeneous, then the characteristic representation of a line segment is:

[0022]

[0023] in, Let be the homogeneous representation of a line segment. This is the normalized result; the error from an endpoint of a line segment feature in the previous frame to the matching line segment in the current frame is expressed as:

[0024]

[0025] in, This is the intrinsic parameter matrix of the color camera; since a line segment feature contains two endpoints, they are respectively represented by... and This indicates that 6 matching point pairs constitute 12 equations, which are the error equations from the point to the line segment:

[0026]

[0027] With four control points unknown The final reprojection error equation is obtained by integrating the point reprojection error equation and the point-to-line segment error equation. , Solving the reprojection error equation yields the four control points of the current frame. ;

[0028] Step 8: Based on the known control points of the previous frame and the corresponding current frame control point SVD decomposition is used to obtain the motion transformation matrix observed in the camera coordinate system from the previous frame to the current frame. ;

[0029] Step 9: Based on the motion transformation matrix Using the intrinsic parameters of the color camera, calculate the reprojection error of all matching points from the previous time step to the current frame. ;Regulation Less than the point reprojection error threshold The matching points are identified as interior points, and the number of interior points is counted.

[0030] Step 10: Repeat steps 6-9 until the ratio of the number of interior points to the total number of feature points is greater than 0.8, thus achieving early convergence, or until the maximum number of iterations is reached; calculate the motion transformation matrix obtained in the round with the largest number of interior points. As the final output;

[0031] Step 11: Downsample the 3D point cloud from the previous time step using voxel grid filtering. and the current 3D point cloud ;

[0032] The downsampling result obtained in step 11 and the motion transformation matrix output in step 10 are used as the basis for this process. As input to the point-to-surface ICP algorithm, the motion transformation matrix is... Make corrections; calculate the 3D point cloud from the 3D model of the target object. Motion transformation matrix to the current frame ;

[0033] Step 12: Save the 3D point cloud sampled in the current frame, the detected feature points, and line segment features; save the motion transformation matrix. Update the current frame to the previous frame. for , for Then process the next frame.

[0034] The beneficial effects of this invention are:

[0035] Compared with existing traditional point cloud registration methods based on single features, the 3D point cloud registration method based on point-line feature fusion proposed in this invention has achieved significant technical effects in augmented reality maintenance guidance systems, specifically as follows:

[0036] (1) Significantly improved tracking robustness:

[0037] In traditional methods, point features are easily lost in fast motion, lighting changes, or weak texture environments, leading to tracking failure. This invention introduces LSD-LBD line segment features on top of ORB point features and uses the EPnPL algorithm to fuse point and line features. This enables stable tracking in mechanical parts scenes with missing textures or prominent edge features, effectively avoiding the problems caused by the failure of a single feature.

[0038] (2) Mismatch elimination and enhanced estimation stability:

[0039] A Random Sample Consensus (RANSAC) algorithm is introduced in the initial pose estimation process. Through multiple sampling and reprojection error statistics, a large proportion of mismatched points and line features are eliminated. Even in the presence of many interfering features, the algorithm can still obtain high-precision pose estimation, significantly improving the stability of registration.

[0040] (3) Improved registration accuracy:

[0041] By employing a two-stage strategy combining coarse and fine registration, registration accuracy is improved. In the fine registration stage, this invention uses a point-to-plane ICP method instead of the traditional point-to-point ICP. This method utilizes point cloud normal vector information to constrain the optimization process, transforming the registration error from point-to-point distance to point-to-plane distance, thereby improving convergence speed and global accuracy, and avoiding getting trapped in local optima.

[0042] (4) Enhanced applicability and engineering feasibility:

[0043] This method does not rely on manual labeling and can achieve stable 3D point cloud registration entirely based on natural features, making it widely applicable to complex scenarios such as mechanical product repair and industrial assembly. Furthermore, the algorithm framework is highly scalable, capable of combining different feature detection and matching methods, and possesses good versatility and application prospects. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the three-dimensional point cloud registration method based on point-line feature fusion of the present invention. Detailed Implementation

[0045] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and technical solutions.

[0046] A comparison was made with a baseline model on the YCB-Video dataset. The baseline model uses the ORB-ICP algorithm, which only uses point features in the coarse registration stage and the point-to-surface ICP algorithm in the fine registration stage. The experimental procedure in this embodiment is as follows: First, consistent RGB-D data is generated from the YCB-Video dataset using the depth image and color image alignment steps according to the official camera calibration file. Then, YOLOv8 is used to detect target objects in each frame of the YCB-Video dataset, the region of interest (ROI) of the target object is extracted, and the 3D point cloud of the target object in the current frame is obtained by backprojection based on the depth image. Voxel filtering downsampling is performed on the 3D point cloud of the 3D model of the target object and the 3D point cloud of the target object surface to ensure the consistency of point cloud density. Then, SAC-IA is used for static registration of the 3D point cloud to establish the initial transformation relationship between the 3D point cloud of the 3D model of the target object and the 3D point cloud of the target object surface. Subsequently, the EPnPL algorithm is executed between consecutive frames to estimate the inter-frame motion of the target object. After each estimation, point-to-surface ICP is used for precise optimization, thereby obtaining the final registration transformation from the 3D point cloud of the target object's 3D model to the 3D point cloud of the target object's surface. During the experiment, the officially labeled test sequence is selected for each target object, and the algorithm of this invention and ORB-ICP are run on all test frames respectively. The pose error of each frame is recorded, and the average error is finally calculated for performance comparison.

[0047] Table 1. Accuracy Analysis of 3D Point Cloud Registration Results

[0048]

[0049] Table 1 summarizes the average registration errors for multiple objects. The proposed algorithm achieved lower average errors than ORB-ICP for 7 out of 9 test objects. The errors for the mustard bottle decreased by 7.3%, foam brick by 4.6%, and kettle by 2.3%, showing particularly significant improvements. Although the errors for the luncheon meat can and banana objects increased slightly (both less than 0.04), the overall performance still demonstrates that the registration accuracy was significantly improved by fusing point and line segment features. The experimental results fully validate the effectiveness and superiority of the proposed algorithm. This invention has significant application value and promising prospects in the field of augmented reality maintenance guidance, providing reliable technical support for efficient maintenance and operation of complex mechanical structures.

Claims

1. A three-dimensional point cloud registration method based on point-line feature fusion, characterized in that, The steps are as follows: Step 1: Obtain the 3D model of the target object to be tracked, downsample the 3D model of the target object through voxel filtering to obtain a three-dimensional point cloud, denoted as ; Real-time acquisition of color image and depth image of the working scene using RGB-D camera, and alignment of depth image to color image using coordinate transformation matrix between color camera and depth camera; using YOLOv8 model to detect target object in color image and obtain bounding box of target object; then calculate three-dimensional point cloud of target object surface, denoted as ; Step 2: Before dynamic registration, first perform static registration of the three-dimensional point cloud of the target object surface; The specific implementation process of step 2 is as follows: using SAC-IA to complete the registration between the three-dimensional point cloud of the 3D model of the target object and the three-dimensional point cloud of the surface of the target object, estimate the motion transformation matrix from the position of the 3D model to the position of the target object observed in the camera coordinate system . The specific implementation process of step 2 is as follows: using SAC-IA to complete the registration between the three-dimensional point cloud of the 3D model of the target object and the three-dimensional point cloud of the surface of the target object, estimate the motion transformation matrix from the position of the 3D model to the position of the target object observed in the camera coordinate system . : wherein, and R and t represent the rotation matrix and displacement vector from the 3D model position to the target object position, respectively; Step 3: save the color image, depth image, motion transformation matrix in step 1 and 2 , the bounding box of the target object and the three-dimensional point cloud of the target object surface , to indicate the state of the target object at the previous time, and record the color image at the previous time as the previous frame; the color image, depth image, bounding box of the target object and three-dimensional point cloud of the target object surface at the current time are obtained by the same method as in step 1, and the three-dimensional point cloud of the target object surface at the current time is recorded as , the color image at the current time is the current frame; Step 4: Perform the same feature extraction operation on the previous frame and the current frame; Step 5: Set the maximum number of iterations, point re-projection error threshold, and judge the percentage of inliers that converge early; Step 6: Obtain the re-projection error equation; Step 7: Solve the re-projection error equation; Step 8: The previous frame control points are known and the corresponding current frame control points are used to solve for the motion transformation matrix from the previous frame to the current frame observed in the camera coordinate system using SVD decomposition ; Step 9: Compute the motion transformation matrix according to the motion vector of the current frame and the motion vector of the previous frame and the intrinsic parameters of the color camera, compute the reprojection error of all the matched points in the previous frame to the current frame ; if the reprojection error of a matched point is less than a threshold value, the matched point is an inlier, and the number of inliers is counted ; if the reprojection error of a matched point is less than a threshold value, the matched point is an inlier, and the number of inliers is counted​ Step 10: Repeat steps 6-9 until the number of inliers in this iteration is greater than 0.8 of the total number of feature points or the maximum number of iterations is reached; the motion transformation matrix calculated in the round with the most inliers is taken as the final output. as the final output. Step 11: downsample the three-dimensional point cloud of the previous time instant using a voxel grid filter and the three-dimensional point cloud of the current time instant ; down-sampled result obtained in step 11 and the motion transformation matrix outputted in step 10 as input to the point-to-plane ICP algorithm, the motion transformation matrix is corrected; the three-dimensional point cloud from the 3D model of the target object to the current frame is calculated ; Step 12: save the three-dimensional point cloud of the current frame samples, detected feature points and line segment features; save the motion transformation matrix ; update the current frame as the previous frame, as , as , and process the next frame.

2. The three-dimensional point cloud registration method based on point-line feature fusion according to claim 1, characterized in that, The specific implementation process of step 4 is as follows: first, the ORB operator is used to extract feature points; second, the LSD is used to detect line segment features; third, the LBD is used to encode the line segment features; and finally, the brute force matching algorithm is used to bidirectionally match the feature points and the line segment features, so as to obtain matching point pairs and matching line pairs. The above feature extraction operations are performed in the ROI region within the bounding box of the target object.

3. The three-dimensional point cloud registration method based on point-line feature fusion according to claim 2, characterized in that, In step 5: Set the maximum number of iterations to 200, and the point re-projection error threshold is 5 pixel units, and the percentage of inliers that are judged to converge early is 80%.

4. The three-dimensional point cloud registration method based on point-line feature fusion according to claim 3, characterized in that, The specific implementation process of step 6 is as follows: randomly sample 12 matching point pairs and 6 matching line pairs from the matching point pairs and matching line pairs respectively; use the EPnPL algorithm to estimate the motion transformation matrix between the previous frame and the current frame; initialize the four control points of the previous time step. The four control points are the x-axis, y-axis, z-axis, and origin of the Cartesian coordinate system; the endpoints of the feature points and line segment features are denoted as three-dimensional points. All three points Represented as four control points Linear combination: ;in Given the weights, based on the three points The positions of the four control points are uniquely determined; let the three-dimensional points be... 3D point cloud of the target object surface at the current moment The corresponding three-dimensional points are The projection point corresponding to the current frame is Let the three-dimensional point cloud of the target object's surface at the previous moment be... Control point 3D point cloud of the target object surface at the current moment The corresponding control point is Based on the projection relationship from 3D space to the camera plane and the linear relationship between 3D points and control points, the following point reprojection error equation is obtained: wherein, is the intrinsic parameter of the color camera; each pair of matched points corresponds to two equations, one equation for each pair of matched points one equation.

5. The three-dimensional point cloud registration method based on point-line feature fusion according to claim 4, characterized in that, The specific implementation process of step 7 is as follows: Let the end points of the line segment feature detected in the current frame be , , all in homogeneous representation, then the line segment feature is represented as: wherein, is the homogeneous representation of the line segment, is the normalized result; the error representation of one end point of the line segment feature of the previous frame to the matching line segment in the current frame is: wherein, is the intrinsic matrix of the color camera; since a line segment feature contains two endpoints, respectively denoted by and , 6 matched point pairs form 12 equations, which are error equations from points to line segments: with four control points as unknowns , the point re-projection error equation and the point-to-line segment error equation are integrated to obtain a final re-projection error equation , , four control points of the current frame are solved by solving the re-projection error equation .

Images