A hybrid tracking and registration method

Abstract

A hybrid tracking and registration method relates to the field of computer vision processing technology, including an offline training phase and an online recognition phase; wherein the offline training phase includes obtaining reference images of a three-dimensional model from multiple angles and establishing an information set, extracting and storing directional gradient and normal vector features, and obtaining a model point cloud; the online recognition phase includes acquiring and repairing a depth image, extracting directional gradient and normal vector features of the depth image, using an improved linear parallel multimodal LineMod template matching method to describe the outline and shape of the object, and obtaining a key frame, matching it with the stored image, obtaining a camera pose, and using an improved ICP algorithm to align the environment point cloud and the model point cloud, determine the model pose information, and accurately estimate the camera pose. The present invention can have good real-time and robustness in application scenarios where the environment is cluttered and the operating object lacks texture, and has high alignment accuracy and computational efficiency.

Description

Technical Field

[0001] The present invention relates to the field of computer vision processing technology, and in particular to a hybrid-based tracking and registration method. Background Art

[0002] Tracking and registration algorithms are widely used in the fields of augmented reality and mixed reality, and are the core algorithms in this field. The accuracy and robustness of the algorithm will affect the virtual-reality registration effect in the actual process. The algorithm proposed in this patent plays a key role in assisting industrial assembly.

[0003] Currently, model-based tracking and registration methods have been widely used in the field of augmented reality. This method can solve the problem that objects are difficult to detect due to lack of texture. However, this method needs to process a large number of reference images collected from different perspectives and requires a lot of calculations. Therefore, the real-time performance and usability are poor in practical applications.

[0004] Point cloud-based tracking and registration methods show good robustness in assembly scenarios with low lighting intensity and lack of texture on the object surface. This method usually uses the ICP registration algorithm to iteratively calculate all or part of the point cloud, which can achieve better pose calculation under poor ambient lighting conditions. However, when the depth sensor cannot obtain the correct initial point cloud dataset due to rapid movement, the iterative process of the ICP algorithm is prone to fall into local optimality and even cause "frame loss", thereby interrupting the tracking and registration process. Summary of the Invention

[0005] In view of this, the technical problem to be solved by the present invention is to provide a hybrid tracking and registration method that can have good real-time and robustness in application scenarios where the environment is cluttered and the operating objects lack texture, and has high alignment accuracy and computational efficiency.

[0006] In order to solve the above technical problems, the technical solution of the present invention is:

[0007] A hybrid tracking and registration method includes an offline training phase and an online recognition phase. The offline training phase includes the following steps:

[0008] S10, obtaining reference images of the three-dimensional model from multiple angles through computer-aided design (CAD), and establishing an information set of the model;

[0009] S20, extracting and storing directional gradient and normal vector features of the image;

[0010] S30, obtaining the corresponding model point cloud;

[0011] The online recognition phase includes the following steps:

[0012] S40, collecting and repairing the depth image;

[0013] S50, extracting directional gradient and normal vector features of the depth image;

[0014] S60. Using the improved linear parallel multimodal LineMod template matching method, the directional gradient and normal vector features are calculated from the depth image to describe the contour and shape of the object, and key frames are obtained from the depth image, matched with the stored image, and the camera pose is obtained;

[0015] S70, using the improved ICP algorithm to register the environment point cloud and the model point cloud, determine the model pose information, and accurately estimate the camera pose;

[0016] In step S70, the following steps are included:

[0017] S71, use random sampling algorithm to simplify the environmental point cloud;

[0018] S72, matching the points with the same name, and assigning weights to the points with the same name;

[0019] S73. Filter mismatches of points with the same name.

[0020] Preferably, in step S60, the improved linear parallel multimodal LineMod template matching method includes the following steps:

[0021] S61, input depth image;

[0022] S62, blurring the depth image using a 7×7 Gaussian filter to remove depth jump pixels;

[0023] S63, using a 5×5 Sobel operator to perform gradient calculation on the blurred depth image to obtain the directional gradient and gradient value of each pixel in the image;

[0024] S64, selecting a pixel whose gradient value is greater than a set threshold, and quantizing the directional gradient of the pixel;

[0025] S65, combining the quantized normal direction of the depth image as a feature to create a template;

[0026] S66. Matching is performed based on the template.

[0027] Preferably, step S63 includes the following steps:

[0028] S631, converting the depth image into a grayscale image;

[0029] S632, performing convolution calculations on the grayscale image using Sobel operators in the horizontal and vertical directions, respectively, to obtain a grayscale image in the horizontal direction and a grayscale image in the vertical direction;

[0030] S633. Obtain edge information of the grayscale image using the formula G(I)=sqrt(Gx(I)^2+Gy(I)^2), where Gx(I) and Gy(I) represent the grayscale values of the image after horizontal and vertical edge detection, respectively.

[0031] S634. Calculate the direction angle using the formula θ=atan2(Gy(I),Gx(I)), where the atan2 function is an inverse tangent function, and the value it returns is a radian value between [-π,π].

[0032] Preferably, step S71 includes the following steps:

[0033] S711, calculating the bounding boxes of the target point cloud and the reference point cloud, and calculating the relative position and posture information to obtain a preliminary registration result;

[0034] S712. In a new iteration, adaptively adjust the number of sampling points according to the current registration error.

[0035] S713, use formula 1 to perform registration. In each iteration, by matching the sampling point and the nearest neighbor point on the reference point cloud, the transformation matrix is calculated to update the position and posture information of the target point cloud.

[0036] Formula 1:

[0037] Where RMSE is the root mean square error of the last iteration, d is the density of the point cloud, N is the size of the point cloud, Where x is the screening coefficient. When the RMSE error is relatively large, it means that there are many noise points in the current point cloud. Therefore, fewer points will be selected to participate in the calculation to speed up the convergence. When the RMSE error is relatively small, more points will be selected to participate in the calculation to improve the accuracy of the registration.

[0038] S714. Determine whether to continue iteration by judging whether the registration error has reached a set threshold. If the registration error is small, stop iteration and output the final registration result; if the registration error is still large, continue iteration.

[0039] Preferably, step S72 includes the following steps:

[0040] S721, extracting feature points from the reference point cloud and the target point cloud respectively;

[0041] S722, using the BNN algorithm to match the points with the same name in the reference point cloud and the target point cloud, where the two points with the same name must be each other's nearest neighbors;

[0042] S723. After matching the points with the same name, calculate the normal vector for each point;

[0043] S724, same name point pair c i =(p i ,q i ), where p i ,q i The corresponding normal vectors are v pi ,v qi , using formula 2 for the pair of points c i Assign weights, where formula 2: w i =v pi ·v qi ;

[0044] S725. Calculate the average weight of all points with the same name based on the weight value, and use it as the weight of the final matching result.

[0045] Preferably, step S73 includes the following steps:

[0046] S731, after the matching of the same-name point pairs is completed, i =(p i ,q i ) Calculate the cosine value, p i ,q i The vector connecting two points is represented by v i , for any two pairs of points c i ,c j , the cosine of the angle between the connecting vectors is v i ·v j ;

[0047] S732, for any point c i , use formula 3 to statistically score its compatibility with all remaining point pairs,

[0048] Formula 3: in, The weight is p i ,q i The Euler distance between two points is inversely proportional to d max represents the maximum Euler distance among all pairs of points with the same name;

[0049] S733, for all scores {s i} Perform statistics and calculate the corresponding mean μ and variance σ;

[0050] S734. A threshold ε is set by using the mean μ and the variance σ. Matches with scores less than the threshold ε are determined to be incorrect matches and filtered out.

[0051] After adopting the above technical solution, the beneficial effects of the present invention are:

[0052] Since the present invention discloses a hybrid tracking and registration method, specifically, it includes an offline training stage and an online recognition stage; wherein, the offline training stage includes obtaining reference images of a three-dimensional model from multiple angles to establish an information set of the model, extracting and storing the directional gradient and normal vector features of the image, and obtaining the corresponding model point cloud; the online recognition stage includes collecting and repairing the depth image, extracting the directional gradient and normal vector features of the depth image, using the improved linear parallel multimodal LineMod template matching method, calculating the directional gradient and normal vector features from the depth image to describe the contour and shape of the object, and obtaining key frames from the depth image, matching them with the stored image, obtaining the camera pose, using the improved ICP algorithm to align the environment point cloud and the model point cloud, determining the model pose information, and accurately estimating the camera pose. Directional gradients are calculated from the depth map to describe the contour and shape, and the direction of the surface normal is extracted from the depth image to describe the object shape, which improves the model recognition rate and realizes the coarse positioning and segmentation of the target workpiece. At the same time, an improved ICP algorithm is used to give higher weights to correct matches, effectively filter out incorrect matches, and use an error function that combines point-to-point and point-to-surface to improve the robustness and accuracy of the algorithm. When only some points are screened for alignment, the algorithm time consumption and computational overhead are significantly reduced. When only some points are screened for alignment, the accuracy similar to that of the ICP algorithm is still maintained. Therefore, it has good real-time and robustness, and has high alignment accuracy and computational efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The present invention will be further described below with reference to the accompanying drawings and examples.

[0054] Figure 1 This is an overall flow chart of a hybrid tracking and registration method according to an embodiment of the present invention;

[0055] Figure 2 This is a flowchart of a hybrid tracking and registration method according to an embodiment of the present invention;

[0056] Figure 3 yes Figure 1 Flow chart of the linear parallel multimodal LineMod template matching method;

[0057] Figure 4 yes Figure 1 Flowchart of the ICP algorithm. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0059] The terms "front", "back", "left", "right", "middle", etc. used in this specification are only for the convenience of description and are not intended to limit the scope of the present invention. Changes or adjustments to their relative relationships should be regarded as within the scope of the present invention without substantially changing the technical content.

[0060] like Figures 1 to 3 As shown, the present invention discloses a hybrid tracking and registration method, which includes an offline training phase and an online recognition phase; wherein the offline training phase includes the following steps:

[0061] S10, obtaining reference images of the three-dimensional model from multiple angles through computer-aided design (CAD), and establishing an information set of the model;

[0062] S20, extracting and storing directional gradient and normal vector features of the image;

[0063] S30, obtaining the corresponding model point cloud;

[0064] The online recognition phase includes the following steps:

[0065] S40, collecting and repairing the depth image;

[0066] S50, extracting directional gradient and normal vector features of the depth image;

[0067] S60. Using the improved linear parallel multimodal LineMod template matching method, the directional gradient and normal vector features are calculated from the depth image to describe the contour and shape of the object, and key frames are obtained from the depth image, matched with the stored image, and the camera pose is obtained;

[0068] S70, using the improved ICP algorithm to register the environment point cloud and the model point cloud, determine the model pose information, and accurately estimate the camera pose;

[0069] In step S70, the following steps are included:

[0070] S71, use random sampling algorithm to simplify the environmental point cloud;

[0071] S72, matching points with the same name, assigning weights to pairs of points with the same name;

[0072] S73. Filter mismatches of points with the same name.

[0073] It should be noted that step S10 includes the following steps:

[0074] S11. Model the three-dimensional model using CAD software;

[0075] S12. Setting views at different angles in the CAD software and taking screenshots or saving them as pictures. Preferably, the pictures include a top view, a front view, a side view, and a 45-degree view.

[0076] S13, importing the image into the image processing algorithm and performing preprocessing operations, the preprocessing operations including denoising, filtering, and rotation adjustment;

[0077] S14, extracting directional gradient and normal vector features, wherein the directional gradient is calculated using a gradient operator such as a Canny operator or a Sobel operator, and the normal vector feature is calculated using an algorithm in a point cloud library (such as PCL, Open3D, etc.);

[0078] S15. For each reference image, an information set is established and stored. When establishing the information set, relevant attributes of the current reference image, such as image ID, directional gradient, and normal vector features, need to be recorded.

[0079] In step S40, the following steps are included:

[0080] S41, preprocessing the depth image, including removing invalid pixels and eliminating noise;

[0081] S42: Repairing the pre-processed depth image, including but not limited to using interpolation, filling or correction algorithms to repair the depth information.

[0082] In step S60, the improved linear parallel multimodal LineMod template matching method includes the following steps:

[0083] S61, input depth image;

[0084] S62, blurring the depth image using a 7×7 Gaussian filter to remove depth jump pixels;

[0085] S63, using a 5×5 Sobel operator to perform gradient calculation on the blurred depth image to obtain the directional gradient and gradient value of each pixel in the image;

[0086] S64, selecting a pixel whose gradient value is greater than a set threshold, and quantizing the directional gradient of the pixel;

[0087] S65, combining the quantized normal direction of the depth image as a feature to create a template;

[0088] S66. Matching is performed based on the template.

[0089] Wherein, in step S63, the following steps are included:

[0090] S631, converting the depth image into a grayscale image;

[0091] S632, performing convolution calculations on the grayscale image using Sobel operators in the horizontal and vertical directions, respectively, to obtain a grayscale image in the horizontal direction and a grayscale image in the vertical direction;

[0092] S633. Obtain edge information of the grayscale image using the formula G(I)=sqrt(Gx(I)^2+Gy(I)^2), where Gx(I) and Gy(I) represent the grayscale values of the image after horizontal and vertical edge detection, respectively.

[0093] S634. Calculate the direction angle using the formula θ=atan2(Gy(I),Gx(I)), where the atan2 function is an inverse tangent function, and the value it returns is a radian value between [-π,π].

[0094] Directional gradients are calculated from the depth image to describe the contour and shape, and the direction of the surface normal is extracted from the depth image to describe the object shape, which improves the model recognition rate and realizes the coarse positioning and segmentation of the target workpiece.

[0095] In the present invention, step S71 includes the following steps:

[0096] S711, calculating the bounding boxes of the target point cloud and the reference point cloud, and calculating the relative position and posture information to obtain a preliminary registration result;

[0097] S712. In a new iteration, adaptively adjust the number of sampling points according to the current registration error.

[0098] S713, use formula 1 to perform registration. In each iteration, by matching the sampling point and the nearest neighbor point on the reference point cloud, the transformation matrix is calculated to update the position and posture information of the target point cloud.

[0099] Formula 1:

[0100] Where RMSE is the root mean square error of the last iteration, d is the density of the point cloud, N is the size of the point cloud, Where x is the screening coefficient. When the RMSE error is relatively large, it means that there are many noise points in the current point cloud. Therefore, fewer points will be selected to participate in the calculation to speed up the convergence. When the RMSE error is relatively small, more points will be selected to participate in the calculation to improve the accuracy of the registration.

[0101] S714. Determine whether to continue iteration by judging whether the registration error has reached a set threshold. If the registration error is small, stop iteration and output the final registration result; if the registration error is still large, continue iteration.

[0102] Wherein, in step S72, the following steps are included:

[0103] S721, extracting feature points from the reference point cloud and the target point cloud respectively;

[0104] S722, using the BNN algorithm to match the points with the same name in the reference point cloud and the target point cloud, where the two points with the same name must be each other's nearest neighbors;

[0105] S723. After matching the points with the same name, calculate the normal vector for each point;

[0106] S724, same name point pair c i =(p i ,q i ), where p i ,q i The corresponding normal vectors are v pi ,v qi , using formula 2 for the pair of points c i Assign weights, where formula 2: w i =v pi ·v qi ;

[0107] S725. Calculate the average weight of all points with the same name based on the weight value, and use it as the weight of the final matching result.

[0108] Preferably, step S73 includes the following steps:

[0109] S731, after the matching of the same-name point pairs is completed, i =(p i ,q i ) Calculate the cosine value, p i ,q i The vector connecting two points is represented by v i , for any two pairs of points c i ,c j , the cosine of the angle between the connecting vectors is v i ·v j ;

[0110] S732, for any point c i , use formula 3 to statistically score its compatibility with all remaining point pairs,

[0111] Formula 3: in, The weight is p i ,q i The Euler distance between two points is inversely proportional to d max represents the maximum Euler distance among all pairs of points with the same name;

[0112] S733, for all scores {s i} Perform statistics and calculate the corresponding mean μ and variance σ;

[0113] S734. A threshold ε is set by using the mean μ and the variance σ. Matches with scores less than the threshold ε are determined to be incorrect matches and filtered out.

[0114] An improved ICP algorithm is used to assign higher weights to correct matches, effectively filter out incorrect matches, and use a point-to-point and point-to-surface error function to improve the robustness and accuracy of the algorithm. When only some points are screened for registration, the algorithm time consumption and computational overhead are significantly reduced. When only some points are screened for registration, the accuracy is still similar to that of the ICP algorithm. Therefore, the algorithm has good real-time and robustness, and has high registration accuracy and computational efficiency.

[0115] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A hybrid tracking and registration method, characterized in that: It includes an offline training phase and an online recognition phase; wherein the offline training phase includes the following steps: S10, obtaining reference images of the three-dimensional model from multiple angles through computer-aided design (CAD), and establishing an information set of the model; S20, extracting and storing directional gradient and normal vector features of the image; S30, obtaining the corresponding model point cloud; The online recognition phase includes the following steps: S40, collecting and repairing the depth image; S50, extracting directional gradient and normal vector features of the depth image; S60. Using the improved linear parallel multimodal LineMod template matching method, the directional gradient and normal vector features are calculated from the depth image to describe the contour and shape of the object, and key frames are obtained from the depth image, matched with the stored image, and the camera pose is obtained; In step S60, the improved linear parallel multimodal LineMod template matching method includes the following steps: S61, input depth image; S62, blurring the depth image using a 7×7 Gaussian filter to remove depth jump pixels; S63, using a 5×5 Sobel operator to perform gradient calculation on the blurred depth image to obtain the directional gradient and gradient value of each pixel in the image; S64, selecting a pixel whose gradient value is greater than a set threshold, and quantizing the directional gradient of the pixel; S65, combining the quantized normal direction of the depth image as a feature to create a template; S66, matching based on template; S70, using the improved ICP algorithm to register the environment point cloud and the model point cloud, determine the model pose information, and accurately estimate the camera pose; In step S70, the following steps are included: S71, use random sampling algorithm to simplify the environmental point cloud; S72, matching the points with the same name, and assigning weights to the points with the same name; In step S72, the following steps are included: S721, extracting feature points from the reference point cloud and the target point cloud respectively; S722, using the BNN algorithm to match the points with the same name in the reference point cloud and the target point cloud, where the two points with the same name must be each other's nearest neighbors; S723. After matching the points with the same name, calculate the normal vector for each point; S724, same name point pair c i =(p i ,q i ), where p i ,q i The corresponding normal vectors are v pi ,v qi , using formula 2 for the pair of points c i Assign weights, where formula 2: w i =v pi ·v qi ; S725. Calculate the average weight of all points with the same name based on the weight value, and use it as the weight of the final matching result; S73, filtering out mismatched pairs of homonymous points; In step S73, the following steps are included: S731, after the matching of the same-name point pairs is completed, i =(p i ,q i ) Calculate the cosine value, p i ,q i The vector connecting two points is represented by v i , for any two pairs of points c i ,c j , the cosine of the angle between the connecting vectors is v i ·v j ; S732, for any point c i , use formula 3 to statistically score its compatibility with all remaining point pairs, Formula 3: in, The weight is p i ,q i The Euler distance between two points is inversely proportional to d max represents the maximum Euler distance among all pairs of points with the same name; S733, for all scores {s i } Perform statistics and calculate the corresponding mean μ and variance σ; S734. A threshold ε is set by using the mean μ and the variance σ. Matches with scores less than the threshold ε are determined to be incorrect matches and filtered out.

2. The hybrid tracking and registration method according to claim 1, wherein: In step S63, the following steps are included: S631, converting the depth image into a grayscale image; S632, performing convolution calculations on the grayscale image using Sobel operators in the horizontal and vertical directions, respectively, to obtain a grayscale image in the horizontal direction and a grayscale image in the vertical direction; S633. Obtain edge information of the grayscale image using the formula G(I)=sqrt(Gx(I)^2+Gy(I)^2), where Gx(I) and Gy(I) represent the grayscale values of the image after horizontal and vertical edge detection, respectively. S634. Calculate the direction angle using the formula θ=atan2(Gy(I),Gx(I)), where the atan2 function is an inverse tangent function, and the value it returns is a radian value between [-π,π].

3. The hybrid tracking and registration method according to claim 1, wherein: In step S71, the following steps are included: S711, calculating the bounding boxes of the target point cloud and the reference point cloud, and calculating the relative position and posture information to obtain a preliminary registration result; S712. In a new iteration, adaptively adjust the number of sampling points according to the current registration error. S713, use formula 1 to perform registration. In each iteration, by matching the sampling point and the nearest neighbor point on the reference point cloud, the transformation matrix is calculated to update the position and posture information of the target point cloud. Formula 1: Where RMSE is the root mean square error of the last iteration, d is the density of the point cloud, N is the size of the point cloud, Where x is the screening coefficient. When the RMSE error is relatively large, it means that there are many noise points in the current point cloud. Therefore, fewer points will be selected to participate in the calculation to speed up the convergence. When the RMSE error is relatively small, more points will be selected to participate in the calculation to improve the accuracy of the registration. S714. Determine whether to continue iteration by judging whether the registration error has reached a set threshold. If the registration error is small, stop iteration and output the final registration result; if the registration error is still large, continue iteration.

Images