A robot vision three-dimensional positioning millimeter-level error correction method and system based on interior point robust screening and compensation matrix learning
Patent Information
- Application Number
- CN202610923720.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-06-25
- Publication Date
- 2026-10-13
AI Technical Summary
[0006]发明目的：为克服现有机器人视觉定位方法中手眼标定残余误差难以消除、深度点噪声影响大、离群点易导致补偿失效以及工业现场重新标定成本高的问题，本发明提供一种基于内点鲁棒筛选与补偿矩阵学习的机器人视觉三维定位毫米级误差校正方法及系统
本发明不要求全部三维点对均准确，而是利用RANSAC筛选满足空间刚体一致性的内点。该设计符合工业现场实际情况，因为真实场景中常存在深度异常、遮挡、反光、误匹配和测量噪声。只要存在足够数量的可信内点，即可学习稳定的补偿矩阵。
Smart Images
Figure CN122900608A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to robot visual positioning, RGB-D 3D perception, hand-eye calibration error compensation, 3D point cloud registration, and precision operation technology methods and systems for industrial robots. In particular, it relates to a method and system for correcting millimeter-level errors in robot visual 3D positioning based on robust interior point screening and compensation matrix learning. Background Technology
[0002] In tasks such as grasping, assembly, inspection, loading and unloading, grinding, welding, and visual servoing, industrial robots typically need to use cameras to acquire the 3D position of target objects or key points, and then transform the 3D points in the camera coordinate system to the robot's base coordinate system using hand-eye calibration. Theoretically, as long as the camera intrinsic parameters, depth data, and hand-eye transformation relationship are accurate, the robot can complete the positioning operation based on the transformed spatial coordinates.
[0003] However, in real-world industrial environments, robot vision positioning errors are often difficult to completely eliminate. Sources of these errors include, but are not limited to: camera depth noise, RGB-D edge depth distortion, calibration board corner point extraction errors, residual errors in the hand-eye calibration matrix, minute camera mounting offsets, robot repetitive positioning errors, target object occlusion, localized reflections, missing point clouds, and 3D point matching errors. These errors can cause millimeter- to centimeter-level deviations in camera coordinates when transformed to the robot's base coordinate system.
[0004] In existing technologies, common practices include recalibrating hand-eye connections, increasing the number of calibration images, improving camera accuracy, or employing complex point cloud registration algorithms to reduce errors. However, these methods still have the following shortcomings: First, the recalibration process is cumbersome and the downtime cost is high, making it unsuitable for frequent adjustments in industrial settings. Second, some errors do not come purely from hand-eye calibration, but from the combined effects of depth camera, mounting structure, mechanical system and environmental factors, which cannot be completely eliminated by recalibration alone; Third, traditional least squares fitting methods are easily affected by outliers. When there are a few mismatched points or deep outliers, the overall compensation matrix will shift. Fourth, existing methods usually aim to use all points for fitting, but in industrial settings, not all points are reliable, and forcing erroneous points to participate can actually reduce the accuracy of compensation. Fifth, existing visual positioning systems lack an interpretable and reusable millimeter-level error compensation mechanism for engineering applications.
[0005] Therefore, a method is needed to automatically select reliable interior points from 3D point pair data containing noise and outliers, and learn a stable compensation matrix based on these interior points. This method should not rely on all points being accurate, but rather establish a consistent rigid body compensation relationship on the set of reliable interior points, thereby achieving millimeter-level correction of the robot's visual localization results. Summary of the Invention
[0006] Objective: To overcome the problems of difficult elimination of residual hand-eye calibration errors, significant influence of depth point noise, easy compensation failure due to outliers, and high recalibration costs in industrial settings in existing robot vision localization methods, this invention provides a millimeter-level error correction method and system for robot vision 3D localization based on robust interior point screening and compensation matrix learning. The core idea of this invention is: instead of assuming all 3D point pairs are reliable, it automatically obtains a set of interior points that satisfy spatial rigidity consistency through robust screening mechanisms such as RANSAC, and primarily utilizes these interior points to learn the compensation matrix. Since interior points themselves represent the stable part of the system error, even if a certain proportion of outliers exist in the original data, a stable, interpretable, and suitable compensation matrix for online correction can be obtained.
[0007] The method of this invention first obtains 3D points in the camera coordinate system, the original points in the robot base coordinate system obtained through initial hand-eye relationship transformation, and the corresponding real base coordinate points; then, it constructs 3D point pair relationships between the original base points and the real base points; further, it employs a RANSAC interior point filtering strategy based on rigid body transformation to remove outlier interference points, and learns a compensation matrix on the interior point set; finally, it uses the compensation matrix to perform online correction on subsequent robot visual localization results. This method does not require all points to be high-quality matching points, but rather completes stable compensation matrix estimation by filtering interior points with consistent geometric relationships, which can effectively suppress the influence of depth noise, occlusion, mismatches, and abnormal annotations. Experimental verification shows that in the 3D point pair verification scenario constructed using the BOP-LMO dataset, the RMSE of the verification set before compensation is 17.289 mm, and the RMSE of all points after compensation is reduced to 5.373 mm; on the 3 mm threshold interior point set, the RMSE is reduced to 1.361 mm, the average error is 1.256 mm, the maximum error is 2.888 mm, and the verification interior point rate reaches 91.11%. This invention can achieve millimeter-level positioning error correction required for industrial applications and is suitable for scenarios such as robot gripping, assembly, inspection, loading and unloading, and visual servoing.
[0008] Technical Solution: This invention provides a method for correcting millimeter-level errors in robot vision 3D localization based on interior-point robust screening and compensation matrix learning, comprising the following steps: S1: Obtain the 3D point set in the camera coordinate system Based on the initial hand-eye transformation relationship, the three-dimensional point set is converted into the original base point set in the robot base coordinate system. ; S2: Obtain the set of real base coordinate points corresponding to the original base point set. Construct a set of point pairs between the original base point and the real base point (in this invention, the real base coordinate point is a reference reference point obtained by a high-precision external measuring device). S3: Using the original base point set The source point set is the set of coordinate points of the actual base. For the target point set, establish a rigid body compensation model: ; in, Let be the compensation matrix to be solved; S4: The random sampling consensus algorithm is used to screen the point pair set for interior points. Each time, no less than three non-collinear point pairs are randomly selected from the point pair set, the candidate rigid body compensation matrix is calculated, and the residual of each point pair is calculated based on the candidate compensation matrix. Point pairs with residuals less than a preset threshold are determined as interior points. After multiple random samplings, the interior point set corresponding to the candidate model with the most interior points is selected as the screening result. S5: Based on the selected set of interior points, solve for the rotation matrix and translation vector using singular value decomposition to obtain the compensation matrix. ; S6: Apply the compensation matrix to the subsequent robot vision positioning results to correct the original base coordinate points and obtain the compensated base coordinate points.
[0009] Further, in step S3, the compensation matrix Here is a fourth-order homogeneous transformation matrix: ; in, It is a three-dimensional rotation matrix. It is a three-dimensional translation vector.
[0010] Further, in step S4, the formula for calculating the residual is: ; when When the value is less than the preset threshold, determine the first... Each pair of points is an interior point.
[0011] Furthermore, the preset threshold is 1 mm to 5 mm, preferably 3 mm.
[0012] Furthermore, in step S4, the random sampling consensus algorithm iterates no less than 5000 times.
[0013] Furthermore, in step S5, the specific steps for solving the compensation matrix based on the set of interior points using singular value decomposition include: Calculate the centroids of the set of points A inside the source point and the set of points B inside the target point: ; After decentering the two point sets, construct the covariance matrix: ; Perform singular value decomposition on H: ; Calculate the rotation matrix: ; like Then, the singular value decomposition result is corrected by reflection; The translation vector is: ; The compensation matrix T_comp is formed by the rotation matrix and translation vector.
[0014] Furthermore, the method does not use all point pairs as the basis for solving the compensation matrix, but uses the internal point pairs that satisfy the spatial rigid body consistency after being screened by the random sampling consistency algorithm as the main basis for solving, thereby reducing the influence of outliers on the compensation matrix.
[0015] Furthermore, the three-dimensional point set in the camera coordinate system is obtained by an RGB-D camera, a structured light camera, a binocular camera, a ToF camera, or a three-dimensional vision sensor; The set of real base coordinate points is obtained through robot teaching, standard tooling, precision measuring equipment, known CAD models or calibration reference systems; the method is applied to robot grasping, robot assembly, robot loading and unloading, visual inspection, hand-eye calibration error compensation, end effector positioning compensation or robot visual servo control.
[0016] Another aspect of the present invention provides a robot vision 3D localization error correction system based on interior point robust screening and compensation matrix learning, comprising: The 3D point acquisition module is used to acquire 3D points in the camera coordinate system; The initial coordinate transformation module is used to convert the camera coordinate points into the original base points in the robot base coordinate system according to the initial hand-eye transformation relationship; The real reference point acquisition module is used to acquire the real base coordinate points corresponding to the original base points; The interior point filtering module is used to filter interior point pairs that satisfy rigid body consistency based on the random sampling consistency algorithm; The compensation matrix learning module is used to solve the compensation matrix based on the internal point pairs; An online compensation module is used to correct the robot's visual positioning results in real time using the compensation matrix; The error assessment module is used to calculate the average error, root mean square error, maximum error, and interior point rate before and after compensation.
[0017] Furthermore, it is used to calculate the average error, root mean square error, maximum error, and inlier rate before and after compensation, and outputs the compensation matrix, residual table, outlier list, and coordinates of points after compensation.
[0018] Compared with the prior art, the present invention has the following significant advantages.
[0019] 1. Focus on trusted interior points, rather than blindly relying on all points. This invention does not require all 3D point pairs to be accurate; instead, it utilizes RANSAC to filter interior points that satisfy spatial rigidity consistency. This design aligns with real-world industrial scenarios, as depth anomalies, occlusion, reflections, mismatches, and measurement noise are common in real-world environments. As long as a sufficient number of reliable interior points exist, a stable compensation matrix can be learned.
[0020] 2. Capable of achieving millimeter-level positioning compensation in industrial applications. Experimental results show that, in the 3D point pair validation set constructed using the BOP-LMO dataset, the RMSE of the validation set before compensation was 17.289 mm, and after compensation, the RMSE of all points decreased to 5.373 mm. On the point set within the 3 mm threshold, the RMSE decreased to 1.361 mm, with an average error of 1.256 mm and a maximum error of 2.888 mm. These results demonstrate that the present invention can meet the millimeter-level positioning compensation requirements commonly encountered in industrial applications.
[0021] 3. It exhibits strong robustness to outliers. In the verification experiment, the original data contained simulated noise and outliers, with an in-point rate of 91.11% in the verification set. This invention avoids outliers dominating the solution of the compensation matrix by screening in-points, making the compensation results more stable and reliable.
[0022] 4. The compensation matrix has a clear physical meaning. The compensation matrix obtained by this invention consists of a rotation matrix and a translation vector, and can be interpreted as a unified correction for the initial hand-eye transformation, camera installation error, and systematic spatial bias. Compared with black-box neural network methods, this matrix is easier to deploy, debug, and verify.
[0023] 5. Low engineering deployment cost This invention eliminates the need for frequent hand-eye recalibration and does not rely on expensive, high-precision external measurement systems. Once the compensation matrix is learned using a small number of training points, it can be used for subsequent online corrections, making it suitable for deployment in robot workstations, vision-guided grasping systems, and industrial inspection units.
[0024] 6. Wide range of applications This invention is applicable to various types of 3D vision sensors, such as RGB-D cameras, structured light cameras, binocular cameras, and ToF cameras. It can also be applied to industrial robots and collaborative robots of different brands, and has good versatility and scalability. Attached Figure Description
[0025] Figure 1 This is a flowchart illustrating the overall process of robot vision 3D positioning error compensation according to the present invention. Figure 2 A schematic diagram is constructed for the data between the camera coordinate points, the original base points, and the real base points; Figure 3 A flowchart for RANSAC-based interior point robust selection and compensation matrix learning. Figure 4 Flowchart for online application of compensation matrix; Figure 5 This is a diagram showing the comparison of results before and after error compensation. Detailed Implementation
[0026] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions of the present invention will be further described below.
[0027] Example 1: The robot vision 3D localization millimeter-level error correction method based on interior point robust screening and compensation matrix learning provided in this example includes the following steps.
[0028] S1: Construct camera coordinates The 3D coordinates of the target point in the camera coordinate system are obtained through an RGB-D camera, structured light camera, binocular camera, ToF camera, or other 3D vision sensors, and denoted as: ; in, Number the points.
[0029] In this embodiment, the three-dimensional coordinates can be obtained through key points of the target object model, point cloud feature points, detection box center points, mechanical positioning points, or manual calibration points.
[0030] S2: Generate the original base point Based on the initial hand-eye transformation matrix Convert 3D points in the camera coordinate system to original points in the robot base coordinate system: ; The origin point contains a systematic deviation caused by a combination of hand-eye calibration residual error, camera depth error, and mechanical installation error.
[0031] S3: Obtain the actual base point Obtain the actual base coordinates corresponding to the original base points through robot teaching, standard tooling, precision measuring equipment, known CAD models, simulation ground truth, or calibration reference systems:
[0032] This forms a point pair: ; The actual base coordinates are obtained through high-precision external measuring equipment (such as laser trackers), robot precision teaching, or standard tooling, and serve as a reference point for error correction. Their accuracy should be higher than the original positioning accuracy to be corrected.
[0033] S4: Establish the compensation matrix model Construct the compensation relationship from the original base point to the actual base point: ; in: ; To compensate for the rotation matrix, To compensate for the translation vector.
[0034] S5: RANSAC Interior Point Robust Screening Randomly sample at least three non-collinear point pairs from the set of point pairs, calculate the candidate rigid body compensation matrix, and calculate the residuals for all points: ; When the residual If the value is less than a preset threshold, the point pair is identified as an inner point; otherwise, it is identified as an outer point. The preset threshold can be set from 1 mm to 5 mm according to the industrial positioning accuracy requirements, preferably 3 mm.
[0035] After multiple random samplings, the candidate model with the largest number of interior points and the smallest root mean square error of interior points was selected as the initial robust model.
[0036] S6: Solving the compensation matrix based on interior points The optimal rotation matrix and translation vector are solved by using the Kabsch algorithm or the singular value decomposition rigid body registration algorithm on the set of interior points obtained by RANSAC screening.
[0037] Let the set of interior points of the source point be... The set of points inside the target point is Calculate the centroid separately: ; ; After decentralization, the covariance matrix is constructed as follows: ; right Perform singular value decomposition: ; The rotation matrix is then: ; like Then, a reflection correction is applied to the singular value decomposition result. The translation vector is: ; The final compensation matrix is obtained as follows: ; S7: Optional interior point quadratic optimization Based on the RANSAC inlier set, a small number of inliers with large residuals are enumerated and iteratively removed. After each candidate point is removed, the compensation matrix is refitted. If the root mean square error of the inliers decreases by more than a preset improvement threshold, the removal result is accepted. This step is used to further reduce the impact of boundary outliers on matrix estimation.
[0038] S8: Online Compensation Application For a new robot visual localization point, the initial base point is first obtained through the initial hand-eye relationship. Then, the corrected base points are obtained through the compensation matrix: ; The robot control system performs grasping, assembly, inspection, or visual servoing tasks based on the corrected base point.
[0039] Example 2: Construction of 3D point pairs based on the BOP-LMO dataset This embodiment uses the BOP-LMO dataset to construct verification data for robot visual localization error compensation. The dataset includes RGB images, depth images, camera intrinsic parameters, a 3D model of the object, and the object's true pose in the camera coordinate system.
[0040] The camera intrinsic parameter matrix is: ; The depth map resolution is The depth unit is millimeters. The translation vectors of object poses in the dataset are in millimeters, but are uniformly converted to meters during the calculation process.
[0041] This embodiment extracts key points from the bounding box of the object's 3D model, including 8 bounding box corner points, 1 center point, and 6 face center points, for a total of 15 3D key points. Based on the rotation matrix and translation vector from the object to the camera coordinate system provided in the dataset, the local key points of the object are converted into camera coordinate points. in, This represents the rotation matrix from the object coordinate system to the camera coordinate system. This represents the translation vector from the object coordinate system to the camera coordinate system.
[0042] Example 3: Generation of Original Base Points and Real Base Points To verify the compensation algorithm, this embodiment sets the transformation matrix from the real camera to the robot base. And set the original transformation matrix with systematic errors. The true base point is obtained by the following formula: ; The original base point is obtained by the following formula: ; in, It is Gaussian noise. The outlier is a random outlier. In this embodiment, the standard deviation of the Gaussian noise is 0.8 mm, the outlier probability is 8%, and the outlier range is 8 mm to 25 mm.
[0043] The final result is 300 sample files, each typically containing 15 3D point pairs. The training set contains 210 files, and the validation set contains 90 files. The training set contains 3150 point pairs, and the validation set contains 1350 point pairs.
[0044] Example 4: Compensation Matrix Learning The compensation matrix was learned using 3150 point pairs from the training set. As the source point, with For the target point, interior points are selected using RANSAC. The RANSAC interior point threshold is set to 3.0 mm, and the number of iterations is set to 5000.
[0045] The training results are as follows: ; The compensation matrix obtained through learning is: ; The matrix shows that the systematic errors mainly manifest as small-angle rotational deviations and millimeter- to centimeter-level translational deviations. The compensation matrix can correct the original base point to a position closer to the true base point.
[0046] Example 5: Validation set compensation effect The compensation matrix obtained during training was used to evaluate 1350 point pairs in 90 validation files. The validation set was not involved in the compensation matrix learning and was only used to test the generalization effect.
[0047] The verification results are as follows: ; As shown in the table above, the RMSE of the validation set before compensation was 17.289 mm; after the compensation matrix correction, the RMSE of all points decreased to 5.373 mm, a reduction of 68.92%. If an inlier threshold of 3 mm is used to filter inliers, the RMSE of inliers decreases to 1.361 mm, a reduction of approximately 92.13% compared to before compensation.
[0048] These results demonstrate that the present invention does not rely on all points being reliable, but rather achieves stable millimeter-level error correction on a set of reliable interior points. Even with approximately 8% outliers in the data, the compensation matrix still maintains good generalization ability and engineering applicability.
[0049] Example 6: Industrial Application In a real robot workstation, this invention can be deployed in the following manner: The first step is for the robot vision system to acquire several three-dimensional feature points on the target object or tooling; The second step is to obtain the original base coordinates based on the initial hand-eye calibration matrix; The third step is to obtain some of the actual base coordinates through robot teaching points, standard tooling, or precision measuring equipment. The fourth step is to learn the compensation matrix using the method of this invention; Fifth, in subsequent operations, real-time compensation is performed on all base coordinates output by the vision system: ; The sixth step involves the robot performing grasping, insertion, detection, or trajectory tracking based on the compensated coordinates.
[0050] This deployment method eliminates the need for complete hand-eye calibration each time, and can quickly restore positioning accuracy when there are slight changes in camera installation position, system deviations in depth cameras, or long-term drift in robot vision positioning.
[0051] in conclusion This invention proposes a method and system for correcting millimeter-level errors in robot vision 3D localization based on interior point robust screening and compensation matrix learning. Its core contribution lies in: identifying stable and reliable spatial geometric relationships through interior point screening, and learning a rigid body compensation matrix with clear physical meaning based on interior points, thereby achieving millimeter-level correction of robot vision localization results even in the presence of noise and outliers.
[0052] Experiments show that this invention can reduce the RMSE from 17.289 mm before validation set compensation to 5.373 mm across all points, and further reduce it to 1.361 mm on the interior set. This method combines robustness, interpretability, ease of deployment, and industrial applicability, making it suitable for scenarios such as robot grasping, assembly, inspection, visual servoing, and residual error compensation in hand-eye calibration.
[0053] The above are merely preferred embodiments of the present invention and do not constitute any limitation on the present invention. Any equivalent substitutions or modifications made by those skilled in the art to the technical solutions and content disclosed in the present invention without departing from the scope of the present invention shall be deemed to have remained within the protection scope of the present invention.
Claims
1. A method for correcting millimeter-level errors in robot vision 3D localization based on interior-point robust screening and compensation matrix learning, characterized in that, Includes the following steps: S1: Obtain the 3D point set in the camera coordinate system Based on the initial hand-eye transformation relationship, the three-dimensional point set is converted into the original base point set in the robot base coordinate system. ; S2: Obtain the original base point set. A set of real base coordinate points that correspond one-to-one And construct a set of three-dimensional point pairs between the original base points and the real base points; S3: Using the original base point set The source point set is the set of coordinate points of the actual base. For the target point set, establish a rigid body compensation model consisting of rotation and translation: ； in, Let be the compensation matrix to be solved; S4: The random sampling consensus algorithm is used to screen the point pair set for interior points. Each time, no less than three non-collinear point pairs are randomly selected from the point pair set, the candidate rigid body compensation matrix is calculated, and the residual of each point pair is calculated based on the candidate compensation matrix. Point pairs with residuals less than a preset threshold are determined as interior points. After multiple random samplings, the interior point set corresponding to the candidate model with the most interior points is selected as the screening result. S5: Based on the selected set of interior points, solve for the rotation matrix and translation vector using singular value decomposition to obtain the compensation matrix. ; S6: Apply the compensation matrix to the subsequent robot vision positioning results to correct the original base coordinate points and obtain the compensated base coordinate points.
2. The method according to claim 1, characterized in that, In step S3, the compensation matrix Here is a fourth-order homogeneous transformation matrix: ； in, It is a three-dimensional rotation matrix. It is a three-dimensional translation vector.
3. The method according to claim 1, characterized in that, In step S4, the formula for calculating the residual is: ； when When the value is less than the preset threshold, determine the first... Each pair of points is an interior point.
4. The method according to claim 3, characterized in that, The preset threshold is 1 mm to 5 mm, preferably 3 mm.
5. The method according to claim 1, characterized in that, In step S4, the random sampling consensus algorithm iterates no less than 5000 times.
6. The method according to claim 1, characterized in that, In step S5, the specific steps for solving the compensation matrix based on the set of interior points using singular value decomposition include: Calculate the centroids of the set of points A inside the source point and the set of points B inside the target point: ； ； After decentering the two point sets, construct the covariance matrix: ； Perform singular value decomposition on H: ； Calculate the rotation matrix: ； like Then, the singular value decomposition result is corrected by reflection; The translation vector is: ； The compensation matrix T_comp is formed by the rotation matrix and translation vector.
7. The method according to claim 1, characterized in that, The method does not use all point pairs as the basis for solving the compensation matrix, but uses the internal point pairs that satisfy the spatial rigid body consistency after being screened by the random sampling consistency algorithm as the main basis for solving, thereby reducing the influence of outliers on the compensation matrix.
8. The method according to claim 1, characterized in that, The three-dimensional point set in the camera coordinate system is obtained by an RGB-D camera, a structured light camera, a binocular camera, a ToF camera, or a three-dimensional vision sensor; The set of real base coordinate points is obtained through robot teaching, standard tooling, precision measuring equipment, known CAD models or calibration reference systems; the method is applied to robot grasping, robot assembly, robot loading and unloading, visual inspection, hand-eye calibration error compensation, end effector positioning compensation or robot visual servo control.
9. A robot vision 3D localization error correction system based on interior point robust screening and compensation matrix learning, characterized in that, include: The 3D point acquisition module is used to acquire 3D points in the camera coordinate system; The initial coordinate transformation module is used to convert the camera coordinate points into the original base points in the robot base coordinate system according to the initial hand-eye transformation relationship; The real reference point acquisition module is used to acquire the real base coordinate points corresponding to the original base points; The interior point filtering module is used to filter interior point pairs that satisfy rigid body consistency based on the random sampling consistency algorithm; The compensation matrix learning module is used to solve the compensation matrix based on the internal point pairs; An online compensation module is used to correct the robot's visual positioning results in real time using the compensation matrix; The error assessment module is used to calculate the average error, root mean square error, maximum error, and interior point rate before and after compensation.
10. The system according to claim 9, characterized in that, It is used to calculate the average error, root mean square error, maximum error, and inlier rate before and after compensation, and outputs the compensation matrix, residual table, outlier list, and coordinates of points after compensation.