Robot calibration method and device based on fusion algorithm, collaborative robot and storage medium

Abstract

The application relates to the technical field of robot calibration control, in particular to a robot calibration method and device based on a fusion algorithm, a collaborative robot and a storage medium. The method comprises the following steps: determining a plurality of calibration positions required to be reached by a robot, and determining a motion path of the robot for reaching the calibration positions in sequence; collecting image data of a calibration board and determining pose data of the robot at a current calibration position, and generating a calibration sample pair corresponding to the current calibration position based on the pose data and the image data; solving a calibration sample data set in parallel based on a preset fusion algorithm, obtaining a plurality of candidate transformation matrices; and performing multidimensional quality assessment and optimization on each candidate transformation matrix to obtain a target transformation matrix. The application can improve the accuracy, reliability and stability of the calibration result.

Description

Technical Field

[0001] This application relates to the field of robot calibration and control technology, and in particular to robot calibration methods, devices, collaborative robots and storage media based on fusion algorithms. Background Technology

[0002] Hand-eye calibration is a crucial step in vision guidance for industrial robots, used to establish the transformation relationship between the camera coordinate system and the robot coordinate system. The accuracy of the calibration results directly affects the accuracy of subsequent visual localization, path planning, and grasping operations. Existing calibration methods involve operators manually teaching multiple calibration poses, causing the robot to carry the calibration board or camera to different positions and postures. This process is time-consuming, cumbersome, and relies on manual calibration, resulting in low accuracy and efficiency.

[0003] Existing automated calibration methods typically employ a two-step hand-eye calibration algorithm. The calibration results obtained through this method are not sufficiently accurate and stable, leading to unreliable calibration results and affecting calibration precision. Summary of the Invention

[0004] To address the aforementioned issues, embodiments of this application provide a robot calibration method, apparatus, collaborative robot, and storage medium based on a fusion algorithm, which can solve at least one of the aforementioned technical problems.

[0005] According to one aspect of the embodiments of this application, a robot calibration method based on a fusion algorithm is proposed, the method comprising: Determine multiple calibrated locations that the robot needs to reach, and determine the movement path of the robot to reach each calibrated location sequentially based on the nearest neighbor strategy; When the robot reaches any calibration position according to the motion path, image data of the calibration board is collected and pose data of the robot at the current calibration position is determined. Based on the pose data and the image data, a calibration sample pair corresponding to the current calibration position is generated. The calibration sample pairs corresponding to each calibration position are used as the calibration sample dataset. The calibration sample dataset is solved in parallel based on a preset fusion algorithm to obtain multiple candidate transformation matrices. The candidate transformation matrices are subjected to multi-dimensional quality evaluation and optimization to obtain the target transformation matrix. The preset fusion algorithm is generated based on the two-step hand-eye calibration algorithm, the hand-eye calibration algorithm based on dual quaternions, and the Ceres nonlinear optimization method.

[0006] In the above scheme, determining the multiple calibration locations that the robot needs to reach includes: After moving the robot's camera directly above the calibration plate, the position of the camera is used as a reference position. Multiple sets of calibration poses are generated with the reference reference position as the center. Each set of calibration poses is obtained by changing the height and attitude angle of the camera relative to the reference reference position. For each set of calibration poses, the position of the robot when it reaches the calibration pose is taken as the calibration position that the robot needs to reach. Wherein, at the reference reference position, the complete plane of the calibration plate is within the field of view of the camera, and the optical axis of the camera is perpendicular to the complete plane of the calibration plate.

[0007] In the above scheme, generating a calibration sample pair corresponding to the current calibration position based on the pose data and the image data includes: The corner points of the calibration board are determined based on the image data; Acquire depth data for each corner point, the depth data being used to characterize the distance between the corner point and the robot's camera; The interquartile range method is used to remove outlier depth data, resulting in the remaining depth data. The median of the remaining depth data is taken as the final depth estimation result of the image data; Based on the final depth estimation result and the pose data, a calibration sample pair corresponding to the current calibration position is generated.

[0008] In the above scheme, the method of using interquartile range to remove abnormal depth data includes: The lower and upper quartiles of each depth data point are determined using the interquartile range method. The difference between the upper quartile and the lower quartile is taken as the target difference. The lower threshold for removing abnormal depth data is determined based on the lower quartile and the target difference. The upper threshold for removing abnormal depth data is determined based on the upper quartile and the target difference. For each depth data point, if the value of the depth data is less than the lower threshold or greater than the upper threshold, then the depth data is determined to be abnormal depth data.

[0009] In the above scheme, the calibration sample dataset is solved in parallel using a preset fusion algorithm to obtain multiple candidate transformation matrices, including: The rotation matrix and translation vector for the calibration sample dataset are obtained by solving the two-step hand-eye calibration algorithm, and the first candidate transformation matrix obtained by the two-step hand-eye calibration algorithm is generated based on the rotation matrix and the translation vector. A hand-eye calibration algorithm based on dual quaternions is used to generate a second candidate transformation matrix obtained by solving the hand-eye calibration algorithm based on dual quaternions, according to the rotation matrix and the translation vector. The candidate transformation matrix with the smaller root mean square error of the samples is selected from the first candidate transformation matrix and the second candidate transformation matrix as the initial transformation matrix; The initial transformation matrix is ​​fine-tuned iteratively by minimizing the reprojection error using the Ceres nonlinear optimization method to obtain the third candidate transformation matrix solved by the Ceres nonlinear optimization method.

[0010] In the above scheme, the step of performing multi-dimensional quality evaluation and optimization on each of the candidate transformation matrices to obtain the target transformation matrix includes: The root mean square error, maximum error, average error, standard deviation, translation error, rotation error, error of each axis component, and condition number are all calculated for the first candidate transformation matrix, the second candidate transformation matrix, and the third candidate transformation matrix to obtain the multi-dimensional quality evaluation results of the first candidate transformation matrix, the second candidate transformation matrix, and the third candidate transformation matrix. Among the multi-dimensional quality evaluation results of the first candidate transformation matrix, the second candidate transformation matrix, and the third candidate transformation matrix, the optimal multi-dimensional quality evaluation result is selected as the optimal calibration result. The optimal calibration result includes multiple samples, and each sample corresponds to a calibration sample pair in the calibration sample dataset. Calculate the reprojection error of each sample in the optimal calibration result, and calculate the mean and standard deviation of each reprojection error; The maximum and minimum thresholds for removing samples are determined based on the mean and the standard deviation. For each of the samples, if the reprojection error of the sample is greater than the maximum threshold or less than the minimum threshold, then the sample is determined to be an abnormal sample. Each of the aforementioned abnormal samples is removed, and the remaining samples after removing the abnormal samples are used as the calibration sample dataset for the next iteration round. Based on the preset fusion algorithm, the calibration sample dataset of the next iteration round is solved in parallel to obtain multiple candidate transformation matrices for the next iteration round. The candidate transformation matrices of each of the next iteration rounds are subjected to multi-dimensional quality evaluation and optimization until no abnormal samples are eliminated or the preset maximum number of iterations is reached, and the target transformation matrix is ​​output.

[0011] In the above scheme, after obtaining the target transformation matrix, the method further includes: The true position of the calibration plate in the robot's base coordinate system is calculated using the first coordinate system transformation chain, and the true position is taken as the true value; Calculate the first standard deviation of each of the first positional deviations of the calibration samples relative to the true value; The simulated position of the calibration plate in the robot's base coordinate system is calculated using the second coordinate system transformation chain, and the second standard deviation of each second position deviation is calculated based on the second position deviation of each calibration sample relative to the simulated position. If the first standard deviation is greater than a preset standard deviation threshold or the second standard deviation is greater than a preset standard deviation threshold, then the calibration accuracy of the target transformation matrix is ​​determined to be substandard. The first coordinate system transformation chain is obtained through the transformation matrix from the robot end effector to the robot base, the transformation matrix from the camera to the robot end effector, and the transformation matrix from the calibration plate to the camera coordinate system. The second coordinate system transformation chain is from pixel coordinates to camera coordinates to flange coordinates to base coordinates.

[0012] According to one aspect of the embodiments of this application, a robot calibration device based on a fusion algorithm is proposed. The device is used to implement the aforementioned robot calibration method based on a fusion algorithm. The device includes: The determining unit is used to determine multiple calibration positions that the robot needs to reach, and to determine the motion path of the robot to reach each calibration position in sequence based on the nearest neighbor strategy; The acquisition unit is used to acquire image data of the calibration board and determine the pose data of the robot at the current calibration position when the robot reaches any calibration position according to the motion path, and generate a calibration sample pair corresponding to the current calibration position based on the pose data and the image data. The solving unit is used to take the calibration sample pairs corresponding to each calibration position as the calibration sample dataset, and solve the calibration sample dataset in parallel based on a preset fusion algorithm to obtain multiple candidate transformation matrices. An evaluation and optimization unit is used to perform multi-dimensional quality evaluation and optimization on each of the candidate transformation matrices to obtain the target transformation matrix. The preset fusion algorithm is generated based on the two-step hand-eye calibration algorithm, the hand-eye calibration algorithm based on dual quaternions, and the Ceres nonlinear optimization method. According to one aspect of the embodiments of this application, a collaborative robot is provided. The collaborative robot is provided with a robotic arm, a tool holder, a tool holder for assembling the tool holder, a processor for controlling the robotic arm to grasp the tool holder, and a memory. The memory stores a computer program, and when the processor executes the computer program, it is used to implement the above-mentioned robot calibration method based on the fusion algorithm.

[0013] According to one aspect of the embodiments of this application, a computer-readable storage medium is provided, on which a computer program is stored, wherein the computer program executes the above-described robot calibration method based on the fusion algorithm when it is run.

[0014] The beneficial effects of this application are as follows: This application automatically controls the robot to sequentially reach each calibration position according to the motion path. When it reaches the calibration position, it collects the image data of the calibration board and the current position data of the robot to form a calibration sample pair. In this way, after the robot reaches each calibration position, it obtains multiple calibration sample pairs to form a calibration sample dataset. This reduces the dependence on operators, automatically forms calibration sample pairs, and improves calibration efficiency.

[0015] The calibration sample dataset is solved in parallel using a pre-defined fusion algorithm (including a two-step hand-eye calibration algorithm, a hand-eye calibration algorithm based on dual quaternions, and the Ceres nonlinear optimization method) to obtain candidate transformation matrices (candidate calibration results) for each algorithm. By performing multi-dimensional quality evaluation on each candidate transformation matrix, the overall accuracy of each candidate transformation matrix can be evaluated, avoiding the failure of a single algorithm in a specific scenario and improving the reliability and stability of the calibration results.

[0016] By optimizing the calibration results (candidate transformation matrix) after multi-dimensional quality assessment, the data quality of the calibration sample dataset can be improved, resulting in higher accuracy of the final output target transformation matrix. Attached Figure Description

[0017] Figure 1 This is a system architecture diagram of the robot calibration method based on fusion algorithm provided in the embodiments of this application; Figure 2 A flowchart illustrating the robot calibration method based on a fusion algorithm provided in this application embodiment; Figure 3 This is a logic diagram for generating calibration poses provided in the embodiments of this application; Figure 4A logic diagram for removing abnormal depth data provided in the embodiments of this application; Figure 5 A logic diagram for multi-dimensional quality assessment and optimization provided in the embodiments of this application; Figure 6 A logic diagram for dual verification provided in the embodiments of this application; Figure 7 A block diagram of a robot calibration device based on a fusion algorithm provided in an embodiment of this application; Figure 8 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application; Figure 9 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Detailed Implementation

[0018] To enable those skilled in the art to better understand the solutions of this application, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] It should be noted that while some processes described in the specification, claims, and accompanying drawings include multiple steps appearing in a specific order, it should be clearly understood that these steps may not be performed in the order they appear herein, or may be performed in parallel. The step numbers are merely used to distinguish different steps and do not themselves represent any execution order. Furthermore, descriptions such as "first," "second," or "objective" in this document are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. "Multiple" in this document refers to at least two.

[0020] It is worth noting that in the specific embodiments of this application, data such as calibration position, pose, and image data are involved. When the above embodiments of this application are applied to specific products or technologies, permission or consent from the target object is required, and the collection, use, and processing of related data must comply with relevant laws, regulations, and standards. For example, when an embodiment of this application needs to obtain data such as calibration position, pose, and image data, separate permission or consent from the target object can be obtained through pop-up windows or redirection to a confirmation page. After obtaining separate permission or consent from the target object, the calibration position, pose, image data, and other related data used to enable the embodiment of this application to operate normally can then be obtained.

[0021] Please see Figure 1 , Figure 1This is a system architecture diagram of the robot calibration method based on fusion algorithm provided in the embodiments of this application. It includes a terminal 140, an Internet connection 130, a gateway 120, a server 110, etc.

[0022] Terminal 140 can take various forms, including desktop computers, laptops, PDAs (personal digital assistants), mobile phones, vehicle terminals, and dedicated terminals. Furthermore, it can be a single device or a collection of multiple devices. For example, multiple desktop computers can be interconnected via a local area network, sharing a single monitor to work collaboratively, forming a single terminal 140. Terminal 140 can communicate with the Internet 130 via wired or wireless means to exchange data.

[0023] Server 110 refers to a computer system capable of providing certain services to terminal 140. Compared to ordinary terminal 140, server 110 has higher requirements in terms of stability, security, and performance. Server 110 can be a single high-performance computer in a network platform, a cluster of multiple high-performance computers, a portion of a single high-performance computer (e.g., a virtual machine), or a combination of portions of multiple high-performance computers (e.g., virtual machines). Server 110 can also communicate with the Internet 130 via wired or wireless means to exchange data.

[0024] Gateway 120, also known as an internetwork connector or protocol converter, is a computer system or device that acts as a translator, enabling network interconnection at the transport layer. It bridges the gap between two systems using different communication protocols, data formats, languages, or even completely different architectures. Gateways can also provide filtering and security functions. Messages sent from terminal 140 to server 110 are forwarded to the corresponding server 110 via gateway 120. Messages sent from server 110 to terminal 140 are also forwarded to the corresponding terminal 140 via gateway 120.

[0025] The specific implementation methods of the embodiments of this application are described in detail below: Please see Figure 2 , Figure 2 This is a flowchart illustrating the robot calibration method based on a fusion algorithm provided in this application embodiment. The robot calibration method based on the fusion algorithm can be implemented by server 110 and / or terminal 140. Figure 2 The robot calibration method based on fusion algorithms shown includes: Step 210: Determine multiple calibrated locations that the robot needs to reach, and determine the motion path of the robot to reach each calibrated location in sequence based on the nearest neighbor strategy; Step 220: When the robot reaches any calibration position according to the motion path, collect image data of the calibration board and determine the pose data of the robot at the current calibration position, and generate a calibration sample pair corresponding to the current calibration position based on the pose data and the image data; Step 230: Take the calibration sample pairs corresponding to each calibration position as the calibration sample dataset, and solve the calibration sample dataset in parallel based on the preset fusion algorithm to obtain multiple candidate transformation matrices; Step 240: Perform multi-dimensional quality evaluation and optimization on each of the candidate transformation matrices to obtain the target transformation matrix.

[0026] Specifically, the ultimate goal of the hand-eye calibration method is to obtain the transformation matrix between the camera coordinate system and the robot's own base coordinate system. Therefore, this application uses different algorithms in the fusion algorithm to solve for each candidate transformation matrix, and finally compares, evaluates and optimizes them to obtain a target transformation matrix with high calibration accuracy and reliability.

[0027] This application automates the process by controlling a robot to sequentially reach each calibration position along a motion path. Upon reaching a calibration position, it collects image data from the calibration board and the robot's current position data to form calibration sample pairs. In this way, after reaching each calibration position, the robot obtains multiple calibration sample pairs, thereby forming a calibration sample dataset. This reduces reliance on operators, automates the formation of calibration sample pairs, and improves calibration efficiency.

[0028] The calibration sample dataset is solved in parallel using a pre-defined fusion algorithm (including a two-step hand-eye calibration algorithm, a hand-eye calibration algorithm based on dual quaternions, and the Ceres nonlinear optimization method) to obtain candidate transformation matrices (candidate calibration results) for each algorithm. By performing multi-dimensional quality evaluation on each candidate transformation matrix, the overall accuracy of each candidate transformation matrix can be evaluated, avoiding the failure of a single algorithm in a specific scenario and improving the reliability and stability of the calibration results.

[0029] By optimizing the calibration results (candidate transformation matrix) after multi-dimensional quality assessment, the data quality of the calibration sample dataset can be improved, resulting in higher accuracy of the final output target transformation matrix.

[0030] In some embodiments, determining the multiple calibrated locations that the robot needs to reach includes: After moving the robot's camera directly above the calibration plate, the position of the camera is used as a reference position. Multiple sets of calibration poses are generated with the reference reference position as the center. Each set of calibration poses is obtained by changing the height and attitude angle of the camera relative to the reference reference position. For each set of calibration poses, the position of the robot when it reaches the calibration pose is taken as the calibration position that the robot needs to reach. Wherein, at the reference reference position, the complete plane of the calibration plate is within the field of view of the camera, and the optical axis of the camera is perpendicular to the complete plane of the calibration plate.

[0031] Specifically, by setting a reference position, multiple sets of calibration poses are automatically generated around this reference position, and the robot is controlled to move sequentially to each calibration pose. The nearest neighbor strategy is used to automatically plan the motion path. The nearest neighbor strategy minimizes the amount of motion between adjacent path points (adjacent calibration positions). Since the nearest neighbor strategy is a commonly used existing technique, it will not be elaborated further here. Specifically, it includes: Set the reference position: Move the robot's TCP (Tool Center Point) directly above the calibration plate. For example, move the robot's TCP to a position 200 mm directly above the calibration plate and record this position as the reference position. This position should ensure that the calibration plate is completely within the camera's field of view, and that the camera's optical axis is perpendicular or approximately perpendicular to the plane of the calibration plate.

[0032] Multiple poses are generated around a reference position: Multiple different calibration poses are automatically generated centered on the reference position. Each pose is obtained by changing the robot's TCP height and attitude angle relative to the reference position, including combinations of different height levels and rotation angles, ensuring that the calibration poses are uniformly distributed in space. Figure 3 As shown, different height layers are selected, including the plane layer (0mm) at the reference position and the +50mm and -50mm height layers in the height direction of the reference position. The attitude angles can be obtained by rotating the camera around the X, Y, and Z axes of the camera coordinate system at different angles within a given height layer. Based on the height layers and attitude angles, the target calibration position that the robot needs to reach can be determined.

[0033] In some embodiments, generating a calibration sample pair corresponding to the current calibration position based on the pose data and the image data includes: The corner points of the calibration board are determined based on the image data; Acquire depth data for each corner point, the depth data being used to characterize the distance between the corner point and the robot's camera; The interquartile range method is used to remove outlier depth data, resulting in the remaining depth data. The median of the remaining depth data is taken as the final depth estimation result of the image data; Based on the final depth estimation result and the pose data, a calibration sample pair corresponding to the current calibration position is generated.

[0034] Specifically, corner detection is performed on the image data of the calibration board to determine each corner of the calibration board, for example... Figure 4 As shown, a 5×5 neighborhood sampling strategy is used to acquire depth data at the corner points of the calibration board, at the target point ( Figure 4 Twenty-five depth values ​​were collected around the center point (marked in red); outlier depth values ​​were removed using the interquartile range (IQR) method; and the median was used as the final depth estimate, effectively suppressing noise from the depth camera.

[0035] The specific logic of the interquartile range (IQR) method is as follows: sort the values ​​of each depth data from smallest to largest, take the 25th percentile as the lower quartile (Q1), take the 75th percentile as the upper quartile (Q3), and take the value of Q3-Q1 as the interquartile range (IQR). In this embodiment, IQR is also known as the target difference.

[0036] Furthermore, Q1 - 1.5 × IQR is used as the lower threshold for removing abnormal depth data, and Q3 + 1.5 × IQR is used as the upper threshold for removing abnormal depth data. If the value of a depth data is less than the lower threshold or greater than the upper threshold, the depth data is considered abnormal and removed. The median of the remaining data is taken as the final depth estimation result. Based on the final depth estimation result and the robot's current pose data, calibration sample pairs corresponding to the current calibration position are generated. Each time the robot reaches a calibration position, that calibration position is taken as the current calibration position, thus obtaining calibration sample pairs corresponding to each calibration position.

[0037] In some embodiments, the calibration sample dataset is solved in parallel based on a preset fusion algorithm to obtain multiple candidate transformation matrices, including: The rotation matrix and translation vector for the calibration sample dataset are obtained by solving the two-step hand-eye calibration algorithm, and the first candidate transformation matrix obtained by the two-step hand-eye calibration algorithm is generated based on the rotation matrix and the translation vector. A hand-eye calibration algorithm based on dual quaternions is used to generate a second candidate transformation matrix obtained by solving the hand-eye calibration algorithm based on dual quaternions, according to the rotation matrix and the translation vector. The candidate transformation matrix with the smaller root mean square error of the samples is selected from the first candidate transformation matrix and the second candidate transformation matrix as the initial transformation matrix; The initial transformation matrix is ​​fine-tuned iteratively by minimizing the reprojection error using the Ceres nonlinear optimization method to obtain the third candidate transformation matrix solved by the Ceres nonlinear optimization method.

[0038] Specifically, the preset fusion algorithms include a two-step hand-eye calibration algorithm, a hand-eye calibration algorithm based on dual quaternions, and the Ceres nonlinear optimization method. Hand-eye calibration is solved by simultaneously using the following three algorithms: Figure 4 As shown, robustness and accuracy are improved by fusing multiple algorithms: Tsai-Lenz algorithm (two-step hand-eye calibration algorithm): A classic two-step hand-eye calibration algorithm that first solves the rotation part (rotation matrix) and then solves the translation part (translation vector). It is fast and widely used.

[0039] The Daniilidis algorithm (a hand-eye calibration algorithm based on dual quaternions) solves the rotation matrix and translation vector simultaneously within the framework of dual quaternions, complementing the Tsai-Lenz algorithm.

[0040] Ceres nonlinear optimization: The optimal solution results of the Tsai-Lenz and Danilidis algorithms are selected as the initial values ​​(initial transformation matrix). The Ceres optimizer is used for nonlinear optimization and fine-tuning. The calibration accuracy is further improved by minimizing the reprojection error.

[0041] Three algorithms solve the hand-eye transformation relationship from different mathematical perspectives. Two classic algorithms (a two-step hand-eye calibration algorithm and a hand-eye calibration algorithm based on dual quaternions) provide complementary initial values. The Ceres optimizer is then fine-tuned based on these initial values, forming a three-level solution strategy of "classical complementarity + optimization fine-tuning". Finally, the optimal transformation matrix is ​​selected as the target transformation matrix from the candidate transformation matrices obtained by the three algorithms. Furthermore, continue to refer to Figure 4 As shown, the target transformation matrix can only be solved through multi-dimensional quality assessment and iterative optimization. Specifically: For each candidate transformation matrix, calculate the following indicators: RMS error (root mean square error), maximum error, average error, standard deviation, translation and rotation errors, errors of each component along the X / Y / Z axes, and condition number (to assess numerical stability). Select the optimal calibration result based on the comprehensive score. The comprehensive score can be calculated as follows: assign corresponding weights to RMS error, maximum error, average error, standard deviation, translation and rotation errors, errors of each component along the X / Y / Z axes, and condition number (to assess numerical stability). Multiply these weights by their respective values ​​to obtain the comprehensive score.

[0042] In some embodiments, the optimization process is as follows: Based on the optimal calibration results, the reprojection error of each sample is calculated. The 3σ principle (Laida criterion) is used to identify outliers, i.e., samples whose reprojection error exceeds the mean ± 3 times the standard deviation are identified as outliers and removed. The maximum threshold for removing samples is the result of calculating the mean + standard deviation multiplied by 3, and the minimum threshold for removing samples is the result of calculating the mean - standard deviation multiplied by 3.

[0043] After removing abnormal samples, the above steps of parallel solution of fusion algorithm, multi-dimensional quality assessment and optimization are repeated until no abnormal samples are removed or the preset maximum number of iterations is reached. Finally, Ceres nonlinear optimization is used to fine-tune the optimal result and output the final calibration result (target transformation matrix).

[0044] In some embodiments, after obtaining the target transformation matrix, the method further includes: The true position of the calibration plate in the robot's base coordinate system is calculated using the first coordinate system transformation chain, and the true position is taken as the true value; Calculate the first standard deviation of each of the first positional deviations of the calibration samples relative to the true value; The simulated position of the calibration plate in the robot's base coordinate system is calculated using the second coordinate system transformation chain, and the second standard deviation of each second position deviation is calculated based on the second position deviation of each calibration sample relative to the simulated position. If the first standard deviation is greater than a preset standard deviation threshold or the second standard deviation is greater than a preset standard deviation threshold, then the calibration accuracy of the target transformation matrix is ​​determined to be substandard. The first coordinate system transformation chain is obtained through the transformation matrix from the robot end effector to the robot base, the transformation matrix from the camera to the robot end effector, and the transformation matrix from the calibration plate to the camera coordinate system. The second coordinate system transformation chain is from pixel coordinates to camera coordinates to flange coordinates to base coordinates.

[0045] Here, the reliability of the final calibration result (target transformation matrix) is mainly verified using the first coordinate system transformation chain and the second coordinate system transformation chain. Specifically, this is combined with... Figure 5 As shown, the algorithm specifically includes: Algorithm 1 (Truth calculation, corresponding to the first coordinate system transformation chain): Calculate the position of the calibration board in the robot base coordinate system using T_robot (transformation matrix from robot end effector to base) × T_handeye (transformation matrix from camera to robot end effector) × T_target2cam (transformation matrix from calibration board to camera coordinate system). This position should remain unchanged under different robot poses, and its deviation (standard deviation) reflects the calibration accuracy.

[0046] Method 2 (end-to-end calculation, corresponding to the second coordinate system transformation chain): Simulate the actual use scenario, calculate the position of the calibration plate in the base coordinate system through a chain transformation of pixel coordinates → camera coordinates → flange coordinates → base coordinates, and compare it with the true value calculation result.

[0047] Accuracy evaluation: The deviation between the two algorithms is calculated as the calibration accuracy evaluation index. If the deviation calculated by either algorithm exceeds the preset standard deviation threshold (e.g., 0.6), it indicates that the calibration result is unreliable and recalibration is required.

[0048] This application provides a parameter customization function, allowing users to adjust the following parameters according to actual application scenarios: the number of calibration poses (calibration positions, or calibration sample pairs), the selection range and number of height layers, the angle range, the size of the depth sampling neighborhood, the IQR coefficient, the 3σ threshold (upper and lower thresholds), the quality assessment weight, the verification accuracy threshold, etc., to adapt to the needs of different robot models and different calibration board specifications.

[0049] This application is implemented in the hand-eye calibration module of the SmartHand flexible grasping system software. The system has a four-layer architecture: UI layer (CalibrationWidget): Provides a calibration settings interface, including tabs for 3D camera data display, robot simulation and path preview, intrinsic parameter calibration, extrinsic parameter calibration (manual / automatic / simulation), calibration quality assessment, static verification, etc.

[0050] The business control layer (CalibrationController) coordinates various functional modules and manages the calibration process, including path generation, data acquisition, algorithm scheduling, and result output.

[0051] Algorithm layer: Includes core algorithm modules such as HandEyeMultiSolver, CalibrationQualityEvaluator, and RobustDepthEstimator.

[0052] Hardware Abstraction Layer: Includes the 3D camera interface (SP_Camera3D) and the robot control interface (SP_RobotControl), providing a unified hardware access interface. The following examples combine numerical data and... Figure 6 The embodiments of this application will be described below: Taking the Eye-in-Hand calibration mode as an example, using an Intel RealSense D435 depth camera and a UR10 six-axis robot, with an 11×8 checkerboard calibration board (20mm side length), the specific implementation steps are as follows: Step S1 (Path Planning): The operator moves the robot's end effector to approximately 200mm directly above the calibration board using the teach pendant, adjusting its posture so that the camera's optical axis is approximately perpendicular to the calibration board plane. The system records the current TCP pose as the reference position Pref. Centered on Pref, the system uniformly selects three height layers within the height direction [-50mm, +50mm]. On each height layer, different pose combinations are generated around the X, Y, and Z axes with preset angle steps within the range of [-15°, +15°], generating multiple sets of candidate calibration poses. The candidate poses are sorted and optimized, using a nearest neighbor strategy to minimize the motion between adjacent path points. The robot is then controlled to move sequentially to each calibration pose.

[0053] Step S2 (Data Acquisition and Depth Estimation): The system controls the robot to sequentially reach the calibration positions corresponding to each calibration pose according to the generated motion trajectory. At each calibration position, after the robot stops moving, it triggers the camera to acquire image data of the calibration board, and simultaneously reads the current pose data of the robot. For each image, checkerboard corner points (11×8=88 interior corner points) are detected, and robust depth estimation is performed for each corner point: 25 depth values ​​are acquired within a 5×5 neighborhood around the corner point, and outliers (values ​​below Q1-1.5×IQR and above Q3+1.5×IQR) are removed using the IQR method. The median of the remaining data is taken as the final depth estimate. Multiple sets of calibration sample pairs are acquired.

[0054] Step S3 (Solving with three algorithms): The system uses three algorithms simultaneously to solve the hand-eye transformation matrix: (a) The Tsai-Lenz algorithm first solves the rotation matrix R and then the translation vector t; (b) The Daniilidis algorithm solves R and t simultaneously in the dual quaternion; (c) Using the result with the smaller RMS (root mean square) error between (a) and (b) as the initial value, the Ceres optimizer is used to minimize the reprojection error of all samples for nonlinear optimization and fine-tuning.

[0055] Step S4 (Quality Assessment and Anomaly Removal): Calculate the RMS error, maximum error, average error, standard deviation, X / Y / Z axis component errors, and condition number for the results of the three algorithms. The comprehensive score shows that the Ceres optimization result is the best (RMS = 0.85mm). Based on this result, calculate the reprojection error of each sample, and identify one outlier sample according to the 3σ principle (reprojection error 2.6mm, exceeding the mean + 3σ = 2.1mm). After removal, re-execute the three algorithms and perform multi-dimensional quality assessment. After iteration, the RMS of the Ceres optimization result drops to 0.52mm, with no new outlier samples. The final calibration result is then output.

[0056] Step S5 (Dual Verification): (1) Truth Value Calculation: Calculate the position of the calibration board in the base coordinate system using T_robot×T_handeye×T_target2cam. The standard deviation of the position deviation of the 12 sets of pose calculation results is 0.48mm; (2) End-to-End Calculation: Simulate the actual usage scenario, calculate the position of the calibration board from the pixel coordinates through a chain transformation of camera coordinates → flange coordinates → base coordinates. The deviation from the truth value calculation result is 0.52mm. The deviations of both methods are in the sub-millimeter level, confirming the reliability of the calibration results. Dual verification is also known as cross-verification.

[0057] In summary, the beneficial effects of this application are as follows: Fully automated process: From motion path planning to verification and confirmation, no manual point-by-point teaching is required, which greatly shortens the calibration preparation time. The entire calibration process can be completed in a few minutes, reducing the dependence on the operator's experience.

[0058] Multi-algorithm fusion improves robustness: Three algorithms solve the problem from different mathematical perspectives, avoiding the failure of a single algorithm in a specific scenario; two classic algorithms complement each other to provide initial values, and Ceres optimization and fine-tuning improve accuracy.

[0059] Multi-dimensional quality assessment provides comprehensive diagnostics: it not only evaluates overall accuracy but also analyzes the errors and numerical stability of each axis component, providing comprehensive diagnostic information for calibration results.

[0060] Automatic removal of outliers improves data quality: Based on the 3σ principle, outlier data is automatically identified and removed, and calibration accuracy is continuously improved through iterative optimization.

[0061] Dual verification ensures reliability: The calibration results are cross-validated using two independent methods to ensure the reliability and credibility of the calibration results.

[0062] Ensure uniform distribution of calibration poses: By systematically generating multiple sets of poses around the reference position, ensure uniform coverage of calibration poses in space, and provide high-quality input data for the calibration algorithm.

[0063] Please see Figure 7 , Figure 7 This is a schematic diagram of the structure of a robot calibration device based on a fusion algorithm provided in an embodiment of this application. This robot calibration device based on a fusion algorithm is applied to a computer device, and may include: The determining unit 701 is used to determine multiple calibration positions that the robot needs to reach, and to determine the motion path of the robot to reach each calibration position in sequence based on the nearest neighbor strategy; The acquisition unit 702 is used to acquire image data of the calibration board and determine the pose data of the robot at the current calibration position when the robot reaches any calibration position according to the motion path, and generate a calibration sample pair corresponding to the current calibration position based on the pose data and the image data. The solving unit 703 is used to take the calibration sample pairs corresponding to each calibration position as the calibration sample dataset, and solve the calibration sample dataset in parallel based on a preset fusion algorithm to obtain multiple candidate transformation matrices. The evaluation and optimization unit 704 is used to perform multi-dimensional quality evaluation and optimization on each of the candidate transformation matrices to obtain the target transformation matrix. The preset fusion algorithm is generated based on the two-step hand-eye calibration algorithm, the hand-eye calibration algorithm based on dual quaternions, and the Ceres nonlinear optimization method.

[0064] Reference Figure 8 , Figure 8 To implement the structural block diagram of a portion of the terminal 140 in this application embodiment, the terminal 140 includes: a radio frequency (RF) circuit 710, a memory 715, an input unit 730, a display unit 740, a sensor 750, an audio circuit 760, a wireless fidelity (WiFi) module 770, a processor 780, and a power supply 790, among other components. Those skilled in the art will understand that... Figure 8 The terminal 140 structure shown does not constitute a limitation on a mobile phone or computer, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0065] The RF circuit 710 can be used to receive and transmit signals during information transmission or calls. In particular, it receives downlink information from the base station and processes it with the processor 780; in addition, it transmits uplink data to the base station.

[0066] The memory 715 can be used to store software programs and modules. The processor 780 executes various functional applications of the terminal and robot calibration processing based on fusion algorithms by running the software programs and modules stored in the memory 715.

[0067] The input unit 730 can be used to receive input numeric or character information, and to generate key signal inputs related to the terminal's settings and function control. Specifically, the input unit 730 may include a touch panel 731 and other input devices 732.

[0068] The display unit 740 can be used to display input or provided information, as well as various menus of the terminal. The display unit 740 may include a display panel 741.

[0069] Audio circuitry 760, speaker 761, and microphone 762 provide an audio interface.

[0070] In this embodiment, the processor 780 included in the terminal 140 can execute the robot calibration method based on the fusion algorithm in the previous embodiment.

[0071] The terminal 140 in this application embodiment includes, but is not limited to, mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, and aircraft. This application embodiment can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0072] Figure 9 This is a partial structural block diagram of a server 110 implementing an embodiment of this application. The server 110 can vary significantly due to different configurations or performance characteristics, and may include one or more central processing units (CPUs) 822 (e.g., one or more processors) and memory 832, and one or more storage media 830 (e.g., one or more mass storage devices) for storing application programs 842 or data 844. The memory 832 and storage media 830 can be temporary or persistent storage. The program stored in the storage media 830 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the server 110. Furthermore, the CPU 822 may be configured to communicate with the storage media 830 and execute the series of instruction operations in the storage media 830 on the server 110.

[0073] Server 110 may also include one or more power supplies 826, one or more wired or wireless network interfaces 850, one or more input / output interfaces 858, and / or one or more operating systems 841, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0074] The central processing unit 822 in server 110 can be used to execute the robot calibration method based on fusion algorithm according to the embodiments of this application.

[0075] This application also provides a computer-readable storage medium for storing program code for executing the robot calibration method based on the fusion algorithm in the foregoing embodiments.

[0076] This application also provides a computer program product, which includes a computer program. A processor of a computer device reads and executes the computer program, causing the computer device to perform the robot calibration method based on the fusion algorithm described above.

[0077] Furthermore, the terms “comprising” and “including”, and any variations thereof, are intended to cover non-exclusive inclusion, such that a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or apparatus.

[0078] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0079] It should be understood that in the description of the embodiments of this application, "multiple" means two or more, "greater than", "less than", "exceeding" etc. are understood to exclude the number itself, and "above", "below", "within" etc. are understood to include the number itself.

[0080] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0081] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0082] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0083] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0084] It should also be understood that the various implementation methods provided in this application can be combined arbitrarily to achieve different technical effects.

[0085] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0086] The above is a detailed description of the embodiments of this application. However, this application is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A robot calibration method based on a fusion algorithm, characterized in that, The method includes: Determine multiple calibrated locations that the robot needs to reach, and determine the movement path of the robot to reach each calibrated location sequentially based on the nearest neighbor strategy; When the robot reaches any calibration position according to the motion path, image data of the calibration board is collected and pose data of the robot at the current calibration position is determined. Based on the pose data and the image data, a calibration sample pair corresponding to the current calibration position is generated. The calibration sample pairs corresponding to each calibration position are used as the calibration sample dataset. The calibration sample dataset is solved in parallel based on a preset fusion algorithm to obtain multiple candidate transformation matrices. The candidate transformation matrices are subjected to multi-dimensional quality evaluation and optimization to obtain the target transformation matrix. The preset fusion algorithm is generated based on the two-step hand-eye calibration algorithm, the hand-eye calibration algorithm based on dual quaternions, and the Ceres nonlinear optimization method.

2. The robot calibration method based on fusion algorithm according to claim 1, characterized in that, The determination of the multiple calibrated locations that the robot needs to reach includes: After moving the robot's camera directly above the calibration plate, the position of the camera is used as a reference position. Multiple sets of calibration poses are generated with the reference reference position as the center. Each set of calibration poses is obtained by changing the height and attitude angle of the camera relative to the reference reference position. For each set of calibration poses, the position of the robot when it reaches the calibration pose is taken as the calibration position that the robot needs to reach. Wherein, at the reference reference position, the complete plane of the calibration plate is within the field of view of the camera, and the optical axis of the camera is perpendicular to the complete plane of the calibration plate.

3. The robot calibration method based on a fusion algorithm according to claim 1, characterized in that, The step of generating a calibration sample pair corresponding to the current calibration position based on the pose data and the image data includes: The corner points of the calibration board are determined based on the image data; Acquire depth data for each corner point, the depth data being used to characterize the distance between the corner point and the robot's camera; The interquartile range method is used to remove outlier depth data, resulting in the remaining depth data. The median of the remaining depth data is taken as the final depth estimation result of the image data; Based on the final depth estimation result and the pose data, a calibration sample pair corresponding to the current calibration position is generated.

4. The robot calibration method based on fusion algorithm according to claim 3, characterized in that, The method of removing outlier depth data using the interquartile range includes: The lower and upper quartiles of each depth data point are determined using the interquartile range method. The difference between the upper quartile and the lower quartile is taken as the target difference. The lower threshold for removing abnormal depth data is determined based on the lower quartile and the target difference. The upper threshold for removing abnormal depth data is determined based on the upper quartile and the target difference. For each depth data point, if the value of the depth data is less than the lower threshold or greater than the upper threshold, then the depth data is determined to be abnormal depth data.

5. The robot calibration method based on a fusion algorithm according to claim 1, characterized in that, The preset fusion algorithm is used to solve the calibration sample dataset in parallel to obtain multiple candidate transformation matrices, including: The rotation matrix and translation vector for the calibration sample dataset are obtained by solving the two-step hand-eye calibration algorithm, and the first candidate transformation matrix obtained by the two-step hand-eye calibration algorithm is generated based on the rotation matrix and the translation vector. A hand-eye calibration algorithm based on dual quaternions is used to generate a second candidate transformation matrix obtained by solving the hand-eye calibration algorithm based on dual quaternions, according to the rotation matrix and the translation vector. The candidate transformation matrix with the smaller root mean square error of the samples is selected from the first candidate transformation matrix and the second candidate transformation matrix as the initial transformation matrix; The initial transformation matrix is ​​fine-tuned iteratively by minimizing the reprojection error using the Ceres nonlinear optimization method to obtain the third candidate transformation matrix solved by the Ceres nonlinear optimization method.

6. The robot calibration method based on fusion algorithm according to claim 5, characterized in that, The step of performing multi-dimensional quality evaluation and optimization on each of the candidate transformation matrices to obtain the target transformation matrix includes: The root mean square error, maximum error, average error, standard deviation, translation error, rotation error, error of each axis component, and condition number are all calculated for the first candidate transformation matrix, the second candidate transformation matrix, and the third candidate transformation matrix to obtain the multi-dimensional quality evaluation results of the first candidate transformation matrix, the second candidate transformation matrix, and the third candidate transformation matrix. Among the multi-dimensional quality evaluation results of the first candidate transformation matrix, the second candidate transformation matrix, and the third candidate transformation matrix, the optimal multi-dimensional quality evaluation result is selected as the optimal calibration result. The optimal calibration result includes multiple samples, and each sample corresponds to a calibration sample pair in the calibration sample dataset. Calculate the reprojection error of each sample in the optimal calibration result, and calculate the mean and standard deviation of each reprojection error; The maximum and minimum thresholds for removing samples are determined based on the mean and the standard deviation. For each of the samples, if the reprojection error of the sample is greater than the maximum threshold or less than the minimum threshold, then the sample is determined to be an abnormal sample. Each of the aforementioned abnormal samples is removed, and the remaining samples after removing the abnormal samples are used as the calibration sample dataset for the next iteration round. Based on the preset fusion algorithm, the calibration sample dataset of the next iteration round is solved in parallel to obtain multiple candidate transformation matrices for the next iteration round. The candidate transformation matrices of each of the next iteration rounds are subjected to multi-dimensional quality evaluation and optimization until no abnormal samples are eliminated or the preset maximum number of iterations is reached, and the target transformation matrix is ​​output.

7. The robot calibration method based on a fusion algorithm according to claim 1, characterized in that, After obtaining the target transformation matrix, the method further includes: The true position of the calibration plate in the robot's base coordinate system is calculated using the first coordinate system transformation chain, and the true position is taken as the true value; Calculate the first standard deviation of each of the first positional deviations of the calibration samples relative to the true value; The simulated position of the calibration plate in the robot's base coordinate system is calculated using the second coordinate system transformation chain, and the second standard deviation of each second position deviation is calculated based on the second position deviation of each calibration sample relative to the simulated position. If the first standard deviation is greater than a preset standard deviation threshold or the second standard deviation is greater than a preset standard deviation threshold, then the calibration accuracy of the target transformation matrix is ​​determined to be substandard. The first coordinate system transformation chain is obtained through the transformation matrix from the robot end effector to the robot base, the transformation matrix from the camera to the robot end effector, and the transformation matrix from the calibration plate to the camera coordinate system. The second coordinate system transformation chain is from pixel coordinates to camera coordinates to flange coordinates to base coordinates.

8. A robot calibration device based on a fusion algorithm, characterized in that, The apparatus is used to implement the robot calibration method based on the fusion algorithm as described in any one of claims 1-7, the apparatus comprising: The determining unit is used to determine multiple calibration positions that the robot needs to reach, and to determine the motion path of the robot to reach each calibration position in sequence based on the nearest neighbor strategy; The acquisition unit is used to acquire image data of the calibration board and determine the pose data of the robot at the current calibration position when the robot reaches any calibration position according to the motion path, and generate a calibration sample pair corresponding to the current calibration position based on the pose data and the image data. The solving unit is used to take the calibration sample pairs corresponding to each calibration position as the calibration sample dataset, and solve the calibration sample dataset in parallel based on a preset fusion algorithm to obtain multiple candidate transformation matrices. An evaluation and optimization unit is used to perform multi-dimensional quality evaluation and optimization on each of the candidate transformation matrices to obtain the target transformation matrix. The preset fusion algorithm is generated based on the two-step hand-eye calibration algorithm, the hand-eye calibration algorithm based on dual quaternions, and the Ceres nonlinear optimization method.

9. A collaborative robot, characterized in that, The collaborative robot is equipped with a robotic arm, a tool holder, a tool holder for assembling the tool holder, a processor for controlling the robotic arm to grasp the tool holder, and a memory. The memory stores a computer program, and when the processor executes the computer program, it implements the robot calibration method based on the fusion algorithm as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed, it performs the robot calibration method based on the fusion algorithm as described in any one of claims 1 to 7.

Images