Methods for determining the relationship model of the hand-eye matrix, hand-eye calibration methods and equipment

Abstract

This application applies to the field of robotics technology, providing a method for determining a hand-eye matrix relationship model, a hand-eye calibration method, and a device. The method for determining the hand-eye matrix relationship model includes: determining the coordinate offset between the origin of the robot arm's base coordinate system and the origin of the gimbal's base coordinate system, and determining the coordinate transformation matrix between the gimbal's base coordinate system and the camera coordinate system; based on the coordinate offset and the coordinate transformation matrix, determining the hand-eye matrix relationship model. This relationship model characterizes the relationship between the hand-eye matrix and the gimbal angle, where the gimbal angle is the angle of the gimbal relative to the gimbal base on different planes. The hand-eye matrix is ​​used to characterize the coordinate transformation relationship from the robot arm's base coordinate system to the camera coordinate system. Through this method, a relationship model between the gimbal angle and the hand-eye matrix can be established, thereby simplifying the robot's hand-eye calibration.

Description

Technical Field

[0001] This application belongs to the field of robotics technology, and in particular relates to a method for determining the relationship model of the hand-eye matrix, a hand-eye calibration method, and a device. Background Technology

[0002] With the rapid development of high technology and the application of robots in various industries, the intelligence level of robotic arms in conjunction with environmental perception is constantly improving, especially visual guidance, which is changing the way robots are used.

[0003] In single-arm, dual-arm, or dual-arm anthropomorphic robotic systems, a depth camera with a gimbal is used for visual positioning. The gimbal provides at least two degrees of freedom for rotational motion: horizontal and pitch. By adjusting the camera's position and attitude, target positioning can be achieved from different angles, and at a fixed position, positioning can cover a wider range of targets. After positioning, a hand-eye matrix converts the target point's pose in the camera coordinate system into its pose in the robotic arm's base coordinate system. Once the robotic arm acquires the target point's pose in its base coordinate system, it can move to that target point to complete the task.

[0004] Visual guidance optimization solutions, typically used in service robots, inspection robots, industrial robots, and robotic arm systems, provide a wider-ranging, more intelligent, and more human-like visual guidance solution for robot operation, building upon the existing fixed-angle positioning of cameras.

[0005] In this process, hand-eye calibration is very important. The usual way to obtain the hand-eye matrix is ​​to identify the pose matrix of the camera relative to the robot arm base coordinates using the common method of eye outside hand, with the horizontal and pitch angles of the gimbal fixed. This method can obtain an accurate hand-eye matrix when the gimbal angle is fixed. However, due to the needs of the operation, the horizontal and pitch angles of the gimbal are not fixed. Every time the horizontal and pitch angles are changed, the camera needs to be recalibrated, which is very labor-intensive. In addition, storing these matrices also requires a lot of space. Summary of the Invention

[0006] In view of this, embodiments of this application provide a method for determining the relationship model of the hand-eye matrix, a hand-eye calibration method, and a device, which are used to determine the relationship model between the hand-eye matrix and the gimbal angle, thereby calibrating the hand-eye matrix based on the relationship model and the gimbal angle, reducing the difficulty of hand-eye calibration.

[0007] The first aspect of this application provides a method for determining a hand-eye matrix relationship model, comprising:

[0008] Determine the coordinate offset between the origin of the robotic arm base coordinate system and the origin of the gimbal base coordinate system, and determine the coordinate transformation matrix between the gimbal base coordinate system and the camera coordinate system. The gimbal base coordinate system is a coordinate system established with the center of the rotation axis of the gimbal base as the origin. The camera coordinate system is a coordinate system established with the focus center of the camera as the origin. The robotic arm base coordinate system is a coordinate system established with the center of the rotation axis of the robotic arm base as the origin.

[0009] Based on the coordinate offset and the coordinate transformation matrix, a relationship model of the hand-eye matrix is ​​determined. The relationship model can characterize the relationship between the hand-eye matrix and the gimbal angle. The gimbal angle is the angle of the gimbal relative to the gimbal base on different planes. The hand-eye matrix is ​​used to characterize the coordinate transformation relationship from the robot arm base coordinate system to the camera coordinate system.

[0010] A second aspect of this application provides a hand-eye calibration method applied to a terminal device, the terminal device including a gimbal and a robotic arm, the gimbal having a camera mounted on it, the method comprising:

[0011] Determine the target relationship model of the terminal device. The target relationship model is the relationship model between the gimbal angle of the terminal device and the hand-eye matrix. The hand-eye matrix is ​​used to characterize the coordinate transformation relationship from the robot arm base coordinate system to the camera coordinate system.

[0012] The current gimbal angle is input into the target relationship model to obtain the current target hand-eye matrix. The target hand-eye matrix is ​​used to characterize the coordinate transformation relationship from the robot arm base coordinate system to the camera coordinate system under the current gimbal angle.

[0013] The target relationship model is established according to the method described in the first aspect above.

[0014] A third aspect of this application provides an apparatus for determining a hand-eye matrix relationship model, comprising:

[0015] The coordinate transformation determination module is used to determine the coordinate offset between the origin of the robot arm base coordinate system and the origin of the gimbal base coordinate system, and to determine the coordinate transformation matrix between the gimbal base coordinate system and the camera coordinate system. The gimbal base coordinate system is a coordinate system established with the center of the rotation axis of the gimbal base as the origin, the camera coordinate system is a coordinate system established with the focus center of the camera as the origin, and the robot arm base coordinate system is a coordinate system established with the center of the rotation axis of the robot arm base as the origin.

[0016] The relation model determination module is used to determine the relation model of the hand-eye matrix based on the coordinate offset and the coordinate transformation matrix. The relation model can characterize the relationship between the hand-eye matrix and the gimbal angle. The gimbal angle is the angle of the gimbal relative to the gimbal base on different planes. The hand-eye matrix is ​​used to characterize the coordinate transformation relationship from the robot arm base coordinate system to the camera coordinate system.

[0017] A second aspect of this application provides a hand-eye calibration device applied to a terminal device, the terminal device including a gimbal and a robotic arm, the gimbal having a camera mounted on it, the device comprising:

[0018] The target relationship model determination module is used to determine the target relationship model of the terminal device. The target relationship model is the relationship model between the gimbal angle of the terminal device and the hand-eye matrix. The hand-eye matrix is ​​used to characterize the coordinate transformation relationship from the robot arm base coordinate system to the camera coordinate system.

[0019] The target hand-eye matrix calibration module is used to input the current gimbal angle into the target relationship model to obtain the current target hand-eye matrix. The target hand-eye matrix is ​​used to characterize the coordinate transformation relationship from the robot arm base coordinate system to the camera coordinate system under the current gimbal angle.

[0020] The target relationship model is established according to the method described in the first aspect above.

[0021] A fifth aspect of this application provides a terminal device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first or second aspect above.

[0022] A sixth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first or second aspect above.

[0023] A seventh aspect of this application provides a computer program product that, when run on a terminal device, causes the terminal device to execute the method described in the first or second aspect.

[0024] Compared with the prior art, the embodiments of this application have the following advantages:

[0025] The hand-eye matrix is ​​used to characterize the coordinate transformation relationship between the robotic arm base coordinate system and the camera coordinate system. In this embodiment, the hand-eye matrix can be decomposed into the coordinate transformation relationship between the robotic arm base coordinate system and the gimbal base coordinate system, and the coordinate transformation relationship between the gimbal base coordinate system and the camera coordinate system. Therefore, the coordinate offset between the origin of the robotic arm base coordinate system and the origin of the camera coordinate system can be determined, and the coordinate transformation matrix between the gimbal base coordinate system and the camera coordinate system can be determined. Then, the coordinate offset and the coordinate transformation matrix are combined to obtain the relationship model of the hand-eye matrix. The relationship model of the hand-eye matrix includes relevant parameters of the gimbal angle; therefore, this relationship model can characterize the relationship between the hand-eye matrix and the gimbal angle. Based on the decomposition of the hand-eye matrix, this embodiment can easily obtain the relationship model between the hand-eye matrix and the gimbal angle.

[0026] Based on the aforementioned method for determining the hand-eye matrix relationship model, the terminal device can determine the target relationship model between the gimbal angle and the hand-eye matrix during hand-eye calibration. Based on this target relationship model and the current gimbal angle, hand-eye calibration can be achieved, yielding the current hand-eye matrix. This method allows for rapid hand-eye calibration based on the gimbal angle, avoiding the need for calibration personnel to calibrate the hand-eye matrix every time the gimbal angle changes. This significantly reduces the number of calibrations and improves the efficiency and flexibility of robot operations. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0028] Figure 1 This is a flowchart illustrating the steps of a method for determining a hand-eye matrix relationship model provided in an embodiment of this application.

[0029] Figure 2 This is a schematic diagram of coordinate transformation provided in an embodiment of this application;

[0030] Figure 3 This is a flowchart illustrating the steps of a hand-eye calibration method provided in an embodiment of this application;

[0031] Figure 4 This is a schematic diagram of a device for determining a hand-eye matrix relationship model provided in an embodiment of this application;

[0032] Figure 5 This is a schematic diagram of a hand-eye calibration device provided in an embodiment of this application;

[0033] Figure 6 This is a schematic diagram of a terminal device provided in an embodiment of this application. Detailed Implementation

[0034] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0035] The technical solution of this application will be described below through specific embodiments.

[0036] Reference Figure 1 This diagram illustrates a step-by-step flowchart of a method for determining a hand-eye matrix relationship model according to an embodiment of this application. Specifically, it may include the following steps:

[0037] S101, determine the coordinate offset between the origin of the robot arm base coordinate system and the origin of the gimbal base coordinate system, and determine the coordinate transformation matrix between the gimbal base coordinate system and the camera coordinate system.

[0038] The execution subject of this application embodiment can be a terminal device that needs to determine the hand-eye matrix. This application embodiment does not limit the specific type of terminal device. For example, the terminal device can be a robot, which may include a gimbal and a robotic arm. A camera can be mounted on the gimbal. The gimbal includes a gimbal base, the position of which is fixed, and the gimbal can rotate around the rotation axis center of the gimbal base. The robotic arm may include a robotic arm base, and the rotation axis center of the robotic arm base is fixed.

[0039] The hand-eye matrix is ​​used to characterize the coordinate transformation relationship between the robotic arm base coordinate system and the camera coordinate system. In this embodiment, the hand-eye matrix can be decomposed into the coordinate transformation relationship between the robotic arm base coordinate system and the gimbal base coordinate system, and the coordinate transformation relationship between the gimbal base coordinate system and the camera coordinate system. Specifically, the gimbal base coordinate system is a coordinate system established with the center of the rotation axis of the gimbal base as its origin; the camera coordinate system is a coordinate system established with the focus center of the camera as its origin and the optical axis as its Z-axis; and the robotic arm base coordinate system is a coordinate system established with the center of the rotation axis of the robotic arm base as its origin. Since the rotation axis centers of the robotic arm base and the gimbal base are fixed, the robotic arm base coordinate system and the gimbal base coordinate system are also fixed. The robotic arm base coordinate system and the gimbal base coordinate system can have the same x, y, and z axis directions, so the coordinate transformation between them is equivalent to only requiring a positional offset between their origins. Therefore, determining the coordinate transformation relationship between the robotic arm's base coordinate system and the gimbal's base coordinate system is equivalent to determining the coordinate offset between the origin of the robotic arm's base coordinate system and the origin of the camera's coordinate system. This coordinate offset can include offsets along the x, y, and z axes.

[0040] In one possible implementation, in this embodiment of the application, for ease of processing, the coordinate offset between the origin of the robotic arm's base coordinate system and the origin of the camera coordinate system may only include the offsets on the x and y axes. The offset on the z-axis may be included in the coordinate transformation relationship between the gimbal base coordinate system and the camera coordinate system. That is, the aforementioned coordinate offset may only include lateral and longitudinal offsets.

[0041] As the gimbal angle changes, the camera angle also changes, and consequently, the camera's focus center and optical axis also change. Therefore, when converting the gimbal base coordinate system to the camera coordinate system, coordinate system rotation and translation are required. Based on the coordinate system rotation and translation, the coordinate transformation matrix between the gimbal base coordinate system and the camera coordinate system can be determined.

[0042] Figure 2 This is a schematic diagram of coordinate transformation provided in an embodiment of this application. For example... Figure 2 As shown, the coordinate transformation from the robotic arm base coordinate system to the gimbal base coordinate system can include:

[0043] The robot arm's base coordinate system is translated in the x and y directions, so that the origin of the robot arm's base coordinate system moves to the z-axis, where the origin of the gimbal's base coordinate system is located. Since the positions of the robot arm base and the gimbal base are fixed, the coordinate transformation between the robot arm's base coordinate system and the gimbal's base coordinate system only requires coordinate transformation based on the positional offset between the origins.

[0044] The transformation between the gimbal base coordinate system and the camera coordinate system can include:

[0045] The gimbal base coordinate system is rotated by an angle θ along its z-axis to obtain the first coordinate system; the first coordinate system is rotated by an angle β along its y-axis to obtain the second coordinate system; the second coordinate system is moved upwards by an angle h along its z-axis to obtain the third coordinate system; the third coordinate system, while maintaining its attitude, is translated by an angle d along its x-axis to obtain the fourth coordinate system; the fourth coordinate system is rotated by an angle α along its x-axis to obtain the fifth coordinate system; and the fifth coordinate system is moved by an angle s along its z-axis to obtain the camera coordinate system. The first, second, third, fourth, and fifth coordinate systems are all intermediate coordinate systems in the coordinate transformation process; each coordinate transformation is performed based on the previous one.

[0046] Assumption Matrix , representing the coordinate transformation matrix from coordinate system a to coordinate system b. Based on the transformation steps from the gimbal base coordinate system to the camera coordinate system described above, the formula for calculating the coordinate transformation matrix from the gimbal base to the camera origin is as follows:

[0047]

[0048] in, Let be the coordinate transformation matrix, where θ is the rotation angle of the gimbal base coordinate system along the z-axis of the gimbal base coordinate system during the transformation from the gimbal base coordinate system to the first coordinate system, β is the rotation angle along the y-axis of the first coordinate system during the transformation from the first coordinate system to the second coordinate system, h is the translation along the z-axis of the second coordinate system during the transformation from the second coordinate system to the third coordinate system, d is the translation along the x-axis of the third coordinate system during the transformation from the third coordinate system to the fourth coordinate system, α is the rotation angle along the x-axis of the fourth coordinate system during the transformation from the fourth coordinate system to the fifth coordinate system, and s is the translation along the z-axis of the fifth coordinate system during the transformation from the fifth coordinate system to the camera coordinate system. The process of transforming from the gimbal base coordinate system to the camera coordinate system is as follows: transformation from the gimbal coordinate system to the first coordinate system, transformation from the first coordinate system to the second coordinate system, transformation from the second coordinate system to the third coordinate system, transformation from the third coordinate system to the fourth coordinate system, transformation from the fourth coordinate system to the fifth coordinate system, and transformation from the fifth coordinate system to the camera coordinate system.

[0049] The above h may include the distance on the z-axis from the origin of the robot arm's base coordinate system to the origin of the gimbal's base coordinate system.

[0050] Assuming the coordinate transformation from the robotic arm base to the camera end effector, i.e., the hand-eye matrix, is as follows: Based on the above, This can be broken down into the transformation from the robotic arm's base coordinate system to the gimbal base coordinate system and the transformation from the gimbal base to the camera origin, namely:

[0051] = (2)

[0052] in This is a coordinate transformation between the robotic arm base and the camera base, which is generated by offsets in the x and y directions.

[0053] S102, Based on the coordinate offset and the coordinate transformation matrix, determine the relationship model of the hand-eye matrix, which can characterize the relationship between the hand-eye matrix and the gimbal angle.

[0054] The camera is mounted on a gimbal and can move with it. After the gimbal is mounted on its base, it can rotate horizontally around its axis of rotation and tilt vertically. Therefore, the gimbal angle can include both horizontal and tilt angles. The horizontal and tilt angles can be determined using the angular relationship between the gimbal and a preset reference position.

[0055] When the gimbal angle changes, the camera's focal center and optical axis also change, and consequently, the camera coordinate system changes. Therefore, the camera coordinate system is related to the gimbal angle, and correspondingly, the hand-eye matrix can be correlated with the gimbal angle.

[0056] Therefore, the relationship model of the hand-eye matrix determined based on the above coordinate offset and coordinate transformation matrix can characterize the relationship between the hand-eye matrix and the gimbal angle. In other words, based on this relationship model, the hand-eye matrix can be determined by the gimbal angle.

[0057] For example, by combining Formula 1 and Formula 2 above, the relationship between the hand-eye matrix and the gimbal angle can be obtained:

[0058]

[0059] in, Let be the hand-eye matrix, θ be the rotation angle of the gimbal base coordinate system along the z-axis of the gimbal base coordinate system during the transformation from the gimbal base coordinate system to the first coordinate system, β be the rotation angle along the y-axis of the first coordinate system during the transformation from the first coordinate system to the second coordinate system, h be the translation displacement along the z-axis of the second coordinate system during the transformation from the second coordinate system to the third coordinate system, d be the translation displacement along the x-axis of the third coordinate system during the transformation from the third coordinate system to the fourth coordinate system, α be the rotation angle along the x-axis of the fourth coordinate system during the transformation from the fourth coordinate system to the fifth coordinate system, s be the translation displacement along the z-axis of the fifth coordinate system during the transformation from the fifth coordinate system to the camera coordinate system, x be the lateral offset, and y be the longitudinal offset. The process of transforming from the gimbal base coordinate system to the camera coordinate system is as follows: transformation from the gimbal coordinate system to the first coordinate system, transformation from the first coordinate system to the second coordinate system, transformation from the second coordinate system to the third coordinate system, transformation from the third coordinate system to the fourth coordinate system, transformation from the fourth coordinate system to the fifth coordinate system, and transformation from the fifth coordinate system to the camera coordinate system.

[0060] In the above relational model, for the terminal device, the horizontal and pitch angles of the gimbal matrix are variables, while other parameters are fixed values ​​for a given terminal device. For example, once the gimbal is installed, its installation angle is fixed, meaning β is a fixed value. Since all other parameters are fixed, and only the gimbal angle is a variable, the above relational model is a relationship model between the hand-eye matrix and the gimbal angle.

[0061] In the above relational model, β, x, y, h, d, and s are fixed, but they need to be solved. To facilitate the solution, the matrices in the above relational model can be converted into camera poses, resulting in:

[0062] (4)

[0063] (5)

[0064] (6)

[0065] Where α is the pitch angle of the gimbal, β is the mounting angle of the camera on the gimbal, and θ is the horizontal angle of the gimbal. Let x, y, h, d, and s be the camera's pose in the robot arm's base coordinate system, and let x, y, h, d, and s be undetermined coefficients. It can be seen that the camera's final pose in the robot arm's coordinate system, specifically the roll and yaw angles, are the α and θ angles in the gimbal angles, while the β angle is the value of the calibrated pitch angle.

[0066] By performing position term analysis on the above relationship and using coordinate transformation, the position coordinates of the camera relative to the robotic arm base can be derived as follows:

[0067] (7)

[0068] (8)

[0069] (9)

[0070] Where (cam_x, cam_y, cam_z) are the coordinates in the camera coordinate system corresponding to the coordinates in the robot arm coordinate system, α is the pitch angle in the gimbal angle, θ is the horizontal angle in the gimbal angle, β is the mounting angle of the camera on the gimbal, x is the lateral offset in the coordinate offset, y is the longitudinal offset in the coordinate offset, and h is the displacement along the z-axis of the gimbal base coordinate system required to transform from the gimbal base coordinate system to the camera coordinate system.

[0071] Based on the above relationships, when the gimbal angle is fixed, α, β, and θ are constants. Substituting them into formulas (7), (8), and (9) yields three equations. This set of equations has five unknowns: x, y, h, d, and s. Therefore, if two sets of data are calibrated, six equations can be obtained, which can solve for the values ​​of x, y, h, d, and s, and thus obtain the relationship model between the hand-eye matrix and the gimbal angle.

[0072] In this embodiment, when determining the hand-eye matrix, it can be decomposed into the coordinate transformation relationship between the robot arm base coordinate system and the gimbal base coordinate system, and the coordinate transformation relationship between the gimbal base coordinate system and the camera coordinate system. This allows determining the coordinate offset between the origin of the robot arm base coordinate system and the origin of the camera coordinate system, and also the coordinate transformation matrix between the gimbal base coordinate system and the camera coordinate system. The coordinate offset and the coordinate transformation matrix are then combined to obtain the relationship model of the hand-eye matrix. This relationship model includes relevant parameters of the gimbal angle; therefore, it can characterize the relationship between the hand-eye matrix and the gimbal angle. Based on the established relationship model, the hand-eye matrix can be determined using the gimbal angle, facilitating subsequent hand-eye calibration, reducing calibration difficulty, and decreasing the required parameters.

[0073] Reference Figure 3 This diagram illustrates a step-by-step flowchart of another hand-eye calibration method provided in an embodiment of this application. This method can be applied to a terminal device, which may include a gimbal and a robotic arm. A camera may be mounted on the gimbal. Specifically, the method may include the following steps:

[0074] S301, Determine the target relationship model of the terminal device, wherein the target relationship model is the relationship model between the gimbal angle and the hand-eye matrix of the terminal device.

[0075] For example, the terminal device can be a robot, which may include a gimbal and a robotic arm. A camera can be mounted on the gimbal. The gimbal includes a gimbal base, the position of which is fixed, and the gimbal can rotate around the rotation axis of the gimbal base. The robot's camera can be considered as the robot's "eye," and the robotic arm can be considered as the robot's "hand." Hand-eye calibration determines the coordinate transformation relationship between the robot's robotic arm base coordinate system and the camera coordinate system. This coordinate transformation relationship can be a hand-eye matrix. Based on the hand-eye matrix, the robot can transform the pose obtained in the camera coordinate system to the robotic arm base coordinate system, thereby controlling the robotic arm's operation based on the camera's positioning results.

[0076] The target relationship model is established according to the method described in the first aspect. As can be seen from the previous embodiment, when determining the target relationship model, it is necessary to solve for the undetermined coefficients in the relationship model. To solve for the undetermined coefficients, the corresponding equations can be obtained based on multiple manually calibrated data, and then the equations can be solved to obtain the undetermined coefficients.

[0077] For example, at least two initial hand-eye matrices can be determined at multiple different initial gimbal angles. The determination of the initial hand-eye matrices can be based on a calibration board.

[0078] Based on the hand-eye matrix relationship model and multiple initial hand-eye matrices determined in the previous embodiment, multiple corresponding relational expressions can be obtained. This relationship model includes multiple undetermined coefficients, and the multiple relational expressions characterize the relationship between the undetermined coefficients and the gimbal angle. Specifically, the relationship model can be transformed to obtain multiple transformation relationships regarding the undetermined coefficients; thus, based on these multiple transformation relationships, multiple initial hand-eye matrices can be converted into multiple corresponding relational expressions regarding the undetermined coefficients. These multiple transformation relationships may include:

[0079]

[0080]

[0081]

[0082]

[0083]

[0084]

[0085] Where (cam_x, cam_y, cam_z) are the coordinates in the camera coordinate system corresponding to the coordinates in the robot arm coordinate system, α is the pitch angle of the gimbal, β is the mounting angle of the camera on the gimbal, and θ is the horizontal angle of the gimbal. Let be the pose of the camera in the base coordinate system of the robotic arm, and x, y, h, d and s be undetermined coefficients.

[0086] Based on multiple initial hand-eye matrices, by substituting the data into the above transformation relationships, we can obtain multiple corresponding relational expressions.

[0087] After determining the relationships, multiple coefficients can be calculated based on these relationships. The initial gimbal angle is the user-inputted angle; however, there may be an error between the user-inputted initial gimbal angle and the actual gimbal angle. Therefore, the initial gimbal angle can be corrected when calculating the coefficients. For example, the gimbal angle can be corrected using the following formula:

[0088]

[0089] in, For horizontal angles, For pitch angle, Let α be the installation angle, β be the corrected pitch angle, θ be the corrected horizontal angle, and N be the number of initial hand-eye matrices, where N is greater than or equal to 2. One initial hand-eye matrix can determine 3 equations, requiring the solution of 5 unknowns. Therefore, at least 2 initial hand-eye matrices are needed to determine 6 equations, thus solving for the 5 unknowns.

[0090] After correcting the initial gimbal angle, multiple equations for solving the coefficient values ​​can be determined based on several transformation relationships and the corrected initial gimbal angle. Solving these equations yields the coefficient values.

[0091] Substituting the coefficient values ​​into the relational model yields the target relational model. This target relational model is equivalent to the relationship model between the gimbal angle and the hand-eye angle.

[0092] S302, input the current gimbal angle into the target relationship model to obtain the current target hand-eye matrix. The target hand-eye matrix is ​​used to characterize the coordinate transformation relationship from the robot arm base coordinate system to the camera coordinate system under the current gimbal angle.

[0093] After determining the target relationship model, when the gimbal angle changes, the current gimbal angle can be input into the target relationship model to quickly obtain the current target hand-eye matrix. Based on the target hand-eye matrix, the terminal device can control the robotic arm to perform corresponding operations.

[0094] In this embodiment, when performing hand-eye calibration, the terminal device only needs to determine the target relationship model corresponding to the terminal device after installation. Then, when the gimbal angle of the terminal device changes, the current gimbal angle is simply input into the target relationship model to obtain the hand-eye matrix. Based on the method in this embodiment, hand-eye calibration becomes simple, eliminating the need for extensive calculations every time the gimbal angle changes, thus reducing the difficulty of determining the hand-eye matrix. Furthermore, since extensive calculations are not required, computational resources are saved.

[0095] It should be noted that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0096] This application also provides a robot that determines the relationship model between the gimbal angle and the hand-eye matrix using the aforementioned method for determining the hand-eye matrix relationship model, and performs hand-eye calibration using the aforementioned hand-eye calibration method to determine the hand-eye matrix. The robot may include a gimbal and a robotic arm, with a camera mounted on the gimbal. The gimbal includes a gimbal base, the position of which is fixed, and the gimbal can rotate around the rotation axis of the gimbal base. When the gimbal rotates, it can cause the camera to rotate, thereby changing the camera coordinate system. When the camera coordinate system changes, the robot can perform hand-eye calibration using the aforementioned hand-eye calibration method to obtain the hand-eye matrix.

[0097] Reference Figure 4 The diagram illustrates a device for determining a hand-eye matrix relationship model according to an embodiment of this application. Specifically, it may include a coordinate transformation determination module 41 and a relationship model determination module 42, wherein:

[0098] The coordinate transformation determination module 41 is used to determine the coordinate offset between the origin of the robot arm base coordinate system and the origin of the gimbal base coordinate system, and to determine the coordinate transformation matrix between the gimbal base coordinate system and the camera coordinate system. The gimbal base coordinate system is a coordinate system established with the center of the rotation axis of the gimbal base as the origin. The camera coordinate system is a coordinate system established with the focus center of the camera as the origin. The robot arm base coordinate system is a coordinate system established with the center of the rotation axis of the robot arm base as the origin.

[0099] The relation model determination module 42 is used to determine the relation model of the hand-eye matrix based on the coordinate offset and the coordinate transformation matrix. The relation model can characterize the relationship between the hand-eye matrix and the gimbal angle. The gimbal angle is the angle of the gimbal relative to the gimbal base on different planes. The hand-eye matrix is ​​used to characterize the coordinate transformation relationship from the robot arm base coordinate system to the camera coordinate system.

[0100] In one possible implementation, the coordinate transformation matrix between the gimbal base coordinate system and the camera coordinate system is:

[0101]

[0102] in, Let θ be the coordinate transformation matrix, where θ is the angle of rotation along the z-axis of the gimbal base coordinate system during the transformation from the gimbal base coordinate system to the first coordinate system, β is the angle of rotation along the y-axis of the first coordinate system required for the transformation from the first coordinate system to the second coordinate system, h is the displacement along the z-axis of the second coordinate system required for the transformation from the second coordinate system to the third coordinate system, d is the displacement along the x-axis of the third coordinate system required for the transformation from the third coordinate system to the fourth coordinate system, and α is the displacement along the x-axis of the fourth coordinate system required for the transformation from the fifth coordinate system to the fifth coordinate system. The rotation angle of the x-axis of the fourth coordinate system, s, is the displacement along the z-axis of the fifth coordinate system required to transform from the fifth coordinate system to the camera coordinate system; wherein, the process of transforming from the gimbal base coordinate system to the camera coordinate system is as follows: transforming from the gimbal coordinate system to the first coordinate system, transforming from the first coordinate system to the second coordinate system, transforming from the second coordinate system to the third coordinate system, transforming from the third coordinate system to the fourth coordinate system, transforming from the fourth coordinate system to the fifth coordinate system, and transforming from the fifth coordinate system to the camera coordinate system.

[0103] In one possible implementation, the coordinate offset includes a lateral and a longitudinal offset between the origin of the robotic arm's base coordinate system and the origin of the camera's coordinate system; the gimbal angle includes a horizontal angle and a pitch angle; and the relationship model is as follows:

[0104]

[0105] in, Let θ be the hand-eye matrix, θ be the angle of rotation along the z-axis of the gimbal base coordinate system during the transformation from the gimbal base coordinate system to the first coordinate system, β be the angle of rotation along the y-axis of the first coordinate system required for the transformation from the first coordinate system to the second coordinate system, h be the displacement along the z-axis of the second coordinate system required for the transformation from the second coordinate system to the third coordinate system, d be the displacement along the x-axis of the third coordinate system required for the transformation from the third coordinate system to the fourth coordinate system, and α be the displacement along the x-axis of the fourth coordinate system required for the transformation from the fourth coordinate system to the fifth coordinate system. The rotation angle is s, which is the displacement along the z-axis of the fifth coordinate system required to transform from the fifth coordinate system to the camera coordinate system; x is the lateral offset; and y is the longitudinal offset. The process of transforming from the gimbal base coordinate system to the camera coordinate system is as follows: transforming from the gimbal coordinate system to the first coordinate system, transforming from the first coordinate system to the second coordinate system, transforming from the second coordinate system to the third coordinate system, transforming from the third coordinate system to the fourth coordinate system, transforming from the fourth coordinate system to the fifth coordinate system, and transforming from the fifth coordinate system to the camera coordinate system.

[0106] In one possible implementation, the device further includes:

[0107] The transformation module is used to perform matrix transformation on the relationship model to obtain the relationship model between the camera coordinates and the gimbal angle as follows:

[0108]

[0109]

[0110]

[0111] Where (cam_x, cam_y, cam_z) are the coordinates in the camera coordinate system corresponding to the coordinates in the robotic arm coordinate system, α is the pitch angle in the gimbal angle, θ is the horizontal angle in the gimbal angle, β is the mounting angle of the camera on the gimbal, x is the lateral offset in the coordinate offset, y is the longitudinal offset in the coordinate offset, and h is the displacement along the z-axis of the gimbal base coordinate system required to transform from the gimbal base coordinate system to the camera coordinate system.

[0112] Reference Figure 5 This illustration shows a schematic diagram of a hand-eye calibration device provided in an embodiment of this application. The device is applied to a terminal device, which includes a gimbal and a robotic arm. A camera is mounted on the gimbal. Specifically, the device may include a target relationship model determination module 51 and a target hand-eye matrix calibration module 52, wherein:

[0113] The target relationship model determination module 51 is used to determine the target relationship model of the terminal device. The target relationship model is the relationship model between the gimbal angle of the terminal device and the hand-eye matrix. The hand-eye matrix is ​​used to characterize the coordinate transformation relationship from the robot arm base coordinate system to the camera coordinate system.

[0114] The target hand-eye matrix calibration module 52 is used to input the current gimbal angle into the target relationship model to obtain the current target hand-eye matrix. The target hand-eye matrix is ​​used to characterize the coordinate transformation relationship from the robot arm base coordinate system to the camera coordinate system under the current gimbal angle.

[0115] The target relationship model is established based on the method for determining the relationship model of the hand-eye matrix described above.

[0116] In one possible implementation, the target relation model determination module 51 mentioned above includes:

[0117] The initial calibration submodule is used to determine at least two initial hand-eye matrices under multiple different initial gimbal angles;

[0118] The relation determination submodule is used to obtain multiple corresponding relational expressions based on the relational model of the hand-eye matrix and multiple initial hand-eye matrices. The relational model includes multiple undetermined coefficients, and the relational expressions are used to characterize the relationship between the undetermined coefficients and the gimbal angle.

[0119] The coefficient value calculation submodule is used to calculate the coefficient values ​​of multiple undetermined coefficients based on multiple of the aforementioned relational formulas.

[0120] The target relation model determination submodule is used to substitute the coefficient values ​​into the relation model to obtain the target relation model;

[0121] The relationship model is a relationship model of the hand-eye matrix established by the method for determining the relationship model of the hand-eye matrix.

[0122] In one possible implementation, the above relational determination submodule includes:

[0123] A transformation relationship determination unit is used to transform the relationship model to obtain multiple transformation relationships regarding the undetermined coefficients;

[0124] The relation determination unit is used to convert multiple initial hand-eye matrices into multiple corresponding relation expressions with respect to the undetermined coefficients based on multiple transformation relations.

[0125] In one possible implementation, the multiple transformation relationships include:

[0126]

[0127]

[0128]

[0129]

[0130]

[0131]

[0132] Where (cam_x, cam_y, cam_z) are the coordinates in the camera coordinate system corresponding to the coordinates in the robotic arm coordinate system, α is the pitch angle of the gimbal, β is the mounting angle of the camera on the gimbal, and θ is the horizontal angle of the gimbal. ) represents the pose of the camera in the base coordinate system of the robotic arm, and x, y, h, d and s are the undetermined coefficients.

[0133] In one possible implementation, the above coefficient value calculation submodule includes:

[0134] The correction unit is used to correct the initial gimbal angle;

[0135] The equation-establishing unit is used to determine multiple equations for solving the coefficient values ​​based on the multiple transformation relationships and the corrected initial gimbal angle.

[0136] The solving unit is used to solve multiple of the equations to obtain the coefficient values.

[0137] In one possible implementation, the initial gimbal angle includes a horizontal angle, an installation angle, and a pitch angle, and the aforementioned correction unit includes:

[0138] The correction subunit is used to correct the gimbal angle using the following formula:

[0139]

[0140] in, The horizontal angle is... The pitch angle is... Let α be the installation angle, β be the corrected pitch angle, θ be the corrected horizontal angle, and N be the number of the initial hand-eye matrices, where N is greater than or equal to 2.

[0141] As the apparatus embodiments are basically similar to the method embodiments, they are described in a relatively simple manner. For relevant details, please refer to the description in the method embodiment section.

[0142] Figure 6 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Figure 6 As shown, the terminal device 6 in this embodiment includes: at least one processor 60 ( Figure 6 (Only one is shown in the diagram) a processor, a memory 61, and a computer program 62 stored in the memory 61 and executable on the at least one processor 60, wherein the processor 60 executes the computer program 62 to implement the steps in any of the above method embodiments.

[0143] The terminal device 6 may be a robot or other intelligent device. This terminal device may include, but is not limited to, a processor 60 and a memory 61. Those skilled in the art will understand that... Figure 6 This is merely an example of terminal device 6 and does not constitute a limitation on terminal device 6. It may include more or fewer components than shown in the figure, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0144] The processor 60 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0145] In some embodiments, the memory 61 may be an internal storage unit of the terminal device 6, such as a hard disk or memory of the terminal device 6. In other embodiments, the memory 61 may be an external storage device of the terminal device 6, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the terminal device 6. Furthermore, the memory 61 may include both internal and external storage units of the terminal device 6. The memory 61 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory 61 can also be used to temporarily store data that has been output or will be output.

[0146] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0147] This application provides a computer program product that, when run on a terminal device, enables the terminal device to implement the steps described in the various method embodiments above.

[0148] The embodiments described above are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A hand-eye calibration method, characterized in that, Applied to a terminal device, the terminal device including a gimbal and a robotic arm, the gimbal having a camera mounted on it, the method includes: Determine at least two initial hand-eye matrices under multiple different initial gimbal angles; Based on the hand-eye matrix relationship model and multiple initial hand-eye matrices, multiple corresponding relational expressions are obtained. The relationship model includes multiple undetermined coefficients, and the relational expressions are used to characterize the relationship between the undetermined coefficients and the gimbal angle. Based on the multiple relationships described, calculate the coefficient values ​​of the multiple undetermined coefficients; Substituting the coefficient values ​​into the relational model yields the target relational model, which is the relational model between the gimbal angle of the terminal device and the hand-eye matrix. The hand-eye matrix is ​​used to characterize the coordinate transformation relationship from the robot arm base coordinate system to the camera coordinate system. The current gimbal angle is input into the target relationship model to obtain the current target hand-eye matrix. The target hand-eye matrix is ​​used to characterize the coordinate transformation relationship from the robot arm base coordinate system to the camera coordinate system under the current gimbal angle. The step of calculating the coefficient values ​​of multiple undetermined coefficients based on multiple relational expressions includes: The initial gimbal angle is corrected; Based on the multiple transformation relationships and the corrected initial gimbal angle, multiple equations are determined for solving the coefficient values; Solve the multiple equations to obtain the coefficient values; The initial gimbal angle includes the horizontal angle, the mounting angle, and the pitch angle. Correcting the gimbal angle includes: The gimbal angle is corrected using the following formula: in, The horizontal angle is... The pitch angle is... Let α be the installation angle, β be the corrected pitch angle, θ be the corrected horizontal angle, and N be the number of the initial hand-eye matrices, where N is greater than or equal to 2. ( ) represents the pose of the camera in the robot arm's base coordinate system.

2. The method as described in claim 1, characterized in that, The relationship model of the hand-eye matrix is ​​determined in the following way: Determine the coordinate offset between the origin of the robotic arm base coordinate system and the origin of the gimbal base coordinate system, and determine the coordinate transformation matrix between the gimbal base coordinate system and the camera coordinate system. The gimbal base coordinate system is a coordinate system established with the center of the rotation axis of the gimbal base as the origin. The camera coordinate system is a coordinate system established with the focus center of the camera as the origin. The robotic arm base coordinate system is a coordinate system established with the center of the rotation axis of the robotic arm base as the origin. Based on the coordinate offset and the coordinate transformation matrix, a relationship model of the hand-eye matrix is ​​determined. The relationship model can characterize the relationship between the hand-eye matrix and the gimbal angle. The gimbal angle is the angle of the gimbal relative to the gimbal base on different planes. The hand-eye matrix is ​​used to characterize the coordinate transformation relationship from the robot arm base coordinate system to the camera coordinate system.

3. The method as described in claim 2, characterized in that, The coordinate transformation matrix between the gimbal base coordinate system and the camera coordinate system is: in, Let θ be the coordinate transformation matrix, where θ is the rotation angle of the gimbal base coordinate system along the z-axis of the gimbal base coordinate system during the transformation from the gimbal base coordinate system to the first coordinate system, β is the rotation angle along the y-axis of the first coordinate system during the transformation from the first coordinate system to the second coordinate system, h is the translation along the z-axis of the second coordinate system during the transformation from the second coordinate system to the third coordinate system, d is the translation along the x-axis of the third coordinate system required for the transformation from the third coordinate system to the fourth coordinate system, and α is the translation along the x-axis of the third coordinate system during the transformation from the fourth coordinate system to the fifth coordinate system. The angle of rotation along the x-axis of the fourth coordinate system, s is the displacement along the z-axis of the fifth coordinate system during the transformation from the fifth coordinate system to the camera coordinate system; wherein, the process of transforming from the gimbal base coordinate system to the camera coordinate system is as follows: transforming from the gimbal coordinate system to the first coordinate system, transforming from the first coordinate system to the second coordinate system, transforming from the second coordinate system to the third coordinate system, transforming from the third coordinate system to the fourth coordinate system, transforming from the fourth coordinate system to the fifth coordinate system, and transforming from the fifth coordinate system to the camera coordinate system.

4. The method as described in claim 2, characterized in that, The coordinate offset includes the lateral and longitudinal offsets between the origin of the robotic arm's base coordinate system and the origin of the gimbal's base coordinate system; the gimbal angle includes the horizontal angle and the pitch angle; and the relationship model is as follows: in, Let θ be the hand-eye matrix, θ be the angle of rotation of the gimbal base coordinate system along the z-axis of the gimbal base coordinate system during the transformation from the gimbal base coordinate system to the first coordinate system, β be the angle of rotation along the y-axis of the first coordinate system during the transformation from the first coordinate system to the second coordinate system, h be the displacement along the z-axis of the second coordinate system during the transformation from the second coordinate system to the third coordinate system, d be the displacement along the x-axis of the third coordinate system during the transformation from the third coordinate system to the fourth coordinate system, and α be the displacement along the x-axis of the fourth coordinate system during the transformation from the fourth coordinate system to the fifth coordinate system. The rotation angle of the axis, s is the displacement along the z-axis of the fifth coordinate system during the transformation from the fifth coordinate system to the camera coordinate system, x is the lateral offset, and y is the longitudinal offset; wherein, the process of transforming from the gimbal base coordinate system to the camera coordinate system is as follows: transforming from the gimbal coordinate system to the first coordinate system, transforming from the first coordinate system to the second coordinate system, transforming from the second coordinate system to the third coordinate system, transforming from the third coordinate system to the fourth coordinate system, transforming from the fourth coordinate system to the fifth coordinate system, and transforming from the fifth coordinate system to the camera coordinate system.

5. The method as described in claim 3 or 4, characterized in that, After determining the hand-eye matrix relationship model based on the coordinate offset and the coordinate transformation matrix, the method further includes: Performing a matrix transformation on the aforementioned relationship model yields the following relationship model between the camera coordinates and the gimbal angle: Where (cam_x, cam_y, cam_z) are the coordinates in the camera coordinate system corresponding to the coordinates in the robotic arm coordinate system, α is the pitch angle in the gimbal angle, θ is the horizontal angle in the gimbal angle, β is the mounting angle of the camera on the gimbal, x is the lateral offset in the coordinate offset, y is the longitudinal offset in the coordinate offset, and h is the displacement along the z-axis of the gimbal base coordinate system required to transform from the gimbal base coordinate system to the camera coordinate system.

6. The method as described in claim 5, characterized in that, The relationship model based on the hand-eye matrix and multiple initial hand-eye matrices yields multiple corresponding relational expressions, including: The relational model is transformed to obtain multiple transformation relations concerning the undetermined coefficients; Based on the multiple transformation relationships, the multiple initial hand-eye matrices are converted into multiple corresponding relational expressions with respect to the undetermined coefficients.

7. The method as described in claim 6, characterized in that, The multiple transformation relationships include: Where (cam_x, cam_y, cam_z) are the coordinates in the camera coordinate system corresponding to the coordinates in the robotic arm coordinate system, α is the pitch angle of the gimbal, β is the mounting angle of the camera on the gimbal, and θ is the horizontal angle of the gimbal. ) represents the pose of the camera in the base coordinate system of the robotic arm, and x, y, h, d and s are the undetermined coefficients.

8. A robot, characterized in that, The robot performs hand-eye calibration using the method described in any one of claims 1-7.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1-7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-7.

Images