Hand-eye calibration method, hand-eye calibration system, computer device and storage device

Abstract

The application discloses a hand-eye calibration method, a hand-eye calibration system, a computer device and a storage device. The method comprises the following steps: setting a marker on a robot, obtaining first sampling data, and obtaining a rotation relationship matrix of a camera coordinate system of the robot relative to a calibration coordinate system by using the first sampling data, wherein the camera coordinate system corresponds to a camera device, the calibration coordinate system corresponds to a calibration position of the robot, and the calibration position is a base position or an end position; controlling the end position of the robot to move around the coordinate axis of the camera coordinate system by using the rotation relationship matrix, and obtaining second sampling data; and obtaining a translation relationship between the camera coordinate system and the calibration coordinate system of the robot by using the rotation relationship matrix and the second sampling data, wherein the rotation relationship matrix and the translation relationship are used for hand-eye calibration of the camera coordinate system and the calibration coordinate system. The above scheme can improve the applicability of hand-eye calibration.

Description

Technical Field

[0001] This application relates to the field of robotics technology, and in particular to a hand-eye calibration method, a hand-eye calibration system, a computer device, and a storage device. Background Technology

[0002] With the development of artificial intelligence technology and people's increasing demands for quality of life, intelligent robots are gradually appearing in people's daily lives, such as cleaning robots, industrial robots, service robots, and robots that move goods in warehouses.

[0003] For example, in industrial applications, robots have vision systems. Using the images acquired by these systems, robots can control end effectors to perform machining and assembly actions. In other words, the vision system is like the human eye, and the end effector is like the human hand; through the coordination of hand and eye, pre-set action tasks are completed.

[0004] Existing robot hand-eye calibration methods typically require manual collection of calibration parameters, resulting in low automation, high manpower requirements, and a high risk of errors and poor repeatability. Furthermore, existing hand-eye calibration methods are not suitable for robots with complex movements. Summary of the Invention

[0005] The main technical problem addressed by this application is to provide a hand-eye calibration method, a hand-eye calibration system, a computer device, and a storage device, which can improve the applicability of hand-eye calibration.

[0006] To address the aforementioned problems, the first aspect of this application provides a hand-eye calibration method, comprising: setting markers on a robot, acquiring first sampling data, and using the first sampling data to acquire a rotation relationship matrix of the robot's camera coordinate system relative to a calibration coordinate system, wherein the camera coordinate system corresponds to a camera device, and the calibration coordinate system corresponds to a calibration part of the robot, the calibration part being a base part or an end effector part; using the rotation relationship matrix to control the robot's end effector part to move around the coordinate axes of the camera coordinate system, and acquiring second sampling data; and using the rotation relationship matrix and the second sampling data to acquire a translation relationship between the camera coordinate system and the robot's calibration coordinate system, wherein the rotation relationship matrix and the translation relationship are used for hand-eye calibration of the camera coordinate system and the calibration coordinate system.

[0007] To address the aforementioned problems, a second aspect of this application provides a hand-eye calibration system, comprising a robot and a camera device. The robot includes an end effector and a base. The camera device is disposed on the end effector of the robot; or, the camera device is disposed outside the end effector of the robot. The combination of the camera device and the robot is used to perform any step of the aforementioned hand-eye calibration method to determine the hand-eye calibration relationship between the camera device and the calibration part of the robot.

[0008] To address the aforementioned problems, a third aspect of this application provides a computer device comprising a memory and a processor coupled to each other, wherein the memory stores program data and the processor executes the program data to implement any step of the aforementioned hand-eye calibration method.

[0009] To address the aforementioned problems, a fourth aspect of this application provides a storage device that stores program data executable by a processor, the program data being used to implement any step of the aforementioned hand-eye calibration method.

[0010] The above scheme involves setting a marker on the robot, acquiring first sampling data, and using this first sampling data to obtain the rotation relationship matrix of the robot's camera coordinate system relative to the calibration coordinate system. Since the rotation relationship matrix describes the position and orientation relationship between the camera and the calibration coordinate system (base coordinate system or end effector coordinate system), by using the rotation relationship matrix to control the robot's end effector to move around the coordinate axes of the camera coordinate system to acquire second sampling data, the marker can move within the field of view of the camera device, avoiding the cantilever effect and acquiring second sampling data over a larger range. In addition, the rotation relationship matrix and the second sampling data are used to obtain the translation relationship between the camera coordinate system and the robot's calibration coordinate system. The rotation relationship matrix and the translation relationship are used for hand-eye calibration of the camera coordinate system and the calibration coordinate system, which can improve the applicability and accuracy of hand-eye calibration. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in this application, the accompanying drawings required in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Among them:

[0012] Figure 1 This is a schematic diagram of the structure of an embodiment of the eye-in-hand external system of this application;

[0013] Figure 2 This is a schematic diagram of the structure of an embodiment of the Eye in Hand system of this application;

[0014] Figure 3This is a flowchart illustrating the first embodiment of the hand-eye alignment method of this application;

[0015] Figure 4 This application Figure 3 A flowchart illustrating an embodiment of step S11;

[0016] Figure 5 This application Figure 3 A flowchart illustrating an embodiment of step S12;

[0017] Figure 6 This application Figure 3 A flowchart illustrating an embodiment of step S13;

[0018] Figure 7 This is a flowchart illustrating the second embodiment of the hand-eye alignment method of this application;

[0019] Figure 8 This is a flowchart illustrating the third embodiment of the hand-eye alignment method of this application;

[0020] Figure 9 This is a schematic diagram of the structure of an embodiment of the hand-eye calibration system provided in this application;

[0021] Figure 10 This is a schematic diagram of another embodiment of the hand-eye calibration system provided in this application;

[0022] Figure 11 This is a schematic diagram of the structure of an embodiment of the computer device of this application;

[0023] Figure 12 This is a schematic diagram of the structure of an embodiment of the storage device of this application. Detailed Implementation

[0024] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0025] The terms "first" and "second" in this application are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or apparatus.

[0026] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0027] Currently, before using a robot, it is usually necessary to perform hand-eye calibration between the robot and the camera device, that is, to unify the coordinate system of the vision system and the robot's coordinate system. This allows the object pose determined by the coordinate system corresponding to the camera device's vision system to be transformed into the robot's coordinate system, so that the robot's robotic arm can complete the task on the object. Examples of robots include SCARA (Selective Compliance Assembly Robot Arm) robots, six-axis robotic arms, and other industrial robots. This application uses a six-axis robotic arm robot as an example for explanation below. It should be understood that the methods of the embodiments of this application are also applicable to other robots, and this application is not limited thereto.

[0028] Please see Figures 1-2 The hand-eye calibration system can include two hand-eye calibration methods. For example, robot 100 can include a base portion 101 and an end effector portion 102. The end effector portion 102 can be the end of the robotic arm of robot 100, and can include manipulators (grippers, suction cups, etc.) for performing operations on objects. Furthermore, markers 300 can be placed on the end effector portion 102. Camera device 200 can be mounted on the end effector portion 102 or fixed elsewhere outside the robot 100; this application does not impose any limitations on this.

[0029] Please see Figure 1In an eye-in-hand system, the camera device 200 is mounted outside the robot 100, while the robot 100 remains within the field of view of the camera device 200. In this mounting configuration, the camera device 200 is stationary relative to the base 101, allowing for the calibration of the hand-eye alignment between the coordinate system of the camera device 200 and the coordinate system of the base 101. In other words, the hand-eye alignment refers to the coordinate transformation relationship between the camera coordinate system and the base coordinate system.

[0030] Please see Figure 2 For the eye-on-hand system, the camera device 200 is mounted on the end effector 102 of the robot 100. The camera device 200 is stationary relative to the end effector 102, which allows for the calibration of the hand-eye calibration relationship between the coordinate system corresponding to the camera device 200 and the coordinate system corresponding to the end effector 102. In other words, the hand-eye calibration relationship refers to the coordinate transformation relationship from the camera coordinate system to the end effector coordinate system.

[0031] Through long-term research, the inventors of this application have discovered that most existing hand-eye calibration methods are designed for robots with relatively simple movements. However, when calibrating hand-eye movements for robots with more complex movements, the cantilever effect often causes the robot's end effector to shift or position part of the calibrated object outside the camera's field of view. Therefore, during hand-eye calibration, to avoid moving the object outside the camera's field of view, sampling must be limited to a small area, making it unsuitable for robots with complex movements and affecting the accuracy of the hand-eye calibration results for both the camera and the robot.

[0032] To address the aforementioned problems, this application provides the following embodiments, which are described in detail below.

[0033] Please see Figure 3 , Figure 3 This is a flowchart illustrating the first embodiment of the hand-eye alignment method of this application. The method may include the following steps:

[0034] S11: Set markers for the robot, acquire the first sampled data, and use the first sampled data to obtain the rotation relationship matrix of the robot's camera coordinate system relative to the calibration coordinate system.

[0035] Among them, the camera coordinate system corresponds to the camera device, and the calibration coordinate system corresponds to the calibration part of the robot, which is the base part or the end part.

[0036] Before performing hand-eye calibration, a hand-eye calibration system is constructed. This system can be an eye-outside-hand system or an eye-on-hand system. A marker can be placed at the end of the robot. The marker can be a calibration board, calibration paper, or other calibration item. The marker contains at least one identifiable marker point and is within the field of view of the camera device. This application does not impose any limitations on this.

[0037] In a hand-eye alignment system, the robot can be an industrial robot, and the camera device can be a 3D (3-dimensional) camera. Different parts can correspond to different coordinate systems. The robot's base corresponds to the base coordinate system, which can be represented by 'b'; the end effector corresponds to the end effector coordinate system, which can be represented by 'f'; the camera device corresponds to the camera coordinate system, which can be represented by 'c'; and additionally, markers can be represented by 'm'.

[0038] In some embodiments, there is a connection between the camera device and the robot, and each of the camera device and the robot can be connected to a processor or processing terminal, etc., so that the robot and the camera device can be controlled separately. Alternatively, processors can be set in the robot and the camera device respectively. This application does not limit this.

[0039] In some implementations, the robot can be controlled to perform motion sampling to obtain sampling data, such as controlling the end effector of the robot's robotic arm to move or rotate. Specifically, the end effector of the robot can be controlled to move along the coordinate axes of the end effector coordinate system to obtain at least one first sampling data.

[0040] When the hand-eye calibration system is an eye-on-hand system, the calibration location is the base location, meaning the calibration coordinate system is the base coordinate system. When the hand-eye calibration system is an eye-on-hand system, the calibration location is the end-effector location, meaning the calibration coordinate system is the end-effector coordinate system. The following embodiments of this application refer to this description, with this embodiment primarily using the eye-on-hand system as an example.

[0041] In some implementations, the rotation matrix of the camera coordinate system relative to the calibration coordinate system can be obtained by using the unit vector of pose movement before and after the movement. When the calibration coordinate system is the base coordinate system, this rotation matrix can be expressed as: c R b This refers to the rotation relationship between the base coordinate system (or base part) and the camera coordinate system. The following description of the coordinate system relationship between the two parts can be referenced from this description.

[0042] In some embodiments, please refer to Figure 4 The above embodiment can be further extended by step S11. Acquiring first sampling data and using the first sampling data to obtain the rotation relationship matrix of the robot's camera coordinate system relative to the calibration coordinate system can include the following steps:

[0043] S111: Obtain the initial pose of the robot's end effector and the initial sampling data of the marker at the initial point in the camera coordinate system.

[0044] Control the robot's movement to bring the marker within the camera's field of view, and set this initial point as the initial position. Obtain the initial pose of the end effector at this initial point. b T f 0. The initial pose is the positional relationship between the end effector coordinate system and the base coordinate system. The initial pose can be a 6-dimensional vector including rotation and translation. It can be directly read from the readings of the robot arm controller. For example, the initial pose is ( b T f )0(X0, Y0, z0, Rx0, Ry0, Rz0). And obtain the initial sampling data of the marker in the camera coordinate system, that is, the initial position of the marker in the camera coordinate system ( c P m )0, this initial position only includes a three-dimensional translation vector. When the end effector of the robotic arm moves to a certain position, it triggers the camera to capture an image of the marker. Based on the image, image processing can be used to obtain the marker's coordinates in the camera coordinate system. For example, if the initial position is ( c P m )0(x0, y0, z0).

[0045] S112: Control the end effector of the robot to move a preset distance along at least one coordinate axis in the calibrated coordinate system to obtain the first sampling data of the marker at at least one first sampling point in the camera coordinate system.

[0046] The robot is moved sequentially along at least one coordinate axis (X-axis, Y-axis, Z-axis) of the calibration coordinate system by a preset distance to obtain at least one first sampled data of the marker in the camera coordinate system, that is, the position coordinates.

[0047] Taking the eye-to-hand system as an example, the end effector of the robot is moved sequentially along its X-axis, Y-axis and Z-axis in the base coordinate system by a preset distance. The preset distance along each axis can be different, and the coordinate position of the marker in the camera coordinate system before and after the movement is obtained.

[0048] For example, the pose of the front end portion is ( b T f )0(X0, Y0, z0, Rx0, Ry0, Rz0), the coordinate position of the marker in the camera coordinate system is ( c P m )0(x0, y0, z0). The end part moves a preset distance along the X-axis in the base coordinate system to reach the first sampling point corresponding to the X-axis. After the movement, the pose of the end part is ( b T f )1(X1, Y0, Z0, Rx0, Ry0, Rz0), the coordinate position of the marker in the camera coordinate system is ( cP m )1(x1, y1, z1), that is, the first sampled data is ( c P m )1(x1, y1, z1).

[0049] In some implementations, if it is an eye-on-hand system, the end effector of the robot is moved sequentially along its X-axis, Y-axis, and Z-axis in the end effector coordinate system by a preset distance. The preset distance along each axis can be different, and the coordinate position of the marker in the camera coordinate system before and after the movement is obtained.

[0050] S113: Based on the initial sampling data of the marker and at least one first sampling data, obtain the displacement vector of the marker corresponding to the first sampling point in the camera coordinate system.

[0051] By using the initial sampling data of the marker and the first sampling data obtained after moving along each axis, the displacement vector of the marker at the first sampling point in the camera coordinate system can be obtained. This displacement vector can be the distance vector (v) of the movement. x v y v z It can also be a moving unit vector (v) x v y v z This application does not impose any restrictions on this.

[0052] Taking the displacement vector as the unit vector of movement and the movement along the X-axis by a preset distance as an example, the initial sampling data of the marker can be used ( c P m )0(x0, y0, z0), the first sampled data is ( c P m The displacement vector Vx(v) can be obtained from 1(x1, y1, z1) using the following formula. x v y v z The specific expression of this formula is as follows:

[0053]

[0054] In the above formula (1), V x This represents the displacement vector obtained by moving along the X-axis.

[0055] In some embodiments, by repeating step S112 above, the end portion is controlled to move sequentially along the Y-axis and Z-axis in the base coordinate system by a preset distance, respectively, and the first sampling data corresponding to the Y-axis and Z-axis are obtained respectively. Thus, the components V of the displacement vectors corresponding to the Y-axis and Z-axis can be obtained respectively through the above formula (1). y V z .

[0056] S114: Using at least one displacement vector, obtain the rotation relationship matrix of the robot's camera coordinate system relative to the calibration coordinate system.

[0057] Taking the eye-in-hand extra-system as an example, with the calibration coordinate system as the base coordinate system, the above can yield three displacement vectors, namely displacement vector V. x Displacement vector V y Displacement vector V z These three displacement vectors can be used as the rotation matrix of the robot's camera coordinate system relative to the base coordinate system. c R b =[V x V y V z ].

[0058] In some implementations, if it is an eye-on-the-hand system, referring to the specific implementation process of the eye-on-the-hand external system described above, the rotation relationship matrix of the camera coordinate system relative to the end-effector coordinate system can be obtained. c R f =[-V x -V y -V z ].

[0059] In this embodiment, the displacement vector of the marker corresponding to the first sampling point in the camera coordinate system is obtained by using the initial sampling data of the marker and the first sampling data obtained along the coordinate axes of the calibration coordinate system. Thus, the rotation relationship matrix of the robot's camera coordinate system relative to the calibration coordinate system is obtained, and the position and attitude of the camera device relative to the calibration coordinate system can be obtained.

[0060] S12: Using the rotation relation matrix, control the robot's end effector to move around the coordinate axes of the camera coordinate system to obtain the second sampling data.

[0061] The transpose of the rotation relation matrix can be used to obtain the transposed rotation relation matrix. b R c Using the transpose rotation relation matrix b R c Construct a new coordinate system such that the robot's end effector revolves around a matrix with respect to the transpose of the rotation relation. b R c The robot's end effector is controlled to rotate and translate multiple times around the coordinate axes of the camera coordinate system to obtain multiple second sampling data, including attitude data and position data.

[0062] In some implementations, step S12 can be repeated at least three times to obtain multiple second sample data. The number of samplings can be greater than three to obtain more second sample data, and this application does not impose any limitation on this.

[0063] In some embodiments, please refer to Figure 5 This embodiment can be further extended to step S12 of the above embodiment. Using a rotation relation matrix, the robot's end effector is controlled to move around the coordinate axes of the camera coordinate system to acquire second sampling data. This embodiment may include the following steps:

[0064] S121: Obtain the first attitude data of the end part in the base coordinate system at the first movement point; wherein, the first movement point includes the initial point.

[0065] Obtain the first attitude data of the end part in the base coordinate system at the first moving point, that is, the attitude of the end part before rotation.

[0066] In some implementations, the first movement point includes an initial point, and the attitude data of the initial pose of the end portion can be used as the first attitude data. b R f )0.

[0067] S122: Construct a first rotation coordinate system based on the rotation relationship matrix, control the end part of the robot to rotate around any coordinate axis of the first rotation coordinate system by a first preset angle, and obtain the second attitude data and the second position data of the end part of the second movement point in the base coordinate system.

[0068] Rotation relation matrix can be used c R b Transpose the matrix to obtain the transpose rotation relation matrix. b R c The rotation relation matrix is ​​derived from the transpose. b R c A first rotational coordinate system is constructed, allowing the end effector of the robot to rotate around the coordinate axes (X-axis, Y-axis, Z-axis) of this system. The rotation angle can be a first preset angle, which is a range of angles that ensures the marker remains within the camera's field of view, allowing for small-angle rotations. For example, if the first preset angle is 5°, and the camera cannot capture a valid image of the marker at 5°, the first preset angle can be set to 4°.

[0069] In some implementations, rotations can be performed sequentially around the X-axis, Y-axis, and Z-axis of the first rotating coordinate system. Each rotation is performed around any one of the coordinate axes of the first rotating coordinate system, and multiple rotations include rotations around the X-axis, Y-axis, and Z-axis.

[0070] In some implementations, after rotation to the second displacement point, the second attitude data of the end portion in the base coordinate system at this point can be recorded. b R f )1, and the second position data of the rotated end portion ( b t f 1.

[0071] For example, by controlling the end effector of the robot to rotate sequentially around the X, Y, and Z axes of the first rotation coordinate system by a1, a2, and a3 respectively (where a1, a2, and a3 are angle values), the second posture data of the end effector in the base coordinate system can be obtained. b R f 1. Details are as follows:

[0072] ( b R f )1= b R c Rz(a3)Ry(a2)Rx(a1) c R b ( b R f )0 (2)

[0073] In this step, since the transpose rotation relation matrix can describe the position and orientation of the camera in the calibration coordinate system (base coordinate system or end coordinate system), such as the transpose rotation relation matrix... b R c The rotational relationship between the base coordinate system and the camera coordinate system is described. Therefore, this step can control the marker to always rotate in front of the camera device, so that the marker is within the field of view of the camera device and the camera device can capture an effective image of the marker, thus avoiding the cantilever effect.

[0074] S123: Using the rotation relation matrix, control the end part to move to the first moving point and obtain the first position data of the end part in the base coordinate system.

[0075] Using the transpose of the rotation relation matrix, we can deduce the vector that the robot's end effector needs to move back to the first movement point in the base coordinate system. b t m )1, thereby controlling the end part to move to the position of the first moving point ( c P m 0, obtain the first position data of the end part in the base coordinate system at this time. b t f )0.

[0076] For example, the initial position of the marker in the camera coordinate system is ( c P m0. After the robot's end effector rotates 5° around the X-axis of the first rotation coordinate system, the second attitude data of the end effector is ( b R f 1. The coordinates of the marker in the camera coordinate system are ( c P m 1. The translation amount of the marker's movement in the camera coordinate system can be obtained from the following formula:

[0077] ( c t m )1=( c P m )1-( c P m )0 (3)

[0078] Since the marker is fixed at the end, the translation amount of the marker can be equal to the translation amount of the end, that is:

[0079] ( b t m )1= b R c ( c t m )1 (4)

[0080] From the above formula (4), it can be seen that the vector that the robot's end effector (marker) needs to move back to the first moving point is, that is, the translation amount. b t m )1, then using this translation amount ( b t m 1. The marker at the end of the robot can be controlled to move back to its initial position. c P m )0.

[0081] S124: Repeat the above steps at least three times to obtain at least three sets of second sampling data, wherein each set of second sampling data includes attitude data and position data, the attitude data includes first attitude data and second attitude data, and the position data includes first position data and second position data.

[0082] Repeating steps S121 to S123 above yields at least three sets of second sampled data. From steps S121 to S123 above, one set of second sampled data is obtained, wherein the second sampled data may include the acquired attitude data and position data, the attitude data also including the first attitude data (…). b R f )0 and second attitude data, position data, that is, including first position data ( b t f )0 and second position data ( b tf 1.

[0083] In this embodiment, a first rotation coordinate system is constructed based on a rotation relationship matrix, and the end part of the robot is controlled to rotate around any coordinate axis of the first rotation coordinate system. This ensures that the control marker always rotates towards the camera device, avoiding the cantilever effect and allowing sampling to be performed over a larger range. The obtained sampling data is used for hand-eye calibration, which can improve the accuracy of hand-eye calibration between the robot and the camera device.

[0084] S13: Using the rotation relation matrix and the second sampled data, obtain the translation relationship between the camera coordinate system and the robot's calibration coordinate system. The rotation relation matrix and the translation relationship are used to perform hand-eye calibration between the camera coordinate system and the calibration coordinate system.

[0085] Based on multiple samplings of second-sample data, the calibrated end-effector position of the marker in the end-effector coordinate system corresponding to the robot's end-effector is obtained. Using the calibrated end-effector position and the pose of the end-effector in the base coordinate system, the calibrated base position of the marker in the base coordinate system is obtained. Knowing the marker's position in the camera coordinate system, i.e., the calibrated camera position, the translation relationship between the camera coordinate system and the calibrated coordinate system can be obtained using the rotation relation matrix, the calibrated base position, and the calibrated camera position, through the known transformation relationships between the camera coordinate system, the base coordinate system, and the end-effector coordinate system.

[0086] In some implementations, rotation and translation relationships are used for hand-eye calibration between the camera coordinate system and the calibration coordinate system. For example, the translation relationship can be used as the hand-eye calibration relationship between the camera coordinate system and the calibration coordinate system. For instance, if the second sample data is sampled more than three times, multiple translation relationships between the camera coordinate system and the calibration coordinate system can be obtained. The optimal translation relationship can be obtained using the least squares method, and then used as the hand-eye calibration relationship between the camera coordinate system and the calibration coordinate system.

[0087] In other implementations, refined sampling can be performed using rotation and translation matrices, and the translation relationship can be optimized using the sampled data. For example, the target sampling pose of the third sampling point can be determined based on the translation relationship; the third sampling data can then be obtained using the target sampling pose of the third sampling point, and the hand-eye calibration relationship of the camera coordinate system relative to the calibration coordinate system can be obtained using the third sampling data. Specific implementation details can be found in the following embodiments.

[0088] In this embodiment, a marker is set on the robot to obtain first sampling data. The first sampling data is used to obtain the rotation relationship matrix of the robot's camera coordinate system relative to the calibration coordinate system. Since the rotation relationship matrix describes the position and attitude relationship between the camera and the calibration coordinate system (base coordinate system or end effector coordinate system), by using the rotation relationship matrix, the robot's end effector is controlled to move around the coordinate axes of the camera coordinate system to obtain second sampling data. This allows the marker to move within the field of view of the camera device, avoiding the cantilever effect and obtaining second sampling data over a larger range. In addition, the rotation relationship matrix and the second sampling data are used to obtain the translation relationship between the camera coordinate system and the robot's calibration coordinate system. The rotation relationship matrix and the translation relationship are used for hand-eye calibration of the camera coordinate system and the calibration coordinate system, which can improve the applicability and accuracy of hand-eye calibration.

[0089] In some embodiments, please refer to Figure 6 This embodiment can further extend step S13 of the above embodiment. Using the rotation relationship matrix and the second sampling data, the translation relationship between the camera coordinate system and the robot's calibration coordinate system is obtained. This embodiment may include the following steps:

[0090] S131: Based on the second sampling data, obtain the calibrated end position of the marker in the end coordinate system corresponding to the end part of the robot.

[0091] By using the steps of the above embodiments, at least three sets of second sampling data are obtained. These at least three sets of second sampling data can be used to obtain the position coordinates of the marker in the end-effector coordinate system corresponding to the end-effector of the robot, that is, to calibrate the end-effector position. f P m .

[0092] In some implementations, the calibration end-effector position can be obtained using the attitude difference between at least three sets of attitude data and the position difference between position data from at least three sets of second-acquired data. Specifically, obtaining the calibration end-effector position... f P m The following formula is as follows:

[0093] (( b R f )0-( b R f )1) f P m =( b t f )1-( b t f )0 (5)

[0094] For example, using at least three sets of second sampling data, the calibration end position of the marker in the end coordinate system can be obtained using the following formula.f P m The formula is expressed as follows:

[0095]

[0096] In the above formula (6), the specific acquisition process of the first group of second sample data, the second group of second sample data and the third group of second sample data can refer to the process of acquiring a group of second sample data in step S12 of the above embodiment, which will not be repeated here.

[0097] In some implementations, since steps S121 to S123 are repeated multiple times in the above embodiments, more than three sets of second sampling data can be obtained, that is, the calibration end positions of multiple markers in the end coordinate system can be obtained. f P m Therefore, the least squares method can be used to obtain the values ​​of multiple calibration end positions. f P m This is used for subsequent calculations of the translation relationship between the camera coordinate system and the calibration coordinate system.

[0098] S132: By using the position of the calibration end and the pose of the end part in the base coordinate system, the position of the calibration base of the marker in the base coordinate system is obtained.

[0099] Since the pose of the end portion in the base coordinate system is known. b T f And the calibration end position of the marker in the end coordinate system. f P m The position of the marker in the base coordinate system can be obtained using the following formula, which is also the formula for calibrating the base position. b P m The formula is expressed as follows:

[0100] b P m = b T f f P m (7)

[0101] S133: Using the rotation relationship matrix, the calibration base position, and the calibration camera position, the translation relationship between the camera coordinate system and the calibration coordinate system is obtained; where the calibration camera position represents the position of the marker in the camera coordinate system.

[0102] By utilizing the known coordinate transformation relationships between the base coordinate system, the rotation relationship matrix, and the camera coordinate system, the translation relationship between the camera coordinate system and the calibration coordinate system can be obtained, that is, the relative pose transformation relationship or position relationship between the camera coordinate system and the calibration coordinate system.

[0103] In some implementations, if the system is an eye-outside-hand system, i.e., when the camera device is located outside the robot, the translation relationship between the camera coordinate system and the robot's base coordinate system can be obtained by subtracting the product of the calibration camera position and the rotation relation matrix and the calibration base position. Specifically, the coordinate transformation relationship between the base coordinate system, the rotation relation matrix, and the camera coordinate system can be expressed by the following formula:

[0104]

[0105] In the above formula (8), c t b This indicates the translation relationship between the camera coordinate system and the robot's base coordinate system. c R b This represents the rotation matrix between the camera coordinate system and the base coordinate system. b P m Indicates the position of the calibration base. c P m This indicates the position of the calibrated camera, that is, the position of the marker in the camera coordinate system.

[0106] The above formula (8) can be converted into the following formula:

[0107] c t b = c P m - c R b b P m (9)

[0108] From the above formula (9), the translation relationship between the camera coordinate system and the robot's base coordinate system can be obtained. c t b .

[0109] In some implementations, if it is an eye-on-hand system, i.e., when the camera device is located at the end effector of the robot, the translation relationship between the camera coordinate system and the robot's calibration coordinate system is obtained using the calibration camera position, rotation relationship matrix, calibration base position, attitude transformation matrix between the end effector and the base coordinate system, and the attitude of the end effector in the base coordinate system. This can be obtained using the following formula:

[0110]

[0111] In the above formula (10), b R f This represents the attitude transformation matrix relative to the end-effector coordinate system and the base coordinate system. b tf This indicates the attitude of the end-effector in the base coordinate system. f R c This represents the rotation matrix between the camera coordinate system and the end coordinate system. f t c This represents the translational relationship between the camera coordinate system and the robot's end effector coordinate system, that is, the attitude transformation relationship between the camera coordinate system and the end effector coordinate system. c P m Indicates the calibrated camera position. b P m Indicates the location of the calibration base.

[0112] From the above formula (10), we can obtain the following expression:

[0113] f t c = f R b ( b P m - b R f f R c c P m - b t f (11)

[0114] From the above formula (9), the translation relationship between the camera coordinate system and the robot's end effector coordinate system can be obtained. f t c That is, the relative pose transformation relationship or positional relationship between the camera coordinate system and the end-effector coordinate system.

[0115] In some implementations, after obtaining the translation relationship between the camera coordinate system and the calibration coordinate system, steps S11 to S13 can be executed multiple times to obtain multiple translation relationships between the camera coordinate system and the calibration coordinate system. The optimal translation relationship among the multiple translation relationships can be obtained by methods such as the least squares method. The obtained optimal translation relationship can be used as the hand-eye calibration relationship between the camera coordinate system and the calibration coordinate system.

[0116] In some embodiments, the translation relationship between the camera coordinate system and the calibration coordinate system obtained above can be used for hand-eye calibration of the camera coordinate system and the calibration coordinate system. This translation relationship can be used as the hand-eye calibration relationship between the camera coordinate system and the calibration coordinate system. In addition, this translation relationship can be refined to make the hand-eye calibration relationship between the camera coordinate system and the calibration coordinate system more accurate.

[0117] Please see Figure 7 , Figure 7This is a flowchart illustrating a second embodiment of the hand-eye calibration method of this application. In this embodiment, the translation relationship between the acquired camera coordinate system and the robot's calibration coordinate system can be further refined to make the hand-eye calibration relationship between the camera coordinate system and the calibration coordinate system more accurate. The method may include the following steps:

[0118] S21: Set markers on the robot, obtain the first sampling data, and use the first sampling data to obtain the rotation relationship matrix of the robot's camera coordinate system relative to the calibration coordinate system.

[0119] Among them, the camera coordinate system corresponds to the camera device, and the calibration coordinate system corresponds to the calibration part of the robot, which is the base part or the end part.

[0120] S22: Using the rotation relationship matrix, control the movement of the robot's end effector around the coordinate axes of the camera coordinate system to obtain the second sampling data;

[0121] S23: Using the rotation relation matrix and the second sampled data, obtain the translation relationship between the camera coordinate system and the robot's calibration coordinate system. The rotation relation matrix and the translation relationship are used to perform hand-eye calibration between the camera coordinate system and the calibration coordinate system.

[0122] In this embodiment, the specific implementation process of steps S21 to S23 can refer to the implementation process of steps S11 to S13 in the above embodiment, and will not be repeated here.

[0123] After step S23, the method of this embodiment may further include the following steps:

[0124] S24: Determine the target sampling pose of the third sampling point based on the translation relationship.

[0125] Based on the translation relationship between the obtained camera coordinate system and the robot's calibration coordinate system, the target sampling pose of the third sampling point is determined, including the target sampling position and the target sampling posture.

[0126] In some implementations, grid sampling can be performed on images captured by a camera device, and preset sampling points can be obtained on the pixel plane. Each sampling point on the pixel plane can arbitrarily take the pixel value of one grid point. The preset sampling points can be pre-set sampling points or sampling points obtained by the robot during its movement; this application does not impose any restrictions on this.

[0127] Using the pixel coordinates (u, v) of the preset sampling point on the pixel plane and the camera device's configuration parameters (such as the camera device's intrinsic parameter K), the coordinate position of the third sampling point corresponding to the preset sampling point in actual space is obtained in the camera coordinate system. c P f (xc y c z c ), where z c The value can be z0 (where z0 is the position value in the initial pose). Then, using the translation relationship between the camera coordinate system and the calibration coordinate system, the preset sampling point on the pixel plane is transformed from the camera coordinate system to the calibration coordinate system to obtain the target sampling position of the third sampling point. b P f .

[0128] In some implementations, z can be adjusted based on z0, taking into account the depth-of-field range of the camera device. c The value can be randomly sampled within the pixel plane and within a height range based on z0. For example, if the pixel plane is divided into an 8x8 grid, a sampling point can take the pixel value of any grid point, and the zc value can take any value within the range (z0-d0, z0+d1), thus obtaining the position coordinates of more other third sampling points. The sampling method of this application is not limited to this.

[0129] If the eye is outside the hand system, and the calibration coordinate system is the base coordinate system, the above can be used to obtain the target sampling position of the third sampling point in the base coordinate system. b P f .

[0130] If it is an eye-on-hand system, the calibration coordinate system is the end coordinate system, and the above can be used to obtain the target sampling position of the third sampling point in the end coordinate system.

[0131] In some implementations, the attitude of the camera device in the calibration coordinate system is determined as the attitude of the marker in the calibration coordinate system, thus obtaining the target sampling attitude of the third sampling point.

[0132] If it is an eye-to-hand system, the calibration coordinate system is the base coordinate system, and the orientation of the camera device in the base coordinate system is determined. b R c The orientation of the marker in the base coordinate system is determined. b R m The target sampling pose of the third sampling point is obtained as follows: b R c .

[0133] If it is an eye-on-hand system, the calibration coordinate system is the end-effector coordinate system, and the camera device's orientation in the end-effector coordinate system is determined. f R c Determine the attitude of the marker in the end coordinate system. f R m The target sampling pose of the third sampling point is obtained as follows: f R c .

[0134] The target sampling pose of the third sampling point can be obtained through the above method. The target sampling pose includes the target sampling position and the target sampling posture.

[0135] S25: Use the target sampling pose of the third sampling point to obtain the third sampling data, and use the third sampling data to obtain the hand-eye calibration relationship between the camera coordinate system and the calibration coordinate system.

[0136] The pose of the robot's end effector is adjusted to the target sampling pose of multiple third sampling points, where the target sampling pose includes the target sampling position and the target sampling orientation. b R c or f R c ).

[0137] In some implementations, the robot's end effector is controlled to move to the target sampling position of the third sampling point. b P f The location of the target, and in the target sampling pose. b R c Construct a second rotating coordinate system and control the robot's end effector to rotate around the second rotating coordinate system. b R c The coordinate axes (X-axis, Y-axis, Z-axis) are rotated sequentially by a second preset angle. The second preset angle for each coordinate axis can be different, and the second preset angle can be an angle within the field of view of the camera device. This allows several rotations centered on the second rotating coordinate system to be applied to the third sampling point.

[0138] In other embodiments, the robot's end effector can be controlled to perform a preset movement, and then the robot's end effector can be rotated around a second rotational coordinate system. b R c The coordinate axes (X-axis, Y-axis, Z-axis) are rotated so that the end effector of the robotic arm moves to the target sampling position of the third sampling point. b P f The location of [the object / component] is not restricted in this application.

[0139] During the aforementioned movement and rotation of the robot's end effector, for example, at the nth (n is a positive integer) sampling point, the pose of the end effector (the third sampling point) is obtained. b T f ) n That is, the pose of the end part in the base coordinate system; and the image of the marker is obtained by taking a picture of the marker using a camera device, and the image sampling data of the marker (pixel coordinates (u, v) in the marker image can be identified from the marker image. # nThe camera sampling position of the marker in the camera coordinate system is obtained based on the corresponding image sampling data. c P m ) n This means that the third sample data pair corresponding to each third sample point can be obtained. The third sample data pair of the third sample point includes the pose of the third sample point ( b T f ) n and camera sampling position ( c P m ) n .

[0140] In the eye-on-hand system, the specific process for acquiring the third sampling data pair can refer to the process described above, and will not be repeated here. For example, after determining the pose of the camera device in the end-effector coordinate system as the pose of the marker in the end-effector coordinate system, when controlling the robot to move to the third sampling point, the camera sampling position of the marker in the camera coordinate system is obtained. c P m and the pose of the third sampling point f T b That is, the pose of the base in the end coordinate system.

[0141] Substitute multiple third-sampled data pairs into the preset optimization equation and solve for the optimization equation. For an eye-in-hand external system, with the calibration coordinate system being the base coordinate system, the expression of the preset optimization equation is as follows:

[0142] b T f f P m = b T c c P m (12)

[0143] The above formula, preset optimization equation (12), also includes translation relationship and calibration reference position. The initial value of the translation relationship is the translation relationship between the camera coordinate system and the base coordinate system. b T c The initial value of the calibration reference position is the calibration end position of the marker in the end coordinate system. f P m .

[0144] If it's an eye-on-hand system, with the coordinate system set as the end coordinate system, the default optimized equation expression is as follows:

[0145] f T b b P m = fT c c P m (13)

[0146] The above formula, preset optimization equation (12), also includes translation relationship and calibration reference position. The initial value of the translation relationship is the translation relationship between the camera coordinate system and the end coordinate system. f T c The initial value of the calibration reference position is the position of the calibration base of the marker in the base coordinate system. b P m . f T b , c P m For the third sampled data pair, where, f T b This indicates the pose of the third sampling point. c P m This indicates the camera sampling position of the marker in the camera coordinate system.

[0147] Substitute multiple third-sampled data pairs into the aforementioned preset optimization equation, optimize and solve the preset optimization equation, and obtain the final translation relationship between the optimized camera coordinate system and the robot's calibration coordinate system and the final calibration reference position of the marker. The final translation relationship is then used as the hand-eye calibration relationship of the camera coordinate system relative to the calibration coordinate system.

[0148] In this embodiment, by substituting the acquired third sampling data into the preset optimization equation for optimization, the optimized translation relationship between the camera coordinate system and the robot's calibration coordinate system is obtained and used as the hand-eye calibration relationship, which can improve the accuracy of hand-eye calibration between the camera coordinate system and the robot's calibration coordinate system.

[0149] Please see Figure 8 , Figure 8 This is a flowchart illustrating the third embodiment of the hand-eye calibration method of this application. After obtaining the hand-eye calibration relationship between the camera coordinate system and the calibration coordinate system in step S13 or step S25, the hand-eye calibration relationship can be sampled and verified. The method may include the following steps:

[0150] S31: Using the hand-eye calibration relationship, adjust the robot's end effector to move to multiple third sampling points again, and obtain the sampling verification data corresponding to the multiple third sampling points.

[0151] In some implementations, using the final hand-eye calibration relationship and calibration reference position obtained above, and referring to steps 24 to S25 in the above embodiments, the robot's end effector is adjusted to move again to the corresponding multiple third sampling points, and sampling verification data corresponding to the multiple third sampling points is obtained. The sampling verification data includes the pose of the third sampling point when it moves again to the Nth (N is a positive integer) third sampling point. b T f ) N and camera sampling position ( c P m ) N The process of obtaining the sample verification data can refer to the process of obtaining the third sample data pair described above, and will not be repeated here.

[0152] In other embodiments, a camera device can be controlled to capture an image of the marker, thereby identifying the first pixel coordinates (u, v) of the marker from the image. * Furthermore, in step S25 above, it is known that the third sampled data of the marker is identified based on the marker image, that is, the second pixel coordinates (u, v) in the marker image. # The first pixel coordinates and the second pixel coordinates can be used as sampling verification data.

[0153] S32: Using the sampling verification data corresponding to each third sampling point and the hand-eye calibration relationship, obtain the calibration error of each third sampling point.

[0154] In some implementations, the hand-eye calibration relationship, calibration reference position, and sampling verification data corresponding to each third sampling point obtained above can be substituted into the above preset optimization equation (12) or preset optimization equation (13), and the calibration error of each third sampling point can be obtained based on the difference between the left and right sides of the preset optimization equation.

[0155] In other implementations, the difference in the sampled verification data at each third sampling point can be used to obtain the first pixel coordinates (u, v). * and the second pixel coordinates (u, v) # The difference (u, v) * -(u, v) # This serves as the calibration error for each third sampling point.

[0156] S33: Based on the calibration error of each third sampling point, determine whether to perform preset processing on the hand-eye calibration relationship.

[0157] Based on the calibration error of each third sampling point, the average calibration error of all third sampling points can be obtained. This average value can measure the overall calibration accuracy and verify the calibration effect of hand-eye calibration.

[0158] In some implementations, if the mean value of the calibration error is less than a preset calibration threshold, then there is no need to perform preset processing on the hand-eye calibration relationship. If the mean value of the calibration error is not less than the preset calibration threshold, then preset processing on the hand-eye calibration relationship is performed.

[0159] In some implementations, the preset processing includes removing third sampling points whose calibration error is greater than a preset error threshold, and re-acquiring the third sampling data corresponding to the remaining third sampling points to optimize the hand-eye calibration relationship.

[0160] In some implementations, the preset processing includes re-executing steps S21 to S25, or re-executing steps S24 to S25, selecting a new third sampling point, obtaining the third sampling data corresponding to the new third sampling point, and optimizing the hand-eye calibration relationship in step S25 to obtain a more accurate hand-eye calibration relationship.

[0161] Regarding the above embodiments, this application provides a hand-eye calibration system; please refer to [link to relevant documentation]. Figures 9-10 , Figure 9 This is a schematic diagram of an embodiment of the hand-eye calibration system provided in this application. Figure 10 This is a schematic diagram of another embodiment of the hand-eye calibration system provided in this application. The hand-eye calibration system 40 includes a robot 41 and a camera device 42. The robot 41 includes an end effector 411 and a base 412, wherein the end effector 411 may be provided with markers (not shown). The robot 41 and the camera device 42 may or may not be coupled; this application does not impose any restrictions on this. The camera device 42 may be the vision system of the robot 41, that is, the camera device 42 may exist within the robot 41. The combination of the camera device 42 and the robot 41 can be used to perform any step in any embodiment of the above-described hand-eye calibration method to determine the hand-eye calibration relationship between the coordinate system corresponding to the camera device and the coordinate system corresponding to the calibration part of the robot.

[0162] Please see Figure 9 When performing hand-eye calibration, if it is an eye-on-hand system, the camera device 42 can be set on the end part 411 of the robot 41, and the calibration part is the end part 411.

[0163] Please see Figure 10 If it is an eye-in-hand system, the camera device 42 is located outside the end part 411 of the robot 41, and the calibration part is the base part 412.

[0164] In some implementations, please refer to Figures 9-10The aforementioned hand-eye calibration system 40 may further include a processor 43, which can be connected to the robotic arm of the robot 41, that is, a processor 43 that can be connected to the robot 41. Specifically, the sensors and controllers of the robotic arm can be connected to the processor 43.

[0165] The processor 43 can control the operation of the robot 41. In addition, the processor 43 can also be connected to the camera device 42, thereby controlling the camera device 42 to take pictures.

[0166] Processor 43 can also be referred to as CPU (Central Processing Unit). Processor 43 may be an integrated circuit chip with signal processing capabilities. Processor 43 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), off-the-shelf programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. A general-purpose processor can be a microprocessor or any conventional processor, etc.

[0167] The processor 43 is used to execute instructions to implement the method provided by any embodiment of the hand-eye calibration method described above in this application or a combination thereof.

[0168] Optionally, the robot 41 may further include a memory 44 coupled to the processor 43 for storing instructions executed by the processor 43 and data required to execute the instructions. The processor 43 and the memory 44 may exist in other terminals (not shown) preceding the robot 41 to control the operation of the robot 41 and / or the camera device 42 through other terminals.

[0169] In addition, the robot 41 can also include display devices (not shown), input / output devices and other equipment according to actual needs.

[0170] In other embodiments, the hand-eye calibration system 40 described above may also include separate processors 43 for the robot 41 and the camera device 42, that is, it may include multiple processors 43, so that the processor 43 connected to the robot 41 can control the operation of the robot 41. The processor 43 connected to the camera device 42 can control the operation of the camera device 42. The specific connection method can be set according to the application scenario, and this application is not limited to this.

[0171] In other embodiments, in the hand-eye calibration system 40 described above, the processor 43 may reside within the robot 41, which is coupled to the camera device 42. The processor 43 can execute instructions to control the operation of the robot 41 and the camera device 42. Additionally, a memory 44 coupled to the processor 43 is also located in the robot 41 to store the instructions executed by the processor 43 and the data required for executing those instructions. This application does not impose any limitations on the configuration of the processor 43 and the memory 44.

[0172] The specific implementation of this embodiment can be referred to the implementation process of the above embodiments, and will not be repeated here.

[0173] Regarding the above embodiments, this application provides a computer device; please refer to [link / reference]. Figure 11 , Figure 11 This is a schematic diagram of the structure of a computer device according to an embodiment of the present application. The computer device 50 includes a memory 51 and a processor 52, wherein the memory 51 and the processor 52 are coupled to each other. The memory 51 stores program data, and the processor 52 is used to execute the program data to implement the steps in any embodiment of the hand-eye calibration method described above.

[0174] In this embodiment, processor 52 can also be referred to as CPU (Central Processing Unit). Processor 52 may be an integrated circuit chip with signal processing capabilities. Processor 52 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. The general-purpose processor can be a microprocessor, or processor 52 can be any conventional processor.

[0175] The specific implementation of this embodiment can be referred to the implementation process of the above embodiments, and will not be repeated here.

[0176] The methods described in the above embodiments can be implemented as computer programs; therefore, this application proposes a storage device. Please refer to [link to relevant documentation]. Figure 12 , Figure 12 This is a schematic diagram of a storage device according to an embodiment of the present application. The storage device 60 stores program data 61 that can be executed by a processor. The program data 61 can be executed by the processor to implement the steps of any embodiment of the hand-eye calibration method described above.

[0177] The specific implementation of this embodiment can be referred to the implementation process of the above embodiments, and will not be repeated here.

[0178] In this embodiment, the storage device 60 can be a medium that can store program data 61, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk. Alternatively, it can be a server that stores the program data 61. The server can send the stored program data 61 to other devices for execution, or it can run the stored program data 61 itself.

[0179] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0180] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0181] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0182] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage device, which is a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application.

[0183] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device, or fabricating them separately as individual integrated circuit modules, or fabricating multiple modules or steps as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0184] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A hand-eye calibration method, characterized in that, The method includes: Markers are set on the robot, first sampling data is obtained, and the rotation relationship matrix of the robot's camera coordinate system relative to the calibration coordinate system is obtained using the first sampling data. The camera coordinate system corresponds to the camera device, and the calibration coordinate system corresponds to the calibration part of the robot, which is the base part or the end part. Using the rotation relationship matrix, the end effector of the robot is controlled to move around the coordinate axes of the camera coordinate system to acquire second sampling data; Using the rotation relationship matrix and the second sampling data, the translation relationship between the camera coordinate system and the robot's calibration coordinate system is obtained, wherein the rotation relationship matrix and the translation relationship are used to perform hand-eye calibration of the camera coordinate system and the calibration coordinate system; Based on the translation relationship, the target sampling pose of the third sampling point is determined; The process involves obtaining third sampling data using the target sampling pose of the third sampling point, and then using the third sampling data to obtain the hand-eye calibration relationship between the camera coordinate system and the calibration coordinate system, including: The pose of the robot's end effector is adjusted to the target sampling pose of multiple third sampling points, and the camera device is used to take pictures of the marker to obtain image sampling data. Multiple third sampling data pairs of the third sampling points are obtained. The third sampling data pairs of the third sampling points include the pose of the third sampling point and the camera sampling position of the marker in the camera coordinate system obtained according to the corresponding image sampling data. Substitute the multiple third sampling data pairs into the preset optimization equation; The preset optimization equation is optimized and solved to obtain the final translation relationship between the camera coordinate system and the robot's calibration coordinate system and the final calibration reference position of the marker. The final translation relationship is then used as the hand-eye calibration relationship between the camera coordinate system and the calibration coordinate system.

2. The method according to claim 1, characterized in that, The step of acquiring the first sampled data and using the first sampled data to obtain the rotation relationship matrix of the robot's camera coordinate system relative to the calibration coordinate system includes: The initial pose of the robot's end effector is obtained, as well as the initial sampling data of the marker at the initial point in the camera coordinate system is obtained; The robot's end effector is controlled to move a preset distance along at least one coordinate axis in the calibrated coordinate system to obtain first sampling data of the marker at at least one first sampling point in the camera coordinate system. Based on the initial sampling data of the marker and the at least one first sampling data, the displacement vector of the marker corresponding to the first sampling point in the camera coordinate system is obtained; Using at least one of the displacement vectors, obtain the rotation relationship matrix of the robot's camera coordinate system relative to the calibration coordinate system.

3. The method according to claim 1, characterized in that, The step of controlling the robot's end effector to move around the coordinate axes of the camera coordinate system using the rotation relationship matrix to acquire second sampling data includes: Acquire the first attitude data of the end part in the base coordinate system at the first movement point; wherein, the first movement point includes the initial point; Based on the rotation relationship matrix, a first rotation coordinate system is constructed, and the end part of the robot is controlled to rotate around any coordinate axis of the first rotation coordinate system by a first preset angle to obtain the second posture data and the second position data of the end part in the base coordinate system at the second movement point. Using the rotation relationship matrix, control the end part to move to the first moving point, and obtain the first position data of the end part in the base coordinate system; Repeat the above steps at least three times to obtain at least three sets of the second sampling data. Each set of the second sampling data includes attitude data and position data. The attitude data includes the first attitude data and the second attitude data, and the position data includes the first position data and the second position data.

4. The method according to claim 3, characterized in that, The step of obtaining the translation relationship between the camera coordinate system and the robot's calibration coordinate system using the rotation relationship matrix and the second sampling data includes: Based on the second sampling data, the calibrated end position of the marker in the end coordinate system corresponding to the end part of the robot is obtained; Using the calibration end position and the pose of the end portion in the base coordinate system, the calibration base position of the marker in the base coordinate system is obtained; Using the rotation relationship matrix, the calibration base position, and the calibration camera position, the translation relationship between the camera coordinate system and the calibration coordinate system is obtained; wherein, the calibration camera position represents the position of the marker in the camera coordinate system.

5. The method according to claim 4, characterized in that, The step of obtaining the calibrated end position of the marker in the end-effector coordinate system corresponding to the end-effector of the robot based on the second sampling data includes: The calibration end position is obtained by using the attitude difference between at least three sets of attitude data and the position difference between the position data in at least three sets of second adopted data.

6. The method according to claim 4, characterized in that, The step of obtaining the translation relationship between the camera coordinate system and the calibration coordinate system using the rotation relationship matrix, the calibration base position, and the calibration camera position includes: When the camera device is located outside the end of the robot, the translation relationship between the camera coordinate system and the robot's calibration coordinate system is obtained by subtracting the product of the calibration camera position, the rotation relationship matrix, and the calibration base position. When the camera device is located on the end part of the robot, the translation relationship between the camera coordinate system and the robot's calibration coordinate system is obtained by using the calibrated camera position, the rotation relationship matrix, the calibrated base position, the mutual attitude transformation matrix between the end part and the base coordinate system, and the attitude of the end part in the base coordinate system.

7. The method according to claim 1, characterized in that, Determining the target sampling pose of the third sampling point based on the translation relationship includes: Using the translation relationship between the camera coordinate system and the calibration coordinate system, and the configuration parameters of the camera device, the preset sampling point on the pixel plane is transformed from the camera coordinate system to the calibration coordinate system to obtain the target sampling position of the third sampling point; The orientation of the camera device in the calibration coordinate system is set to the orientation of the marker in the calibration coordinate system to obtain the target sampling orientation of the third sampling point.

8. The method according to claim 1, characterized in that, The preset optimization equation includes a translation relationship and a calibration reference position; the initial value of the translation relationship is the translation relationship between the camera coordinate system and the calibration coordinate system; when the calibration part is a base part, the initial value of the calibration reference position is the calibration end position of the marker in the end coordinate system; when the calibration part is an end part, the initial value of the calibration reference position is the calibration base position of the marker in the base coordinate system.

9. The method according to claim 1, characterized in that, After obtaining the hand-eye calibration relationship between the camera coordinate system and the calibration coordinate system using the third sampling data, the method further includes: Using the hand-eye calibration relationship, the end effector of the robot is adjusted to move to multiple third sampling points again, and sampling verification data corresponding to multiple third sampling points are obtained; The calibration error of each third sampling point is obtained by using the sampling verification data corresponding to each third sampling point and the hand-eye calibration relationship. Based on the calibration error of each of the third sampling points, it is determined whether to perform preset processing on the hand-eye calibration relationship.

10. A hand-eye calibration system, characterized in that, The system includes: A robot and camera device, the robot including an end effector and a base; The camera device is disposed on the end portion of the robot; or, the camera device is disposed outside the end portion of the robot. The combination of the camera device and the robot is used to perform the steps of the method according to any one of claims 1 to 9, to determine the hand-eye calibration relationship between the calibration points of the camera device and the robot.

11. A computer device, characterized in that, The method includes a memory and a processor coupled to each other, the memory storing program data and the processor executing the program data to implement the steps of the method according to any one of claims 1 to 9.

12. A storage device, characterized in that, The system stores program data that can be executed by a processor, the program data being used to implement the steps of the method according to any one of claims 1 to 9.

Images