Hand-eye calibration method and device and mechanical arm system

Abstract

The invention provides a hand-eye calibration method and device and a mechanical arm system.The method comprises the steps that first coordinates of a calibration object of a mechanical arm in various configurations are determined based on mechanical arm kinematics data, and second coordinates of the calibration object of the mechanical arm in various configurations are determined based on collected point cloud data; and performing hand-eye calibration based on the first coordinate and the second coordinate to obtain a hand-eye calibration result. In the embodiment of the invention, the hand-eye calibration of the robot can be realized only based on the depth information, a 2D camera does not need to be deployed, and the hardware cost is reduced. Moreover, compared with a 2D hand-eye calibration scheme in the related technology, the method is not affected by distortion of a camera lens, and the calibration precision and effect are better.

Description

Technical Field

[0001] The present disclosure relates to the field of robot control technology, and in particular to a hand-eye calibration method and device, and a robotic arm system. Background Art

[0002] In the system combining robots and computer vision (CV), hand-eye calibration is very important. The hand-eye calibration of the robot refers to determining the homogeneous transformation relationship between the camera coordinate system and the robot coordinate system. Summary of the Invention

[0003] To achieve depth-based robot hand-eye calibration and improve calibration accuracy, the present disclosure provides a hand-eye calibration method and device, a robotic arm system, a storage medium, and a computer program product.

[0004] In a first aspect, an embodiment of the present disclosure provides a hand-eye calibration method, comprising:

[0005] determining, based on the kinematic data of the manipulator, a first coordinate of a calibration object relative to a base coordinate system of the manipulator in a plurality of configurations of the manipulator, wherein the calibration object is fixed to the manipulator;

[0006] Collecting point cloud data of the robotic arm in each configuration, and determining a second coordinate of the calibration object relative to a camera coordinate system based on the point cloud data;

[0007] Hand-eye calibration is performed based on the first coordinates and the second coordinates of the calibration object in the multiple configurations to obtain a hand-eye calibration result.

[0008] In some embodiments, collecting point cloud data of the robotic arm in each configuration includes:

[0009] The point cloud data of the robotic arm in each configuration is collected by a depth camera, wherein the calibration object is located within the field of view of the depth camera.

[0010] In some embodiments, the calibration object includes a calibration sphere; and determining, based on the kinematic data of the manipulator, a first coordinate of the calibration object relative to a base coordinate system of the manipulator in a plurality of configurations of the manipulator includes:

[0011] Pre-establishing the spherical center coordinate system of the calibration sphere;

[0012] Based on the kinematic data of the robotic arm and the mapping relationship between the spherical center coordinate system and the base coordinate system, the first coordinate of the center of the calibration sphere of the robotic arm in each configuration is determined.

[0013] In some embodiments, the calibration object includes a calibration sphere; and determining the second coordinate of the calibration object relative to the camera coordinate system based on the point cloud data includes:

[0014] Randomly sampling a preset number of points from the point cloud data;

[0015] Performing spherical fitting based on the three-dimensional coordinates of the preset number of points to obtain a candidate spherical equation;

[0016] In response to the number of points in the point cloud data that satisfy the candidate spherical equation being greater than or equal to a preset threshold, the second coordinate of the center of the calibration sphere is determined based on the candidate spherical equation.

[0017] In some embodiments, the hand-eye calibration method of the present disclosure further includes:

[0018] In response to the number of points in the point cloud data that satisfy the candidate spherical equation being less than the preset threshold, a preset number of points are randomly sampled from the point cloud data again, and a spherical fitting is performed based on the three-dimensional coordinates of the preset number of points to obtain the candidate spherical equation until the convergence condition is met.

[0019] In some embodiments, randomly sampling from the point cloud data to obtain a preset number of points includes:

[0020] Performing image segmentation on the calibration sphere based on the point cloud data to obtain target point cloud data corresponding to the calibration sphere;

[0021] A preset number of points are obtained by random sampling from the target point cloud data.

[0022] In some embodiments, the hand-eye calibration result includes a homogeneous transformation relationship between the first coordinate and the second coordinate; and performing hand-eye calibration based on the first coordinate and the second coordinate of the calibration object in the multiple configurations to obtain the hand-eye calibration result includes:

[0023] constructing an error equation between the first coordinate and the second coordinate based on a least squares method;

[0024] Determining optimal rotation matrix parameters and translation matrix parameters in the error equation based on singular value decomposition;

[0025] A homogeneous transformation relationship between the first coordinate and the second coordinate is determined based on the rotation matrix parameters and the translation matrix parameters.

[0026] In a second aspect, an embodiment of the present disclosure provides a hand-eye calibration device, comprising:

[0027] a first coordinate determination module configured to determine, based on the kinematic data of the manipulator, first coordinates of a calibration object relative to a base coordinate system of the manipulator in a plurality of configurations of the manipulator, wherein the calibration object is fixed to the manipulator;

[0028] a second coordinate determination module, configured to collect point cloud data of the robotic arm in each configuration, and determine a second coordinate of the calibration object relative to the camera coordinate system based on the point cloud data;

[0029] The hand-eye calibration module is configured to perform hand-eye calibration based on the first coordinates and the second coordinates of the calibration object in the multiple configurations to obtain a hand-eye calibration result.

[0030] In some embodiments, the second coordinate determination module is configured to:

[0031] The point cloud data of the robotic arm in each configuration is collected by a depth camera, wherein the calibration object is located within the field of view of the depth camera.

[0032] In some embodiments, the calibration object includes a calibration sphere; and the first coordinate determination module is configured to:

[0033] Pre-establishing the spherical center coordinate system of the calibration sphere;

[0034] Based on the kinematic data of the robotic arm and the mapping relationship between the spherical center coordinate system and the base coordinate system, the first coordinate of the center of the calibration sphere of the robotic arm in each configuration is determined.

[0035] In some embodiments, the calibration object includes a calibration sphere; and the second coordinate determination module is configured to:

[0036] Randomly sampling a preset number of points from the point cloud data;

[0037] Performing spherical fitting based on the three-dimensional coordinates of the preset number of points to obtain a candidate spherical equation;

[0038] In response to the number of points in the point cloud data that satisfy the candidate spherical equation being greater than or equal to a preset threshold, the second coordinate of the center of the calibration sphere is determined based on the candidate spherical equation.

[0039] In some embodiments, the second coordinate determination module is configured to:

[0040] In response to the number of points in the point cloud data that satisfy the candidate spherical equation being less than the preset threshold, a preset number of points are randomly sampled from the point cloud data again, and a spherical fitting is performed based on the three-dimensional coordinates of the preset number of points to obtain the candidate spherical equation until the convergence condition is met.

[0041] In some embodiments, the second coordinate determination module is configured to:

[0042] Performing image segmentation on the calibration sphere based on the point cloud data to obtain target point cloud data corresponding to the calibration sphere;

[0043] A preset number of points are obtained by random sampling from the target point cloud data.

[0044] In some embodiments, the hand-eye calibration result includes a homogeneous transformation relationship between the first coordinate and the second coordinate; and the hand-eye calibration module is configured to:

[0045] constructing an error equation between the first coordinate and the second coordinate based on a least squares method;

[0046] Determining optimal rotation matrix parameters and translation matrix parameters in the error equation based on singular value decomposition;

[0047] A homogeneous transformation relationship between the first coordinate and the second coordinate is determined based on the rotation matrix parameters and the translation matrix parameters.

[0048] In a third aspect, an embodiment of the present disclosure provides a robotic arm system, comprising:

[0049] A robotic arm comprising a fixed end and a moving end, wherein the fixed end is fixed to a target object;

[0050] A calibration object is fixed to the moving end of the robotic arm;

[0051] A depth camera, wherein the calibration object is located within the field of view of the depth camera, and the depth camera is used to collect point cloud data of the calibration object;

[0052] The controller includes a processor and a memory, wherein the memory stores computer instructions, and the computer instructions are used to enable the processor to execute the method according to any embodiment of the first aspect.

[0053] In a fourth aspect, an embodiment of the present disclosure provides a storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the method according to any embodiment of the first aspect.

[0054] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, which implements the method described in any embodiment of the first aspect when running.

[0055] The hand-eye calibration method of the disclosed embodiment includes determining the first coordinates of a calibration object of the robotic arm in various configurations based on the robotic arm's kinematic data, determining the second coordinates of the calibration object of the robotic arm in various configurations based on the collected point cloud data, and performing hand-eye calibration based on the first coordinates and the second coordinates to obtain a hand-eye calibration result. In the disclosed embodiment, the robot's hand-eye calibration can be achieved based solely on depth information, eliminating the need for deploying a 2D camera and reducing hardware costs. Furthermore, compared to the 2D hand-eye calibration scheme in the related art, it is not affected by camera lens distortion, and has better calibration accuracy and effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0057] Figure 1 It is a schematic structural diagram of a robotic arm system according to some embodiments of the present disclosure.

[0058] Figure 2 It is a schematic structural diagram of a robotic arm system according to some embodiments of the present disclosure.

[0059] Figure 3 is a flowchart of a hand-eye calibration method according to some embodiments of the present disclosure.

[0060] Figure 4 is a flowchart of a hand-eye calibration method according to some embodiments of the present disclosure.

[0061] Figure 5 is a flowchart of a hand-eye calibration method according to some embodiments of the present disclosure.

[0062] Figure 6 is a schematic diagram of a hand-eye calibration method according to some embodiments of the present disclosure.

[0063] Figure 7 is a flowchart of a hand-eye calibration method according to some embodiments of the present disclosure.

[0064] Figure 8 is a flowchart of a hand-eye calibration method according to some embodiments of the present disclosure.

[0065] Figure 9 is a schematic diagram of a hand-eye calibration method according to some embodiments of the present disclosure.

[0066] Figure 10 It is a structural block diagram of a hand-eye calibration device according to some embodiments of the present disclosure.

[0067] Figure 11 It is a structural block diagram of a robotic arm system according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0068] The technical solutions of the present disclosure will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure. In addition, the technical features involved in the different embodiments of the present disclosure described below can be combined with each other as long as they do not conflict with each other.

[0069] Related technologies typically use a 2D camera to capture a calibration plate with known pattern dimensions. By identifying features in the image, the correspondence between the camera coordinate system and the robot coordinate system is established to achieve hand-eye calibration. However, in some scenarios, the robot's vision system lacks a 2D camera, making this approach unsuitable for hand-eye calibration.

[0070] For example, in some scenarios, a robotic arm system includes a depth camera instead of a 2D camera. The main difference between a depth camera and a regular 2D camera is that, in addition to collecting 2D plane information, a depth camera can also collect depth information of an object. Depth information represents the distance between a point in space and the camera.

[0071] The data collected by the depth camera includes point cloud data, and each point in the point cloud data includes (x, y, z) three-dimensional coordinates. Due to the difference in the working principle of depth cameras and ordinary 2D cameras, the above 2D hand-eye calibration solution cannot be used for hand-eye calibration.

[0072] Based on the above, the embodiments of the present disclosure provide a hand-eye calibration method, device, robotic arm system, storage medium and computer program product, which aim to achieve hand-eye calibration based on depth information and improve calibration accuracy compared to the 2D hand-eye calibration scheme in the related art.

[0073] Figure 1 The schematic diagram of the structure of the robot arm system in some embodiments of the present disclosure is shown below. Figure 1 The application scenarios of the method according to the embodiment of the present disclosure are described.

[0074] See also Figure 1As shown, in the example of the present disclosure, the robotic arm system includes a camera 101 and a robotic arm 102. Camera 101 includes at least a depth camera, which can be, for example, a binocular stereo camera, a structured light camera, a ToF (Time of Flight) camera, etc., and the present disclosure does not limit this. In the embodiment of the present disclosure, camera 101 can include only a depth camera, or it can include both a depth camera and a 2D camera. The 2D camera can be, for example, an RGB camera.

[0075] The robot arm 102 can be a multi-axis robot arm, which includes a fixed end and a moving end. The fixed end can generally be fixed on a platform, and the moving end refers to the working end of the robot arm. By controlling the movement of each axis of the robot arm 102, the moving end can be moved to various spatial positions to realize spatial operations.

[0076] In the disclosed embodiment, the robotic arm 102 and the camera 101 are arranged separately from each other, that is, "eye-to-hand", and the robotic arm 102 is located within the field of view of the camera 101, so that when the robotic arm 102 moves, the camera 101 can always collect depth data of the robotic arm 102.

[0077] In this scenario, the relationship between the camera coordinate system {C} of the camera 101 and the base coordinate system {B} of the robotic arm 102 is fixed. The goal of hand-eye calibration of the robotic arm system is to find the correspondence between the camera coordinate system {C} and the base coordinate system {B}.

[0078] In the embodiment scenario of the present disclosure, in order to achieve hand-eye calibration of the robotic arm system, a calibration object needs to be set at the moving end of the robotic arm. The calibration object can be a regular-shaped object of known size, and the calibration object can be fixed at the moving end of the robotic arm 102.

[0079] For example, see Figure 2 As shown in (a), the calibration object can be a calibration sphere 103 with a known radius r, see Figure 2 As shown in (b), the calibration ball 103 can be fixedly mounted on the moving end of the robotic arm 102 via the flange 104, so that when the robotic arm 102 moves, the calibration ball 103 moves accordingly.

[0080] Of course, those skilled in the art will appreciate that the calibration object is not limited to the calibration sphere 103, and any other suitable object may be used, which will not be described in detail in this disclosure. Based on the above-mentioned robotic arm system, the hand-eye calibration method of the embodiment of this disclosure is described below.

[0081] like Figure 3 As shown, in some embodiments, the hand-eye calibration method of the present disclosure includes:

[0082] S310 , based on the kinematic data of the robotic arm, determining the first coordinate of the calibration object relative to the robotic arm base coordinate system in multiple configurations of the robotic arm.

[0083] Combine Figure 1 As shown, the robotic arm 102 can change its configuration in a controlled manner. During the movement of the robotic arm 102, the robotic arm system will continuously collect various kinematic data during the movement, and solve the motion state of the robotic arm 102 in real time based on the kinematic model to achieve precise control of the robotic arm 102.

[0084] For example, in one example, the robotic arm system collects kinematic data such as the angle and angular velocity of each joint of the robotic arm 102 and performs closed-loop control of the robotic arm based on the kinematic data, thereby achieving motion control of any configuration of the robotic arm 102.

[0085] In the embodiments of the present disclosure, it is necessary to control the robotic arm 102 to transform into multiple configurations. For example, in one example, the robotic arm 102 can be controlled to transform into any n configurations, where n is a positive integer greater than 1. It will be appreciated that in the embodiments of the present disclosure, since the calibration object is fixed to the end of the robotic arm 102, the calibration object always moves with the robotic arm 102 when the robotic arm 102 transforms its configuration.

[0086] Combine Figure 1 As shown in the figure, during the robot's motion, the base coordinate system {B} remains fixed, while the tool coordinate system {t} of the calibration object at the end changes with the motion of the robot arm 102. When the robot arm 102 is in a certain configuration, the corresponding relationship between the tool coordinate system {t} at the end of the robot arm 102 and the base coordinate system {B} can be calculated in real time through the aforementioned kinematic solution process, thereby obtaining the first coordinate of the calibration object at the end of the robot arm 102 relative to the base coordinate system {B}.

[0087] For example, in one example, the object is calibrated with Figure 2 As an example, the calibration ball 103 shown in FIG. Figure 4 The embodiment describes a process of determining the first coordinates.

[0088] like Figure 4 As shown, in some embodiments, the hand-eye calibration method of the present disclosure example, the process of determining the first coordinate of the calibration ball 103 of the robotic arm in each configuration includes:

[0089] S311. Pre-establish the center coordinate system of the calibration sphere.

[0090] S312. Based on the kinematic data of the manipulator and the mapping relationship between the spherical center coordinate system and the base coordinate system, determine the first coordinate of the center of the calibration ball of the manipulator in each configuration.

[0091] In this disclosure example, Figure 1 As shown, the spherical center coordinate system of the calibration sphere 103 can be pre-established, and the spherical center coordinate system is Figure 1 The coordinate system {t} shown in FIG. 1 is a spherical center coordinate system. The origin of the coordinate system {t} is always located at the center of the calibration sphere 103 , and the direction of the initial coordinate axis is the same as that of the base coordinate system {B}.

[0092] When the manipulator 102 moves to a certain configuration, the current calibration ball center coordinates O can be calculated in real time based on the above-mentioned method process by collecting the kinematic data of the manipulator and combining the mapping relationship between the spherical center coordinate system {t} and the base coordinate system {B}. base , the coordinates of the sphere center O base That is, the coordinate of the center O of the calibration ball relative to the base coordinate system, which is also the first coordinate mentioned in this disclosure.

[0093] The above description only uses one configuration of the robot arm 102 as an example. For n configurations of the robot arm 102, the above method can be repeated to obtain the first coordinate of the center of the calibration sphere 103 relative to the base coordinate system {B} in each configuration, which can be expressed as: Among them, P base represents the first coordinate set, Represents the first coordinate of the sphere center corresponding to the i-th configuration.

[0094] The above method disclosed in the present invention explains the process of determining the first coordinate of the center of the calibration ball relative to the base coordinate system. As for the parts of the kinematic calculation process not described in detail, those skilled in the art can undoubtedly understand and fully implement them in combination with relevant technologies, and the present invention will not elaborate on them.

[0095] S320: Collect point cloud data of the robotic arm in each configuration, and determine the second coordinate of the calibration object relative to the camera coordinate system based on the point cloud data.

[0096] Combine Figure 1 、 Figure 2 It can be seen from the scene that during the movement of the robotic arm 102 , the end of the robotic arm 102 is located within the field of view of the camera 101 , so that the camera 101 can collect depth information of the robotic arm 102 and the calibration ball 103 .

[0097] In the embodiments of the present disclosure, the camera 101 must at least include a depth camera. The present disclosure does not limit the type of depth camera, which can be, for example, a ToF, structured light, or binocular camera. The data collected by the depth camera 101 is a depth image containing pixel depth. The depth image is essentially point cloud data. Each point included in the point cloud data has three-dimensional coordinates (x, y, z). The three-dimensional coordinates of all points constitute the point cloud data.

[0098] When the robotic arm 102 moves to a certain configuration, the depth camera 101 can be used to collect point cloud data, and the second coordinate of the calibration object relative to the camera coordinate system {C} can be determined based on the point cloud data.

[0099] For example, in one example, the calibration object is still Figure 2 As an example, the calibration ball 103 shown in FIG. Figure 5 The embodiment describes the process of determining the second coordinates.

[0100] like Figure 5 As shown, in some embodiments, the hand-eye calibration method of the present disclosure example, the process of determining the second coordinate of the calibration ball 103 of the robotic arm in each configuration includes:

[0101] S321. Randomly sample from the point cloud data to obtain a preset number of points.

[0102] S322 : Perform spherical fitting based on the three-dimensional coordinates of a preset number of points to obtain a candidate spherical equation.

[0103] S323 : In response to the number of points in the point cloud data that satisfy the candidate spherical equation being greater than or equal to a preset threshold, determine the second coordinates of the center of the calibration sphere based on the candidate spherical equation.

[0104] In some embodiments of the present disclosure, when the robotic arm 102 moves to a certain configuration, point cloud data P can be acquired by the depth camera 101. The point cloud data P represents a collection of a series of three-dimensional points, where each point includes three-dimensional coordinates (x, y, z).

[0105] In the embodiment of the present disclosure, the second coordinate of the calibration object is the three-dimensional coordinate (x o ,y o ,z o ), and the mathematical model of the known sphere can be expressed using the coordinates of the sphere center (x o ,y o ,z o ) and radius r, so in some embodiments of the present disclosure, the second coordinate O of the center of the calibration sphere 103 in the camera coordinate system can be determined by fitting the spherical equation camera (x o ,y o ,z o ).

[0106] First, a preset number of points P' can be randomly sampled from the point cloud data P, where P'∈P. For example, in an example, the point cloud data P includes 100,000 points in total, and 10,000 points can be randomly sampled from the point cloud data P to form a set P'.

[0107] Then, a spherical surface fitting is performed based on the three-dimensional coordinates of the points in P' to obtain the spherical surface equation f(x, y, z). It is understood that since the points in P' are randomly sampled, the spherical surface equation obtained by fitting may not necessarily converge to match the sphere of the calibration sphere 103. Therefore, the spherical surface equation f(x, y, z) obtained by fitting needs to be further verified. Therefore, in the embodiment of the present disclosure, the spherical surface equation f(x, y, z) obtained by fitting is defined as a candidate spherical surface equation.

[0108] Afterwards, all points in the point cloud data P can be substituted into the candidate spherical equation f(x, y, z) for verification, and the number of points that satisfy the candidate spherical equation f(x, y, z) can be counted.

[0109] It can be understood that the more points that satisfy the candidate spherical equation f(x, y, z), the better the spherical fitting effect is, that is, the closer the candidate spherical equation obtained by fitting is to the spherical surface of the calibration sphere 103; conversely, the fewer points that satisfy the candidate spherical equation f(x, y, z), the worse the spherical fitting effect is, that is, the greater the difference between the candidate spherical equation obtained by fitting and the spherical surface of the calibration sphere 103.

[0110] Based on this, in some embodiments of the present disclosure, a preset threshold can be pre-set. The preset threshold represents the critical value for convergence of the candidate spherical equation. The specific value of the preset threshold can be selected based on the scenario requirements and is not limited by the present disclosure. For example, in one example, the preset threshold can be set to 10,000.

[0111] In one example, if the number of points in the point cloud data that satisfy the candidate spherical equation f(x, y, z) is greater than or equal to a preset threshold, it means that the spherical fitting effect is good at this time, and the candidate spherical equation f(x, y, z) can be determined as the target spherical equation f(x, y, z) of the calibration sphere 103.

[0112] In another example, if the number of points in the point cloud data that satisfy the candidate spherical equation f(x, y, z) is less than a preset threshold, it indicates that the spherical fitting effect is very poor. Then, the aforementioned method process S321 to S323 can be executed again, that is, a preset number of points are randomly sampled from the point cloud data again, and the candidate spherical equation is refitted based on the three-dimensional coordinates of these points. Then, all points included in the point cloud data are substituted into the candidate spherical equation, and the number of points that satisfy the candidate spherical equation is again compared with the preset threshold.

[0113] This cycle repeats until a convergence condition is met, and the final candidate spherical equation is determined as the target spherical equation. In some embodiments, the convergence condition can be, for example, "until the number of points that satisfy the candidate spherical equation is greater than or equal to a preset threshold," or "until the number of fits meets a preset upper limit."

[0114] After determining the target spherical equation f(x, y, z), the coordinates of the sphere center relative to the camera coordinate system can be determined based on the target spherical equation f(x, y, z) camera (x o ,y o ,z o ), which is the second coordinate O camera (x o ,y o ,z o ).

[0115] The above description only uses one configuration of the robot arm 102 as an example. For n configurations of the robot arm 102, the above method can be repeated to obtain the second coordinate of the center of the calibration sphere 103 relative to the camera coordinate system {C} in each configuration, which can be expressed as: Among them, P camera represents the second coordinate set, Represents the second coordinate of the sphere center corresponding to the i-th configuration.

[0116] S330: Perform hand-eye calibration based on the first coordinates and the second coordinates of the calibration object in multiple configurations to obtain a hand-eye calibration result.

[0117] In the embodiment of the present disclosure, through the above method steps, the first coordinate O of the center of the calibration ball in each configuration can be obtained. base and the second coordinate O camera , expressed as:

[0118]

[0119] It is understandable that the first coordinate It represents the coordinates of the center of the calibration ball 103 relative to the base coordinate system {B} in the i-th configuration. The second coordinate It represents the coordinates of the center of the calibration sphere 103 relative to the camera coordinate system {C} in the i-th configuration. Since the coordinates of the center of the sphere in the camera coordinate system and the coordinates of the center of the sphere in the base coordinate system actually represent the same spatial point, the above P base The first coordinate of the point set With P camera The second coordinate of the point set There is a one-to-one correspondence. For example, see Figure 6 As shown in (a), the point set P base With point set P camera The points in the correspondence are one to one, see Figure 6 As shown in (b), the points in one point set coincide with the points in the other point set after spatial transformation.

[0120] The goal of hand-eye calibration for the robotic arm system is to find the Euclidean space transformation R∈SO(3), t∈R 2 , so that for all o base =R·o camera +t, where R represents the rotation matrix, t represents the translation matrix, and SO(3) is a special orthogonal group. In other words, the goal of hand-eye calibration is to calculate the rotation matrix R and the translation matrix t, thereby constructing the following homogeneous transformation relationship:

[0121] o base =R·o camera +t (2)

[0122] The homogeneous transformation relationship shown in formula (2) is the hand-eye calibration result described in this disclosure.

[0123] In some embodiments, considering that solving the above homogeneous transformation relationship is a 3D point set to 3D point set registration problem, that is, the iterative closest point (ICP) problem, some embodiments of the present disclosure propose a solution based on singular value decomposition (SVD). Figure 7 Provide explanation.

[0124] like Figure 7 As shown, in some embodiments, the hand-eye calibration method of the present disclosure example, the process of determining the hand-eye calibration result includes:

[0125] S331. Construct an error equation between the first coordinate and the second coordinate based on the least squares method.

[0126] S332. Determine the optimal rotation matrix parameters and translation matrix parameters in the error equation based on singular value decomposition.

[0127] S333. Determine a homogeneous transformation relationship between the first coordinate and the second coordinate based on the rotation matrix parameters and the translation matrix parameters.

[0128] First, to solve the ICP problem, we can construct the least squares error equation, which is expressed as:

[0129]

[0130] That is, solving the ICP problem is to find an optimal set of rotation matrices R and translation matrices t so that J is minimized.

[0131] Secondly, we can calculate the point set P base The center of mass And the point set P camera The center of mass That is, the point set P base The average of the three-dimensional coordinates of the midpoint, That is, the point set P camera The average of the 3D coordinates of the midpoints.

[0132] Then calculate the centroid coordinates of each point Substituting it into formula (3) it becomes:

[0133]

[0134] In formula (4), each error is Assuming that the optimal rotation matrix R and translation matrix t are found, then The term should be zero, so formula (4) simplifies to:

[0135]

[0136] Expand formula (5):

[0137]

[0138] It can be seen that only the rotation matrix R remains in formulas (5) and (6), so the rotation matrix R and the translation matrix t are solved separately.

[0139] In formula (6), only the last term is related to the rotation matrix R, and the others are constant terms. Therefore, the target change is maximized as follows:

[0140]

[0141] Pair Matrix Perform singular value decomposition and get:

[0142]

[0143] In formula (8), U represents a unitary matrix, Λ represents a diagonal matrix, and V represents a conjugate transpose.

[0144] Then, after calculating the unitary matrix U and the conjugate transpose V, the rotation matrix can be expressed as R = VU T After determining the rotation matrix R, substitute the rotation matrix R into the aforementioned , the translation matrix t can be determined.

[0145] Thus, the optimal rotation matrix parameter R and the optimal translation matrix parameter t can be determined. Then, R and t are substituted into the above formula (2) to determine the first coordinate o base With the second coordinate o cameraThe homogeneous transformation relationship between them is the hand-eye calibration result described in this disclosure.

[0146] As can be seen from the above, in the disclosed embodiments, robot hand-eye calibration can be achieved based solely on depth information, eliminating the need for a 2D camera and reducing hardware costs. Furthermore, compared to 2D hand-eye calibration solutions in related technologies, this approach is unaffected by camera lens distortion, resulting in improved calibration accuracy and effectiveness. Furthermore, determining the sphere center coordinates through spherical fitting effectively reduces camera noise, lowers feature detection errors, and improves calibration accuracy.

[0147] In some embodiments, taking into account Figure 5 In the embodiment, when a preset number of points are randomly sampled from the point cloud data P for spherical fitting, in order to accelerate the fitting convergence process, the calibration sphere 103 can be image segmented based on the point cloud data. Figure 8 Provide explanation.

[0148] like Figure 8 As shown, in some embodiments, the hand-eye calibration method of the present disclosure samples a preset number of points from point cloud data, including:

[0149] S810 , performing image segmentation on the calibration sphere based on the point cloud data to obtain target point cloud data corresponding to the calibration sphere.

[0150] S820: Randomly sample the target point cloud data to obtain a preset number of points.

[0151] Combined with the above Figure 5 It can be seen from the embodiment that the point cloud data P of the robot arm 102 can be collected by the depth camera. For example, in one example, the point cloud image corresponding to the point cloud data P can be as follows: Figure 9 As shown in (a).

[0152] See also Figure 9 As shown in (a), the points in the point cloud data P include not only the points on the calibration sphere 103, but also the points on the robotic arm 102. If P' is randomly sampled from the point cloud data P, it is easy to sample a large number of points outside the calibration sphere 103, making it difficult to converge during spherical fitting.

[0153] Therefore, in the embodiment of the present disclosure, the calibration sphere 103 on the point cloud image can first be segmented based on the point cloud data, for example Figure 9 As shown in (c), by performing image segmentation on the calibration sphere 103 area, target point cloud data including only three-dimensional points on the calibration sphere 103 can be segmented.

[0154] After obtaining the target point cloud data, a preset number of points can be obtained by random sampling from the target point cloud data. It can be understood that since the points in the target point cloud data are all points on the calibration sphere 103, when performing spherical fitting based on the preset number of sampled points, the spherical equation can converge quickly, and the fitted spherical equation has higher accuracy, providing an accurate data basis for the subsequent calibration process. For the process of spherical fitting, those skilled in the art can refer to the aforementioned Figure 5 The implementation method is sufficient and this disclosure will not elaborate on it any more.

[0155] From the above, it can be seen that in the embodiment of the present disclosure, by performing image segmentation on the point cloud data and fitting the spherical equation based on the target point cloud data of the segmented calibration sphere, the efficiency and accuracy of spherical fitting can be improved, thereby improving the hand-eye calibration accuracy.

[0156] To further illustrate the effect of the hand-eye calibration method according to the embodiment of the present disclosure, the following Table 1 provides actual test data of the robotic arm 102 in eight different configurations.

[0157] Table 1

[0158]

[0159]

[0160] For the data shown in Table 1, after being processed by the hand-eye calibration method described above, the homogeneous transformation matrix obtained is for:

[0161]

[0162] In formula (9), represents the homogeneous transformation matrix, R represents the rotation matrix, and t represents the translation matrix.

[0163] Substitute the data in Table 1 above into the following formula to calculate the calibration root mean square error:

[0164]

[0165] In the example disclosed herein, the calculated root mean square error RMSE = 0.49 mm, while in related technologies, the calibration root mean square error based on the 2D calibration scheme is above 1 mm. The hand-eye calibration scheme of the embodiment disclosed herein greatly improves the calibration effect and accuracy.

[0166] From the above, it can be seen that in the embodiment of the present disclosure, the hand-eye calibration of the robot can be achieved based only on depth information, without the need to deploy a 2D camera, thereby reducing hardware costs. Moreover, compared with the 2D hand-eye calibration scheme in the related art, it is not affected by the distortion of the camera lens, and the calibration accuracy and effect are better. In addition, by determining the coordinates of the center of the sphere by spherical fitting, the camera noise can be effectively reduced, the feature detection error can be reduced, and the calibration accuracy can be improved. By performing image segmentation on the point cloud data and fitting the spherical equation based on the target point cloud data of the segmented calibration sphere, the efficiency and accuracy of the spherical fitting can be improved, thereby improving the hand-eye calibration accuracy.

[0167] In some embodiments, the present disclosure provides a hand-eye calibration device, see Figure 10 As shown, the device includes:

[0168] A first coordinate determination module 10 is configured to determine, based on the kinematic data of the manipulator, a first coordinate of a calibration object relative to a base coordinate system of the manipulator in a plurality of configurations of the manipulator, wherein the calibration object is fixed to the manipulator;

[0169] a second coordinate determination module 20 configured to collect point cloud data of the robotic arm in each configuration, and determine a second coordinate of the calibration object relative to the camera coordinate system based on the point cloud data;

[0170] The hand-eye calibration module 30 is configured to perform hand-eye calibration based on the first coordinates and the second coordinates of the calibration object in the multiple configurations to obtain a hand-eye calibration result.

[0171] In some embodiments, the second coordinate determination module 20 is configured to:

[0172] The point cloud data of the robotic arm in each configuration is collected by a depth camera, wherein the calibration object is located within the field of view of the depth camera.

[0173] In some embodiments, the calibration object includes a calibration sphere; and the first coordinate determination module 10 is configured to:

[0174] Pre-establishing the spherical center coordinate system of the calibration sphere;

[0175] Based on the kinematic data of the robotic arm and the mapping relationship between the spherical center coordinate system and the base coordinate system, the first coordinate of the center of the calibration sphere of the robotic arm in each configuration is determined.

[0176] In some embodiments, the calibration object includes a calibration sphere; and the second coordinate determination module 20 is configured to:

[0177] Randomly sampling a preset number of points from the point cloud data;

[0178] Performing spherical fitting based on the three-dimensional coordinates of the preset number of points to obtain a candidate spherical equation;

[0179] In response to the number of points in the point cloud data that satisfy the candidate spherical equation being greater than or equal to a preset threshold, the second coordinate of the center of the calibration sphere is determined based on the candidate spherical equation.

[0180] In some embodiments, the second coordinate determination module 20 is configured to:

[0181] In response to the number of points in the point cloud data that satisfy the candidate spherical equation being less than the preset threshold, a preset number of points are randomly sampled from the point cloud data again, and a spherical fitting is performed based on the three-dimensional coordinates of the preset number of points to obtain the candidate spherical equation until the convergence condition is met.

[0182] In some embodiments, the second coordinate determination module 20 is configured to:

[0183] Performing image segmentation on the calibration sphere based on the point cloud data to obtain target point cloud data corresponding to the calibration sphere;

[0184] A preset number of points are obtained by random sampling from the target point cloud data.

[0185] In some embodiments, the hand-eye calibration result includes a homogeneous transformation relationship between the first coordinate and the second coordinate; the hand-eye calibration module 30 is configured to:

[0186] constructing an error equation between the first coordinate and the second coordinate based on a least squares method;

[0187] Determining optimal rotation matrix parameters and translation matrix parameters in the error equation based on singular value decomposition;

[0188] A homogeneous transformation relationship between the first coordinate and the second coordinate is determined based on the rotation matrix parameters and the translation matrix parameters.

[0189] From the above, it can be seen that in the embodiment of the present disclosure, the hand-eye calibration of the robot can be achieved based only on depth information, without the need to deploy a 2D camera, thereby reducing hardware costs. Moreover, compared with the 2D hand-eye calibration scheme in the related art, it is not affected by the distortion of the camera lens, and the calibration accuracy and effect are better. In addition, by determining the coordinates of the center of the sphere by spherical fitting, the camera noise can be effectively reduced, the feature detection error can be reduced, and the calibration accuracy can be improved. By performing image segmentation on the point cloud data and fitting the spherical equation based on the target point cloud data of the segmented calibration sphere, the efficiency and accuracy of the spherical fitting can be improved, thereby improving the hand-eye calibration accuracy.

[0190] In some embodiments, the present disclosure provides a robotic arm system, see Figure 1 and Figure 2 As shown, the robotic arm system includes:

[0191] A robotic arm comprising a fixed end and a moving end, wherein the fixed end is fixed to a target object;

[0192] A calibration object is fixed to the moving end of the robotic arm;

[0193] A depth camera, wherein the calibration object is located within the field of view of the depth camera, and the depth camera is used to collect point cloud data of the calibration object;

[0194] The controller includes a processor and a memory, wherein the memory stores computer instructions, and the computer instructions are used to enable the processor to execute the method according to any embodiment of the first aspect.

[0195] In some embodiments, the present disclosure provides an example storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method described in any of the above embodiments.

[0196] In some embodiments, the examples of the present disclosure provide a computer program product, which implements the method described in any of the above embodiments when running.

[0197] Specifically, Figure 11 A schematic diagram of the structure of a robotic arm system 600 suitable for implementing the method disclosed herein is shown. Figure 11 The system shown can realize the corresponding functions of the above-mentioned processor and storage medium.

[0198] like Figure 11 As shown, system 600 includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a memory 602 or a program loaded from a storage unit 608 into the memory 602. Various programs and data required for the operation of system 600 are also stored in the memory 602. The processor 601 and the memory 602 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0199] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, and the like; an output section 607 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 608 including a hard disk; and a communication section 609 including a network interface card such as a LAN card or a modem. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 610 as needed, so that computer programs read therefrom can be installed into the storage section 608 as needed.

[0200] In particular, according to embodiments of the present disclosure, the above method process can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program tangibly embodied on a machine-readable medium, the computer program comprising program code for executing the above method. In such embodiments, the computer program can be downloaded and installed from a network via the communication portion 609 and / or installed from a removable medium 611.

[0201] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0202] Obviously, the above embodiments are merely examples for clarity of explanation and are not intended to limit the embodiments. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all embodiments here. Obvious variations or modifications arising therefrom remain within the scope of protection of the present disclosure.

Claims

1. A hand-eye calibration method, characterized in that: include: determining, based on the kinematic data of the manipulator, a first coordinate of a calibration object relative to a base coordinate system of the manipulator in a plurality of configurations of the manipulator, wherein the calibration object is fixed to the manipulator; Collecting point cloud data of the robotic arm in each configuration, and determining a second coordinate of the calibration object relative to a camera coordinate system based on the point cloud data; Hand-eye calibration is performed based on the first coordinates and the second coordinates of the calibration object in the multiple configurations to obtain a hand-eye calibration result.

2. The hand-eye calibration method according to claim 1, characterized in that: The collecting of point cloud data of the robotic arm in each configuration includes: The point cloud data of the robotic arm in each configuration is collected by a depth camera, wherein the calibration object is located within the field of view of the depth camera.

3. The hand-eye calibration method according to claim 1, characterized in that: The calibration object includes a calibration sphere; and determining, based on the kinematic data of the manipulator, a first coordinate of the calibration object relative to the manipulator base coordinate system in a plurality of configurations of the manipulator, includes: Pre-establishing the spherical center coordinate system of the calibration sphere; Based on the kinematic data of the robotic arm and the mapping relationship between the spherical center coordinate system and the base coordinate system, the first coordinate of the center of the calibration sphere of the robotic arm in each configuration is determined.

4. The hand-eye calibration method according to claim 1, characterized in that: The calibration object includes a calibration sphere; and determining a second coordinate of the calibration object relative to a camera coordinate system based on the point cloud data includes: Randomly sampling a preset number of points from the point cloud data; Performing spherical fitting based on the three-dimensional coordinates of the preset number of points to obtain a candidate spherical equation; In response to the number of points in the point cloud data that satisfy the candidate spherical equation being greater than or equal to a preset threshold, the second coordinate of the center of the calibration sphere is determined based on the candidate spherical equation.

5. The hand-eye calibration method according to claim 4, characterized in that: Also includes: In response to the number of points in the point cloud data that satisfy the candidate spherical equation being less than the preset threshold, a preset number of points are randomly sampled from the point cloud data again, and a spherical fitting is performed based on the three-dimensional coordinates of the preset number of points to obtain the candidate spherical equation until the convergence condition is met.

6. The hand-eye calibration method according to claim 4, characterized in that: The randomly sampling from the point cloud data to obtain a preset number of points includes: Performing image segmentation on the calibration sphere based on the point cloud data to obtain target point cloud data corresponding to the calibration sphere; A preset number of points are obtained by random sampling from the target point cloud data.

7. The hand-eye calibration method according to claim 1, characterized in that: The hand-eye calibration result includes a homogeneous transformation relationship between the first coordinate and the second coordinate; the hand-eye calibration is performed based on the first coordinate and the second coordinate of the calibration object in the multiple configurations to obtain the hand-eye calibration result, including: constructing an error equation between the first coordinate and the second coordinate based on a least squares method; Determining optimal rotation matrix parameters and translation matrix parameters in the error equation based on singular value decomposition; A homogeneous transformation relationship between the first coordinate and the second coordinate is determined based on the rotation matrix parameters and the translation matrix parameters.

8. A hand-eye calibration device, characterized in that: include: a first coordinate determination module configured to determine, based on the kinematic data of the manipulator, first coordinates of a calibration object relative to a base coordinate system of the manipulator in a plurality of configurations of the manipulator, wherein the calibration object is fixed to the manipulator; a second coordinate determination module, configured to collect point cloud data of the robotic arm in each configuration, and determine a second coordinate of the calibration object relative to the camera coordinate system based on the point cloud data; The hand-eye calibration module is configured to perform hand-eye calibration based on the first coordinates and the second coordinates of the calibration object in the multiple configurations to obtain a hand-eye calibration result.

9. A robotic arm system, characterized in that: include: A robotic arm comprising a fixed end and a moving end, wherein the fixed end is fixed to a target object; A calibration object is fixed to the moving end of the robotic arm; A depth camera, wherein the calibration object is located within the field of view of the depth camera, and the depth camera is used to collect point cloud data of the calibration object; A controller comprises a processor and a memory, wherein the memory stores computer instructions, and the computer instructions are used to enable the processor to execute the method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: Computer instructions are stored, and the computer instructions are used to make a computer execute the method according to any one of claims 1 to 7.

11. A computer program product, characterized in that The computer program product implements the method according to any one of claims 1 to 7 when running.

Images