Hand-eye calibration method and device and motion mechanism system

Abstract

The embodiment of the application provides a hand-eye calibration method, device and motion mechanism system, the method comprises the steps that a first homography matrix and an image principal point coordinate are acquired; a ratio of a first distance to a distance between a workpiece to be grabbed and released and an aperture of a camera is calculated as a first scale factor; a conversion relationship between the first homography matrix and a target homography matrix is determined based on the first scale factor and the image principal point coordinate as a first conversion relationship; and the target homography matrix is determined based on the first conversion relationship and the first homography matrix. The technical scheme provided by the embodiment of the application can improve the precision of hand-eye calibration.

Description

Technical Field

[0001] This application relates to the field of vision technology, and in particular to a hand-eye calibration method, device, and motion mechanism system. Background Technology

[0002] The principal point of an image is the intersection of the camera's optical axis and the imaging plane. Theoretically, the principal point is the center point of the pixel coordinate system. However, in practice, due to installation accuracy issues, the principal point may deviate from the image center. Therefore, in reality, the principal point may not be the center point of the pixel coordinate system, especially in scenarios where the optical axis of the camera lens is not perpendicular to the plane of the object's feature points.

[0003] In related technologies, for 2.5D positioning and guidance scenarios, the principal point of the image is usually taken as the center point of the pixel coordinate system. The transformation relationship between the robot's pixel coordinate system and the physical coordinate system of the motion mechanism is solved, that is, hand-eye calibration is performed. It can be seen that the hand-eye calibration results obtained in this way are not accurate. Summary of the Invention

[0004] The purpose of this application is to provide a hand-eye calibration method, device, and motion mechanism system to improve the accuracy of hand-eye calibration. The specific technical solution is as follows:

[0005] In a first aspect, embodiments of this application provide a hand-eye calibration method, the method comprising:

[0006] Obtain the first homography matrix and the image principal point coordinates; wherein, the image principal point coordinates are the coordinates of the intersection point of the camera's optical axis and the pixel plane calculated based on the first homography matrix and the second homography matrix; the first homography matrix is ​​the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism, determined by multiple images obtained by the camera taking multiple shots of the first reference object in the first scene; the second homography matrix is ​​the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism, determined by multiple images obtained by the camera taking multiple shots of the second reference object in the second scene; the first reference object and the second reference object are carried on the motion mechanism, or the camera is carried on the motion mechanism; the first distance and the second distance are different, the first distance is the distance between the first reference object and the aperture of the camera, and the second distance is the distance between the second reference object and the aperture of the camera;

[0007] Calculate the ratio of the first distance to the distance between the workpiece to be gripped and the aperture of the camera, and use it as the first scaling factor;

[0008] Based on the first scaling factor and the image principal point coordinates, the transformation relationship between the first homography matrix and the target homography matrix is ​​determined as the first transformation relationship; wherein, the target homography matrix is ​​the homography matrix between the pixel coordinate system corresponding to the workpiece to be gripped and placed and the physical coordinate system of the motion mechanism;

[0009] Based on the first transformation relationship and the first homography matrix, the target homography matrix is ​​determined.

[0010] In some embodiments, the method further includes obtaining the second homography matrix; calculating the ratio of the distance between the first reference object and the aperture of the camera to the distance between the workpiece to be gripped and the aperture of the camera, as a first scaling factor, includes:

[0011] Based on the first homography matrix and the second homography matrix, the transformation relationship between the first homography matrix and the second homography matrix is ​​determined as the second transformation relationship;

[0012] Based on the second conversion relationship, the ratio of the first distance to the second distance is determined as the second scaling factor;

[0013] The first scale factor is determined based on the second scaling factor, the height difference between the workpiece to be gripped and the first reference object, and the height difference between the first reference object and the second reference object.

[0014] In some embodiments, determining the first scale factor based on the second scale factor, the height difference between the workpiece to be gripped and the first reference object, and the height difference between the first reference object and the second reference object includes:

[0015] Calculate the ratio of the height difference between the workpiece to be gripped and the first reference object to the height difference between the first reference object and the second reference object, and use it as the depth difference ratio;

[0016] The first scaling factor is determined based on the second scaling factor and the depth difference ratio; wherein the first scaling factor is positively correlated with the second scaling factor and positively correlated with the depth difference ratio.

[0017] In some embodiments, the first transformation relationship is represented in matrix form, wherein the linear component of the first transformation relationship is positively correlated with the first scaling factor; the rotation component of the first transformation relationship is negatively correlated with the first scaling factor and positively correlated with the value of the principal point coordinates of the image.

[0018] In some embodiments, determining the transformation relationship between the first homography matrix and the second homography matrix as a second transformation relationship based on the first homography matrix and the second homography matrix includes: calculating the inverse matrix of either the first homography matrix or the second homography matrix, and calculating the product of the inverse matrix and the other as the second transformation relationship;

[0019] The second transformation relationship is represented in matrix form. The second scaling factor is positively correlated with the main diagonal element of the 2×2 matrix in the upper left corner of the second transformation relationship, and negatively correlated with the secondary diagonal element of the 2×2 matrix.

[0020] In some embodiments, the second transformation relationship is represented in matrix form, and determining the ratio of the first distance to the second distance as a second scaling factor based on the second transformation relationship includes:

[0021] Calculate the mean of the main diagonals of the top-left 2×2 block matrix in the matrix representing the second transformation relationship, and use this as the second scaling factor; or,

[0022] Calculate the square root of the determinant of the top-left 2×2 block matrix in the matrix used to represent the second transformation relationship, and use it as the second scaling factor.

[0023] In some embodiments, when the reference object is carried on the motion mechanism, the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism in any scene is obtained in the following ways, including:

[0024] The motion mechanism is controlled to carry the reference object to the first photographing position, and the camera is controlled to photograph the reference object to obtain an image of the reference object at the first photographing position, which is used as the first image;

[0025] The pixel coordinates of at least one feature point of the reference object are determined based on the first image, and are used as the pixel coordinates of the feature point corresponding to the first image.

[0026] Determine whether the number of times the motion mechanism carries the reference object to the first photographing position has reached a preset number. If the preset number has not been reached, determine a new photographing position and use the new photographing position as the first photographing position. Repeat the above steps of obtaining the feature point pixel coordinates corresponding to the first image.

[0027] If the preset number of times has been reached, then the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism is determined based on the feature point pixel coordinates corresponding to each first image and each first shooting position;

[0028] Wherein, the distance between each of the first shooting positions and the aperture of the camera is the same in a preset direction, and the preset direction is the z-axis orientation of the camera coordinate system.

[0029] In some embodiments, when the camera is mounted on the motion mechanism, the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism in any scene is obtained through the following methods, including:

[0030] The motion mechanism is controlled to carry the camera to the second shooting position, and the camera is controlled to take a picture of the reference object to obtain an image of the reference object at the second shooting position, which is used as the second image;

[0031] The pixel coordinates of at least one feature point of the reference object are determined based on the second image, and used as the pixel coordinates of the feature point corresponding to the second image.

[0032] Determine whether the number of times the motion mechanism carries the camera to the second shooting position has reached a preset number. If the preset number has not been reached, determine a new shooting position and use the new shooting position as the second shooting position. Repeat the above steps of obtaining the feature point pixel coordinates corresponding to the second image.

[0033] If the preset number of times has been reached, then the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism is determined based on the feature point pixel coordinates corresponding to each second image and each second shooting position;

[0034] Wherein, the distance between each of the second shooting positions and the reference object is the same in a preset direction, and the preset direction is the z-axis orientation of the camera coordinate system.

[0035] Secondly, embodiments of this application also provide a hand-eye calibration device, the device comprising:

[0036] A homography matrix acquisition module is used to acquire a first homography matrix and image principal point coordinates; wherein, the image principal point coordinates are the coordinates of the intersection point of the camera's optical axis and the pixel plane calculated based on the first and second homography matrices; the first homography matrix is ​​a homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism, determined by multiple images obtained by the camera taking multiple shots of a first reference object in a first scene; the second homography matrix is ​​a homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism, determined by multiple images obtained by the camera taking multiple shots of a second reference object in a second scene; the first and second reference objects are carried on the motion mechanism, or the camera is carried on the motion mechanism; the first distance and the second distance are different, the first distance is the distance between the first reference object and the aperture of the camera, and the second distance is the distance between the second reference object and the aperture of the camera;

[0037] The scaling factor calculation module is used to calculate the ratio of the distance between the first reference object and the aperture of the camera to the distance between the workpiece to be gripped and the aperture of the camera, as the first scaling factor;

[0038] The transformation relationship determination module is used to determine the transformation relationship between the first homography matrix and the target homography matrix based on the first scaling factor and the coordinates of the principal point of the image, as the first transformation relationship; wherein, the target homography matrix is ​​the homography matrix between the pixel coordinate system corresponding to the workpiece to be gripped and placed and the physical coordinate system of the motion mechanism;

[0039] The homography matrix determination module is used to determine the target homography matrix based on the first transformation relationship and the homography matrix corresponding to the first reference object.

[0040] In some embodiments, the homography matrix acquisition module is further configured to acquire the second homography matrix;

[0041] The scaling factor calculation module includes:

[0042] The first determining unit is used to determine the transformation relationship between the first homography matrix and the second homography matrix based on the first homography matrix and the second homography matrix, as the second transformation relationship;

[0043] The second determining unit is used to determine the ratio of the first distance to the second distance based on the second conversion relationship, as a second scaling factor;

[0044] The third determining unit is used to determine the first scaling factor based on the second scaling factor, the height difference between the workpiece to be gripped and the first reference object, and the height difference between the first reference object and the second reference object.

[0045] In some embodiments, the third determining unit is specifically used for,

[0046] Calculate the ratio of the height difference between the workpiece to be gripped and the first reference object to the height difference between the first reference object and the second reference object, and use it as the depth difference ratio;

[0047] The first scaling factor is determined based on the second scaling factor and the depth difference ratio; wherein the first scaling factor is positively correlated with the second scaling factor and positively correlated with the depth difference ratio.

[0048] In some embodiments, the first transformation relationship is represented in matrix form, wherein the linear component of the first transformation relationship is positively correlated with the first scaling factor; the rotation component of the first transformation relationship is negatively correlated with the first scaling factor and positively correlated with the value of the principal point coordinates of the image.

[0049] In some embodiments, the first determining unit is specifically configured to calculate the inverse matrix of either the first homography matrix or the second homography matrix, and calculate the product of the inverse matrix and the other homography matrix as the second transformation relationship;

[0050] The second transformation relationship is represented in matrix form. The second scaling factor is positively correlated with the main diagonal element of the 2×2 matrix in the upper left corner of the second transformation relationship, and negatively correlated with the secondary diagonal element of the 2×2 matrix.

[0051] In some embodiments, the second transformation relationship is represented in matrix form, and determining the ratio of the first distance to the second distance as a second scaling factor based on the second transformation relationship includes:

[0052] Calculate the mean of the main diagonals of the top-left 2×2 block matrix in the matrix representing the second transformation relationship, and use this as the second scaling factor; or,

[0053] Calculate the square root of the determinant of the top-left 2×2 block matrix in the matrix used to represent the second transformation relationship, and use it as the second scaling factor.

[0054] In some embodiments, when the reference object is carried on the motion mechanism, the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism in any scene is obtained in the following ways, including:

[0055] The motion mechanism is controlled to carry the reference object to the first photographing position, and the camera is controlled to photograph the reference object to obtain an image of the reference object at the first photographing position, which is used as the first image;

[0056] The pixel coordinates of at least one feature point of the reference object are determined based on the first image, and are used as the pixel coordinates of the feature point corresponding to the first image.

[0057] Determine whether the number of times the motion mechanism carries the reference object to the first photographing position has reached a preset number. If the preset number has not been reached, determine a new photographing position and use the new photographing position as the first photographing position. Repeat the above steps of obtaining the feature point pixel coordinates corresponding to the first image.

[0058] If the preset number of times has been reached, then the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism is determined based on the feature point pixel coordinates corresponding to each first image and each first shooting position;

[0059] Wherein, the distance between each of the first shooting positions and the aperture of the camera is the same in a preset direction, and the preset direction is the z-axis orientation of the camera coordinate system.

[0060] In some embodiments, when the camera is mounted on the motion mechanism, the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism in any scene is obtained through the following methods, including:

[0061] The motion mechanism is controlled to carry the camera to the second shooting position, and the camera is controlled to take a picture of the reference object to obtain an image of the reference object at the second shooting position, which is used as the second image;

[0062] The pixel coordinates of at least one feature point of the reference object are determined based on the second image, and used as the pixel coordinates of the feature point corresponding to the second image.

[0063] Determine whether the number of times the motion mechanism carries the camera to the second shooting position has reached a preset number. If the preset number has not been reached, determine a new shooting position and use the new shooting position as the second shooting position. Repeat the above steps of obtaining the feature point pixel coordinates corresponding to the second image.

[0064] If the preset number of times has been reached, then the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism is determined based on the feature point pixel coordinates corresponding to each second image and each second shooting position;

[0065] Wherein, the distance between each of the second shooting positions and the reference object is the same in a preset direction, and the preset direction is the z-axis orientation of the camera coordinate system.

[0066] Thirdly, embodiments of this application also provide a first motion mechanism system, including a processor, a motion mechanism, and a camera, wherein the camera is disposed outside the motion mechanism.

[0067] The motion mechanism is used to carry the first reference object in a first scene and to carry the second reference object in a second scene.

[0068] The camera is used to take multiple shots of a first reference object carried by the motion mechanism in the first scene to obtain multiple third images; and to take multiple shots of a second reference object carried by the motion mechanism in the second scene to obtain multiple fourth images; wherein the first distance and the second distance are different, the first distance is the distance between the first reference object and the aperture of the camera, and the second distance is the distance between the second reference object and the aperture of the camera;

[0069] The processor is configured to acquire a first homography matrix and image principal point coordinates determined based on the first and second homography matrices; wherein the image principal point coordinates are the coordinates of the intersection of the camera's optical axis and the pixel plane; the first homography matrix is ​​a homography matrix of the pixel coordinate system and the physical coordinate system of the motion mechanism determined based on the plurality of third images; and the second homography matrix is ​​a homography matrix of the pixel coordinate system and the physical coordinate system of the motion mechanism determined based on the plurality of fourth images.

[0070] The processor is further configured to calculate the ratio of the first distance to the distance between the workpiece to be gripped and the aperture of the camera, as a first scaling factor; based on the first scaling factor and the principal point coordinates of the image, determine the transformation relationship between the first homography matrix and the target homography matrix, as a first transformation relationship; based on the first transformation relationship and the first homography matrix, determine the target homography matrix; wherein, the target homography matrix is ​​the homography matrix between the pixel coordinate system corresponding to the workpiece to be gripped and the physical coordinate system of the motion mechanism.

[0071] Fourthly, embodiments of this application also provide a second motion mechanism system, including a processor, a motion mechanism, and a camera, wherein the camera is carried on the motion mechanism.

[0072] The motion mechanism is used to carry the camera in a first scene and to carry the camera in a second scene.

[0073] The camera is used to take multiple photos of a first reference object in the first scene while being carried by the motion mechanism to obtain multiple fifth images; and in the second scene, while being carried by the motion mechanism to obtain multiple sixth images of a second reference object; wherein the first distance and the second distance are different, the first distance being the distance between the first reference object and the aperture of the camera, and the second distance being the distance between the second reference object and the aperture of the camera;

[0074] The processor is configured to acquire a first homography matrix and image principal point coordinates determined based on the first and second homography matrices; wherein the image principal point coordinates are the coordinates of the intersection of the camera's optical axis and the pixel plane; the first homography matrix is ​​a homography matrix of the pixel coordinate system and the physical coordinate system of the motion mechanism determined based on the plurality of fifth images; the second homography matrix is ​​a homography matrix of the pixel coordinate system and the physical coordinate system of the motion mechanism determined based on the plurality of sixth images;

[0075] The processor is further configured to calculate the ratio of the first distance to the distance between the workpiece to be gripped and the aperture of the camera, as a first scaling factor; based on the first scaling factor and the principal point coordinates of the image, determine the transformation relationship between the first homography matrix and the target homography matrix, as a first transformation relationship; based on the first transformation relationship and the first homography matrix, determine the target homography matrix; wherein, the target homography matrix is ​​the homography matrix between the pixel coordinate system corresponding to the workpiece to be gripped and the physical coordinate system of the motion mechanism.

[0076] Fifthly, embodiments of this application also provide an electronic device, including:

[0077] Memory, used to store computer programs;

[0078] The processor, when executing a program stored in memory, implements any of the hand-eye calibration methods described above.

[0079] Sixthly, embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the above-described hand-eye calibration methods.

[0080] In a seventh aspect, embodiments of this application also provide a computer program product containing instructions that, when run on a computer, cause the computer to execute any of the hand-eye calibration methods described above.

[0081] Beneficial effects of the embodiments in this application:

[0082] In the technical solution provided in this application embodiment, a first homography matrix and image principal point coordinates are obtained. The image principal point coordinates are the coordinates of the intersection point of the optical axis and the pixel plane, calculated based on the first and second homography matrices. Specifically, the first homography matrix is ​​the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism, determined based on multiple images of a first reference object captured by the camera multiple times in the first scene. The second homography matrix is ​​the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism, determined based on multiple images of a second reference object captured by the camera multiple times in the second scene. The first and second reference objects are carried by the motion mechanism, or the camera is carried by the motion mechanism. The difference between the first and second scenes lies in the distance between the first reference object and the second reference object in the first scene. The distance between the camera apertures (i.e., the first distance) is different from the distance between the second reference object and the camera aperture in the second scene (i.e., the second distance). Since the principal point coordinates of the image are the coordinates of the intersection of the camera's optical axis and the pixel plane, the principal point coordinates of the image can be obtained based on the first homography matrix and the second homography matrix. Then, the ratio of the first distance to the distance between the workpiece to be gripped and the camera aperture (i.e., the first scaling factor) is calculated. Based on the first scaling factor and the principal point coordinates of the image, the transformation relationship between the first homography matrix and the target homography matrix (i.e., the first transformation relationship) can be obtained. Then, based on the first transformation relationship and the first homography matrix, the target homography matrix can be determined, that is, the homography matrix between the pixel coordinate system corresponding to the workpiece to be gripped and the physical coordinate system of the motion mechanism.

[0083] Understandably, in this application, the target homography matrix is ​​solved based on the first homography matrix and the first transformation relation. The image principal point coordinates used to solve the first transformation relation are the actual image principal point coordinates calculated based on the first homography matrix and the second homography matrix, rather than using the theoretical pixel coordinate system center point as the image principal point coordinates. Therefore, the accuracy of determining the target homography matrix based on the hand-eye calibration method shown in this application is relatively high.

[0084] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0085] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0086] Figure 1 A schematic diagram illustrating the positional relationship between pixel coordinates and world coordinates in an embodiment of this application.

[0087] Figure 2 for Figure 1 A detailed schematic diagram illustrating the positional relationship between midpoint P' and point P".

[0088] Figure 3 This is a schematic diagram of the first flowchart of the hand-eye calibration method provided in the embodiments of this application;

[0089] Figure 4 A flowchart illustrating the determination of a first scaling factor provided in an embodiment of this application;

[0090] Figure 5 A flowchart illustrating the determination of a first homography matrix or a second homography matrix provided in an embodiment of this application;

[0091] Figure 6 Another flowchart illustrating the determination of the first homography matrix or the second homography matrix provided for embodiments of this application;

[0092] Figure 7 This is a schematic diagram of a second process for the hand-eye calibration method provided in an embodiment of this application;

[0093] Figure 8 A schematic diagram of the reference depth difference provided in the embodiments of this application;

[0094] Figure 9 A schematic diagram of the running depth difference provided in the embodiments of this application;

[0095] Figure 10 A schematic diagram of a software interface for a hand-eye calibration method provided in an embodiment of this application;

[0096] Figure 11 A schematic diagram of another software interface for the hand-eye calibration method provided in the embodiments of this application;

[0097] Figure 12 A schematic diagram of the hand-eye calibration device provided in the embodiments of this application;

[0098] Figure 13 A schematic diagram of a first motion mechanism system provided in an embodiment of this application;

[0099] Figure 14 A schematic diagram of a second motion mechanism system provided in an embodiment of this application;

[0100] Figure 15 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0101] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0102] Hand-eye calibration is a crucial step in robotics, involving the intersection of computer vision and robot control. Typically, the "eye" refers to a camera mounted on the robotic arm or fixed in the environment, while the "hand" refers to the actuator at the end effector of the robot arm, such as a gripper or suction cup. The main function of hand-eye calibration is to establish the coordinate transformation relationship between the camera's pixel coordinate system and the coordinate system of the robot arm's end effector, thereby achieving precise localization of the target object.

[0103] For 2D guided scenarios, N-point calibration is typically used; for 3D scenarios, camera calibration and hand-eye calibration are typically used. The purpose of camera calibration is to solve for the intrinsic and extrinsic parameters of the camera; the purpose of hand-eye calibration is to solve for the coordinate transformation relationship between the camera coordinate system and the coordinate system of the robotic arm (also known as the motion mechanism); N-point calibration refers to solving for the coordinate transformation relationship between the pixel coordinate system and the physical coordinate system of the motion mechanism by using the coordinates of N corresponding points in the pixel coordinate system and the physical coordinate system of the motion mechanism (N is a positive integer greater than or equal to 2).

[0104] For 2.5D positioning scenarios, related technologies typically employ N-point calibration to solve the transformation relationship from the pixel coordinate system to the physical coordinate system of the motion mechanism. Then, using this transformation relationship, along with the distances from the aperture of the reference camera position to the feature points of the workpiece to be grasped and placed, and the distances from the aperture of the camera position to the feature points of the workpiece to be grasped and placed, the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism during actual operation is calculated. In 2.5D positioning scenarios, the pose of the workpiece to be grasped and placed changes only within a two-dimensional plane, while its distance (or height) from the camera also changes. However, this approach has the following drawbacks.

[0105] a) This scheme uses only a single homography matrix, making it impossible to decompose and obtain the camera's intrinsic and extrinsic parameters. Therefore, this scheme is based on the premise that the image principal point coordinates are the center point of the pixel coordinate system. However, while ideally the optical axis of the lens (i.e., the straight line passing through the lens and perpendicular to the sensor plane) should pass through the exact center of the image, in reality, due to installation accuracy issues, errors always exist. Therefore, the image principal point coordinates cannot be the center point coordinates of the pixel coordinate system. It is evident that using the pixel coordinate system center point as the image principal point is inaccurate. Consequently, the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism obtained by using the image principal point as the pixel coordinate system center point is also inaccurate.

[0106] b. When the optical axis of the lens is not perpendicular to the plane of the workpiece feature point, and the height of the plane where the workpiece feature point is located changes in the z-direction, the feature point in the camera coordinate system not only changes in the Z-direction coordinate, but this method cannot be compatible with this type of scenario.

[0107] c. The exact position of the camera aperture is difficult to determine. Therefore, the distance from the camera aperture to the feature point of the workpiece to be gripped and placed is difficult to measure accurately. Based on this, the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism is naturally inaccurate.

[0108] To improve the accuracy of the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism, i.e. to improve the accuracy of hand-eye calibration, this application provides a hand-eye calibration method.

[0109] To facilitate subsequent understanding, the following will be explained in advance... Figure 1 The example shown (i.e., the pinhole imaging model) illustrates the transformation relationship between the pixel coordinate system and the physical coordinate system of the moving mechanism. To facilitate understanding of the following content, we will first explain... Figure 1 and Figure 2 Some special expressions in the text will be explained. Figure 1 A point P" (Puv) is both the projection point of point P(x, y, z) onto the image plane coordinate system (P"(x", y") and the projection point of point P"(x", y") onto the pixel coordinate system (p). uv (u, v); for example Figure 1 and Figure 2 The axis represented by Z' (z") in the equation means that the axis is both the Z' axis and the z" axis; Figure 1 In this context, p(p') represents a point that is both a point P(x, y, z) in the world coordinate system and a point P'(x', y', z') in the camera coordinate system. Figure 1 In this context, Z'(p') represents the coordinates of point P(x, y, z) in the world coordinate system along the z' direction in the camera coordinate system. Figure 2 Z'p' in Figure 1 The meaning of Z' (p') is the same. The circle at the position of O' represents the optical center.

[0110] Figure 1 In this context, the world coordinate system is denoted as O. xyz The camera coordinate system is O. x＇y＇z＇ (It can also be written as O'(x'y'z'), that is...) Figure 1 The plane is represented by the coordinate system denoted by x', y', and z'; the image plane coordinate system is O. x"y" (It can also be written as O"(x"y"z"), that is Figure 1The plane represented by the coordinate system denoted by x"", y"", z" is, in other words, like Figure 1 (A plane in which horizontal and vertical lines intersect within the image plane); the pixel coordinate system is O. uv (Right now Figure 1 The plane represented by the coordinate system denoted by u and v).

[0111] In the world coordinate system, there exists a point P(x, y, z). The projection point of point P(x, y, z) in the image plane coordinate system is P"(x", y"). The projection point of point P"(x", y") in the pixel coordinate system is p. uv If (u, v), then P"(x", y") and point p uv The relationship between (u, v) can be expressed in the form shown in formulas (1) and (2) below.

[0112] u=αx"+c x (1)

[0113] v=βy"+c y (2)

[0114] In the above formulas (1) and (2), (cx, cy) are the coordinates of the principal point of the image (e.g., ...). Figure 1 The point indicated by O"(cx, cy) in the image plane is the intersection of the camera's optical axis and the imaging plane, denoted as c; α is the scaling ratio along the u-axis required to convert image plane coordinates to pixel coordinates, or, it can be understood as how many pixels exist per unit length along the u-axis when converting image plane coordinates to pixel coordinates; β is the scaling ratio along the v-axis required to convert image plane coordinates to pixel coordinates, or, it can be understood as how many pixels exist per unit length along the v-axis when converting image plane coordinates to pixel coordinates; where the unit length can be 1 millimeter (mm), 1 meter (m), etc.

[0115] Figure 2 for Figure 1 A detailed schematic diagram illustrating the positional relationship between midpoint P' and point P'", by... Figure 2 It can be seen that point P"(x", y") in the image coordinate system is obtained by passing through similar triangles to point P'(x', y', z') in the camera coordinate system. Therefore, there is a relationship between point P'(x', y', z') and point P"(x", y") as shown in the following formula (3).

[0116] (3)

[0117] Where f is the focal length of the camera, i.e., the distance from the camera center (e.g., O") or aperture to the image plane (e.g., O'); z' is the coordinate value of point P(x, y, z) in the world coordinate system in the z' direction in the camera coordinate system. Figure 2The notation is Z'p', which can also be understood as... Figure 1 Z'(p') in the middle.

[0118] Combining the above formulas (1)-(3), we can obtain the coordinates of point P'(x', y', z') in the camera coordinate system and point p in the pixel coordinate system. uv The conversion relationship between (u, v) is shown in the following formulas (4)-(5).

[0119] (4)

[0120] (5)

[0121] In the above formulas (4)-(5), f x =αf,f y =βf; f is the focal length of the camera, therefore, f x and f y It can be referred to as the equivalent focal length.

[0122] To make the above formulas (4)-(5) linear transformations, homogeneous coordinates can be introduced to express them, resulting in the following formula (6):

[0123] (6)

[0124] Among them, f x and f y For equivalent focal length; (c x c y () represents the coordinates of the principal point of the image.

[0125] Since point P'(x', y', z') in the camera coordinate system is obtained by rotation and translation transformation of point P(x, y, z) in the world coordinate system, the relationship between point P'(x', y', z') in the camera coordinate system and point P(x, y, z) in the world coordinate system can be expressed as shown in the following formula (7).

[0126] (7)

[0127] Among them, R 3×3 R is a 3×3 rotation matrix from the world coordinate system to the camera coordinate system. 3×1 It is a 3×1 translation vector from the world coordinate system to the camera coordinate system.

[0128] Combining the above formulas (6) and (7), the distance from point P(x, y, z) in the world coordinate system to point p in the pixel coordinate system is... uv The conversion relationship between (u, v) can be expressed in the form shown in the following formula (8).

[0129] (8)

[0130] Among them, f x f y c x c y This can be understood as a camera's intrinsic parameter, a fixed and unchanging quantity; R 3×3 and T 3×1 It can be understood as the camera's extrinsic parameters; the camera's intrinsic parameter matrix is ​​denoted as K, and the camera's extrinsic parameter matrix is ​​denoted as [RT], which completes the transformation in the above formula (8); at this time, M is the projection matrix.

[0131] In the above formula (8), the rotation matrix R 3×3 One matrix representation of it is shown in the following formula (9).

[0132] (9)

[0133] In the above formula (9), α is the angle of rotation around the X-axis; β is the angle of rotation around the Y-axis; and γ is the angle of rotation around the Z-axis.

[0134] Since in a 2D scene, the workpiece to be gripped and the motion mechanism only rotate around the Z-axis, for example, a four-axis selective compliance assembly robot arm (SCARA robot) or an XY motion module can only rotate around the Z-axis, the matrix R shown in the above formula (9) can be expressed in the form shown in the following formula (10).

[0135] (10)

[0136] In a 2D positioning-guided scenario, combining the above formulas (8) and (10), z c Let z' be the distance in the z-direction from the feature point plane of the reference object to the lens aperture (or camera center). c Then we can obtain the following formula (11).

[0137] (11)

[0138] Simplifying the above formula (11) yields the following formula (12).

[0139] (12)

[0140] The above formula (12) can be understood as the transformation relationship between the pixel coordinate system and the world coordinate system, that is, the transformation relationship between the pixel coordinate system and the physical coordinate system of the motion mechanism.

[0141] The hand-eye calibration method provided in this application will be described in detail below through specific embodiments.

[0142] The hand-eye calibration method provided in this application is applied to electronic devices, such as desktop computers, portable computers, and smart mobile terminals.

[0143] See Figure 3 , Figure 3 A first flowchart illustrating the hand-eye calibration method provided in this application includes the following steps:

[0144] Step S31: Obtain the first homography matrix and the coordinates of the principal point of the image;

[0145] Step S32: Calculate the ratio of the first distance to the distance between the workpiece to be gripped and the camera aperture, and use it as the first scaling factor;

[0146] Step S33: Based on the first scaling factor and the coordinates of the principal point of the image, determine the transformation relationship between the first homography matrix and the target homography matrix, and use it as the first transformation relationship;

[0147] Step S34: Determine the target homography matrix based on the first transformation relationship and the first homography matrix.

[0148] In the technical solution provided in this application embodiment, a first homography matrix and image principal point coordinates are obtained. The image principal point coordinates are the coordinates of the intersection point of the optical axis and the pixel plane, calculated based on the first and second homography matrices. Specifically, the first homography matrix is ​​the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism, determined based on multiple images of the first reference object captured by the camera in the first scene. The second homography matrix is ​​the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism, determined based on multiple images of the second reference object captured by the camera in the second scene. The first and second reference objects are carried by the motion mechanism, or the camera is carried by the motion mechanism. The difference between the first and second scenes lies in the distance between the first reference object and the second reference object in the first scene. The distance between the camera apertures (i.e., the first distance) is different from the distance between the second reference object and the camera aperture in the second scene (i.e., the second distance). Since the principal point coordinates of the image are the coordinates of the intersection of the camera's optical axis and the pixel plane, the principal point coordinates of the image can be obtained based on the first homography matrix and the second homography matrix. Then, the ratio of the first distance to the distance between the workpiece to be gripped and the camera aperture (i.e., the first scaling factor) is calculated. Based on the first scaling factor and the principal point coordinates of the image, the transformation relationship between the first homography matrix and the target homography matrix (i.e., the first transformation relationship) can be obtained. Then, based on the first transformation relationship and the first homography matrix, the target homography matrix can be determined, that is, the homography matrix between the pixel coordinate system corresponding to the workpiece to be gripped and the physical coordinate system of the motion mechanism.

[0149] Understandably, in this application, the target homography matrix is ​​solved based on the first homography matrix and the first transformation relation. The image principal point coordinates used to solve the first transformation relation are the actual image principal point coordinates calculated based on the first homography matrix and the second homography matrix, rather than using the theoretical pixel coordinate system center point as the image principal point coordinates. Therefore, the accuracy of determining the target homography matrix based on the hand-eye calibration method shown in this application is relatively high.

[0150] In step S31 above, the motion mechanism can be understood as the robot's robotic arm; the homography matrix is ​​usually used to identify the transformation relationship between two coordinate systems. In this embodiment, the homography matrix can be understood as the transformation relationship between the pixel coordinate system and the physical coordinate system of the motion mechanism; the first homography matrix is ​​the transformation relationship between the pixel coordinate system and the physical coordinate system of the motion mechanism determined in the first scene, and the second homography matrix is ​​the transformation relationship between the pixel coordinate system and the physical coordinate system of the motion mechanism determined in the second scene.

[0151] The first scenario includes a first reference object, a camera, and a motion mechanism. In one possible instance, the first reference object is carried on the motion mechanism, while the camera is independent of the motion mechanism (also known as the "eye" outside the "hand"). That is, the camera may not be directly mounted on the motion mechanism but placed in a fixed position. In this case, when acquiring the homography matrix, the first reference object can be moved by controlling the motion mechanism, but the camera will not move with the motion mechanism. In another possible instance, the first reference object is independent of the motion mechanism, while the camera is carried on the motion mechanism (also known as the "eye" on the "hand"). That is, the camera may be directly mounted on the motion mechanism. In this case, when acquiring the homography matrix, the camera can be moved by controlling the motion mechanism. The second scenario is similar to the first scenario, except that the first reference object becomes the second reference object.

[0152] Furthermore, to facilitate the subsequent calculation of the principal point coordinates of the image based on the first and second homography matrices, the distances between the reference objects in the first and second scenes and the camera aperture must be different. That is, if the first distance represents the distance between the first reference object and the camera aperture, and the second distance represents the distance between the second reference object and the camera aperture, then the first distance and the second distance must be different. The distance between the reference object and the camera aperture can also be understood as the distance between the feature point plane of the reference object and the camera aperture.

[0153] Understandably, for any two specific reference objects, there is no specific limitation on which reference object is used as the first reference object and which as the second reference object. The specific first and second distances depend on which reference object is used as the first reference object and which as the second reference object. For example, assuming there are reference objects A and B, in some embodiments, reference object A can be used as the first reference object and reference object B as the second reference object. In this case, the first distance is actually the distance between reference object A and the camera aperture, and the second distance is actually the distance between reference object B and the camera aperture. In other embodiments, reference object B can be used as the first reference object and reference object A as the second reference object. In this case, the first distance is actually the distance between reference object B and the camera aperture, and the second distance is actually the distance between reference object A and the camera aperture.

[0154] As can be seen from the above examples, in this application, the first distance and the second distance refer to the first reference object and the second reference object, not to a specific reference object. For example, the first distance represents the distance between the first reference object and the camera aperture, not the distance between reference object A and the camera aperture, because it can also be the distance between any specific reference object, such as reference object B or reference object C, and the camera aperture. The same applies to the first scale factor, second scale factor, first homography matrix, and second homography matrix mentioned below.

[0155] It is understandable that there can be multiple specific reference points. For example, if there are n specific reference points, such as reference point 1, reference point 2, ..., reference point n, then any two of them can be taken as the first reference point and the second reference point to determine the first homography matrix and the second homography matrix, and then the target homography matrix can be determined.

[0156] Alternatively, in some embodiments, any two reference objects may be selected from a plurality of reference objects as the first reference object and the second reference object. After determining the first homography matrix and the second homography matrix, a target homography matrix may be determined in any of the ways shown below. Then, another two reference objects may be selected from a plurality of reference objects as the first reference object and the second reference object. After determining the first homography matrix and the second homography matrix, a target homography matrix may be determined in any of the ways shown below. Finally, the target homography matrix may be determined based on the two determined target homography matrices.

[0157] The electronic device can determine a first homography matrix based on multiple images obtained by the camera taking multiple shots of a first reference object in a first scene, and determine a second homography matrix based on multiple images obtained by the camera taking multiple shots of a second reference object in a second scene. For details on how to determine the homography matrix, please refer to the following... Figure 5 and Figure 6The description in the text will not be repeated here.

[0158] Regarding the description of the principal point in the embodiments of this application, the principal point is the intersection of the camera's optical axis and the imaging plane, and the coordinates of the principal point are the coordinates of the intersection of the camera's optical axis and the pixel plane, representing the coordinate values ​​of a fixed point in the physical world projected onto the pixel coordinate system. Since the physical coordinates of the principal point remain fixed, the coordinates of the principal point projected onto the physical coordinate system remain unchanged when performing hand-eye calibration at different heights. Based on this principle, the coordinates of the principal point can be calculated using the first homography matrix and the second homography matrix obtained in step S31 above.

[0159] It is understood that, in the embodiments of this application, obtaining the principal point coordinates of the image can be achieved by the electronic device used to calculate the target homography matrix directly obtaining the first homography matrix and the second homography matrix, and then calculating the principal point coordinates of the image based on the first homography matrix and the second homography matrix; or, it can be achieved by the electronic device used to calculate the target homography matrix directly obtaining the first homography matrix and the principal point coordinates of the image, without obtaining the second homography matrix, and the obtained principal point coordinates of the image are calculated by other devices (such as devices used to determine the first homography matrix and the second homography matrix) based on the first homography matrix and the second homography matrix.

[0160] The principle and method of determining the principal point coordinates of an image based on the first and second homography matrices will be explained in detail below using formulas (13)-(18).

[0161] The essence of solving the transformation relationship between the pixel coordinate system and the physical coordinate system of the motion mechanism is to solve the homography matrix between two planes (i.e., the image plane and the motion plane of the motion mechanism). For example, the relationship between the pixel coordinate system and the physical coordinate system of the motion mechanism can be expressed as shown in the following formula (13):

[0162] (13)

[0163] Where (u, v) can be understood as coordinates in the pixel coordinate system; (x, y) can be understood as coordinates in the physical coordinate system of the motion mechanism; H= , representing the transformation relationship from the pixel coordinate system to the physical coordinate system of the motion mechanism, can be understood as a homography matrix, where h 11 h 12 ... h 32 h 33 Here are the parameters in the homography matrix H; it is understandable that the degree of freedom of the transformation relationship from the pixel coordinate system to the physical coordinate system of the motion mechanism is 8, therefore, h 33 with h 11 ~h 32 It is linearly dependent, therefore, we can apply this to h. 33Normalization is performed, that is, h 33 Set it to 1.

[0164] If the homography matrix obtained through N-point calibration from the pixel coordinate system to the physical coordinate system of the motion mechanism is denoted as M... Base Then, by inverting the matrix and normalizing the ninth element, we can obtain matrix M. Base_inv , of which M Base_inv This can be understood as the homography matrix from the physical coordinate system of the motion mechanism to the pixel coordinate system. M Base With M Base_inv The relationship between them can be expressed in the form shown in the following formula (14).

[0165] M Base_inv =(M Base ) -1 (14)

[0166] Combining the above formulas (13) and (14), the transformation relationship between the pixel coordinate system and the physical coordinate system of the motion mechanism can be expressed as shown in the following formula (15).

[0167] (15)

[0168] From the above formulas (12) and (15), it can be seen that M Base_inv It is equivalent to the projection matrix M, and matrix M is equivalent to the first 3×3 matrix on the right side of equation (12) above.

[0169] Since the physical coordinates of the principal point in the image remain constant, the coordinates of the principal point projected onto the physical coordinate system remain unchanged during hand-eye calibration at different heights. Therefore, the following equation (16) exists.

[0170] M Base1 ×c=M Base2 ×c (16)

[0171] Among them, M Base1 M is the first homography matrix obtained in step S12 above; Base1 The second homography matrix obtained in step S12 above; c is the coordinate of the principal point of the image, which can be denoted as... .

[0172] Let homography matrix , Then we have the equation shown in formula (17).

[0173] (17)

[0174] Where A is a 2×2 matrix, t represents the offset of the first homography matrix; A' is a 2×2 matrix, t' represents the offset of the second homography matrix; and c represents the coordinates of the principal point of the image.

[0175] Simplifying the above formula (17), we can obtain the equation shown in the following formula (18).

[0176] Ac+t= A'c+t' (18)

[0177] The coordinates of the principal point of the image, c, can be obtained by using the above formula (18).

[0178] In step S32 above, the workpiece to be grasped and placed can be understood as the object that the robot needs to grasp and place using its motion mechanism during operation; the first distance is the distance between the first reference object and the camera aperture; the first scaling factor is the ratio of the first distance to the distance between the workpiece to be grasped and placed and the camera aperture, that is, the first scaling factor is the ratio of the distance between the first reference object and the camera aperture to the distance between the workpiece to be grasped and placed and the camera aperture. The distance between the workpiece to be grasped and placed and the camera aperture can also be understood as the distance between the plane containing the feature point of the workpiece to be grasped and placed and the camera aperture; both the distance between the first reference object and the camera aperture, and the distance between the workpiece to be grasped and placed and the camera aperture, can be obtained by measurement, thus directly yielding the first scaling factor; however, in some examples, the distance between the first reference object and the camera aperture, and the distance between the workpiece to be grasped and placed and the camera aperture, may not be measured, but the first scaling factor may be calculated in other forms, as detailed below. Figure 4 The description in the text.

[0179] Referring to the description in step S31 above, for two specific reference objects, there is no specific limitation on which reference object is the first reference object and which is the second reference object. In step S32, continuing with the example of having reference object A and reference object B, when reference object A is the first reference object and reference object B is the second reference object, the first distance is actually the distance between reference object A and the camera aperture, and the first scaling factor is actually the ratio of the distance between reference object A and the camera aperture to the distance between the workpiece to be gripped and the camera aperture; when reference object B is the first reference object and reference object A is the second reference object, the first distance is actually the distance between reference object B and the camera aperture, and the first scaling factor is actually the ratio of the distance between reference object B and the camera aperture to the distance between the workpiece to be gripped and the camera aperture.

[0180] In step S33 above, the target homography matrix is ​​the homography matrix between the pixel coordinate system corresponding to the workpiece to be gripped and the physical coordinate system of the motion mechanism. It is understandable that although the target homography matrix, the first homography matrix, and the second homography matrix all represent the transformation relationship between the pixel coordinate system and the physical coordinate system of the motion mechanism, the distance between the object (such as the first reference object, the second reference object, and the workpiece to be gripped) and the camera aperture affects the transformation relationship between the pixel coordinate system and the physical coordinate system of the motion mechanism. Since the distance between the first reference object and the camera aperture (i.e., the first distance) and the distance between the second reference object and the camera aperture (i.e., the second distance) are not the same, the first homography matrix and the second homography matrix are not the same. Similarly, the distance between the workpiece to be gripped and the camera aperture is not necessarily the same as the first distance and the second distance; therefore, the target homography matrix is ​​not necessarily the same as the first homography matrix and the second homography matrix.

[0181] The first transformation relationship is the transformation relationship between the first homography matrix and the target homography matrix. The electronic device can solve for this first transformation relationship based on the first scaling factor and the coordinates of the principal point of the image.

[0182] In some embodiments, the first transformation relationship can be represented in the form of a matrix. The linear component in the first transformation relationship can be understood as the main diagonal element in the 2×2 block matrix in the upper left corner of the matrix representing the first transformation relationship, which is positively correlated with the first scaling factor. The rotation component in the first transformation relationship can be understood as the element in the 2×1 matrix in the upper right corner of the matrix representing the first transformation relationship, which is negatively correlated with the first scaling factor and positively correlated with the value of the principal point coordinates of the image.

[0183] Specifically, in some embodiments, the electronic device can determine the transformation relationship (i.e., the first transformation relationship) between the first homography matrix and the target homography matrix using the following formula (19-1).

[0184] (19-1)

[0185] Where T is the first transformation relationship; k' is the first scaling factor; c is the coordinates of the principal point of the image; and E is the identity matrix.

[0186] It is understood that the above formula (19-1) is only one possible calculation formula. In some embodiments, the first transformation relationship can also be calculated by other formulas obtained by simple transformation of the above formula (19-1). For example, see the following formula (19-2).

[0187] (19-2)

[0188] Where T is the first transformation relation; k' is the first scaling factor; c is the coordinates of the principal point of the image; and J is a matrix similar to the identity matrix.

[0189] As can be understood, based on the content shown in step S32 above, the actual value of the first scaling factor depends on which reference object is used as the first reference object and which reference object is used as the second reference object. When reference object A is used as the first reference object and reference object B is used as the second reference object, the first scaling factor is actually the ratio of the distance between reference object A and the camera aperture to the distance between the workpiece to be gripped and the camera aperture. At this time, substituting the first scaling factor into the formula (19-1) or (19-2) above, the calculated T is actually the conversion relationship between the homography matrix determined in the scene where reference object A is the reference object and the target homography matrix. Then, according to the subsequent step S34, based on the homography matrix determined in the scene where reference object A is the reference object and the first scaling factor (i.e., the distance between reference object A and the camera aperture), The target homography matrix can be determined by the ratio of the distance between the reference object B and the aperture of the camera to the distance between the reference object B and the aperture of the camera. When the reference object B is the first reference object and the reference object A is the second reference object, the first scaling factor is actually the ratio of the distance between the reference object B and the aperture of the camera to the distance between the workpiece to be gripped and the aperture of the camera. At this time, the first scaling factor is substituted into the formula (19-1) or (19-2) above to calculate T, which is actually the conversion relationship between the homography matrix determined in the scenario where the reference object is the reference object B and the target homography matrix. Then, according to the subsequent step S34, the target homography matrix can be determined based on the homography matrix determined in the scenario where the reference object is the reference object B and the first scaling factor (i.e., the ratio of the distance between the reference object B and the aperture of the camera to the distance between the workpiece to be gripped and the aperture of the camera).

[0190] Below, we will explain the principle behind the validity of formula (19-1) in this embodiment, using formulas (20)-(22).

[0191] Understandably, the first, second, and third homography matrices represent transformations between the pixel coordinate system and the physical coordinate system of the motion mechanism. The difference lies in the distance between the corresponding objects (such as the first reference object, the second reference object, and the workpiece to be gripped and placed) and the camera aperture. Therefore, the first, second, and third homography matrices can be converted to each other. The conversion relationship is related to the distance between the corresponding object and the camera aperture. The following explanation focuses on the conversion relationship between the first and second homography matrices.

[0192] Based on the above Figure 2 As shown in the diagram, the similar triangle relationship between the camera coordinate system and the image plane coordinate system indicates that when the distance between the reference object and the camera aperture (denoted as T) is... zWhen the distance changes from the first distance to the second distance, the following equation (20) can be obtained.

[0193] (20)

[0194] in, This can be understood as the ratio of the first distance to the second distance. Let it be denoted as k.

[0195] The above formula (20) can be transformed into the form shown in the following formula (21).

[0196] (twenty one)

[0197] in, , is a 2×2 identity matrix.

[0198] Therefore, the transformation relationship between the first homography matrix and the second homography matrix can be expressed as shown in the following formula (22).

[0199] (twenty two)

[0200] From the above formula (22), it can be seen that, ,and The transformation relationship between the first and second homography matrices is given, where k is the ratio of the first distance to the second distance. Based on this ratio and the coordinates c of the principal point in the image, the transformation can be obtained. Therefore, based on the relationship between the first homography matrix, the second homography matrix, and the target homography matrix, the transformation relationship between the first homography matrix and the target homography matrix (i.e., the first transformation relationship) can be obtained by solving the ratio of the first distance to the distance between the workpiece to be gripped and the aperture of the camera, and the coordinates of the principal point of the image. That is, the above formula (19-1) holds.

[0201] In step S34 above, the first transformation relationship is the transformation relationship between the first homography matrix and the target homography matrix. Given the first homography matrix and the first transformation relationship, the electronic device can solve for the target homography matrix based on these relationships. After obtaining the target homography matrix, the electronic device can control the robot to precisely grasp the workpiece according to the target homography matrix.

[0202] Based on the above Figure 3As shown, the distance between the first reference object and the camera aperture, and the distance between the workpiece to be gripped and the camera aperture, can be obtained by measurement. However, in practical applications, the center position of the camera aperture is not easy to determine. Therefore, the distance between the camera aperture and the first reference object and the workpiece to be gripped is difficult to measure accurately. Thus, the first scale factor determined by directly measuring the distance between the first reference object and the camera aperture, and the distance between the workpiece to be gripped and the camera aperture, as shown in step S32 above, is not accurate. Consequently, the target homography matrix calculated based on the first scale factor is also inaccurate.

[0203] To address the issue of inaccurate calculation of the first scaling factor, embodiments of this application also provide a method for calculating the first scaling factor. In this method, the electronic device acquires a second homography matrix. (See [link to relevant documentation]). Figure 4 This may include the following steps:

[0204] Step S41: Based on the first homography matrix and the second homography matrix, determine the transformation relationship between the first homography matrix and the second homography matrix, and use it as the second transformation relationship;

[0205] Step S42: Based on the second transformation relationship, determine the ratio of the first distance to the second distance as the second scaling factor;

[0206] Step S43: Determine the first scale factor based on the second scale factor, the height difference between the workpiece to be gripped and the first reference object, and the height difference between the first reference object and the second reference object.

[0207] In the technical solution provided in this application embodiment, the transformation relationship between the first homography matrix and the second homography matrix can be calculated based on the first homography matrix and the second homography matrix, i.e., the second transformation relationship; based on the second transformation relationship and the principal point of the image, the second scaling factor can be determined; then, the height difference between the workpiece to be gripped and the first reference object is calculated, and the first scaling factor can be calculated based on the second scaling factor and the height difference. This avoids measuring the distances of the first reference object, the second reference object, and the workpiece to be gripped from the camera aperture, i.e., it avoids the error caused by the difficulty in determining the center point of the camera aperture. Therefore, solving the target homography matrix based on the first scaling factor can improve the accuracy of the target homography matrix.

[0208] As is understandable, in this article, the height difference between two objects refers to the difference in height of the plane containing the feature points to be photographed and identified on the two objects; the height of an object refers to the distance between the feature points to be photographed and identified on the object and the placement plane in the object's normal placement state; the normal placement state refers to the placement state that the workpiece should be in when the motion mechanism grasps the object. Here, the object refers to the first reference object, the second reference object, and the workpiece to be grasped and placed.

[0209] In step S41 above, the second transformation relationship is the transformation relationship between the first homography matrix and the second homography matrix. The second transformation relationship can be directly obtained from the first homography matrix and the second homography matrix obtained in step S31 above.

[0210] Specifically, in some embodiments, the electronic device can calculate the inverse matrix of either the first homography matrix or the second homography matrix, and calculate the product of the inverse matrix and the other as a second transformation relationship.

[0211] For example, in some embodiments, the electronic device can calculate the inverse of the first homography matrix and the product of the inverse of the first homography matrix and the second homography matrix, using the product of the inverse of the first homography matrix and the second homography matrix as the second transformation relation. Specifically, this can be calculated using the following formula (23-1).

[0212] (23-1)

[0213] Where T' is the second transformation relation; M Base1 M is the first homography matrix; Base2 This is the second homography matrix.

[0214] Alternatively, in other embodiments, the electronic device can calculate the inverse of the second homography matrix and the product of the inverse of the second homography matrix and the first homography matrix, using the product of the inverse of the second homography matrix and the first homography matrix as the second transformation relation. Specifically, this can be calculated using the following formula (23-2).

[0215] (23-2)

[0216] Where T' represents the second transformation relation; M Base1 M is the first homography matrix; Base2 This is the second homography matrix.

[0217] For ease of description, the following content will be explained using T' calculated by formula (23-1) as an example. If you need to perform the subsequent steps using T' calculated by formula (23-2), then replace T' calculated by formula (23-1) with the inverse matrix of T' calculated by formula (23-2) in the following content.

[0218] In step S42 above, the second scaling factor is the ratio of the distance between the first reference object and the camera aperture (i.e., the first distance) to the distance between the second reference object and the camera aperture (i.e., the second distance). Given the second transformation relationship and the coordinates of the principal point of the image, the second scaling factor can be obtained by referring to the above formula (22).

[0219] In some embodiments, when the second transformation relationship is represented in the form of a matrix, the second scaling factor is positively correlated with the main diagonal element of the 2×2 matrix in the upper left corner of the second transformation relationship, and negatively correlated with the secondary diagonal element of the 2×2 matrix (i.e., the two elements in the 2×2 block matrix along the diagonal from the lower left corner to the upper right corner).

[0220] Specifically, the second scaling factor of the electronic device can be solved using the following formula (24).

[0221] (twenty four)

[0222] Where T' is the second transformation relationship; k is the second scaling factor; c is the coordinates of the principal point of the image; and E is the identity matrix.

[0223] In some embodiments, the second transformation relationship obtained in step S41 above can be represented in the form of a matrix. In this way, the electronic device can calculate the mean of the main diagonal of the 2×2 block matrix in the upper left corner of the matrix used to represent the second transformation relationship (i.e., the mean of the two elements in the 2×2 block matrix from the upper left corner to the lower right corner) as the second scaling factor; or, the electronic device can calculate the square root of the determinant of the 2×2 block matrix in the upper left corner of the matrix used to represent the second transformation relationship as the second scaling factor.

[0224] For example, the matrix used to represent the second transformation relation is denoted as T 11 That is, the 2×2 block matrix in the top left corner of the matrix, specifically, If the mean of the main diagonal elements of the block matrix is ​​used as the second scaling factor, then the second scaling factor... Alternatively, if the square root of the determinant of the partitioned matrix is ​​used as the second scaling factor, then the second scaling factor... Alternatively, the main diagonal elements of the block matrix can be taken as the second scaling factor, i.e., k can be wx and wy. This application does not restrict the way k can be taken.

[0225] In step S43 above, the electronic device determines the first scaling factor based on the second scaling factor, the height difference between the workpiece to be gripped and the first reference object, and the height difference between the first reference object and the second reference object.

[0226] In some embodiments, the electronic device can calculate the ratio of the height difference between the workpiece to be gripped and the first reference object to the height difference between the first reference object and the second reference object, as the depth difference ratio; then, based on the second scaling factor and the depth difference ratio, the electronic device can determine the first scaling factor; wherein, the first scaling factor is positively correlated with the second scaling factor and positively correlated with the depth difference ratio.

[0227] Specifically, in some embodiments, the electronic device can solve for the first scaling factor using the following formula (25-1).

[0228] (25-1)

[0229] Where k' is the first scaling factor; k is the second scaling factor; The height difference between the workpiece to be grasped and the first reference object; The height difference between the first reference object and the second reference object.

[0230] It is understood that the above formula (25-1) is only one possible calculation formula. In some embodiments, the first scaling factor can also be calculated by other formulas obtained by simple transformation of the above formula (25-1). For example, see the following formula (25-2).

[0231] (25-2)

[0232] Where k' is the first scaling factor; k is the second scaling factor; The height difference between the workpiece to be grasped and the first reference object; The height difference between the first reference object and the second reference object.

[0233] Understandably, to avoid measuring the distance between the object and the camera aperture, the height difference between the first reference object and the second reference object in this embodiment is obtained based on the heights of the first and second reference objects. Furthermore, the height difference between the first reference object and the workpiece to be gripped is also obtained based on the heights of the first reference object and the workpiece to be gripped. The height differences between the first reference object, the second reference object, and the workpiece to be gripped can be obtained from the workpiece drawing, or determined using measuring tools such as a ruler, tape measure, height gauge, or micrometer. Alternatively, a height measuring sensor can be used to improve measurement accuracy. The specific measuring tool used can be determined based on the positioning and guidance accuracy.

[0234] The principle behind the validity of the above formula (25-1) will be explained below using formulas (26)-(29).

[0235] Let H be the distance from the feature point plane of the first reference object to the camera aperture in the z-direction. Mark1 The distance from the feature point plane of the second reference object to the camera aperture in the z-direction is H. Mark2 Then we can obtain the following formulas (26)-(27).

[0236] Z c_Mark1 = H Mark1 (26)

[0237] Z c_Mark2 = H Mark2 (27)

[0238] The height difference between the first reference object and the second reference object is denoted as . Then we have △H Mark = H Mark2 -H Mark1 Then the second scaling factor k can be expressed in the form shown in the following formula (28).

[0239] (28)

[0240] Let H be the distance in the z-direction from the feature point plane of the workpiece to be gripped and placed to the camera aperture. Cur Then Z c_Cur =H Cur The height difference between the first reference object and the workpiece to be gripped and placed is denoted as △H. Cur Then we have △H Cur = H Cur -H Mark1 At this point, the first scaling factor It can be expressed in the form shown in the following formula (29),

[0241] (29)

[0242] Therefore, the above formula (25-1) holds true.

[0243] In some embodiments, based on this application Figure 4 The example shown provides a method to determine the first scaling factor. Then, following step S33 above, the first transformation relationship is determined based on the first scaling factor and the coordinates of the principal point of the image. In step S34 above, each element in the target homography matrix can be calculated in the following way:

[0244] Let the target homography matrix be denoted as

[0245] Then we have:

[0246]

[0247]

[0248]

[0249]

[0250]

[0251]

[0252]

[0253]

[0254]

[0255] Where k' is based on Figure 4 The first scaling factor obtained as shown; h 11 ~h 23 These are elements in the first homography matrix.

[0256] In some other embodiments, when the main diagonal elements of the 2×2 block matrix are taken as the second scaling factor (i.e., k takes wx and wy) according to the content shown in step S42 above, the calculation formula of the first scaling factor k' (i.e., k'x' and k'y' below) can be expressed as shown in the following formulas (30) and (31) (it is understood that if formula (25-2) is used here instead of formula (25-1), then the constant 1 in the following formulas (30) and (31) can be replaced with 0.9):

[0257] (30)

[0258] (31)

[0259] In this case, M Cur_inv The formula for calculating each element in the matrix is:

[0260]

[0261]

[0262]

[0263]

[0264]

[0265]

[0266]

[0267]

[0268]

[0269] Below, through the following Figure 5 and Figure 6 The method of obtaining the first homography matrix and the second homography matrix in step S31 of this application will be described below.

[0270] See Figure 5 , Figure 5 This is a schematic flowchart illustrating the process of obtaining a first homography matrix or a second homography matrix according to an embodiment of this application. When a reference object is carried on a moving mechanism, the homography matrix between the pixel coordinate system in any scene and the physical coordinate system of the moving mechanism can be obtained through the following... Figure 5 The method shown can be obtained by including the following steps:

[0271] Step S51: Control the motion mechanism to carry the reference object to the first photographing position, and control the camera to take a picture of the reference object to obtain an image of the reference object at the first photographing position, which is used as the first image;

[0272] The electronic device can control the motion mechanism to carry the reference object to the first photographing position, and control the camera to take a picture of the reference object, thereby obtaining an image of the reference object at the first position, i.e., the first image.

[0273] Understandably, to facilitate obtaining the homography matrix, a new shooting position will be determined as the first shooting position. However, for the first shooting position moved to for the first time (referred to as the "initial first shooting position"), it is only necessary to ensure that the reference object is within the camera's field of view at the initial shooting position; the distance between the initial first shooting position and the camera's aperture is not important. The first shooting positions determined in subsequent processes (referred to as the "new first shooting positions") must ensure that the reference object is within the camera's field of view, and also ensure that the distance between each new first shooting position and the camera's aperture is the same as the distance between the initial first shooting position and the camera's aperture. In this embodiment, the distances refer to distances in a preset direction, which is the z-axis orientation of the camera coordinate system. This ensures that the distances between each first shooting position and the camera's aperture are the same in the preset direction. This can be further explained in step S53 below.

[0274] In addition, the reference objects in the embodiments of this application can be a checkerboard calibration plate, a circular calibration plate, etc.

[0275] Step S52: Determine the pixel coordinates of at least one feature point of the reference object based on the first image, and use them as the pixel coordinates of the feature point corresponding to the first image.

[0276] The electronic device can determine the pixel coordinates of at least one feature point of the reference object based on the first image, and use these coordinates as the pixel coordinates of the feature point corresponding to the first image. It is understood that the feature points obtained by the electronic device from each of the first images are the same. For example, if there are two first images, image 1 and image 2, and the pixel coordinates of feature point A and feature point B of reference object 1 are obtained from image 1, then the pixel coordinates of the feature point obtained from image 2 should also be the pixel coordinates of feature point A and feature point B of reference object 1.

[0277] Understandably, to improve the accuracy of the obtained feature point pixel coordinates, in this embodiment, the first image can be first distorted to eliminate radial and tangential distortion of the camera lens, and then the pixel coordinates (i.e., feature point pixel coordinates) of multiple feature points of the reference object can be determined based on the distorted first image. The electronic device can perform distortion correction on the first image using polar coordinate transformation, polynomial fitting, or a correction method based on a parametric model.

[0278] Step S53: Determine whether the number of times the motion mechanism carries the reference object to the first photographing position has reached the preset number. If the preset number has not been reached, determine a new photographing position and use the new photographing position as the first photographing position. Repeat the above steps of obtaining the feature point pixel coordinates corresponding to the first image.

[0279] The electronic device can determine whether the number of times the motion mechanism carrying the reference object moves to the first photographing position has reached a preset number. If the preset number has not been reached, the photographing position can be redefined and the redefined photographing position can be used as the first photographing position. The above steps S51-S53 are repeated until the number of times the motion mechanism carrying the reference object moves to the first photographing position reaches the preset number.

[0280] Understandably, in this embodiment, the preset number of times refers to the number of times required to obtain a homography matrix, such as the number of times required to obtain the first homography matrix, or the number of times required to obtain the second homography matrix. Therefore, within the preset number of times, when determining a new shooting position, the distance from each determined new shooting position to the camera aperture must be the same in a preset direction as the distance from the first shooting position to the camera aperture in the first instance; that is, the distance between each first shooting position and the camera aperture in the preset direction is the same. However, when the preset number of times has been reached, and the homography matrix of the current scene can be determined, and it is necessary to obtain the homography matrix of another scene (e.g., when obtaining the second homography matrix after obtaining the first homography matrix), the distance between the determined new shooting position and the camera aperture in the preset direction must be different from the distance between the previous first shooting positions and the camera aperture in the preset direction.

[0281] Step S54: If the preset number of times has been reached, determine the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism based on the pixel coordinates of the feature points corresponding to each first image and the first image capture position.

[0282] When the preset number of times is reached, the electronic device can determine the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism based on the pixel coordinates of the feature points corresponding to each first image and the first shooting position.

[0283] See Figure 6 , Figure 6 This is a schematic diagram illustrating a second process for obtaining a first or second homography matrix according to an embodiment of this application. When the camera is mounted on a motion mechanism, the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism in any scene can be obtained through the following... Figure 6 The method shown can be obtained by including the following steps:

[0284] Step S61: Control the motion mechanism to move the camera to the second shooting position, and control the camera to take a picture of the reference object to obtain an image of the reference object at the second shooting position, which is used as the second image;

[0285] In this embodiment, the camera is located on the motion mechanism. Therefore, when the motion mechanism moves to the second photographing position, the camera is also located at the second photographing position. The electronic device can control the motion mechanism to carry the camera to the second photographing position and control the camera to photograph the reference object, thereby obtaining an image of the reference object at the second position, i.e., the second image.

[0286] To facilitate the acquisition of the homography matrix, a new shooting position will be determined as the second shooting position. However, for the first second shooting position (referred to as the "initial second shooting position"), it is only necessary to ensure that the camera can capture the reference object when it is located at the initial second shooting position; the distance between the initial second shooting position and the camera's aperture is not important. The second shooting positions determined in subsequent processes (referred to as the "new second shooting positions") must ensure that the reference object is within the camera's field of view, and also ensure that the distance between each new second shooting position and the camera's aperture is the same as the distance between the initial second shooting position and the camera's aperture. In this embodiment, the distances refer to distances in a preset direction, which is the z-axis orientation of the camera coordinate system. This ensures that the distances between each first shooting position and the camera's aperture are the same in the preset direction. See step S63 below for details.

[0287] In addition, the reference objects in the embodiments of this application can be a checkerboard calibration plate, a dot calibration plate, etc.

[0288] Step S62: Determine the pixel coordinates of at least one feature point of the reference object based on the second image, and use them as the pixel coordinates of the feature point corresponding to the second image;

[0289] The electronic device can determine the pixel coordinates of at least one feature point of the reference object based on the second image, and use these coordinates as the pixel coordinates of the feature point corresponding to the second image. Similarly to step S52 above, the feature points obtained by the electronic device from each second image are the same; and, in some embodiments, to improve the accuracy of the pixel coordinates of multiple feature points obtained from the second image, distortion correction can be performed on the second image before determining the pixel coordinates of the feature points.

[0290] Step S63: Determine whether the number of times the motion mechanism carries the camera to the second shooting position has reached the preset number. If the preset number has not been reached, determine a new shooting position and use the new shooting position as the second shooting position. Repeat the above steps to obtain the feature point pixel coordinates corresponding to the second image.

[0291] The electronic device can determine whether the number of times the motion mechanism carrying the camera moves to the second shooting position has reached a preset number. If the preset number has not been reached, the shooting position can be redefined and the new position can be used as the second shooting position. The above steps S61-S63 are repeated until the number of times the motion mechanism carrying the camera moves to the second shooting position reaches the preset number.

[0292] Understandably, in this embodiment, the preset number of times refers to the number of times required to obtain a homography matrix. Therefore, within the preset number of times, the distances from each new photographing position to the reference object must be kept the same in the preset direction as the distances from the first second photographing position to the reference object. That is, the distances from each second photographing position to the reference object are the same in the preset direction. However, when the preset number of times is reached, enough to determine the homography matrix of the current scene, and it is necessary to obtain the homography matrix of another scene (e.g., when the first homography matrix has been obtained and the second homography matrix needs to be obtained), the distances from the new photographing positions to the reference object are different from the distances from each of the previous second photographing positions to the reference object in the preset direction.

[0293] Step S64: If the preset number of times has been reached, determine the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism based on the pixel coordinates of the feature points corresponding to each second image and the second shooting position.

[0294] When the preset number of times is reached, the electronic device can determine the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism based on the pixel coordinates of the feature points corresponding to each second image and each second shooting position.

[0295] Understandably, the target homography matrix in this paper is obtained by solving for the first and second homography matrices. Both the first and second homography matrices are determined based on the pixel coordinates of the feature points of the reference objects acquired by the camera. Therefore, the descriptions of height differences and distances mentioned in this paper, such as the height difference between objects (e.g., the first reference object, the second reference object, and the workpiece to be gripped), and the distance between the object and the camera aperture, can actually be understood as the height difference between the planes containing the feature points of the objects, and the distance between the planes containing the feature points of the objects and the camera aperture. For example, the height difference between the first and second reference objects can be understood as the height difference between the planes containing the feature points of the first and second reference objects; similarly, the distance between the workpiece to be gripped and the camera aperture can be understood as the distance between the plane containing the feature points of the workpiece and the camera aperture.

[0296] Below, taking the example of a reference object carried on a moving mechanism and a camera fixedly mounted outside the moving mechanism, we will combine... Figures 7-11 The hand-eye calibration method provided in the embodiments of this application will be further described.

[0297] See Figure 7 , Figure 7 This is a schematic diagram of a second flowchart of the hand-eye calibration method provided in the embodiments of this application. Figure 7 Steps S71-S75 belong to the offline calibration stage, while steps S76-S77 belong to the online operation stage.

[0298] Step S71: The motion mechanism reaches the predetermined shooting position, moves and takes pictures with the calibration reference object 1, and collects several sets of image coordinates and physical coordinate data (no less than two sets).

[0299] In this embodiment, the predetermined shooting position can be understood as the aforementioned first shooting position, the several sets of image coordinates can be understood as the feature point pixel coordinates corresponding to each first image, and the physical coordinate data can be understood as the coordinates of each first shooting position. See the description in step S51 above for details.

[0300] Step S72: Solve for the homography matrix 1M from the pixel coordinate system to the physical coordinate system of the motion mechanism. Base1 For details, please refer to the descriptions in steps S52-S54 above.

[0301] Step S73: The motion mechanism reaches the predetermined shooting position, moves and takes pictures with the calibration reference object 2, and collects several sets of image coordinates and physical coordinate data (no less than two sets).

[0302] In this embodiment, the predetermined shooting position can be understood as the aforementioned second shooting position, the several sets of image coordinates can be understood as the feature point pixel coordinates corresponding to each second image, and the physical coordinate data can be understood as the coordinates of each second shooting position. See the description in step S51 above for details. Step S73 is the same as step S71, except that reference object 1 is changed to reference object 2.

[0303] Step S74: Solve for the homography matrix 2M from the pixel coordinate system to the physical coordinate system of the motion mechanism. Base2 .

[0304] Step S74 is the same as step S72, except that reference object 1 is changed to reference object 2.

[0305] Step S75: Obtain the height difference between the feature points of calibration reference 2 and the feature points of calibration reference 1. .

[0306] In this embodiment of the application, the height difference between the feature point of the calibration reference 2 and the feature point of the calibration reference 1 can be understood as the height difference between the first reference and the second reference in step S43 above, which can also be called the reference depth difference. Figure 8 As shown, Figure 8 There are reference object 1 and reference object 2. Camera 81 is fixedly mounted on motion mechanism 82. Motion mechanism 82 can carry camera 81 to move. The height difference between reference object 1 and reference object 2 is the reference depth difference (e.g., ...). Figure 8 (The value of the height referred to in 3).

[0307] Step S76: Obtain the height difference between the feature point of the workpiece to be gripped and the feature point of the calibration reference 1. .

[0308] In this embodiment of the application, the height difference between the feature point of the workpiece to be gripped and the feature point of the reference object 1 can be understood as the height difference between the workpiece to be gripped and the first reference object in step S43 above. This height difference can also be called the running depth difference. Figure 9 As shown, Figure 9 There is a reference object 1 and a workpiece 4 to be gripped and placed. The camera 81 is fixedly mounted on the motion mechanism 82. The height difference between the workpiece 4 to be gripped and placed and the reference object 1 is the running depth difference (e.g., ...). Figure 9 The value of the height indicated by 5 in the middle), that is .

[0309] Step S77: Using the reference homography matrix 1M Base1 and the reference homography matrix 2M Base2Solve for the coordinates of the principal point of the image and the reference height scaling factor (i.e., the second scaling factor). The method for solving the coordinates of the principal point of the image can be found in step S31 above, and the method for solving the reference height scaling factor can be found in step S42 above.

[0310] Then use the reference depth difference Difference between running depth and running depth Solve for the running height scaling factor (i.e., the first scaling factor), as described in step S43 above. Finally, solve for the running homography matrix M from the pixel coordinate system to the physical coordinate system of the motion mechanism. cur .

[0311] In the technical solution provided in this application embodiment, for different workpieces to be gripped and placed, a first homography matrix, a second homography matrix, and a reference depth difference can be determined before each determination of the target homography matrix. The target homography matrix is ​​determined based on the first homography matrix, the second homography matrix, and the reference depth difference determined in real time, thereby improving the accuracy of the target homography matrix. Here, different workpieces to be gripped and placed can be understood as different items.

[0312] To illustrate with an example, suppose a factory produces two items, A and B. A robotic arm (i.e., a motion mechanism) and a camera are installed on a conveyor belt. The robotic arm is used to grab items and place them in the next process area when they pass its position. To achieve accurate positioning of item A or item B by the robotic arm, when the conveyor belt is used to transport item A, the electronic equipment can follow the procedures described above in this application. Figure 5 or Figure 6 The first and second homography matrices are determined as shown, and the difference in reference depth is recorded. This is then combined with the above-mentioned provisions of this application. Figure 3 and Figure 4 The method shown obtains the target homography matrix based on the first and second homography matrices, which is the homography matrix between the pixel coordinate system of item A and the physical coordinate system of the motion mechanism. Then, based on the homography matrix corresponding to item A, precise grasping of item A is achieved. When the conveyor belt finishes transporting item A and begins transporting item B, the electronic device can again follow the procedures described above. Figure 5 or Figure 6 The first and second homography matrices are determined as shown, and the difference in reference depth is recorded. This is then combined with the above-mentioned provisions of this application. Figure 3 and Figure 4The method shown is based on solving the first and second homography matrices to obtain the target homography matrix, which is the homography matrix between the pixel coordinate system of item B and the physical coordinate system of the motion mechanism. Then, based on the homography matrix corresponding to item B, precise grasping of item B is achieved. It should be noted that in this example, regardless of whether the installation positions of the robotic arm and camera change when the item being transported on the conveyor belt changes from item A to item B, the first and second homography matrices are redefined, and the reference depth difference is recorded. That is, for different workpieces to be grasped, regardless of whether the installation positions of the robotic arm and camera change, the first and second homography matrices, as well as the reference depth difference, are redefined once.

[0313] Alternatively, in some embodiments, for different workpieces to be gripped and placed, when the installation positions of the robotic arm and camera do not change, it is not necessary to redetermine the first homography matrix, the second homography matrix, and the reference depth difference. The target homography matrix can be determined based on the previously determined first homography matrix, the second homography matrix, and the reference depth difference. Only when the installation positions of the robotic arm and camera change, the first homography matrix and the second homography matrix are redetermined, and the target homography matrix is ​​determined based on the new first homography matrix, the second homography matrix, and the reference depth difference.

[0314] Continuing with the previous example, when using the conveyor belt to transport item A, the electronic device determines the first homography matrix and the second homography matrix, and records the reference depth difference. Based on the first homography matrix, the second homography matrix, and the recorded reference depth difference, the target homography matrix corresponding to item A is determined, and the precise grasping of item A is achieved based on the homography matrix corresponding to item A. Subsequently, when using the conveyor belt to transport item B, if the installation positions of the robotic arm and camera remain unchanged (i.e., the installation positions are the same as when transporting item A), then it is only necessary to determine the operating depth difference corresponding to item B. Based on the first homography matrix, the second homography matrix, and the reference depth difference determined when transporting item A, the target homography matrix corresponding to item B can be determined, thus enabling precise grasping of item B based on the homography matrix corresponding to item B. If the installation positions of the robotic arm and camera change (i.e., the installation positions are different from when transporting item A), then the first homography matrix and the second homography matrix need to be re-determined, and the reference depth difference recorded. Then, based on the newly determined first homography matrix, the second homography matrix, and the reference depth difference, the target homography matrix corresponding to item B is determined, and precise grasping of item B is achieved based on this target homography matrix.

[0315] In this embodiment, when the installation positions of the robotic arm and the camera remain unchanged, regardless of any changes to the workpiece to be gripped or placed, it is only necessary to determine the difference in operating depth corresponding to the actual workpiece to be gripped or placed. Then, the target homography matrix corresponding to the actual workpiece to be gripped or placed can be determined by combining the previously determined first homography matrix, second homography matrix, and reference depth difference. Based on this, the solution provided by this embodiment has a wide range of applications, high accuracy, and simple operation.

[0316] See Figure 10 , Figure 10 A schematic diagram of a software interface for the hand-eye calibration method provided in this application embodiment is shown below. Figure 10 As shown, from top to bottom of the interface, the top displays the words "9. Multi-planar hand-eye calibration based on image principal points"; below this text are two options: "Basic parameters" and "Result display". Currently... Figure 10 The page shown should be the one indicated by the "Basic Parameters" option. This page displays "Category Input," and below it are four lines of text: "Baseline Depth Difference," "Running Depth Difference," "Baseline Calibration Matrix 1," and "Baseline Calibration Matrix 2." Each of these lines is followed by a long box, and to the right of each long box is a link symbol 101 (which is understandable). Figure 10 (Only one link symbol is marked in the text). Currently, the long box corresponding to "Reference Depth Difference" displays "100", the long box corresponding to "Running Depth Difference" displays "200", the long box corresponding to "Reference Calibration Matrix 1" displays "8 1810mm. Calibration Matrix []", and the long box corresponding to "Reference Calibration Matrix 2" displays "12 1910mm. Calibration Matrix []". There are three options at the bottom of the interface: "Continuous Execution", "Execute", and "OK".

[0317] Understandably, the above Figure 10 In the interface shown, the long boxes following "Reference Depth Difference," "Operating Depth Difference," "Reference Calibration Matrix 1," and "Reference Calibration Matrix 2" correspond to their respective values. For example, the value in the long box following "Reference Calibration Matrix 1" is the determined first homography matrix. Users can directly input the corresponding values ​​into the long boxes, or they can click the link symbol 101 on the right side of the long box. The electronic device will respond to the user's click on the link symbol 101 and import the corresponding values ​​into the corresponding long boxes.

[0318] In some embodiments, in response to a user clicking the "Continuously Execute" button, the electronic device can continuously acquire data from each long frame and determine the image principal point coordinates and target homography matrix corresponding to each set of long frame data. Alternatively, in response to a user clicking the "Execute" button, the electronic device can determine the image principal point coordinates and target homography matrix corresponding to a set of data input by the user in a long frame; or, the electronic device can acquire data from a set of long frames through link symbol 101 and determine the image principal point coordinates and target homography matrix corresponding to that set of data.

[0319] See Figure 11 , Figure 11 This is a schematic diagram of a software interface for a hand-eye calibration method provided in an embodiment of this application. Figure 11 It can also be understood as being with Figure 10 The diagram shows one type of output interface corresponding to the input interface shown. Figure 11 The interface shown contains five rows of data. The first row is “ˇMultiplaneHomoMat1”; the second row is “Module Status 1”; and the third row is: CurCalibMatrix -0.1950009;-0.006203677;1567.321;-0.005978366;0.1952618;-294.1291;0;0;1;

[0320] The fourth row of data is “Cx 2570.27”; the fifth row of data is “Cy 2617.838”.

[0321] In summary, the technical solutions provided in this application have the following beneficial effects:

[0322] a. This application does not restrict the principal point of the image to be the center point of the pixel coordinate system, the calibration model is closer to the actual model, and the multi-plane hand-eye calibration accuracy is higher;

[0323] b. This application is compatible with scenarios where the optical axis of the lens is not perpendicular to the plane of the workpiece feature point, thus having a wider range of applications;

[0324] c. This application does not require measuring the height distance from the camera aperture to the feature point of the workpiece to be gripped and placed. The reference depth difference and the running depth difference can be obtained from the drawings. The depth difference measurement steps are simpler, which effectively reduces the estimation error of multi-plane hand-eye calibration and improves the accuracy of hand-eye calibration.

[0325] Corresponding to the above-described hand-eye calibration method, this application also provides a hand-eye calibration device, such as... Figure 12 As shown, the device includes:

[0326] The homography matrix acquisition module 121 is used to acquire a first homography matrix and the image principal point coordinates; wherein, the image principal point coordinates are the coordinates of the intersection point of the camera's optical axis and the pixel plane calculated based on the first homography matrix and the second homography matrix; the first homography matrix is ​​the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism determined by multiple images obtained by the camera taking multiple shots of the first reference object in the first scene; the second homography matrix is ​​the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism determined by multiple images obtained by the camera taking multiple shots of the second reference object in the second scene; the first reference object and the second reference object are carried on the motion mechanism, or the camera is carried on the motion mechanism; the first distance and the second distance are different, the first distance is the distance between the first reference object and the camera's aperture, and the second distance is the distance between the second reference object and the camera's aperture;

[0327] The scaling factor calculation module 122 is used to calculate the ratio of the distance between the first reference object and the camera aperture to the distance between the workpiece to be gripped and the camera aperture, as the first scaling factor.

[0328] The transformation relationship determination module 123 is used to determine the transformation relationship between the first homography matrix and the target homography matrix based on the first scaling factor and the image principal point coordinates, as the first transformation relationship; wherein, the target homography matrix is ​​the homography matrix between the pixel coordinate system corresponding to the workpiece to be gripped and placed and the physical coordinate system of the motion mechanism;

[0329] The homography matrix determination module 124 is used to determine the target homography matrix based on the first transformation relationship and the first homography matrix.

[0330] In the technical solution provided in this application embodiment, a first homography matrix and image principal point coordinates are obtained. The image principal point coordinates are the coordinates of the intersection point of the optical axis and the pixel plane, calculated based on the first and second homography matrices. Specifically, the first homography matrix is ​​the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism, determined based on multiple images of the first reference object captured by the camera in the first scene. The second homography matrix is ​​the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism, determined based on multiple images of the second reference object captured by the camera in the second scene. The first and second reference objects are carried by the motion mechanism, or the camera is carried by the motion mechanism. The difference between the first and second scenes lies in the distance between the first reference object and the second reference object in the first scene. The distance between the camera apertures (i.e., the first distance) is different from the distance between the second reference object and the camera aperture in the second scene (i.e., the second distance). Since the principal point coordinates of the image are the coordinates of the intersection of the camera's optical axis and the pixel plane, the principal point coordinates of the image can be obtained based on the first homography matrix and the second homography matrix. Then, the ratio of the first distance to the distance between the workpiece to be gripped and the camera aperture (i.e., the first scaling factor) is calculated. Based on the first scaling factor and the principal point coordinates of the image, the transformation relationship between the first homography matrix and the target homography matrix (i.e., the first transformation relationship) can be obtained. Then, based on the first transformation relationship and the first homography matrix, the target homography matrix can be determined, that is, the homography matrix between the pixel coordinate system corresponding to the workpiece to be gripped and the physical coordinate system of the motion mechanism.

[0331] Understandably, in this application, the target homography matrix is ​​solved based on the first homography matrix and the first transformation relation. The image principal point coordinates used to solve the first transformation relation are the actual image principal point coordinates calculated based on the first homography matrix and the second homography matrix, rather than using the theoretical pixel coordinate system center point as the image principal point coordinates. Therefore, the accuracy of determining the target homography matrix based on the hand-eye calibration method shown in this application is relatively high.

[0332] In some embodiments, the homography matrix acquisition module 121 is further configured to acquire the second homography matrix;

[0333] The scaling factor calculation module 122 includes:

[0334] The first determining unit is used to determine the transformation relationship between the first homography matrix and the second homography matrix based on the first homography matrix and the second homography matrix, as the second transformation relationship;

[0335] The second determining unit is used to determine the ratio of the first distance to the second distance based on the second transformation relationship, as the second scaling factor;

[0336] The third determining unit is used to determine the first scaling factor based on the second scaling factor, the height difference between the workpiece to be gripped and the first reference object, and the height difference between the first reference object and the second reference object.

[0337] In some embodiments, the third determining unit is specifically used to calculate the ratio of the height difference between the workpiece to be gripped and the first reference object and the height difference between the first reference object and the second reference object, as the depth difference ratio;

[0338] Based on the second scaling factor and the depth difference ratio, the first scaling factor is determined; wherein the first scaling factor is positively correlated with the second scaling factor and positively correlated with the depth difference ratio.

[0339] In some embodiments, the first transformation relationship is represented in matrix form, and the linear component in the first transformation relationship is positively correlated with the first scaling factor; the rotation component in the first transformation relationship is negatively correlated with the first scaling factor and positively correlated with the value of the principal point coordinates of the image.

[0340] In some embodiments, the first determining unit is specifically used to calculate the inverse matrix of either the first homography matrix or the second homography matrix, and to calculate the product of the inverse matrix and the other as a second transformation relationship;

[0341] The second transformation relationship is represented in matrix form. The second scaling factor is positively correlated with the main diagonal element of the 2×2 matrix in the upper left corner of the second transformation relationship, and negatively correlated with the secondary diagonal element of the 2×2 matrix.

[0342] In some embodiments, the second transformation relationship is represented in matrix form, and the second determination of a single eye is further used for,

[0343] Calculate the mean of the main diagonals of the top-left 2×2 block matrix in the matrix representing the second transformation relationship, and use it as the second scaling factor; or,

[0344] Calculate the square root of the determinant of the top-left 2×2 block matrix in the matrix used to represent the second transformation relationship, and use it as the second scaling factor.

[0345] In some embodiments, when a reference object is carried on the motion mechanism, the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism in any scene is obtained in the following ways, including:

[0346] The motion control mechanism carries the reference object to the first photographing position, and controls the camera to photograph the reference object, obtaining an image of the reference object at the first photographing position, which is used as the first image;

[0347] The pixel coordinates of at least one feature point of the reference object are determined based on the first image, and used as the pixel coordinates of the feature point corresponding to the first image.

[0348] Determine whether the number of times the motion mechanism carries the reference object to the first photographing position has reached the preset number. If the preset number has not been reached, determine a new photographing position and use the new photographing position as the first photographing position. Repeat the above steps of obtaining the feature point pixel coordinates corresponding to the first image.

[0349] If the preset number of times has been reached, then the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism is determined based on the pixel coordinates of the feature points corresponding to each first image and the position of each first image capture.

[0350] Among them, the distance between each first shooting position and the camera aperture is the same in a preset direction, which is the z-axis orientation of the camera coordinate system.

[0351] In some embodiments, when the camera is mounted on a motion mechanism, the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism in any scene is obtained in the following ways, including:

[0352] The motion control mechanism carries the camera to the second shooting position and controls the camera to take a picture of the reference object, thereby obtaining an image of the reference object at the second shooting position, which is used as the second image;

[0353] The pixel coordinates of at least one feature point of the reference object are determined based on the second image, and used as the pixel coordinates of the feature point corresponding to the second image.

[0354] If the number of times the motion mechanism moves the camera to the second shooting position has reached a preset number, then a new shooting position is determined and used as the second shooting position. The above steps of obtaining the feature point pixel coordinates corresponding to the second image are repeated. If the preset number has been reached, then the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism is determined based on the feature point pixel coordinates corresponding to each second image and each second shooting position.

[0355] Among them, the distance between each second shooting position and the reference object is the same in a preset direction, which is the z-axis orientation of the camera coordinate system.

[0356] Corresponding to the above-described hand-eye calibration method, this application also provides a first motion mechanism system, such as... Figure 13 As shown (which is understandable), Figure 13 (The processor is not shown in the diagram). The first motion mechanism system includes a processor, a motion mechanism 131, and a camera 132, which is disposed outside the motion mechanism 131.

[0357] The motion mechanism 131 is used to carry the first reference object in a first scene and to carry the second reference object in a second scene.

[0358] The camera 132 is used to take multiple shots of the first reference object carried by the motion mechanism 131 in the first scene to obtain multiple third images; and to take multiple shots of the second reference object carried by the motion mechanism 131 in the second scene to obtain multiple fourth images; wherein the first distance and the second distance are different, the first distance is the distance between the first reference object and the aperture of the camera, and the second distance is the distance between the second reference object and the aperture of the camera.

[0359] The processor is used to acquire a first homography matrix and image principal point coordinates determined based on the first homography matrix and the second homography matrix; wherein the image principal point coordinates are the coordinates of the intersection of the camera's optical axis and the pixel plane; the first homography matrix is ​​a homography matrix of the pixel coordinate system and the physical coordinate system of the motion mechanism determined based on multiple third images; the second homography matrix is ​​a homography matrix of the pixel coordinate system and the physical coordinate system of the motion mechanism determined based on multiple fourth images.

[0360] The processor is also used to calculate the ratio of the first distance to the distance between the workpiece to be gripped and the aperture of the camera, as a first scaling factor; based on the first scaling factor and the principal point coordinates of the image, to determine the transformation relationship between the first homography matrix and the target homography matrix, as a first transformation relationship; based on the first transformation relationship and the first homography matrix, to determine the target homography matrix; wherein, the target homography matrix is ​​the homography matrix between the pixel coordinate system corresponding to the workpiece to be gripped and the physical coordinate system of the motion mechanism.

[0361] In the technical solution provided in this application embodiment, a first homography matrix and image principal point coordinates are obtained. The image principal point coordinates are the coordinates of the intersection point of the optical axis and the pixel plane, calculated based on the first and second homography matrices. Specifically, the first homography matrix is ​​the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism, determined by multiple images (i.e., the third image) obtained from multiple shots of the first reference object by the camera in the first scene. The second homography matrix is ​​the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism, determined by multiple images (i.e., the fourth image) obtained from multiple shots of the second reference object by the camera in the second scene. In the first scene, the first reference object is carried by the motion mechanism; in the second scene, the second reference object is carried by the motion mechanism. The difference between the first and second scenes lies in the first... The distance between the first reference object in the scene and the camera aperture (i.e., the first distance) is different from the distance between the second reference object in the second scene and the camera aperture (i.e., the second distance). Since the principal point coordinates of the image are the coordinates of the intersection of the camera's optical axis and the pixel plane, the principal point coordinates of the image can be obtained based on the first homography matrix and the second homography matrix. Then, the ratio of the first distance to the distance between the workpiece to be gripped and the camera aperture (i.e., the first scaling factor) is calculated. Based on the first scaling factor and the principal point coordinates of the image, the transformation relationship between the first homography matrix and the target homography matrix (i.e., the first transformation relationship) can be obtained. Then, based on the first transformation relationship and the first homography matrix, the target homography matrix can be determined, that is, the homography matrix between the pixel coordinate system corresponding to the workpiece to be gripped and the physical coordinate system of the motion mechanism.

[0362] Understandably, in this application, the target homography matrix is ​​solved based on the first homography matrix and the first transformation relation. The image principal point coordinates used to solve the first transformation relation are the actual image principal point coordinates calculated based on the first homography matrix and the second homography matrix, rather than using the theoretical pixel coordinate system center point as the image principal point coordinates. Therefore, the accuracy of determining the target homography matrix based on the hand-eye calibration method shown in this application is relatively high.

[0363] In some embodiments, the processor is further configured to acquire a second homography matrix;

[0364] The calculation of the ratio of the distance between the first reference object and the camera aperture to the distance between the workpiece to be gripped and the camera aperture, as the first scaling factor, includes:

[0365] Based on the first homography matrix and the second homography matrix, the transformation relationship between the first homography matrix and the second homography matrix is ​​determined, which is used as the second transformation relationship;

[0366] Based on the second transformation relationship, the ratio of the first distance to the second distance is determined as the second scaling factor;

[0367] The first scale factor is determined based on the second scale factor, the height difference between the workpiece to be gripped and the first reference object, and the height difference between the first reference object and the second reference object.

[0368] In some embodiments, determining the first scaling factor based on a second scaling factor, the height difference between the workpiece to be gripped and the first reference object, and the height difference between the first reference object and the second reference object includes:

[0369] Calculate the ratio of the height difference between the workpiece to be gripped and the first reference object to the height difference between the first reference object and the second reference object, and use it as the depth difference ratio;

[0370] Based on the second scaling factor and the depth difference ratio, the first scaling factor is determined; wherein the first scaling factor is positively correlated with the second scaling factor and positively correlated with the depth difference ratio.

[0371] In some embodiments, the first transformation relationship is represented in matrix form, and the linear component in the first transformation relationship is positively correlated with the first scaling factor; the rotation component in the first transformation relationship is negatively correlated with the first scaling factor and positively correlated with the value of the principal point coordinates of the image.

[0372] In some embodiments, determining the transformation relationship between the first homography matrix and the second homography matrix as a second transformation relationship based on the first homography matrix and the second homography matrix includes: calculating the inverse matrix of either the first homography matrix or the second homography matrix, and calculating the product of the inverse matrix and the other as the second transformation relationship;

[0373] The second transformation relationship is represented in matrix form. The second scaling factor is positively correlated with the main diagonal element of the 2×2 matrix in the upper left corner of the second transformation relationship, and negatively correlated with the secondary diagonal element of the 2×2 matrix.

[0374] In some embodiments, the second transformation relationship is represented in matrix form. Based on the second transformation relationship, the ratio of the first distance to the second distance is determined as a second scaling factor, including:

[0375] Calculate the mean of the main diagonals of the top-left 2×2 block matrix in the matrix representing the second transformation relationship, and use this as the second scaling factor; or,

[0376] Calculate the square root of the determinant of the top-left 2×2 block matrix in the matrix used to represent the second transformation relationship, and use it as the second scaling factor.

[0377] In some embodiments, the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism in any scene is obtained through the following methods, including:

[0378] The motion control mechanism carries the reference object to the first photographing position, and controls the camera to photograph the reference object, obtaining an image of the reference object at the first photographing position, which is used as the first image;

[0379] The pixel coordinates of at least one feature point of the reference object are determined based on the first image, and used as the pixel coordinates of the feature point corresponding to the first image.

[0380] Determine whether the number of times the motion mechanism carries the reference object to the first photographing position has reached the preset number. If the preset number has not been reached, determine a new photographing position and use the new photographing position as the first photographing position. Repeat the above steps of obtaining the feature point pixel coordinates corresponding to the first image.

[0381] If the preset number of times has been reached, then the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism is determined based on the pixel coordinates of the feature points corresponding to each first image and the position of each first image capture.

[0382] Among them, the distance between each first shooting position and the camera aperture is the same in a preset direction, which is the z-axis orientation of the camera coordinate system.

[0383] Corresponding to the above-described hand-eye calibration method, this application also provides a second motion mechanism system, such as... Figure 14 As shown (which is understandable), Figure 14 (The processor is not shown in the image). The second motion mechanism system includes a processor, a motion mechanism 141, and a camera 142, which is carried on the motion mechanism 141.

[0384] The motion mechanism 141 is used to carry the camera 142 in a first scene and to carry the camera 142 in a second scene.

[0385] Camera 142 is used to take multiple photos of a first reference object in a first scene while being carried by motion mechanism 141 to obtain multiple fifth images; and in a second scene, to take multiple photos of a second reference object while being carried by motion mechanism 141 to obtain multiple sixth images; wherein the first distance and the second distance are different, the first distance is the distance between the first reference object and the aperture of the camera, and the second distance is the distance between the second reference object and the aperture of the camera;

[0386] The processor is used to acquire a first homography matrix and image principal point coordinates determined based on the first homography matrix and the second homography matrix; wherein the image principal point coordinates are the coordinates of the intersection of the camera's optical axis and the pixel plane; the first homography matrix is ​​a homography matrix of the pixel coordinate system and the physical coordinate system of the motion mechanism determined based on multiple fifth images; the second homography matrix is ​​a homography matrix of the pixel coordinate system and the physical coordinate system of the motion mechanism determined based on multiple sixth images.

[0387] The processor is also used to calculate the ratio of the first distance to the distance between the workpiece to be gripped and the aperture of the camera, as a first scaling factor; based on the first scaling factor and the principal point coordinates of the image, to determine the transformation relationship between the first homography matrix and the target homography matrix, as a first transformation relationship; based on the first transformation relationship and the first homography matrix, to determine the target homography matrix; wherein, the target homography matrix is ​​the homography matrix between the pixel coordinate system corresponding to the workpiece to be gripped and the physical coordinate system of the motion mechanism.

[0388] In the technical solution provided in this application embodiment, a first homography matrix and image principal point coordinates are obtained. The image principal point coordinates are the coordinates of the intersection of the optical axis and the pixel plane calculated based on the first homography matrix and the second homography matrix. Specifically, the first homography matrix is ​​the homography matrix of the pixel coordinate system and the physical coordinate system of the motion mechanism determined by multiple images (i.e., the fifth image) obtained by the camera taking multiple shots of the first reference object in the first scene. The second homography matrix is ​​the homography matrix of the pixel coordinate system and the physical coordinate system of the motion mechanism determined by multiple images (i.e., the sixth image) obtained by the camera taking multiple shots of the second reference object in the second scene. The camera is carried on the motion mechanism. The difference between the first scene and the second scene lies in the distance between the first reference object and the camera aperture in the first scene (i.e., the first distance) and the distance between the second reference object and the camera aperture in the second scene (i.e., the second distance). Since the principal point coordinates of the image are the coordinates of the intersection of the camera's optical axis and the pixel plane, the principal point coordinates of the image can be obtained based on the first homography matrix and the second homography matrix. Then, the ratio of the first distance to the distance between the workpiece to be gripped and the camera aperture (i.e., the first scaling factor) is calculated. Based on the first scaling factor and the principal point coordinates of the image, the transformation relationship between the first homography matrix and the target homography matrix (i.e., the first transformation relationship) can be obtained. Then, based on the first transformation relationship and the first homography matrix, the target homography matrix can be determined, which is the homography matrix between the pixel coordinate system corresponding to the workpiece to be gripped and the physical coordinate system of the motion mechanism.

[0389] Understandably, in this application, the target homography matrix is ​​solved based on the first homography matrix and the first transformation relation. The image principal point coordinates used to solve the first transformation relation are the actual image principal point coordinates calculated based on the first homography matrix and the second homography matrix, rather than using the theoretical pixel coordinate system center point as the image principal point coordinates. Therefore, the accuracy of determining the target homography matrix based on the hand-eye calibration method shown in this application is relatively high.

[0390] In some embodiments, the processor is further configured to acquire a second homography matrix;

[0391] The calculation of the ratio of the distance between the first reference object and the camera aperture to the distance between the workpiece to be gripped and the camera aperture, as the first scaling factor, includes:

[0392] Based on the first homography matrix and the second homography matrix, the transformation relationship between the first homography matrix and the second homography matrix is ​​determined, which is used as the second transformation relationship;

[0393] Based on the second transformation relationship, the ratio of the first distance to the second distance is determined as the second scaling factor;

[0394] The first scale factor is determined based on the second scale factor, the height difference between the workpiece to be gripped and the first reference object, and the height difference between the first reference object and the second reference object.

[0395] In some embodiments, determining the first scale factor based on the second scale factor, the height difference between the workpiece to be gripped and the first reference object, and the height difference between the first reference object and the second reference object includes:

[0396] Calculate the ratio of the height difference between the workpiece to be gripped and the first reference object to the height difference between the first reference object and the second reference object, and use it as the depth difference ratio;

[0397] Based on the second scaling factor and the depth difference ratio, the first scaling factor is determined; wherein the first scaling factor is positively correlated with the second scaling factor and positively correlated with the depth difference ratio.

[0398] In some embodiments, the first transformation relationship is represented in matrix form, and the linear component in the first transformation relationship is positively correlated with the first scaling factor; the rotation component in the first transformation relationship is negatively correlated with the first scaling factor and positively correlated with the value of the principal point coordinates of the image.

[0399] In some embodiments, determining the transformation relationship between the first homography matrix and the second homography matrix as a second transformation relationship based on the first homography matrix and the second homography matrix includes: calculating the inverse matrix of either the first homography matrix or the second homography matrix, and calculating the product of the inverse matrix and the other as the second transformation relationship;

[0400] The second transformation relationship is represented in matrix form. The second scaling factor is positively correlated with the main diagonal element of the 2×2 matrix in the upper left corner of the second transformation relationship, and negatively correlated with the secondary diagonal element of the 2×2 matrix.

[0401] In some embodiments, the second transformation relationship is represented in matrix form, and determining the ratio of the first distance to the second distance as a second scaling factor based on the second transformation relationship includes:

[0402] Calculate the mean of the main diagonals of the top-left 2×2 block matrix in the matrix representing the second transformation relationship, and use this as the second scaling factor; or,

[0403] Calculate the square root of the determinant of the top-left 2×2 block matrix in the matrix used to represent the second transformation relationship, and use it as the second scaling factor.

[0404] In some embodiments, the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism in any scene is obtained through the following methods, including:

[0405] The motion control mechanism carries the camera to the second shooting position and controls the camera to take a picture of the reference object, thereby obtaining an image of the reference object at the second shooting position, which is used as the second image;

[0406] The pixel coordinates of at least one feature point of the reference object are determined based on the second image, and used as the pixel coordinates of the feature point corresponding to the second image.

[0407] Determine whether the number of times the motion mechanism carries the camera to the second shooting position has reached a preset number. If the preset number has not been reached, determine a new shooting position and use the new shooting position as the second shooting position. Repeat the above steps of obtaining the feature point pixel coordinates corresponding to the second image.

[0408] If the preset number of times has been reached, then the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism is determined based on the pixel coordinates of the feature points corresponding to each second image and the position of each second image capture.

[0409] Among them, the distance between each first shooting position and the reference object is the same in a preset direction, which is the z-axis orientation of the camera coordinate system.

[0410] This application also provides an electronic device, such as... Figure 15 As shown, it includes:

[0411] Memory 151 is used to store computer programs;

[0412] When the processor 152 executes the program stored in the memory 151, it implements any of the hand-eye calibration methods described above.

[0413] Furthermore, the aforementioned electronic device may also include a communication bus and / or a communication interface. The processor 152, the communication interface, and the memory 151 communicate with each other through the communication bus. The communication bus mentioned in the aforementioned electronic device may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used to represent it in the figure, but this does not mean that there is only one bus or one type of bus.

[0414] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0415] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0416] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0417] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described hand-eye calibration methods.

[0418] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the hand-eye calibration methods described above.

[0419] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a solid-state drive (SSD), etc.

[0420] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0421] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for devices, first motion mechanism systems, second motion mechanism systems, electronic devices, storage media, and computer program products, since they are basically similar to the method embodiments, the descriptions are relatively simple, and relevant parts can be referred to the descriptions of the method embodiments.

[0422] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A hand-eye calibration method, characterized in that, The method includes: Obtain the first homography matrix and the image principal point coordinates; wherein, the image principal point coordinates are the coordinates of the intersection point of the camera's optical axis and the pixel plane calculated based on the first homography matrix and the second homography matrix; the first homography matrix is ​​the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism, determined by multiple images obtained by the camera taking multiple shots of the first reference object in the first scene; the second homography matrix is ​​the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism, determined by multiple images obtained by the camera taking multiple shots of the second reference object in the second scene; the first reference object and the second reference object are carried on the motion mechanism, or the camera is carried on the motion mechanism; the first distance and the second distance are different, the first distance is the distance between the first reference object and the aperture of the camera, and the second distance is the distance between the second reference object and the aperture of the camera; Calculating the ratio of the first distance to the distance between the workpiece to be gripped and the aperture of the camera, as a first scaling factor, includes: determining a transformation relationship between the first homography matrix and the second homography matrix based on the first homography matrix and the second homography matrix, as a second transformation relationship; determining the ratio of the first distance to the second distance based on the second transformation relationship, as a second scaling factor; and determining the first scaling factor based on the second scaling factor, the height difference between the workpiece to be gripped and the first reference object, and the height difference between the first reference object and the second reference object. Based on the first scaling factor and the coordinates of the principal point of the image, the transformation relationship between the first homography matrix and the target homography matrix is ​​determined as the first transformation relationship; wherein, the target homography matrix is ​​the homography matrix between the pixel coordinate system corresponding to the workpiece to be gripped and placed and the physical coordinate system of the motion mechanism; Based on the first transformation relationship and the first homography matrix, the target homography matrix is ​​determined.

2. The method according to claim 1, characterized in that, The method further includes obtaining the second homography matrix.

3. The method according to claim 2, characterized in that, The determination of the first scale factor based on the second scale factor, the height difference between the workpiece to be gripped and the first reference object, and the height difference between the first reference object and the second reference object includes: Calculate the ratio of the height difference between the workpiece to be gripped and the first reference object to the height difference between the first reference object and the second reference object, and use it as the depth difference ratio; The first scaling factor is determined based on the second scaling factor and the depth difference ratio; wherein the first scaling factor is positively correlated with the second scaling factor and positively correlated with the depth difference ratio.

4. The method according to claim 1, characterized in that, The first transformation relationship is represented in matrix form. The linear component in the first transformation relationship is positively correlated with the first scaling factor. The rotation component in the first transformation relationship is negatively correlated with the first scaling factor and positively correlated with the value of the principal point coordinates of the image.

5. The method according to claim 2, characterized in that, The step of determining the transformation relationship between the first homography matrix and the second homography matrix as the second transformation relationship based on the first homography matrix and the second homography matrix includes: calculating the inverse matrix of either the first homography matrix or the second homography matrix, and calculating the product of the inverse matrix and the other as the second transformation relationship; The second transformation relationship is represented in matrix form. The second scaling factor is positively correlated with the main diagonal element of the 2×2 matrix in the upper left corner of the second transformation relationship, and negatively correlated with the secondary diagonal element of the 2×2 matrix.

6. The method according to claim 2, characterized in that, The second transformation relationship is represented in matrix form. The step of determining the ratio of the first distance to the second distance, as a second scaling factor, based on the second transformation relationship includes: Calculate the mean of the main diagonal of the 2×2 block matrix in the upper left corner of the matrix used to represent the second transformation relationship, and use it as the second scaling factor; or, Calculate the square root of the determinant of the top-left 2×2 block matrix in the matrix used to represent the second transformation relationship, and use it as the second scaling factor.

7. The method according to claim 1, characterized in that, When the reference object is carried by the motion mechanism, the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism in any scene is obtained in the following ways, including: The motion mechanism is controlled to carry the reference object to the first photographing position, and the camera is controlled to photograph the reference object to obtain an image of the reference object at the first photographing position, which is used as the first image; The pixel coordinates of at least one feature point of the reference object are determined based on the first image, and are used as the pixel coordinates of the feature point corresponding to the first image. Determine whether the number of times the motion mechanism carries the reference object to the first photographing position has reached a preset number. If the preset number has not been reached, determine a new photographing position and use the new photographing position as the first photographing position. Repeat the above steps of obtaining the feature point pixel coordinates corresponding to the first image. If the preset number of times has been reached, then the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism is determined based on the feature point pixel coordinates corresponding to each first image and each first shooting position; Wherein, the distance between each of the first shooting positions and the aperture of the camera is the same in a preset direction, and the preset direction is the z-axis orientation of the camera coordinate system.

8. The method according to claim 1, characterized in that, When the camera is mounted on the motion mechanism, the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism in any scene is obtained in the following ways, including: The motion mechanism is controlled to carry the camera to the second shooting position, and the camera is controlled to take a picture of the reference object to obtain an image of the reference object at the second shooting position, which is used as the second image; The pixel coordinates of at least one feature point of the reference object are determined based on the second image, and used as the pixel coordinates of the feature point corresponding to the second image. Determine whether the number of times the motion mechanism carries the camera to the second shooting position has reached a preset number. If the preset number has not been reached, determine a new shooting position and use the new shooting position as the second shooting position. Repeat the above steps of obtaining the feature point pixel coordinates corresponding to the second image. If the preset number of times has been reached, then the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism is determined based on the feature point pixel coordinates corresponding to each second image and each second shooting position; Wherein, the distance between each of the second shooting positions and the reference object is the same in a preset direction, and the preset direction is the z-axis orientation of the camera coordinate system.

9. A hand-eye calibration device, characterized in that, The device includes: A homography matrix acquisition module is used to acquire a first homography matrix and image principal point coordinates; wherein, the image principal point coordinates are the coordinates of the intersection point of the camera's optical axis and the pixel plane calculated based on the first and second homography matrices; the first homography matrix is ​​a homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism, determined by multiple images obtained by the camera taking multiple shots of a first reference object in a first scene; the second homography matrix is ​​a homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism, determined by multiple images obtained by the camera taking multiple shots of a second reference object in a second scene; the first and second reference objects are carried on the motion mechanism, or the camera is carried on the motion mechanism; the first distance and the second distance are different, the first distance is the distance between the first reference object and the aperture of the camera, and the second distance is the distance between the second reference object and the aperture of the camera; The scaling factor calculation module is used to calculate the ratio of the distance between the first reference object and the aperture of the camera to the distance between the workpiece to be gripped and the aperture of the camera, as the first scaling factor; The transformation relationship determination module is used to determine the transformation relationship between the first homography matrix and the target homography matrix based on the first scaling factor and the coordinates of the principal point of the image, as the first transformation relationship; wherein, the target homography matrix is ​​the homography matrix between the pixel coordinate system corresponding to the workpiece to be gripped and placed and the physical coordinate system of the motion mechanism; A homography matrix determination module is used to determine the target homography matrix based on the first transformation relationship and the first homography matrix; The scaling factor calculation module is specifically used to determine the transformation relationship between the first homography matrix and the second homography matrix based on the first homography matrix and the second homography matrix, as a second transformation relationship; to determine the ratio of the first distance to the second distance based on the second transformation relationship, as a second scaling factor; and to determine the first scaling factor based on the second scaling factor, the height difference between the workpiece to be gripped and the first reference object, and the height difference between the first reference object and the second reference object.

10. A first motion mechanism system, characterized in that, It includes a processor, a motion mechanism, and a camera, wherein the camera is disposed outside the motion mechanism. The motion mechanism is used to carry a first reference object and move in the first scene; And move while carrying a second reference object in the second scene; The camera is used to take multiple photos of the first reference object carried by the motion mechanism in the first scene to obtain multiple third images; In the second scene, multiple images are taken of the second reference object carried by the motion mechanism to obtain multiple fourth images; wherein, the first distance and the second distance are different, the first distance is the distance between the first reference object and the aperture of the camera, and the second distance is the distance between the second reference object and the aperture of the camera; The processor is configured to acquire a first homography matrix and image principal point coordinates determined based on the first and second homography matrices; wherein the image principal point coordinates are the coordinates of the intersection of the optical axis of the camera and the pixel plane; the first homography matrix is ​​a homography matrix of the pixel coordinate system and the physical coordinate system of the motion mechanism determined based on the plurality of third images; and the second homography matrix is ​​a homography matrix of the pixel coordinate system and the physical coordinate system of the motion mechanism determined based on the plurality of fourth images. The processor is further configured to calculate the ratio of the first distance to the distance between the workpiece to be gripped and the aperture of the camera, as a first scaling factor; based on the first scaling factor and the principal point coordinates of the image, determine the transformation relationship between the first homography matrix and the target homography matrix, as a first transformation relationship; based on the first transformation relationship and the first homography matrix, determine the target homography matrix; wherein, the target homography matrix is ​​the homography matrix between the pixel coordinate system corresponding to the workpiece to be gripped and the physical coordinate system of the motion mechanism; The processor is specifically configured to: determine a transformation relationship between the first homography matrix and the second homography matrix as a second transformation relationship; determine the ratio of the first distance to the second distance as a second scaling factor based on the second transformation relationship; and determine the first scaling factor based on the second scaling factor, the height difference between the workpiece to be gripped and the first reference object, and the height difference between the first reference object and the second reference object.

11. A second motion mechanism system, characterized in that, It includes a processor, a motion mechanism, and a camera, the camera being carried on the motion mechanism. The motion mechanism is used to carry the camera in a first scene and to carry the camera in a second scene. The camera is used to take multiple photos of a first reference object in the first scene while being carried by the motion mechanism to obtain multiple fifth images; and in the second scene, while being carried by the motion mechanism to obtain multiple sixth images of a second reference object; wherein the first distance and the second distance are different, the first distance being the distance between the first reference object and the aperture of the camera, and the second distance being the distance between the second reference object and the aperture of the camera; The processor is configured to acquire a first homography matrix and image principal point coordinates determined based on the first and second homography matrices; wherein the image principal point coordinates are the coordinates of the intersection of the camera's optical axis and the pixel plane; the first homography matrix is ​​a homography matrix of the pixel coordinate system and the physical coordinate system of the motion mechanism determined based on the plurality of fifth images; the second homography matrix is ​​a homography matrix of the pixel coordinate system and the physical coordinate system of the motion mechanism determined based on the plurality of sixth images; The processor is further configured to calculate the ratio of the first distance to the distance between the workpiece to be gripped and the aperture of the camera, as a first scaling factor; based on the first scaling factor and the principal point coordinates of the image, determine the transformation relationship between the first homography matrix and the target homography matrix, as a first transformation relationship; based on the first transformation relationship and the first homography matrix, determine the target homography matrix; wherein, the target homography matrix is ​​the homography matrix between the pixel coordinate system corresponding to the workpiece to be gripped and the physical coordinate system of the motion mechanism; The processor is specifically configured to: determine a transformation relationship between the first homography matrix and the second homography matrix as a second transformation relationship; determine the ratio of the first distance to the second distance as a second scaling factor based on the second transformation relationship; and determine the first scaling factor based on the second scaling factor, the height difference between the workpiece to be gripped and the first reference object, and the height difference between the first reference object and the second reference object.

Images