Hand-eye calibration method and device and movement mechanism system

Abstract

The embodiment of the invention provides a hand-eye calibration method and device and a movement mechanism system. The method comprises the steps that a first homography matrix and image principal point coordinates are acquired; calculating the ratio of the first distance to the distance between the to-be-grabbed workpiece and the aperture of the camera as a first scale factor; based on the first scale factor and the image principal point coordinates, determining a conversion relation between the first homography matrix and the target homography matrix as a first conversion relation; and determining a target homography matrix based on the first conversion relation and the first homography matrix. By applying the technical scheme provided by the embodiment of the invention, the precision of hand-eye calibration can be improved.

Description

Technical Field

[0001] The present application relates to the field of visual technology, and in particular to a hand-eye calibration method, device and motion mechanism system. Background Art

[0002] The principal point of the image is the intersection of the camera optical axis and the imaging plane. Theoretically, the principal point of the image is the center point of the pixel coordinate system. However, in practice, due to installation accuracy issues, the principal point may deviate from the center of the image. Therefore, in practice, the principal point of the image may not be the center point of the pixel coordinate system, especially in scenes where the optical axis of the camera lens is not perpendicular to the plane of the object's feature points.

[0003] In the related art, for 2.5D positioning and guidance scenarios, the main point of the image is usually used as the center point of the pixel coordinate system to solve the conversion relationship between the robot's pixel coordinate system and the physical coordinate system of the motion mechanism, that is, hand-eye calibration is performed. It can be seen that the hand-eye calibration result obtained in this way is not accurate. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a hand-eye calibration method, device and motion mechanism system to improve the accuracy of hand-eye calibration. The specific technical solution is as follows:

[0005] In a first aspect, an embodiment of the present application provides a hand-eye calibration method, the method comprising:

[0006] Obtain a first homography matrix and image principal point coordinates; wherein the image principal point coordinates are coordinates of the intersection of the optical axis of the camera and the pixel plane calculated based on the first homography matrix and the second homography matrix; the first homography matrix is ​​a homography matrix of a pixel coordinate system determined by multiple images obtained by the camera taking multiple shots of the first reference object in the first scene and a physical coordinate system of the motion mechanism; the second homography matrix is ​​a homography matrix of a pixel coordinate system determined by multiple images obtained by the camera taking multiple shots of the second reference object in the second scene and a physical coordinate system of the motion mechanism; the first reference object and the second reference object are carried by the motion mechanism, or the camera is carried by the motion mechanism; the first distance is different from the second distance, the first distance is the distance between the first reference object and the aperture of the camera, and the second distance is the distance between the second reference object and the aperture of the camera;

[0007] Calculating a ratio of the first distance to the distance between the workpiece to be captured and released and the aperture of the camera as a first proportional factor;

[0008] Based on the first scale factor and the coordinates of the principal point of the image, a conversion relationship between the first homography matrix and a target homography matrix is ​​determined as a first conversion relationship; wherein the target homography matrix is ​​a homography matrix between the pixel coordinate system corresponding to the workpiece to be grasped and released and the physical coordinate system of the motion mechanism;

[0009] The target homography matrix is ​​determined based on the first conversion relationship and the first homography matrix.

[0010] In some embodiments, the method further includes obtaining the second homography matrix; and calculating the ratio of the distance between the first reference object and the aperture of the camera to the distance between the workpiece to be captured and released and the aperture of the camera as the first proportional factor, including:

[0011] Based on the first homography matrix and the second homography matrix, determining a conversion relationship between the first homography matrix and the second homography matrix as a second conversion relationship;

[0012] Based on the second conversion relationship, determine a ratio of the first distance to the second distance as a second proportional factor;

[0013] The first scale factor is determined based on the second scale factor, a height difference between the workpiece to be picked and placed and the first reference object, and a height difference between the first reference object and the second reference object.

[0014] In some embodiments, determining the first proportional factor based on the second proportional factor, the height difference between the workpiece to be picked and placed and the first reference object, and the height difference between the first reference object and the second reference object includes:

[0015] Calculating a ratio of a height difference between the workpiece to be grasped and placed and the first reference object to a height difference between the first reference object and the second reference object as a depth difference ratio;

[0016] The first scale factor is determined based on the second scale factor and the depth difference ratio; wherein the first scale factor is positively correlated with the second scale factor and is positively correlated with the depth difference ratio.

[0017] In some embodiments, the first transformation relationship is represented in the form of a matrix, the linear component in the first transformation relationship is positively correlated with the first scale factor; the rotation component in the first transformation relationship is negatively correlated with the first scale factor and positively correlated with the value of the image principal point coordinates.

[0018] In some embodiments, the determining, based on the first homography matrix and the second homography matrix, a conversion relationship between the first homography matrix and the second homography matrix as a second conversion relationship comprises: calculating an inverse matrix of either the first homography matrix or the second homography matrix, and calculating a product of the inverse matrix and the other as the second conversion relationship;

[0019] The second conversion relationship is represented in the form of a matrix, and the second proportional factor is positively correlated with the main diagonal elements in the 2×2 matrix at the upper left corner of the second conversion relationship, and negatively correlated with the sub-diagonal elements in the 2×2 matrix.

[0020] In some embodiments, the second conversion relationship is represented in the form of a matrix, and determining the ratio of the first distance to the second distance based on the second conversion relationship as a second proportional factor includes:

[0021] Calculate the mean of the main diagonal of the 2×2 block matrix in the upper left corner of the matrix used to represent the second conversion relationship as the second proportional factor; or,

[0022] The square root of the determinant of the upper left 2×2 block matrix in the matrix used to represent the second conversion relationship is calculated as the second scaling factor.

[0023] In some embodiments, when the reference object is carried by the motion mechanism, the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism in any scene is obtained by the following method, including:

[0024] Controlling the motion mechanism to carry the reference object to a first photographing position, and controlling the camera to photograph the reference object, to obtain an image of the reference object at the first photographing position as a first image;

[0025] Determine pixel coordinates of at least one feature point of the reference object according to the first image as pixel coordinates of the feature point corresponding to the first image;

[0026] Determine whether the number of times the motion mechanism carries the reference object to move to the first photographing position reaches a preset number, and if not, determine a new photographing position, use the new photographing position as the first photographing position, and repeat the above step of obtaining the pixel coordinates of the feature points corresponding to the first image;

[0027] If the preset number of times has been reached, determining a homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism according to the pixel coordinates of the feature points corresponding to each of the first images and each of the first photographing positions;

[0028] The distances between each of the first photographing positions and the aperture of the camera are the same in a preset direction, and the preset direction is the z-axis direction of the camera coordinate system.

[0029] In some embodiments, when the camera is carried on the motion mechanism, the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism in any scene is obtained by the following method, including:

[0030] Controlling the motion mechanism to carry the camera to move to a second photographing position, and controlling the camera to photograph the reference object, to obtain an image of the reference object at the second photographing position as a second image;

[0031] Determine pixel coordinates of at least one feature point of the reference object according to the second image as pixel coordinates of the feature point corresponding to the second image;

[0032] Determine whether the number of times the motion mechanism carries the camera to move to the second photographing position reaches a preset number, and if not, determine a new photographing position, use the new photographing position as the second photographing position, and repeat the above step of obtaining the pixel coordinates of the feature points corresponding to the second image;

[0033] If the preset number of times has been reached, determining a homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism according to the pixel coordinates of the feature points corresponding to each of the second images and each of the second photographing positions;

[0034] Wherein, the distances between each of the second photographing positions and the reference object are the same in a preset direction, and the preset direction is the z-axis direction of the camera coordinate system.

[0035] In a second aspect, an embodiment of the present application further provides a hand-eye calibration device, the device comprising:

[0036] A homography matrix acquisition module, used to acquire a first homography matrix and image principal point coordinates; wherein the image principal point coordinates are the intersection coordinates of the optical axis of the camera and the pixel plane calculated based on the first homography matrix and the second homography matrix; the first homography matrix is ​​a homography matrix of a pixel coordinate system determined by multiple images obtained by the camera taking multiple shots of a first reference object in a first scene and a physical coordinate system of a motion mechanism; the second homography matrix is ​​a homography matrix of a pixel coordinate system determined by multiple images obtained by the camera taking multiple shots of a second reference object in a second scene and a physical coordinate system of a motion mechanism; the first reference object and the second reference object are carried by the motion mechanism, or the camera is carried by the motion mechanism; the first distance is different from the second distance, the first distance is the distance between the first reference object and the aperture of the camera, and the second distance is the distance between the second reference object and the aperture of the camera;

[0037] A proportional factor calculation module, used for calculating the ratio of the distance between the first reference object and the aperture of the camera to the distance between the workpiece to be captured and released and the aperture of the camera as a first proportional factor;

[0038] A transformation relationship determination module, used to determine, based on the first scale factor and the image principal point coordinates, a transformation relationship between the first homography matrix and a target homography matrix as a first transformation relationship; wherein the target homography matrix is ​​a homography matrix between the pixel coordinate system corresponding to the workpiece to be grasped and released and the physical coordinate system of the motion mechanism;

[0039] A homography matrix determination module is used to determine the target homography matrix based on the first conversion relationship and the homography matrix corresponding to the first reference object.

[0040] In some embodiments, the homography matrix acquisition module is further used to acquire the second homography matrix;

[0041] The scale factor calculation module comprises:

[0042] A first determining unit, configured to determine, based on the first homography matrix and the second homography matrix, a conversion relationship between the first homography matrix and the second homography matrix as a second conversion relationship;

[0043] a second determining unit, configured to determine, based on the second conversion relationship, a ratio of the first distance to the second distance as a second proportional factor;

[0044] The third determining unit is configured to determine the first proportional factor based on the second proportional factor, a height difference between the workpiece to be picked and placed and the first reference object, and a height difference between the first reference object and the second reference object.

[0045] In some embodiments, the third determining unit is specifically configured to:

[0046] Calculating a ratio of a height difference between the workpiece to be grasped and placed and the first reference object to a height difference between the first reference object and the second reference object as a depth difference ratio;

[0047] The first scale factor is determined based on the second scale factor and the depth difference ratio; wherein the first scale factor is positively correlated with the second scale factor and is positively correlated with the depth difference ratio.

[0048] In some embodiments, the first transformation relationship is represented in the form of a matrix, the linear component in the first transformation relationship is positively correlated with the first scale factor; the rotation component in the first transformation relationship is negatively correlated with the first scale factor and positively correlated with the value of the image principal point coordinates.

[0049] In some embodiments, the first determining unit is specifically configured to calculate an inverse matrix of either the first homography matrix or the second homography matrix, and calculate a product of the inverse matrix and the other matrix as the second conversion relationship;

[0050] The second conversion relationship is represented in the form of a matrix, and the second proportional factor is positively correlated with the main diagonal elements in the 2×2 matrix at the upper left corner of the second conversion relationship, and negatively correlated with the sub-diagonal elements in the 2×2 matrix.

[0051] In some embodiments, the second conversion relationship is represented in the form of a matrix, and determining the ratio of the first distance to the second distance based on the second conversion relationship as a second proportional factor includes:

[0052] Calculate the mean of the main diagonal of the 2×2 block matrix in the upper left corner of the matrix used to represent the second conversion relationship as the second proportional factor; or,

[0053] The square root of the determinant of the upper left 2×2 block matrix in the matrix used to represent the second conversion relationship is calculated as the second scaling factor.

[0054] In some embodiments, when the reference object is carried by the motion mechanism, the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism in any scene is obtained by the following method, including:

[0055] Controlling the motion mechanism to carry the reference object to a first photographing position, and controlling the camera to photograph the reference object, to obtain an image of the reference object at the first photographing position as a first image;

[0056] Determine pixel coordinates of at least one feature point of the reference object according to the first image as pixel coordinates of the feature point corresponding to the first image;

[0057] Determine whether the number of times the motion mechanism carries the reference object to move to the first photographing position reaches a preset number, and if not, determine a new photographing position, use the new photographing position as the first photographing position, and repeat the above step of obtaining the pixel coordinates of the feature points corresponding to the first image;

[0058] If the preset number of times has been reached, determining a homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism according to the pixel coordinates of the feature points corresponding to each of the first images and each of the first photographing positions;

[0059] The distances between each of the first photographing positions and the aperture of the camera are the same in a preset direction, and the preset direction is the z-axis direction of the camera coordinate system.

[0060] In some embodiments, when the camera is carried on the motion mechanism, the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism in any scene is obtained by the following method, including:

[0061] Controlling the motion mechanism to carry the camera to move to a second photographing position, and controlling the camera to photograph the reference object, to obtain an image of the reference object at the second photographing position as a second image;

[0062] Determine pixel coordinates of at least one feature point of the reference object according to the second image as pixel coordinates of the feature point corresponding to the second image;

[0063] Determine whether the number of times the motion mechanism carries the camera to move to the second photographing position reaches a preset number, and if not, determine a new photographing position, use the new photographing position as the second photographing position, and repeat the above step of obtaining the pixel coordinates of the feature points corresponding to the second image;

[0064] If the preset number of times has been reached, determining a homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism according to the pixel coordinates of the feature points corresponding to each of the second images and each of the second photographing positions;

[0065] Wherein, the distances between each of the second photographing positions and the reference object are the same in a preset direction, and the preset direction is the z-axis direction of the camera coordinate system.

[0066] In a third aspect, an embodiment of the present application further provides a first motion mechanism system, comprising a processor, a motion mechanism, and a camera, wherein the camera is disposed outside the motion mechanism.

[0067] The motion mechanism is used to carry the first reference object to move in a first scene; and to carry the second reference object to move in a second scene;

[0068] The camera is used to photograph the first reference object carried by the motion mechanism multiple times in the first scene to obtain multiple third images; and photograph the second reference object carried by the motion mechanism multiple times in the second scene to obtain multiple fourth images; wherein the first distance is different from the second distance, the first distance is the distance between the first reference object and the aperture of the camera, and the second distance is the distance between the second reference object and the aperture of the camera;

[0069] The processor is used to obtain a first homography matrix, and image principal point coordinates determined based on the first homography matrix and the second homography matrix; wherein the image principal point coordinates are the intersection coordinates of the optical axis of the camera and the pixel plane; the first homography matrix is ​​a homography matrix of a pixel coordinate system determined based on the plurality of third images and a physical coordinate system of a motion mechanism; the second homography matrix is ​​a homography matrix of a pixel coordinate system determined based on the plurality of fourth images and a physical coordinate system of a motion mechanism;

[0070] The processor is further used to calculate the ratio of the first distance to the distance between the workpiece to be captured and the aperture of the camera as a first proportional factor; based on the first proportional factor and the coordinates of the principal point of the image, determine the transformation relationship between the first homography matrix and the target homography matrix as a first transformation relationship; based on the first transformation relationship and the first homography matrix, determine the target homography matrix; wherein the target homography matrix is ​​the homography matrix of the pixel coordinate system corresponding to the workpiece to be captured and the physical coordinate system of the motion mechanism.

[0071] In a fourth aspect, an embodiment of the present application further provides a second motion mechanism system, comprising a processor, a motion mechanism, and a camera, wherein the camera is carried by the motion mechanism.

[0072] The motion mechanism is used to carry the camera for motion in the first scene; and to carry the camera for motion in the second scene;

[0073] The camera is used to photograph the first reference object multiple times in the first scene while being carried by the motion mechanism to obtain multiple fifth images; and to photograph the second reference object multiple times in the second scene while being carried by the motion mechanism to obtain multiple sixth images; wherein the first distance is different from the second distance, the first distance is the distance between the first reference object and the aperture of the camera, and the second distance is the distance between the second reference object and the aperture of the camera;

[0074] The processor is used to obtain a first homography matrix, and image principal point coordinates determined based on the first homography matrix and the second homography matrix; wherein the image principal point coordinates are the intersection coordinates of the optical axis of the camera and the pixel plane; the first homography matrix is ​​a homography matrix of a pixel coordinate system determined based on the multiple fifth images and a physical coordinate system of a motion mechanism; the second homography matrix is ​​a homography matrix of a pixel coordinate system determined based on the multiple sixth images and a physical coordinate system of a motion mechanism;

[0075] The processor is further used to calculate the ratio of the first distance to the distance between the workpiece to be captured and the aperture of the camera as a first proportional factor; based on the first proportional factor and the coordinates of the principal point of the image, determine the transformation relationship between the first homography matrix and the target homography matrix as a first transformation relationship; based on the first transformation relationship and the first homography matrix, determine the target homography matrix; wherein the target homography matrix is ​​the homography matrix of the pixel coordinate system corresponding to the workpiece to be captured and the physical coordinate system of the motion mechanism.

[0076] In a fifth aspect, an embodiment of the present application further provides an electronic device, including:

[0077] Memory, used to store computer programs;

[0078] The processor is used to implement any of the above-mentioned hand-eye calibration methods when executing the program stored in the memory.

[0079] In a sixth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the steps of any of the above-mentioned hand-eye calibration methods are implemented.

[0080] In a seventh aspect, an embodiment of the present application further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the hand-eye calibration methods described above.

[0081] Beneficial effects of the embodiments of the present application:

[0082] In the technical solution provided in the embodiment of the present application, a first homography matrix and image principal point coordinates are obtained, wherein the image principal point coordinates are the coordinates of the intersection of the optical axis and the pixel plane calculated based on the first homography matrix and the second homography matrix; specifically, the first homography matrix is ​​a homography matrix of a pixel coordinate system and a physical coordinate system of a motion mechanism determined based on multiple images obtained by repeatedly shooting a first reference object with a camera in a first scene; the second homography matrix is ​​a homography matrix of a pixel coordinate system and a physical coordinate system of a motion mechanism determined based on multiple images obtained by repeatedly shooting a second reference object with a camera in a second scene; wherein the first reference object and the second reference object are carried on the motion mechanism, or the camera is carried on the motion mechanism; the difference between the first scene and the second scene is that the first reference object in the first scene is at a distance from the first reference object to the camera. The distance between the aperture of the camera and the second reference object in the second scene (i.e., the first distance) is different from the distance between the aperture of the camera and the second reference object in the second scene (i.e., the second distance). Since the image principal point coordinates are the coordinates of the intersection of the optical axis of the camera and the pixel plane, the image principal point coordinates can be obtained based on the first homography matrix and the second homography matrix. After that, the ratio of the first distance to the distance between the workpiece to be captured and the aperture of the camera (i.e., the first proportional factor) is calculated. According to the first proportional factor and the image principal point coordinates, the conversion relationship between the first homography matrix and the target homography matrix (i.e., the first conversion relationship) can be obtained. Furthermore, based on the first conversion relationship and the first homography matrix, the target homography matrix can be determined, that is, the homography matrix of the pixel coordinate system corresponding to the workpiece to be captured and the physical coordinate system of the motion mechanism.

[0083] It can be understood that in the present application, the target homography matrix is ​​solved based on the first homography matrix and the first transformation relationship, and the image principal point coordinates used to solve the first transformation relationship are the real image principal point coordinates calculated based on the first homography matrix and the second homography matrix, rather than the theoretical center point of the pixel coordinate system as the image principal point coordinates. Therefore, the accuracy of determining the target homography matrix based on the hand-eye calibration method shown in this application is higher.

[0084] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0086] Figure 1 A schematic diagram of the relationship between the positions of coordinate points in the pixel coordinate system and the world coordinate system provided in an embodiment of the present application;

[0087] Figure 2 for Figure 1 A detailed schematic diagram of the positional relationship between the midpoint P' and the point P";

[0088] Figure 3 A schematic diagram of a first flow chart of a hand-eye calibration method provided in an embodiment of the present application;

[0089] Figure 4 A schematic diagram of a process for determining a first proportional factor provided in an embodiment of the present application;

[0090] Figure 5 A schematic diagram of a process for determining a first homography matrix or a second homography matrix provided in an embodiment of the present application;

[0091] Figure 6 Another schematic diagram of a process for determining a first homography matrix or a second homography matrix provided in an embodiment of the present application;

[0092] Figure 7 A second flow chart of the hand-eye calibration method provided in an embodiment of the present application;

[0093] Figure 8 A schematic diagram of a reference depth difference provided in an embodiment of the present application;

[0094] Fig. 9 A schematic diagram of the operating depth difference provided in an embodiment of the present application;

[0095] Fig.10 A schematic diagram of a software interface for the hand-eye calibration method provided in an embodiment of the present application;

[0096] Fig.11 A schematic diagram of another software interface of the hand-eye calibration method provided in an embodiment of the present application;

[0097] Fig.12 A schematic diagram of the structure of a hand-eye calibration device provided in an embodiment of the present application;

[0098] Fig.13 A schematic diagram of a first motion mechanism system provided in an embodiment of the present application;

[0099] Fig.14 A schematic diagram of a second motion mechanism system provided in an embodiment of the present application;

[0100] Fig.15 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0101] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0102] Hand-eye calibration is an important part of robotics technology, involving the intersection of computer vision and robot control. Usually, the "eye" refers to the camera installed on the robot arm or fixed in the environment, and the "hand" refers to the actuator at the end of the robot arm, such as the gripper, suction cup, etc. The main function of hand-eye calibration is to establish the coordinate conversion relationship between the pixel coordinate system of the camera and the coordinate system of the end effector of the robot arm, so as to achieve accurate positioning of the target object.

[0103] For 2D guidance scenes, N-point calibration is usually used; for 3D scenes, camera calibration and hand-eye calibration are usually used. The purpose of camera calibration is to solve the intrinsic and extrinsic parameters of the camera; the purpose of hand-eye calibration is to solve the coordinate transformation relationship between the camera coordinate system and the robotic arm (also called the motion mechanism) coordinate system; N-point calibration refers to solving the coordinate transformation relationship between the pixel coordinate system and the physical coordinate system of the motion mechanism through N groups of corresponding point coordinates in the pixel coordinate system and the physical coordinate system of the motion mechanism (N is a positive integer greater than or equal to 2).

[0104] As for 2.5D positioning scenarios, in related technologies, N-point calibration is usually used to solve the conversion relationship between the pixel coordinate system and the physical coordinate system of the motion mechanism, and then the conversion relationship, as well as the distance from the reference camera lens aperture to the feature point of the workpiece to be captured and released, and the distance from the operating camera lens aperture to the feature point of the workpiece to be captured and released are used to solve the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism during the actual operation process. Among them, the 2.5D positioning scenario refers to the position and posture of the workpiece to be captured and released only changes in the two-dimensional plane, and the distance (or height) from the camera also changes. However, this solution has the following defects:

[0105] a. This solution only uses a single homography matrix, and it is impossible to decompose the camera intrinsic and extrinsic parameters. Therefore, this solution is based on the premise that the image principal point coordinates are the center point of the pixel coordinate system. However, although the optical axis of the lens (i.e., the straight line through the lens perpendicular to the sensor plane) should ideally pass through the middle of the image, in reality, due to the problem of installation accuracy, there are always errors. Therefore, the image principal point coordinates cannot be the center point coordinates of the pixel coordinate system. It can be seen that it is inaccurate to use the center point of the pixel coordinate system as the image principal point. Furthermore, the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism obtained by solving the pixel coordinate system with the image principal point as the center point of the pixel coordinate system is also inaccurate;

[0106] b. When the optical axis of the lens is not perpendicular to the plane of the workpiece feature point, and the plane where the workpiece feature point is located changes in height in the z direction, the feature point in the camera coordinate system changes not only in the z direction, and this method is not compatible with this type of scenario;

[0107] c. The exact position of the camera aperture is difficult to determine. Therefore, the distance from the camera aperture to the feature point of the workpiece to be captured and released is difficult to measure accurately. The homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism obtained based on this solution is naturally inaccurate.

[0108] In order to improve the accuracy of the solved homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism, that is, to improve the accuracy of hand-eye calibration, an embodiment of the present application provides a hand-eye calibration method.

[0109] To facilitate subsequent understanding, the following Figure 1 The example shown (i.e., the pinhole imaging model) illustrates the conversion relationship between the pixel coordinate system and the physical coordinate system of the motion mechanism. Figure 1 and Figure 2 Some special expressions are explained in Figure 1 The point P" (Puv) is the projection point of point P (x, y, z) in the image plane coordinate system, which is P" (x", y"), and the projection point of point P" (x", y") in the pixel coordinate system, which is P" (x", y") uv (u, v); Figure 1 and Figure 2 The axis represented by Z' (z") in the formula indicates that the axis is both the Z' axis and the z" axis. Figure 1 Here, p (p') indicates that the point is both the point P (x, y, z) in the world coordinate system and the point P' (x', y', z') in the camera coordinate system. Figure 1 The Z' (p') in the world coordinate system is the coordinate value of the point P (x, y, z) in the z' direction in the camera coordinate system. Figure 2 Z'p' and Figure 1 The meaning of Z'(p') in is the same as that of Z'(p'). The circle at the position of O' represents the optical center.

[0110] Figure 1 In the above example, the world coordinate system is denoted as O. xyz ; The camera coordinate system is O x＇y＇z＇ (It can also be written as O'(x'y'z'), that is Figure 1 The plane shown by the coordinate system represented by x', y', z'; the image plane coordinate system is O x"y" (It can also be written as O"(x"y"z"), that is Figure 1The plane shown by the coordinate system represented by x", y", and z", that is, Figure 1 The plane in which horizontal lines and vertical lines intersect in the middle image plane); the pixel coordinate system is O uv (Right now Figure 1 The plane shown by the coordinate system represented by u and v in ).

[0111] There is a point P(x, y, z) in the world coordinate system. The projection point of point P(x, y, z) in the image plane coordinate system is P"(x", y"), and the projection point of point P"(x", y") in the pixel coordinate system is p uv (u, v), then P"(x", y") and point p uv The relationship between (u, v) can be expressed as shown in the following formula (1) and formula (2):

[0112] u=αx"+c x (1)

[0113] v=βy"+c y (2)

[0114] In the above formulas (1) and (2), (cx, cy) is the coordinate of the principal point of the image (e.g. Figure 1 The point indicated by O"(cx, cy) in , that is, the intersection of the camera optical axis and the imaging plane, is denoted by c; α is the scaling ratio along the u-axis direction when the image plane coordinates are converted into pixel coordinates, or it can also be understood as the number of pixels per unit length along the u-axis direction when the image plane coordinates are converted into pixel coordinates; β is the scaling ratio along the v-axis direction when the image plane coordinates are converted into pixel coordinates, or it can also be understood as the number of pixels per unit length along the v-axis direction when the image plane coordinates are converted into pixel coordinates; the unit length can be 1 millimeter (mm), 1 meter (m), etc.

[0115] Figure 2 for Figure 1 A detailed schematic diagram of the positional relationship between the midpoint P' and the point P", Figure 2 It can be seen that the point P" (x", y") in the image plane coordinate system is obtained by the point P' (x', y', z') in the camera coordinate system through the similar triangle relationship. Therefore, there is a relationship between the point P' (x', y', z') and the point P" (x", y") as shown in the following formula (3):

[0116] (3)

[0117] Where f is the focal length of the camera, that is, the distance from the camera center (such as O') or aperture to the image plane (such as O'); z' is the coordinate value of the point P (x, y, z) in the world coordinate system in the z' direction in the camera coordinate system ( Figure 2It is denoted as Z'p', which can also be understood as Figure 1 Z'(p') in.

[0118] Combining the above formulas (1)-(3), we can get the point P'(x', y', z') in the camera coordinate system and the point p in the pixel coordinate system uv The conversion relationship between (u, v) is shown in the following formulas (4)-(5):

[0119] (4)

[0120] (5)

[0121] In the above formulas (4)-(5), f x =αf,f y =βf; f is the focal length of the camera, so f x and f y It can be called the equivalent focal length.

[0122] In order to make the above formulas (4)-(5) into linear transformations, homogeneous coordinates can be introduced for expression, and the following formula (6) is obtained:

[0123] (6)

[0124] Among them, f x and f y is the equivalent focal length; (c x , c y ) are the coordinates of the principal point of the image.

[0125] Since the point P'(x', y', z') in the camera coordinate system is obtained by rotating and translating the point P(x, y, z) in the world coordinate system, the relationship between the point P'(x', y', z') in the camera coordinate system and the point P(x, y, z) in the world coordinate system can be expressed as shown in the following formula (7):

[0126] (7)

[0127] Among them, R 3×3 is the 3×3 rotation matrix from the world coordinate system to the camera coordinate system; R 3×1 is the 3×1 translation vector from the world coordinate system to the camera coordinate system.

[0128] Combining the above formulas (6)-(7), the distance from point P(x, y, z) in the world coordinate system to point p in the pixel coordinate system is uv The conversion relationship between (u, v) can be expressed as shown in the following formula (8):

[0129] (8)

[0130] Among them, f x 、f y 、c x 、c y It can be understood as a camera internal parameter, which is a fixed quantity; R 3×3 and T 3×1 It can be understood as the camera external parameters. The camera internal parameter matrix is ​​recorded as K, and the camera external parameter matrix is ​​recorded as [RT], and the transformation in the above formula (8) can be completed. At this time, M is the projection matrix.

[0131] In the above formula (8), the rotation matrix R 3×3 A matrix expression of is shown in the following formula (9):

[0132] (9)

[0133] In the above formula (9), α is the angle of rotation around the X-axis; β is the angle of rotation around the Y-axis; and γ is the angle of rotation around the Z-axis.

[0134] In a 2D scenario, the workpiece to be picked and placed and the motion mechanism can only rotate around the Z axis. For example, a four-axis selective compliance assembly robot arm (SCARA robot) or an XY motion module can only rotate around the Z axis. Therefore, the matrix R shown in the above formula (9) can be expressed as shown in the following formula (10):

[0135] (10)

[0136] In the 2D positioning guidance scenario, by combining the above formula (8) and formula (10), z c is the distance from the feature point plane of the reference object to the lens aperture (or camera center) in the z direction, let z'=z c , we can get the following formula (11):

[0137] (11)

[0138] Simplifying the above formula (11) can obtain the following formula (12):

[0139] (12)

[0140] The above formula (12) can be understood as the conversion relationship between the pixel coordinate system and the world coordinate system, that is, the conversion relationship between the pixel coordinate system and the physical coordinate system of the motion mechanism.

[0141] The hand-eye calibration method provided by the present application is described in detail below through specific embodiments.

[0142] The hand-eye calibration method provided in the embodiment of the present application is applied to an electronic device, which may be a desktop computer, a portable computer, a smart mobile terminal, etc.

[0143] See also Figure 3 , Figure 3 The first flow chart of the hand-eye calibration method provided in the embodiment of the present application includes the following steps:

[0144] Step S31: Obtain the first homography matrix and the coordinates of the principal point of the image;

[0145] Step S32: Calculate the ratio of the first distance to the distance between the workpiece to be captured and released and the aperture of the camera as a first proportional factor;

[0146] Step S33: Based on the first scale factor and the image principal point coordinates, determine a conversion relationship between the first homography matrix and the target homography matrix as a first conversion relationship;

[0147] Step S34: determining a target homography matrix based on the first conversion relationship and the first homography matrix.

[0148] In the technical solution provided in the embodiment of the present application, a first homography matrix and image principal point coordinates are obtained, wherein the image principal point coordinates are the coordinates of the intersection of the optical axis and the pixel plane calculated based on the first homography matrix and the second homography matrix; specifically, the first homography matrix is ​​a homography matrix of a pixel coordinate system and a physical coordinate system of a motion mechanism determined based on a plurality of images obtained by taking multiple shots of a first reference object by a camera in a first scene; the second homography matrix is ​​a homography matrix of a pixel coordinate system and a physical coordinate system of a motion mechanism determined based on a plurality of images obtained by taking multiple shots of a second reference object by a camera in a second scene; wherein the first reference object and the second reference object are carried on the motion mechanism, or the camera is carried on the motion mechanism; the difference between the first scene and the second scene is that the first reference object in the first scene is at a distance of 100 mm from the moving object. The distance between the apertures of the cameras (i.e., the first distance) is different from the distance between the aperture of the second reference object in the second scene and the camera (i.e., the second distance). Since the image principal point coordinates are the coordinates of the intersection of the optical axis of the camera and the pixel plane, the image principal point coordinates can be obtained based on the first homography matrix and the second homography matrix. After that, the ratio of the first distance to the distance between the workpiece to be captured and the aperture of the camera (i.e., the first proportional factor) is calculated. According to the first proportional factor and the image principal point coordinates, the conversion relationship between the first homography matrix and the target homography matrix (i.e., the first conversion relationship) can be obtained. Furthermore, based on the first conversion relationship and the first homography matrix, the target homography matrix can be determined, that is, the homography matrix of the pixel coordinate system corresponding to the workpiece to be captured and the physical coordinate system of the motion mechanism.

[0149] It can be understood that in the present application, the target homography matrix is ​​solved based on the first homography matrix and the first transformation relationship, and the image principal point coordinates used to solve the first transformation relationship are the real image principal point coordinates calculated based on the first homography matrix and the second homography matrix, rather than the theoretical center point of the pixel coordinate system as the image principal point coordinates. Therefore, the accuracy of determining the target homography matrix based on the hand-eye calibration method shown in this application is higher.

[0150] In the above step S31, the motion mechanism can be understood as the robotic arm of the robot; the homography matrix is ​​usually used to identify the conversion relationship between two coordinate systems. In the embodiment of the present application, the homography matrix can be understood as the conversion relationship between the pixel coordinate system and the physical coordinate system of the motion mechanism; the first homography matrix is ​​the conversion relationship between the pixel coordinate system determined in the first scene and the physical coordinate system of the motion mechanism, and the second homography matrix is ​​the conversion relationship between the pixel coordinate system determined in the second scene and the physical coordinate system of the motion mechanism.

[0151] The first scene includes a first reference object, a camera, and a motion mechanism. In one possible example, the first reference object is carried on the motion mechanism, and the camera and the motion mechanism are independent of each other (also called "eyes" outside the "hand"), that is, the camera may not be directly installed on the motion mechanism, but placed in a fixed position. In this case, when obtaining the homography matrix, the first reference object can be driven to move by controlling the motion mechanism, but the camera will not move with the motion mechanism; in another possible example, the first reference object is independent of the motion mechanism, and the camera is carried on the motion mechanism (also called "eyes" on the "hand"), that is, the camera may be directly installed on the motion mechanism. In this case, when obtaining the homography matrix, the camera can be driven to move by controlling the motion mechanism. The second scene is similar to the first scene, with the only difference that the first reference object becomes the second reference object.

[0152] In addition, in order to facilitate the subsequent calculation of the coordinates of the principal point of the image based on the first homography matrix and the second homography matrix, when determining the first homography matrix and the second homography matrix, the distance between the reference object and the aperture of the camera in the first scene and the second scene needs to be different, that is, if the first distance represents the distance between the first reference object and the aperture of the camera, and the second distance represents the distance between the second reference object and the aperture of the camera, then the first distance and the second distance need to be different. The distance between the reference object and the aperture of the camera can also be understood as the distance between the feature point plane of the reference object and the aperture of the camera.

[0153] It can be understood that, for two specific reference objects, there is no specific limitation on which reference object is used as the first reference object and which reference object is used as the second reference object. The specific first distance and second distance depend on which reference object is used as the first reference object and which reference object is used as the second reference object. For example, taking reference object A and reference object B as an example, in some embodiments, reference object A can be used as the first reference object and reference object B can be used as the second reference object. In this case, the first distance is actually the distance between reference object A and the aperture of the camera, and the second distance is actually the distance between reference object B and the aperture of the camera. In other embodiments, reference object B can also be used as the first reference object and reference object A can be used as the second reference object. In this case, the first distance is actually the distance between reference object B and the aperture of the camera, and the second distance is actually the distance between reference object A and the aperture of the camera.

[0154] As can be seen from the above examples, in this application, the first distance and the second distance refer to the first reference object and the second reference object, rather than to a specific reference object. For example, the first distance represents the distance between the first reference object and the aperture of the camera, rather than the distance between reference object A and the aperture of the camera, because it can also be the distance between any specific reference object used as the first reference object, such as reference object B, reference object C, and the aperture of the camera. The first scale factor, second scale factor, first homography matrix, and second homography matrix mentioned in the following content are the same.

[0155] It can be understood that there can be multiple specific reference objects. For example, if there are n specific reference objects, such as reference object 1, reference object 2,..., reference object n, any two of the reference objects can be taken as the first reference object and the second reference object to determine the first homography matrix and the second homography matrix, and then determine the target homography matrix.

[0156] Alternatively, in some embodiments, any two reference objects may be selected from a plurality of reference objects as the first reference object and the second reference object, and after determining the first homography matrix and the second homography matrix, a target homography matrix may be determined in any of the ways shown below, and then, another two reference objects may be selected from the plurality of reference objects as the first reference object and the second reference object, and after determining the first homography matrix and the second homography matrix, a target homography matrix may be determined in any of the ways shown below, and then, the final target homography matrix may be determined based on the two determined target homography matrices.

[0157] The electronic device can determine the first homography matrix based on multiple images of the first reference object obtained by the camera in the first scene, and determine the second homography matrix based on multiple images of the second reference object obtained by the camera in the second scene. The specific method of determining the homography matrix can be found in the following Figure 5 and Figure 6The description in will not be repeated here.

[0158] Regarding the description of the principal point of the image in the embodiment of the present application, the principal point of the image is the intersection of the optical axis of the camera and the imaging plane, and the coordinates of the principal point of the image are the coordinates of the intersection of the optical axis of the camera and the pixel plane, which represent the coordinate values ​​of a fixed point in the physical world projected into the pixel coordinate system. Since the physical coordinates of the principal point of the image are fixed, the coordinates of the principal point of the image projected into the physical coordinate system remain unchanged when hand-eye calibration is performed on planes at different heights. According to this principle, the coordinates of the principal point of the image can be calculated based on the first homography matrix and the second homography matrix obtained in the above step S31.

[0159] It can be understood that the acquisition of the image principal point coordinates in the embodiments of the present application can be that the electronic device used to calculate the target homography matrix directly obtains the first homography matrix and the second homography matrix, and then calculates the image principal point coordinates based on the first homography matrix and the second homography matrix; or, it can also be that the electronic device used to calculate the target homography matrix directly obtains the first homography matrix and the image principal point coordinates, and does not obtain the second homography matrix, and the acquired image principal point coordinates are calculated by other devices (such as devices used to determine the first homography matrix and the second homography matrix) based on the first homography matrix and the second homography matrix.

[0160] The principle and method of determining the coordinates of the principal point of the image based on the first homography matrix and the second homography matrix are described in detail below through formulas (13) to (18).

[0161] The essence of solving the transformation relationship between the pixel coordinate system and the physical coordinate system of the motion mechanism is to solve the homography matrix between two planes (i.e., the image plane and the motion plane of the motion mechanism). For example, the relationship between the pixel coordinate system and the physical coordinate system of the motion mechanism can be expressed as shown in the following formula (13):

[0162] (13)

[0163] Among them, (u, v) can be understood as the coordinates in the pixel coordinate system; (x, y) can be understood as the coordinates in the physical coordinate system of the motion mechanism; H = , represents the transformation relationship from the pixel coordinate system to the physical coordinate system of the motion mechanism, which can be understood as a homography matrix, where h 11 、h 12 ,…,h 32 、h 33 is the parameter in the homography matrix H; it can be understood that the degree of freedom of the transformation relationship from the pixel coordinate system to the physical coordinate system of the motion mechanism is 8, so h 33 With h 11 ~h 32 is linearly related, and thus, h 33Normalize h 33 Set to 1.

[0164] If the homography matrix from the pixel coordinate system to the physical coordinate system of the motion mechanism obtained by N-point calibration is recorded as M Base , then the matrix is ​​inverted and the ninth element is normalized to obtain the matrix M Base_inv , where M Base_inv It can be understood as the homography matrix from the physical coordinate system of the motion mechanism to the pixel coordinate system. Base With M Base_inv The relationship between can be expressed as shown in the following formula (14):

[0165] M Base_inv =(M Base ) -1 (14)

[0166] Combining the above formulas (13) and (14), the transformation relationship between the pixel coordinate system and the physical coordinate system of the motion mechanism can be expressed as shown in the following formula (15):

[0167] (15)

[0168] From the above formulas (12) and (15), we can see that M Base_inv is equivalent to the projection matrix M, and the matrix M is equivalent to the first 3×3 matrix on the right side of the above formula (12).

[0169] According to the fact that the physical coordinates of the principal point of the image remain unchanged, it can be seen that in the hand-eye calibration at different height planes, the coordinates of the principal point projected into the physical coordinate system remain unchanged, so there exists the following equation as shown in formula (16):

[0170] M Base1 ×c=M Base2 ×c (16)

[0171] Among them, M Base1 is the first homography matrix obtained in the above step S12; M Base1 is the second homography matrix obtained in the above step S12; c is the coordinate of the main point of the image, which can be recorded as .

[0172] Let the homography matrix , , then we have the following equation (17):

[0173] (17)

[0174] Wherein, A is a 2×2 matrix, t represents the offset of the first homography matrix; A' is a 2×2 matrix, t' represents the offset of the second homography matrix; c represents the coordinates of the principal point of the image.

[0175] By simplifying the above formula (17), we can get the following equation (18):

[0176] Ac+t= A'c+t' (18)

[0177] According to the above formula (18), the coordinates of the principal point of the image, i.e. c, can be obtained.

[0178] In the above step S32, the workpiece to be captured can be understood as an object that requires a moving mechanism to capture and release when the robot is put into use; the first distance is the distance between the first reference object and the aperture of the camera; the first proportional factor is the ratio of the first distance to the distance between the workpiece to be captured and released and the aperture of the camera, that is, the first proportional factor is the ratio of the distance between the first reference object and the aperture of the camera to the distance between the workpiece to be captured and released and the aperture of the camera. Among them, the distance between the workpiece to be captured and released and the aperture of the camera can also be understood as the distance between the plane where the characteristic point of the workpiece to be captured and released is located and the aperture of the camera; the distance between the first reference object and the aperture of the camera, and the distance between the workpiece to be captured and released and the aperture of the camera can both be obtained by measurement, and then the first proportional factor can be directly obtained; and in some examples, the distance between the first reference object and the aperture of the camera, and the distance between the workpiece to be captured and released and the aperture of the camera can be obtained by measurement, and then the first proportional factor can be obtained directly; and in some examples, the distance between the first reference object and the aperture of the camera, and the distance between the workpiece to be captured and released and the aperture of the camera can be obtained by measurement, and the first proportional factor can be obtained by calculating in other forms instead of measuring. For details, please refer to the following Figure 4 Description in .

[0179] Referring to the description in the above step S31, for two specific reference objects, there is no specific limitation on which reference object is the first reference object and which reference object is the second reference object. In step S32, continuing to take the existence of reference object A and reference object B as an example, when reference object A is the first reference object and reference object B is the second reference object, the first distance is actually the distance between reference object A and the aperture of the camera, and the first proportional factor is actually the ratio of the distance between reference object A and the aperture of the camera to the distance between the workpiece to be captured and released and the aperture of the camera; when reference object B is the first reference object and reference object A is the second reference object, the first distance is actually the distance between reference object B and the aperture of the camera, and the first proportional factor is actually the ratio of the distance between reference object B and the aperture of the camera to the distance between the workpiece to be captured and released and the aperture of the camera.

[0180] In the above step S33, the target homography matrix is ​​the homography matrix of the pixel coordinate system corresponding to the workpiece to be captured and the physical coordinate system of the motion mechanism. It can be understood that although the target homography matrix, the first homography matrix, and the second homography matrix all represent the conversion relationship between the pixel coordinate system and the physical coordinate system of the motion mechanism, the distance between the object (such as the first reference object, the second reference object, and the workpiece to be captured) and the aperture of the camera will affect the conversion relationship between the pixel coordinate system and the physical coordinate system of the motion mechanism, and the distance between the first reference object and the aperture of the camera (i.e., the first distance) and the distance between the second reference object and the aperture of the camera (i.e., the second distance) are not the same, therefore, the first homography matrix and the second homography matrix are not the same. Similarly, the distance between the workpiece to be captured and the aperture of the camera is not necessarily the same as the first distance and the second distance, therefore, the target homography matrix is ​​not necessarily the same as the first homography matrix and the second homography matrix.

[0181] The first conversion relationship is a conversion relationship between the first homography matrix and the target homography matrix. The electronic device can solve and obtain the first conversion relationship based on the first scale factor and the coordinates of the principal point of the image.

[0182] In some embodiments, the first transformation relationship can be represented in the form of a matrix, and the linear component in the first transformation relationship can be understood as the main diagonal elements in the upper left 2×2 block matrix in the matrix used to represent the first transformation relationship, which are positively correlated with the first scale factor; the rotation component in the first transformation relationship can be understood as the elements in the upper right 2×1 matrix in the matrix used to represent the first transformation relationship, which are negatively correlated with the first scale factor and positively correlated with the value of the image principal point coordinates.

[0183] Specifically, in some embodiments, the electronic device may determine the conversion relationship (ie, the first conversion relationship) between the first homography matrix and the target homography matrix by using the following formula (19-1):

[0184] (19-1)

[0185] Among them, T is the first conversion relationship; k' is the first scale factor; c is the coordinate of the main point of the image; and E is the unit matrix.

[0186] It can be understood that the above formula (19-1) is only a possible calculation formula. In some embodiments, the first conversion relationship can also be calculated by other formulas obtained by simply transforming the above formula (19-1). For example, see the following formula (19-2):

[0187] (19-2)

[0188] Among them, T is the first transformation relationship; k' is the first scale factor; c is the coordinate of the main point of the image; and J is a matrix similar to the unit matrix.

[0189] It can be understood that, based on the content shown in the above step S32, the actual value of the first scale factor depends on which reference object is used as the first reference object and which reference object is used as the second reference object. When reference object A is used as the first reference object and reference object B is used as the second reference object, the first scale factor is actually the ratio of the distance between reference object A and the aperture of the camera to the distance between the workpiece to be captured and released and the aperture of the camera. At this time, the first scale factor is substituted into the above formula (19-1) or (19-2) to obtain T, which is actually the conversion relationship between the homography matrix determined in the scene where the reference object is reference object A and the target homography matrix. Then, according to the subsequent step S34, based on the homography matrix determined in the scene where the reference object is reference object A and the first scale factor (i.e., the distance between reference object A and the aperture of the camera), the target homography matrix is ​​converted into the target homography matrix. The target homography matrix can be determined by substituting the first proportional factor (i.e., the ratio of the distance between the reference object B and the aperture of the camera to the distance between the workpiece to be captured and placed and the aperture of the camera) for the target homography matrix; when reference object B is used as the first reference object and reference object A is used as the second reference object, the first proportional factor is actually the ratio of the distance between reference object B and the aperture of the camera to the distance between the workpiece to be captured and placed and the aperture of the camera. At this time, the first proportional factor is substituted into the above formula (19-1) or (19-2) to obtain T, which is actually the conversion relationship between the homography matrix determined in the scene where the reference object is reference object B and the target homography matrix. Then, according to the subsequent step S34, the target homography matrix can be determined based on the homography matrix determined in the scene where the reference object is reference object B and the first proportional factor (i.e., the ratio of the distance between reference object B and the aperture of the camera to the distance between the workpiece to be captured and placed and the aperture of the camera).

[0190] Next, in combination with formulas (20)-(22), the principle of the validity of formula (19-1) in the present embodiment is explained.

[0191] It can be understood that the first homography matrix, the second homography matrix, and the third homography matrix are all conversion relationships between the pixel coordinate system and the physical coordinate system of the motion mechanism. The difference between them is that the distances between the corresponding objects (such as the first reference object, the second reference object, and the workpiece to be captured and released) and the aperture of the camera are different. Therefore, the first homography matrix, the second homography matrix, and the third homography matrix can be converted to each other, and the conversion relationship is related to the distance between the objects corresponding to each homography matrix and the aperture of the camera. The conversion relationship between the first homography matrix and the second homography matrix is ​​explained below.

[0192] Based on the above Figure 2 From the similar triangle relationship between the camera coordinate system and the image plane coordinate system shown in the figure, it can be seen that when the distance between the reference object and the aperture of the camera (denoted as T z) changes from the first distance to the second distance, the following equation (20) can be obtained:

[0193] (20)

[0194] in, It can be understood as the ratio of the first distance to the second distance. Denoted as k.

[0195] The above formula (20) can be transformed into the form shown in the following formula (21):

[0196] (twenty one)

[0197] in, , which is a 2×2 identity matrix.

[0198] Therefore, the conversion relationship between the first homography matrix and the second homography matrix can be expressed as shown in the following formula (22):

[0199] (twenty two)

[0200] From the above formula (22), we can see that ,and is the conversion relationship between the first homography matrix and the second homography matrix, k is the ratio of the first distance to the second distance, and based on the ratio of the first distance to the second distance and the image principal point coordinate c, we can get ; Therefore, based on the relationship between the first homography matrix, the second homography matrix, and the target homography matrix, the transformation relationship between the first homography matrix and the target homography matrix (i.e., the first transformation relationship) can be obtained by solving the ratio of the first distance to the distance between the workpiece to be captured and the aperture of the camera, and the coordinates of the principal point of the image, that is, the above formula (19-1) holds.

[0201] In the above step S34, the first conversion relationship is the conversion relationship between the first homography matrix and the target homography matrix. When the first homography matrix and the first conversion relationship are known, the electronic device can solve the target homography matrix based on the first homography matrix and the first conversion relationship. After solving the target homography matrix, the electronic device can control the robot to accurately grasp the workpiece to be grasped and placed according to the target homography matrix.

[0202] Based on the above Figure 3It can be seen from the shown content that the distance between the first reference object and the aperture of the camera, and the distance between the workpiece to be captured and the aperture of the camera can be obtained by measurement, but in actual applications, the center position of the camera aperture is not easy to determine. Therefore, the distance between the camera aperture and the first reference object and the workpiece to be captured is difficult to measure accurately. It can be seen that the first proportional factor determined by directly measuring the distance between the first reference object and the aperture of the camera, and the distance between the workpiece to be captured and the aperture of the camera as shown in the above step S32 is not accurate, and further, the target homography matrix calculated according to the first proportional factor is also not accurate.

[0203] To solve the problem that the calculation of the first scale factor is not accurate enough, the embodiment of the present application further provides a method for calculating the first scale factor. In this method, the electronic device obtains a second homography matrix. Figure 4 , which may include the following steps:

[0204] Step S41: Based on the first homography matrix and the second homography matrix, determining a conversion relationship between the first homography matrix and the second homography matrix as a second conversion relationship;

[0205] Step S42: Based on the second conversion relationship, determine the ratio of the first distance to the second distance as a second proportional factor;

[0206] Step S43: determining a first proportional factor based on a second proportional factor, a height difference between the workpiece to be picked and placed and the first reference object, and a height difference between the first reference object and the second reference object.

[0207] In the technical solution provided in the embodiment of the present application, the conversion relationship between the first homography matrix and the second homography matrix, that is, the second conversion relationship, can be calculated based on the first homography matrix and the second homography matrix; the second scale factor can be determined based on the second conversion relationship and the image principal point; thereafter, the height difference between the workpiece to be grasped and the first reference object is calculated, and then the first scale factor can be calculated based on the second scale factor and the height difference, thereby avoiding measuring the distance between the first reference object, the second reference object and the workpiece to be grasped and the camera aperture, that is, avoiding the error caused by the difficulty in determining the center point of the camera aperture, thereby solving the target homography matrix based on the first scale factor can improve the accuracy of the target homography matrix.

[0208] It can be understood that, in this article, the height difference between two objects refers to the height difference between the planes where the feature points of the two objects to be photographed and identified are located; the height of the object refers to the distance between the feature points to be photographed and identified in the object and the placement plane when the object is placed normally; the normal placement state refers to the placement state that the workpiece should be in when the motion mechanism grabs the object. Here, the object refers to the first reference object, the second reference object, and the workpiece to be grasped and placed.

[0209] In the above step S41, the second conversion relationship is the conversion relationship between the first homography matrix and the second homography matrix. The second conversion relationship can be directly obtained according to the first homography matrix and the second homography matrix obtained in the above step S31.

[0210] Specifically, in some embodiments, the electronic device may calculate the inverse matrix of either the first homography matrix or the second homography matrix, and calculate the product of the inverse matrix and the other as the second conversion relationship.

[0211] For example, in some embodiments, the electronic device may calculate the inverse matrix of the first homography matrix, and calculate the product of the inverse matrix of the first homography matrix and the second homography matrix, and use the product of the inverse matrix of the first homography matrix and the second homography matrix as the second conversion relationship. Specifically, the calculation may be performed using the following formula (23-1):

[0212] (23-1)

[0213] Among them, T' is the second conversion relationship; M Base1 is the first homography matrix; M Base2 is the second homography matrix.

[0214] Alternatively, in some other embodiments, the electronic device may calculate the inverse matrix of the second homography matrix, and calculate the product of the inverse matrix of the second homography matrix and the first homography matrix, and use the product of the inverse matrix of the second homography matrix and the first homography matrix as the second conversion relationship. Specifically, the calculation may be performed using the following formula (23-2):

[0215] (23-2)

[0216] Among them, T' is the second conversion relationship; M Base1 is the first homography matrix; M Base2 is the second homography matrix.

[0217] For ease of description, the following contents are all explained using T' calculated by formula (23-1) as an example. If you need to use T' calculated by formula (23-2) to perform subsequent steps, replace T' calculated by formula (23-1) in the following content with the inverse matrix of T' calculated by formula (23-2).

[0218] In the above step S42, the second proportional factor is the ratio of the distance between the first reference object and the aperture of the camera (i.e., the first distance) to the distance between the second reference object and the aperture of the camera (i.e., the second distance). When the second conversion relationship and the coordinates of the principal point of the image are known, the second proportional factor can be solved by referring to the above formula (22).

[0219] In some embodiments, when the second conversion relationship is represented in the form of a matrix, the second scaling factor is positively correlated with the main diagonal elements in the 2×2 matrix in the upper left corner of the second conversion relationship, and is negatively correlated with the sub-diagonal elements in the 2×2 matrix (i.e., the two elements in the diagonal direction from the lower left corner to the upper right corner in the 2×2 block matrix).

[0220] Specifically, the electronic device can solve the second proportional factor by the following formula (24):

[0221] (twenty four)

[0222] Among them, T' is the second conversion relationship; k is the second scale factor; c is the coordinate of the main point of the image; and E is the unit matrix.

[0223] In some embodiments, the second conversion relationship solved in the above step S41 can be represented in the form of a matrix, so that the electronic device can calculate the mean of the main diagonal of the 2×2 block matrix in the upper left corner of the matrix used to represent the second conversion relationship (that is, the mean of the two elements in the diagonal direction from the upper left corner to the lower right corner of the 2×2 block matrix) as the second proportional factor; or, the electronic device can calculate the square root of the determinant of the 2×2 block matrix in the upper left corner of the matrix used to represent the second conversion relationship as the second proportional factor.

[0224] For example, the matrix used to represent the second transformation relationship is recorded as , where T 11 That is, the 2×2 block matrix in the upper left corner of the matrix. Specifically, , if the mean of the main diagonal elements of the block matrix is ​​used as the second scale factor, then the second scale factor Alternatively, if the square root of the determinant of the block matrix is ​​used as the second scaling factor, then the second scaling factor Alternatively, the main diagonal elements of the block matrix may be taken as the second scale factor, that is, k is wx or wy. This application does not limit the value of k.

[0225] In the above step S43, the electronic device can determine the first proportionality factor based on the second proportionality factor, the height difference between the workpiece to be picked and placed and the first reference object, and the height difference between the first reference object and the second reference object.

[0226] In some embodiments, the electronic device can calculate the ratio of the height difference between the workpiece to be grasped and placed and the first reference object, and the height difference between the first reference object and the second reference object, as a depth difference ratio; then, based on the second proportional factor and the depth difference ratio, the electronic device can determine the first proportional factor; wherein the first proportional factor is positively correlated with the second proportional factor, and is positively correlated with the depth difference ratio.

[0227] Specifically, in some embodiments, the electronic device may solve the first proportional factor by the following formula (25-1):

[0228] (25-1)

[0229] Wherein, k' is the first proportional factor; k is the second proportional factor; is the height difference between the workpiece to be grasped and placed and the first reference object; is the height difference between the first reference object and the second reference object.

[0230] It can be understood that the above formula (25-1) is only a possible calculation formula. In some embodiments, the first proportional factor can also be calculated by other formulas obtained by simply transforming the above formula (25-1). For example, see the following formula (25-2):

[0231] (25-2)

[0232] Wherein, k' is the first proportional factor; k is the second proportional factor; is the height difference between the workpiece to be grasped and placed and the first reference object; is the height difference between the first reference object and the second reference object.

[0233] It is understandable that in order to avoid measuring the distance between the object and the camera aperture, the height difference between the first reference object and the second reference object in the embodiment of the present application is obtained based on the height of the first reference object and the second reference object, and the height difference between the first reference object and the workpiece to be captured and released is also obtained based on the height of the first reference object and the workpiece to be captured and released. Among them, the height difference between the first reference object, the second reference object, and the workpiece to be captured and released can be obtained from the workpiece drawing, or it can be determined using measuring tools such as a ruler, a tape measure, a height gauge, or a micrometer, or a height measurement sensor can be used to improve the measurement accuracy. The specific measurement tool to be used can be determined based on the positioning guidance accuracy.

[0234] The following is an explanation of the principle of the validity of the above formula (25-1) through formulas (26) to (29).

[0235] The distance from the feature point plane of the first reference object to the camera aperture in the z direction is H Mark1 , the z-direction distance from the feature point plane of the second reference object to the camera aperture is H Mark2 , we can get the following formula (26)-formula (27)

[0236] Z c_Mark1 = H Mark1 (26)

[0237] Z c_Mark2 = H Mark2 (27)

[0238] The height difference between the first reference object and the second reference object is recorded as , then △H Mark = H Mark2 -H Mark1 , then the second proportional factor k can be expressed as shown in the following formula (28):

[0239] (28)

[0240] The distance in the z direction from the feature point plane of the workpiece to be picked up to the camera aperture is H. Cur , then Z c_Cur =H Cur ; The height difference between the first reference object and the workpiece to be picked up is recorded as △H Cur , then △H Cur = H Cur -H Mark1 , at this time, the first scale factor It can be expressed as shown in the following formula (29):

[0241] (29)

[0242] Therefore, the above formula (25-1) holds.

[0243] In some embodiments, based on this application Figure 4 The first scale factor is determined in the manner provided by the example shown, and then the first conversion relationship is determined based on the first scale factor and the image principal point coordinates according to the above step S33. Then, in the above step S34, each element in the target homography matrix can be calculated in the following manner:

[0244] The target homography matrix is ​​recorded as

[0245] Then we have:

[0246]

[0247]

[0248]

[0249]

[0250]

[0251]

[0252]

[0253]

[0254]

[0255] Among them, k' is based on Figure 4 The first proportional factor obtained in the manner shown; h 11 ~h 23 are the elements in the first homography matrix.

[0256] In other embodiments, when the main diagonal elements in the 2×2 block matrix are respectively taken as the second scale factor (i.e., k is wx, wy) according to the content shown in the above step S42, then combined with the above formula (25-1), the calculation formula of the first scale factor k' (i.e., the following k'x', k'y') can be expressed as shown in the following formulas (30) and (31) (it can be understood that if formula (25-2) is combined here instead of formula (25-1), it is only necessary to replace the constant 1 in the following formulas (30) and (31) with 0.9):

[0257] (30)

[0258] (31)

[0259] In this case, M Cur_inv The calculation formula for each element in the matrix is:

[0260]

[0261]

[0262]

[0263]

[0264]

[0265]

[0266]

[0267]

[0268]

[0269] Next, through the following Figure 5 and Figure 6 The method of obtaining the first homography matrix and the second homography matrix in the above step S31 of the present application is described.

[0270] See also Figure 5 , Figure 5 A schematic diagram of a process for obtaining a first homography matrix or a second homography matrix provided in an embodiment of the present application. When the reference object is carried by a motion mechanism, the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism in any scene can be obtained by the following method: Figure 5 The method shown in the figure may include the following steps:

[0271] Step S51: controlling the motion mechanism to carry the reference object to move to a first photographing position, and controlling the camera to photograph the reference object, to obtain an image of the reference object at the first photographing position as a first image;

[0272] The electronic device can control the motion mechanism to carry the reference object to move to the first photographing position, and control the camera to photograph the reference object to obtain an image of the reference object at the first position, namely, a first image.

[0273] It can be understood that, in order to facilitate the acquisition of the homography matrix, a new photographing position will be determined as the first photographing position later, but for the first photographing position moved to for the first time (referred to as the "initial first photographing position"), it is only necessary to keep the reference object within the camera's field of view at the initial photographing position, and there is no need to care about the distance between the initial first photographing position and the camera's aperture; and the first photographing position determined in the subsequent process (referred to as the "new first photographing position"), while ensuring that the reference object is within the camera's field of view, also needs to ensure that the distance between each new first photographing position and the camera's aperture is the same as the distance between the initial first photographing position and the camera's aperture; wherein, the distance in the embodiment of the present application refers to the distance in the preset direction, and the preset direction is the z-axis direction of the camera coordinate system; in this way, the distance between each first photographing position and the camera's aperture is the same in the preset direction. The specific description can be combined with the following step S53.

[0274] In addition, the reference object in the embodiment of the present application may be a checkerboard calibration plate, a circular calibration plate, or the like.

[0275] Step S52: Determine pixel coordinates of at least one feature point of the reference object according to the first image as pixel coordinates of the feature point corresponding to the first image.

[0276] The electronic device may determine the pixel coordinates of at least one feature point of the reference object based on the first image as the pixel coordinates of the feature point corresponding to the first image. It is understandable that the feature points obtained by the electronic device from each first image are the same. For example, there are two first images, namely image 1 and image 2. The pixel coordinates of feature point A of reference object 1 and the pixel coordinates of feature point B are obtained in image 1. Then, the pixel coordinates of the feature point obtained in image 2 should also be the pixel coordinates of feature point A of reference object 1 and the pixel coordinates of feature point B.

[0277] It is understandable that, in order to improve the accuracy of the acquired pixel coordinates of the feature points, in the embodiment of the present application, the first image may be firstly subjected to distortion correction to eliminate the radial distortion and tangential distortion of the camera lens, and then the pixel coordinates of the multiple feature points of the reference object (i.e., the pixel coordinates of the feature points) may be determined based on the first image after the distortion correction. Among them, the electronic device may perform distortion correction on the first image by a polar coordinate conversion method, a polynomial fitting method, or a correction method based on a parameter model.

[0278] Step S53: determining whether the number of times the motion mechanism carries the reference object to move to the first photographing position reaches a preset number; if not, determining a new photographing position, taking the new photographing position as the first photographing position, and repeating the above step of obtaining the pixel coordinates of the feature points corresponding to the first image;

[0279] The electronic device can determine whether the number of times the moving mechanism carries the reference object to move to the first photographing position reaches a preset number. If it does not reach the preset number, the photographing position can be re-determined, and the re-determined photographing position is used as the first photographing position. The above steps S51-S53 are repeated until the number of times the moving mechanism carries the reference object to move to the first photographing position reaches the preset number.

[0280] It can be understood that in this embodiment, the preset number of times refers to the number of times required to obtain a homography matrix, for example, the number of times required to obtain the first homography matrix, or the number of times to obtain the second homography matrix. Therefore, within the preset number of times, when determining a new photographing position, it is necessary to keep the distance between each new photographing position determined and the aperture of the camera the same as the distance between the first first photographing position and the aperture of the camera in the preset direction, that is, the distance between each first photographing position and the aperture of the camera in the preset direction is the same. However, when the preset number of times is reached, the homography matrix of the current scene can be determined, and the homography matrix of another scene needs to be obtained (for example, the first homography matrix has been obtained, and the second homography matrix is ​​obtained), the distance between the determined new photographing position and the aperture of the camera in the preset direction needs to be different from the distance between each previous first photographing position and the aperture of the camera in the preset direction.

[0281] Step S54: if the preset number of times has been reached, then according to the pixel coordinates of the feature points corresponding to each first image and each first photographing position, a homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism is determined.

[0282] When the preset number of times is reached, the electronic device can determine the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism according to the pixel coordinates of the feature points corresponding to each first image and each first photographing position.

[0283] See also Figure 6 , Figure 6 A second flow chart of obtaining the first homography matrix or the second homography matrix provided in the embodiment of the present application, when the camera is carried on the motion mechanism, the homography matrix of the pixel coordinate system and the physical coordinate system of the motion mechanism in any scene can be obtained by the following Figure 6 The method shown in the figure may include the following steps:

[0284] Step S61: controlling the motion mechanism to carry the camera to move to a second photographing position, and controlling the camera to photograph the reference object, to obtain an image of the reference object at the second photographing position as a second image;

[0285] In the embodiment of the present application, the camera is located on the motion mechanism, so the motion mechanism moves to the second photographing position, that is, the camera is located at the second photographing position. The electronic device can control the motion mechanism to carry the camera to the second photographing position, and control the camera to photograph the reference object to obtain an image of the reference object at the second position, that is, the second image.

[0286] In order to facilitate the acquisition of the homography matrix, a new photographing position will be determined as the second photographing position later. However, for the second photographing position moved to for the first time (referred to as the "initial second photographing position"), it is only necessary to keep the camera at the initial second photographing position to be able to photograph the reference object, and there is no need to care about the distance between the initial second photographing position and the aperture of the camera; while the second photographing position determined in the subsequent process (referred to as the "new second photographing position") ensures that the distance between each new second photographing position and the aperture of the camera is the same as the distance between the initial second photographing position and the aperture of the camera; wherein, the distance in the embodiment of the present application refers to the distance in the preset direction, and the preset direction is the z-axis direction of the camera coordinate system; in this way, the distance between each first photographing position and the aperture of the camera is the same in the preset direction. The specific description can be combined with the following step S63.

[0287] In addition, the reference object in the embodiment of the present application may be a checkerboard calibration plate, a dot calibration plate, etc.

[0288] Step S62: determining pixel coordinates of at least one feature point of the reference object according to the second image as pixel coordinates of the feature point corresponding to the second image;

[0289] The electronic device may determine the pixel coordinates of at least one feature point of the reference object according to the second image as the pixel coordinates of the feature point corresponding to the second image. Similar to the above step S52, the feature points obtained by the electronic device from each second image are the same; and, in some embodiments, to improve the accuracy of the pixel coordinates of the multiple feature points obtained from the second image, the second image may be distorted and then the pixel coordinates of the feature points may be determined.

[0290] Step S63: determining whether the number of times the motion mechanism carries the camera to move to the second photographing position reaches a preset number; if not, determining a new photographing position, taking the new photographing position as the second photographing position, and repeating the above step of obtaining the pixel coordinates of the feature points corresponding to the second image;

[0291] The electronic device can determine whether the number of times the motion mechanism carries the camera to the second photographing position reaches a preset number. If it does not reach the preset number, the photographing position can be re-determined, and the new position can be used as the second photographing position. The above steps S61-S63 are repeated until the number of times the motion mechanism carries the camera to the second photographing position reaches the preset number.

[0292] It can be understood that in the embodiment of the present application, the preset number of times refers to the number of times required to obtain a homography matrix. Therefore, within the preset number of times, the distance between each new photographing position and the reference object must be kept the same as the distance between the first second photographing position and the reference object in the preset direction, that is, the distance between each second photographing position and the reference object is the same in the preset direction. However, when the preset number of times is reached, the homography matrix of the current scene is determined, and the homography matrix of another scene needs to be obtained (for example, the first homography matrix has been obtained, and the second homography matrix needs to be obtained), the distance between the new photographing position and the reference object is different from the distance between each second photographing position and the reference object in the preset direction.

[0293] Step S64: if the preset number of times has been reached, then the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism is determined according to the pixel coordinates of the feature points corresponding to each second image and each second photographing position.

[0294] When the preset number of times is reached, the electronic device can determine the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism according to the pixel coordinates of the feature points corresponding to each second image and each second photographing position.

[0295] It can be understood that the target homography matrix in this article is obtained by solving the first homography matrix and the second homography matrix, and the first homography matrix and the second homography matrix are both determined based on the pixel coordinates of the feature points of the reference object acquired by the camera. Therefore, the descriptions of the height difference and distance mentioned in this article, such as the height difference between the objects (such as the first reference object, the second reference object, and the workpiece to be captured and released), and the distance between the object and the aperture of the camera, can actually be understood as the height difference between the planes where the feature points of the object are located, and the distance between the plane where the feature points of the object are located and the aperture of the camera. For example, the height difference between the first reference object and the second reference object can actually be understood as the height difference between the plane where the feature points of the first reference object are located and the plane where the feature points of the second reference object are located; the distance between the workpiece to be captured and released and the aperture of the camera can actually be understood as the distance between the plane where the feature points of the workpiece to be captured and released and the aperture of the camera.

[0296] In the following, we take the reference object carried by the motion mechanism and the camera fixedly installed outside the motion mechanism as an example. Figure 7-Figure 11 , the hand-eye calibration method provided in the embodiment of the present application is further explained.

[0297] See also Figure 7 , Figure 7 A second flow chart of the hand-eye calibration method provided in the embodiment of the present application is shown in FIG. Figure 7 Steps S71 to S75 belong to the offline calibration phase, and steps S76 to S77 belong to the online operation phase:

[0298] Step S71: The motion mechanism reaches a predetermined photographing position, carries the calibration reference object 1 and moves to take photographs, and collects several sets of image coordinates and physical coordinate data (no less than two sets).

[0299] In the embodiment of the present application, the predetermined photographing position can be understood as the first photographing position, the plurality of sets of image coordinates can be understood as the pixel coordinates of the feature points corresponding to the first images, and the physical coordinate data can be understood as the coordinates of the first photographing positions. For details, please refer to the description in the above step S51.

[0300] Step S72: Solve the homography matrix 1M from the pixel coordinate system to the physical coordinate system of the motion mechanism Base1 For details, please refer to the description in the above steps S52 to S54.

[0301] Step S73: The motion mechanism reaches the predetermined photographing position, carries the calibration reference object 2 and moves to take photographs, and collects several sets of image coordinates and physical coordinate data (no less than two sets).

[0302] In the embodiment of the present application, the predetermined photographing position can be understood as the second photographing position, the plurality of sets of image coordinates can be understood as the pixel coordinates of the feature points corresponding to each second image, and the physical coordinate data can be understood as the coordinates of each second photographing position. For details, please refer to the description in the above step S51. Step S73 is the same as step S71, except that reference object 1 is changed to reference object 2.

[0303] Step S74: Solve the homography matrix 2M from the pixel coordinate system to the physical coordinate system of the motion mechanism Base2 .

[0304] Step S74 is the same as step S72, except that reference object 1 is changed to reference object 2.

[0305] Step S75: Obtain the height difference between the feature point of calibration reference object 2 and the feature point of calibration reference object 1 .

[0306] In the embodiment of the present application, the height difference between the characteristic point of the calibration reference object 2 and the characteristic point of the calibration reference object 1 can be understood as the height difference between the first reference object and the second reference object in the above step S43, and the height difference can also be called the reference depth difference. Figure 8 As shown, Figure 8 There are reference objects 1 and 2 in the image processing system, and the camera 81 is fixedly mounted on the motion mechanism 82. The motion mechanism 82 can carry the camera 81 to move. The height difference between the reference objects 1 and 2 is the reference depth difference (e.g. Figure 8 The value of the height indicated by 3 in the figure).

[0307] Step S76: Obtain the height difference between the feature point of the workpiece to be picked up and the feature point of the calibration reference object 1 .

[0308] In the embodiment of the present application, the height difference between the feature point of the workpiece to be captured and placed and the feature point of the reference object 1 can be understood as the height difference between the workpiece to be captured and placed and the first reference object in the above step S43, and the height difference can also be called the running depth difference. Fig. 9 As shown, Fig. 9 There is a reference object 1 and a workpiece 4 to be picked up and placed, the camera 81 is fixedly mounted on the motion mechanism 82, and the height difference between the workpiece 4 to be picked up and placed and the reference object 1 is the running depth difference (such as Fig. 9 The value of the height indicated by 5 in the figure), that is .

[0309] Step S77: Using the reference homography matrix 1M Base1 and the base homography matrix 2M Base2, solve the image principal point coordinates and the reference height scale factor (i.e., the second scale factor). The method for solving the image principal point coordinates can refer to the description in the above step S31, and the method for solving the reference height scale factor can refer to the description in the above step S42.

[0310] Reuse the base depth difference and running depth difference , solve the running height scale factor (ie, the first scale factor), for details, please refer to the description in the above step S43. Finally, solve the running homography matrix M from the pixel coordinate system to the physical coordinate system of the motion mechanism cur .

[0311] In the technical solution provided in the embodiment of the present application, for different workpieces to be grasped and released, a first homography matrix, a second homography matrix, and a reference depth difference can be determined before determining the target homography matrix each time. The target homography matrix is ​​determined based on the first homography matrix, the second homography matrix, and the reference depth difference determined in real time, so as to improve the accuracy of the target homography matrix, wherein different workpieces to be grasped and released can be understood as different objects.

[0312] To illustrate with an example, for example, a factory produces two kinds of items, item A and item B. A robotic arm (i.e., a motion mechanism) and a camera are respectively arranged on the conveyor belt. The robotic arm is used to grab the item when the item passes the position where the robotic arm is located and place it in the next process area. In order to achieve accurate positioning of item A or item B by the robotic arm, when the conveyor belt is used to convey item A, the electronic device can be arranged according to the aforementioned Figure 5 or Figure 6 The first homography matrix and the second homography matrix are determined in the manner shown, and the reference depth difference is recorded, and then combined with the above-mentioned Figure 3 and Figure 4 The method shown is based on the first homography matrix and the second homography matrix to solve the target homography matrix, that is, the homography matrix of the pixel coordinate system corresponding to the object A and the physical coordinate system of the motion mechanism, and then based on the homography matrix corresponding to the object A, the object A is accurately grasped; when the conveyor belt finishes conveying the object A and starts conveying the object B, the electronic device can again follow the aforementioned method of the present application Figure 5 or Figure 6 The first homography matrix and the second homography matrix are determined in the manner shown, and the reference depth difference is recorded, and then combined with the above-mentioned Figure 3 and Figure 4The method shown is based on the first homography matrix and the second homography matrix to solve the target homography matrix, that is, the homography matrix of the pixel coordinate system corresponding to the object B and the physical coordinate system of the motion mechanism, and then based on the homography matrix corresponding to the object B, the object B is accurately grasped. It should be noted that in this example, regardless of whether the installation position of the robot arm and the camera changes when the object conveyed on the conveyor belt is switched from object A to object B, the first homography matrix and the second homography matrix are re-determined, and the reference depth difference is recorded. That is, for different workpieces to be grasped and released, regardless of whether the installation position of the robot arm and the camera changes, the first homography matrix, the second homography matrix, and the reference depth difference are re-determined once.

[0313] Alternatively, in some embodiments, for different workpieces to be grasped and released, when the installation positions of the robot arm and the camera have not changed, there is no need to redetermine the first homography matrix, the second homography matrix, and the reference depth difference, and the target homography matrix can be determined based on the previously determined first homography matrix, the second homography matrix, and the reference depth difference; only when the installation positions of the robot arm and the camera have changed, the first homography matrix and the second homography matrix are redetermined, and then the target homography matrix is ​​determined based on the new first homography matrix, the second homography matrix, and the reference depth difference.

[0314] Continuing with the above example, when the conveyor belt is used to transport item A, the electronic device determines the first homography matrix, the second homography matrix, and records the reference depth difference, determines the target homography matrix corresponding to item A based on the first homography matrix, the second homography matrix, and the recorded reference depth difference, and achieves accurate grasping of item A based on the homography matrix corresponding to item A. Subsequently, when the conveyor belt is used to convey object B, if the installation positions of the robot arm and the camera have not changed, that is, the installation positions of the robot arm and the camera are the same as those when conveying object A, it is only necessary to determine the running depth difference corresponding to object B, and then determine the target homography matrix corresponding to object B according to the first homography matrix, the second homography matrix, and the reference depth difference determined when conveying A, and then accurately grasp object B based on the homography matrix corresponding to object B; if the installation positions of the robot arm and the camera have changed, that is, the installation positions of the robot arm and the camera are different from those when conveying object A, it is necessary to re-determine the first homography matrix, the second homography matrix, and record the reference depth difference, and then determine the target homography matrix corresponding to object B according to the newly determined first homography matrix, the second homography matrix, and the reference depth difference, and accurately grasp object B based on the target homography matrix.

[0315] In the embodiment of the present application, when the installation positions of the robotic arm and the camera have not changed, no matter what changes occur in the workpiece to be grasped and placed, it is only necessary to determine the running depth difference corresponding to the actual workpiece to be grasped and placed, and then the target homography matrix corresponding to the actual workpiece to be grasped and placed can be determined by combining the previously determined first homography matrix, the second homography matrix, and the reference depth difference. Based on this, the solution provided in the embodiment of the present application has a wide range of applications, high accuracy and simple operation.

[0316] See also Fig.10 , Fig.10 A schematic diagram of a software interface of the hand-eye calibration method provided in the embodiment of the present application, such as Fig.10 As shown in the figure, from the top to the bottom of the interface, the top shows the words "9 Multi-plane hand-eye calibration based on image principal points"; below the words there are two options, namely "Basic parameters" and "Result display". Fig.10 The page shown should be the page indicated by the "Basic Parameters" option. The page shows "Category Input". There are four lines of text below "Category Input", namely "Benchmark Depth Difference", "Operation Depth Difference", "Benchmark Calibration Matrix 1", and "Benchmark Calibration Matrix 2". There is a long box behind each of these four lines of text. There is a link symbol 101 on the right side of each long box (understandable, Fig.10 Only one link symbol is marked in the figure). Currently, "100" is displayed in the long box corresponding to "Benchmark depth difference", "200" is displayed in the long box corresponding to "Running depth difference", "8 1810mm.Calibration matrix []" is displayed in the long box corresponding to "Benchmark calibration matrix 1", and "12 1910mm.Calibration matrix []" is displayed in the long box corresponding to "Benchmark calibration matrix 2". There are three options at the bottom of the interface, namely "Continuous execution", "Execute", and "OK".

[0317] Understandably, the above Fig.10 In the interface shown, the corresponding long boxes behind "reference depth difference", "operating depth difference", "reference calibration matrix 1", and "reference calibration matrix 2" correspond to their corresponding values, such as the corresponding value in the long box behind "reference calibration matrix 1", such as the determined first homography matrix. The user can directly input the corresponding value into the long box, or click the link symbol 101 on the right side of the long box. In response to the user's operation of clicking the link symbol 101, the electronic device can import the corresponding value into the corresponding long box.

[0318] In some embodiments, in response to the user clicking the "Continuous Execution" operation, the electronic device can continuously obtain the data in each long frame and determine the image principal point coordinates and the target homography matrix corresponding to each group of long frame data. In response to the user clicking the "Execute" operation, the electronic device can determine the image principal point coordinates and the target homography matrix corresponding to a group of data in the long frame input by the user; or, the electronic device can obtain the data in a group of long frames through the link symbol 101, and determine the image principal point coordinates and the target homography matrix corresponding to the group of data.

[0319] See also Fig.11 , Fig.11 A schematic diagram of a software interface of the hand-eye calibration method provided in an embodiment of the present application, Fig.11 It can also be understood as Fig.10 A schematic diagram of an output interface corresponding to the input interface shown, Fig.11 There are five rows of data in the interface shown. The first row of data is "ˇMultiplaneHomoMat1"; the second row of data is "Module Status 1"; the third row of data is: CurCalibMatrix -0.1950009;-0.006203677;1567.321;-0.005978366;0.1952618;-294.1291;0;0;1;

[0320] The fourth row of data is "Cx 2570.27"; the fifth row of data is "Cy 2617.838".

[0321] In summary, the technical solution provided by the embodiments of the present application has the following beneficial effects:

[0322] a. This application does not limit the image principal point to the center point of the pixel coordinate system. The calibration model is closer to the actual model, and the multi-plane hand-eye calibration accuracy is higher;

[0323] b. This application is compatible with scenarios where the optical axis of the lens is not perpendicular to the plane of the feature point of the workpiece, and has a wider range of usage scenarios;

[0324] c. This application does not need to measure the height distance from the camera aperture to the feature point of the workpiece to be grasped and released. The reference depth difference and the running depth difference can be obtained through the drawings. The depth difference measurement steps are simpler, which effectively reduces the estimation error of the multi-plane hand-eye calibration and improves the accuracy of the hand-eye calibration.

[0325] Corresponding to the above-mentioned hand-eye calibration method, the present application embodiment also provides a hand-eye calibration device, such as Fig.12 As shown, the device comprises:

[0326] The homography matrix acquisition module 121 is used to acquire a first homography matrix and image principal point coordinates; wherein the image principal point coordinates are coordinates of the intersection of the optical axis and the pixel plane obtained by the camera calculated based on the first homography matrix and the second homography matrix; the first homography matrix is ​​a homography matrix of a pixel coordinate system determined by multiple images obtained by the camera taking multiple shots of the first reference object in the first scene and a physical coordinate system of the motion mechanism; the second homography matrix is ​​a homography matrix of a pixel coordinate system determined by multiple images obtained by the camera taking multiple shots of the second reference object in the second scene and a physical coordinate system of the motion mechanism; the first reference object and the second reference object are carried by the motion mechanism, or the camera is carried by the motion mechanism; the first distance is different from the second distance, the first distance is the distance between the first reference object and the aperture of the camera, and the second distance is the distance between the second reference object and the aperture of the camera;

[0327] A proportional factor calculation module 122 is used to calculate the ratio of the distance between the first reference object and the aperture of the camera to the distance between the workpiece to be captured and released and the aperture of the camera as a first proportional factor;

[0328] The conversion relationship determination module 123 is used to determine the conversion relationship between the first homography matrix and the target homography matrix as the first conversion relationship based on the first scale factor and the image principal point coordinates; wherein the target homography matrix is ​​the homography matrix of the pixel coordinate system corresponding to the workpiece to be picked up and placed and the physical coordinate system of the motion mechanism;

[0329] The homography matrix determining module 124 is configured to determine a target homography matrix based on the first conversion relationship and the first homography matrix.

[0330] In the technical solution provided in the embodiment of the present application, a first homography matrix and image principal point coordinates are obtained, wherein the image principal point coordinates are the coordinates of the intersection of the optical axis and the pixel plane calculated based on the first homography matrix and the second homography matrix; specifically, the first homography matrix is ​​a homography matrix of a pixel coordinate system and a physical coordinate system of a motion mechanism determined based on a plurality of images obtained by taking multiple shots of a first reference object by a camera in a first scene; the second homography matrix is ​​a homography matrix of a pixel coordinate system and a physical coordinate system of a motion mechanism determined based on a plurality of images obtained by taking multiple shots of a second reference object by a camera in a second scene; wherein the first reference object and the second reference object are carried on the motion mechanism, or the camera is carried on the motion mechanism; the difference between the first scene and the second scene is that the first reference object in the first scene is at a distance of 100 mm from the moving object. The distance between the apertures of the cameras (i.e., the first distance) is different from the distance between the aperture of the second reference object in the second scene and the camera (i.e., the second distance). Since the image principal point coordinates are the coordinates of the intersection of the optical axis of the camera and the pixel plane, the image principal point coordinates can be obtained based on the first homography matrix and the second homography matrix. After that, the ratio of the first distance to the distance between the workpiece to be captured and the aperture of the camera (i.e., the first proportional factor) is calculated. According to the first proportional factor and the image principal point coordinates, the conversion relationship between the first homography matrix and the target homography matrix (i.e., the first conversion relationship) can be obtained. Furthermore, based on the first conversion relationship and the first homography matrix, the target homography matrix can be determined, that is, the homography matrix of the pixel coordinate system corresponding to the workpiece to be captured and the physical coordinate system of the motion mechanism.

[0331] It can be understood that in the present application, the target homography matrix is ​​solved based on the first homography matrix and the first transformation relationship, and the image principal point coordinates used to solve the first transformation relationship are the real image principal point coordinates calculated based on the first homography matrix and the second homography matrix, rather than the theoretical center point of the pixel coordinate system as the image principal point coordinates. Therefore, the accuracy of determining the target homography matrix based on the hand-eye calibration method shown in this application is higher.

[0332] In some embodiments, the homography matrix acquisition module 121 is further used to acquire the second homography matrix;

[0333] The scale factor calculation module 122 includes:

[0334] A first determining unit, configured to determine, based on the first homography matrix and the second homography matrix, a conversion relationship between the first homography matrix and the second homography matrix as a second conversion relationship;

[0335] A second determining unit, configured to determine, based on a second conversion relationship, a ratio of the first distance to the second distance as a second proportional factor;

[0336] The third determining unit is used to determine the first proportional factor based on the second proportional factor, the height difference between the workpiece to be picked and placed and the first reference object, and the height difference between the first reference object and the second reference object.

[0337] In some embodiments, the third determination unit is specifically used to calculate the ratio of the height difference between the workpiece to be grasped and placed and the first reference object and the height difference between the first reference object and the second reference object as the depth difference ratio;

[0338] Based on the second scale factor and the depth difference ratio, a first scale factor is determined; wherein the first scale factor is positively correlated with the second scale factor and is positively correlated with the depth difference ratio.

[0339] In some embodiments, the first transformation relationship is represented in the form of a matrix, the linear component in the first transformation relationship is positively correlated with the first scale factor; the rotation component in the first transformation relationship is negatively correlated with the first scale factor and positively correlated with the value of the image principal point coordinates.

[0340] In some embodiments, the first determining unit is specifically configured to calculate an inverse matrix of either the first homography matrix or the second homography matrix, and calculate a product of the inverse matrix and the other as the second conversion relationship;

[0341] The second conversion relationship is expressed in the form of a matrix, and the second proportional factor is positively correlated with the main diagonal elements in the 2×2 matrix at the upper left corner of the second conversion relationship, and is negatively correlated with the sub-diagonal elements in the 2×2 matrix.

[0342] In some embodiments, the second conversion relationship is represented in the form of a matrix, and the second monocular determination is further used to:

[0343] Calculate the mean of the main diagonal of the 2×2 block matrix in the upper left corner of the matrix used to represent the second conversion relationship as the second scaling factor; or,

[0344] The square root of the determinant of the upper left 2×2 block matrix in the matrix used to represent the second conversion relationship is calculated as the second scaling factor.

[0345] In some embodiments, when the reference object is carried by the motion mechanism, the homography matrix between the pixel coordinate system in any scene and the physical coordinate system of the motion mechanism is obtained by the following method, including:

[0346] Control the motion mechanism to carry the reference object to move to a first photographing position, and control the camera to photograph the reference object, to obtain an image of the reference object at the first photographing position as a first image;

[0347] Determine pixel coordinates of at least one feature point of the reference object according to the first image as pixel coordinates of the feature point corresponding to the first image;

[0348] Determine whether the number of times the motion mechanism carries the reference object to move to the first photographing position reaches a preset number of times; if not, determine a new photographing position, use the new photographing position as the first photographing position, and repeat the above step of obtaining the pixel coordinates of the feature points corresponding to the first image;

[0349] If the preset number of times has been reached, determining a homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism according to the pixel coordinates of the feature points corresponding to each first image and each first photographing position;

[0350] The distances between each first photographing position and the aperture of the camera are the same in a preset direction, and the preset direction is the z-axis direction of the camera coordinate system.

[0351] In some embodiments, when the camera is carried on a motion mechanism, the homography matrix between the pixel coordinate system in any scene and the physical coordinate system of the motion mechanism is obtained by the following methods, including:

[0352] Control the motion mechanism to carry the camera to move to a second photographing position, and control the camera to photograph the reference object to obtain an image of the reference object at the second photographing position as a second image;

[0353] Determine pixel coordinates of at least one feature point of the reference object according to the second image as pixel coordinates of the feature point corresponding to the second image;

[0354] Determine whether the number of times the motion mechanism carries the camera to move to the second photographing position reaches a preset number; if not, determine a new photographing position, use the new photographing position as the second photographing position, and repeat the above step of obtaining the pixel coordinates of the feature points corresponding to the second image; if the preset number has been reached, determine the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism according to the pixel coordinates of the feature points corresponding to each second image and each second photographing position;

[0355] The distances between each second photographing position and the reference object are the same in a preset direction, and the preset direction is the z-axis direction of the camera coordinate system.

[0356] Corresponding to the above-mentioned hand-eye calibration method, the embodiment of the present application further provides a first motion mechanism system, such as Fig.13 As shown (understandably, Fig.13 The processor is not shown in the figure), the first motion mechanism system includes a processor, a motion mechanism 131, and a camera 132, and the camera 132 is arranged outside the motion mechanism 131.

[0357] The motion mechanism 131 is used to carry the first reference object to move in the first scene; and to carry the second reference object to move in the second scene;

[0358] The camera 132 is used to take multiple shots of the first reference object carried by the motion mechanism 131 in the first scene to obtain multiple third images; and to take multiple shots of the second reference object carried by the motion mechanism 131 in the second scene to obtain multiple fourth images; wherein the first distance is different from the second distance, the first distance is the distance between the first reference object and the aperture of the camera, and the second distance is the distance between the second reference object and the aperture of the camera;

[0359] A processor, configured to obtain a first homography matrix, and image principal point coordinates determined based on the first homography matrix and the second homography matrix; wherein the image principal point coordinates are the intersection coordinates of the optical axis of the camera and the pixel plane; the first homography matrix is ​​a homography matrix of a pixel coordinate system determined based on a plurality of third images and a physical coordinate system of a motion mechanism; and the second homography matrix is ​​a homography matrix of a pixel coordinate system determined based on a plurality of fourth images and a physical coordinate system of a motion mechanism;

[0360] The processor is also used to calculate the ratio of the first distance to the distance between the workpiece to be captured and the aperture of the camera as a first proportional factor; based on the first proportional factor and the coordinates of the principal point of the image, determine the conversion relationship between the first homography matrix and the target homography matrix as a first conversion relationship; based on the first conversion relationship and the first homography matrix, determine the target homography matrix; wherein the target homography matrix is ​​the homography matrix of the pixel coordinate system corresponding to the workpiece to be captured and the physical coordinate system of the motion mechanism.

[0361] In the technical solution provided by the embodiment of the present application, a first homography matrix and image principal point coordinates are obtained, wherein the image principal point coordinates are the coordinates of the intersection of the optical axis and the pixel plane calculated based on the first homography matrix and the second homography matrix; specifically, the first homography matrix is ​​a homography matrix of a pixel coordinate system and a physical coordinate system of a motion mechanism determined based on a plurality of images (i.e., a third image) obtained by taking multiple shots of a first reference object by a camera in a first scene; the second homography matrix is ​​a homography matrix of a pixel coordinate system and a physical coordinate system of a motion mechanism determined based on a plurality of images (i.e., a fourth image) obtained by taking multiple shots of a second reference object by a camera in a second scene; wherein, in the first scene, the first reference object is carried by the motion mechanism, and in the second scene, the second reference object is carried by the motion mechanism; the difference between the first scene and the second scene is that the first The distance between the first reference object in the scene and the aperture of the camera (i.e., the first distance) is different from the distance between the second reference object in the second scene and the aperture of the camera (i.e., the second distance). Since the coordinates of the principal point of the image are the coordinates of the intersection of the optical axis of the camera and the pixel plane, the coordinates of the principal point of the image can be obtained based on the first homography matrix and the second homography matrix. After that, the ratio of the first distance to the distance between the workpiece to be captured and the aperture of the camera (i.e., the first proportional factor) is calculated. According to the first proportional factor and the coordinates of the principal point of the image, the conversion relationship between the first homography matrix and the target homography matrix (i.e., the first conversion relationship) can be obtained. Furthermore, based on the first conversion relationship and the first homography matrix, the target homography matrix can be determined, that is, the homography matrix of the pixel coordinate system corresponding to the workpiece to be captured and the physical coordinate system of the motion mechanism.

[0362] It can be understood that in the present application, the target homography matrix is ​​solved based on the first homography matrix and the first transformation relationship, and the image principal point coordinates used to solve the first transformation relationship are the real image principal point coordinates calculated based on the first homography matrix and the second homography matrix, rather than the theoretical center point of the pixel coordinate system as the image principal point coordinates. Therefore, the accuracy of determining the target homography matrix based on the hand-eye calibration method shown in this application is higher.

[0363] In some embodiments, the processor is further configured to obtain a second homography matrix;

[0364] The calculating, as a first proportional factor, a ratio of a distance between the first reference object and the aperture of the camera to a distance between the workpiece to be captured and released and the aperture of the camera comprises:

[0365] Based on the first homography matrix and the second homography matrix, determining a conversion relationship between the first homography matrix and the second homography matrix as a second conversion relationship;

[0366] Based on the second conversion relationship, determining a ratio of the first distance to the second distance as a second proportional factor;

[0367] The first proportional factor is determined based on the second proportional factor, the height difference between the workpiece to be picked and placed and the first reference object, and the height difference between the first reference object and the second reference object.

[0368] In some embodiments, determining the first proportional factor based on the second proportional factor, the height difference between the workpiece to be picked and placed and the first reference object, and the height difference between the first reference object and the second reference object includes:

[0369] Calculate the ratio of the height difference between the workpiece to be grasped and placed and the first reference object to the height difference between the first reference object and the second reference object as the depth difference ratio;

[0370] Based on the second scale factor and the depth difference ratio, a first scale factor is determined; wherein the first scale factor is positively correlated with the second scale factor and is positively correlated with the depth difference ratio.

[0371] In some embodiments, the first transformation relationship is represented in the form of a matrix, the linear component in the first transformation relationship is positively correlated with the first scale factor; the rotation component in the first transformation relationship is negatively correlated with the first scale factor and positively correlated with the value of the image principal point coordinates.

[0372] In some embodiments, based on the first homography matrix and the second homography matrix, determining a conversion relationship between the first homography matrix and the second homography matrix as a second conversion relationship includes: calculating an inverse matrix of either the first homography matrix or the second homography matrix, and calculating a product of the inverse matrix and the other as the second conversion relationship;

[0373] The second conversion relationship is expressed in the form of a matrix, and the second proportional factor is positively correlated with the main diagonal elements in the 2×2 matrix at the upper left corner of the second conversion relationship, and negatively correlated with the sub-diagonal elements in the 2×2 matrix.

[0374] In some embodiments, the second conversion relationship is represented in the form of a matrix, and based on the second conversion relationship, a ratio of the first distance to the second distance is determined as a second proportional factor, including:

[0375] Calculate the mean of the main diagonal of the 2×2 block matrix in the upper left corner of the matrix used to represent the second conversion relationship as the second scaling factor; or,

[0376] The square root of the determinant of the upper left 2×2 block matrix in the matrix for representing the second conversion relationship is calculated as the second scale factor.

[0377] In some embodiments, the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism in any scene is obtained by the following method, including:

[0378] Control the motion mechanism to carry the reference object to move to a first photographing position, and control the camera to photograph the reference object, to obtain an image of the reference object at the first photographing position as a first image;

[0379] Determine pixel coordinates of at least one feature point of the reference object according to the first image as pixel coordinates of the feature point corresponding to the first image;

[0380] Determine whether the number of times the motion mechanism carries the reference object to move to the first photographing position reaches a preset number of times; if not, determine a new photographing position, use the new photographing position as the first photographing position, and repeat the above step of obtaining the pixel coordinates of the feature points corresponding to the first image;

[0381] If the preset number of times has been reached, determining a homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism according to the pixel coordinates of the feature points corresponding to each first image and each first photographing position;

[0382] The distances between each first photographing position and the aperture of the camera are the same in a preset direction, and the preset direction is the z-axis direction of the camera coordinate system.

[0383] Corresponding to the above-mentioned hand-eye calibration method, the embodiment of the present application further provides a second motion mechanism system, such as Fig.14 As shown (understandably, Fig.14 The processor is not shown in the figure), the second motion mechanism system includes a processor, a motion mechanism 141, and a camera 142, the camera 142 is carried by the motion mechanism 141,

[0384] The motion mechanism 141 is used to carry the camera 142 to move in the first scene; and to carry the camera 142 to move in the second scene;

[0385] The camera 142 is used to photograph the first reference object multiple times in the first scene while being carried by the moving mechanism 141 to obtain multiple fifth images; and to photograph the second reference object multiple times in the second scene while being carried by the moving mechanism 141 to obtain multiple sixth images; wherein the first distance is different from the second distance, the first distance is the distance between the first reference object and the aperture of the camera, and the second distance is the distance between the second reference object and the aperture of the camera;

[0386] A processor, configured to obtain a first homography matrix, and image principal point coordinates determined based on the first homography matrix and the second homography matrix; wherein the image principal point coordinates are the coordinates of the intersection of the optical axis of the camera and the pixel plane; the first homography matrix is ​​a homography matrix of a pixel coordinate system determined based on a plurality of fifth images and a physical coordinate system of a motion mechanism; and the second homography matrix is ​​a homography matrix of a pixel coordinate system determined based on a plurality of sixth images and a physical coordinate system of a motion mechanism;

[0387] The processor is also used to calculate the ratio of the first distance to the distance between the workpiece to be captured and the aperture of the camera as a first proportional factor; based on the first proportional factor and the coordinates of the principal point of the image, determine the conversion relationship between the first homography matrix and the target homography matrix as a first conversion relationship; based on the first conversion relationship and the first homography matrix, determine the target homography matrix; wherein the target homography matrix is ​​the homography matrix of the pixel coordinate system corresponding to the workpiece to be captured and the physical coordinate system of the motion mechanism.

[0388] In the technical solution provided in the embodiment of the present application, a first homography matrix and image principal point coordinates are obtained, wherein the image principal point coordinates are the intersection coordinates of the optical axis and the pixel plane calculated based on the first homography matrix and the second homography matrix; specifically, the first homography matrix is ​​a homography matrix of a pixel coordinate system and a physical coordinate system of the motion mechanism determined based on multiple images (i.e., the fifth image) obtained by multiple shots of the first reference object by the camera in the first scene; the second homography matrix is ​​a homography matrix of a pixel coordinate system and a physical coordinate system of the motion mechanism determined based on multiple images (i.e., the sixth image) obtained by multiple shots of the second reference object by the camera in the second scene; wherein the camera is carried on the motion mechanism. The difference between the first scene and the second scene is that the distance between the first reference object and the aperture of the camera in the first scene (i.e., the first distance) is different from the distance between the second reference object and the aperture of the camera in the second scene (i.e., the second distance). Since the coordinates of the image principal point are the coordinates of the intersection of the optical axis of the camera and the pixel plane, the coordinates of the image principal point can be solved based on the first homography matrix and the second homography matrix; then, the ratio of the first distance to the distance between the workpiece to be captured and the aperture of the camera (i.e., the first proportional factor) is calculated, and the conversion relationship between the first homography matrix and the target homography matrix (i.e., the first conversion relationship) can be solved based on the first proportional factor and the coordinates of the image principal point. Furthermore, the target homography matrix, i.e., the homography matrix of the pixel coordinate system corresponding to the workpiece to be captured and the physical coordinate system of the motion mechanism, can be determined based on the first conversion relationship and the first homography matrix.

[0389] It can be understood that in the present application, the target homography matrix is ​​solved based on the first homography matrix and the first transformation relationship, and the image principal point coordinates used to solve the first transformation relationship are the real image principal point coordinates calculated based on the first homography matrix and the second homography matrix, rather than the theoretical center point of the pixel coordinate system as the image principal point coordinates. Therefore, the accuracy of determining the target homography matrix based on the hand-eye calibration method shown in this application is higher.

[0390] In some embodiments, the processor is further configured to obtain a second homography matrix;

[0391] The calculating, as a first proportional factor, a ratio of a distance between the first reference object and the aperture of the camera to a distance between the workpiece to be captured and released and the aperture of the camera comprises:

[0392] Based on the first homography matrix and the second homography matrix, determining a conversion relationship between the first homography matrix and the second homography matrix as a second conversion relationship;

[0393] Based on the second conversion relationship, determine a ratio of the first distance to the second distance as a second proportional factor;

[0394] The first proportional factor is determined based on the second proportional factor, the height difference between the workpiece to be picked and placed and the first reference object, and the height difference between the first reference object and the second reference object.

[0395] In some embodiments, determining the first proportional factor based on the second proportional factor, the height difference between the workpiece to be picked and placed and the first reference object, and the height difference between the first reference object and the second reference object includes:

[0396] Calculating the ratio of the height difference between the workpiece to be grasped and placed and the first reference object to the height difference between the first reference object and the second reference object as the depth difference ratio;

[0397] Based on the second scale factor and the depth difference ratio, a first scale factor is determined; wherein the first scale factor is positively correlated with the second scale factor and is positively correlated with the depth difference ratio.

[0398] In some embodiments, the first transformation relationship is represented in the form of a matrix, the linear component in the first transformation relationship is positively correlated with the first scale factor; the rotation component in the first transformation relationship is negatively correlated with the first scale factor and positively correlated with the value of the image principal point coordinates.

[0399] In some embodiments, the determining, based on the first homography matrix and the second homography matrix, a conversion relationship between the first homography matrix and the second homography matrix as the second conversion relationship comprises: calculating an inverse matrix of either the first homography matrix or the second homography matrix, and calculating a product of the inverse matrix and the other as the second conversion relationship;

[0400] The second conversion relationship is represented in the form of a matrix, and the second proportional factor is positively correlated with the main diagonal elements in the 2×2 matrix at the upper left corner of the second conversion relationship, and negatively correlated with the sub-diagonal elements in the 2×2 matrix.

[0401] In some embodiments, the second conversion relationship is represented in the form of a matrix, and the determining, based on the second conversion relationship, of the ratio of the first distance to the second distance as the second proportional factor includes:

[0402] Calculate the mean of the main diagonal of the 2×2 block matrix in the upper left corner of the matrix used to represent the second conversion relationship as the second scaling factor; or,

[0403] The square root of the determinant of the upper left 2×2 block matrix in the matrix for representing the second conversion relationship is calculated as the second scale factor.

[0404] In some embodiments, the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism in any scene is obtained by the following method, including:

[0405] Control the motion mechanism to carry the camera to move to a second photographing position, and control the camera to photograph the reference object to obtain an image of the reference object at the second photographing position as a second image;

[0406] Determine pixel coordinates of at least one feature point of the reference object according to the second image as pixel coordinates of the feature point corresponding to the second image;

[0407] Determine whether the number of times the motion mechanism carries the camera to move to the second photographing position reaches a preset number of times; if not, determine a new photographing position, use the new photographing position as the second photographing position, and repeat the above steps of obtaining the pixel coordinates of the feature points corresponding to the second image;

[0408] If the preset number of times has been reached, determining a homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism according to the pixel coordinates of the feature points corresponding to each second image and each second photographing position;

[0409] The distances between each first photographing position and the reference object are the same in a preset direction, and the preset direction is the z-axis direction of the camera coordinate system.

[0410] The present application also provides an electronic device, such as Fig.15 As shown, including:

[0411] A memory 151, for storing computer programs;

[0412] The processor 152 is used to implement any of the above-mentioned hand-eye calibration methods when executing the program stored in the memory 151.

[0413] In addition, the electronic device may further include a communication bus and / or a communication interface, and the processor 152, the communication interface, and the memory 151 communicate with each other through the communication bus. The communication bus mentioned in the electronic device may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The communication bus may be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0414] The communication interface is used for communication between the above electronic device and other devices.

[0415] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0416] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0417] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned hand-eye calibration methods are implemented.

[0418] In another embodiment provided in the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any hand-eye calibration method in the above embodiments.

[0419] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.

[0420] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0421] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, the first motion mechanism system, the second motion mechanism system, the electronic device, the storage medium, and the computer program product, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0422] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A hand-eye calibration method, characterized in that: The method comprises: Obtain a first homography matrix and image principal point coordinates; wherein the image principal point coordinates are coordinates of the intersection of the optical axis of the camera and the pixel plane calculated based on the first homography matrix and the second homography matrix; the first homography matrix is ​​a homography matrix of a pixel coordinate system determined by multiple images obtained by the camera taking multiple shots of the first reference object in the first scene and a physical coordinate system of the motion mechanism; the second homography matrix is ​​a homography matrix of a pixel coordinate system determined by multiple images obtained by the camera taking multiple shots of the second reference object in the second scene and a physical coordinate system of the motion mechanism; the first reference object and the second reference object are carried by the motion mechanism, or the camera is carried by the motion mechanism; the first distance is different from the second distance, the first distance is the distance between the first reference object and the aperture of the camera, and the second distance is the distance between the second reference object and the aperture of the camera; Calculating a ratio of the first distance to the distance between the workpiece to be captured and released and the aperture of the camera as a first proportional factor; Based on the first scale factor and the coordinates of the principal point of the image, a conversion relationship between the first homography matrix and a target homography matrix is ​​determined as a first conversion relationship; wherein the target homography matrix is ​​a homography matrix between the pixel coordinate system corresponding to the workpiece to be grasped and released and the physical coordinate system of the motion mechanism; The target homography matrix is ​​determined based on the first conversion relationship and the first homography matrix.

2. The method according to claim 1, characterized in that The method further includes obtaining the second homography matrix; The calculating, as a first proportional factor, a ratio of a distance between the first reference object and the aperture of the camera to a distance between the workpiece to be captured and released and the aperture of the camera includes: Based on the first homography matrix and the second homography matrix, determining a conversion relationship between the first homography matrix and the second homography matrix as a second conversion relationship; Based on the second conversion relationship, determine a ratio of the first distance to the second distance as a second proportional factor; The first scale factor is determined based on the second scale factor, a height difference between the workpiece to be picked and placed and the first reference object, and a height difference between the first reference object and the second reference object.

3. The method according to claim 2, characterized in that The determining of the first proportional factor based on the second proportional factor, the height difference between the workpiece to be grasped and placed and the first reference object, and the height difference between the first reference object and the second reference object includes: Calculating a ratio of a height difference between the workpiece to be grasped and placed and the first reference object to a height difference between the first reference object and the second reference object as a depth difference ratio; The first scale factor is determined based on the second scale factor and the depth difference ratio; wherein the first scale factor is positively correlated with the second scale factor and is positively correlated with the depth difference ratio.

4. The method according to claim 1, characterized in that: The first transformation relationship is expressed in the form of a matrix, the linear component in the first transformation relationship is positively correlated with the first scale factor; the rotation component in the first transformation relationship is negatively correlated with the first scale factor and positively correlated with the value of the image principal point coordinate.

5. The method according to claim 2, characterized in that: The determining, based on the first homography matrix and the second homography matrix, a conversion relationship between the first homography matrix and the second homography matrix as a second conversion relationship comprises: calculating an inverse matrix of either the first homography matrix or the second homography matrix, and calculating a product of the inverse matrix and the other as the second conversion relationship; The second conversion relationship is represented in the form of a matrix, and the second proportional factor is positively correlated with the main diagonal elements in the 2×2 matrix at the upper left corner of the second conversion relationship, and negatively correlated with the sub-diagonal elements in the 2×2 matrix.

6. The method according to claim 2, characterized in that The second conversion relationship is represented in the form of a matrix, and the ratio of the first distance to the second distance is determined based on the second conversion relationship as a second proportional factor, including: Calculate the mean of the main diagonal of the 2×2 block matrix in the upper left corner of the matrix used to represent the second conversion relationship as the second proportional factor; or, The square root of the determinant of the upper left 2×2 block matrix in the matrix used to represent the second conversion relationship is calculated as the second scaling factor.

7. The method according to claim 1, characterized in that When the reference object is carried by the motion mechanism, the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism in any scene is obtained by the following method, including: Controlling the motion mechanism to carry the reference object to a first photographing position, and controlling the camera to photograph the reference object, to obtain an image of the reference object at the first photographing position as a first image; Determine pixel coordinates of at least one feature point of the reference object according to the first image as pixel coordinates of the feature point corresponding to the first image; Determine whether the number of times the motion mechanism carries the reference object to move to the first photographing position reaches a preset number, and if not, determine a new photographing position, use the new photographing position as the first photographing position, and repeat the above step of obtaining the pixel coordinates of the feature points corresponding to the first image; If the preset number of times has been reached, determining a homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism according to the pixel coordinates of the feature points corresponding to each of the first images and each of the first photographing positions; The distances between each of the first photographing positions and the aperture of the camera are the same in a preset direction, and the preset direction is the z-axis direction of the camera coordinate system.

8. The method according to claim 1, characterized in that: When the camera is carried on the motion mechanism, the homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism in any scene is obtained by the following method, including: Controlling the motion mechanism to carry the camera to move to a second photographing position, and controlling the camera to photograph the reference object, to obtain an image of the reference object at the second photographing position as a second image; Determine pixel coordinates of at least one feature point of the reference object according to the second image as pixel coordinates of the feature point corresponding to the second image; Determine whether the number of times the motion mechanism carries the camera to move to the second photographing position reaches a preset number, and if not, determine a new photographing position, use the new photographing position as the second photographing position, and repeat the above step of obtaining the pixel coordinates of the feature points corresponding to the second image; If the preset number of times has been reached, determining a homography matrix between the pixel coordinate system and the physical coordinate system of the motion mechanism according to the pixel coordinates of the feature points corresponding to each of the second images and each of the second photographing positions; Wherein, the distances between each of the second photographing positions and the reference object are the same in a preset direction, and the preset direction is the z-axis direction of the camera coordinate system.

9. A hand-eye calibration device, characterized in that: The device comprises: A homography matrix acquisition module, used to acquire a first homography matrix and image principal point coordinates; wherein the image principal point coordinates are the intersection coordinates of the optical axis of the camera and the pixel plane calculated based on the first homography matrix and the second homography matrix; the first homography matrix is ​​a homography matrix of a pixel coordinate system determined by multiple images obtained by the camera taking multiple shots of a first reference object in a first scene and a physical coordinate system of a motion mechanism; the second homography matrix is ​​a homography matrix of a pixel coordinate system determined by multiple images obtained by the camera taking multiple shots of a second reference object in a second scene and a physical coordinate system of a motion mechanism; the first reference object and the second reference object are carried by the motion mechanism, or the camera is carried by the motion mechanism; the first distance is different from the second distance, the first distance is the distance between the first reference object and the aperture of the camera, and the second distance is the distance between the second reference object and the aperture of the camera; A proportional factor calculation module, used for calculating the ratio of the distance between the first reference object and the aperture of the camera to the distance between the workpiece to be captured and released and the aperture of the camera as a first proportional factor; A transformation relationship determination module, used to determine, based on the first scale factor and the image principal point coordinates, a transformation relationship between the first homography matrix and a target homography matrix as a first transformation relationship; wherein the target homography matrix is ​​a homography matrix between the pixel coordinate system corresponding to the workpiece to be grasped and released and the physical coordinate system of the motion mechanism; A homography matrix determination module is used to determine the target homography matrix based on the first conversion relationship and the homography matrix corresponding to the first reference object.

10. A first motion mechanism system, characterized in that: The invention comprises a processor, a motion mechanism, and a camera, wherein the camera is arranged outside the motion mechanism. The motion mechanism is used to carry the first reference object to move in the first scene; and moving with a second reference object in a second scene; The camera is used to take multiple shots of the first reference object carried by the motion mechanism in the first scene to obtain multiple third images; and photographing the second reference object carried by the motion mechanism multiple times in the second scene to obtain multiple fourth images; wherein the first distance is different from the second distance, the first distance is the distance between the first reference object and the aperture of the camera, and the second distance is the distance between the second reference object and the aperture of the camera; The processor is used to obtain a first homography matrix, and image principal point coordinates determined based on the first homography matrix and the second homography matrix; wherein the image principal point coordinates are the intersection coordinates of the optical axis of the camera and the pixel plane; the first homography matrix is ​​a homography matrix of a pixel coordinate system determined based on the plurality of third images and a physical coordinate system of a motion mechanism; the second homography matrix is ​​a homography matrix of a pixel coordinate system determined based on the plurality of fourth images and a physical coordinate system of a motion mechanism; The processor is further used to calculate the ratio of the first distance to the distance between the workpiece to be captured and the aperture of the camera as a first proportional factor; based on the first proportional factor and the coordinates of the principal point of the image, determine the transformation relationship between the first homography matrix and the target homography matrix as a first transformation relationship; based on the first transformation relationship and the first homography matrix, determine the target homography matrix; wherein the target homography matrix is ​​the homography matrix of the pixel coordinate system corresponding to the workpiece to be captured and the physical coordinate system of the motion mechanism.

11. A second motion mechanism system, characterized in that: The invention comprises a processor, a motion mechanism, and a camera, wherein the camera is carried by the motion mechanism. The motion mechanism is used to carry the camera for motion in the first scene; and to carry the camera for motion in the second scene; The camera is used to photograph the first reference object multiple times in the first scene while being carried by the motion mechanism to obtain multiple fifth images; and to photograph the second reference object multiple times in the second scene while being carried by the motion mechanism to obtain multiple sixth images; wherein the first distance is different from the second distance, the first distance is the distance between the first reference object and the aperture of the camera, and the second distance is the distance between the second reference object and the aperture of the camera; The processor is used to obtain a first homography matrix, and image principal point coordinates determined based on the first homography matrix and the second homography matrix; wherein the image principal point coordinates are the intersection coordinates of the optical axis of the camera and the pixel plane; the first homography matrix is ​​a homography matrix of a pixel coordinate system determined based on the multiple fifth images and a physical coordinate system of a motion mechanism; the second homography matrix is ​​a homography matrix of a pixel coordinate system determined based on the multiple sixth images and a physical coordinate system of a motion mechanism; The processor is further used to calculate the ratio of the first distance to the distance between the workpiece to be captured and the aperture of the camera as a first proportional factor; based on the first proportional factor and the coordinates of the principal point of the image, determine the transformation relationship between the first homography matrix and the target homography matrix as a first transformation relationship; based on the first transformation relationship and the first homography matrix, determine the target homography matrix; wherein the target homography matrix is ​​the homography matrix of the pixel coordinate system corresponding to the workpiece to be captured and the physical coordinate system of the motion mechanism.

Images